Once upon a time, ‘evidence’ was widely confused with randomised controlled trials (RCTs). The latter were held to be the paragon of rigour, and other forms of evidence, though not necessarily dismissed, were certainly devalued. Whenever observation, experience, reasoning, or even statistical analysis suggested one thing and some RCT another, there was a tendency to assume that the RCT got it right.
If not with evidence tout court, RCTs became synonymous with ‘rigorous evidence’ or ‘hard evidence’. The privileged status of RCTs was expressed in statements such as “all too often development policy is based on fads, and randomised evaluations could allow it to be based on evidence” (Duflo and Kremer 2003).
Today, the fever has come down and most economists would take a more nuanced view of what constitutes evidence. We also have a better understanding of the uses and misuses of RCTs. For this we owe a great deal to Angus Deaton and Nancy Cartwright, as well to many others who have contributed to the debate on both sides. Deaton and Cartwright (2018) present a particularly lucid and insightful assessment of the logic, power, and limits of RCTs.
Related article: Single studies cannot inform policy-making
As the authors argue, what ultimately matters is not just evidence (broadly defined) but understanding. That, it seems to me, is a far-reaching insight, because understanding can grow in many ways. Evidence certainly contributes to “the process of cumulative understanding”, as Cartwright and Deaton call it, but so do, say, personal experience and public debate. Even reading fiction can enhance our understanding of real-world matters – John Le Carré’s novels, for instance, are no less enlightening (and more fun to read!) than the best RCTs when it comes to political economy.
Just as the meaning of evidence in the term ‘evidence-based policy’ has gone through some healthy revision, it seems to me that the relation between evidence and policy needs further thought. Evidence is about facts, policy is a political decision – in what sense is it to be ‘based’ on evidence?
An extremist view, in this regard, would be that RCTs – or evidence – can be held as an impartial arbiter of public policy. As one commentator puts it: “The proof of the pudding is in the eating: if something works, it is worth doing” (Rodrik 2009). It is easy to see that this proposition, taken literally, is untenable. Even if something works, something else may work even better.
More importantly, whether or not something should be done involves value judgements that no RCT, or for that matter no evidence, can settle on its own.
More importantly, whether or not something should be done involves value judgements that no RCT, or for that matter no evidence, can settle on its own (as Deaton and Cartwright aptly note, “what works is not equivalent to what should be”). I do not think that the leading lights of the RCT school, so to speak, subscribe to this extremist view, but the trap is worth mentioning because it is easy to fall into it.
Aside from this glaring trap, however, it is important to avoid a myopic view of the long bridge that takes us from evidence to policy. Let me share some thoughts on this, with reference to development economics, based on recent experiences of involvement with social policy in India. For several reasons, I believe that economists ought to be cautious in offering ‘evidence-based’ policy advice in this field.
First, as noted earlier, giving advice requires some serious reflection about values and objectives. No value judgements are required to conduct an RCT aimed at examining whether adding eggs in school meals helps to enhance pupil attendance or child nutrition. But advocating the inclusion of eggs in school meals is a very different ballgame.
To conduct an RCT examining whether eggs in school meals help child nutrition, can be devoid of politics. But, advocating the inclusion of eggs in school meals, based on an RCT’s results cannot be apolitical. | Photo courtesy: Akshaya Patra, Pixabay
It means dealing with the arguments of upper-caste vegetarian lobbies (eggs are considered non-vegetarian in India) and animal-rights activists, aside from those of the finance ministry, the education department, and teachers’ unions. Commercial interests, too, are likely to come into play as the poultry business eyes big contracts.
The debate can easily get very charged. Any ‘advice’ offered in this charged atmosphere may have serious repercussions, good or bad. A worst-case but not uncommon scenario is that a piece of advice turns out to be counterproductive. Dealing with these choices, conflicts, and dilemmas require much more than ‘evidence’.
Related article: The ‘science-policy-practitioner’ divide
Quite often, public policy is flawed due to wrong priorities rather than lack of evidence. Just to give one example, the central government’s contribution to social security pensions in India has stagnated at an abysmal Rs 200 (barely US$ 3) per month since 2006, even as the salaries and pensions of government employees went up by leaps and bounds.
Most economists, I believe, would support an increase in the pension amounts. In fact, 60 eminent economists wrote to the finance minister in December 2017, recommending a modest increase (from Rs 200 to Rs 500 per month), also backed by some of the minister’s key advisors.
Yet the 2018-19 Union budget ignored this recommendation and upheld the measly norm of Rs 200 per month. More evidence on pension schemes could certainly help to overcome this resistance, but the crux of the matter is ethical and political.
Second, the advice one might give on a particular issue generally depends on who one advises. The general presumption, in development economics, is that in matters of public policy the researcher is advising the government (indeed, the words ‘policy’ and ‘government policy’ are more or less interchangeable in much of that literature). This is understandable, because government policy does look – rightly or wrongly – like the most powerful lever one can activate to change the world.
But there are also other means of achieving practical change. And even government policy, in a country like India, is the outcome of democratic processes that involve not only the government but also a range of non-government institutions. There is no reason for economists to restrict their advice to the government.
Even advising the government calls for reflecting on one’s position in the system. For instance, including eggs in midday meals may be good for the government in terms of its stated objective of improving child nutrition, but bad in terms of re-election prospects (for example, because of the power of vegetarian lobbies).
In practice, however, conflicts of objectives may be rife, both within the government as well as between the government and the researchers.
In the world-view of mainstream economics, the ‘policymakers’ have a ‘social welfare function’ that integrates these diverse objectives, and policy recommendations are essentially geared to this objective function. In practice, however, conflicts of objectives may be rife, both within the government as well as between the government and the researchers.
Another example may help to convey how our best advice often depends on who we are advising and in what capacity. There is a lively debate, in India, on the pros and cons of cash transfers vis-à-vis food transfers through the public distribution system (PDS). It is easy to imagine an economist giving the following sort of advice to the government:
Our RCT shows that people essentially treat food transfers as an implicit cash transfer. Considering the high transaction costs of food subsidies, a transition to cash transfers seems advisable. Since the banking infrastructure is still weak in many areas, the transition should be gradual, starting with urban areas and the more developed districts. Cash transfers should be indexed, so that their real value is not eroded by inflation.
Related article: Do evaluations actually serve a useful purpose?
It is possible, however, that based on the same research a person who addresses herself to poor people (not to the government) would give the following – very different – advice:
The government is planning to replace food transfers with cash transfers. You should resist this at all cost. Our work shows that the banking infrastructure is not ready. If you get cash instead of food, you will have to travel long distances and queue for hours to collect your meagre benefits. The government is paying no attention to this. Also, as you know from the experience of social security pensions, you cannot trust the government to index cash transfers, or even to ensure timely payments. Right now, food subsidies are a lifeline for many of you, do not give them up unless and until the government proves that it has something better to offer.
In the first instance, the researcher positions herself as a government advisor who has (or thinks she has) an opportunity to influence the modalities of cash transfers. In the second, he or she looks at the same problem from the point of view of people who are powerless to influence the way things are done, and have a sobering experience of how the government works. The evidence is the same, but the advice that flows from it is contingent on where we stand.
Policy design is analogous to ‘plumbing’ in so far as it requires tinkering with nuts and bolts until the contraption works.
Third, policy advice often requires going into difficult operational, legal, ethical, political, and other issues that need not be familiar to development economists. Just to pursue the first angle, a development scheme can stand or fall on minor details such as whether the monthly cheques are signed by the district magistrate or village head, whether a government-sponsored latrine has one pit or two pits, or whether biometric authentication is necessary to apply for benefits.
This point is well discussed in a recent paper by Esther Duflo (2017), who rightly stresses the importance of ‘detail’ in policy design. Duflo argues that policy design is analogous to “plumbing” in so far as it requires tinkering with nuts and bolts until the contraption works.
So far so good. Where I would differ with Duflo is the idea that economists are generally good plumbers. She argues that they are well-qualified, by virtue of their disciplinary training, to influence if not take charge of policy design. I am not so sure. Economists certainly have much to contribute, but in many cases, they have no special competence on the relevant details.
Local administrators and MNREGA workers tend to be well-informed, being in the thick of the implementation process. In academic seminars on this subject, by contrast, the ignorance of operational matters is epic. | Photo courtesy: Bishnu Sarangi, Pixabay
This has often struck me in the context of discussions of India’s MNREGA (Mahatma Gandhi National Rural Employment Guarantee Act), a complex programme that makes exacting demands on the administrative machinery. In my experience, it is possible to have enlightening discussions on the operational details with local administrators, village-level functionaries, and even MNREGA workers.
For one thing, they tend to be well-informed, being in the thick of the implementation process. In academic seminars on this subject, by contrast, the ignorance of operational matters is epic. Economists do have valuable expert knowledge (for example, of the international experience with public works programmes), and nothing prevents them from learning about operational details as well, but it is not something for which they are particularly well-trained.
Sometimes, in fact, their training puts them at a disadvantage. Returning, for instance, to the issue of eggs in school meals, one of the critical operational issues is whether eggs should be supplied in bulk by a private contractor or bought locally by the school cook. There is much to be said for local purchase, but private contractors often lobby really hard for centralised procurement.
Economists are not well-placed to realise the anti-social aspect of corporate interests, partly because of the cosy relationship between the corporate sector and academia.
Despite Adam Smith’s historic warning that policy proposals from business sources should be examined “not only with the most scrupulous, but with the most suspicious attention”, economists are not well-placed to recognise the anti-social aspect of corporate interests, partly because of their theoretical training and partly because of the cosy relationship between the corporate sector and academia. Even if they have valuable expertise on other aspects of this procurement issue, their advice may need a pinch of salt.
Fourth, policy advice can have all sorts of unintended consequences. If you advise A, you may get a.A (a fraction of A), or A’ (a variant of A), or B (or an alternative to A), or even -A (the opposite of A). In the first case, should one actually advise (1/a).A, in the hope of getting A? That is, indeed, a common tactic among activists – ask for the loaf, settle for a slice.
To put this in a different way (familiar to economists), policy advice can be seen as a kind of ‘game’, where the outcome depends on the strategies of all players, and the players must take each other’s strategies into account. Or perhaps it would be unprincipled to look at it that way, and economists should just give the advice they think is right, irrespective of the consequences? It is hard to tell.
Related article: Integrating the qualitative and quantitative
Here again, the point can be illustrated with reference to the issue of eggs in school meals. In some Indian states with a strong vegetarian lobby, the government responded to the demand for eggs with milk. But milk has a short shelf life, and is not always safe. It is quite possible that in these states, it would have been better to argue for a fruit than for an egg, even if eggs are better in principle.
These caveats have varying relevance, depending on the context. If an RCT suggests that the colour version of an application form is more effective than the black-and-white version, perhaps some useful advice can be given without worrying about these complications. They do matter, however, whenever economic research deals with issues such as teacher incentives, alternative payment systems for pensioners, the grouping of students in schools, the case for cash versus in-kind transfers, the design of vaccination programmes, and so on. In general, there is no escape from the fact that research is a scientific act, and policy advice a political act.
RCTs cannot de-politicise public policy, because policy decisions would still require values and objectives, which are political by nature.
Some RCT enthusiasts may respond that the purpose of RCTs is precisely to de-politicise public policy and shield it from the pernicious influence of ideology, fads and such. Perhaps they would prefer public policy to be left to experts, sitting across the table with the policymakers. But going to bed with the government is hardly an apolitical step. In any case, this approach would not de-politicise public policy, because policy decisions would still require values and objectives, which are political by nature. What it would do is to shield these value judgements from public scrutiny and debate.
That is the trend in India today, with more and more policy decisions being taken behind the ramparts of the finance ministry, the Prime Minister’s Office, and NITI Aayog (handsomely peppered with “consultants” from international agencies and the corporate sector). It does not strike me as a healthy trend. For one thing, most of the social benefits that poor people have won in India over the years came their way in the context of electoral politics. De-politicising social policy – if such a thing were possible – would deprive them of this vital foothold.
In short, I feel that economists need to be cautious and modest when it comes to giving policy advice, let alone getting actively involved in ‘policy design’. Their expertise and research can certainly contribute to more informed policy discussions and public debates. But if they give advice, it is best done as concerned citizens rather than plumber-like economists, in collaboration with others from different disciplines and walks of life. In the field of social policy, at least, I see no reason to privilege the advice of economists.
None of this detracts from the value of RCTs (correctly understood), or from the case for evidence-based policy. If the idea is to bring more evidence to bear on public policy, there is much to be said for it. This endeavour, however, is likely to be all the more useful if we bear in mind that evidence involves more than RCTs, understanding more than evidence, and policy more than understanding.
This was originally published on The Wire.
