The experimental evaluation of a public policy uses the same principle as clinical trials in medicine: two groups are randomly selected from the same population, one benefiting from the policy while the other serves as a control. The impact of the policy is measured by comparing the two groups: beneficiaries and non-beneficiaries. In practice, there are several types of randomized interventions that allow experiments to be adapted to suit the policy being evaluated.
Quasi-experimental methods use pre-existing observational data to estimate the effect of a public policy by attempting to get closer to experimental conditions. Quasi-experimental methods are useful because they mobilize fewer resources and avoid the ethical, political and behavioral problems that can result from randomized allocation.
The French context
Experimental methods have long been used by US regulatory agencies and international bodies such as the World Bank. In France, the development of these methods to aid public decision-making is much more recent and modest. The progress of experiments appears to be limited by various factors, including the political and ethical problems raised in French culture by the random selection of samples and the accessibility of individual data. The political limits have been greatly reduced since the Constitutional Law of 28 March 2003 allowing derogations from the principle of equal treatment for experimentation purposes. The ethical pitfalls have also been reduced by universities’ use of ethical committees to evaluate experiment plans. However, another obstacle is that few public decision-makers are familiar with the different design possibilities and new statistical techniques.
Continuous learning
Traditional policy evaluation tools are largely based on ex ante methods, such as cost-benefit analysis, using assumptions about the expected effects on the economy. In practice, ex anteevaluation is complicated, especially for social policies (assistance to return to work, encouragement to study, response to subsidies, etc.) whose expected effects depend on the human factor. It is difficult to anticipate beneficiaries’ reactions because they vary according to characteristics specific to each individual.
Consequently, it is important to experiment with proposed measures by first applying them to small groups. Experimentation must be seen as a continuous learning process with both ex ante and ex post evaluations which are revised throughout the life cycle of the policy studied.
Ex post evaluation has been used extensively in the areas of work, education and development, where a thorough understanding of behavior is critical. Ex ante evaluation, meanwhile, has been widely applied in areas such as transport, energy and the environment, where policies can have significant long-term macroeconomic effects and the value of ex post approaches is less clear. For example, it is hard to imagine how to simulate long-term impacts for a planet with or without a climate policy. However, recent ex post studies may help us better understand the long-term impacts of climate policy by exploring the links between climate policy and greenhouse gas emissions, or between temperature and climate damage.
Economist as plumber
The empirical approach in economics, and especially the evaluation of public policies, is not limited to experimental and quasi-experimental methods. Structural econometric methods, which use behavioral models to predict the consequences of public interventions, are also very useful. Structural models have also benefited from the credibility revolution, as their ex ante or model-based predictions are increasingly compared with ex post observations.
These advances have contributed to the evolution of the economist who advises, assists and evaluates public policy. For a long time, economists guided public policy mainly by proposing robust and rigorous principles to evaluate policy decisions. Now, economists have become involved in the details of the implementation of policies, working upstream with decision-makers to design the trial, implementation and evaluation of feasible and credible policy alternatives. After the advent of the economist as engineer, we see the emergence of the economist as plumber (Duflo 2017).
Extract of the TSE Mag#18 Winter 2018
