Impact Evaluation. The case of the European project Woodie
1. Outline of Impact Evaluation
1.1. Introduction and key concepts
Impact Evaluation (IE) is a relatively recent field of research that inherits the knowledge of different disciplines such as social science, sociology, statistics and econometrics.
It is a specific type of evaluation used for:
- demonstrating the impact of a policy by measuring changes in short term, intermediate and long-term outcomes
- determining whether changes in outcomes can be attributed to the policy
- comparing relative impacts of policies with different components
- identifying the relative cost-benefit or cost-effectiveness of a policy
The focus on changes brought about by a policy is at the core of the IE, which intervenes after the implementation phase of a policy as shown in the picture below.
Here we are discussing about the impact evaluation to a policy with a specific reference to the work performed on the whistleblowing protection and open data policy in the framework of the European project titled “Woodie – Whistleblowing open data impact - An Implementation and impact assessment (see following paragraphs of UD 7). However, it has to be remarked that the IE can be undertaken on policy or interventions at different scale such as a project, a local intervention or even a wider program performed by an organization, e.g. by Non-governmental organizations carrying out civil society strengthening program in less developed countries.
The focus on impact makes crucial to understand clearly its meaning. The concept of impact is very challenging and complex by nature. How impact is defined necessarily determines the scope and content of the impact evaluation because different definitions put emphasis on different aspects of impact, imply different concept of causality (what produces the impact) and how to measure the impact (evaluation design).
In literature, there are several definitions of impact provided by different institutions and organizations, each of them with strengths and weaknesses. The most widely shared definition is that of the OECD-DAC related to development interventions, which defines impact as “the positive and negative, primary and secondary effects produced by a development intervention, directly or indirectly, intended or unintended” (Organisation for Economic Co-operation and Development – Development Assistance Committee Glossary, 2010).
The focus on impact makes crucial to understand clearly its meaning. The concept of impact is very challenging and complex by nature. How impact is defined necessarily determines the scope and content of the impact evaluation because different definitions put emphasis on different aspects of impact, imply different concept of causality (what produces the impact) and how to measure the impact (evaluation design).
In literature, there are several definitions of impact provided by different institutions and organizations, each of them with strengths and weaknesses. The most widely shared definition is that of the OECD-DAC related to development interventions, which defines impact as “the positive and negative, primary and secondary effects produced by a development intervention, directly or indirectly, intended or unintended” (Organisation for Economic Co-operation and Development – Development Assistance Committee Glossary, 2010).
This definition has gained wider recognition as it:
- stresses the importance to search any effect, not only the expected ones;
- recognises that effects may be positive or negative;
- recognises that effects of interests are produced (or caused) by the intervention;
- suggests the possibility of different kind of links between all kinds of development intervention (project, program or policy) and effect;
- focuses on the longer-term effects of the development interventions.
Assuming this broad definition of impact, the definition of impact evaluation that best fits with it includes:
The primary purpose of IE is to determine whether a policy (or program, project, intervention according to the specific case) has an impact on a few key outcomes and, more specifically, to quantify how large that impact is. It puts emphasis on long-term effects as well as on the search of cause and effect relationships between policy and results. It also acknowledges the logic of attribution (= the idea that a change is solely due to the policy or intervention under investigation) as much as contribution (= the idea that the influence of the policy or intervention investigated is just one of many factors which contribute to a change). This latter aspect is notably relevant for policies that address issues with a complex and multidimensional nature or are multi-actor as in the case of the policies addressed by the project Woodie or in specific circumstances including impact evaluation applied to development interventions.
- evaluating the positive and negative, primary and secondary long term effects on final beneficiaries that result from a policy or an intervention;
- assessing the direct and indirect causal contribution claims of the policy to such effects whether intended or unintended;
- explaining how policy contribute to an effect so that lessons can be learnt.
The primary purpose of IE is to determine whether a policy (or program, project, intervention according to the specific case) has an impact on a few key outcomes and, more specifically, to quantify how large that impact is. It puts emphasis on long-term effects as well as on the search of cause and effect relationships between policy and results. It also acknowledges the logic of attribution (= the idea that a change is solely due to the policy or intervention under investigation) as much as contribution (= the idea that the influence of the policy or intervention investigated is just one of many factors which contribute to a change). This latter aspect is notably relevant for policies that address issues with a complex and multidimensional nature or are multi-actor as in the case of the policies addressed by the project Woodie or in specific circumstances including impact evaluation applied to development interventions.