The increase in rigorous impact evaluations (RIEs) over the past twenty years has been accompanied by a growing interest among development actors worldwide. (See our blog post on what RIEs are and their importance for evidence-based learning in international development). In Germany too, interest in RIEs has grown in recent years.
1. German Development Cooperation (GDC)
By way of background, Germany is the second largest provider of Development Cooperation in the OECD’s Development Assistance Committee, after the United States. In 2022, the country’s total Official Development Assistance amounted to USD 35 billion, or 0.83% of its Gross National Income (OECD, 2022). German Development Cooperation consists mainly of the German Federal Ministry for Economic Cooperation and Development (BMZ), which coordinates German Development Cooperation; the Deutsche Gesellschaft für Internationale Zusammenarbeit GmbH (GIZ), the main German development agency for technical cooperation; and the KfW, Germany’s development bank for financial cooperation.
1.1. Status-Quo: Unsystematic Implementation and Use of RIEs in GDC
Despite growing interest in and use of RIEs in Germany in recent years, there is currently no systematic approach or guidance for when to conduct RIEs in German Development Cooperation. The number of Rigorous Impact Evaluations (RIEs) in the portfolio of German development projects remains rather low and unsystematic, and evidence from RIEs is often not used strategically in project design or strategic decision-making (Krämer et al., 2021).
In its 2021 Evaluation Guidelines (available here in German), the BMZ has, however, formally endorsed the role of RIEs, paving the way for a more systematic integration of RIEs into policy-making.
1.2. Key Barriers to Systematic Implementation and Use of RIEs in GDC
The BMZ also funded a research and advisory project conducted by the German Institute for Development Evaluation (DEval) to increase the application of RIE methods and the use of evidence. This research culminated in a report by DEval (Krämer et al, 2021), which reviewed the use of evidence and RIEs in German Development Cooperation. The report identified the following key barriers to rigorously evaluating projects and the use of this evidence for future planning:
Lack of time,
Lack of methodological skills, and
Lack of financial resources.
The report also highlighted that the perceived usefulness of rigorously evaluating projects and using this evidence for future planning is unclear to decision-makers. Despite this, 94% of the German development professionals who responded to the survey believe that evidence is useful for improving the effectiveness of development cooperation. This suggests that further knowledge brokering with key decision-makers in the German development sector is needed. Other barriers to the uptake of evidence include the low priority given to evidence from RIEs in the design of new projects, the lack of formal requirements for the use of RIEs, and the scarce and unsystematic dissemination of findings.
1.3. Actions to Increase the Systematic Implementation and Use of RIEs in GDC
Based on the empirical research commissioned by BMZ, DEval recommends a number of actions for the German development sector to systematically become more evidence-based. Key findings are listed below (a full list can be found here):
Identify relevant topics and evidence demands to guide RIE implementation.
Raise awareness and develop capacity on RIE implementation and evidence take-up.
Install a support structure for knowledge brokering and to support RIE implementation.
Require or incentivise a systematic check of whether projects are suitable for RIE.
Require or incentivise the take-up of the best available (rigorous) evidence.
To address the lack of dedicated funding, the BMZ has also committed to funding up to nine RIEs by 2025 as part of the DEval pilot project. Almost all of these are carried out by German universities, which are used to apply for research funding and aim at publishing RIEs (ideally Randomised Control Trials) with a scientifically sound methodology in peer-reviewed journals. In most cases, however, development agencies (including the GIZ) often do not commission universities, but rather individual consultants or consulting companies to conduct different types of evaluations.
2. The Role of Consulting Companies in Driving RIEs
Consulting companies have several advantages, which could help drive the systematic implementation and use of RIEs:
Donor client orientation: Consultants want to provide the evaluation methodology that best suits the needs of donor clients. Consultancies do not have a pre-defined methodological preference and can fully adapt the method to the client’s needs. They can provide answers to multiple research questions and not only one research question, not just one, as is the case for publications in peer-reviewed journals. Consultants do not have a conflict of interest in publishing in the highest-ranked peer-reviewed journals and do not have to meet journal requirements for personal academic career development (such as experimental designs or innovative topics that help fill a research gap).
Target group-centred evaluations: Consulting companies will prioritize the interests of the target group over trying to use the most scientifically sound methodology. This ensures a target-group-centred approach to evaluation is taken.
Highest methodological standards: Consultants often have a strong academic background, enabling them to apply scientifically sound RIE methods, including experimental, quasi-experimental and non-experimental designs.
Highest technical, financial and contractual project management standards: Additionally, large consulting companies can provide very efficient financial and technical project management support. Their evaluation services ensure the correct contracting of international and national experts and partners in accordance with local taxation requirements, ethical, compliance and data protection standards, making quick use of an extensive network of experts and partners around the world.
Therefore, there is a need to combine academically sound RIE methodologies with donor/client/target group-oriented RIE solutions with the highest project management standards.
3. Partnership between GOPA and Impactloop
To this end, Impactloop LTD and GOPA Worldwide Consultants GmbH have signed a Memorandum of Understanding (MoU). The aim of the MoU is to promote the generation of evidence and to jointly conduct academically sound, target-group-centred, donor/client-oriented and efficient RIEs in the German development sector.
Impactloop works at the intersection of academic and applied research in international development and specialises in conducting mixed-methods RIE. GOPA is Germany’s leading consulting company in the field of international development cooperation. GOPA’s large monitoring, evaluation and learning departments work efficiently and have an extensive network of international experts and partners and project management experience in quasi-experimental and non-experimental impact evaluations for various international donors.
Bibliography
BMZ. (2021). Evaluierung der Entwicklungszusammenarbeit Leitlinien des BMZ. https://www.bmz.de/de/aktuelles/publikationen/92884-92884
DEval. Rigorous Impact Evaluation. https://rie.deval.org/
GIZ/GOPA 2023. Measuring Results of Skills Development Interventions: Experiences of Impact Evaluations by German, Swiss and Austrian Development Cooperation. Esser, Ursula, Holzaepfel, Franziska, Zamalloa-Huegel, Patricia. Bonn: GIZ. Available at:
OECD. Development Cooperation Profiles. Germany. https://www.oecd-ilibrary.org/sites/0079f636-en/index.html?itemId=/content/component/0079f636-en Accessed 22 August 2023.
J-PAL at 20: Supporting the use of rigorous evidence in German development cooperation. https://www.povertyactionlab.org/blog/4-6-23/j-pal-20-supporting-use-rigorous-evidence-german-development-cooperation. Accessed 2 August 2023.
Krämer et al. (2021), Rigorous Impact Evaluation: Evidence generation and take-up in German development cooperation, German Institute for Development Evaluation (DEval), Bonn.
Comments