Evaluability assessment of EBRD’s transition qualities

Background

The EBRD’s redefinition of its transition concept in 2016 was a significant change in how its core purpose was identified. The concept has had important implications for many core operational issues, and it remains a work in progress in key respects. A first independent review of the Bank’s evaluation systems completed in 2019 (the Kirk Report) asked the core question of the degree to which intended operational results could be reliably and credibly verified. It observed that the qualities were broad and abstract and therefore challenging to translate into a coherent approach to results identification and management. The EBRD has made major efforts in recent years to build a stronger and more systematic approach to transition measurement and monitoring. This evaluation seeks to address evaluability in principle and evaluability in practice. The first asks questions based on the internal logic of the Bank’s approach and systems as they exist, and the second focuses closely on the measurement approach, and tools and indicators used within those systems to reflect performance and the results achieved.

Main findings

While the transition qualities are broad and abstract, there is a range of well-established indices and metrics that, in principle, provide the basis for benchmarking and tracking transition quality performance at the macro level.

Evaluability requires a framework setting out plausible connections between operations and higher-level objectives: a theory of change connecting activities with ultimate objectives. Country strategies were intended to be the essential framework connecting operations to wider transition progress, but despite important improvements, country strategies do not now play this role.

The Bank has added substantially to the number and quality of its tools to track progress at the operational level, but these components have for the most part been developed individually rather than systematically.

For example, there is a substantial gap between country results indicators and the country-level context; policy work and technical cooperation are responsible for a substantial amount of predicted transition effects but are not meaningfully integrated into key parts of the results architecture.

Project results are monitored only until financial completion/repayment; externalities, demonstration effects and indirect and induced results are not monitored; nor are possible multiplier or synergistic effects of collaboration with other institutions.

The EBRD does not produce an overall unified report on development (or transition) effectiveness, and results reporting is scattered 27 across different programmes and instruments. Efforts to improve knowledge management have been isolated, under-resourced and not efficiently in centivised.