By Rob D. van den Berg
On 20 November I attended a workshop on realistevaluations, organised by IDS Fellow Inka Barnett and co-sponsored by the Centre for Development Impact. The focus was ongoing work in evaluations and how this could be improved. Realist evaluation experts Bruno Marchal and Sara van Belle, who came over especially from Antwerp, provided an excellent overview of the realist evaluation paradigm and actively engaged with workshop participants. Throughout the day it became clear that the realist paradigm is the most detailed and sophisticated of the theory of change / theory based evaluation approaches. These are various theoretical frameworks that aim to look at assumptions that underlie policies, programmes and interventions. These assumptions can then be evaluated to see what works, for whom, how and under which circumstances.
The examples of realist evaluations explored
during the workshop were rich and varied. They ranged from using mobile phone
technology for nutrition surveillance to a realist synthesis of evaluative
evidence on water and sanitation issues, to an evaluation of influencing the
Chinese position on global health issues. Participants struggled a bit with
some of the jargon: assumptions are especially framed in terms of ‘CMOs’:
context-mechanism-outcome configurations that describe how a specific mechanism
is thought to bring change (the desired outcome) in a specific context.
However, this is the element that makes realist evaluation potentially richer
than other theory based approaches.
As an IDS Visiting Fellow I am involved in
supporting the Centre for Development Impact, so I was interested in how
realist evaluation could be positioned in the range of impact evaluation
methodologies that the Centre promotes. For me, the realist evaluation approach
scores high on a number of issues. It is the richest and theoretically most
satisfying version of the theory based approaches that have been developed over
the past decades. The approach accommodates the greatest range of potential
evaluation questions. This is immediately clear in the questions that are often
used to characterise realist evaluation: what works for whom, how, when and
where under which circumstances. The focus on causality, on mechanisms that are
supposed to bring change, makes it especially worthwhile for impact evaluations
of complex interventions. And a great advantage is that realist evaluation is
not dogmatic as regards evidence, counterfactuals and tools and methods, and
has a theoretical underpinning for this. In other words: realist evaluators
have rigorous justification for the use of data and evidence through the
realist evaluation framework.
Some drawbacks also need to be mentioned.
The jargon seems somewhat forced – CMOs need to be explained to outsiders. As
usual with these frameworks, the need to clearly define what you are talking
about may actually turn against you – the best way to prevent this in
evaluations is to maintain a strong focus on the questions that need to be
answered. If the jargon stands between your question and your answer, the
realist framework may not be the best to follow.
A second more profound difficulty is the
fact that the realist evaluation framework is all about social change. This is
a limitation that becomes a bit of an obstacle when other types of change need
to be taken into account. Evaluation is increasingly confronted with change
processes in other domains: climate change, environmental degradation, use of
natural resources, value chains, and so on. If the follow up to the millennium
development goals in 2015 will integrate use of natural resources and
environmental issues into new sustainable development goals, the realist
framework in its current incarnation will not help them. A new challenge
emerges: that of adapting realist evaluation to include non-social processes of
change.
The evaluations that were discussed are
very relevant to the work of the Centre for Development Impact. Compliments to
Inka Barnett for organising this! The Centre for Development Impact should
invite these evaluations to become part of a community of practice on impact
evaluations, hosted by the Centre, where evaluators may exchange questions,
solutions, issues to further explore and learning on new developments. I hope
to see more of this in the near future!
Rob D. van den Berg is a Visiting Fellow at
the Institute of Development Studies.
