Showing posts with label Evaluation. Show all posts
Showing posts with label Evaluation. Show all posts

Friday, 8 November 2013

Looking for a tool to analyse and 'compare' policies? Check our our lessons from conducting a QDA

By Elise Wach

As part of our team efforts to maintain a reflective practice and share learning to others, one of our latest ‘Practice Papers in Brief’ provides some insights from conducting a Qualitative Document Analysis (QDA) on policy documents for the rural water sector.

The QDA was undertaken as part of the Triple-S (Sustainable Services at Scale) initiative, for which the Impact and Learning Team (ILT) at IDS facilitates learning. 

Qualitative Document Analysis (QDA) is a research method for systematically analysing the contents of written documents.  The approach is used in political science research to facilitate impartial and consistent analysis of written policies. 

Given that Triple-S is aiming to change policies and practices in the rural water sector, the initiative decided to undertake a QDA on policy documents at the international level in order to understand trends and progress in the sector and also to engage development partners in identifying possible changes to policies and practices to move the sector closer to achieving ‘sustainable services at scale.’  Later, we decided to expand this to ‘practice’ documents as well. 

Consistent with Triple S’s ‘theory of change’, generating discussion on these issues and catalysing change was just as much of a priority as generating reliable evidence about policy trends.

In the paper, we discuss the strengths and weaknesses of the methodology, and provide some pointers that might be helpful if it is a tool you might consider using.

Overall, we found that the QDA exercise provided useful information about trends and gaps in the rural water sector, helped to refine the Triple-S engagement strategy, and served as a useful platform for engagement with partner organisations.

Some of our lessons related to issues of defining our 'themes' and scoring, inclusion criteria for documents, unclear or zero scoring, and the relationships between the research team and the organisations included in the review.

Next week, Triple-S will be kicking off another QDA for the Ghana Workstream, to analyse government rural water policies, and will incorporate many of the lessons that we’ve learned on QDA so far.  We’ll also be conducting another round of QDA at the international level next year to analyse the ways in which the rural water sector policies have shifted over the course of the Triple-S project, and to understand what to focus on moving forward.

Elise Wach is Monitoring, Evaluation and Learning Adviser with the Impact and Learning Team at the Institute of Development Studies. 

Other blogs by Elise on Impact and Learning: 

Tuesday, 28 August 2012

Can a policy brief be an effective tool for policy influence?

By Catherine Fisher


Popular wisdom suggests that busy policy makers don’t want to read dense academic journal articles or books. Instead they want something short that summarises findings into accessible language and draws out the main implications. Consequently, policy briefs; short jargon-free summaries of research findings, have become an increasingly popular tool for researchers trying to achieve policy influence. So popular that a in a recent discussion on the Knowledge Brokers Forum, Nasreen Jesani, commented "there is policy brief fever…. People feel like it is the silver bullet and so there is an upsurge of policy brief creation."

Yet, anyone who has tried to produce one will know that policy briefs can be time consuming and expensive to create and disseminate, and study by IDRC Think Tank Initiative (pdf) raised questions about their popularity with policy makers


So how effective are policy briefs at influencing beliefs and prompting people to act differently?

Penelope Beynon from the Impact and Learning Team joined forces with others at IDS, 3ie,and the Norwegian Agency for Development Cooperation (Norad) to try to find out. The team used a randomised control design to explore three research questions:
  • Do policy briefs influence readers?
  • Does the presence of an op-ed type commentary within the brief lead to more or less influence?
  • Does it matter if the commentary is assigned to a well known name in the field?
And it is this last question where IDS Director Lawrence Haddad bravely stepped in to test what has described as “the Haddad effect”. Did having his name onthe commentary make a difference?

Full details of the study findings and methodology

Download summary of study findings (PDF)

Download full report (PDF)

Key findings

Funnily enough, the study did not find that policy briefs are a "silver bullet". However, the findings are striking and have implications for how communication experts design policy briefs and how we evaluate research communication. The study found that:

  • The policy brief was more effective in creating 'evidence-accurate' beliefs amongst those with no prior opinion than among those who already held an opinion
  • Messengers matter when it comes to readers' intended actions: the authority (Haddad) effect influenced likelihood of taking certain actions but not on beliefs
  • Gender and self-perceived levels of influence affect people’s intention to act after reading the policy brief : women were less likely to report that they would act differently after reading the brief.

The report offers some recommendations for those creating policy briefs. In particular they recommend including opinion and authority features as they may help to ensure briefs are shared and passed on. They also suggest that the startling difference in response to the brief between men and women in the study should be investigated further.

The authors have acknowledged the methodological limitations of the study, not least that a policy brief is rarely consulted in isolation. However this study is a contribution towards understanding the effectiveness of different approaches to communicating research.

It is important for those involved in trying to strengthen the connections between research and policy to think about the tools they are using and the change that they are trying to effect. As research uptake becomes increasingly important we need to invest more in understanding what works and how we can meaningfully test that.


This study is a contribution to that important debate, let us know what you think.

Comments on this study:
Is there a "Haddad" effect? Results from a randomised controlled trial Lawrence Haddad
Should think tanks write policy briefs? What an RCT can tell us Jeff Knezovitch 
Summary of e-discussion on policy briefs and information needs of decision-makers Yaso Kunaratnam on KBF

Friday, 30 March 2012

What should the post-2015 MDG (on water and sanitation) look like, and how would we measure it?

By Elise Wach
IDS Bulletin 43.2

On World Water Day, I had the opportunity to attend the IDS STEPS Centre launch of the IDS Bulletin on Politics and Pathways in Water and Sanitation. The discussion focused strongly around the MDGs: it was recently announced that the target for water had already been met, but there are a lot of questions about what that really means and how it was determined that we have ‘halved the proportion of the population without sustainable access to safe drinking water’.

The panellists all lambasted the fact that the goal says nothing about equity.  And while the word ‘sustainability’ is included the target, there are serious doubts about how that is actually measured.  Katharina Welle’s research, for example, revealed that neither the neither the method used by the Ministry of Water and Energy (based on infrastructure completed) nor the method used by the JMP (based on household use of water) accurately captures the access issues that people in rural areas are grappling with.

This resonates quite strongly with the work I’m doing with the International Water and Sanitation Centre (IRC) to synthesize the learning streams for the Triple-S Initiative, which is attempting to completely transform the rural water sector.  And I have been asking myself, if we wanted another round of MDGs, and if we kept the same sector-based approach (both are big ‘if’s’ and there are more), what would we want the water sector goal to look like, and how would we measure progress towards it?

I essentially worked backwards through a simplified theory of change, starting with the end goal. Based on the principles of Triple-S, I went ahead and defined the end goal to be:

Everyone has sustainable access to safe, adequate, and reliable water.

Essentially, there are five core components here (the five words in bold).  While they are all intrinsically linked to one another, let’s attempt to look at just one aspect of this: sustainability.

How do you measure sustainability?  You could go back and see if water services are still there in ten years’ time.  That’s useful, and I think it should be done (and so does Water for People) but how do we know now if water services will continue sustainably in ten years time?   

Perhaps we’d want to think about what is needed to ensure sustainability, and try to measure that. According to Triple-S, one prerequisite for ensuring that water services last is to ensure that there is the capacity, financing, and planning for major replacements in the future (i.e. not just maintenance). And there are a variety of other direct and indirect requirements for sustainability: ownership, inclusion, accountability, transparency, government capacity, and coordination to name a few.

So a big challenge would be to agree on the requirements for sustainability.  Assuming we can overcome this (daunting) hurdle, we’d then need to assess whether these are in place.  But measuring these won’t be as straightforward as measuring the number of people who live within a certain radius of a borehole.  And there are other issues as well, such as the issue of Multiple-Use Systems, as Stef Smits discusses.

It will be, in a word, complex.   

But if the issues we’re trying to address are complex (and they are - read more about "complexity" in this ODI working paper (PDF)), then it isn’t surprising that measuring progress and achievements is complex as well. 

While the simplicity of the MDGs may have helped mobilise support for development, that simplicity comes at a cost.  As Lindsey Nelson discussed in her STEPS presentation on multi-modal discourse last week, there are consequences to basing your strategy on a bumper sticker slogan.  Something to think about as we discuss the post-2015 development agenda.

Monday, 19 March 2012

(Still) Seeking a cure for Portal Proliferation Syndrome

By Susie Page

As co-editor to a forthcoming issue of the IDS Bulletin (autumn 2012) which will be focusing on ‘research communications’ (all facets of it!), I am currently reading about some very exciting work around this.

During our Call for Submissions, Geoff Barnard, former Head of the Information Department here at IDS, and now Head of Knowledge Management at the Climate and Development Knowledge Network (CDKN) sent me a link his blog on Seeking a Cure for Portal Proliferation Syndrome.

Geoff aptly captures the dilemma that anyone working in research communication and knowledge brokering will be familiar with – the temptation to solve some of the challenges around research communication and uptake in development policymaking and practice by gathering all the relevant research into a super, sophisticated website. The underlying assumption being – if only people could access the research (at the click of a button), then the rest will follow.

He obviously hit a nerve, as there was a stream of responses to his blog, including one from Catherine Fisher who also contributes to this blog, highlighting her work “Ten Portal Pitfalls” – I would urge you to read Geoff's blog and contribute to the debate. 

Can we 'scientifically' test for what works when it comes to research uptake?

Image from: http://72.167.62.13/
Coming from a medical background, the words “cure” and “syndrome” had immediate resonance for me: in the medical world, we are acutely aware of persistent diseases and syndromes with millions of pounds spent on testing for cures (although take it from me the health sector struggles equally with the business of research uptake! See Lomas on this, for example).

But does similar testing occur in this sector – the one that wants to get good research results out of the lab and into development policy and practice? Is it even feasible to conceive of a scientific test for something as amorphous as “knowledge” and “evidence”?


Going back to 'the cure', we should perhaps be asking whether portals are a syndrome or a symptom. If they are symptom, the problem could be that we think research is not being used in policymaking and practice because people don’t have access to it. Yet, surely the very proliferation of portals in itself highlights that this isn’t the problem – after all, how will one more portal succeed where others have failed? What do we know about the success and failures of portals? What actually do we know about the relationship between portals and research uptake?

With people still wracking their brains over measuring impact of research, there is room for some robust ‘scientific’ testing what is and isn’t effective for supporting research uptake and the place (or otherwise) of portals within this. We recently teamed up with 3ie to carry out an experiment on the effectiveness of the ubiquitous ‘policy brief’ (even more ubiquitous than portals, I would argue). The results are just beginning to come through. Watch this space for more on this – we will of course be sharing our findings with you!

Tuesday, 7 February 2012

Monitoring and evaluation in partnerships: why learning comes first

Guest post: Andre Ling, Research Officer/Technical Assistance with the Agricultural Learning and Impacts Network (ALINe) at the Institute of Development Studies (IDS)

The Impact and Learning Team and the ALINe project members share insights and good practice during bi-monthly Learning Labs, an afternoon of learning and reflection framed by our project work and esearch questions. Andre shares his reflections in this blog post.

Last week's Learning Lab with the Impact and Learning Team (ILT) touched on two topics that are probably relevant to just about all actors involved in development processes: partnerships and sustainability. Both are buzzwords, frequently used and misused, open to a range of interpretations and often obscuring the hard realities that confront development practice.

This post looks specifically at 'partnership' and what approaches to monitoring and evaluation (M&E) may be most appropriate in a partnership context.


The term 'partnership' frequently glosses over the complexity of inter-organisational relationships:
  • The many reasons for which organisations find themselves in a partnership to begin with
  • The power asymmetries inherent in the common model of grantor-grantees that defines many partnerships
  • The different interests, values or ideological positions of partners
  • The specific organisational development needs of different partners


Image from: http://www.ramsar.org
The prevalence of such factors can lead one to the conclusion that most partnerships are marriages of convenience as those joining in such ventures do so largely to serve their own interests. But, at the same time, it barely deserves mentioning that an individual actor can do little to address the complex problems of our times.

So how do we go on together honestly? How do we make partnerships work?

And what kind of M&E makes sense in partnership contexts?



To begin with, taking stock of power asymmetries within the partnership and mapping out the divergent interests, values, worldviews, spheres of concern and needs of partner organisations may be a good place to start. Often the leading agency in a partnership (usually a grant-maker of sorts) will have goals of achieving some kind of systemic change. The other partners (frequently grantees) may be more concerned with implementing specific activities to contribute to more localised changes and be less comfortable with confronting systemic change. Such a divergence creates a critical disjuncture, resulting in the leading agency wanting to make changes in the other partners; to make them see the same way. Failure to engage with this rift in a sensitive manner can lead to a variety of problems, for example low trust and limited ownership.

One response is, perhaps, to establish a joint M&E framework; a system of indicators and reporting requirements that can be deployed to ensure that all organisations are 'on the same page' and are held accountable to achieving desired outcomes in a standardised way. The danger here is that compliance and standardisation – both power mechanisms – ride roughshod over what might be considered the more crucial goal: learning how to work together effectively to achieve mutually desired change.

This is not to dismiss the contribution of joint M&E frameworks but rather to put them in their rightful place as servants of learning (individual, organisational or institutional) rather than formalities implemented for their own sake. To clarify, the point is that learning encompasses a far wider set of practices and activities than what usually goes by the name of M&E and, furthermore, that regular M&E is not a sufficient pre-condition for learning to take place. Just think of all the evaluation reports that have been shelved and forgotten.

An emphasis on learning prior to M&E opens the door to a potentially more diverse set of tools, techniques and practices that can be used to (a) build relationships of mutual trust; (b) reveal and question entrenched assumptions; (c) share and cultivate more systemic ways of thinking about the nature of the problems that the partnership is seeking to tackle.

This can prepare the ground for co-creating an M&E system that is both oriented toward learning and situated within a partnership culture that is supportive of learning. A significant consideration here is that this means taking the partnership itself as a unit of analysis to be monitored, evaluated and learned from and through over time.

Monday, 30 January 2012

Change is hard but not impossible with a little help from ELFs (part 2)

Guest post: We welcome back Elise Wach, Evaluation Consultant at the Institute of Development Studies (IDS)

To follow on from my post last week on how change is hard to achieve, even when you know an approach isn't working, here's an update on the Learning Retreat that the Impact and Learning Team facilitated for the IRC (International Water and Sanitation Centre) Triple-S Initiative. 

We kicked off the 2-day retreat by hearing reports back on the variety of learning streams that had taken place so far and assessing progress.  This led into discussions about the approach of the initiative as well as the approach of the evaluation and learning.  Were course corrections needed in order to achieve the desired outcomes of the initiative?  Were adjustments to the evaluation and learning methodologies needed in order to better capture initiative progress?

In terms of the approach of the initiative, one of the primary concerns was related to external communication - getting the right information out to the right people in a timely manner.  IRC’s key staff immediately made commitments to improve this:
  1. Set up an organisational blog to get discussions started and put information out on the web (see Emilie Wilson’s entry about the merits of blogging).  
  2. Post more information and resources on the website, even if not yet polished.  A key phrase throughout the retreat was to ‘not let the perfect be the enemy of the perfectly good’.  Resources and information may take the form of videos, slide decks, and less formal reports.
  3. Improve the layout of the website to make it more user-friendly, and the search engine tags to make it easier to find.
Other commitments were also made to improve the approach of the initiative, including a re-visit to the Theory of Change.  This will surely be discussed in detail at the next Learning Retreat which will go into more depth into the data and its synthesis and also include the advisory group for the initiative (scheduled for the end of April).

See more at http://www.waterservicesthatlast.org  

In terms of the evaluation and learning approaches, a few changes will be made to the timing and scope of data collection and analysis.  For example, given the disconnect between policy and practice discussed in last week’s entry, it was decided that in addition to analysing the policy documents of key agencies and organisations in the water sector, IRC and ILT will also analyse documents that might indicate that shifts in policies are being reflected in practice, such as calls for proposals and project reports.  That information will of course be posted on the Triple-S website in an effort to give these agencies a little extra nudge towards sustainable services that last.  



It was also determined that impact weighting for different outcomes and milestones might prove useful, along the lines of DFID’s revised approach to logframes (PDF) (though Triple-S is looking to move away from a linear/tabular format and towards more mind mapping and video).  

What was interesting to me about the retreat was the fact that none of the data discussed revealed anything incredibly new or surprising to the Triple-S team.  But for some reason, getting a group of people in a room together and setting aside time to specifically discuss progress and obstacles can be extremely effective for getting decisions made…especially if External Learning Facilitators (endearingly referred to as ELFs) are there to help the process along.

IRC has committed to starting their blog in the next week and will soon be posting more resources on their website, including the full report from the Learning Retreat (which will cover much more than I’m able to include here). As for making it easier to find the Triple-S website, you can try to Google it for yourself, but I think they’re still working on this one (unfortunately there’s a clothing company that goes by the same name!), so just in case, the "Water Services that Last" website is here.

Tuesday, 17 January 2012

Change is hard

By Elise Wach

Elise Wach is currently working with the Impact and Learning team to support the IRC International Water and Sanitataion Centre through a learning process around its Triple-S Initiative 

Penelope Beyon’s blog about failure and learning brings up some interesting and very valid points about the recent attention to failure, evaluation and learning in development. While it is essential (and quite difficult) for the development community to know when their programming is unsuccessful, and admit this, it doesn’t do any good if we don’t then learn from our failures and change our approaches so as not to repeat them.  

But how does change happen?

The IRC International Water and Sanitation Centre is in the middle of a six-year initiative which is attempting to shift the rural water sector away from a one-off infrastructure-based approach towards a Service Delivery Approach; what they term, Sustainable Service Delivery at Scale (Triple-S). How to enact change is exactly what they’re trying to figure out.

For decades, most development organisations and agencies in the rural water sector (like most sectors) did not know about, or did not want to know about, their failures. They were oblivious to the fact that the majority of their boreholes fell into disrepair within five to seven years, or that many were never even used at all.  Without this knowledge, it is easy to see why development organisations and agencies charged ahead with the same unsuccessful approaches.  

However, in a recent round of interviews I conducted with key stakeholders in the sector (as part of the Impact and Learning Team's support to IRC on this initiative),  it was overwhelmingly apparent that now, everyone in the water sector knows that the standard approach to rural water supply has been ineffective and unsustainable. It is common knowledge that the sector has been failing.  

So the rural water sector has overcome that essential, but difficult first hurdle of finding out about and acknowledging failure. 

Source unknown, but widely available
And in the most recent round of interviews, key stakeholders in the sector generally agreed that the discourse at the top – the policy-level – is starting to reflect these revelations.  But funding practices and implementation on the ground seem to have continued relatively unchanged: the same infrastructure-focused, unsuccessful approaches continue to dominate. Why?

Because change is hard.

One interviewee explained, ‘We’ve been engineered to do small-scale piecemeal interventions…so of course shifting to more of a sustainable approach at scale (vis-à-vis financial flows, regulations, norms, and standards) is going to take time.  There will be resistance to change.’

Changing approaches to realising change


To date, IRC ‘s Triple-S initiative has been attempting to accelerate changes within the sector through three main approaches:
  • Relationship-led (i.e. using champions to mobilise change)
  • Value-led (i.e. leveraging peer pressure and creating coalitions for change)
  • Evidence-led (i.e. providing proof that the current approaches don’t work and proof that other ones do)
The initiative has also been exploring the relationships between policy, funding and practice.

This week, IRC and the Impact and Learning team are holding a learning retreat to go over the findings from the most recent round of stakeholder interviews and other evaluation data.  Based on this, IRC may refine its Theory of Change and tweak its approach to help maximise the efficacy of the initiative moving forward. 

We’ll report back on the outcomes of that learning retreat next week.

In the meantime, a final thought. If more development actors followed a similar approach to IRC (i.e. if they thought through Theories of Change for their approaches and periodically revised them based on real-time evaluations and analysis), it’s not unrealistic to think that the way we work in ‘development’ would be quite different. That is, the phenomenon that Penelope termed as ‘reinventing broken wheels’ might not be as common. Change is hard, but not impossible, and it is certainly needed.

Wednesday, 11 January 2012

Are we reinventing broken wheels? Let’s talk about the ‘F’ word

By Penelope Beynon

A common saying goes "The only real failure in life is the failure to try."  I disagree.

I think the worst failure in life (and in knowledge brokering) is the repetition of an established mistake. That is to say, the worst failure is the failure to learn.

In recent months, I have come across an increasing number of websites, discussions and articles that almost celebrate failure, in an effort to foster a culture of sharing and learning from others’ mistakes. The Engineers Without Borders (EWB) website Admitting Failures is a good example. In their own words:

"By hiding our failures, we are condemning ourselves to repeat them and we are stifling innovation. In doing so, we are condemning ourselves to continue under-performance in the development sector.

Conversely, by admitting our failures – publicly sharing them not as shameful acts, but as important lessons – we contribute to a culture in development where failure is recognized as essential to success."

While I agree with the premise, often times it is not fully realised.
Image from: http://st-anley.blogspot.com


Ironically, perhaps, several of the ‘failures’ admitted on the EWB website are, in fact, examples of people’s failure to learn from past mistakes – their own and those of others. That is, they are reinventing broken wheels, sometimes under the guise of 'innovation'.



Innovation is important for progress, and with innovation comes a certain level of risk. But I think these risks need to be calculated and one of the key considerations should be a thorough investigation of whether this particular experiment is truly an innovation or whether it has already been tested elsewhere. That is, an honest commitment to learning before doing as well as learning after doing. I hear the echo of Catherine’s recent blog where challenges knowledge brokers to practice what they preach .

Lessons identified or lessons learnt? 


Learning is a big theme for the Impact and Learning Team at IDS  and we have recently been thinking a lot about the difference between a lesson identified and a lesson learned.

In our view, a lesson is only really 'learned' when the implications of the lesson are acted upon. Far too often we see After Action Reviews and evaluation documents that recite from their own experience ‘lessons’ that are insights long established internally and already documented in the experience of others (e.g. developing partnerships takes time, communication matters, etc.). Very seldom does anyone pick up that the worst failure here was not the failure to communicate but the failure to identify ahead of time that communication matters and to learn from others’ experiences about how to do it well.

One outstanding example of a lesson that was learned (albeit the hard way) is retold by Lieven Claessen, a researcher from the International Potato Centre (CIP),s  in two short videos produced the Consultative Group on International Agricultural Research (CGIAR)'s ICT-KM programme.

In the first video, Claessens identifies the lessons by bravely telling a rather sobering story about his failure to communicate research findings in a way that people likely to be affected could understand and use for decision making. Had the findings of his 2007 research been acted on, the devastating effects of the 2010 mudslides in Eastern Uganda could have been mitigated, potentially saving the lives of hundreds of people and the livelihoods of hundreds more.  In his second video, Claessens evidences his learning by telling how he has changed his approach and commitment to communicating research to ensure he does not repeat this same mistake.

I find Claessens' story deeply moving for two reasons.

Firstly, I take my hat off to anyone who owns up to their part in a failure with such devastating consequences. Especially where that failure could as easily have been passed off to someone else.

Secondly, I find the story unique in its clarity about the link between research communication and wellbeing outcomes. Or, in this case failure to communicate research and negative outcomes. Often that link is much less clear for knowledge brokering. In fact, just as it is difficult (if not impossible) to evidence attribution of development outcomes to knowledge brokering work, it is equally difficult (if not impossible) to evidence negative development outcomes to failure in the same area. Perhaps this provides something of a safety net that allows us to distance ourselves from consequences, or maybe it is one of the reasons that it is apparently so hard to talk about failure in the knowledge brokering arena.

Tuesday, 15 November 2011

Are evaluators too fixated on objectives? The case of conflict-affected states

By Chris Barnett


It is a while since I was co-facilitating a workshop outside Brussels on evaluations in conflict-affected situations. We were exploring alternative approaches to evaluation, building on recent experiences of multi-donor evaluations in South Sudan and the DRC. Previously I’ve raised the issue about whether theories of change have their limitations (see my earlier post on evaluation approaches). In the end, we spent much more time exploring the use and limitations of objectives in evaluation practice. Here is a brief summary of those discussions.

Objectives form the reference point for most mainstream (development) evaluations. Objectives underpin the OECD/DAC evaluation criteria. For instance:

  • Relevance: Are the programme objectives aligned to the situation?
  • Effectiveness: Were the programme objectives met?
In development evaluation,[1] we sometimes come across programmes that are radically re-designed during the mid-term. When this happens, the evaluator is faced with a dilemma: do you measure success against the original objectives, but risk undervaluing a genuinely flexible and responsive programme? Or, do you measure success against the re-designed objectives, but risk excusing an initially poor design?  The former often seems too harsh, the latter too lenient.

In conflict-affected situations, flexibility is a must and re-design occurs more often. Many things cannot be planned in advance, and managers respond to constraints and opportunities as they arise. Yet, being responsive to a dynamic context can undermine plans made three or five years previous. There are also other reasons why clearly planned objectives may be undermined in such fragile contexts: 
  • Terms such as peace and conflict prevention may not be well defined. Indeed, not all stakeholders may share the same understanding within a single programme (e.g. differing civilian and military perspectives). 
  • Peace may not be an explicitly-stated objective. The Utstein palette (PDF) is helpful in this regard, as it unpacks the full range of activities that may contribute to conflict prevention and peace-building. Yet for the evaluator, this all-encompassing view of peace-building activities presents a challenge; with interventions contributing to peace even when this is not stated.
  • Objectives may be politically-sensitive, and may be hidden or masked. For the ex post evaluator, a retrospective interpretation of such objectives opens up the real possibility of contested findings. This is especially so where the ‘evaluation space’ is highly contested, as was the case in the CPPB evaluation in SriLanka (PDF).
So, what provides an alternative point of reference?

Well, the perspectives of local people: their needs, and their understanding of what drives the conflict. The draft OECD Guidance on Evaluating Conflict Prevention and Peacebuilding Activities (currently under revision), provides an entry point for developing such an approach. It layers Conflict Analysis onto the traditional DAC evaluation approach, although it doesn’t fully integrate the two approaches. Instead, a more radical view would be to use the conflict analysis (especially an analysis by local people about what drives the conflict) as a reference point for evaluating the intervention. For instance:

  • Relevance: To what extent does the programme reflect a local understanding of the conflict?
  • Effectiveness: To what extent does the programme address the drivers of the conflict?
  • Impact: To what extent is the programme leading to reduced conflict, by building peace?
Some will argue that this shifts accountability away from programme managers. After all, managers should be accountable for their design, and therefore measured against the objectives they stated. But, this suggested approach is more about making managers accountable to their constituents (their target audience), and focuses on whether they are addressing local needs and factors behind the conflict.


[1] By development evaluation, I am simply referring to evaluations conducted in relatively stable developing contexts, in contrast to those undertaken in fragile or conflict-affected states.

Wednesday, 2 November 2011

Exploring the black box together: evaluating the impact of knowledge brokers

Cartoon by Sidney Harris (2007)
By Catherine Fisher

I love this cartoon! 

It seems to capture the idea of the "black box" that lies between the activities knowledge brokers and intermediaries undertake and the outcomes and impacts they seek to achieve. That’s not to say that they don’t achieve outcomes in the real world, rather that the pathways by which their work brings about change are difficult to unpack and evaluate.

The Knowledge Broker’s Forum (KBF) has started exploring this "black box" of how to evaluate the impact of knowledge brokers and intermediaries in an e-discussion running from 31 October until 9 November. I am (lightly) facilitating this discussion, along with Yaso Kunaratnam from IDS Knowledge Services.

If you would like to participate, you can sign up on the forum's website, it's open to anyone with an interest in this area.

Challenges in evaluating impact

We know there are a lot of challenges to evaluating impact of knowledge brokering. Some challenges stem from the processes (psychological, social and political) in which knowledge and information bring about change, the contested nature of the relationship between research and better development results, and the challenges of identifying contribution to any changes in real world contexts. This is particularly challenging for actors that seek to convene, facilitate and connect rather than persuade or influence.

As well as these quite high level challenges, there are the very practical issues around lack of time and resources to dedicate to effectively understanding impact. These challenges are explored in a background paper (PDF) I prepared as food for thought for those taking part in the e-discussion.

Being an e-discussion amongst 400+ knowledge brokers from all over the world, I am not sure yet where discussions will go, but I am hoping that it will shed some light on the following areas:

Breadth and depth of impact and outcomes  

How far do people go to identify ultimate outcomes of knowledge brokering work? I feel we can certainly go beyond immediate impact (e.g. personal learning) to push towards what that resulted in, however I wonder if it is meaningful to start looking at human development and wellbeing indicators. It will be interesting to see how far others are going.

Understanding behaviour change

If knowledge brokering is about behaviour changes that ensure greater engagement with research evidence, how are people defining those behaviour changes and are how are they measuring them? Are we too easily impressed with stories of information use when these could in fact hide some very poor decision-making behaviours?

Opportunities for standardisation of approaches and data collection

If people have come up with ways of doing this, is there any appetite for standardising approaches to enable greater comparison of data between different knowledge brokering initiatives? This would help us build a greater understanding of the contribution of knowledge brokers beyond the scope of any one broker’s evaluation.

I’ll also be interested to explore and challenge some of my assumptions – in particular that building some kind of theory or map of change is an important starting point for defining and then seeking to evaluate impact. This has been discussed previously on this blog and is a hot topic at the moment.

Our discussion will face challenges – not least the huge variety of types of knowledge brokering and contexts in which it is undertaken may mean there is not enough common interest. But I am sure that there is a lot of experience in the group that can be brought to bear on these questions and, in 10 days time, we will have a better idea of what is known, who is keen to explore this further and and hopefully how we could move forward to develop our understanding in this area.

Monday, 24 October 2011

Beyond happy sheets: outcome-focused event evaluation

By Penelope Beynon

Since joining the knowledge for development sector in June last year, I have participated in no less than 2 international conferences, 3 regional workshops and a host of cross-organisational meetings (and sent apologies for three times as many of each). Some cost money (for international or intercity travel), all have opportunity costs (being here instead of there) and they all cost time. 

As a participant, I find there is something innately attractive and energising about being together in a room with experts and peers that just cannot be simulated through online alternatives; but as a taxpayer I can’t quite shake that uncomfortable question – was it worth it?

In my role as M&E advisor I am occasionally asked how to evaluate events – while I haven’t yet found a tried and tested method that fits every event, I thought I’d share a few things I have learnt along the way.

With a few notable exceptions (e.g. A Process and Outcomes Evaluation of the International AIDS Conference, Lalonde et al 2007), most organisers fail to evaluate their events beyond a cursory feedback form that gauges audience satisfaction (commonly referred to as a ‘happy sheet’). But, if an organiser did want to push their evaluation to a new level and address the ‘uncomfortable’ question of worth – where would they begin?

In its most simplistic form, I propose that a worthwhile event evaluation needs to gather three types of information:
  • Costs 
  • Outcomes 
  • Reasonable alternatives

The full financial cost of events is rarely included in evaluation

The table below shows a summary of some areas where events incur costs. Unsurprisingly few organisers publish even the full financial costs of their events (grey box) or even add up their own financial and time costs (grey + purple boxes) for purposes of evaluation, let alone start to consider the sectoral costs of their event to participants and contributors.



Focusing on desired outcomes
Learning events may benefit all of these groups (P. Beynon, IDS)

1.    Spread your net wide when looking for outcomes

A common short coming of most outcome-focused event evaluations that I have unearthed (of which there are few to begin with) is a narrow concept of where benefits will occur and an almost exclusive focus on participants as the subjects for evaluation. Just as there are at least three groups who can incur costs for an event, these same groups could feasibly incur benefits (see diagram).



2.    Tailor your evaluation tools to match desired outcomes

Like all interventions, face-to-face events do not happen in isolation, they are usually part of a wider set of strategies intended (implicitly or explicitly) to contribute in some way to a programme's overall theory of change. Unfortunately, more often than not this link is not properly explored and event objectives read like either a) a less-than-ambitious list of activities, or b) an overly ambitious set of development aspirations well beyond anything the event could possibly deliver. Work closely with organisers to get to flesh out their theory of change and to situate the conference objectives within the wider programme context - then you will be able to tailor your evaluation tools to match the desired outcomes. While some organisers are coming up with interesting tools and approaches for outcome-focused event evaluation (e.g. network mapping (PDF), 3-test self-assessment) which I explore along with a few of our own attempts in a forthcoming ILT Practice In-Brief paper, most still limit their data sources to attendance records and the standard ‘happy sheet’.

3.    Follow through on your follow up!

The biggest limitation for most event evaluations is a lack of meaningful follow up. Change takes time, and unless you follow up with participants when they are back in their workplace you will only be able to capture intended behaviour change or the initial step towards an extended network. Be disciplined – schedule event follow ups for 3, 6 even 12 months after the fact.

Is there a cheaper way to achieve the same outcomes?
Well, this really is the million dollar question, and without a clear picture of our costs and benefits it just cannot be answered. But when you do have this level of information for one event, you will be able to start comparing that event with another and maybe even progress on to comparing all your face to face events with other strategies that use different tactics to achieve similar aims: such as ongoing rather than one-off events; online rather than face to face convening; 1 to 1 rather than convened events...

To conclude
As the saying goes, “If it’s worth doing at all it is worth doing properly” - so I urge organisers to go beyond ‘happy sheets’ and really scrutinise the worth of their events for their own sake and for the sake of the sector.

Wednesday, 14 September 2011

Exploring evaluation approaches: Are there any limits to theories of change?

By Chris Barnett

I’m in Brussels co-facilitating a course on evaluation in conflict-affected countries, with Channel Research. We are exploring new and alternative approaches to evaluation, building on recent experiences of multi-donor evaluations in South Sudan and the Democratic Republic of Congo (DRC). The South Sudan and DRC evaluations are part of a suite of evaluations that sought to test the draft OECD Guidance on Evaluating Conflict Prevention and Peacebuilding Activities.

While the context is very specific, I’m hoping that the discussions will raise some interesting issues around the way we approach evaluation and particularly how we use theories of change. The term “theory of change” is a much overused phrase at the moment, and one that seems to have different meanings to different people. In this case it is being defined as, “the set of beliefs [and assumptions] about how and why an initiative will work to change the conflict” (OECD Guidance, page 35). Duncan Green, in his blog, also helpfully points out the difference between a theory of change (a classic, linear intervention logic, or results chain, used as a basis for logical frameworks) from theories of change (such as a range of theories about the political economy of how and why change occurs).

Photo courtesy of Jon Bennett
Working in conflict-affected states poses many challenges for evaluation, not least the changing context, instability and insecurity. It most cases it is not feasible to set up a controlled experiment and maintain it over a reasonable period of time. Not only are there the cost and ethical issues of distributing benefits randomly, but also the sheer technical difficulty of maintaining a robust counterfactual in a context where there is so much change. It is not impossible of course (e.g. IRC’s evaluation of Community Driven Reconstruction in Liberia); just often not appropriate or feasible.

Hence, the OECD Guidance focuses on a theory-based approach to evaluation (NB: Henry Lucas and Richard Longhurst, IDS Bulletin 41:6, provide a useful overview of different evaluation approaches). At the heart of the OECD Guidance is the need to identify theories of change, against which to evaluate performance.

But in South Sudan and DRC we found a number of limitations to this approach:

1. Firstly, we found it challenging to apply a theory of change approach to the policy or strategic level. Most donors did not articulate a transparent, evidence-based rationale for intervening – sometimes intentionally so, given the dynamic and sensitive context. This meant that reconstructing theories of change for evaluation purposes became highly interpretive and open to being challenged – particularly when drawing out differences between actual and de facto policies.

2. Secondly, we found that different theories of change existed at different levels. As one moved down from the headquarters level to the capital city, and onto local government and field levels, then views differed about the drivers of conflict and the theories of change necessary to address these. This presented the evaluator with a dilemma – and sometimes wrongfully placed as arbiter to different perspectives and realities.

3. Thirdly, while lots of activities contribute to conflict prevention and peacebuilding, many were not be explicit about such objectives. Again, the reconstruction of the de facto theories of change against which to assess performance becomes highly interpretive and more open to being challenged.

So do we hope to do this week? We will be exploring alternatives to such Objective or Goal-Based evaluations that seek to assess performance against the stated (or reconstructed) theories behind an intervention. Rather, we’ll explore some Goal-Free alternatives – where data is gathered to compare outcomes with the actual needs of the target audience, using reality as the reference point rather than a programme theory. After all, in many walks of life we do not “evaluate” performance against the stated objectives: When we assess whether a car is good or not, we do not consider whether the design team fulfilled its objectives! Rather, we are interested in whether it fulfils our needs.