Impact and Learning blog will no longer be updated but you can read all the latest opinions from the IDS community on our website

This will be the last post to be published on this blog. The Institute of Development Studies now publishes all our members’ and guest bloggers’ posts directly onto our website.

You can keep up to date with the latest opinion and expert analysis from your favourite development bloggers, comment on and share their posts, by visiting our website, following us on social media or subscribing to updates from the following IDS units:

• Communications and Engagement Unit
• Knowledge Mobilisation and Impact Unit
• Open Knowledge and Digital Services Unit

Alternatively, you may be interested in these IDS research and knowledge clusters:

• Business, Markets and the State in Development
• Cities
• Conflict, Violence and Development
• Digital
• Gender and Sexuality
• Governance
• Green transformations
• Health and Nutrition
• Participation
• Power and Popular Politics
• Resource Politics
• Rural Futures

We are sure you will continue to enjoy the conversation.

Best Wishes

The IDS Communications and Engagement Unit

Realist Evaluation

By Rob D. van den Berg

On 20 November I attended a workshop on realistevaluations, organised by IDS Fellow Inka Barnett and co-sponsored by the Centre for Development Impact. The focus was ongoing work in evaluations and how this could be improved. Realist evaluation experts Bruno Marchal and Sara van Belle, who came over especially from Antwerp, provided an excellent overview of the realist evaluation paradigm and actively engaged with workshop participants. Throughout the day it became clear that the realist paradigm is the most detailed and sophisticated of the theory of change / theory based evaluation approaches. These are various theoretical frameworks that aim to look at assumptions that underlie policies, programmes and interventions. These assumptions can then be evaluated to see what works, for whom, how and under which circumstances.

The examples of realist evaluations explored during the workshop were rich and varied. They ranged from using mobile phone technology for nutrition surveillance to a realist synthesis of evaluative evidence on water and sanitation issues, to an evaluation of influencing the Chinese position on global health issues. Participants struggled a bit with some of the jargon: assumptions are especially framed in terms of ‘CMOs’: context-mechanism-outcome configurations that describe how a specific mechanism is thought to bring change (the desired outcome) in a specific context. However, this is the element that makes realist evaluation potentially richer than other theory based approaches.

As an IDS Visiting Fellow I am involved in supporting the Centre for Development Impact, so I was interested in how realist evaluation could be positioned in the range of impact evaluation methodologies that the Centre promotes. For me, the realist evaluation approach scores high on a number of issues. It is the richest and theoretically most satisfying version of the theory based approaches that have been developed over the past decades. The approach accommodates the greatest range of potential evaluation questions. This is immediately clear in the questions that are often used to characterise realist evaluation: what works for whom, how, when and where under which circumstances. The focus on causality, on mechanisms that are supposed to bring change, makes it especially worthwhile for impact evaluations of complex interventions. And a great advantage is that realist evaluation is not dogmatic as regards evidence, counterfactuals and tools and methods, and has a theoretical underpinning for this. In other words: realist evaluators have rigorous justification for the use of data and evidence through the realist evaluation framework.

Some drawbacks also need to be mentioned. The jargon seems somewhat forced – CMOs need to be explained to outsiders. As usual with these frameworks, the need to clearly define what you are talking about may actually turn against you – the best way to prevent this in evaluations is to maintain a strong focus on the questions that need to be answered. If the jargon stands between your question and your answer, the realist framework may not be the best to follow.

A second more profound difficulty is the fact that the realist evaluation framework is all about social change. This is a limitation that becomes a bit of an obstacle when other types of change need to be taken into account. Evaluation is increasingly confronted with change processes in other domains: climate change, environmental degradation, use of natural resources, value chains, and so on. If the follow up to the millennium development goals in 2015 will integrate use of natural resources and environmental issues into new sustainable development goals, the realist framework in its current incarnation will not help them. A new challenge emerges: that of adapting realist evaluation to include non-social processes of change.

The evaluations that were discussed are very relevant to the work of the Centre for Development Impact. Compliments to Inka Barnett for organising this! The Centre for Development Impact should invite these evaluations to become part of a community of practice on impact evaluations, hosted by the Centre, where evaluators may exchange questions, solutions, issues to further explore and learning on new developments. I hope to see more of this in the near future!

Rob D. van den Berg is a Visiting Fellow at the Institute of Development Studies. 

It’s stakeholder mapping Jim but not as we know it

By James Georgalakis

What happens when you create fictitious organisations working on make-believe influencing scenarios and ask a bunch of people who have never worked together before to develop stakeholder maps for them? Quite cool stuff actually.

At a recent IDS short course designed to provide a broad overview of research and policy communications we included a section on stakeholder mapping tools. Over the past decade I have run many workshops which included some form of stakeholder mapping exercise. But whether the sessions took place with civil society organisations in Malawi, researchers in Nepal or social workers in Ukraine, context was always key. During all these capacity building events we worked on real scenarios. So what were we to do in a single day with 25 participants from a broad range of think tanks, universities, NGOs and consultancies, two thirds of whom had never undertaken any kind of stakeholder mapping before? Make it up of course!

We simply created five pretend scenarios based on very different policy contexts inspired by the range of participants we were expecting. We surveyed them first checking what kind of policy actors they typically targeted. We then got pretty creative making up an organisational profile, an influencing objective and we even suggested some potential stakeholders they might want to kick things off with. The exercise was otherwise pretty much like any other network mapping style process. They identified further stakeholders, they placed them on a large map to indicate their relationship to their made up institution and they looked at their relationship to one another. They then scored each one in terms of their level of influence on the issue and the likelihood of them being allies, opponents or neutral in relation to the hoped for influencing goal.

One of the main differences from a more conventional session was it was faster. Detailed maps were produced in under an hour and a half – although they would have all happily taken much longer if we’d let them. With less time spent on reviewing objectives (we provided these) and dissecting deep rooted institutional issues around identity, legitimacy, power and profile, the participants quickly explored different external stakeholders and their potential usefulness. However, the discussions still contained much of the richness that more conventional sessions do. Our approach meant that we had mixed groups learning from one another’s institutional and sector perspectives. They quickly discovered they had quite different ideas about how change happens and the impact of research, and begun exploring hidden power relationships. In the group I facilitated they challenged the narrow list of parliamentary and governmental stakeholders we had suggested and wanted to extend their map to include civil society organisations, social scientists and the media. This in turn helped them unpack what it means to try to influence the quality of a particular public policy discourse with evidence (they were pretending that they were going to try and engage with the controversial debate on the impact of immigration on the UK in the run up to a General Election!)

The participants really seemed to gain an appreciation of the importance of mapping your policy environment and how documenting the discussion is almost as useful as the map itself. They revisited their maps in the afternoon and used them to select priority audiences for whom a communications plan was then developed. It all felt pretty real by the end and the fictitious scenarios appeared to deliver much the same learning and tools, that they could apply back in their own organisations, as any more realistic grounded exercise would have done.

You can download all the course materials including the stakeholder mapping scenarios and facilitators notes here. IDS is currently developing an Achieving Influence and Impact Series, so do let us know if you would like to be kept informed of future courses and free resources on this topic: training@ids.ac.uk

James Georgalakis, is the Head of Communications at IDS
Follow James @ www.twitter.com/bloggs74

Other posts by James Georgalakis on research communications:

The Guardian
Has Twitter killed the media star?
Marketing: still the dirty word of development?

On Think Tanks
Is it wrong to herald the death of the institutional website?
How can we make research communications stickier? 

Impact and Learning 
Digital repositories – reaching the parts other websites cannot reach
Influencing and engagement: why let research programmes have all the fun?
Going for gold: why and how is IDS bringing our journal back in house and making it open access?

Going for gold: Why and how is IDS bringing our journal back in-house and making it open access?

By James Georgalakis

The recent announcement by IDS that we are not planning to renew our contract with Wiley Blackwell for the publication of our journal, the IDS Bulletin, will have delighted some and baffled others. Re-launching our flagship publication as a gold open access digital journal means the end of subscription income and the end of a large publisher’s marketing support. From January 2016 the Bulletin will be produced in-house and will be available to all for free.

Since 1968 the IDS Bulletin has been an integral part of IDS’ research dissemination strategy, covering the major themes and influencing debates within international development. As we move forward we will build on its unique characteristics including the thematic issues that mobilise scholars from multiple disciplines, around key development issues. However, for the first time in its history from 2016 there will be no paywall, no embargos and few licencing restrictions to obstruct researchers, students, policy actors and activists from using the Bulletin to support their work.

This new open access IDS Bulletin will be supported by robust editorial and peer review processes with an editorial steering group made up of IDS fellows from all of our key research areas plus an advisory body to provide oversight. Academic editors of issues will be drawn from across the IDS community, including our partners, and a small in-house production team will provide a high quality publication available for free digitally and in print for those that need it.

Who Pays?
Of course the key conundrum of the open access movement has always been who pays? If not the subscriber then surely the researcher (via their funder of course). What this means in practice for conventional social science journals is paying article processing charges (APCs) of around £2000 per article to commercial publishers. In the case of the new IDS Bulletin we are dispensing with this process altogether and simply bringing production and distribution in-house and charging projects and programmes a fixed sum of just under £6000 for a whole issue of up to 9 articles and then fundraising to meet the shortfall. With its long history, policy focused thematic issues, not-for-profit financial model and full compliance with even the most stringent open access policies, we are confident that the IDS open access Bulletin will attract the financial support it needs to continue to provide fresh new thinking on key development issues.

Why not just publish a hybrid?
A hybrid publishing route involves placing submitted versions (post peer review but pre editing and formatting) of your articles in a repository and making others gold open access with APCs. Many publishers including Wiley have developed hybrid systems. However, the Bulletin is quite different to other journals. With thematic issues that are often built around a specific research project or programme the use of APCs is almost impossible. We need to fund each whole issue otherwise it is game over. Hence the charge we apply to projects wanting to commission an issue. Plus, simply releasing the submitted versions of articles still fails to meet the strictest of open access mandates such as DFID’s. In fact, this hybrid route was originally recommended by the UK Government as part of a five-year transition (from 2012 to 2017) towards fully Gold Open Access. It does not work for us and it may not represent a long term solution for other learned societies in the social science sector.

An open access strategy fit for fifty years in development studies
Our emerging open access strategy and our desire to pursue engaged excellence demands that we open access to as much of our evidence and knowledge as possible. The schemes to open up access to journals to southern institutions such as HINARI, AGORA and OARE under the Research4Life programme are good but no longer go far enough. The ongoing evolution of the IDS Bulletin is in part thanks to Wiley Blackwell themselves who have over the last six years helped build its credibility and reach. However, the expiry of our contract with Wiley at the end of 2015 marked an opportunity to take the journal into the next exciting phase of its development. It will be re-launched in our fiftieth year and form a key part of our anniversary celebrations as we release the entire back catalogue. This major publishing event will form part of the narrative around the Institute’s fiftieth birthday as we explore our future role in a changing world in which development knowledge is generated globally and we seek to share it with all those that need it.

James Georgalakis, is the Head of Communications at IDS
Follow James @ www.twitter.com/bloggs74

Other posts by James Georgalakis on research communications:

The Guardian
Has Twitter killed the media star?
Marketing: still the dirty word of development?

On Think Tanks
Is it wrong to herald the death of the institutional website?
How can we make research communications stickier? 

Impact and Learning 
Digital repositories – reaching the parts other websites cannot reach
Influencing and engagement: Why let research programmes have all the fun?

Hosting webinars: lessons from a recent ‘on air’ experience engaging stakeholders in real-time

By Adrian Bannister

As part of a recent e-dialogues week delivered by the Making All Voices Count programme team, colleagues from IDS Research and Knowledge Services departments worked to convene an entirely web-based audience of invited stakeholders to two online events. 

These book-ended 5 days of asynchronous online discussion that took place on the Eldis Communities platform – you can read my top tips for facilitating online discussion here.

So what’s a webinar?

For the uninitiated a webinar is simply defined as ‘a seminar conducted over the Internet’. Like a real-world seminar this generally involves one or more presenters speaking to, and receiving questions from, an audience. 

It happens in real-time and typically involves the streaming of audio and video. For the audience it enables them to be ‘virtually there’ - for the project team it means working like a television production unit.

By Jorge Díaz (Flickr: On air) CC-BY-SA-2.0 

One key distinctive feature, and advantage, is that nobody (from the panel of speakers or the audience) has to be in the same location. However, each participant must have a broadband Internet connection and knowledge of ‘where the virtual room is’ i.e. a web link. In some cases it is also necessary to download software e.g. At&T Connect.

Rationale and approach

We chose to open the e-dialogues week using a webinar event for three reasons:

:- to present the programme’s own thinking on the questions the e-dialogues week was posing (the focus was on the experience of Technology for Transparency and Accountability Initiatives - T4TAIs) 

:- to provide an opportunity for programme stakeholders to ‘meet’ the programme staff and hear them speak directly, and;

:- to begin to stimulate thinking and interaction among participants around aspects of the online discussion to come

Both events were an hour-long and followed a broadly similar format. A panel of IDS colleagues sat together in the same room, though they could have each been disparately located, and each shared a short presentation that was live-streamed to the audience. 

The audience were able to interact with each other via a chat-room and, and also to queue-up questions to the panel. These were fed to the speakers during an extended question and answer session. 

The closing webinar provided a mechanism for the panel to verbally summarise and provide commentary on the areas of the discussion that they had facilitated. 

A note about technology

To host these webinars we setup dedicated and private virtual rooms using the Click Meeting platform, which was provided and supported by Webgathering. Participants were able to access the space via a web browser, which only required a minimum software download, the popular Adobe Flash plug-in.

Though there were only 80 or so invitees for our events, Click Meeting can host up to 1000 attendees for large webinars and has useful text translation features for a variety of languages. It allows for meetings to be recorded as a video file (in MP4 format) which makes it accessible and shareable with most computer users afterwards. 

Web Gathering has an extended facility with Click Meeting so they can accommodate more presenters than a standard plan however a free trial is also available.

Key lessons

Based largely on these experiences, we’ve gathered our thoughts into the following: 15 top-tips for managing webinars which may be useful for your own projects.

Adrian Bannister is the Web Innovations Convenor & Eldis Communities Coordinator at IDS