Showing posts with label Evidence. Show all posts
Showing posts with label Evidence. Show all posts

Wednesday, 7 August 2013

Open data and increasing the impact of research? It's a piece of cake!

By Duncan Edwards

I talk to a lot of friends and colleagues who work in research, knowledge intermediary, and development organisations about some of the open data work I’ve been doing in relation research communications. Their usual response is “so it’s about technology?” or “open data is about governance and transparency, right?”. Well no, it’s not just about technology and it’s broader than governance and transparency.

I believe that there is real potential for open data approaches in increasing the impact of research knowledge for poverty reduction and social justice. In this post I outline how I see Open Data fitting within a theory of change of how research knowledge can influence development.

Every year thousands of datasets, reports and articles are generated about development issues. Yet much of this knowledge is kept in ‘information silos’ and remains unreachable and underused by broader development actors. Material is either not available or difficult to find online. There can be upfront fees, concerns regarding intellectual property rights, fears that institutions/practitioners don’t have the knowhow, means or time to share, or political issues within an organisation that can mean this material is not used.

What is “Open data”? What is “Linked Open Data”? 

The Open Knowledge Foundation says “a piece of content or data is open if anyone is free to use, reuse, and redistribute it — subject only, at most, to the requirement to attribute and/or share-alike.”

The Wikipedia entry for Linked Data describes it as“a method of publishing structured data so that it can be interlinked and become more useful. It builds upon standard Web technologies such as HTTP and URIs, but rather than using them to serve web pages for human readers, it extends them to share information in a way that can be read automatically by computers. This enables data from different sources to be connected and queried…. the idea is very old and is closely related to concepts including database network models, citations between scholarly articles, and controlled headings in library catalogs.”

So Linked Open Data can be described as Open Data which is published in a way that can be interlinked with other datasets. Think about two data sets with country categorisation – if you publish these as linked data, you can then make the link between related content between different datasets for any given country.

For more definitions and discussion on data see Tim Davies post "Untangling the data debate: definitions and implications".


Why should Open Data be of interest to research producers? 

The way in which the Internet and technology has evolved means that instead of simply producing a website from which people can consume your content, you can open up your content so that others can make use of, and link it in new and exciting ways.

There are many theories of change which look to articulate how research evidence can affect development policy and practice. The Knowledge Services department at the Institute of Development Studies (IDS) works with a theory of change which views access to, and demand for, research knowledge, along with the capacity to engage effectively with it, as critical elements to research evidence uptake and use in relation to decision-making within development. Open Data has significant potential in relation to the ‘access to’ element of this theory of change.

Contextualisation and new spaces 

When we think about access to research knowledge – we should go beyond simply having access to a research document. Instead we must look at whether research knowledge is available in a suitable format and language, and whether it has been contextualised in a way which makes sense to an audience within a given environment.



I like to use a Data cake metaphor developed by Mark Johnstone to illustrate this - if we consider research outputs to be the data/ingredients for the cake, then we organise, summarise and catalogue this (i.e. add meta-data) to ‘bake’ into our information cake. We then present this information in a way in which we feel is most useful and “palatable” to our intended audiences with the intention they will consume it and be able to make use of new knowledge. It’s in this area that Open Data approaches can really increase the potential uptake of research – if you make your information/ content open it creates the possibility that other intermediaries can easily make use of this content to contextualise and present it to their own users in a way that is more likely to be consumed.

Essentially by opening up datasets of research materials you can reduce duplication, allow people to reuse, repurpose, remix this content in many more spaces thereby increasing the potential for research findings to be taken up and influencing change in the world.

While I see significant benefits in researchers making their outputs available and accessible in an open manner we must redress the dominance of knowledge generated in the global North. We need to continue to invest in the strengthening of intermediaries at local, national, and international levels to make use of research material and Open Data to influence positive change.

Duncan Edwards is the ICT Innovations Manager at the Institute of Development Studies (IDS) - you can follow him on Twitter: @duncan_ids

NOTE: an admission on Open Access - The original article this post is based on, “Davies, T. and Edwards, D. (2012) 'Emerging Implications of Open and Linked Data for Knowledge Sharing in Development', IDS Bulletin 43 (5) 117-127”, published in the IDS Bulletin: “New Roles for Communication in Development?”. Ironically, considering it’s subject matter, is only partially open access (two free articles per issue). But you can access this article as green open access in Open Docs - http://opendocs.ids.ac.uk/opendocs/handle/123456789/2247

Tuesday, 28 August 2012

Can a policy brief be an effective tool for policy influence?

By Catherine Fisher


Popular wisdom suggests that busy policy makers don’t want to read dense academic journal articles or books. Instead they want something short that summarises findings into accessible language and draws out the main implications. Consequently, policy briefs; short jargon-free summaries of research findings, have become an increasingly popular tool for researchers trying to achieve policy influence. So popular that a in a recent discussion on the Knowledge Brokers Forum, Nasreen Jesani, commented "there is policy brief fever…. People feel like it is the silver bullet and so there is an upsurge of policy brief creation."

Yet, anyone who has tried to produce one will know that policy briefs can be time consuming and expensive to create and disseminate, and study by IDRC Think Tank Initiative (pdf) raised questions about their popularity with policy makers


So how effective are policy briefs at influencing beliefs and prompting people to act differently?

Penelope Beynon from the Impact and Learning Team joined forces with others at IDS, 3ie,and the Norwegian Agency for Development Cooperation (Norad) to try to find out. The team used a randomised control design to explore three research questions:
  • Do policy briefs influence readers?
  • Does the presence of an op-ed type commentary within the brief lead to more or less influence?
  • Does it matter if the commentary is assigned to a well known name in the field?
And it is this last question where IDS Director Lawrence Haddad bravely stepped in to test what has described as “the Haddad effect”. Did having his name onthe commentary make a difference?

Full details of the study findings and methodology

Download summary of study findings (PDF)

Download full report (PDF)

Key findings

Funnily enough, the study did not find that policy briefs are a "silver bullet". However, the findings are striking and have implications for how communication experts design policy briefs and how we evaluate research communication. The study found that:

  • The policy brief was more effective in creating 'evidence-accurate' beliefs amongst those with no prior opinion than among those who already held an opinion
  • Messengers matter when it comes to readers' intended actions: the authority (Haddad) effect influenced likelihood of taking certain actions but not on beliefs
  • Gender and self-perceived levels of influence affect people’s intention to act after reading the policy brief : women were less likely to report that they would act differently after reading the brief.

The report offers some recommendations for those creating policy briefs. In particular they recommend including opinion and authority features as they may help to ensure briefs are shared and passed on. They also suggest that the startling difference in response to the brief between men and women in the study should be investigated further.

The authors have acknowledged the methodological limitations of the study, not least that a policy brief is rarely consulted in isolation. However this study is a contribution towards understanding the effectiveness of different approaches to communicating research.

It is important for those involved in trying to strengthen the connections between research and policy to think about the tools they are using and the change that they are trying to effect. As research uptake becomes increasingly important we need to invest more in understanding what works and how we can meaningfully test that.


This study is a contribution to that important debate, let us know what you think.

Comments on this study:
Is there a "Haddad" effect? Results from a randomised controlled trial Lawrence Haddad
Should think tanks write policy briefs? What an RCT can tell us Jeff Knezovitch 
Summary of e-discussion on policy briefs and information needs of decision-makers Yaso Kunaratnam on KBF

Thursday, 3 May 2012

More on Change: systems, principles, and learning

By Elise Wach

I am back to talk about change following on from my previous postings (Change is hard and Change is hard but not impossible) on how you change a sector, Here are some reflections from the latest IRC Triple-S learning retreat.

IRC is attempting to change the way water is provided in rural communities by:
  1. changing the way things work at the level of rural communities so that water is available to everyone indefinitely, and
  2. changing the water sector to enable this to happen. 
I think it is easy to get lost in the frameworks and theories that attempt to explain how to achieve these changes. For example, there are a number of different frameworks proposed for influencing policy and measuring that influence (for example Crewe and Young 2002, Court and Young 2003, Steven 2007, Gladwell 2000, etc.). These provide useful insights as to what might make a difference, but at the end of the day we need to remember that these are complex and unique systems that we are trying to change: so there is no ‘best practice’! 

Danny Burns’ seminar at IDS yesterday on "How Change Happens" helped remind me that while it is not necessarily labelled as such, Triple-S is essentially using a Systemic Action Research (PDF) approach: their larger (systems) view of the water sector and iterative learning processes enable them to recognise and respond to opportunities for change.

Image from: http://www.rallytorestoresanity.com 
In attempting to influence policy, for example, Triple-S is not just looking at written policy documents (although this is one piece of Triple-S work).  But they recognise that policy change results from and is indicated by changes in discourse, perceptions, agendas, networks, political contexts, and institutions.  And that a multitude of stakeholders are involved in those changes, including journalists, NGO workers, researchers, finance ministers, and even people who post on Twitter. They recognise that certain events (such as a change in government) can greatly accelerate or completely block policy changes. 

And that the right evidence and information at the right time delivered to the right people could make a difference.  So the Triple-S approach is built on the assumption that changing policy doesn’t entail following a formula but instead recognising and responding to opportunities and trigger points.

At the rural community level, Triple-S is trying to ensure that the rural water sector takes into account a variety of factors in order to ensure that water services are provided to everyone indefinitely.  So this means looking at life-cycle costs, mechanisms for transparency and accountability, possible alternative service providers, accounting for the multiple uses of water, etc. etc.  

But does viewing these issues with a systemic lens mean that we become paralysed by the complexity?  Danny Burns pointed out yesterday that the key is to focus on action rather than on consensus.  To focus on the actions that different actors can take that can change the system.  Or as Bob Williams explains in his the Ottawa Charter approach (doc), it will be a ‘strategically selected jigsaw of people and organisations doing what they are most effective at’ that will create lasting change, rather than Triple-S trying to change the sector on its own.

Triple-S isn’t trying to get consensus around a specific approach to achieving sustainable rural water supply, but is instead trying to get everyone on board with basic set of principles for sustainable services and providing a range of resources and tools and building capacities (look out for new trainings in the near future) to put those principles in action. They are leveraging existing institutions and structures, and working closely with individuals and organisations to facilitate ownership.

But getting people to wrap their heads around the concept of changing their principles is a big obstacle.  People want tools and approaches that they can go put into action, and while Triple-S is providing a range of these, success starts with viewing rural water supply completely differently: it isn’t ‘the Service Delivery Approach’ but ‘a Service Delivery Approach’. 

Another obstacle Triple-S is facing relates to the way in which evidence is perceived.  So there are people who say, ‘this is all fine and good in theory, but is it really possible? Can we really achieve both sustainability and scale? Where is the evidence?’  Evidence is a strong word.  Today, it usually refers to a call for a ‘rigorous’ approach like a randomised control trial.  But if you want to find out if services are provided forever, then how long do you have to wait for the RCT results?  And here is where I cannot resist but refer to the brilliant example of the limitations of RCTs – would you doubt that a parachute would make jumping out of a plane safer just because an RCT has not proven it?  I think this highlights the need for the development community to reflect on what we consider to be evidence.     

But I don’t think that these obstacles are insurmountable, especially given that Triple-S’ approach enables it to recognise and respond to opportunities and challenges while remaining focused.  One of the Triple-S pillars is for the rest of the rural water sector to have ‘a strong learning and adaptive capacity’.  I see this as pre-requisite for success in the other two pillars, and in the rural water sector in general.  But achieving this is....well, complex.

Monday, 24 October 2011

Beyond happy sheets: outcome-focused event evaluation

By Penelope Beynon

Since joining the knowledge for development sector in June last year, I have participated in no less than 2 international conferences, 3 regional workshops and a host of cross-organisational meetings (and sent apologies for three times as many of each). Some cost money (for international or intercity travel), all have opportunity costs (being here instead of there) and they all cost time. 

As a participant, I find there is something innately attractive and energising about being together in a room with experts and peers that just cannot be simulated through online alternatives; but as a taxpayer I can’t quite shake that uncomfortable question – was it worth it?

In my role as M&E advisor I am occasionally asked how to evaluate events – while I haven’t yet found a tried and tested method that fits every event, I thought I’d share a few things I have learnt along the way.

With a few notable exceptions (e.g. A Process and Outcomes Evaluation of the International AIDS Conference, Lalonde et al 2007), most organisers fail to evaluate their events beyond a cursory feedback form that gauges audience satisfaction (commonly referred to as a ‘happy sheet’). But, if an organiser did want to push their evaluation to a new level and address the ‘uncomfortable’ question of worth – where would they begin?

In its most simplistic form, I propose that a worthwhile event evaluation needs to gather three types of information:
  • Costs 
  • Outcomes 
  • Reasonable alternatives

The full financial cost of events is rarely included in evaluation

The table below shows a summary of some areas where events incur costs. Unsurprisingly few organisers publish even the full financial costs of their events (grey box) or even add up their own financial and time costs (grey + purple boxes) for purposes of evaluation, let alone start to consider the sectoral costs of their event to participants and contributors.



Focusing on desired outcomes
Learning events may benefit all of these groups (P. Beynon, IDS)

1.    Spread your net wide when looking for outcomes

A common short coming of most outcome-focused event evaluations that I have unearthed (of which there are few to begin with) is a narrow concept of where benefits will occur and an almost exclusive focus on participants as the subjects for evaluation. Just as there are at least three groups who can incur costs for an event, these same groups could feasibly incur benefits (see diagram).



2.    Tailor your evaluation tools to match desired outcomes

Like all interventions, face-to-face events do not happen in isolation, they are usually part of a wider set of strategies intended (implicitly or explicitly) to contribute in some way to a programme's overall theory of change. Unfortunately, more often than not this link is not properly explored and event objectives read like either a) a less-than-ambitious list of activities, or b) an overly ambitious set of development aspirations well beyond anything the event could possibly deliver. Work closely with organisers to get to flesh out their theory of change and to situate the conference objectives within the wider programme context - then you will be able to tailor your evaluation tools to match the desired outcomes. While some organisers are coming up with interesting tools and approaches for outcome-focused event evaluation (e.g. network mapping (PDF), 3-test self-assessment) which I explore along with a few of our own attempts in a forthcoming ILT Practice In-Brief paper, most still limit their data sources to attendance records and the standard ‘happy sheet’.

3.    Follow through on your follow up!

The biggest limitation for most event evaluations is a lack of meaningful follow up. Change takes time, and unless you follow up with participants when they are back in their workplace you will only be able to capture intended behaviour change or the initial step towards an extended network. Be disciplined – schedule event follow ups for 3, 6 even 12 months after the fact.

Is there a cheaper way to achieve the same outcomes?
Well, this really is the million dollar question, and without a clear picture of our costs and benefits it just cannot be answered. But when you do have this level of information for one event, you will be able to start comparing that event with another and maybe even progress on to comparing all your face to face events with other strategies that use different tactics to achieve similar aims: such as ongoing rather than one-off events; online rather than face to face convening; 1 to 1 rather than convened events...

To conclude
As the saying goes, “If it’s worth doing at all it is worth doing properly” - so I urge organisers to go beyond ‘happy sheets’ and really scrutinise the worth of their events for their own sake and for the sake of the sector.

Friday, 14 October 2011

Early headlines from research on policy makers and ICTs: "persistent and curious enquirers" (with smartphones)

By Simon Batchelor

Just to keep you up to date on the country studies that I mentioned in my first blog….(in which I spoke about research we were conducting on policy makers and their use of ICTs).... a lot of data is in. Some countries found it easier to get interviews with senior policy makers than others, so some countries have still to deliver their full quota.

However we have now begun analysis and we begin to find some interesting headlines. As I write, my colleagues Jon Gregson and Manu Tyagi are presenting some headlines back to a portion of the intermediary sector in India and Nepal, and Chris Barnett presented last week in Ghana.  I would like to acknowledge the work of our partners ODC Inc in Nepal, and Delink Services  in Ghana.

So what are some of those headlines?

We will upload the slideshare soon, but in brief here are some of the things that attracted my attention:-

Policy actors have access to ICT, and a considerable number of them have smartphones, and to my mind more importantly, know how to use them!

Image from: http://bestsmartphone2011.info/

Of course they almost all have access to computers and the internet, and cellphones.  But Ghana 52%, Nepal 49% and North India 35% of the samples have smartphones.  In Ghana 25% had more than one smartphone!  And of those that have a smartphone, almost all in Ghana and Nepal have explored sending emails, surfing in the internet on the phone, recording a video and instant messaging.  Only in North India did it seem that there were a significant portion of people who had a smartphone and yet did not explore these ‘features’ (about 50%).

What does this mean to us in the intermediary sector? It suggests that if you are developing an app to push research into the policy environment, then the baseline of smartphone use is there.

Policy actors are surfing the internet themselves – the idea that policy makers wait for an assistant to brief them seems to be diminishing.  

In all three countries, the majority of policy makers agreed with statements surrounding their own use of ICT and surfing the internet.  They described themselves as ‘a "persistent and curious" enquirer’ and noted that they ‘often "discover" other relevant information when searching’ (Phrases used by the PEW Internet studies in USA).  They also agreed to a lesser extent with ‘I tend to get my briefings face to face officially, in meetings’.  In Ghana, where there was a significant portion of private sector executives, there were a significant number who actually disagreed with the idea that they got their information from ‘official briefings’.

What does this mean to us in the intermediary sector? It suggests that policy actors are looking for information themselves, and, I presume, therefore need to find it easily, in an accessible form, and I guess, quickly.  

Yes, I know that searching for information online is evolving, and that social networks now tend to push information within the network.  This changes the way those of us who are well connected get our information.  We did investigate whether the policy actors are connected to social media networks and to some extent looked at their searching behaviours, but we are not there yet in the analysis to be able to comment on it.  Watch this space.

Policy actors do have an appetite for research – or at least they say they do 

There was a consistent strong agreement with the need for facts and figures, and that these need to be up to date.  We explored what information they were actually looking for and we looked at whether they trust the sources and channels for the information.  Again, these details will come out as the analysis proceeds.  However there was an interesting difference between the three countries.  In India there is a strong trust for ‘local research’ (as opposed to international research), however in Ghana  and Nepal they rate international research much higher than local research.
 
What does this mean to us in the intermediary sector ? In our MK4D programme, we are working on the idea that local intermediaries understand the context of research and policy in their location, and therefore have a strong ground with which to communicate research to policy makers. However, we also work with the idea of ‘co-construction’ working alongside and with our colleagues in the South.   If ‘local research’ is trusted less by policy actors, then that would seem to endorse the approach of co-construction – where local and international bodies work together to provide quality insights.   It also suggests that our programme to support the exposure of research published in the South onto the global internet is heading in the right direction.

Anyway, those are some insights from the first week of analysis.  More to come.

Friday, 7 October 2011

Getting serious about the evidence in policy making

By Nick Perkins
 
See www.impactevaluation2011.org
Earlier this year, the International Initiative for Impact Evaluation - better known as 3ie - convened a conference in Mexico called
Mind the Gap: From Evidence to Policy Impact.

I liked the idea of dedicating 3 days, dozens of presentations and hundreds of blog posts to that little ‘leap of faith’ which characterises so many theories of change about what research can do for development.


The problem that we are faced with is that the normative idea about how policy should be made – based on objective evidence – is seldom the reality that we are faced with - i.e. policy through political expediency. Political expediency is understood to be a range of contextual influences on the decision-making process. When described this way, there is something inevitable about it.

Current thinking is that this expediency can be addressed through mediation of research knowledge. This has given rise to the research mediation sector- institutions and individuals within institutions who seek to frame research in a way that it is accessible and relevant to people working in key policy spheres.

What this reveals is a kind of contradiction at the heart of the development knowledge sector. While we call for evidence-based policy making, there is also increasing investment in the complex process that shapes decision making. A way through this may have been revealed through a closer look at what research mediation actually entails.

A couple years ago, IDS held a series of ‘influencing seminars’ which revealed how different disciplinary communities nuanced their approaches to policy influence depending on how they understood change happened. None of them declared disdain for value of quality evidence. Instead they all expressed differing views of what constitutes ‘quality’ evidence and how to gain traction with those who might need it.

What emerged was a framework of four different ways of building an effective relationship between research and quality policy making.

The first is about generating as many policy options as possible. This emphasises the use of repositories to allow users to sift through the options for themselves.

The second is evidence-based and prioritises the familiar idea that the quality of the research evidence is what will best inform the quality of the decision. Systematic reviews are seen as crucial in the research mediation process here.

Third is the value-led idea of policy-making. There are many examples of this leading to bad science, but it is by far the most common type of public policy making. Networks and epistemic communities are critical to the mediation process in this case.

Finally we have the relational model of influence, which maintains that no amount of research will influence a policymaker if there is not a relationship which reflects equity and a balance of power -where a researcher or a mediator are themselves subject to some influence.

Clearly though, none of these frames are mutually exclusive. Perhaps the point is that we can support the complex reality of policy influence which draws on these without losing sight of the where we ultimately need to get to. In fact using a little political expediency ourselves can go a long way to cross what is too often seen as a small gap.