Can a policy brief be an effective tool for policy influence?

By Catherine Fisher


Popular wisdom suggests that busy policy makers don’t want to read dense academic journal articles or books. Instead they want something short that summarises findings into accessible language and draws out the main implications. Consequently, policy briefs; short jargon-free summaries of research findings, have become an increasingly popular tool for researchers trying to achieve policy influence. So popular that a in a recent discussion on the Knowledge Brokers Forum, Nasreen Jesani, commented "there is policy brief fever…. People feel like it is the silver bullet and so there is an upsurge of policy brief creation."

Yet, anyone who has tried to produce one will know that policy briefs can be time consuming and expensive to create and disseminate, and study by IDRC Think Tank Initiative (pdf) raised questions about their popularity with policy makers


So how effective are policy briefs at influencing beliefs and prompting people to act differently?

Penelope Beynon from the Impact and Learning Team joined forces with others at IDS, 3ie,and the Norwegian Agency for Development Cooperation (Norad) to try to find out. The team used a randomised control design to explore three research questions:

• Do policy briefs influence readers?
• Does the presence of an op-ed type commentary within the brief lead to more or less influence?
• Does it matter if the commentary is assigned to a well known name in the field?

And it is this last question where IDS Director Lawrence Haddad bravely stepped in to test what has described as “the Haddad effect”. Did having his name onthe commentary make a difference?

Full details of the study findings and methodology

Download summary of study findings (PDF)

Download full report (PDF)

Key findings

Funnily enough, the study did not find that policy briefs are a "silver bullet". However, the findings are striking and have implications for how communication experts design policy briefs and how we evaluate research communication. The study found that:

• The policy brief was more effective in creating 'evidence-accurate' beliefs amongst those with no prior opinion than among those who already held an opinion

• Messengers matter when it comes to readers' intended actions: the authority (Haddad) effect influenced likelihood of taking certain actions but not on beliefs

• Gender and self-perceived levels of influence affect people’s intention to act after reading the policy brief : women were less likely to report that they would act differently after reading the brief.

The report offers some recommendations for those creating policy briefs. In particular they recommend including opinion and authority features as they may help to ensure briefs are shared and passed on. They also suggest that the startling difference in response to the brief between men and women in the study should be investigated further.

The authors have acknowledged the methodological limitations of the study, not least that a policy brief is rarely consulted in isolation. However this study is a contribution towards understanding the effectiveness of different approaches to communicating research.

It is important for those involved in trying to strengthen the connections between research and policy to think about the tools they are using and the change that they are trying to effect. As research uptake becomes increasingly important we need to invest more in understanding what works and how we can meaningfully test that.


This study is a contribution to that important debate, let us know what you think.

Comments on this study:
Is there a "Haddad" effect? Results from a randomised controlled trial Lawrence Haddad
Should think tanks write policy briefs? What an RCT can tell us Jeff Knezovitch 
Summary of e-discussion on policy briefs and information needs of decision-makers Yaso Kunaratnam on KBF

Do policymakers in the South text more than talk?

By Simon Batchelor

In a news story today on the BBC website, the headline reads “Texting overtakes talking in UK, says Ofcom study”.  The article states that “While 58% of people communicated via texts on a daily basis in 2011, only 47% made a daily mobile call, [according to the UK's] communications industry regulator [Ofcom].”

As readers of this blog will know, we have been conducting our own study in 6 countries on the information ecosystem* of policy actors.  You can find our previous blogs, here and here.

While we asked participants in the study about what ICT technology they owned, how frequently they used the Internet, how they searched for information, and so on, we did not ask them specifically about their preferred form of mobile phone use (texting or talking).  However, the Ofcom report which prompted the BBC news story, focuses on much more than texting – in fact, it covers very similar set of questions to our own study (indeed we used the Pew US studies to help shape our own study).

For example, UK Ofcom report states that “39% of (UK) adults now own a smartphone, a 12% increase on 2010.”  How does this compare with policy actors in the South? The graph below shows that of our sample of 100 actors in each country, a similar proportion have at least one smartphone. In fact some people have more than one, since cross network calls are often expensive enough that it makes it worth carrying two phones to keep call costs down.  This is particularly true of say Ghana, where several respondents had 2 or 3 smartphones.

The UK report also states that “Tablet ownership is also on the rise, with 11% owning such a device, up from 2% last year.”  Interestingly as this blog showed, Tablet ownership among policy actors is at an equivalent level – currently at about 15% (from the updated data set). We concluded in that blog that this ownership of technology suggests that information intermediaries seeking to get evidence in front of policy actors in order to inform their decisions, should indeed be using these channels.

The Ofcom report also offers detail on internet behaviour of UK households – contrasting the behaviour of 16 to 24 year olds with others. Of course, in our study we didn’t have many 16–24 year olds as we were interviewing those in leadership and management positions.  However we did note that the younger respondents tended to do more with their smartphones and tablets than older respondents – with the exception of where older respondents let their children play with their phones. Where this occurred, the older respondents themselves had as good a knowledge of the more tricky smartphone activities such as downloading an app, or uploading a video, as the younger respondents. 

And finally the Ofcom report states that “Two thirds of internet users have accessed Facebook.”.  From our preliminary data almost 70% of our respondents have accessed internet communities. Keeping in mind that actually there are alternative social networks to Facebook on the world scene (hard to believe!), this again shows an equivalent figure. What is much more surprising from our data is that nearly 60% of those who have smartphones have accessed social networks from their phones. Unfortunately, the Ofcom report doesn’t say how people access their social network. 

Conclusion?  As we have previously noted in our work, the respondents to our survey show that technology use among policy actors in the South pretty much mirrors the average household in USA (Pew Internet and American Life Project research). With this Ofcom report on UK behaviour, we can see that our research findings also appear to mirror technology use in the UK.   

If this is so, (and this is where I am speculating) then perhaps it is reasonable to assume that the details of how people surf might also be true?  For instance, Ofcom report that “With two-thirds of internet users on Facebook, it generates almost a quarter of all referred traffic to YouTube (23.7%), in contrast to Google’s 32.3%. Facebook also refers traffic to other popular websites: BBC (11.2%), eBay (6.7%), Twitter (3.8%), and Wikipedia (3.6%).”  If policy actors are members of social networking sites, then perhaps Facebook is driving them to sites as much as their own searching of Google. If this is the case, then it becomes all the more important that social networks are used by knowledge intermediaries to bring evidence to the attention of policy actors.

When policy actors engage with the internet, what do they actually do?

By Simon Batchelor

Ever wondered what those serious men and women who describe themselves as involved in policymaking actually use the internet for?

From our study of 360 policy actors across 4 countries, Ehtiopia, Nepal, India and Ghana, the graph below shows the percentage of the whole sample undertaking certain actions on the internet.

Unsurprisingly, almost all policy actors engage with the internet for emails. As you can see from the graph, over 80% download official forms, obtain information from public authorities’ websites and read or download newspapers or online news. A majority undertake the remaining options Instant Messaging, internet communities, video and audio podcasts, uploading self-created content and seeking health-related information. Just under half have telephoned over the internet.

Yet, when you compare these responses, coming from a wide variety of actors in the global South, some of whom have quite poor connectivity, to that pioneer of internet engagement, the USA, you find that their internet use is almost same as that of the average American household! According to recent Pew Research Center on households in the USA, 92% read email, or use a search engine, 76% read online news and 65% use social networking. Only 24% of USA households have used Voice of Internet (PDF link) – so in this case the policy actors are ahead in their use than the average US household.

Are differences still about connectivity?

There are of course differences across the countries not captured by the above graph. We would expect the use of the internet to relate to the connectivity of the country, and to some extent this is what we see in our findings. According to the 2011 ICT Development Index (IDI) (PDF), undertaken by the International Telecommunication Union, Ethiopia ranks 150th, Nepal 134th, Ghana 120th and India 116th - out of a total of 154 countries covered by the index. (We also speculated that connectivity in the South of India was better than the North, and divided our sample accordingly.)

In the graph above we see that telephoning over internet (VOIP), which demands good connectivity, follows our understanding of the connectivity of the countries surveyed. Ethiopia with its poor connectivity makes telephoning a difficult option even for those on the best connectivity the country can provide. Whereas in South India it has now become common across most respondents to use the internet for telephoning (80%).

Data around obtaining information from public authorities’ websites also follows levels of connectivity at the country level, with the exception of Ethiopia. The unusual high use seems to be explained from other data as being about ‘formality’ and need to know official government positions.

However, not all behaviours are best explained by connectivity. When we consider instant messaging and uploading video – which other data suggests are emergent behaviours more dependent on early adoption  – there is no significant difference between countries. Whereas for the use of emails, while there is some difference between countries, the overwhelming uptake makes such differences of no practical meaning.

Finally just considering early adopters; unsurprisingly early adopters have a higher use on Telephoning over the Internet, video calls (via webcam), entertainment, videos or audio podcasts and ‘uploading self-created content to be shared’. We didn’t dare ask these senior policy actors what entertainment they were accessing!

So why should we be interested in this? 

We often talk as though policy actors are a unique species, that because they are such important people, they behave in a unique way. To me the data suggests they are human as the rest of us when it comes to communication and information-seeking behaviours. They use the internet in much the same way as the average UK and USA household uses it. And there is a spread of use, in the same way we see a spread of use among US households. Some have invested in new technologies (see my earlier post on the adoption of Tablet PCs) and some of them are early adopters, just as some households are early adopters. Policy actors seem to demonstrate a spread of human interest in the new communication tools in much the same way as a population with wealth and access might show.

When I first came to IDS I wondered whether ‘policy actors’ in countries of poor connectivity would ever push the Facebook button on a webpage or use instant chat for a helpdesk. I thought that perhaps some of our efforts using the latest communications technologies were a little too North-centric, and that policy actors in the South might be left cold.

However, our research suggests otherwise. My scepticism has been duly quashed.

Philosopher-craftsmen: interesting times for research communications professionals

Plato, the Greek philosopher

By Emilie Wilson

Two exciting new publications have landed on my desk today :
(1)  Knowledge, policy and power in international development: a practical guide and the latest edition of the IDS Bulletin,
(2)  Action research for development and social change.

Knowledge, policy and power in international development: a practical guide, not a definitive model

The first, a book by researchers at the Overseas Development Institute (ODI), aims to be a "practical guide to understanding how knowledge, policy and power interact to promote or prevent change". However, the authors are quick to put in a disclaimer:

"...we acknowledge that, although some models provide useful analyses of some aspects of the interface between knowledge and policy, it is impossible to construct a single one size-fits-all template for understanding such a complex set of relationships".

That is not to say the authors aren’t aiming high: "this book seeks to provide: 

• a state-of-the-art overview of current thinking about knowledge, policy and power in international development 
• present empirical case studies that provide concrete examples of how these issues play out in reality 
• offer practical guidance on the implications of this knowledge base” 

I’m looking forward to getting stuck in, and am particularly intrigued by their “Questions this section will help you to answer” approach to structuring some of the content. I’m also looking out for references to work by IDS Knowledge Services around knowledge intermediation (well, of course I am!).

Action research for development and social change

The second, edited by Danny Burns, who heads up the Participation, Power and Social Change team at IDS, is the latest edition of the IDS Bulletin.

IDS Bulletins come in a variety of shapes and sizes – some very theoretical, others with more practical examples. This one appears to provide a nice balance of both, and has a stellar cast of leading lights at IDS on action research and participatory approaches.

Again, there is a disclaimer "we have not sought to draw firm conclusions or a single 'theory of practice'" but then a helpful identification of recurrent themes around which to hang your reflections as you read along: power and complex power relations, learning, and action.

Both these works, I think demonstrate what an exciting time it is to be working in the realm of research uptake, weaving analysis into practice, and giving us communications professionals space to reflect on the impact of our work.

I’m not a development practitioner, I’m a communications professional...

In my early days at IDS, when I had more enthusiasm than experience, I remember a conversation with a colleague in which I referred to us as “development practitioners” and she responded “I am not a development practitioner, I am a librarian”. She’s quite right, in many ways – a librarian with a whole heap of experience in international development.

I guess that description could apply to me too: a communications professional experienced in international development. Just as others are engineers, agronomists, doctors, project managers...experienced in international development.

That is, we should not forget, while we muse on power, complexity and social change, that we are also master craftsmen. Our understanding of communication, our craft, is based on an understanding of human behaviour. While it needs to be nuanced by peoples culture, worldview, literacy, all manner of contextual factors - we remain craftsmen who understand what to look for and how to build it in different contexts. It provides us with a lens through which to see the world.

Hopefully, with my bedside reading all set up now for the next month, the theory (and practical guidance) will percolate into my communications practice and I can aspire (grossly paraphrasing Plato) to being a ‘philosopher-communicator’...(albeit with less beard!)

Reflections on the K* summit: beyond K-Star-wars?

By Catherine Fisher

It was only a matter of time before someone made the KStarWars joke at the K* Conference that took place at the end of April in Canada. I’m only sorry it wasn’t me!

However, the K* Conference was notable not for its battles, but for the sense of commonality that emerged among the participants and for the momentum for future action it generated. 


The K* summit aimed to connect practitioners working across the knowledge-policy-practice interfaces to advance K* theory and practice. Its aim was to span the different sectors and contexts and different terms under which this kind of work is undertaken, for example Knowledge Mobilisation (KMb), Knowledge Sharing (KS), Knowledge Transfer and Translation (KTT).  Hence K*:  an umbrella term that attempts to bypass terminology discussions. 

This blog post provides links to some of the great reporting from the event, acknowledges some of the critiques that the event raised and points to the next steps for K*.    

• K* about process not outcome (blog link)

The opening presentation highlighted how K* is about supporting processes of exchange and engagement between knowledge-policy-practice interfaces not the achievement of particular outcomes. It was great to hear this point made by John Lavis, who has something of a guru status in K* in health. Other important points were about learning about context and what that means, not just saying its important!

• K* about shining a light on multiple sources of knowledge (video link)

Another great metaphor courtesy of Charles Dhewa. The importance of multiple knowledges, knowledge hierarchies and the role of K* actors in helping to facilitate interactions between those knowledges was a recurring theme. E.g. see video by Laurens Klerxx talking about multiple knowledges and innovation brokers. 

• Transnational comparison of K* practices (video link) 

As David Phipps explains in this video, participants from Canada, Ghana and Argentina were able to find considerable commonalities in their work with communities. This transnational comparison may be familiar to those of us who work in international development but it was a first for many of the Canadian participants who are doing really interesting work, for example, in government ministries or communities. I think this points to a strength of the K* movement in connecting people that might not otherwise talk.

• Exploring the range of K* work and how the different pieces of K* work fit together (video link)

The conference illustrated the range and scope of K* work. For example, Jacquie Brown, National Implementation Research Network who works helping communities to implement science, has learnt how this piece fits within the broader scope of K*.  For me, this seeing how different kinds of K* roles are played and how they intersect is important.  

In this video, I share some of my reflections at the time: brokering in the Canadian context including an  examples of brokering at the point of research commissioning:  power dynamics in brokering; and the way that informing role of knowledge brokering is getting a “bum rap” compared to more relational knowledge brokering work. I also get distracted by bangs, crashes and the emergence of breakfast!  

Critiques and the importance of engaging with them

The conference has generated some robust critiques. For example, Enrique Mendizabal sparked a discussion on his blog, On Think Tanks with a range of critiques including whether knowledge brokers are required, how knowledge is shared, and a critique of elitist professionalisation of this field. Scroll to the bottom of his blog post to read the responses, including mine. Meanwhile, Jap Pels argued that the nature of the debate at K* was pretty basic knowledge-sharing stuff.

I think both of these critiques raise interesting points but I think they constitute arguments For K*, not against it. K* recognises that the knowledge work is changing and proliferating, that there is considerable experience and understanding that is not shared across the different spaces in which the role is played. It aims to bring together bodies of expertise (for example that which Jaap Pels points to) to raise the game of all practitioners. It will hopefully provide spaces for debates and engagement with the kinds of critiques that Enrique raises.   

So what next for K*?

The conference generated a range of areas for further collaborative action, and plans for taking the K* initiative goes from here. 

Areas for further collaborative action included:

• Understanding impact: a group agreed to share the tools data collection tools they are already using, I’ll be participating in this group, building on work of Knowledge Brokers Forum
  
• K* in developing countries: a predominantly African group explored the particular dimensions of K* work in their contexts generating a number of action points

A group of participants gathered on Saturday to work out what next for K* as a whole. Consolidation of the K* Green Paper is considered an important next step – co-organiser  Louise Shaxson will be leading this work. There are ideas of developing a more formalised network, which will be led by UNU-INWEH in the first instance.   

UNU, who have led this process so far, remain committed and aim to get the support of the UNU governance. The World Bank has already provided financial support. Support from such international bodies is important as it will embed the international nature of this initiative, it is not without its risks!    


So to borrow again from StarWars, the force is, for now, with K*.  The scale and ambition of the initiative together with some indications of funding and high profile support suggest it has a future. However it faces both practical and fundamental challenges.

Practical challenges include maintaining ownership and momentum on behalf of the largely volunteer force taking it forward for now, identifying its niche and building connections around such a fragmented field of practice.

More fundamental challenges lie in ensuring that it really can generate value that will improve knowledge-policy-practice interfaces, rather than providing a talking shop for elitist actors.   




Catherine Fisher is a member of the K* Conference International Advisory Committee.