Evidencing Impact from Media Engagement

January 7, 2020

Many of us think that media engagement and impact go hand in hand, but this is not necessarily the case. Engaging with social and mass media can be one of the most powerful pathways to impact, but because the effects are so hard to measure, it can be challenging to evidence impacts. I meet social media and television stars, researchers who have had documentaries made about their research, and experts who appear on news channels and front pages around the world multiple times per year. But even at these impressive levels of media exposure, I am often the bearer of bad news: “Great, you’ve got global reach and you’re famous”, I tell them. “But until you can tell me that someone somewhere actually understood something that you said, and ideally as a result of that that, they benefited in some way, then your impressive reach has very limited significance or meaning”.

 

This is a real challenge. How do you know that anyone actually was listening, let alone understood what you said in that media interview? If they understood something, then how do you know that it was important to them in some way? For example, do they understand something new, are they now aware of an important issue, has it changed their attitude, or have they done something differently as a result?

 

Many researchers see it as their responsibility to respond to every media request their press office sends their way. They do the media interviews and give little thought to what happens next for those who listened. Of course, your press offices can give you lots of metrics around circulation numbers, viewer numbers and social media metrics, but this only gives you evidence of reach. If you have no evidence of significant impacts arising from your media engagement, then all your reach has little meaning. You might have reached every person on the planet, but if they didn’t understand a word you said, what was the point?

 

 

Plan for your impact and how you will evidence it at the same time

 

What What you need is some kind of plan to evidence whether your media engagement actually makes a difference or not. Just doing more and more media work, unless you have a plan to evaluate it, may not be a great use of your time. Clearly there are reputational benefits as you get more widely known, and being visible may mean that you get opportunities for impact that you would otherwise not get. Your university will certainly want you to say “yes” to the media opportunities they give you, as it increases their visibility too, and has benefits for them in terms of their reputation, and press offices are driven by quite different metrics from researchers.

 

The number of people hearing about research from your institution is a key metric for press offices. But in addition to knowing how many people were reached, researchers need to know whether it actually made a difference to those people. To do that, you need to have an impact plan and as part of that you need to think about how you might evaluate the significance of the impacts you get through media (or any other form of) engagement.

 

Logic models are a fast and powerful way of planning for impact in ways you can evidence. By thinking ahead with a tool like this, it is possible to identify things you would expect to see on the pathway to impact (milestones) and consider indicators of impact or other evidence that might show when expected benefits have been achieved.

 

Read my guide to impact planning and download the Fast Track Impact planning template here

 

1. Identify impact goals for specific beneficiaries 

 

The first step in this tool is to identify specific impact goals and the group who will benefit from the achievement of those impacts. An impact goal should always be a benefit. Based on that definition, reaching people, “dissemination” or social media metrics are not impact goals, because there is no way of knowing whether anyone who has been reached in these ways actually benefited as a result. For all you know, they may have been unable to understand what reached them, misunderstood it or been bored or offended by it. You need to engage with the media with a clear impact goal in mind, for example, you may want to: 

  • Raise awareness about an important but little-known issue 

  • Enable people to understand something or change their thinking about 

  • something important 

  • Change opinions and attitudes that are limiting people or causing harm 

  • Mobilise action or change behaviours in individuals or households 

  • Influence decisions by people with power to affect broader positive change 

 

2. Design media activities to reach those goals for those beneficiaries 

 

Now you know what you are trying to achieve, it is possible to harness the media in very different ways that are specifically designed to deliver those outcomes. For example:

  • You may turn down an opportunity for an interview with CNN International and spend the time writing a press release for Farmers Weekly if you need to raise the awareness of farmers about a threat, technology or action they can take based on your research 

  • You might plan a workshop or open day for farmers and integrate that into your article to advertise the event and get greater engagement with the group 

  • Alternatively, you might link the article to a new online toolkit or other resource that enables wider engagement with the group across the country 

  • Then, you might use the coverage in Farmers Weekly as a route into getting a speaking slot at the Royal Welsh Show 

  • While you’re there, you might arrange to do an interview with the Farmer’s Guardian at the show to get wider coverage across that community 

  • While you’re doing this, you might work with your press office to get the coverage on social media, and target influential accounts who engage with large numbers of farmers and hashtags like #agrichat to get wider readership and funnel social media readers to your workshop/open day, online resource and show event 

  • At each of the events and via your website, you would incentivise people to leave their contact details so you can follow up with them long-term to provide more benefits and ultimately be able to follow up with a short questionnaire to evidence your impact, all GDPR compliant 

 

This highly targeted approach to media engagement stands in contrast to the usual approach which tends to maximise reach without being particularly sensitive to the audience. Raising awareness amongst millions of members of the public about something farmers need to do may have little benefit if few farmers actually hear or understand the message. 

 

Having said that, there is still a role for media opportunities that get you significant coverage, as the more prominent the coverage the more likely you are to include your target groups. You are also like to reach additional groups that you may not previously have been aware were interested in your work. This can be an important way of extending the reach of benefits from your research. However, you have to be ready to walk through the doors that open to you after a major media opportunity, so it is important to make sure that you are easily discoverable via a Google search and have time in your schedule to follow up with the opportunities as they arise.

 

3. Identify indicators you can use to evidence impact 

 

Evaluation is built into my impact planning template. As you consider what impacts you want to see (point 1 above), and the kind of media engagement you’ll need to get those impacts (point 2 above), you need to consider how you’ll know if the media engagement is working, and whether or not it is generating the benefits you’re aiming for. Press offices are good at collecting data on the success (or otherwise) of your media engagement, collecting all the coverage and telling you the circulation and viewing figures, and social media metrics. Increasingly there are a range of other altmetrics that you can get around your media engagement that will tell you more about the number and types of people who are talking about the work, and where they come from. These kinds of metrics can be a useful starting point for evaluating the significance of your impact. However, in isolation all this data really tells you is your reach. 

 

This is a key weakness of traditional media monitoring, because without evidence of any benefit for the millions of people you reached, it is difficult to argue that there is in fact any impact if you use technical definitions of impact (e.g. from REF) or academic definitions of impact, which all emphasise the significance of benefits as well as their reach. As a result, there is widespread scepticism across the impact community of the value of media engagement as a pathway to impact, and there are increasing reports of academics being told by impact teams to re-prioritise their time away from engaging with their press office, towards more targeted approaches that can generate evidence of significant benefits. This is unfortunate, given the potential scale of benefits that can arise from well-managed media engagement that has been embedded in a well thought out pathway to impact. Based on the points above, you have now created a plan that uses media engagement in a much more targeted way. Now, you just need to make sure you can measure the significance of those benefits in addition to their reach. 

 

The logic model approach makes this easy, because it asks you to identify indicators of success, which do not require any specialist expertise. If your impact goal is to change people’s awareness of the benefits of maggot therapy and shift attitudes to reduce levels of disgust and increase the likelihood that a patient might accept the treatment on the NHS, then you know exactly what you are looking for as an indicator of impact. In this case, you would expect a sample of people exposed to your media coverage to have different levels of awareness and changed attitudes before and after engaging with the coverage, and you would then ultimately expect to see an increase in uptake of the therapy in hospitals. Developing indicators like this does not require any social science expertise – just common sense. Ask yourself what success would look like, and then ask how you might be able to evaluate that. 

 

In some cases there might be things you would be able to quantify, and in other cases you might expect people to say certain things qualitatively if your work had made a difference to them. What methods would you then need to collect that data? 

 

The final section of this guide considers a non-exhaustive range of options for evaluating the significance of impacts arising from media engagement, ranging from accurate but expensive methods to cheaper methods that will take more of your time, but that can still provide robust evidence. 

 

 

Methods for evidencing the impact of media engagement 

 

Methods for evidencing impact from media engagement range from highly accurate but costly methods to methods you can use for free if you plan ahead and have enough time. All the methods I’m going to suggest have three things in common:

  1. They demonstrate the significance of impact by identifying and describing the benefits that arise from media engagement 

  2. They use a sample of the population who engaged with the media. It isn’t possible to survey every person who listened, watched or read about the research, but it is possible to find out if a sample of those people benefited in any way. The more expensive methods work with statistically robust samples, but if you are evaluating impact on a tight budget it is possible to triangulate your findings using other methods, so that you can still say something with confidence with a biased sample 

  3. They demonstrate cause and effect between the research and the benefits, so any impacts claimed are clearly attributable to the research. There are two types of causation you might want to try and prove. Necessary causation shows that the research was necessary to generate the impacts that arose via the media. Sufficient causation shows that the research could in theory have generated the impact, and you then need further evidence to build an argument that despite other factors also being plausible (or even probable), the research did indeed play a significant role. A third, weaker form is contributory causation, where the research may have been one of many contributing factors, and it is not possible to disentangle these competing/confounding factors to show that the research played a significant role. 

In this guide, I will suggest three ways of evaluating impacts arising from media engagement, though there are many alternatives available.

 

 

1. Social media analysis

 

The first approach is inexpensive and may not take too much time (depending on the sort of analysis you do), but has some important limitations. Social media analysis might sound like a daunting prospect if you have not done it before, but it can be surprisingly accessible: 

  • Quantitative content analysis can be used to count the frequency with which particular words or phrases appear in media (and trends can be tracked over time, looking for peaks that might correspond to public engagement activities). Similarly, this technique can be used to characterise a body of text that is known to relate to the public engagement that is being evaluated (e.g. newspaper cuttings or tweets on an event hashtag), based on the frequency of words within that body of text. 

  • Although more time consuming, qualitative analysis of text that has been aggregated using a hashtag or keyword search can offer more nuanced insights into the nature of debate stimulated by public engagement. Changes in the amount and nature of discourse may be tracked over time. 

  • Evidence of reach may be gathered for particular messages (e.g. numb