Evidencing impacts from public engagement

Updated: May 18, 2020

Researchers are engaging with the public more than ever before, but most have no idea how they are benefiting the people they interact with. Many researchers cite lack of time or funding as reasons for not evaluating the benefits of their work with publics. Those who do seek this information, often discover that impacts are complex, long-term and difficult to attribute to their work. Commonly used evaluation techniques (such as visitor surveys) rarely capture the breadth or depth of impacts arising from public engagement, or the long-term benefits.

Despite these challenges, researchers are increasingly expected to be able to articulate the value of their work with publics, to justify funding and demonstrate the impact of their research. This guide explains how to do this easily and effectively, based on research funded by Queen Mary University of London (Reed et al., 2018).

Why is it important to evaluate public engagement?

You can do better public engagement: Evaluating public engagement is essential if we want to improve our practice in this area. Evaluation enables us to better understand the interests and priorities of different publics, so we can better meet their needs and provide them with opportunities that they find meaningful, enriching and valuable. Knowing what works (and what doesn’t) can help us choose methods and activities that will engage publics more effectively, wasting less time and generating more impacts from the time we do invest. Evaluating the design of our public engagement can help us anticipate challenges and avoid using methods that are unlikely to work or that might lead to unintended negative consequences. When things don’t go according to plan, our evaluation can give us ideas about how to get things back on track or do things better next time.

You can communicate the benefits of public engagement more effectively: Evaluating public engagement can also provide important evidence of the benefits of research, informing funders and others with a stake in research impact. There is growing interest in evaluating the positive impacts of research for society, and this includes the benefits of engaging the public with research. The UK has led the way with the inclusion of impact in the Research Excellence Framework in 2014, which has subsequently been used to allocate pubic funding to Higher Education Institutes. Australia is now trialing the evaluation of research impact in its Engagement and Impact Assessment as part of Excellence for Research in Australia, and the European Commission are looking at how to evaluate impact more effectively in the successor to its Horizon 2020 funding programme.

What should I evaluate?

Evaluate the delivery and immediate outputs of engagement: Most evaluations of public engagement focus only on evaluating the delivery of an event or activity and its immediate outcomes. Indicators may be identified and tracked using a range of techniques that can quickly and easily tell you if you are getting the outcomes you expect.

However, if public engagement is as much about improving practice as it is about communicating benefits (see previous section), then it is just as important to evaluate the design of your public engagement. In particular, it is important to evaluate the extent to which the design follows known good practice principles, is adapted to your particular context and is underpinned by sound ethics. A well-designed public engagement process should typically:

  • Identify publics and stakeholders systematically, for example using the Fast Track Impact public and stakeholder analysis template

  • Understand and manage the expectations of these groups

  • Deliver tangible benefits that will be valued by each group in ways that are sensitive to their social and cultural context

  • Identify risks and assumptions and be prepared to adapt to changing circumstances

  • Engage experienced personnel who can manage events, facilitate workshops and organize engagement effectively

Evaluate the impacts of public engagement: Evaluations that focus only on the delivery of activities on the day and the immediate outputs of engagement often fail to articulate the broader, deeper and longer-term benefits of public engagement. However, evaluating the impact of public engagement can be challenging and time-consuming. To make this easier, the next section breaks this down into a series of simple tasks.

How do I evaluate the impact of public engagement?

To evaluate the impacts of public engagement, you need to know what specific impacts you’re looking for, design your public engagement to achieve these impacts, look for evidence that planned impacts are arising at the time and longitudinally, and be prepared to capture additional unanticipated impacts as they arise. Follow these steps:

  • Identify specific and measurable impacts that you would like to see as a result of your public engagement – consider what you might want to achieve in terms of changes in:

  • Understanding and awareness e.g. people understand an issue better than they did before

  • Attitudes e.g. a new appreciation for alternative views and more positive perceptions of people who hold different views

  • Behaviours e.g. more tolerance towards people who hold different views, more engagement with issues via other activities

  • Culture e.g. a change in the nature of public discourse around an issue

  • Capacity or preparedness e.g. access to knowledge or resources that were previously unavailable, which can be used at a time of need

  • Design your public engagement to achieve these impacts: build these goals into a logic model linking goals to specific activities that are tailored to the needs and interests of different publics, with activity and impact indicators you can use to track progress – see the Fast Track Impact Planning Template

  • Look for evidence that planned impacts are arising at the time and longitudinally: while you’re doing public engagement, design your data collection to look for specific planned impacts, and evaluate whether these impacts are arising longitudinally through planned follow-up with participants. For longitudinal work to be possible, you will need to incentivize participants to provide you with their contact details e.g. via a prize draw or joining a mailing list that provides additional free benefits or opportunities

  • Collect additional evidence of unexpected impacts as it arises, ensuring that evidence is safely stored somewhere for later reference

What sort of evidence do I need?

The sort of evidence you will need to convincingly demonstrate impact will differ from project to project. Play “devil’s advocate” and ask yourself what you would need to do to convince someone who does not believe that your public engagement has led any sort of impact.

In some cases, to be convincing you will need quantitative evidence e.g. an increase in visitor numbers and museum revenues after the installation of a new exhibit based on your research. In other cases, qualitative evidence will be more convincing and appropriate, for example illustrative quotes describing how engagement transformed people’s attitudes towards an issue or group of people.

Quantitative evidence might be most appropriate when:

  • You need to demonstrate cause and effect or a relationship between two things e.g. the number of students choosing to study a particular subject at University after watching TV programmes based on your research, or attributing a sales spike in a product to public engagement with your research on its health benefits

  • You need to describe exactly how much something has changed (or is likely to change) as a result of your public engagement e.g. awareness of an issue, or the prevalence of new behaviours

Qualitative evidence might be most appropriate when:

  • You need to explain why public engagement led to observed changes e.g. by challenging assumptions or attitudes that were underpinning previous behaviours

  • You need to represent a range of perspectives, showing how different groups have benefited in different ways from public engagement e.g. differences in the benefits that mothers, fathers or children derive from a public engagement event that targets families

What methods can I use to collect evidence of different types of impact?

Different types of impact often need to be evidenced in different ways. Decide what impact you want to see, and then based on this decide whether you need qualitative, quantitative or mixed methods. For example to evidence: