Lessons learned: Going above and beyond: Public engagement evaluation

− 0 min read

MPCER Workshop 3 December 2020 by Sophie Morris, public engagement officer (UK)

MPCER Workshop 3 December 2020 by Sophie Morris, public engagement officer (UK)

  • Why is evaluation important in community-engaged research?

    To improve our knowledge on the topic and techniques of evaluation, MPCER collaborated again with Sophie Morris, an experienced trainer in public engagement project management and engagement with impact. Last May, Sophie delivered a workshop series on Public Engagement Project Management (read more about the lessons learned here). 

    It was interesting to see how Sophie applied some of the evaluation techniques throughout the workshop as well. For example, she asked the participants to answer a benchmarking question at the beginning of the workshop: “How confident do you feel in your abilities in evaluating your public engagement activities?” When she asked the same question once again at the end of the workshop, she was able to demonstrate that the participants’ level of confidence had significantly grown and that the workshop had been successful.

  • Sophie also asked the participants after each exercise round how they had experienced the exercise and if they had any questions left, before moving on to the next round. She constantly encouraged participants to leave comments and questions in the chat box. After introducing several evaluation techniques, she divided the participants into several breakout rooms and asked them to evaluate the pros and cons of five selected evaluation techniques by filling in a form on a tool called padlet (you can download the PDF entitled 'Evaluation techniques - pros and cons' below).

    Why is evaluation important in community-engaged research?

    The broad dictionary definition of evaluation is the making of a judgement about the amount or value of something.

    Evaluation is particularly important in community-engaged research because researchers are looking for ways to assess the societal impact of their research project. “Public engagement is by definition a two-way process involving interaction and listening. The goal is to generate mutual benefit,” said Sophie Morris.

    Several participants explained why they consider evaluation as an essential aspect of their work:

    • “Evaluation allows us to make sure we are having an impact”
    • “Evaluation allows me to assess if what I do, gives people any benefit at all”
    • “Evaluation enables us to evaluate that we are on the right track, instead of relying on assumptions”
    • “Evaluation allows me to understand how the public views my research and if it reaches my purpose”
    • “It allows me to see what I did right or wrong, and how I can improve it in the future”
    • “It allows me to determine if a change, for example in behaviour or in knowledge happened after I executed my public engagement intervention”

    Summarising, Sophie listed the three main reasons why researchers may want to evaluate their public engagement activities.

    1. Accountability: evaluation can help us to demonstrate what we may have done, for example, with the funding that we received specifically to do public engagement. It can be a powerful way to show what we have delivered on our promise.
    2. Learning: evaluation allows us to see what went well and what did not, and what would need to change in the future.
    3. Monitoring success: evaluation allows us to see if we have achieved the intended change. It can act as an opportunity to celebrate our success. If we haven't quite reached the expected goals, evaluation helps us to reassess our project or to decide what we need to adjust in certain parts of our project. Evaluation can serve as a checkpoint in the research journey.

    What does an evaluation plan look like?

    Sophie explained that in its simplest form, an evaluation plan should have three parts to it:

    1. Which questions are you trying to answer with your evaluation?
    2. What are the best techniques to use to answer these questions?
    3. What might success look like to you in your activity or in your public engagement project?

    Evaluation can help researchers to assess whether the desired outcomes of a project or intervention have been achieved. These outcomes can be changes in behaviour, skill or attitude, and these changes are dependent on the type of public engagement activity or event.

    The participants shared some examples of outcomes that they found important to evaluate in their community-engaged research projects, such as:

    • Connecting people and building relationships
    • Increasing knowledge and awareness
    • changing perceptions
    • Assessing the need for change in the community: does the community feel a need for change, and if so, what kind of change
    • Combating fake news and misinformation, improving the public debate
    • Activating the community, increasing self-reliance and self-determination

  • Which evaluation techniques can you use?

    Sophie showed a comprehensive list of evaluation techniques [H1] that can be used in measuring the effect of an intervention (you can download the PDF entitled 'Evaluation techniques' below). She explained however that in practice, no single method on its own would give us the full answer. For example, some techniques create vast amounts of data that might be tricky to analyse and present. Sophie also advised offering evaluation techniques with opportunities for contributors to remain anonymous.

    “A combination of techniques is usually the best thing,” she said, “and these techniques could be a blend of quantitative and qualitative data collection methods, which is often the most useful way of getting a full picture of your project and its outcomes.”

    She explained that evaluation techniques can be placed on a scale with, on one end, structured and rigid techniques that give a vast breadth of answers, such as quantitative surveys or questionnaires. On the other end of the scale, we can have more unstructured data collection methodologies, that probe a little bit deeper into people's views reactions and opinions.

    “When you're making your own evaluation plans, you may want to pick a few techniques from along this scale,” Sophie said. “Perhaps alongside the surveys, you may want to schedule a focus group or two to explore some issues a little bit deeper.”

    Lastly, Sophie said that it was useful to define success by relating it for example “to the number of activities that you are going to deliver, or the level of positive feedback, a percentage change that you want to bring about.”

  • Lessons learned

    1. In community-engaged research, evaluation is critical because it allows assessing whether the research project has the intended societal impact.

    2. An evaluation plan consists of three parts:

    • What questions are you trying to answer?
    • What are the best techniques to use to answer these questions?
    • What does success look like to you?

    3. Evaluation allows to assess or demonstrate the success of an activity

    4. A combination of evaluation techniques works best

  • Evaluation Techniques

  • Evaluation techniques - pros and cons

Back to News article overview