How can we measure the impact of our policy work?

For years now, Citizens Advice have published an annual Impact report, showing how our advice work makes a difference in people’s lives. It covers all our different services — how many problems we solved, how much our work saved the public purse and put back into people’s pockets. We’re really proud of the impact we make and the steps we take to ensure we’re measuring it correctly.

But Citizens Advice actually has two aims as an organisation — to give advice to people who need it and to advocate for change. We’re good at measuring the impact of the first part, but what about the second?

Over the last year we’ve been working to come up with a way of doing that. We looked for an existing model we could adapt for our purposes but there wasn’t really one out there. So here’s a short explanation of how we developed our own approach.

The total impact of our work

We generally try and calculate the total effect of the changes we advocate for. A big part of the research that goes into our influencing work is setting out how many people will be affected and to what extent. Often, we campaign for material improvements to people’s lives, and we can put a price on those improvements.

In some contexts this is harder than others. For example, if we campaign for an increase in benefits, it’s straightforward to calculate the benefits of that. It’s a bit harder to quantify the cash benefits of, for instance, changing debt collection practices.

So where we can do this, we do it already and use it in our influencing work. But calculating the total effect is only one part of assessing our impact. We need to know how much of it was down to our activity. And that’s where things get a bit complicated.

Advice, advocacy and impact

In a lot of ways, advice giving and advocacy work could not be more different. In advice, we work closely with one individual or family to understand their problem and help them solve it. There may be other people and agencies involved in solving the problem. However, we know that the individual came to us for help, so our role in helping is quite clear.

In our advocacy role, we are working to influence a range of stakeholders and decision makers. We want them to make changes on behalf of our clients and people like them. Those decision makers have their own positions, agency and motivations. We are rarely the only voice trying to convince them of a particular course of action.

So even if ultimately we get the change we advocated for, it’s not obvious how far that was due to our work. If the policy change was likely to happen anyway, it seems clear that our impact was lower than if the decision makers needed a lot of convincing. If we were one of many organisations pushing for change, our impact is lower than if we acted alone.

Mapping out the problem

Fortunately we already use some tools that can help us unpack this problem. At the start of each project, the advocacy teams draw up a map of stakeholders that we would want to influence. It’s actually a grid that looks like this. We plot the positions of stakeholders according to how favourable they are to our suggestions and how much power they have to make changes.

We can start to understand our potential impact based on the position of stakeholders on this grid. If, by the end of the project, stakeholders have moved over to the right and towards our position, we have created impact. If those stakeholders have the power to make the change, we have created more impact.

Working with others

This ‘distance travelled’ measure forms one part of the calculation. Another part is assessing our contribution relative to other campaigns or organisations. This is a bit simpler — if we acted alone, our impact gets greater weight. If we were one of many voices, the impact is less, especially if we were not in the lead.

How we are using the model

So far we have used the model for a few times retrospectively as part of the wash up we do at the end of projects. We’re now starting to embed it from the start, repurposing the stakeholder maps so they can be used to calculate our impact better.

One concern about using this kind of approach is that we might end up focussing on the areas with the biggest aggregate impact, and ignore smaller instances of acute need. To make sure that doesn’t happen we make sure that at least one of our policy changes each year will have a positive impact on groups who experience intense disadvantage, detriment or harm.

More generally though, we are not using this approach to set targets, or even to set direction. Its purpose is to inform us better about where we are successful in our policy work, and where we might need to put our resources.

We developed this approach ourselves as we couldn’t find an ‘off the peg’ version we could adapt to our own work. The approach should be relevant to other organisations, though. We’d be really interested in hearing experiences from other campaigners and policy teams and are planning an online workshop later in the Autumn. If you would be interested in participating, get in touch — tom.macinnes@citizensadvice.org.uk