Predicting the future: The challenges of evaluating preventative impact

In our Investigating our Impact blog series, we have explored different ways we think about ‘impact’ at West London Zone: 

1. Exploring how one size doesn’t always fit all
2. Explaining our early years qualitative developmental evaluation with The Centre for Education and Youth 
3. Developing our evaluation process to support a higher standard of evidence.

These posts describe a range of methodologies that articulate change that has already happened. But as an early intervention initiative, our work is predicated on the assumption that doing something now has a positive impact in the future. Therefore, we want to tackle something more nebulous yet complex: how do you go about ‘proving’ something that hasn’t happened yet, or which – because of positive action taken now – is much less likely to happen? In this blog, we will explore:

• What we mean by 'preventative' impact
• Why we think it is important
• What methods we are developing to understand it 

There is a basic logic to prevention: for example, we know that if we exercise regularly and eat well, we will generally reduce our risk of developing negative health outcomes later in life. We can also develop an individualised sense of it: by adulthood, hopefully we will have identified certain behaviours that we try to maintain (or negate) to prevent negative impacts on our wellbeing.

There is an extensive body of research that shows if a young person exhibits certain characteristics – including low attendance, attainment or social-emotional wellbeing – then they are ‘more at risk’ of experiencing negative outcomes in the future, such as not being in education, employment, or training (NEET), experiencing mental health challenges or becoming involved in antisocial or criminal behaviours. But for some children and young people, there is not an even playing field to figuring out preventative behaviours - they might already be grappling with multiple social, health, or educational challenges – and so need additional support to ensure that these ‘risks’ don’t develop into full-blown crises, which can be much harder to recover from, and that they enter adulthood ready to positively take on life.

We estimate that as many as 20% of young people in our Zone are at risk of experiencing one or more of these outcomes without additional, timely support. Since 2016, we have worked with over 4,500 children and young people to provide exactly that. We have seen positive outcomes for them in this time in their social and emotional wellbeing and their school grades.

Yet this kind of data only tells part of the story. It can be easy to fall into the habit of listing off these kinds of metrics without making the more fundamental meaning behind them explicit, which is: and therefore, there is good reason to think these young people will have every opportunity to be the adult they want to be.

Why is this important?

There has long been a consensus that ‘prevention is better than the cure’ – yet actually funding and delivering services that emphasise prevention remains very difficult, especially in an environment where resources are scarce and return on investment needs to be immediate, or at least made obvious before the next election.

This is more important than ever. Since 2010, combined spending on early intervention services, such as family support services and services for young people, has fallen by 46%, while total expenditure on late interventions, like youth justice, safeguarding and child protection, and children in care, has risen by almost half (47%). In 2021-22, £4 of every £5 spent by Local Authorities on Children’s Services went on ‘late’ interventions.[1] Not only is this hugely financially inefficient, being permanently in ‘crisis mode’ generates the worst outcomes for young people and their families. 

We know that our work is generating long-term positive impact because we hear it in the stories of the young people on our programme time and again, and the testimonies from our school leaders who have excellent understanding of the trajectories that are likely to have been averted through our work. We can see it when a young person who was struggling academically achieves their English and maths GCSEs, or a child struggling with isolation and anxiety leaves school with a strong and supportive friendship network, or when a young person building up their self-belief and confidence steps up to speak in front of their classmates.

At WLZ, we believe that if we can provide evidence that prevention really works, then we can also help make the case for a different approach to supporting children and young people right across the country. It is therefore vital that we can articulate the full impact of the support we deliver, beyond end-of-programme metrics.

What methods are we developing to understand this?

In addition to the more immediate child-level impact previously mentioned (e.g. improvement in grades or mental health, measured at the start and end of a two-year period) we are seeking to develop methods that enable us to demonstrate this preventative impact to our various audiences. Some of these include: 

• More targeted data sharing with Local Authorities: since 2022, we have agreed with our Local Authority partners that they will do a ‘cohort matching’ exercise, to understand how WLZ children engage with their services. Tracking this over time, we would hope to see that once these children start working with WLZ they do not re-engage with services.
• Estimate of future financial savings: we have been working with ATQ Consultants to identify the ‘financial savings or economic benefit’ created by our work. So, for example, if someone who was struggling at school goes on to get their English/Maths GCSEs, their lifetime earnings will be much higher; likewise, if we address a mental health issue early on, it is likely that person won’t need to access a more intensive service down the line – this is a ‘saving’ both to the person and to the state.
• Case studies and testimonies, showcasing a range of different experiences and voices over time – not only young people but families, schools, and community groups/organisations.
• Deep dive ‘working groups’ on attendance and school exclusion, with additional activity happening in a select number of schools to build specific learning and insights.

Evidencing preventative impact is complex: our approach needs to be innovative, curious and wide-ranging. But if we get it right, we can maximise the ability of moving investment downstream and into the preventative sphere – and enable young people to access the support they need at the tipping point that they need it.

[1] Jon Franklin, Jack Larkham and Mariam Mansoor (2023) The well-worn path Children’s services spending 2010-11 to 2021-22, https://www.probonoeconomics.com/Handlers/Download.ashx?IDMF=64274c2e-73c3-4364-b995-4b31b6825dd8