Evidence paradox
March 2, 2021
There’s something called the ‘evidence paradox’ that anyone who has worked in the community sector for any length of time will have bumped up against if not necessarily recognised as such. It arises at that point where you conclude that all you need to do is ‘tell your story’ more compellingly to convince the doubters of the merits of your work. Do that and the walls of resistance will surely crumble. But they don’t. That system which has proved so resistant to your charms, is looking in the wrong place for the evidence. Interesting new report from New Local.
Author: Jessica Studdert, The New Local
Community power produces far-ranging benefits. But an ‘evidence paradox’, hard-wired into our policy-making system, is holding back its potential. As we launch our latest report, Community Power: The Evidence, co-author Jessica Studdert addresses the counteractive way we measure success, why this matters and how to change it.
So you think decisions would be better made locally, not at Westminster? And that communities are best placed to know what would make them thrive, rather than professionals? Prove it.
This is the challenge set by those who hold power within our current system, to anyone who imagines that it could be distributed more evenly across the country. It is the gauntlet laid down by people with vested interests, to dispute the idea that communities know how public services can best support them, rather than the big state or big markets.
And it’s an impossibly high bar. Our new report Community Power: The Evidence demonstrates the wide-ranging positive impacts community-led approaches have, on everything from individual wellbeing, to trust in democracy and reducing demand on services.
But the burden of proof for reform that would disrupt concentrations of power essentially requires demonstrating a counterfactual. It would involve setting up a parallel universe in which power was more diffuse, which would then allow a comparison of outcomes with a centralised model, while controlling for all other variables.
Of course, we can only operate in the real world. And when dealing with human life and complex circumstances, the efficacy of a randomised control trial is unattainable. And yet our system of policymaking holds out for evidence with that level of rigour. What is meaningful to people is often qualitative – relationships or wellbeing, for example. Yet what is required to inform a case for change is largely quantitative – focussed on measurable service outputs. This disconnect between value and metrics is systematically overlooked. And until anyone can establish water-tight proof that a different version of our present reality is better, the status quo holds.
This ‘evidence paradox’ holds back the potential of community power. It is trapped in an impossible situation whereby it is required to prove its worth according to measures that are not set up to recognise its value.
Our research reveals that there is a wealth of evidence that community power initiatives and practice have a real impact on our lives. Whether by having say over resource spending, working as equals with professionals or by taking direct responsibility for assets, communities are playing increasingly proactive roles within our system of public services.
And there is proof that these approaches generate tangible outcomes. Asset-based community development in Ayrshire that sought to strengthen social ties led to an increase in mental wellbeing and a reduced reliance on clinical care. Using deliberative and participatory methods is improving trust and legitimacy. Pioneers in Camden, Barking & Dagenham, Peterhead in North East Scotland and internationally in Poland and Belgium are shifting the power dynamic between people and their governing institutions. And from the perspective of public services, approaches that start with the community can save costs and boost prevention as evaluations of Local Area Coordination have shown.
Although these impacts are real and keenly felt by people, as evidence for policy change on the terms of the current system, they only take us so far. Traditional policymaking is heavily influenced by the evaluation framework of the Treasury Green Book, which requires a particular type of evidence to inform decisions about resource allocation. This is largely data-driven and at scale, in order to inform business cases for shifts in investment and expected outcomes. Of course, a significant onus is on reducing spend. Although savings are only narrowly defined as in-year and within a service silo, knock-on costs in the future or in other services aren’t fully captured.
And yet, as the grid below shows – the nature of community power is almost diametrically opposed to the narrow scope for value our current system recognises. By definition, community power initiatives are small scale and adaptive to particular sets of circumstances, so they are hard to replicate elsewhere. They require a system that is comfortable with pluralism, rather than one which forces standardisation. They may produce a range of benefits which impact on wellbeing and create resilience with less need for professional support. But such potential to bring about a system-wide shift towards prevention isn’t captured by a framework which can’t quantify what doesn’t happen.