, ,

The Problem of Measuring Impact

Impact investors are rightly applauded for the many gifts they undoubtedly bring to the social sector, but what if some of these market-based, social enterprise style gifts, turn out to be unexamined Trojan horses containing ideas that perpetuate the problems we are attempting to transform?

Over the last decade, market-based strategies and social enterprise have come to intellectually dominate philanthropy and what was previously known as the community sector. “The Social Sector” that has arisen in its stead is the result of an exciting cross pollination of traditional community sector approaches with the business world. Business investors have successfully funded projects of enormous scope and scale for decades (think oil rigs, highways, and hydro-electric stations) – surely if they bring all that get-up-and-go to social issues then we will finally have the kind of impact we have always dreamt of, right?

The experience of One Health Organisation (OHO) over the last decade and a half suggests that this all depends upon how you define impact, and on whose terms you measure it. The community sector has implicitly & culturally defined impact qualitatively: equality, empowerment & human rights are the kinds of hard-to-measure ‘deep’ impact it sought.

The business world, by contrast, understands numbers and defines impact accordingly: “If you can’t measure it you can’t manage it,” as all CEOs know. The issue then becomes quantifying the qualitative. Academic articles abound in appropriate methodologies to handle this issue in various disciplines and fields. Yet many continue to voice concerns  that venture philanthropists’ focus on quantifying impact may unwittingly obscure the less tangible, but equally important, qualitative goals such as equality, community and empowerment. As the philanthropy analyst Michael Edwards has asked:

‘Would philanthro-capitalism have helped fund the civil rights movement in the US? I hope so, but it wasn’t “data driven”, it didn’t operate through competition, it couldn’t generate much revenue, and it didn’t measure its impact in terms of the numbers of people who were served each day. Yet it changed the world forever.’

Learning from the Health Sector

In the health sector a similar issue arose when critics of the Primary Health Care-a movement that advocated a qualitative emphasis on the concept of health, argued that promoting health itself just wasn’t practical, and they proposed a “Selective” approach to PHC instead which focused on selective disease markers that could be more easily and economically enacted and monitored. Almost half a century has elapsed under this approach, with many public health authors continuing to affirm that this is the very approach the founders of the PHC movement sought to remedy. Since the WHO has continued to define health as “a complete state of physical, mental and social well-being, and not merely the absence of disease or infirmity” however, the problem of M&E continues to plague public health debates.

One of the hard questions that we as a sector need to ask ourselves is whether current funding practices are unintentionally rewarding ‘good’ M&E rather than good projects in the same way that standardised educational testing rewards students good at doing tests, whilst too many good students fall through the gaps?

What impact is the focus on measurement having on the on-the-ground work of social change?

The recent trend in measuring impact that has come off the back of the social enterprise revolution generally pushes towards goals and impact that are measurable within a single funding cycle in order to qualify for the next round of funding. Whilst this makes sense, the most needed and exciting ventures are attempting to tackle deeply entrenched issues that require different timescales.

Understandably investors have sought more tangible KPI’s than purely values-based assertions, however it may well be that it is this very desire to ‘see’ what are often in the short-term tangible results that subtly shifts deployment of resources within the sector away from difficult-to-measure things like human rights towards more quantifiable outputs like the number of sachets of rehydration salts distributed.

Is there a solution?

To avoid this, measurement and evaluation of impact should ideally be a part of a dialogue and bi-directional skills transfer process between investors and community service providers alike, with both parties contributing their expertise on how to best create appropriate and impactful M&E strategies. At the core of these discussions should be the recipients of the project themselves; ultimately it is the target community that most needs to understand (in appropriate terms) whether the project they participated in had a real impact on their wellbeing.

What this analysis suggests to us is that whilst market-based approaches to social problems indeed provide many wonderful opportunities for our sector to increase the scope of our activities to match the scale of the issues at hand, they need to be carefully adapted to ensure they are fit-for-purpose. The pivotal point that OHO has identified is that in M&E, it is the assessment process that defines a recipient-centred approach. Unless the recipients themselves are empowered to sit in a central position in developing appropriate measurement guidelines, the M&E can come at the cost of the social impact it is trying to measure.

– By Jimi Wollumbin.