What’s Missing in Impact Measurement? Lessons from AVPN 2014

The Asian Philanthropy Forum team is pleased to provide media partner coverage of the Asian Venture Philanthropy Network Conference in Singapore. The post below focuses on impact measurement. See our additional posts on the conference here

Impact_Measurement_Workshop
Jeremy Nicholls of The SROI Network facilitates a workshop at AVPN 2014.

 

Two of my favorite sessions at this year’s AVPN conference were workshops on measuring impact, facilitated by Jeremy Nicholls, CEO of the SROI Network, and Terence Yuen, Project Officer at the Center for Entrepreneurship at Chinese University of Hong Kong (CUHK).

Impact measurement is still a growing practice in philanthropy and social investing. Until recently, much philanthropic giving was like taking a shot in the dark — donors would give to organizations they trusted and hoped for the best, maybe even receiving a few carefully curated stories of successful beneficiaries in progress reports tracking how their money was spent. It was difficult to tell what kind of difference their funding made on a systematic or quantifiable level. Impact measurement took off in response to philanthropists’ desire to better understand the true value of their support and to social purpose organizations’ need to evaluate the efficacy of their interventions. It’s a noble and necessary endeavor.

However, perhaps in an effort to play catch up after years of not measuring, or perhaps out of a desire to appear current with trends in tech and businesses sectors, conversations on measuring impact have begun to skew too far on the technical side, leaving out all too important discussions on how measurement brings applied value to the sector. As Jeremy Nicholls put it, we’ve started to focus on the process of measuring without thinking about what the end results even mean.

Why do we measure?

Consensus on why we even collect data isn’t exactly straightforward either. While we can all agree that it is worthwhile to know the difference our funding and programs are making, different agendas are at play too. When Jeremy asked the simple question, “Why do we measure?” there were a range of responses. Some workshop attendees candidly admitted that nonprofits and social enterprises need to measure impact in order to attract more funding — there’s a validation motive that could potentially be at odds with the evaluation motive. Others pointed at that from a state perspective, governments want to know whether they’re getting value for their money. And if you’re looking at things through the filter of a funder, you’d probably want to measure impact so you can decide where to invest and where to divest. We all have different objectives, but we’re all trying to use the same language.

What can be most disconcerting, though, is that in trying to manage all these different objectives, we often forget to take into account the beneficiary or client perspective.

This was underscored in a thought exercise Jeremy had us do during the workshop. Workshop attendees were split into six groups: social enterprise managers, social enterprise board members, investors offering a loan, investors offering a grant, auditors, and beneficiaries. We were given a hypothetical situation wherein a social enterprise runs programs to help ex-offenders gain employment. Jeremy then asked, “What questions would you want answered to know how much difference has been made or how much value has been created?”

What questions would you want answered to know how much difference has been made or how much value has been created?

After convening and brainstorming in our different groups, we presented our responses. Naturally, in our different roles, we all asked different questions. Some of us wanted to hear business plans and some of us wanted to hear recidivism rates — it made perfect sense given our unique perspectives.

But then there came a moment of realization: after all the groups had all presented their questions, Jeremy pointed out that no one, not a single person, got up to ask the beneficiaries what they believed would bring value to their lives. How can we say we’re creating impact when we don’t even bother to ask beneficiaries what impact means to them?

When we fail to take into account the beneficiaries’ perspective, we fail to account for failure. In other words: Impact metrics are almost always focused on measuring positive changes; rarely do they look at what happens to beneficiaries if programs or projects fail to achieve their goals.

For example, to continue with the ex-con employment social enterprise scenario from the workshop, it would be swell if an ex-con goes through the social enterprise’s programs and is able to find work, but what about the ones who go through the programs and still can’t find steady employment? It’s very likely that this would lead to further demoralization and stigmatization of the ex-con. The ex-con could end up worse off than had he or she never entered the social enterprise’s programs in the first place. How do we account for that?

Once we recognize that measuring impact should not be about looking at a single desired outcome, but about understanding the range of outcomes associated with a project or intervention (be they positive or negative) we can begin to design metrics in a way that brings actual value to practitioners, funders, and beneficiaries alike. We can use data and information to make decisions about whether to strengthen a high-performing program or discontinue a underperforming project, planning exit strategies as necessary. We can make adjustments to existing programsto focus on our core competencies. We can hold ourselves accountable to the beneficiaries we intend to help. Ultimately, impact measurement should be an uncomfortable process that prompts reflection and action, not self-congratulation and adherence to an existing status quo.

Here are a few additional tips Jeremy Nicholls had for making impact measurement matter:

  • The accuracy and rigor applied to the measurements should be proportional funding. Don’t waste money and time on things that don’t influence decision-making, and don’t overburden grantees with needless reporting.
  • There are many different impact measurement frameworks, designed to evaluate different things. Think about which works for you, and know that the different frameworks are not alternatives to one another. You may need to use more than one framework. Examples of frameworks include: social return on investment (SROI), randomized control trials (RCT), cost benefit analysis (CBA), Sustainbility Reporting Framework, etc.
  • Look at causation and correlation. Your intervention is not the only activity going on in a beneficiary’s life. If you don’t ask, “Was all this change caused by us?” you may end up over-claiming your impact.

Leave a Reply

Your email address will not be published. Required fields are marked *