August 3, 2022

Measuring social impact: Who decides what counts?

Arun Maira speaks with Hari Menon and Vineet Rai about how the social sector and the investing world must move beyond numbers when it comes to measuring social impact.

6 min read

Social impact organisations operate as a crucial link between the government and its policies and the people to ensure they have access to basic rights, such as education, water, healthcare, and safety. However, while most of these organisations agree that positive social impact is the goal, they may differ on its definition and how best to create it.

On our podcast ‘On the Contrary by IDR’, host Arun Maira sat down with Hari Menon, the India office lead of the Bill & Melinda Gates Foundation, and Vineet Rai, founder-chairperson of Aavishkaar, to talk about what social impact really means, how it should be measured, and what needs to change in how we think about it.

Below is an edited transcript that provides an overview of the guests’ perspectives on the show.

Defining social impact is complicated

Vineet: This is a debate I’ve had since the time we coined the term ‘impact investing’. The impact that I define for myself is very different from the impact that my investor sitting in New York sees, and [it] is very different for the person I’m trying to impact because their aspiration for impact is very different.

donate banner

So [for instance], whether you go to Reliance or the Tatas, or anybody—they are making some positive impact, some negative impact. We are probably trying to talk about the summation of the negative and positive and see whether there is an end result which is positive, and then you can call it an impactful enterprise.

Acknowledging that what we are doing is probably efficient but less impactful is probably the first way to go.

For example, instead of setting up a factory and trying to actually manufacture shirts, are you going to rural India, working with rural artisans, which is a far more cumbersome, complex process, and getting shirts stitched in the villages and then bringing them to the urban world to sell them? Both approaches are creating impact. But in one, you are not really trying to move all the workers to one place and creating a new ghetto to get a shirt stitched and create livelihood. In the other, probably what you are doing is you are taking livelihood [to] where people live, so you’re not displacing them, and in the process are actually allowing a local ecosystem to thrive. And I think it’s very difficult for a large number of people to actually give this context to the idea of impact, especially when we are talking about impact investing. And, therefore, it has been simplified to create the ghetto because it’s far more efficient, far better. And this is where the math comes in. So acknowledging that what we are doing is probably efficient but less impactful is probably the first way to go.

When we think about impact, we need to factor in how systems work

Hari: Our foundation is driven by that line: ‘All lives have equal value’. So, at one level, the impact we try to track is: Is the work improving the well-being of the communities you’re trying to serve?…Our approach to serving these communities is often through what are seen as vertical programmes. We work with partners or we work with the government on health interventions, on nutrition, and financial inclusion, on sanitation, on agriculture-based livelihoods, increasingly also looking at gender equality as a bedrock. And what we find is that you can get locked into a particular programme and the impact metric for that. So, for example, with vaccines…there’s a very clear metric which is obviously [that] you need to have…a product that’s cost-effective for a country to roll out. But once that’s there, how do you maximise coverage of the populations that will benefit from the particular vaccine?…That’s where things get trickier. And if you start looking at what impact you are having there—is the infrastructure in the right place? Does it have the right kind of quality? Are there people there when communities need them? Are they trained? Do they have empathy on the community side? Do people have the awareness, the knowledge, [and] the information they need to be able to make the best choices? What are the power equations that determine the choices they make? So if you start looking at the system overall and then ask yourself, ‘Okay, did you have impact?’ then it gets a lot more complicated.

Numbers don’t paint the whole picture

Hari: We end up looking at numeric measures of impact, because that’s what the world seems to value and understand. But two problems with that. One is you often default to averages and averages hide inequities, and variations. [For example] India is much better off on many development indicators, visibly [than it was] a couple of decades ago. But if you drill down, you get down to state numbers, you get down to district numbers, you get down to block numbers—the averages hide huge variations. And we don’t often factor in those variations into how we are thinking about improvements and change. And that’s where this focus on numbers and averages can have a distortive effect.

The second is the point of sustainability, and sustainability to my mind only comes if the people are really brought into the change, that is supposedly to improve their lives. And this urge to measure often takes us towards supply side interventions, because those are easier to track, those are easier to control. And we leave out the factors around demand, cultural norms, all of which can have lasting influence on impact, well beyond the time period of interventions.

Hindi Facebook ad banner for English website

So I think there is a need to take a much more integrative approach that is beyond quantitative, in which we spend a lot more time on qualitative conversations, understanding the cultural context within which interventions need to be delivered, and truly getting the communities more empowered and engaged and giving them more agency in determining the kinds of interventions they want to receive.

The urge to measure often takes us towards supply side interventions, because those are easier to track and control. | Picture courtesy: Aseema

The new paradigm of impact moves away from efficiency and focuses on sustainability

Vineet: Most venture capitalists say that we are a partnership but the reality is that’s not the truth…The only skill that they are seeking from me is that ‘When you take over my money, you will make it work on the most efficient paradigm,’ which I said is the old paradigm. The new paradigm is not [about being] the most efficient, but the most sustainable and resilient. Efficiency is not necessarily what will deliver sustainability, and this conflict is a very deep and inherent conflict. And now I have actually had this debate with a lot of very successful and very large corporate CEOs. And I think their ability to understand this conflict is very limited even now, when we are actually challenged with it. And climate is actually a very good example. Growing trees seems to be a very good solution. But now the debate has moved on [to] that people who are actually dealing with creating the biggest amount of pollution are now more interested in buying carbon offsets rather than changing the behaviour of drilling in the Arctic. There’s an inherent culprit. So they will keep drilling in the Arctic, but keep buying carbon offsets—will that actually solve the problem?

Impact at scale can only come through effective partnerships

Hari: When we think of scale, there is often this monolithic vision of scale, which is an…intervention that can be done all across massive geographies, massive populations—that’s what’s going to drive change. In the corporate sector, you often do see that when entities are successful, [they] grow, and they get bigger and bigger and bigger, and swallow up more and more and more. And that’s how scale is thought of. I think, in the social sector, and the development context, scale might actually need to be thought of very differently. The only way we can have impact at scale is through effective partnerships of different actors who understand [and] can engage with communities on the different sectors where the government and partners are trying to make a difference…And right now, I think a lot of mental energy goes into the intervention at scale, right? How do you come up with the frameworks, the operational guidelines, cascade it down? The fact that all of these interventions and guidelines will land in very different contexts, very different realities, and if you’re not creating the intelligence and the ownership to problem-solve, evolve, and morph those interventions and guidelines into whatever is called for locally, we will not have impact.

Listen to the full episode here.

Know more

  • Read this study about accelerating impact and scaling social innovations
  • Read this article to learn why size may not be the right metric to measure impact.
  • Understand how focussing on measuring impact can be counterproductive to collecting other important data here.

We want IDR to be as much yours as it is ours. Tell us what you want to read.
India Development Review-Image
India Development Review

India Development Review (IDR) is India’s first independent online media platform for leaders in the development community. Our mission is to advance knowledge on social impact in India. We publish ideas, opinion, analysis, and lessons from real-world practice.