Impact Methodology
When we vet social impact organizations, we look for the following four things:
- Does the organization have a deep understanding of the social issue that they are working on?
- Does the organization have primary outcome goals that are based on changing the negative consequences of that social issue?
- Does the organization appropriately measure these primary outcome changes in the lives of the people that they work with?
- Lastly, does the organization measure any counterfactual data that helps us understand the likelihood that its intervention is the cause of the primary outcomes it is observing?
Understanding The Social Issue
Yes, this is indeed a thing. We have worked with too many organizations that do not really understand the social problem that they are trying to solve. All too often, they see a simple problem and quickly come up with a fix. Problem → Fix. Or even worse, they come up with a solution and then look for a problem it solves.
Initially, when assessing this, we look for evidence that the organization understands and uses the key terms that others involved with the same social issue use. We also look at how the organization describes the context where their social problem occurs and with whom. We look for the contributing factors of that social problem in that context and determine which of those their solution is addressing. Finally, we determine if the organization is determined to change people's lives for the better, and do they have a way of knowing how much that is and is not occurring.
Ultimate Outcome Goals
Great social impact organizations know the ultimate outcome they are working towards in people's lives. And, they have those outcomes as their primary goals.
For example, many people across the world do not have access to clean water and do not practice adequate sanitation or hygiene. As a result, they become sick and can die of waterborne illnesses. Death from waterborne illness is the leading cause of death for children under five years of age in the developing world. Good organizations in this field will measure their success by the reduction of illness and deaths from waterborne illnesses. Sometimes good, but not yet great organizations, focus only on outputs (intervention milestones). In our example, their goal might be the number of wells they have installed. A well could be an important intermediary goal depending on the context, but it is not the end goal. A reduction in illness and death are the ultimate goals.
Taking the time to understand the ultimate outcome takes time, focus, and a deep understanding of the problem. We do not recommend groups that have not identified their ultimate goal.
Evidence of Success
We assess how the organization measures their progress toward their ultimate outcomes. We are looking for appropriate intervention milestones that describe how clients flow through their solution, where some get stuck, and others may give up. In our example of waterborne illness and deaths, we would want to see how the group is providing access to clean water, how many people actually use the new clean water, and whether or not this access leads to fewer deaths. If they don’t track progress toward fewer deaths, they will never discover whether or not all of their efforts are working as planned.
For example, we know of a group that provided containers of clean water to schools so students would not use nearby contaminated water. They were proud of the gallons of clean water being provided. However, they measured the ultimate outcome - reduced illness and realized their solution had not led to fewer waterborne illnesses. This finding led them to dig in and talk with and observe the students. It turned out, the students did not drink the clean water because the containers were located near where the teachers and other adults congregated. The organization quickly moved the containers to a more student friendly location, measured the number of waterborne illnesses among the students and saw a reduction in illness. If they had not measured their ultimate outcome goal, they never would have known to make these adjustments.
Lastly, we want to make sure that the measurements and assessment methods are valid.
If the organization has no evidence of intermediate and/or ultimate changes in people's lives, we do not recommend them.
Counterfactual Impact
The best organizations measure their outcome success against a control group. A control group is as similar as possible to the group receiving the intervention, but does not have access to the intervention. After the intervention is successfully implemented, the same outcomes for both groups are measured. Similarly to medical research, if there is no difference between the groups, the intervention is not successful. This is hard to do, but very important. Sometimes changes occur naturally, such as an unusually great crop yield. Without a control group, a group supporting farmers might think the increased yield was because of their intervention. But if the control group also has an increased yield, it was probably because of factors that naturally occurred to both groups such as more rain that year.
Similarly, maybe the intervention led to the same crop yield as the year before, but there was little rain that year and the control group had a lower yield. If the intervention group has a higher yield than the control group - even if it didn’t go up, it is a successful intervention. It helped their farmers avoid the devastation of the drought. Most social impact groups operate in dynamic contexts where there are a lot of factors contributing to a problem. Using a control group creates a counterfactual that helps groups better understand if they are effectively solving a problem because it isolates the intervention from all of the other confounding factors.
Creating counterfactual evidence can be resource intensive. It also requires organizational maturity. We don’t expect new organizations that are still piloting and iterating solutions to have robust control group research, but we want to see that they understand its importance and are moving in that direction. First steps include taking baseline measurements, and comparing their groups with the general population in which they are working. However, we expect a mature organization that has been working for more than ten or so years to have counterfactual evidence.
Persistent Humility
Lastly, a number five for our list of four items. We highly recommend organizations that have the humility to work through those four items constantly. Over time, their understanding of the social problem should become clearer. As that occurs, they should be changing their intervention to better address the problem they are trying to solve. Based on their continual, deeper learning, they may even adjust their ultimate outcome goals. Likewise, they will alter their evidence gathering to help them gain an even better understanding of what is actually occurring to their clients. They are in a state of perpetual learning and improving what they do. We are leary of groups that have been offering the same interventions year after year, with no discernible changes and improvements.
