The Playbook on Impact Measurement

For-profits chase revenue like there’s no tomorrow. Nonprofits, on the other hand, chase impact. Impact is your who, what, and why. It’s your reason for being. So you better be sure you’ve defined exactly who your helpful solution is impacting (and how!), because the way you design your impact measurement will define your organization’s architecture.

Landing on an ethical stance to guide you in deciding who you’re going to serve – also known as distribution ethics – will aid you in deciding who your product should be for. Don’t worry if you haven’t figured it all out yet – this chapter provides frameworks for thinking through these big questions.

Plus, we made you this handy worksheet to help you land on an impact framework and think through the distribution ethics of the helpful solution you want to build. Use it as you read through this chapter. Trust us, it will be worth the investment of time 10X over.

Goals for Impact Measurement

Defining impact is critical. It’s how you align your tech nonprofit’s work with the value that you want to create for the world. And, it’s how you can raise support to keep doing your work. Potential funders want to make sure they’re not just giving, but giving smartly. Accordingly, a tech nonprofit’s ability to effectively quantify its impact can have a huge influence on its ability to attract funding.

Defining impact metrics that align your goals with company values enable you to:

• Regularly capture data that reveals your impact performance.

• Measure, monitor, and maximize progress so you regularly improve.

• Compare your work to a benchmark.

• Report your impact to stakeholders.

• Increase your capacity to do more of your good work.

Challenges of Impact Reporting

We’ll start with the hard stuff. Defining success in the nonprofit sector is difficult. Unlike its for-profit counterpart, one cannot simply look to financial returns as a measure of success. Rather, the aims of nonprofits are varied and unique. As such, it is usually not possible to create industry-wide standards for performance. This makes it very difficult to compare the degree of social impact across different organizations and projects.

There are some major challenges to impact measurement in the tech nonprofit sector:

• Lack of industry standards. There are buzzwords abound that mark universal standards for measuring success in the for-profit sector. It’s why you can compare the metrics of a company like Lyft against a wildly different company like Airbnb. While universal industry standards don’t exist in the social sector, the imperative to measure impact does. Read: there’s no simple calculation.

• Qualitative focus. Your work will have qualitative impact; some things can’t be quantified. Said differently, stories about the particular ways your product improved the lives of individuals may often be the most powerful way you can communicate impact. But that’s not enough. More and more, funders expect you to be able to provide quantitative evidence for how you know the helpful thing you’re building works.

• Impact means different things to different people. To complicate matters, different funders look to different types of impact reporting as funding criteria. Therefore, you’ll need to come up with a few different metrics and stories to demonstrate impact.

• Output does not equal impact. We’ll get into that later in the chapter but the basic idea here is that just because you’re getting people to use your helpful solution doesn’t mean it’s having the impact you want it to.

Distribution Ethics

Josh Nesbit, Founder and Former CEO of Medic Mobile, has shared a framework and set of thought experiments that helped him and other tech nonprofit leaders. Here are the basics.

Distribution ethics is a framework that helps you decide what to build and for whom. Whether or not we discuss them openly, moral decisions are at the center of what we do and how we do it. Everyone and every organization has a moral stance – either you picked it explicitly, you intuited it, you inherited it, or someone picked it for you. But being thoughtful about defining your ethical stance will serve as a guiding principle for decision-making, which is one of the things you’ll do most as a founder.

The Classic Nonprofit Model

Idea → Mission → Delivery Model → Business Model → Org Design → Staffing

Distribution ethics inserts a moral stance into the classic nonprofit model. It asks the hard questions like: Who deserves access? Is depth or breadth more important? And based on those answers, what should we work on? The messy problems that most merit our attention usually involve a moral stance.

Model Emphasizing How Moral Stance Affects Distribution

Idea → Mission → Moral Stance → Delivery Model → Business Model → Org Design → Staffing

Classic Ethical Frameworks

Frameworks for thinking about who gets your helpful solution.

The implications of your moral stance will impact everything, including:

• Your Idea: Intervention, product, or service.

• Mission: What you do, for whom, and why.

• Delivery Model: How your organization delivers its helpful solution to people.

• Organization Design: The way your organization is designed to carry out the delivery model.

• Staffing: Who to hire and what they’ll work on.

• Goals: Both your organizational goals and fundraising targets.

• Storytelling: The way you communicate what you do and why you do it.

• End game: Your decision around when it’s acceptable to stop serving.

Now to the big question… which of these ethical frameworks resonates with you?

Deciding Who to Serve

Distribution ethics looks at the types of people you could serve. The spectrum of distribution ethics considers many dimensions, ranging from who you serve and how easy or hard it is to reach them, to how deeply you serve them, to the quantity of individuals you serve, and beyond.

Imagine a circle containing all the possible people who could benefit from your idea, product or service:

In this circle, some people will be easier to reach, and some people will be harder to reach. By harder, we mean that they have less resources; time, money, and social capital. Or, they’re farther away and harder to find, so it costs more to reach and serve them, and it’s riskier to deploy your solution.

You can also identify the people who are easiest to reach. These people still stand to benefit from your solution, but perhaps not as much.

Generally, those who are the very hardest to reach and subsequently serve could benefit from additional resources the most. In some cases, there will be a tension between what the most people need and what the most disadvantaged people need. Serving the most people may lead to great utility. Serving the most needy may lead to progress on specific human rights.

Chain of Causality

Now that you’ve landed on a moral stance that will guide your decision making, it’s time to think about the chain of causality, or the connection between inputs, outputs, outcomes, and impact. The chain of causality is an important element of impact measurement. The main thing to understand is that outputs and outcomes are very different things. It’s one thing to provide a certain number of families with a service (or output) but another to claim that use of the service created the change you set out to effect (or outcome).

Chain of Causality

Below are simplified examples of the chain of causality for two tech nonprofits, Hikma Health and JustFix.

How to Manage Impact Measurement

Measuring impact is not a one and done thing. It’s an iterative process that requires regular attention. In addition to collecting data on a regular basis, you can (and should!) continually return to the metrics you’ve decided to track and reassess if they’re still the metrics that demonstrate impact.

Here’s the general process for defining and keeping track of your impact metrics:

Pick the impact metrics you want to measure from output to outcome. Keep in mind the important questions like:

• What’s my theory of change?

• Do I want to help those who are easiest or hardest to serve?

• What are the key metrics that will demonstrate that I’m succeeding in my mission?

• What processes will I use to keep track of my impact metrics over time?

How frequently should I be tracking? Daily, weekly, monthly?

• What targets will give me confidence that I’m on the right track?

• Who collects this data? Is it me? Someone on my team?

• Where is the data stored? This could be as simple as a spreadsheet or a fancy dashboard.

• How is it audited and how is compliance ensured?

• Are there any AI tools that can help me track these metrics or store them all in a centralized place?

• Do I need an impact measurement tool to collect and analyze impact in real time?

Go back and make sure you’re still measuring the right thing. And ask yourself the same questions you started with:

• Has my theory of change…changed?

• Am I still serving the same group of people?

• Are these still the key metrics that demonstrate I’m succeeding in my mission?

Re-Think Impact

It might be the case that you’ve already built something and can’t change all of your plans tomorrow. That doesn’t mean it’s too late to intentionally choose a moral stance and let that guide how you move into the future. Josh Nesbit provides a helpful thought exercise for doing just that:

1. Draw your circle of people and write “Here’s where we are” over it. Highlight the group of people – either easiest, easier, or hardest – that you’re serving today.
2. Draw a new circle of people and write “Here’s where we’d like to be” over it. Highlight the group of people you’d like to serve in the future, based on your moral stance.
3. In the middle is the delta that you need to solve for. You’re not alone in solving for this delta. The delta represents an opportunity that so many people around you want to help you solve for. By defining an ethical framework and taking a moral stance, you will find resources to help you solve for the delta.

Be the tech nonprofit that is solving for the delta. There is opportunity for transformational impact in the line from where you are today to where you want to be. In the case of Medic Mobile, the organization ended up building an entirely new version of its tools after going through these exercises. Then, it had to do it again and again and again. A moral stance is like a map – it helps you gut check if you’re on the right path with every new decision.

When you’re a tech nonprofit, ethical frameworks matter deeply. Not just in our operations, but to the very core of why and how we do what we do. Tech enables us to scale to any number of people, and its impacts can be far-reaching. It’s by defining a moral stance and letting that guide our decision making that we make smart choices.

There isn’t a right or wrong answer to which ethical stance your organization takes. Each stance has moral value. But you do have to choose to ensure that every decision you make is in alignment. In closing, ethics are complicated, and we can’t do this work alone. We must lean on our peers and community to make sure our moral frameworks are sound. Social impact isn’t a solo sport – we’re in this together.

Additional Resources

• Decide Who You Help and Why (or, Distribution Ethics) with Josh Nesbit, a free workshop from the Fast Forward Academy