Metrics & Analytical Methods

"If you can't measure it, you can't manage it." This common piece of advice from the business world is starting to take hold in the nonprofit/social-e sector, and for good reason. Many nonprofits in the past have existed without knowing or sometimes even trying to quantify the impact they create through their programs. The tide is turning, though, because more foundations and other sources of funding are developing internal metrics for evaluating their investments. This, in turn, is being transferred to the social-e organization and below is the most important information you need to know to understand different forms of analysis as well as developing your own.

Why are metrics so important?

Useful tool for management: Metrics are important because they help management find out which areas of an organization are inefficient, where the organization is facing roadblocks, and where information is not getting to the right people at the right time. As organizations become more dependent on effective coordination between teams of people, the barrier to accessing information should become much smaller and more transparent. While it takes time to develop the internal tools necessary to effectively quantify and qualify the data, the resulting benefit will pay off in numerous ways.

Enable donors to understand relative effectiveness: As competition heats up between organizations for donors limited dollars, having objective data, especially for organizations that operate within the same field, will allow donors to put their money in the hands of organizations that will create the most benefit. Even though it is currently difficult to compare the effectiveness of organizations operating in different fields, such as child literacy and women's rights, having the ability to compare between the same field (child literacy) is relatively easy and gives donors more information.

Increase overall donor confidence thus increasing the amount of total giving: When donors can see how their money is being spent and what the end result of this donation can be, donors may be more willing to give more on effective programs. There are many skeptics to this proposal though, many of which are highlighted below:

• Donors do not see a need for performance measurement.
• Donors do not have time for performance measurement.
• Donors do not have confidence in performance measurement.
• Donors do not want to see nonprofit resources dedicated to performance management.
• Donors look to institutional funders to engage in performance measurement on their behalf.

What are different methods of calculating effectiveness?

Cost-Effectiveness Analysis: The purpose of this form of analysis is to create simple, natural units of measuring impact as a ratio of cost, which can then be compared when appropriate. Cost per child cured of malaria is an example of this method. When developing the metrics, be sure to break out all costs and benefits by stakeholder (identify who pays and who benefits) and over what timeframe. [1][2]

Cost-Benefit Analysis: This form of analysis monetizes the benefits and costs associated with a program and then compares them to one another to see which is greater. The result is the ability to compare any program against any other program and see which one maximizes value monetarily. For example, if it costs the government a total of $10,000 per student over the course of four years to improve their grades and get them into a 4-year college, and the student then earns a total of $900,000 more over the course of his/her lifetime (and guesstimating the government makes an extra $180,000 in taxes at 20%) the cost-benefit analysis comes out to $180,000 / $10,000 = $18 in tax revenue per $1 invested. [1][2]

REDF SROI: REDF is a nonprofit philanthropic social venture fund founded in 1997 in San Francisco, CA, and calculates social return on investment (SROI) by developing a set of metrics that quantify and monetize social return. These returns are then viewed in the context of outcomes that are difficult or impossible to monetize or quantify. This form of measurement draws heavily on cost-benefit analysis and results in total benefits accrued to the public sector, not individual stakeholders, and has thus been less popular with funders because of the lack of detail. [3][4][5]

Robin Hood Foundation Benefit-Cost Ratio: Robin Hood is a nonprofit founded in 1988 to target poverty in New York City (NYC) with the goal of fighting poverty in four general areas: Jobs & Economic Security; Education; Early Childhood & Youth; and Survival. The purpose is to provide a monetary value of the projected and actual outputs of a program and to use these values to compare the effectiveness of programs within its portfolio. Draws heavily on the cost-benefit analysis. [6][7][8][9]

Acumen Fund BACO Ratio: The Acumen Fund is a $50 million nonprofit global venture fund founded in 2001 in New York City with the goal of receiving a payback or exit within five to seven years. To determine the estimated social output, they compare the ROI to the Best Available Charitable Option (BACO) ratio. The driver of this methodology is that any ROI, even if it is only the principal (no interest), allows the fund to reinvest that capital in another opportunity down the road. [10][11]

Hewlett Foundation Expected Return: The William and Flora Hewlett Foundation was founded in 1966 to solve social and environmental problems at home and around the world. The expected return provides a systematic, consistent, quantitative process for evaluating potential giving activities, and is based heavily on cost-effectivness analysis and cost-benefit analysis. [12][13][14]

Center for High Impact Philanthropy Cost per Impact: The Center for High Impact Philanthropy (CHIP) was established in 2006 by alumni of The Wharton School of the University of Pennsylvania, who were frustrated by the difficulty of measuring and maximizing the impact of their charitable gifts. The goal is to provide tools and information that allows philanthropists will have the greatest impact by showing the costs incurred for the provided benefit. Draws on the cost-effectiveness analysis. [15][16]

Foundation Investment Bubble Chart: This form of analysis is more of a visualization tool that plots the quantifiable impact on the x-axis, the percentage of implementation on the y-axis, and the relative size of a foundation's grant in a given field. This results in an easy comparison of the performance of organizations across a portfolio and can have different variables for the x-axis, y-axis and bubble relativeness for flexible data display. [17][18][19]

How can I implement metrics into my organization?

1. Start with your mission - Are you trying to increase access to retroviral drugs for those who have aids? Are you trying to improve student's SAT scores? Be sure you have a great mission that is quantifiable, time bound, and action oriented.
2. Develop metrics and a timeline- Once you know what you're trying to do, you can now develop ways to measure your progress. If you are trying to improve student's SAT scores, first start by having students take a practice or official SAT to develop a baseline. From there, keep track of the number of teaching sessions, total costs incurred for those sessions, length of teaching training and teaching sessions, and finally the increase in SAT score based on a final official test. Put this expected progress on a timeline of expected goals and
3. Quantify Intangibles - Let us say you have a goal of "increasing teamwork" - something that is hard to quantify numerically. To measure these intangibles, it is useful to ask the question, "What would someone with this quality do?" This results in a concrete result, which can then be measured. To continue with the example, an increase in teamwork would result in fewer projects failing or falling off-track. Now you can quantify the number of times a project fails or is late and systematically work towards the goal.
4. Evaluate - Any system for improvement must itself be improved. Asking those who are creating the most results should be asked how they have been creating their successes and then building upon these lessons and sharing them with the rest of the organization.
5. Reward Performance - Find out who has created the most progress towards these goals and reward them appropriately. Also, don't hesitate to take the opportunity to have these individuals share the lessons of their success with the rest of the organization and utilize them as a teacher for the organization. And if the goal is reached on time, take some time to celebrate and reward everyone accordingly!

Conclusion

The development and analysis of metrics is a relatively new and exciting field for nonprofits and social enterprises. While somewhat intimidating and daunting for those with no experience in the area, the best way to start is by asking the question, "If I was considering donating in the cause, what information would I want to know to make sure my money is being put to good use?" Beginning with the end in mind makes the intermediate steps more tangible and manageable. Finally, don't worry about not getting it right the first time! Developing and analyzing the impact of your organization evolves over time but is something that should always be in the back of your mind.

Resources

Primary Source:

Tuan, Melinda. "Profiles of Eight Integrated Cost Approaches to Measuring and/or Estimating Social Value Creation." Bill & Melinda Gates Foundation, Impact Planning and Improvement Division December 15, 2008.

Secondary Sources:

1. Karoly, Lynn A., M. Rebecca Kilburn, James H. Bigelow, Jonathan P. Caulkins, Jill S. Cannon. "Chapter Two: Overview of Cost and Outcome Analysis." RAND: Assessing Costs and Benefits of Early Childhood Intervention Programs: Overview and Applications to the Starting Early, Starting Smart Program.
2. Karoly, Lynn A. "Re: Cost-Benefit Studies of Social Programs" to Melinda Tuan, 20 and 21 May 2008.
3. Gair, Cynthia. "A Report from the Good Ship SROI." The Roberts Foundation 2002.
4. The Roberts Foundation. "SROI Reports." The Roberts Foundation 2000.
5. Tuan, Melinda and Julia Jones. "SROI Reports: Overview and Guide." The Roberts Foundation 2000.
6. Weinstein, Michael. "Measuring Success: How Robin Hood Estimates the Impact of Grants." (working draft) January 31, 2008.
7. Weinstein, Michael. "Re: Robin Hood Foundation Benefit-Cost Ratio and Integrated Cost Approaches to Measuring Social Impact" to Melinda Tuan, 23 April 2008
8. Robin Hood Foundation website.
9. Weinstein, Michael email communications with Melinda Tuan, July, August, September 2008.
10. Acumen Fund Concept Paper: The Best Available Charitable Option (BACO), (Draft), 1/24/07.
11. Trelstad, Brian. "Re: Acumen Fund Best Available Charitable Options (BACO) and Portfolio Data Management System (PDMS)" to Melinda Tuan, 30 April 2008.
12. Redstone Strategy Group, LLC. "Making Every Dollar Count: How Expected Return Can Transform Philanthropy." April 10, 2008.
13. Brest, Paul and Hal Harvey. "Chapter 10: Impact on Steroids: Measuring the Social Return on Your Philanthropic Investment." Money Well Spent: A Strategic Guide to Smart Philanthropy. Forthcoming, Bloomberg Press, November, 2008.
14. Brest, Paul. "Re: Hewlett Foundation‰Û÷s Expected Return Methodology" to Melinda Tuan, 24 April 2008.
15. Rosqueta, Katherina, Hilary Rhodes, and Kathleen Noonan. "Re: Center for High Impact Philanthropy Cost Per Impact Measures and other Integrated Cost Approaches to Measuring Social Impact" to Melinda Tuan, 8 May 2008.
16. Rosqueta, Katherina, Hillary Rhodes to Melinda Tuan, 4 June 2008.
17. Huggett, Jon. The Bridgespan Group presentation "Business Planning: What it Is and Why it Matters." p. 7.
18. Olsen, Sara. "Re: Integrated Cost Approaches to Measuring Social Impact" to Melinda Tuan, 30 April 2008.
19. Fay Twersky, Brian Elliot, Melinda Tuan interpretation.