Shift to Outcomes

Nonobvious Metrics

By Michael Weinstein

The Robin Hood Foundation makes grants of about $150 million a year fighting poverty in New York City. At first glance, its decisions might appear odd. Ponder three grant seekers (with made-up names):

  1. Cool-Tax files for tax refunds on behalf of low-paid workers, recovering an average of $3,000 per worker. Impressive outcome. But Robin Hood does not fund Cool-Tax.
  2. Cool-Loan lends small amounts of money to entrepreneurs to set up daycare services in their basements. Every borrower thrives. Impressive outcome. But Robin Hood does not fund Cool-Loan.
  3. Cool-House spends $600,000 a year to provide shelter for 60 formerly homeless individuals. Not one returns to the streets. Impressive outcome. But Robin Hood does not fund Cool-House.

See a pattern? You might infer that Robin Hood makes dumb decisions. I prefer a different inference: Robin Hood’s way of measuring success — its metrics system, dubbed Relentless Monetization — generates nonobvious decisions. Nonobvious but, I hope to convince you, correct. These decisions are examples of what happens when funders focus on correctly measured outcomes.

Let me explain.

Cool-Tax: Assume the taxpayers who flock to Cool-Tax would, in its absence, flock to H&R Block. Then Cool-Tax accomplishes nothing important other than to save filers a modest fee. The H&R Block option is known as a counterfactual — an assumption, which cannot be directly observed, about what would happen in the absence of an intervention that did indeed take place.

Cool-Loan: Assume the daycare businesses that Cool-Loan makes possible take customers from equally poor daycare providers down the block. If so, Cool-Loan accomplishes nothing important. It changes the name of those who are poor but not their number. The unintended harm that a grant does — in this case, to self-employed neighbors — is known as displacement.

Cool-House: Assume Cool-House spends $600,000 each year to provide services to the same 60 families. That’s prohibitively expensive, compared with spending $600,000 each year providing services to different individuals (as would ordinarily occur if the intervention were training unemployed workers or teaching math to eighth graders). Relentless Monetization’s discipline of trading off benefits against cost taps the workhorse of modern economics: benefit-cost analysis.

Here’s an important point: Nonprofits habitually pay short shrift to counterfactuals, displacement, rate of return, and, as we’ll see later, a correction called the “Robin Hood factor.” These oversights produce not just error, but bias. They make nonprofits look more powerful than they truly are. No wonder these oversights persist.

When deciding whether to fund a nonprofit, Robin Hood applies a complicated algorithm. Let’s start with the basics:

Step #1: Robin Hood identifies outcomes that connect to its mission of fighting poverty.
Step #2: They then assign a dollar value to each outcome (they “monetize” outcomes).
Step #3: They then add up the separate benefits to calculate a total benefit of the intervention: the numerator of a benefit-cost ratio.

For example, imagine a grant to a middle school serving poor children. First, what’s the poverty-fighting outcome? Robin Hood funds the middle school to raise the probability that its students will graduate high school four years hence. Robin Hood does so because it believes earning a high-school diploma is the most important step that any of us take out of poverty.

Let me explain with an equation.

The numerator of the benefit-cost ratio:

45 x [[P.D.V. ($6,500/year for 20 years)] + [1.8 QALYs x $50,000/ QALY]] x [0.6 Robin Hood factor]

45: Robin Hood estimates that the school graduates 45 more students than would have graduated from neighboring schools. Simple enough statement. But behind the figure (45) lurks a lot of research and numerical estimation. It takes account of counterfactual success (how many students would have graduated from neighboring schools). It takes account of displacement (whether Robin Hood’s grant unintentionally harms other low-income students in the host or neighboring schools). And so on.

P.D.V. (present discounted value): This is a reminder to take account of the fact that dollars enter and exit in different years.

$6,500: From research literature, we take this measure of the expected difference in future income between two otherwise identical workers, one of whom earned a diploma and one of whom did not.

1.8 QALYs (quality-adjusted life years): Though the literature does not speak with one voice, we take from it the plausible estimate that folks who graduate high school live, on average, two years longer and in better health than do identical students who don’t graduate high school (separate from the impact of higher income on health). We measure the impact in QALYs, which account for morbidity and mortality.

$50,000:

Problem: If we agree that helping low-income residents live longer and healthier ought to be an important part of alleviating poverty, how do we combine the impact of our grants on income (measured in dollars) and on health status (measured in QALYs)?

Answer: We need to assign a dollar value to improving the health of the individuals we serve. Robin Hood has assigned a value of $50,000 to a QALY (helping an individual live one additional year in good health). They arrived at the $50,000 figure after sifting through the medical-economics literature, reviewing the practices of U.S. and U.K. government agencies, and sometimes making (arguable) judgment calls about inconsistent evidence.

0.6 Robin Hood factor: If Robin Hood is not the sole funder, it cannot claim credit for all of the school’s success. Here, we estimate that without Robin Hood’s help, 60 percent of the rise in graduation rates would disappear.

If you do the arithmetic, the numerator comes to $5.2 million. The cost to Robin Hood — the grant size — is $800,000. The benefit-cost ratio comes to 7:1. What does 7:1 mean? For every $1 Robin Hood spends on this grant to the middle school, the collective wellbeing of low-income New Yorkers rises by $7 (over their lifetimes). Wellbeing takes account of changes in future income and changes in a monetized value of future health status.

Robin Hood can compare the benefit-cost estimate for this middle-school grant to that of any other grant — whether it be, for example, teaching carpentry to chronically underemployed women, providing legal services for abused women, or screening adults for hepatitis C virus and treating those who test positive. The calculations often grow more complicated than the equation presented earlier. But the basic principle remains the same: Steer grant dollars to where they do the most additional good.

Do metrics matter? You bet. This example shows how. Metrics drive us to serve the most disadvantaged — those we can help the most. Metrics drive us away from most standalone after-school programs, steering us instead toward high-performing schools that seek money to run extended hours.

And on and on.

Of course, placing dollar values on services involves arguable judgment. What’s the value of providing lawyers to help parents fight for custody of their children? Of course, monetizing value runs the risk of false precision. But the simple fact is that any funder that allocates a budget across options implicitly assigns a relative value to each option it chooses to fund and to each option it could afford but chooses not to fund. The major advantage of Relentless Monetization is that it makes judgments explicit and, therefore, debatable. Robin Hood posts on its website the 170 or so equations that it uses to assess the impact of grants, alongside the source of all cited numbers. They do so both to share what they’ve learned and to solicit suggestions for improving their metrics. Two key points:

  1. Relentless Monetization focuses on outcomes — outcomes corrected for counterfactual successes and displacement — rather than inputs. Robin Hood doesn’t measure how many students you teach. They measure how many more students you graduate.
  2. Robin Hood’s metrics assess the value of Robin Hood’s grant, not the value of its grantees. The enrollees at Stuyvesant High School in New York are the highest-scoring students on citywide tests. We believe they would all graduate even if you locked them in a dark room for four years before letting them pop out to take exams. The fact that Robin Hood sees no impact from a possible grant to Stuyvesant reflects high regard for these students, not low regard for the school.

The upshot? Robin Hood’s metrics system admittedly doesn’t get everything right. But by tying grant decisions to correctly measured outcomes, Relentless Monetization steers grants where they pack the most wallop. At the very least, Robin Hood has shown that evidence-driven outcomes can move out of the classroom into the streets of New York. All in, Robin Hood estimates that its typical grant generates a ten-fold return: $150 million of annual grantmaking triggers a boost in the collective living standards of low-income New Yorkers by $1.5 billion or so.

Finally, the Robin Hood mantra: Never, ever make a grant on the basis of arithmetic alone. Benefit-cost ratios serve as one, but only one, key piece of information, much as SAT scores serve as one, but only one, key piece of information to college admissions offices. Numbers matter. But other types of information also matter.

Focusing on smartly measured outcomes does not come easily to funders. Adding to their discomfort, smart decisions can steer dollars in counterintuitive directions, away from popular organizations with enviable reputations. But one powerful thought need reign supreme: For funders to spend precious philanthropic dollars on the wrong interventions is unconscionable.