From Evidence-Based Policies to Positive Outcomes: How to Fund What Really Works in Child Welfare

By Susan Snyder

Matthew was 11, John was two, and Lisa was a newborn when they were removed from their mother’s care because of her drug addiction and placed in the foster care system.¹ Instead of helping, the foster care system further traumatized the children. The children’s first foster home was de-licensed after another child was severely physically abused in the home. The siblings were then separated, and John was beaten with a belt in his next foster placement. The newborn was placed in a home with a convicted rapist. Matthew began having behavior problems and engaging in gang activity. Although their mother loved her children, she needed help herself, and the addiction and parenting support programs available to her were ineffective.

Perhaps the most distressing part of this story is how common these circumstances are for child-welfare–involved children. As many as 80 percent of parents whose children are in foster care suffer from a substance abuse disorder.² And while removing children from their original homes can be essential in many circumstances, it also increases their risk of engaging in delinquent behaviors,³ substance use,⁴ and ultimately ending up in prison.⁵ The long-term societal costs of removing children from their original homes, rather than seeking to treat parental addiction issues and preserve families where possible, are enormous — and sometimes continue for generations.

Regardless of whether children are removed from their homes, child maltreatment is a public health problem with substantial costs to society. A study by the Centers for Disease Control and Prevention found that the total cost of child maltreatment is $124 billion each year.⁶ Despite these enormous costs, few evidence-based interventions address the needs of either maltreated children⁷ or their parents.⁸ The situation may be exacerbated because federal child welfare expenditures have decreased, even though the number of child maltreatment reports has increased across the country. As a result, child welfare services are increasingly reliant on dwindling state and local funds.⁹ Thus, social impact bonds may provide a way to address child welfare service needs that cannot be met in the current economic climate.

Children like Lisa, John, and Matthew might be far more likely to have better outcomes if our child welfare policies focused on what really works. Although we spend a great deal on social programs, seldom are the results of programs adequately assessed, which allows programs that don’t work to endure far too long and draw resources from other, more effective interventions.¹⁰ This chapter discusses social impact bonds, which strive to fund child welfare programs that achieve desired outcomes without wasting resources on programs that don’t work.

OVERVIEW

A predecessor to social impact bonds, called performance-based contracts, emerged in the early 1990s as one way to fund what works in child welfare.¹¹ Rather than paying for a unit of a particular service (e.g., an overnight stay in a foster home), performance-based contracts can reward private agencies that achieve or exceed specific outcomes and/or penalize agencies that fail to achieve specific outcomes (e.g., children are reunified with parents, children are adopted, or parents are able to provide a safe and supportive home environment).¹² At least 27 states have experimented with performance-based contracts for one or more child welfare services.¹³

Social impact bonds, part of a broader suite of outcomes-based contracting approaches, constitute a relatively new variation of performance-based contracts that focus on funding effective programs in child welfare. Social impact bonds are public-private partnerships that shift risk from the government to investors: An intermediary raises private capital, chooses service providers to provide an evidence-based intervention, and selects an independent assessor to determine performance outcomes and the evaluation approach.¹⁴ Government is required to pay only if the intervention works as intended.¹⁵

Because of the size and complexity of the child welfare system, ensuring that social impact bonds can be effective in that context requires starting with a well-designed study, with the following components: a clearly defined population and problem, an intervention, a comparison, and the desired outcome.¹⁶

It is best practice for evaluations of social impact bonds to include a comparison or counterfactual, which is a way to see what would happen if the intervention were not given. One way to provide counterfactuals is to use randomized controlled trials.¹⁷ In randomized controlled trials, participants are randomly assigned by an impartial method, such as a flip of a coin, to determine whether they receive the intervention the social impact bond provides. Although randomized controlled trials are considered the gold standard for scientific research, they are sometimes not used because some argue that it is unethical to withhold a treatment that could work. The irony is that failing to use randomized controlled trials may do more to withhold effective interventions, because that can allow treatments that may not work to be scaled up, wasting limited resources.

EXAMPLES

States, counties, and child welfare agencies are beginning to embrace social impact bonds to reduce costs and improve outcomes. Connecticut, for example, obtained $12.5 million in private investments for a social impact bond using the Family-Based Recovery (FBR) intervention, which researchers from Yale University and the University of Connecticut found to be an effective treatment intervention for parental substance abuse.¹⁸ FBR is an in-home, intensive, long-term clinical treatment intervention to keep families of children under the age of eight intact while providing the services needed (e.g., substance use treatment, parent-child relationship support, developmental guidance, and case management) to ensure that the home is a safe and supportive environment.¹⁹ In Cuyahoga County, Ohio, a $4 million social impact bond is housing and providing supportive services to 135 homeless parents with children in foster care, to reduce the children’s length of stay in foster care.²⁰ In Illinois, the Dually-Involved Youth Project was developed to target youth who are involved in both the juvenile justice and child welfare systems, since nearly two-thirds of those youth spend time in costly institutional care and half recidivate within two years.²¹

Although the literature regarding the strengths of child welfare social impact bonds is sparse, there are indications that social impact bonds may have the potential to strengthen the child welfare system. Axford and Morpeth suggest that the outcomes-based focus of social impact bonds may be a better fit for employing programs with a proven track record (i.e., evidence-based practice) than providing traditional public child welfare services without such contracts.²² Liebman asserts that social impact bonds could provide a means of helping foster youth and youth involved in juvenile justice to successfully transition to adulthood.²³ Hawkins et al. contend that social impact bonds could effectively support the widespread use of preventive interventions.²⁴ With this in mind, evidence-based prevention and intervention programs could be initiated to address a wide range of issues pertaining to child welfare over the course of development. One example of a prevention program is SafeCare,²⁵ which is an in-home program for at-risk or maltreating parents that could be implemented in more communities throughout the world.²⁶ Examples of programs that address trauma include the Attachment and Behavioral Catch-Up for children from birth to 24 months, Parent-Child Interaction Therapy for youth from two to 12 years old, and the Safe Harbor program for youth six to 21 years old.²⁷

Despite their potential, social impact bonds also have some weaknesses. Service providers may be deterred by the risk that the intervention could fail, or they (or their investors) may require large fees to assume the risk. Scaling up social impact bonds can be difficult because what works for one population may not work for another.²⁸ Social impact bonds may also cost more than programs that government delivers directly.²⁹ Another possibility is that service providers may avoid providing services in difficult cases that are less likely to succeed when compensation is tied to measurable success.

GIVEN THE EVIDENCE, WHY AREN’T SOCIAL IMPACT BONDS MORE WIDESPREAD?

A few barriers have prevented more widespread use of social impact bonds in funding the child welfare system, including their complexity. Service providers may already struggle to engage in performance-based contracting without the added layers of investors or intermediaries. Additionally, it can be harder to measure real change than other metrics that are easier to quantify but do not necessarily signal improvements in care. According to Stid, it is easier to quantify the resources that go into an intervention (inputs) and the products produced, such as the number of parenting training sessions (outputs), than the actual measurable changes the intervention produces (outcomes).³⁰ Martin’s survey of 614 members of the National Institute of Governmental Purchasing found that only 39.9 percent of new contracts or contract extensions were awarded based on performance outcomes, as opposed to other measures like inputs and outputs.³¹

Another factor that may prevent social impact bonds from spreading in the child welfare field is that children in foster care have complex needs and are served by multiple agencies and programs across local, state, and federal government. This is particularly true of the high-needs users of child welfare services, such as children with severe emotional and behavioral problems. Additionally, individuals at each agency may have disparate and conflicting interests that may prevent needed collaboration and lead to competition for credit. Consequently, it may be difficult to track costs and service utilization, which is essential in order to pay off the intermediaries and investors. Finally, the failure of these programs could attract negative media attention.³² The impact of such attention in child welfare may be particularly profound because the general public may be outraged if children are harmed by a program that was perceived as having a profit motive.

CONCLUSION

Although there are challenges, social impact bonds can play an important role in advancing child welfare. Science and ethics must play a central role in any such venture, and political will must be built among participating stakeholders to ensure optimal designs at every phase. Because the stakes are so high, social impact bonds should be considered as an option for child welfare only when the most rigorous evaluation approaches are used. In addition, child welfare social impact bonds require robust data systems to capture integrated data from each service provider children encounter, so that we can investigate fully what factors influence desired outcomes, both positively and negatively. Finally, it is imperative that all parties involved in social impact bonds are honest when interventions fail, because children like Lisa, John, and Matthew deserve interventions that work.

¹ Pseudonyms have been used to protect the identities of Lisa, John, and Matthew.
² Arazais Oliveros and Joan Kaufman, “Addressing Substance Abuse Treatment Needs of Parents Involved with the Child Welfare System,” Child Welfare 90 (1) (2011): 25–41, available at http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4158612/; Nancy K. Young, Sharon M. Boles, and Cathleen Otero, “Parental Substance Use Disorders and Child Maltreatment: Overlap, Gaps, and Opportunities,” Child Maltreatment 12 (2) (2007): 137–49, available at http://cmx.sagepub.com/content/12/2/137.
³ Susan M. Snyder and Darcey H. Merritt, “Do Childhood Experiences of Neglect Affect Delinquency Among Child Welfare Involved–Youth?” Children and Youth Services Review 46 (2014): 64–71, available at http://www.sciencedirect.com/science/article/pii/S0190740914002941; Joseph P. Ryan and Mark F. Testa, “Child Maltreatment and Juvenile Delinquency: Investigating the Role of Placement and Placement Instability,” Children and Youth Services Review 27 (3) (2005): 227–49, available at http://www.sciencedirect.com/science/article/pii/S0190740904002026.
⁴ Susan M. Snyder and Rachel E. Smith, “Do Physical Abuse, Depression, and Parental Substance Use Influence Patterns of Substance Use Among Child Welfare Involved–Youth?” Substance Use & Misuse 50 (2) (2015): 226–35, available at http://www.ncbi.nlm.nih.gov/pubmed/25338287; Amy Heneghan et al., “Mental Health Problems in Teens Investigated by U.S. Child Welfare Agencies,” Journal of Adolescent Health 52 (5) (2013): 634–40, available at http://www.ncbi.nlm.nih.gov/pubmed/23375826.
⁵ Sara McCarthy and Mark Gladstone, “State Survey of California Prisoners: What Percentage of the State’s Polled Prison Inmates Were Once Foster Care Children?” Policy Matters (California Senate Office of Research: 2011), available at http://www.sor.govoffice3.com/vertical/Sites/%7B3BDD1595-792B-4D20-8D44-626EF05648C7%7D/uploads/Foster_Care_PDF_12-8-11.pdf; Susan M. Snyder and Rose Anne Medeiros, “Typologies of Substance Use and Illegal Behaviors: A Comparison of Emerging Adults with Histories of Foster Care and the General Population,” Children and Youth Services Review 35 (5) (2013): 753–61, available at http://www.sciencedirect.com/science/article/pii/S0190740913000509.
⁶ Xiangming Fang et al., “The Economic Burden of Child Maltreatment in the United States,” Child Abuse & Neglect 36 (2) (2012): 156–165, available at http://www.sciencedirect.com/science/article/pii/S0145213411003140.
⁷ Mary Dozier et al., “Developing Evidence-Based Interventions for Foster Children: An Example of a Randomized Clinical Trial with Infants and Toddlers,” Journal of Social Issues 62 (4) (2006): 765–783, available at http://onlinelibrary.wiley.com/doi/10.1111/j.1540–4560.2006.00486.x/full; Nancy Rolock, Susan M. Snyder, and Cynthia Tate, “Formative Evaluation: A Case Study of the Illinois Birth-to-Three IV-E Waiver,” Journal of Evidence-Informed Social Work 12 (5) (2015): 37–41.
⁸ Joan E. Zweben et al., “Enhancing Family Protective Factors in Residential Treatment for Substance Use Disorders,” Child Welfare 94 (5) (2015): 145–166, available at http://www.ebcrp.org/wp-content/uploads/2016/02/150612_Enhancing-family-protective-factors-in-residential-treatment-for-substanceuse-disorders.pdf; Richard Barth et al., “Parent-Training Programs in Child Welfare Services: Planning for a More Evidence-Based Approach to Serving Biological Parents,” Research on Social Work Practice 15 (5) (2005): 353–371.
⁹ Kristina Rosinsky and Dana Connelly, “Child Welfare Financing SFY 2014: A Survey of Federal, State, and Local Expenditures,” Annie E. Casey Foundation (October 2016), available at https://www.childtrends.org/wp-content/uploads/2016/10/2016-53ChildWelfareFinancingSFY2014.pdf.
¹⁰ Eric Trupin, Nicholas Weiss, and Suzanne U. Kerns, “Social Impact Bonds: Behavioral Health Opportunities,” JAMA Pediatrics 168 (11) (2014): 985–86, available at http://archpedi.jamanetwork.com/article.aspx?articleid=1900479.
¹¹ Crystal Collins-Camargo and Karl Ensign, “Driving Case Outcomes in Child Welfare: Are Performance-Based Contracts the Answer?” Policy & Practice 68 (3) (2010): 11, available at https://www.questia.com/magazine/1G1-229069168/driving-case-outcomes-in-child-welfare-are-performance-based.
¹² Bowen McBeath and William Meezan, “Governance in Motion: Service Provision and Child Welfare Outcomes in a Performance-Based, Managed Care Contracting Environment,” Journal of Public Administration Research & Theory 20 (suppl_1) (2010): i101–i23, available at https://jpart.oxfordjournals.org/content/20/suppl_1/i101.short; Emma Tomkinson, “Outcome-Based Contracting for Human Services,” Evidence Base 1 (1) (2016): 1–20, available at https://journal.anzsog.edu.au/publications/20/EvidenceBase2016Issue1Version1.pdf; Charlotte McCullough, Nancy Pindus, and Elizabeth Lee, “Preparing Effective Contracts in Privatized Child Welfare Systems,” HHS Office of the Assistant Secretary for Planning and Evaluation (2008), available at https://aspe.hhs.gov/basic-report/preparing-effectivecontracts-privatized-child-welfare-systems; Lawrence L. Martin, “Performance-Based Contracting for Human Services: Does It Work?” Administration in Social Work 29 (1) (2005): 63–77.
¹³ Crystal Collins-Camargo and Karl Ensign, “Driving Case Outcomes in Child Welfare: Are Performance-Based Contracts the Answer?” Policy & Practice (3) (2010): 11, available at https://www.questia.com/magazine/1G1-229069168/driving-case-outcomes-in-child-welfare-are-performance-based.
¹⁴ Ibid; Jeffrey B. Liebman, “Social Impact Bonds: A Promising New Financing Model to Accelerate Social Innovation and Improve Government Performance,” Center for American Progress (2011), available at https://www.americanprogress.org/issues/general/report/2011/02/09/9050/social-impact-bonds/.
¹⁵ Laura Callanan, Jonathan Law, and Lenny Mendonca, “From Potential to Action: Bringing Social Impact Bonds to the US,” McKinsey & Company (2012), available at https://nff.org/invest-in-results.
¹⁶ Nancy Rolock, Susan M. Snyder, and Cynthia Tate, “Formative Evaluation: A Case Study of the Illinois Birth-to-Three IV-E Waiver,” Journal of Evidence-Informed Social Work (2015): 1–12, available at http://www.ncbi.nlm.nih.gov/pubmed/25826248; W. Scott Richardson et al., “The Well-Built Clinical Question: A Key to Evidence-Based Decisions,” ACP Journal Club 123 (3) (1995): A12, available at https://acpjc.acponline.org/Content/123/3/issue/ACPJC-1995-123-3-A12.htm.
¹⁷ Liebman, “Social Impact Bonds” (2011).
¹⁸ Kyle Constable and Jacqueline Rabe Thomas, “New Program Announced for Substance Abuse Treatment,” The CT Mirror (February 16, 2016), available at https://ctmirror.org/2016/02/16/new-funding-to-boost-substance-abuse-treatment/.
¹⁹ Yale School of Medicine, “Program for Families Affected by Substance Use (FBR)” (2017), available at https://medicine.yale.edu/childstudy/family/fbr.aspx; Karen E. Hanson et al., “Family-Based Recovery: An Innovative In-Home Substance Abuse Treatment Model for Families with Young Children,” Child Welfare 94 (4) (2015): 161–83.
²⁰ Patrick Lester, “Pay-for-Success in Child Welfare: A Case Study,” Social Innovation Research Center (2015).
²¹ Illinois Department of Children and Family Services, “Illinois Dually-Involved Youth Pay for Success Initiative Ramp-Up Fact Sheet” (2015).
²² Nick Axford and Louis Morpeth, “Evidence-Based Programs in Children’s Services: A Critical Appraisal,” Children and Youth Services Review 35 (2) (2013): 268–77, available at http://www.sciencedirect.com.ezproxy.gsu.edu/science/article/pii/S0190740912003994.
²³ Jeffrey B. Liebman, “Building on Recent Advances in Evidence-Based Policymaking,” Results for America and The Hamilton Project (2013), available at http://www.hamiltonproject.org/assets/legacy/files/downloads_and_links/THP_LiebmanF2_4-13_1.pdf.
²⁴ J. David Hawkins et al., “Unleashing the Power of Prevention,” Institute of Medicine of the National Academies (2015), available at http://aaswsw.org/wp-content/uploads/2013/10/Unleashing-the-Power-of-Prevention-formatted-4.29.15.pdf.
²⁵ For more information see http://safecare.publichealth.gsu.edu/safecare/safecare-research/; Anna Edwards and John R. Lutzker, “Iterations of the SafeCare Model: An Evidence-Based Child Maltreatment Prevention Program,” Behavior Modification 32 (5) (2008): 736–56, available at http://bmo.sagepub.com/content/32/5/736.short.
²⁶ See additional examples for a range of preventative programs at https://www.childwelfare.gov/topics/preventing/programs/.
²⁷ See additional examples at http://www.nctsn.org/resources/topics/treatments-that-work/promising-practices.
²⁸ Liebman, “Social Impact Bonds” (2011).
²⁹ Callanan, “From Potential to Action” (2012).
³⁰ Daniel Stid, “Pay for Success Is Not a Panacea,” Community Development Investment Review (2013): 13–18, available at http://www.frbsf.org/community-development/files/pay-for-success-not-panacea.pdf.
³¹ Lawrence L. Martin, “Performance-Based Contracting for Services: A Survey of NIGP Members,” (Orlando, FL: GSA Training Conference and Expo 2010), available at http://208.112.78.139/gsaSchedule2010/training/ppt/PerformanceBasedContractingForNIGP.ppt.
³² Stid, “Pay for Success Is Not a Panacea” (2013).