Current Genneration Out-of-School Time Programs: What Works, What Doesn't and at What Cost?
Dr. Megan Beckett
Behavioral/Social Scientist, RAND Corporation
Policymakers nationwide are facing the decision about how to best invest in education and related opportunities for their youth populations and, if so, how to allocate these investment across alternative investment opportunities given their respective costs and benefits. In this paper, we review the costs, benefits, and cost and benefits relative to one another for one alternative type of investment: youth programs that are offered during the time that youth are not in school. Such programs are often viewed as a mechanism for addressing the school-age care needs of working parents, for improving youth developmental outcomes, and for reducing the academic achievement gap between advantaged and disadvantaged youth.
Most of the programs considered in this paper are targeted to some extent, with some (which usually provide more intensive programming, such as a case manager, and involve fewer youth) more targeted than others. The youth programs we consider are before and after-school programs, enrichment programs, specialized after-school programs, summer learning programs, and drop-out and teen intervention programs.
What Do We Know About The Reported Costs of youth Programs?
In determining alternative investments in social programs, cost is an important factor. All else equal, policymakers will prefer a program that either serves a larger number of people at the same cost as a program that serves fewer people or a program that serves a fixed number of people at lower cost. Recognizing that cost will feature in the decision of whether to fund new programs or to expand or close existing programs, more and more youth program creators have included cost information into their evaluations and descriptions. We reviewed the costs associated with many of the more prominent youth programs. In order to compare program costs, we compute the cost per hour of service per youth.
We found that most cost data exclude key cost elements and, thus, underestimate the full cost of replicating a program. Most of what is known about program costs relate to operating costs. At the program level, operating costs are likely to account for the majority of costs; some estimates suggest operating costs account for between 60 and 80 percent of total costs. In most cases, even published operating costs are incomplete. Cost estimates of youth programs tend to ignore in-kind resources (e.g., volunteer mentors, community speakers) and omit key operating cost components (e.g., use of donated facilities and janitorial services). Of further concern is that the level of incompleteness in the operating costs varies across the programs making it difficult to compare costs from one program with costs from another.
Given this, our estimated cost per hour of service for the youth programs should be viewed as incomplete and comparisons should be done with care.
Excluding enrichment programs, the lowest-cost programs are the basic before- and after-school programs (and the funding streams that support them): By our best estimates, their cost per hour ranges from $1.17 to $2.57, excluding the two programs that provide a fuller set of services--Beacon’s Initiative and Extended Services Schools Initiatives (which contained several Beacon’s). The costs of these two programs were estimated to be $4.03 and $4.03 per hour per child, respectively.
The specialized after-school programs, summer learning program, and the youth drop-out/intervention programs tend to be much more expensive per hour of service than the lower cost after-school programs. The lowest cost per hour per child of these programs is for group mentoring programs ($3.32); the others range from $5.36 (Children’s Aid Society/Carrera Adolescent Pregnancy Prevention Program) to $8.36 for one-on-one mentoring.
If the aim is to replicate programs throughout a state then the costs of individual programs might not be a good indicator of total program costs across multiple sites. For example, as the program is scaled, operating costs as a proportion of total costs may go down. Additional costs might be incurred, for example to monitor the training and performance of programs. Inclusion of the oft omitted in-kind resources is particularly important if the goal is to replicate a program in many sites, including some sites where it will be difficult to rely on volunteers and donated space. In addition, the costs associated with scaling-up, such as additional training of personal or acquisition of resources would need to be considered.
What Do We Know About the Positive (and Negative) Effects of These Programs?
Our update of an earlier RAND research synthesis of the effects of group-based youth programs on youth participants supports the same broad conclusions: The current generation of youth programs can provide modest positive impacts on academic achievement, academic attainment, and social behaviors, such as pregnancy, and most of the benefits of youth programs are concentrated in programs that are more resource-intensive.
The current synthesis relies on results from evaluations that use the most rigorous design (a controlled experiment) whose integrity was maintained throughout the full evaluation. This is to avoid a bias toward positive results. Research suggests that weaker study designs, such as quasi-experimental and correlation associations, tend to yield more false positives. They are more likely to result in positive results than the more rigorous experimental designs which remove self-selection and other non-observable factors that can contribute to the positive effect. Our synthesis is restricted to evaluations screened for inclusion on either of two websites: “What Works and What Doesn’t Work in Social Policy? Findings from Well-Designed Randomized Controlled Trials” () or a list of studies assigned a “Proven” rating on the Promising Practices Network on Children, Families, and Communities ().
While many after-school programs have undergone less rigorous outcome evaluations, only the evaluation of the 21st Century Community Learning Centers (CCLCs) had a rigorous evaluation design. A key characteristic of 21st CCLC is that like other low-cost after-school programs we reviewed, 21st CCLC is more of an after-school funding stream than a specific after-school program model. Each Center is allowed to design its offerings within some broad guidelines. This means that the results of its evaluation may be considered more akin to what an average (rather than a model) after-school program might expect.
Program participation had an overall negative effect on the participants themselves. In the second year, participants were more likely to be suspended from school and to have been disciplined in school (e.g., missed recess or sent to hall), and their teachers were more likely to have called parents about behavioral problems.
Why might children who participate in average after-school programs act-out during school? There are several hypotheses, such as students may be tired from spending so much time in school, the programs may tolerate behavior that would not be tolerated in school, or the programs are poorly designed and implemented for the effect desired. Further research is necessary to understand what may be happening here and whether this finding is generalizable to other after-school programs and if so, under what circumstances.
Interventions targeting at-risk youth tend to be more research-based and have a longer history of careful program evaluation; for either or both these reasons, more convincing evidence of positive behavioral impacts can be found among targeted (specialized) programs. The Big Brothers Big Sisters of America (BBBSA) program yielded promising results. This program involved matching the intervention group with a volunteer mentor (usually with a college degree) who agreed to meet with the youth (ages 10–16) at least once a week (in most sites) for at least three hours. After 18 months in the program, participants were less likely to have started using illegal drugs or alcohol and less likely to report having hit someone or having skipped school.
Three [PS1] drop-out and teen intervention programs have been rigorously evaluated. The Children’s Aid Society Carrera Adolescent Pregnancy Prevention Program reduced teen pregnancy and births among female participants after four years and increased high school graduation and college enrollment (seven years following program start and three years after program conclusion). Quantum Opportunities Program (QOP) is described as a youth development program for economically disadvantaged youth. Youth, who are called “Associates”, receive year-round services, including comprehensive case management, for high-school years. Associates engage in 250 hours of education, development, and community service activities each year and receive financial incentives for doing so. Compared with the control group, Associates were more likely to graduate from high school, more likely to be in postsecondary school, and less likely to be high school dropouts (Hahn, 1999).
CASASTART is a substance abuse and delinquency prevention program serving high-risk young adolescents and their families. It also involves schools, law enforcement agencies, and social service and health agencies. One year after program completion, CASASTART participants were significantly less likely to have used drugs in the past month, less likely to have reported lifetime sales of drugs and drug sales activity in the last month, and less likely to have committed a violence crime in the year following program completion (Harrell et al., 1998, 1999).
The evaluation results for each of these three drop-out and teen intervention programs suggest potentially powerful impacts if they can be replicated in other settings.
What Do We Know About the Costs Relative to the Benefits
of These Programs?
Policymakers need to decide how to allocate scarce resources among alternatives. Done well, cost-benefit analysis provides useful information for choosing among programs. We review the results of cost-benefit analyses completed on the four youth programs for which positive effects have been found using rigorous evaluation and conclude there is evidence that youth programs may produce benefits that outweigh costs. But limitations in the information available to analysts who wish to conduct cost-benefit analysis restrict their ability to quantify by how much and how consistently effective programs are worth their costs. Primarily, evaluations vary widely in the range of short-term and long-term outcomes they measure. For example, reduction in crime and grade repetition and increase in high school graduation each translate into substantial monetized benefits, yet no evaluation measured all three outcomes. We recommend that future rigorous evaluations of youth programs seek to measure a larger (and consistent) set of outcomes to facilitate cost-benefit analysis.
How Should Policymakers Proceed in Deciding If and How to Invest in Youth Programs Relative to One Another and to Other Alternatives?
At this time, there is enough evidence to suggest that some carefully crafted and implemented youth programs can improve important youth academic and behavioral outcomes. They can reduce drug and alcohol use (BBBSA, CASASTART), violence (BBBSA), and crime (CASASTART), and teen pregnancy and births (The Children’s Aid Society Carrera Adolescent Pregnancy Prevention Program), and they can improve high school graduation rates and enrollment in post-secondary schools (QOP, The Children’s Aid Society Carrera Adolescent Pregnancy Prevention Program). At this time, the evidence is strongest for programs that are costlier and provide more intense resources to youth. It is important to note that all evaluations of these programs are based on at-risk groups. We lack evidence that such programs will benefit youth who are not at-risk or who are less at-risk. Because these programs were designed to provide services for at-risk youth and because other youth are more likely to obtain the needed services elsewhere (like families and schools), we would expect weaker, if any, effects for the average youth.
We also lack evidence that less expensive, less resource-intensive programs, like after-school programs, benefit youth. While evidence from nonrigorous evaluations is largely positive, the one rigorous evaluation of 21st CCLCs suggests that this initiative can produce negative short-term outcomes among program participants, especially boys and children with behavioral problems. More research is needed to assess lower-cost programs and to assess whether there are ways to reduce short-term adverse effects, such as those seen in the 21st CCLCs.
These results do not generalize to the larger population of after-school programs, including those that serve higher-income neighborhoods or those that provide more intensive services (which may include some 21st CCLCs).
Our conclusions about alternative youth programs should be considered preliminary and should be revised as we learn more about the cost and impacts of youth programs. Thus far, only one experimental design evaluation each of after-school and specialized after-school programs has occurred. Given the increasing recognition of the need of additional rigorous evaluation designs in education, such youth programs are likely to use rigorous evaluation designs in the coming years.
There are other considerations policymakers can use in deciding how to allocate across youth programs. In particular, investments can be made to support or improve the quality and content of existing programs.
To reduce the chance of doing harm, investments could be made to understand the circumstances in which after-school programs may contribute to adverse behavioral outcomes. It may be useful to establish model programs--programs that can be laboratories in which practitioners and program developers can observe student behavior and pilot different approaches to avoiding problem behavior and that can be used to evaluate enrichment and other activities before implementing them across multiple programs.
For school-based programs, one can also imagine establishing a continuous quality improvement system that involves monitoring school-based behavioral problems of participants and nonparticipants to immediately identify unintended program consequences, design an intervention to address the problems, and track programs’ success.
Finally, policymakers should keep in mind that moving forward requires advancing the youth programming field and learning what does and does not work. The field will particularly benefit from rigorous and well-done evaluations of large-scale (e.g., statewide) initiatives.
Learn more
| 
