Comprehensive Support and Student Success: Can Out of School Time Make a Difference?

Existing program evaluation research literature that addresses time spent and services provided outside of school typically focuses on specialized interventions that narrowly target specific skill domains or areas of student need among disadvantaged students.² Examples of specialized interventions that target specific skills include academic tutoring (Heinrich, Meyer, and Whitten 2010; Zimmer, Hamilton, and Christina 2010; Heinrich et al. 2014), advising, coaching, and mentoring (Grossman and Tierney 1998; Angrist, Lang, and Oreopoulos 2009; Bettinger and Baker 2014; Castleman, Page, and Schooley 2014), summer reading programs (Kim and Quinn 2013), and behavioral supports (Heller et al. 2013; Cook et al. 2014; Heller et al. 2017). Related interventions address specific areas of student need, such as nutrition, although these interventions are typically carried out within the regular school day (Bhattacharya, Currie, and Haider 2006; Imberman and Kugler 2014; Corcoran, Elbel, and Schwartz 2016).

In contrast, much less is known about the effects of intensive interventions that take a broader view and provide comprehensive education, nutrition, and social support services in a unified setting outside of the regular school day and outside of the existing K–12 system.³ Two existing papers in the literature attempt to address this question, although they do so imperfectly because the programs tested paired the provision of intensive, comprehensive services with generous financial incentives, which have been shown to have strong, independent effects on student outcomes, such as achievement and high school graduation, in some other contexts (Leuven, Oosterbeek, and Klaauw 2010; Ford and Kwakye 2016). Oreopoulos, Brown, and Lavecchia (2017) report quasi-experimental evidence on the effects of the Pathways to Education program, an intensive and comprehensive, out-of-school program for public housing residents in Toronto, Canada. They report that access to Pathways for the duration of high school increased graduation rates by between 5 and 15 percentage points and postsecondary enrollment by between 4 and 19 percentage points. In related work, Rodriguez-Planas (2012) reports results from a multi-site randomized controlled trial of the Quantum Opportunity Program (QOP), which provided intensive and comprehensive services outside of the regular school day to low-achieving high school students. She also finds that the offer of comprehensive services increased high school graduation rates, at least in the short run. However, she finds some evidence to suggest that the control group eventually caught up and graduated high school at rates similar to their treated counterparts.

In this paper, I address this gap in the literature by producing causal evidence on the effects of intensive, comprehensive education, nutrition, and support services provided outside of the regular school day in a unified setting on the early high school outcomes of disadvantaged students. I study this question in the context of StudentU, a nonprofit organization in Durham, North Carolina, that provides these services through a multiyear program that takes advantage of students’ time outside of school after school during the academic year and during summer breaks. To produce evidence on the causal effects of this program, I leverage data from oversubscribed admissions lotteries carried out to allocate slots in the program. By comparing lottery winners and losers, I produce internally valid estimates of the effect of access to comprehensive services on the outcomes of disadvantaged students. I take advantage of the multiyear structure of the program to estimate impacts on student outcomes in the medium term. Specifically, I examine impacts on credit accumulation, grade point average, and suspension as broad measures of how the program affects students’ cognitive and noncognitive skills in early high school.

On average, StudentU lottery winners accumulated more credits and were less likely to be suspended in early high school than their lottery loser counterparts. However, I find strong evidence to suggest that these mean impacts were exclusively driven by lottery winners who entered the program with low levels of baseline achievement. My investigation of treatment effect heterogeneity along this dimension suggests that effects in the full sample of lottery winners were driven by this subgroup. I further find that this subgroup achieved higher grade point averages by the end of ninth grade. Specifically, lottery winners with low baseline achievement accumulated 0.82 more course credits, achieved higher grade point averages (GPAs) by 0.37 grade points, and were 17.1 percentage points less likely to be suspended during ninth grade.

By the end of ninth grade, StudentU lottery winners who entered the program with low baseline achievement had credit accumulation and suspension outcomes that were not only larger on average than their control group counterparts—who had similar achievement test scores at baseline—but were also larger on average than members of the control group who entered the program with high baseline achievement. For GPAs, the increase among StudentU lottery winners with low baseline achievement was large enough to close around 35 percent of the gap between the low- and high-achieving students in the control group.⁴ These findings suggest that comprehensive services provided outside of the regular school may be a particularly effective strategy for improving outcomes of the most disadvantaged students.

To provide evidence on the mechanisms underlying these effects I leverage data on intervening variables to help to distinguish candidate channels of on-time grade progress, course-taking and completion, student achievement, and behavior in school. I find evidence most consistent with on-time grade progress, course completion, and improved behavior in school as primary channels. I do not find any statistically significant evidence of increases in test scores, but I acknowledge that low statistical power is a concern for test scores outcomes. Coupled with the findings on for whom comprehensive services work, these results about mechanisms provide policy-relevant insight into how comprehensive services work to improve student outcomes.

I conclude by presenting the results from back-of-the-envelope calculations designed to investigate how the effects of StudentU on early high school outcomes likely translate into longer-term effects on in high school graduation. Using an index of early high school outcomes and data on past cohorts of first-time ninth-grade students, I predict that lottery winners are around 4 percentage points more likely to graduate from high school than lottery losers. These results are on the low end of the range of effects reported in the previous literature but are still consistent with previous findings about comprehensive programs delivered outside of school. I conclude with a brief discussion of program costs.

This paper contributes to the growing research literature on the effects of comprehensive services provided outside of the regular school day on student outcomes and builds on this previous work by providing evidence from a setting uncontaminated by the addition of financial incentives for participation. The results from this paper provide new, policy-relevant insight into the role that nonprofit and other organizations outside of the K–12 system can play in shaping the outcomes of disadvantaged students outside of the regular school day.

StudentU is a 501(c)(3) nonprofit organization that provides education, nutrition, and related social support services outside of the regular school day to disadvantaged students in Durham, North Carolina. These services include academic programming; healthy meals and snacks; parent/caregiver outreach; coaching, advising, and mentoring; and referrals for children and families to other community services. These comprehensive services are delivered through intensive after-school programming that takes place outside of the regular school day during the academic year and summer programming that occurs between academic years. StudentU's core values are: Energize Your Community, Achieve Greatness, Respect Yourself and Others, Discover Your Best Self, Dream Fearlessly, and Share Your Brilliance. These core values influence the activities of the organization and motivate the organization's mission, which is, “[T]o empower students in the Durham Public Schools to own their education by developing the academic skills and personal well-being necessary to succeed in college and beyond.”⁵

StudentU students are drawn from traditional public schools in Durham Public Schools (DPS), charter schools, and independent (private) schools located within the Durham County, North Carolina. Students enter the program as rising sixth graders—the summer before middle school begins—and remain in the program until high school graduation. Students in the program participate after school (called “School Year Program”) and during the summer (called “Summer Academy”). The School Year Program takes place at a central location on regular attendance days during the academic year. Summer Academy occurs for six weeks during the summer.

StudentU selected its first cohort of students during the spring semester of the 2006–07 academic year and has selected a new cohort of students in the spring of every academic year since then. Following information sessions in schools, directed recruitment through school staff (e.g., principals, teachers, and counselors), and outreach activities conducted by StudentU staff in the community, students are eligible to apply for admission to StudentU in the fall of fifth grade and then receive notice of their admissions decision in the spring of fifth grade. Students formally matriculate into the program during the summer prior to middle school entry (sixth grade). Beginning in the 2011–12 school year, StudentU received applications in excess of its capacity and used lotteries to determine admission to the program. These annual lotteries were conducted separately by gender due to the higher proportion of applications received from female students.⁶ Upon acceptance into the program, StudentU students and their families sign contracts to participate in program activities over the subsequent seven-year period, although in practice sometimes students do not “take-up” the lottery offer or may leave the program before the end of high school. There are no direct costs for students or their families to participate, and transportation is provided for students to attend both the School Year Program and Summer Academy.

Due to the intensity and scope of the comprehensive services provided, StudentU is intentionally small—there are 50 students in each cohort. All students who submit a complete application, which includes short essay questions and a teacher recommendation, are screened against two criteria prior to the lottery: students must be the first person in their immediate family to attend college and/or eligible for free/reduced price lunch. All complete applications that satisfy at least one of these two criteria are entered into that year's admissions lottery. In practice, there are often fewer than 50 slots available each year, since StudentU has historically granted automatic admission to siblings who have an older sibling already enrolled in the program.

Following acceptance into StudentU, rising sixth-grade students attend Summer Academy, an intensive six-week program designed to strengthen students’ academic credentials through small-group instruction and to promote students’ development through elective courses and activities designed to enhance socioemotional learning. During Summer Academy, which meets from 8 a.m. to 4 p.m. five days per week for six weeks, students attend daily, 50-minute classes in Math, Science, English Language Arts, and Global Connect (Social Studies). These small-group classes are led by trained local college students (typically those pursuing a baccalaureate degree in education) or certified teachers from surrounding local school districts who work for StudentU during the summer. In addition to these required courses, students also develop their personal interests by choosing elective courses (e.g., poetry, the arts, or astronomy). Finally, outside of these core and elective courses, StudentU also promotes physical and emotional health through participation in exercise-based activities and wellness classes. Each day also includes one hour of “Family Time,” during which students meet with a regular group of peers and an adult mentor (typically a teacher or other staff member in the program). Family Time is split into three short sessions dispersed throughout the day, thus providing students with regular and sustained adult and peer support throughout the program.

Following Summer Academy, StudentU students participate in 15 hours per week of after school programming each week for 30 weeks during the school year. After school programming includes tutoring and supervised homework time (“Study Skills”), while also allowing students to participate in extracurricular activities ranging from athletics to the arts. Students also participate in structured academic activities outside of their school work each day (“Academic Power Hour”), which are led by program staff and designed to strengthen students’ reading, math, and writing skills. To facilitate the provision of these services, StudentU has 20 full-time employees, including a social worker, a bilingual parent-liaison, and an education specialist. In addition to these full-time staff, additional tutors, mentors, and summer teachers are hired from surrounding universities and local school districts.

Aside from School Year and Summer Academy programming, StudentU also offers supportive services to enrolled students and their families. These services—provided by program staff—include access to counseling, a student support fund (for emergencies), assistance with accessing community resources, one-on-one or small group instruction on academic/study skills, assistance for students/families to access school- or district-provided academic resources, and assistance for families accessing school- or district-provided support services.

Comprehensive programs designed to address multiple areas of skill development or student need may be particularly effective at improving educational outcomes among disadvantaged students because they provide students with access to a suite of services that have been shown to be effective at improving student outcomes in other settings. These include, for example, academic support and nutrition assistance (i.e., access to healthy meals and snacks), which I discuss in turn below. Above and beyond these services, which target skill development and specific areas of need, comprehensive programs also provide support to students in ways that resemble interventions designed to provide coaching, advising, and mentoring. This aspect of comprehensive programs may better allow students to take full advantage of the other services in the program by providing positive relationships, encouragement, and additional assistance when needed. In their analysis of the successes of the Pathways program for public housing residents in Toronto, Oreopoulos, Brown, and Lavecchia (2017) cite qualitative evidence from staff interviews and quantitative evidence from mediation analysis to emphasize the importance of these supportive relationships in facilitating the successes of comprehensive programs.

Much of the previous work on the effects of academic support and instruction provided outside of the regular school day finds that traditional tutoring programs—when delivered in isolation from other supportive services and in “low doses”—are largely ineffective at raising student achievement.⁷ Fryer (2017) reports meta-analytic estimates of the effects of low-dosage tutoring that are 0.074 standard deviation (standard error = 0.045) in math and 0.050 standard deviation (standard error = 0.045) in reading. Using quasi-experimental variation induced by the introduction of supplemental education services (SES) (i.e., tutoring) in school districts under No Child Left Behind, several recent papers investigate the effects of tutoring on student achievement. Heinrich, Meyer, and Whitten (2010) report results from the introduction of SES in Milwaukee Public Schools. They do not find any statistically significant effects on student math or reading gains. In related work, Heinrich et al. (2014) report the results of a multi-site investigation of the same type of services provided to low-income students in four large school districts in the United States. Although they find some evidence of positive math and reading gains in one district, they do not find consistent evidence of positive effects on achievement in any of the other three.

In contrast, tutoring programs that deliver intensive instruction in small group settings (sometimes referred to as “high-dosage” tutoring or tutorials in existing literature) have demonstrated more promising results. Factors such as tutor quality, intensity (days and hours per week), group size, program management, and differentiated instruction are important determinants of whether tutoring programs are successful. In a study of SES in Pittsburgh Public Schools, Zimmer, Hamilton, and Christina (2010) report some evidence of positive test score gains in math and identify differentiated instruction by student skill level, emphasis on knowledge gaps from previous coursework, and tutor experience as key factors affecting success. Fryer (2017) reports meta-analytic estimates of the effects of high-dosage tutoring that are 0.393 (standard error = 0.095) in math and 0.405 (standard error = 0.047) in reading. In related work, Ander, Guryan, and Ludwig (2016) report the results from several randomized controlled trials of intensive “tutorials” (two-on-one tutoring carried out within the regular school day) on the achievement of low-income students. These tutorials, which provide individualized, daily instruction to struggling students, have been shown to increase math test scores by 0.23 standard deviation and reduce the incidence of math course failure by as much as 50 percent. Confirmatory quasi-experimental evidence demonstrates that high-dosage tutoring inside and outside of the regular school day is particularly effective at raising student achievement in math (Fryer 2014; Kraft 2015; Chabrier, Cohodes, and Oreopoulos 2016).

The evidence on the effects of nutrition assistance on student outcomes is more mixed, with some papers reporting positive effects and others reporting null results. In early evaluation of the School Breakfast Program, Bhattacharya, Currie, and Haider (2006) report improvements in nutrition outcomes among students exposed to the program. These included improvements in outcomes like vitamin and mineral intake and decreases in the poor diet outcomes (e.g., the percent of calories obtained from fat). In work investigating the effects of Breakfast in the Classroom (BIC), Imberman and Kugler (2014) report evidence of positive effects on student test scores. By exploiting variation in the timing of BIC introduction, the authors conclude that the likely source of these test score gains is improvements in test performance and not necessarily changes in student learning. In related work, Corcoran, Elbel, and Schwartz (2016) examine the introduction of BIC across schools in New York City, although in that study the authors find no evidence of improvements in student test scores.

Existing literature on the effects of intensive coaching, advising, and mentoring largely comes from interventions designed to improve educational outcomes for students transitioning from high school to postsecondary or from programs for students in the early years of their postsecondary careers. Although the contexts of these interventions are varied, papers in this literature provide insight into the mechanisms that underlie the successes of these interventions. In a study investigating the effects of intensive coaching and mentoring on students’ college application behavior and subsequent enrollment, Carrell and Sacerdote (2017) examine how coaching and mentoring can improve student outcomes when students face intricate and multifaceted challenges that extend over long time horizons (e.g., the college application process). They argue that intensive coaching and mentoring is likely a substitute for the sustained efforts of a parent, teacher, or other support person. Evidence from Bettinger and Baker (2014) provides further insight into the ways in which intensive coaching and mentoring can help students connect their habits, activities, and behaviors to their goals. The authors hypothesize that coaches, advisors, and mentors help students concretely develop strategies and implement action plans in service of their abstract goals. In a randomized controlled trial of coaching for nontraditional college students, the authors find that students exposed to the offer of regular coaching had improved persistence outcomes in a variety of postsecondary settings. Coaching and mentoring effects on student outcomes appear even when the services are provided in small doses. Castleman, Page, and Schooley (2014) find that access to a single 2- to 3-hour session for students transitioning from high school to college improves student enrollment outcomes. In related work, Clotfelter, Hemelt, and Ladd (2018) find that in addition to generous financial aid, nonfinancial supports (e.g., mentoring and social support) improved on-time credit accumulation and grade point averages among low-income, first-generation college students at a large, selective public university.

The data in this paper come from two sources: (1) StudentU application and lottery records and (2) administrative education data from the North Carolina Education Research Data Center (NCERDC) covering the universe of students enrolled in public and charter schools in North Carolina.

All StudentU application and lottery records were matched to existing NCERDC administrative data based on the student's first name, last name, date of birth, and elementary school attended in fifth grade.⁸ The matching procedure carried out by NCERDC staff yielded an overall match rate of 86 percent and was similar to the match rate obtained in other studies that have utilized the same NCERDC data.⁹ StudentU started using admissions lotteries in the spring of 2012, resulting in three separate lottery-based cohorts from the years 2012, 2013, and 2014, respectively. Online appendix Table A1 (available in a separate online appendix that can be accessed on Education Finance and Policy's Web site at https://direct.mit.edu/edfp) summarizes the results from the NCERDC matching procedure. After excluding younger siblings who were granted automatic admission to StudentU outside of the lottery process, 324 out of 376 student applications were matched to a unique student (on the basis of identifiable information) in the NCERDC system. A match rate of 100 percent was not expected, since some students who attend independent schools (private and religious) in Durham County participate in StudentU. These students are not included in the administrative data at NCERDC and therefore could not be matched. The overall match rate was 92 percent for lottery winners and 84 percent for lottery losers, although the pattern of obtaining a higher match rate among lottery winners was not consistent across cohorts. To assess whether treatment status differentially predicted matching, separately by cohort, I regressed a binary indicator for matching (1 = yes) on a binary indicator for treatment. The differences in match rates for the 2012, 2013, and 2014 cohorts were 11.4 percentage points (p = 0.04), −0.6 percentage point (p = 0.92), and 10.9 percentage points (p = 0.12), respectively.¹⁰

Administrative data containing information on student characteristics and outcomes came from NCERDC. These data are provided to NCERDC by the North Carolina Department of Public Instruction and cover the universe of students enrolled in public and charter schools in North Carolina.

The administrative NCERDC data provided baseline student characteristics and post-lottery outcomes. Time-invariant demographics included student gender and race/ethnicity, as well as the following characteristics measured at baseline prior to the lottery (fifth grade): economically disadvantaged,¹¹ age, special education status, gifted designation, English learner (EL) status, and math and reading test scores.

Post-lottery outcomes in early high school included measures constructed from student transcripts and summary measures constructed from the universe of reported disciplinary infractions and their associated consequences (e.g., suspensions). Outcomes constructed from raw transcript data included the number of course credits earned by the end of ninth grade and grade point average.¹² Summary measures constructed from the universe of reported disciplinary infractions included the following binary indicators: any suspension in ninth grade, any violent or weapons-related disciplinary infraction, and any other disciplinary infraction.¹³

To explore patterns of selection into the StudentU applicant pool, table 1 presents descriptive statistics for fifth grade students enrolled in traditional public and charter elementary schools in Durham County, North Carolina and matched, non-sibling StudentU applicants (N = 324; henceforth, I refer to these students as the “matched sample”). Column 1 reports summary statistics for all fifth-grade students in Durham County, while column 4 presents summary statistics for the subset of fifth-grade students who are economically disadvantaged. The subset of students who are economically disadvantaged approximates the population of students in Durham County who are eligible to apply for StudentU, although the approximation is imperfect due to the fact that StudentU requires applicants to be either free or reduced lunch-eligible and/or a first-generation college student not both.¹⁴ Column 7 presents summary statistics for the matched sample of StudentU applicants.¹⁵

Aside from documenting differences between the StudentU applicant pool and other students in Durham County, I provide further context on the educational setting in Durham County by documenting differences between the traditional public and charter sectors. Columns 2 and 3 partition the population of fifth graders in Durham County into those who attend Durham Public Schools and charter schools, respectively. Columns 5 and 6 partition the population of economically disadvantaged fifth graders in Durham County into the same two subgroups. Comparisons of mean characteristics between the traditional public and charter sectors confirm previously documented differences.¹⁶ On average, the charter sector contains lower shares of students who are Hispanic, gifted, ELs, and economically disadvantaged, and a larger share of students who are white. Students attending charter schools also have lower reading achievement on average.

StudentU applicants differ on observables from other public and charter students in Durham County. Column 8 presents differences in means and associated standard errors for StudentU applicants versus all other fifth grade students in Durham County. Relative to all other fifth-grade students in Durham County, StudentU applicants are more likely to be female (8.9 percentage points [pp]), Hispanic (20.8 pp), designated as ELs (4.9 pp), and economically disadvantaged (20.8 pp), and they are less likely to be white (18.6 pp).

Column 9 presents differences in means for StudentU applicants and all economically disadvantaged fifth-grade students in Durham County. StudentU applicants differ on observables from the population of economically disadvantaged students in Durham County. Relative to economically disadvantaged students, StudentU applicants are more likely to be female (9.0 pp), Hispanic (11.3 pp), gifted (5.7 pp), and enrolled in a charter school (4.5 pp). They are less likely to be black (9.9 pp), white (3.4 pp), designated as special education (4.7 pp), and economically disadvantaged (16.7 pp). They are also slightly younger when starting fifth grade (0.059 year) and have higher baseline test scores in math and reading (0.330 and 0.277 SD, respectively) than the population of economically disadvantaged fifth-grade students as a whole.

Table 2 presents results from covariate balance tests designed to compare StudentU lottery winners and losers in the matched sample (N = 324). Columns 1 and 2 report summary statistics to describe lottery winners and lottery losers, respectively, while column 3 presents the difference in means between the lottery winners and lottery losers, conditional on cohort fixed-effects and cohort fixed-effects interacted with gender. Column 4 reports the p-value for the test of statistical significance for the difference in means.

Becaues lottery offers were allocated randomly within gender subgroups, there should not be any differences between the lottery winners and lottery losers, conditional on cohort fixed-effects and cohort fixed-effects interacted with gender. This is the case with two exceptions: There is imbalance between lottery winners and lottery losers in terms of the percent of Hispanic students and share of students who are economically disadvantaged at baseline (fifth grade). Lottery winners are 14.1 percentage points more likely than lottery losers to be Hispanic and 11.8 percentage points more likely to be economically disadvantaged. I do not find any other statistically significant differences between the lottery winners and lottery losers, and I cannot reject the null hypothesis (F-test) that the covariates are jointly zero as predictors of treatment status.¹⁷ As a precaution, I control for student characteristics and covariates measured at baseline in all subsequent analysis. These controls should account for any ways in which these characteristics may have influenced student outcomes. To the extent that this measured imbalance affects my estimates, however, it is likely to bias treatment effect estimates toward zero, since the treatment group is more economically disadvantaged than the control group at baseline.

The main threats to the internal validity of my estimates come from data missingness (i.e., missing baseline covariates) and sample attrition. In this paper, sample attrition (i.e., missing ninth-grade outcome data) is the result of (1) students exiting the public/charter system in North Carolina or (2) repeating a grade following the StudentU lottery but prior to ninth grade. Sample attrition does not result from dropping out of StudentU or leaving Durham County, as outcomes are tracked so long as the student remains enrolled in a public or charter school in North Carolina.

Online appendix table A4 summarizes frequencies and rates of non-missing data by source, overall and separately by lottery group. When considered row by row, the table illustrates how the “matched sample” (N = 324) becomes the “regression sample” (N = 279). From the sequence of rows, two facts emerge: First, missing outcome data account for most of the reduction in sample size, although missing baseline covariates contribute, too. Second, both of these phenomena have larger impacts on lottery losers. The table also makes clear that (1) missing baseline demographic characteristics are not a problem (first row), (2) the issue of missing baseline test scores is small (4 percent) but different between lottery groups (lottery winners are 4.8 percentage points more likely to have non-missing baseline test scores than lottery losers), and (3) after considering both missing baseline covariates and missing ninth-grade outcomes, lottery winners are around 11.8 percentage points more likely to be in the “regression sample” than lottery losers (fourth row).

Missing covariates and missing ninth-grade outcomes could bias my treatment effect estimates if students who have missing covariates or outcomes differ systematically from those who do not. I explore this issue in detail for lottery losers, since the issue is very small for lottery winners (N < 10).¹⁸ Online appendix table A5 presents the results from balance tests among lottery losers comparing baseline characteristics for students who do and do not end up in the regression sample. Reassuringly, I do not find much evidence that these groups differ on observables, except for the percent of students who are white and student age (in the fall of fifth grade). These differences, although statistically significant, are not practically meaningful. Given the small share of students in the StudentU applicant pool who are white (2.5 percent), this 2.7 percentage point difference between groups is not unexpected. The 0.36-year age difference (4 months) is also not practically relevant. The results from these balance tests alleviate some concerns about lottery losers either (i) attriting due to lottery status (e.g., discouragement or failing to make on-time grade progress)¹⁹ or (ii) compensating for lottery status (e.g., exiting the public/charter system for private school), although it is possible that these two effects cancel each other out. As final reassurance that differentially missing outcomes are not responsible for my findings, I augment presentation of main results with results from three standard methods for dealing with missing outcome data: mean imputation, inverse probability weighting, and multiple imputation. Reassuringly, the main results are very similar to those obtained when using these methods. I present the results in online appendix tables D1–D9.

To estimate the effect of a StudentU lottery offer and StudentU enrollment on student outcomes in early high school, I implement a two-stage least-squares (2SLS) approach that exploits the program's lottery-based admissions system. Using student-level information on lottery offers combined with information on program take-up, I obtain both intent-to-treat (ITT) and treatment-on-the-treated (TOT) estimates that capture the effects of lottery offers and enrollment, respectively. Below I present my estimating equations, discuss key threats to validity, and report first-stage results.

In equation 1, $Yic$ is an outcome (e.g., the number of course credits earned by the end of ninth grade) for student i in cohort c. $SUic$ is a binary variable equal to one if student i won the lottery in cohort c. $dij$ are cohort-specific dummies, which are included on their own and interacted with gender (j = 2012, 2013, 2014). The interacted dummies account for the fact that StudentU conducted annual admissions lotteries separately for male and female students. $Xic$ is a vector of student-level covariates obtained at baseline (fifth grade), including: gender, race/ethnicity (black, Hispanic, other), charter enrollment, economically disadvantaged, age, gifted designation, special education placement, and EL status. All baseline covariates were measured during the student's fifth-grade school year, prior to the time when the student learned about whether he or she had received a StudentU lottery offer. The error term, $ɛic$⁠, is assumed to be uncorrelated with all other determinants of the outcome and captures random fluctuations in student-level outcomes.

Equation 2 captures the effect of a lottery offer, $SUic$⁠, on enrollment in StudentU, $Enrollic$⁠, for student i in cohort c, conditional on the same covariates, cohort dummies, and cohort dummies interacted with gender as in equation 1. From this first-stage equation I obtained predicted values of the binary enrollment indicator, $Enroll^ic$⁠, and included these on the right-hand side in the second stage, which is summarized by equation 3. Second-stage estimates of $β1$ can be interpreted as the average effect of enrollment in StudentU on student outcomes among the compliers (i.e., those students induced to enroll in StudentU by the lottery offer). This effect is equal to the ratio of $α1$⁠, the effect of receiving a lottery offer on student outcomes (numerator), to$κ1$⁠, the effect of receiving a lottery offer on the likelihood of enrollment in StudentU (denominator).

Table A6 in the online appendix reports first-stage estimation results from equation 2. The results in column 1 indicate that StudentU lottery winners were around 85.7 percentage points more likely to enroll in StudentU than lottery losers. Enrollment is defined here like “take-up,” reflecting any amount of participation in formal StudentU programming (i.e., Summer Academy or the School Year Program).²⁰ The results in columns 2 and 3 demonstrate that addition of student-level demographic covariates and baseline test scores have minimal effects on the magnitude of this point estimate, as is expected if lottery winners and losers are balanced on observables at baseline. Columns 6 and 7 present results separately by baseline achievement test scores.²¹ Low-achieving lottery winners are slightly more likely to take up the offer of enrollment than high achieving lottery winners (88.0 percent take-up versus 86.6 percent take-up).

Panel A of table 3 presents ITT estimates representing the effect of a lottery offer on the number of credits earned by the end of ninth grade. The point estimate in column 3 indicates that lottery winners accumulated around 0.45 more course credits than lottery losers by the end of ninth grade. This translates into a 6 percent increase relative to the control group mean of 7.04 credits. This point estimate is nearly identical to the point estimates presented in columns 1 and 2, which come from specifications that exclude baseline student covariates and test scores, respectively. Panel B presents TOT results for the same credit accumulation outcome. The TOT estimate represents the effect of enrolling in StudentU on credits earned in ninth grade among the compliers. Due to high take-up rates, the TOT results are very similar to the ITT estimates.

To investigate treatment heterogeneity on the basis of prior achievement, I next present results from a specification in which lottery status is interacted with an index of baseline student achievement. This index of baseline achievement is an average of each student's standardized math and reading test scores from fifth grade. I divided the sample into two groups: low and high baseline achievement, respectively, based on the sample median (−0.26 SD). Students in the high baseline achievement group have an average index score of 0.38 SD (median = 0.26 SD), while students in the low baseline achievement group have an average index score of −0.90 SD (median = −0.80 SD).

Columns 4 and 5 present results from the interacted model and reveal clear evidence of heterogeneous treatment effects. The point estimate for lottery winners with low baseline achievement is 0.822 credits and is statistically significant at the p = 0.05 level, while the point estimate for lottery winners with high baseline achievement is 0.082 credits and is statistically indistinguishable from zero. Column 5 reports the p-value from a statistical test of the difference in these two coefficients, which allows me to reject the null hypothesis of equality (p = 0.003). Once again, panel B presents TOT estimates that are qualitatively very similar to the ITT estimates.

Panel A of table 4 presents ITT estimates representing the effect of a lottery offer on student grade point average at the end of ninth grade. The point estimates for the full sample in columns 1–3 are small in magnitude and statistically insignificant. Columns 4 and 5 present results from a specification in which lottery status is interacted with an index of baseline student achievement and once again reveal evidence of significant treatment heterogeneity. The point estimate for lottery winners with low baseline achievement is 0.370 weighted grade points and is statistically significant at the 0.05 level. In contrast, the point estimate for lottery winners with high baseline achievement is small, slightly negative, and statistically indistinguishable from zero. The p-value from a statistical test of the difference in these two coefficients in column 5 is 0.027, which allows me to once again reject the null hypothesis of equality. Panel B presents TOT estimates, which are again qualitatively similar.

Panel A of table 5 presents estimates of the effect of a lottery offer on the probability of suspension during ninth grade. The point estimate for the full sample in column 3 indicates that lottery winners were around 10.3 percentage points less likely to be suspended during ninth grade than their lottery loser counterparts. In relative terms, this translates into a 47 percent reduction (control mean is 23.7). The point estimates in columns 4 and 5 reveal significant evidence of treatment effect heterogeneity on the basis of baseline achievement. The point estimate in column 4 reports a statistically significant 17.1 percentage point reduction in the likelihood of suspension in ninth grade among lottery winners with low baseline achievement. In contrast, column 4 reports a 3.6 percentage point reduction that is statistically insignificant. The p-value from a statistical test of the null hypothesis of equality is 0.092 (marginally significant). The results from the interacted model once again provide strong evidence to suggest that the mean impact in the full sample of lottery winners is driven by effects on the subgroup of students with low baseline achievement.

I find substantial evidence to suggest that the effects of StudentU on credit accumulation, GPA, and suspension were concentrated among lottery winners who entered the program with low baseline achievement, as measured by fifth-grade math and reading test scores. By the end of ninth grade, these students, who entered the program with average fifth-grade math and reading test scores nearly one standard deviation below the statewide mean (mean = −0.90 SD and median = −0.80 SD among this subgroup), more closely resembled—and in some cases had better outcomes than—their control group (lottery loser) counterparts who entered the program with high baseline achievement. In the cases of credits earned and reduced probability of suspension, lottery winners with low baseline achievement had average outcomes that exceeded the average outcomes of their control group counterparts who entered the program with high baseline achievement. In the case of grade point average, the reduced-form treatment effect was large enough to close around 35 percent of the gap between low baseline and high baseline achievers in the control group. In contrast to Oreopoulos, Brown, and Lavecchia (2017), who report that the effects of the Pathways program are largest among students with high baseline achievement, this paper documents substantial and consistent evidence of effects concentrated among students with low achievement at baseline. Heterogeneous treatment effects by prior achievement are not reported in Rodriguez-Planas (2012), although she does report significant evidence of heterogeneous treatment effects by gender.²² Taken together with previous work, these findings suggest that comprehensive services delivered outside of the regular school day in a unified setting may be most effective at improving educational outcomes among students with low levels of academic achievement at baseline. Extra time outside of school in a small-group and structured setting may improve study skills, provide students with the support needed to keep up with regular assignments and homework, and offer opportunities to ask questions about assigned work. These findings are consistent with results reported in Kraft (2015), who finds that low-achieving students benefit more than high-achieving students from small-group tutorials that offer individualized and differentiated instruction outside of the regular school day. The findings are also consistent with experimental evidence from Schueler (2020), who finds that small-group academic instruction in math—even in small weeklong doses—provided by a single teacher led to an increased likelihood that students were proficient in math and to less exclusionary discipline.

StudentU delivers all of the components of its programming to students simultaneously, which makes it impossible—in this setting—to identify how specific parts of the program (e.g., small group instruction, small-group mentoring, nutrition, child/family services) affected student outcomes. It also makes it impossible to identify whether there were complementarities between the program's components that made them more effective because they were delivered together rather than in isolation. These questions, although unanswered here, are important and deserve additional attention in future research. Despite these limitations, however, it is worth noting that the structure of StudentU programming shares several features identified in previous work as critical to the success of out-of-school time interventions. Among other factors, Schwartz et al. (2018) identify “high-quality enrichment experiences” and “a high level of engagement between adults and students” as factors crucial to the success of out of school time programs.

To gain insight into the mechanisms underlying increased credit accumulation, higher grade point averages, and reduced likelihood of suspension among lottery winners with low baseline achievement, I investigated several intervening variables that provide insight into the causal channels through which StudentU affects student outcomes. To distinguish between channels of on-time grade progress, course-taking and completion, achievement (cognitive skills), and behavior, I reestimated the same models for several intervening variables that serve as proxies for these channels.

Table 6 presents ITT estimates of the effect of a lottery offer on on-time grade progress following admission to StudentU (i.e., for sixth through ninth grades). The point estimate in column 3 indicates that lottery winners were around 2.1 percentage points (2 percent) more likely to make on-time grade progress than lottery losers, although the effect is only marginally statistically significant. The point estimate for lottery winners with low baseline achievement is 3.2 percentage points and statistically significant, while the point estimate for lottery winners with high baseline achievement is 0.1 percentage point and statistically indistinguishable from zero, although I cannot reject the null hypothesis of equality (p = 0.133). For completeness, I present TOT estimates for the same outcomes in online appendix table A8.

Table 7 presents ITT estimates representing the effect of a lottery offer on credits attempted in ninth grade and a binary variable indicating whether the student failed any courses during ninth grade. The point estimates in panel A do not reveal and statistically significant effects on credits attempted, although the point estimates in panel B do reveal substantial reductions in the likelihood of any course failure. The point estimate in column 3 indicates that lottery winners in the full sample are around 8.6 percentage points less likely to fail any courses relative to their lottery loser counterparts. The point estimates in columns 4 and 5 make clear that this effect is entirely driven by lottery winners with low baseline achievement, who are 20.4 percentage points less likely to fail any courses during ninth grade. This contrasts with the small and statistically insignificant point estimate among lottery winners with high baseline achievement. The p-value from a statistical test of the difference between these coefficients is p = 0.008, which means that I can reject the null hypothesis of equality. For completeness, I present TOT estimates for the same outcomes in online appendix table A9. These results are qualitatively similar to the reduced-form effects.

I do not find any statistically significant evidence to suggest that improvements in cognitive skills are a likely mechanism through which lottery winners improve their early high school outcomes, although I acknowledge that low statistical power is a concern for these outcomes.²³ Table 8 presents ITT estimates of the effect of winning the lottery on math and reading test scores in eighth grade.²⁴ For completeness, I present TOT estimates for the same outcomes in online appendix table A10.

Table 9 presents ITT estimates representing the effect of a lottery offer on the likelihood of a violent/weapons-related and other disciplinary infractions in ninth grade. Although I do not find any evidence of reductions in the likelihood of violent/weapons-related infractions, I find substantial evidence of reductions in the likelihood of other disciplinary infractions in panel B. The point estimates in column 4 indicate that lottery winners in the full sample are around 10.3 percentage points less likely to have a reported other infraction in ninth grade. The point estimates in columns 4 and 5 reveal that this main effect is likely driven by lottery winners with low baseline achievement. The point estimate for lottery winners with low baseline achievement is 16.1 percentage points and statistically significant, while the point estimate for lottery winners with high baseline achievement is 4.6 percentage points and statistically indistinguishable from zero, although I cannot reject the null hypothesis of equality (p = 0.146).

Taken together, the results suggest that StudentU improved course completion and reduced disciplinary infractions in school among lottery winners with low baseline achievement. I do not find any evidence of any improvements in student achievement. This is consistent with previous literature on the effects of tutoring outside of the regular school day (Heinrich, Meyer, and Whitten 2010; Zimmer, Hamilton, and Christina 2010; Heinrich et al. 2014). It contrasts with work that reports results from one-on-one or two-on-one, high-dosage tutoring that occurs within the context of the regular school day (see, e.g., Fryer 2014; Kraft 2015; Ander, Guryan, and Ludwig 2016; Chabrier, Cohodes, and Oreopoulos 2016).

To assess the likely effects of improved early high school outcomes on high school graduation, I constructed an index based on students’ early high school outcomes and estimated the empirical relationship between this index and the likelihood of subsequent high school graduation. The binary index (henceforth, the Early High School Outcomes Index) was equal to one if at the end of ninth grade a student (1) had earned the minimum number of credits required to transition to tenth grade and (2) had not received any suspensions. The Early High School Outcomes Index was equal to zero otherwise. I then estimated the relationship between the Early High School Outcomes Index and the likelihood that a student graduated from high school within five years using data from three successive cohorts of first-time ninth graders in Durham County (this includes students in traditional public and charter schools). Although this relationship was estimated among cohorts of first-time ninth graders in Durham County, the statewide longitudinal data permit me to track high school graduation from any public or charter school in the state of North Carolina.

Figure 2a plots the empirical relationship between the Early High School Outcomes Index and the likelihood of graduation from high school within five years. A one-unit change in the Early High School Outcomes Index is associated with a 29 percentage point increase in the likelihood that a student graduates from high school within five years (control mean = 0.59). Although there is some variation across cohorts, the relationship appears to be quite stable.

Figure 2b depicts the effect of winning the StudentU lottery on the likelihood that the Early High School Outcomes Index is equal to one. The pooled estimate indicates that StudentU lottery winners were around 12 percentage points more likely to have an Early High School Outcomes Index equal to one when compared with their lottery loser counterparts (control mean = 0.51). Based on these estimated relationships, I calculate an expected graduation rate of 77 percent for lottery winners and 73 percent for lottery losers, which implies that the effect of winning the lottery on the likelihood of high school graduation is a 4 percentage point increase. This predicted effect on high school graduation is considerably smaller than the actual effect realized in the Pathways program at the original site of implementation (15.3 percentage points), although it is similar in magnitude to the Pathways effect at the expansion site (5.8 percentage points) and to the (statistically insignificant) effect reported for the QOP (3.2 percentage points) (Oreopoulos, Brown, and Lavecchia 2017; Rodriguez-Planas 2012).

The predicted effect of StudentU on high school graduation is large and meaningful but should not be considered in isolation. When evaluating any educational intervention, it is important to consider not only the effects—and their social value—but also the level of investment required to obtain said impacts. To obtain estimates of per student spending for StudentU, I used three years’ worth of annual expenditure data broken down by program and one year's official tax records that reported expenditures broken down by category (e.g., salary/wages, insurance, supplies, food).²⁵ Using these data, I estimated per student spending between $4,790 and $5,905 (2016 dollars) per year. Summing these costs over four years and assuming a real discount rate of 7 percent, I obtained a total cost estimate of $16,225 to $20,002 (2016 dollars) per student, which represents the total cost per student for four years of StudentU programming and services (this corresponds to the number of years between a student's matriculation into the program and the early high school outcomes observed in this paper). This range of cost estimates compares favorably to estimates from other comprehensive programs and related programs in the literature. Oreopoulos, Brown, and Lavecchia (2017) report a per student cost estimate of $13,400 for Pathways and Rodriguez-Planas (2012) reports a per student cost estimate of $25,000 for QOP (4-year program).

In this paper I investigate the effects of StudentU, a comprehensive, multiyear program designed to improve educational outcomes among disadvantaged students, on the early high school outcomes of participating students. To do this, I exploit variation induced by offers from oversubscribed admissions lotteries. Although I find some evidence of effects on early high school outcomes among the full sample of lottery winners, more detailed investigation reveals that these were driven exclusively by the subgroup of lottery winners who entered the program with low baseline achievement.

In an investigation of the mechanisms underlying these effects, I find evidence to suggest that on-time grade progress, course completion, and reduced reports of disciplinary infractions in school are the likely channels through which these effects flowed. Specifically, I find that lottery winners were around 3.2 percentage points more likely to make on-time grade progress, 20.4 percentage points less likely to fail a course during ninth grade, and around 10.3 percentage points less likely to be reported for a miscellaneous (i.e., not violent and not weapons-related) disciplinary infraction during ninth grade.

As a means to forecast longer-run impacts on students’ high school graduation outcomes, I conclude by presenting the results from a simple prediction exercise designed assess how these improvements in early high school outcomes likely translate into successes later in high school. Using an index of students’ early high school outcomes, I predict that lottery winners are around 4 percentage points more likely to graduate from high school than lottery losers (5 percent effect). These results are squarely in line with estimated effects presented in the previous literature. I also report estimates of per-student program costs, which also compare favorably to estimates reported for similar comprehensive programs. Taken together, the results from this paper suggest that comprehensive services delivered outside of the regular school day have the potential to improve the educational outcomes of disadvantaged students outside of the regular school day.

I thank the Editors, Randall Reback and Andrew McEachin, and three anonymous referees for their helpful comments and suggestions. I am grateful to Kara Bonneau and Radhika Deshmukh-McDiarmid for their assistance in matching the data for this project. Liz Ananat, Steve Bednar, Barbara Biasi, Sandy Black, Russ Bittmann, Charlie Clotfelter, Sue Dynarski, Matt Johnson, Ezra Karger, Sunny Ladd, Alicia Modestino, Nolan Pope, Seth Sanders, Elizabeth Setren, Crystal Zhan, and participants at the UNC-CH Educational Inequality Workshop, Elon University, the AEFP 44th Annual Conference, the SEA 88th Annual Meeting, the 2019 Triangle Applied Micro Conference, and the 2019 Carolina Region Empirical Economics Day provided helpful comments on this draft and on earlier versions of this paper. Graeme Peterson provided exceptional research assistance. All errors are my own.