StudentU is a comprehensive program that provides education, nutrition, and social support services to disadvantaged middle and high school students outside of the regular school day. In this paper I investigate the effects of this multiyear program on the early high school outcomes of participating students by exploiting data from oversubscribed admissions lotteries. I find that the subgroup of lottery winners who entered the comprehensive program with low baseline achievement earned more course credits (0.82 credits), achieved higher grade point averages (0.37 grade points), and were less likely to be suspended (17.1 percentage points) during ninth grade than their lottery loser counterparts. Investigation of intervening variables indicates that on-time grade progress and decreases in course failure and disciplinary infractions are potential mediating channels. Using an index of early high school outcomes, I predict that lottery winners are around 4 percentage points more likely to graduate from high school than lottery losers (5 percent effect). These results suggest that comprehensive services delivered outside of the regular school day have the potential to improve the educational outcomes of disadvantaged students.

Gaps in skills between children born into low- versus high-income families exist at all levels of K–12 schooling in the United States. In some cases, these gaps appear prior to kindergarten entry and then widen as students progress through the K–12 system (Duncan and Magnuson 2011; Farkas 2011; Reardon 2011). Long-studied policy interventions designed to ameliorate income-based skill gaps frequently take aim at the existing K–12 system. These include school resource policies, input policies (e.g., class size, innovative curricula, and high-quality teachers), and other market-based reforms (e.g., charter schools, voucher programs). In contrast, much less attention has been devoted to understanding the effects of interventions that target another point of intervention: time spent outside of school. Structured activities and services provided outside of the regular school day—typically after school or during the summer—are increasingly the focus of public investment in the United States, but little is known about their effects on student outcomes. Figure 1 depicts appropriations between 1995 and 2018 for the main federal program that serves children outside of the regular school day in the United States, the 21st Century Community Learning Center program.1 Despite a nearly one thousand–fold increase in real federal support for this program over the past two decades, very little is known about how time spent outside of school affects student success.
Figure 1.
Federal Appropriations for 21st Century Community Learnings Centers in the United States, 1995—2018

Notes: This figure depicts federal appropriations (in 2017 dollars) for the Twenty-First Century Community Learning Center Program, which was authorized by Title I, Part X of the Elementary and Secondary Education Act (ESEA). This program provides funding for schools in the United States to implement activities that provide educational, recreational, cultural, health, and social services outside of the regular school day. For more information, see Stapleton (1998) and McCallion (2001).

Figure 1.
Federal Appropriations for 21st Century Community Learnings Centers in the United States, 1995—2018

Notes: This figure depicts federal appropriations (in 2017 dollars) for the Twenty-First Century Community Learning Center Program, which was authorized by Title I, Part X of the Elementary and Secondary Education Act (ESEA). This program provides funding for schools in the United States to implement activities that provide educational, recreational, cultural, health, and social services outside of the regular school day. For more information, see Stapleton (1998) and McCallion (2001).

Close modal

Existing program evaluation research literature that addresses time spent and services provided outside of school typically focuses on specialized interventions that narrowly target specific skill domains or areas of student need among disadvantaged students.2 Examples of specialized interventions that target specific skills include academic tutoring (Heinrich, Meyer, and Whitten 2010; Zimmer, Hamilton, and Christina 2010; Heinrich et al. 2014), advising, coaching, and mentoring (Grossman and Tierney 1998; Angrist, Lang, and Oreopoulos 2009; Bettinger and Baker 2014; Castleman, Page, and Schooley 2014), summer reading programs (Kim and Quinn 2013), and behavioral supports (Heller et al. 2013; Cook et al. 2014; Heller et al. 2017). Related interventions address specific areas of student need, such as nutrition, although these interventions are typically carried out within the regular school day (Bhattacharya, Currie, and Haider 2006; Imberman and Kugler 2014; Corcoran, Elbel, and Schwartz 2016).

In contrast, much less is known about the effects of intensive interventions that take a broader view and provide comprehensive education, nutrition, and social support services in a unified setting outside of the regular school day and outside of the existing K–12 system.3 Two existing papers in the literature attempt to address this question, although they do so imperfectly because the programs tested paired the provision of intensive, comprehensive services with generous financial incentives, which have been shown to have strong, independent effects on student outcomes, such as achievement and high school graduation, in some other contexts (Leuven, Oosterbeek, and Klaauw 2010; Ford and Kwakye 2016). Oreopoulos, Brown, and Lavecchia (2017) report quasi-experimental evidence on the effects of the Pathways to Education program, an intensive and comprehensive, out-of-school program for public housing residents in Toronto, Canada. They report that access to Pathways for the duration of high school increased graduation rates by between 5 and 15 percentage points and postsecondary enrollment by between 4 and 19 percentage points. In related work, Rodriguez-Planas (2012) reports results from a multi-site randomized controlled trial of the Quantum Opportunity Program (QOP), which provided intensive and comprehensive services outside of the regular school day to low-achieving high school students. She also finds that the offer of comprehensive services increased high school graduation rates, at least in the short run. However, she finds some evidence to suggest that the control group eventually caught up and graduated high school at rates similar to their treated counterparts.

On average, StudentU lottery winners accumulated more credits and were less likely to be suspended in early high school than their lottery loser counterparts. However, I find strong evidence to suggest that these mean impacts were exclusively driven by lottery winners who entered the program with low levels of baseline achievement. My investigation of treatment effect heterogeneity along this dimension suggests that effects in the full sample of lottery winners were driven by this subgroup. I further find that this subgroup achieved higher grade point averages by the end of ninth grade. Specifically, lottery winners with low baseline achievement accumulated 0.82 more course credits, achieved higher grade point averages (GPAs) by 0.37 grade points, and were 17.1 percentage points less likely to be suspended during ninth grade.

By the end of ninth grade, StudentU lottery winners who entered the program with low baseline achievement had credit accumulation and suspension outcomes that were not only larger on average than their control group counterparts—who had similar achievement test scores at baseline—but were also larger on average than members of the control group who entered the program with high baseline achievement. For GPAs, the increase among StudentU lottery winners with low baseline achievement was large enough to close around 35 percent of the gap between the low- and high-achieving students in the control group.4 These findings suggest that comprehensive services provided outside of the regular school may be a particularly effective strategy for improving outcomes of the most disadvantaged students.

To provide evidence on the mechanisms underlying these effects I leverage data on intervening variables to help to distinguish candidate channels of on-time grade progress, course-taking and completion, student achievement, and behavior in school. I find evidence most consistent with on-time grade progress, course completion, and improved behavior in school as primary channels. I do not find any statistically significant evidence of increases in test scores, but I acknowledge that low statistical power is a concern for test scores outcomes. Coupled with the findings on for whom comprehensive services work, these results about mechanisms provide policy-relevant insight into how comprehensive services work to improve student outcomes.

I conclude by presenting the results from back-of-the-envelope calculations designed to investigate how the effects of StudentU on early high school outcomes likely translate into longer-term effects on in high school graduation. Using an index of early high school outcomes and data on past cohorts of first-time ninth-grade students, I predict that lottery winners are around 4 percentage points more likely to graduate from high school than lottery losers. These results are on the low end of the range of effects reported in the previous literature but are still consistent with previous findings about comprehensive programs delivered outside of school. I conclude with a brief discussion of program costs.

This paper contributes to the growing research literature on the effects of comprehensive services provided outside of the regular school day on student outcomes and builds on this previous work by providing evidence from a setting uncontaminated by the addition of financial incentives for participation. The results from this paper provide new, policy-relevant insight into the role that nonprofit and other organizations outside of the K–12 system can play in shaping the outcomes of disadvantaged students outside of the regular school day.

### StudentU's Mission

StudentU is a 501(c)(3) nonprofit organization that provides education, nutrition, and related social support services outside of the regular school day to disadvantaged students in Durham, North Carolina. These services include academic programming; healthy meals and snacks; parent/caregiver outreach; coaching, advising, and mentoring; and referrals for children and families to other community services. These comprehensive services are delivered through intensive after-school programming that takes place outside of the regular school day during the academic year and summer programming that occurs between academic years. StudentU's core values are: Energize Your Community, Achieve Greatness, Respect Yourself and Others, Discover Your Best Self, Dream Fearlessly, and Share Your Brilliance. These core values influence the activities of the organization and motivate the organization's mission, which is, “[T]o empower students in the Durham Public Schools to own their education by developing the academic skills and personal well-being necessary to succeed in college and beyond.”5

StudentU students are drawn from traditional public schools in Durham Public Schools (DPS), charter schools, and independent (private) schools located within the Durham County, North Carolina. Students enter the program as rising sixth graders—the summer before middle school begins—and remain in the program until high school graduation. Students in the program participate after school (called “School Year Program”) and during the summer (called “Summer Academy”). The School Year Program takes place at a central location on regular attendance days during the academic year. Summer Academy occurs for six weeks during the summer.

Due to the intensity and scope of the comprehensive services provided, StudentU is intentionally small—there are 50 students in each cohort. All students who submit a complete application, which includes short essay questions and a teacher recommendation, are screened against two criteria prior to the lottery: students must be the first person in their immediate family to attend college and/or eligible for free/reduced price lunch. All complete applications that satisfy at least one of these two criteria are entered into that year's admissions lottery. In practice, there are often fewer than 50 slots available each year, since StudentU has historically granted automatic admission to siblings who have an older sibling already enrolled in the program.

### StudentU Programming

Following acceptance into StudentU, rising sixth-grade students attend Summer Academy, an intensive six-week program designed to strengthen students’ academic credentials through small-group instruction and to promote students’ development through elective courses and activities designed to enhance socioemotional learning. During Summer Academy, which meets from 8 a.m. to 4 p.m. five days per week for six weeks, students attend daily, 50-minute classes in Math, Science, English Language Arts, and Global Connect (Social Studies). These small-group classes are led by trained local college students (typically those pursuing a baccalaureate degree in education) or certified teachers from surrounding local school districts who work for StudentU during the summer. In addition to these required courses, students also develop their personal interests by choosing elective courses (e.g., poetry, the arts, or astronomy). Finally, outside of these core and elective courses, StudentU also promotes physical and emotional health through participation in exercise-based activities and wellness classes. Each day also includes one hour of “Family Time,” during which students meet with a regular group of peers and an adult mentor (typically a teacher or other staff member in the program). Family Time is split into three short sessions dispersed throughout the day, thus providing students with regular and sustained adult and peer support throughout the program.

Following Summer Academy, StudentU students participate in 15 hours per week of after school programming each week for 30 weeks during the school year. After school programming includes tutoring and supervised homework time (“Study Skills”), while also allowing students to participate in extracurricular activities ranging from athletics to the arts. Students also participate in structured academic activities outside of their school work each day (“Academic Power Hour”), which are led by program staff and designed to strengthen students’ reading, math, and writing skills. To facilitate the provision of these services, StudentU has 20 full-time employees, including a social worker, a bilingual parent-liaison, and an education specialist. In addition to these full-time staff, additional tutors, mentors, and summer teachers are hired from surrounding universities and local school districts.

Aside from School Year and Summer Academy programming, StudentU also offers supportive services to enrolled students and their families. These services—provided by program staff—include access to counseling, a student support fund (for emergencies), assistance with accessing community resources, one-on-one or small group instruction on academic/study skills, assistance for students/families to access school- or district-provided academic resources, and assistance for families accessing school- or district-provided support services.

### Previous Work

Comprehensive programs designed to address multiple areas of skill development or student need may be particularly effective at improving educational outcomes among disadvantaged students because they provide students with access to a suite of services that have been shown to be effective at improving student outcomes in other settings. These include, for example, academic support and nutrition assistance (i.e., access to healthy meals and snacks), which I discuss in turn below. Above and beyond these services, which target skill development and specific areas of need, comprehensive programs also provide support to students in ways that resemble interventions designed to provide coaching, advising, and mentoring. This aspect of comprehensive programs may better allow students to take full advantage of the other services in the program by providing positive relationships, encouragement, and additional assistance when needed. In their analysis of the successes of the Pathways program for public housing residents in Toronto, Oreopoulos, Brown, and Lavecchia (2017) cite qualitative evidence from staff interviews and quantitative evidence from mediation analysis to emphasize the importance of these supportive relationships in facilitating the successes of comprehensive programs.

Much of the previous work on the effects of academic support and instruction provided outside of the regular school day finds that traditional tutoring programs—when delivered in isolation from other supportive services and in “low doses”—are largely ineffective at raising student achievement.7 Fryer (2017) reports meta-analytic estimates of the effects of low-dosage tutoring that are 0.074 standard deviation (standard error = 0.045) in math and 0.050 standard deviation (standard error = 0.045) in reading. Using quasi-experimental variation induced by the introduction of supplemental education services (SES) (i.e., tutoring) in school districts under No Child Left Behind, several recent papers investigate the effects of tutoring on student achievement. Heinrich, Meyer, and Whitten (2010) report results from the introduction of SES in Milwaukee Public Schools. They do not find any statistically significant effects on student math or reading gains. In related work, Heinrich et al. (2014) report the results of a multi-site investigation of the same type of services provided to low-income students in four large school districts in the United States. Although they find some evidence of positive math and reading gains in one district, they do not find consistent evidence of positive effects on achievement in any of the other three.

In contrast, tutoring programs that deliver intensive instruction in small group settings (sometimes referred to as “high-dosage” tutoring or tutorials in existing literature) have demonstrated more promising results. Factors such as tutor quality, intensity (days and hours per week), group size, program management, and differentiated instruction are important determinants of whether tutoring programs are successful. In a study of SES in Pittsburgh Public Schools, Zimmer, Hamilton, and Christina (2010) report some evidence of positive test score gains in math and identify differentiated instruction by student skill level, emphasis on knowledge gaps from previous coursework, and tutor experience as key factors affecting success. Fryer (2017) reports meta-analytic estimates of the effects of high-dosage tutoring that are 0.393 (standard error = 0.095) in math and 0.405 (standard error = 0.047) in reading. In related work, Ander, Guryan, and Ludwig (2016) report the results from several randomized controlled trials of intensive “tutorials” (two-on-one tutoring carried out within the regular school day) on the achievement of low-income students. These tutorials, which provide individualized, daily instruction to struggling students, have been shown to increase math test scores by 0.23 standard deviation and reduce the incidence of math course failure by as much as 50 percent. Confirmatory quasi-experimental evidence demonstrates that high-dosage tutoring inside and outside of the regular school day is particularly effective at raising student achievement in math (Fryer 2014; Kraft 2015; Chabrier, Cohodes, and Oreopoulos 2016).

The evidence on the effects of nutrition assistance on student outcomes is more mixed, with some papers reporting positive effects and others reporting null results. In early evaluation of the School Breakfast Program, Bhattacharya, Currie, and Haider (2006) report improvements in nutrition outcomes among students exposed to the program. These included improvements in outcomes like vitamin and mineral intake and decreases in the poor diet outcomes (e.g., the percent of calories obtained from fat). In work investigating the effects of Breakfast in the Classroom (BIC), Imberman and Kugler (2014) report evidence of positive effects on student test scores. By exploiting variation in the timing of BIC introduction, the authors conclude that the likely source of these test score gains is improvements in test performance and not necessarily changes in student learning. In related work, Corcoran, Elbel, and Schwartz (2016) examine the introduction of BIC across schools in New York City, although in that study the authors find no evidence of improvements in student test scores.

Existing literature on the effects of intensive coaching, advising, and mentoring largely comes from interventions designed to improve educational outcomes for students transitioning from high school to postsecondary or from programs for students in the early years of their postsecondary careers. Although the contexts of these interventions are varied, papers in this literature provide insight into the mechanisms that underlie the successes of these interventions. In a study investigating the effects of intensive coaching and mentoring on students’ college application behavior and subsequent enrollment, Carrell and Sacerdote (2017) examine how coaching and mentoring can improve student outcomes when students face intricate and multifaceted challenges that extend over long time horizons (e.g., the college application process). They argue that intensive coaching and mentoring is likely a substitute for the sustained efforts of a parent, teacher, or other support person. Evidence from Bettinger and Baker (2014) provides further insight into the ways in which intensive coaching and mentoring can help students connect their habits, activities, and behaviors to their goals. The authors hypothesize that coaches, advisors, and mentors help students concretely develop strategies and implement action plans in service of their abstract goals. In a randomized controlled trial of coaching for nontraditional college students, the authors find that students exposed to the offer of regular coaching had improved persistence outcomes in a variety of postsecondary settings. Coaching and mentoring effects on student outcomes appear even when the services are provided in small doses. Castleman, Page, and Schooley (2014) find that access to a single 2- to 3-hour session for students transitioning from high school to college improves student enrollment outcomes. In related work, Clotfelter, Hemelt, and Ladd (2018) find that in addition to generous financial aid, nonfinancial supports (e.g., mentoring and social support) improved on-time credit accumulation and grade point averages among low-income, first-generation college students at a large, selective public university.

The data in this paper come from two sources: (1) StudentU application and lottery records and (2) administrative education data from the North Carolina Education Research Data Center (NCERDC) covering the universe of students enrolled in public and charter schools in North Carolina.

### Matching StudentU Applications to NCERDC Data

All StudentU application and lottery records were matched to existing NCERDC administrative data based on the student's first name, last name, date of birth, and elementary school attended in fifth grade.8 The matching procedure carried out by NCERDC staff yielded an overall match rate of 86 percent and was similar to the match rate obtained in other studies that have utilized the same NCERDC data.9 StudentU started using admissions lotteries in the spring of 2012, resulting in three separate lottery-based cohorts from the years 2012, 2013, and 2014, respectively. Online appendix Table A1 (available in a separate online appendix that can be accessed on Education Finance and Policy's Web site at https://direct.mit.edu/edfp) summarizes the results from the NCERDC matching procedure. After excluding younger siblings who were granted automatic admission to StudentU outside of the lottery process, 324 out of 376 student applications were matched to a unique student (on the basis of identifiable information) in the NCERDC system. A match rate of 100 percent was not expected, since some students who attend independent schools (private and religious) in Durham County participate in StudentU. These students are not included in the administrative data at NCERDC and therefore could not be matched. The overall match rate was 92 percent for lottery winners and 84 percent for lottery losers, although the pattern of obtaining a higher match rate among lottery winners was not consistent across cohorts. To assess whether treatment status differentially predicted matching, separately by cohort, I regressed a binary indicator for matching (1 = yes) on a binary indicator for treatment. The differences in match rates for the 2012, 2013, and 2014 cohorts were 11.4 percentage points (p = 0.04), −0.6 percentage point (p = 0.92), and 10.9 percentage points (p = 0.12), respectively.10

### Student-Level Data from NCERDC

Administrative data containing information on student characteristics and outcomes came from NCERDC. These data are provided to NCERDC by the North Carolina Department of Public Instruction and cover the universe of students enrolled in public and charter schools in North Carolina.

The administrative NCERDC data provided baseline student characteristics and post-lottery outcomes. Time-invariant demographics included student gender and race/ethnicity, as well as the following characteristics measured at baseline prior to the lottery (fifth grade): economically disadvantaged,11 age, special education status, gifted designation, English learner (EL) status, and math and reading test scores.

Post-lottery outcomes in early high school included measures constructed from student transcripts and summary measures constructed from the universe of reported disciplinary infractions and their associated consequences (e.g., suspensions). Outcomes constructed from raw transcript data included the number of course credits earned by the end of ninth grade and grade point average.12 Summary measures constructed from the universe of reported disciplinary infractions included the following binary indicators: any suspension in ninth grade, any violent or weapons-related disciplinary infraction, and any other disciplinary infraction.13

### Descriptive Statistics

To explore patterns of selection into the StudentU applicant pool, table 1 presents descriptive statistics for fifth grade students enrolled in traditional public and charter elementary schools in Durham County, North Carolina and matched, non-sibling StudentU applicants (N = 324; henceforth, I refer to these students as the “matched sample”). Column 1 reports summary statistics for all fifth-grade students in Durham County, while column 4 presents summary statistics for the subset of fifth-grade students who are economically disadvantaged. The subset of students who are economically disadvantaged approximates the population of students in Durham County who are eligible to apply for StudentU, although the approximation is imperfect due to the fact that StudentU requires applicants to be either free or reduced lunch-eligible and/or a first-generation college student not both.14 Column 7 presents summary statistics for the matched sample of StudentU applicants.15

Table 1.

Descriptive Statistics, Fifth-Grade Students in Durham County, 2011—12 to 2013—14

(1)(2)(3)(4)(5)(6)(7)(8)(9)
DurhamDPSCharterStudentUDiff.:Diff.:
DurhamDPSCharterEDEDEDApplicants(7)—(1)(7)—(4)
Female 0.495 0.494 0.496 0.493 0.492 0.505 0.583 0.089*** 0.090***
(0.500) (0.500) (0.500) (0.500) (0.500) (0.501) (0.494) (0.028) (0.028)
Black 0.500 0.499 0.512 0.578 0.559 0.792 0.478 −0.022 −0.099***
(0.500) (0.500) (0.500) (0.494) (0.497) (0.407) (0.500) (0.028) (0.029)
Hispanic 0.224 0.245 0.078 0.319 0.335 0.137 0.432 0.208*** 0.113***
(0.417) (0.430) (0.268) (0.466) (0.472) (0.344) (0.496) (0.028) (0.028)
White 0.211 0.190 0.354 0.059 0.061 0.037 0.025 −0.186*** −0.034***
(0.408) (0.392) (0.478) (0.235) (0.239) (0.189) (0.155) (0.010) (0.009)
Special Ed. 0.173 0.171 0.182 0.189 0.190 0.178 0.142 −0.031 −0.047**
(0.378) (0.377) (0.386) (0.392) (0.392) (0.383) (0.350) (0.020) (0.020)
Gifted 0.237 0.264 0.053 0.149 0.162 0.007 0.207 −0.030 0.057**
(0.425) (0.441) (0.224) (0.357) (0.368) (0.083) (0.406) (0.023) (0.023)
EL 0.118 0.129 0.037 0.167 0.176 0.065 0.167 0.049** −0.001
(0.322) (0.336) (0.190) (0.373) (0.381) (0.246) (0.373) (0.021) (0.021)
Age 10.524 10.530 10.484 10.568 10.568 10.569 10.509 −0.016 −0.059**
(0.464) (0.464) (0.460) (0.490) (0.486) (0.528) (0.484) (0.027) (0.028)
Econ. Disad. 0.625 0.659 0.393 1.000 1.000 1.000 0.833 0.208*** −0.167***
(0.484) (0.474) (0.489) (0.000) (0.000) (0.000) (0.373) (0.021) (0.021)
Charter 0.129 0.000 1.000 0.081 0.000 1.000 0.127 −0.002 0.045**
(0.335) (0.000) (0.000) (0.273) (0.000) (0.000) (0.333) (0.019) (0.019)
Math Ach.a −0.224 −0.221 −0.244 −0.521 −0.501 −0.746 −0.192 0.032 0.330***
(1.042) (1.048) (1.005) (0.927) (0.930) (0.866) (0.863) (0.050) (0.051)
Reading Ach.b −0.241 −0.264 −0.090 −0.603 −0.601 −0.626 −0.338 −0.096* 0.266***
(1.082) (1.083) (1.068) (0.966) (0.966) (0.965) (0.911) (0.053) (0.054)
N 8,576 7,473 1,103 5,328 4,896 432 324
(1)(2)(3)(4)(5)(6)(7)(8)(9)
DurhamDPSCharterStudentUDiff.:Diff.:
DurhamDPSCharterEDEDEDApplicants(7)—(1)(7)—(4)
Female 0.495 0.494 0.496 0.493 0.492 0.505 0.583 0.089*** 0.090***
(0.500) (0.500) (0.500) (0.500) (0.500) (0.501) (0.494) (0.028) (0.028)
Black 0.500 0.499 0.512 0.578 0.559 0.792 0.478 −0.022 −0.099***
(0.500) (0.500) (0.500) (0.494) (0.497) (0.407) (0.500) (0.028) (0.029)
Hispanic 0.224 0.245 0.078 0.319 0.335 0.137 0.432 0.208*** 0.113***
(0.417) (0.430) (0.268) (0.466) (0.472) (0.344) (0.496) (0.028) (0.028)
White 0.211 0.190 0.354 0.059 0.061 0.037 0.025 −0.186*** −0.034***
(0.408) (0.392) (0.478) (0.235) (0.239) (0.189) (0.155) (0.010) (0.009)
Special Ed. 0.173 0.171 0.182 0.189 0.190 0.178 0.142 −0.031 −0.047**
(0.378) (0.377) (0.386) (0.392) (0.392) (0.383) (0.350) (0.020) (0.020)
Gifted 0.237 0.264 0.053 0.149 0.162 0.007 0.207 −0.030 0.057**
(0.425) (0.441) (0.224) (0.357) (0.368) (0.083) (0.406) (0.023) (0.023)
EL 0.118 0.129 0.037 0.167 0.176 0.065 0.167 0.049** −0.001
(0.322) (0.336) (0.190) (0.373) (0.381) (0.246) (0.373) (0.021) (0.021)
Age 10.524 10.530 10.484 10.568 10.568 10.569 10.509 −0.016 −0.059**
(0.464) (0.464) (0.460) (0.490) (0.486) (0.528) (0.484) (0.027) (0.028)
Econ. Disad. 0.625 0.659 0.393 1.000 1.000 1.000 0.833 0.208*** −0.167***
(0.484) (0.474) (0.489) (0.000) (0.000) (0.000) (0.373) (0.021) (0.021)
Charter 0.129 0.000 1.000 0.081 0.000 1.000 0.127 −0.002 0.045**
(0.335) (0.000) (0.000) (0.273) (0.000) (0.000) (0.333) (0.019) (0.019)
Math Ach.a −0.224 −0.221 −0.244 −0.521 −0.501 −0.746 −0.192 0.032 0.330***
(1.042) (1.048) (1.005) (0.927) (0.930) (0.866) (0.863) (0.050) (0.051)
Reading Ach.b −0.241 −0.264 −0.090 −0.603 −0.601 −0.626 −0.338 −0.096* 0.266***
(1.082) (1.083) (1.068) (0.966) (0.966) (0.965) (0.911) (0.053) (0.054)
N 8,576 7,473 1,103 5,328 4,896 432 324

Notes: Columns 1—3 report means and standard deviations (in parentheses) for the variable listed in each row. Column 1 reports descriptive statistics for 5th-grade students in Durham, NC (Durham Public Schools and charter schools in Durham County) pooled across the 2011—12 to 2013—14 school years. Columns 2 and 3 partition 5th-grade students in Durham County into students in the Durham Public Schools (DPS) and students enrolled in charter schools in Durham County. Columns 4—6 present information for the same three groups as in columns 1—3 but restricted to the subsample of students who were economically disadvantaged. Column 7 reports descriptive statistics for StudentU Applicants who applied for admission in the spring semesters of 2012—14. Column 8 reports the difference in means and associated standard error (in parentheses) between

Columns 1 and 7. Column 9 reports the difference in means and associated standard error (in parentheses) between columns 4 and 7. Special Ed. = special education; EL = English learner; Econ. Disad. = economically disadvantaged; Ach. = achievement.

aN = 311 for math achievement.

bN = 310 for reading achievement.

Aside from documenting differences between the StudentU applicant pool and other students in Durham County, I provide further context on the educational setting in Durham County by documenting differences between the traditional public and charter sectors. Columns 2 and 3 partition the population of fifth graders in Durham County into those who attend Durham Public Schools and charter schools, respectively. Columns 5 and 6 partition the population of economically disadvantaged fifth graders in Durham County into the same two subgroups. Comparisons of mean characteristics between the traditional public and charter sectors confirm previously documented differences.16 On average, the charter sector contains lower shares of students who are Hispanic, gifted, ELs, and economically disadvantaged, and a larger share of students who are white. Students attending charter schools also have lower reading achievement on average.

StudentU applicants differ on observables from other public and charter students in Durham County. Column 8 presents differences in means and associated standard errors for StudentU applicants versus all other fifth grade students in Durham County. Relative to all other fifth-grade students in Durham County, StudentU applicants are more likely to be female (8.9 percentage points [pp]), Hispanic (20.8 pp), designated as ELs (4.9 pp), and economically disadvantaged (20.8 pp), and they are less likely to be white (18.6 pp).

Column 9 presents differences in means for StudentU applicants and all economically disadvantaged fifth-grade students in Durham County. StudentU applicants differ on observables from the population of economically disadvantaged students in Durham County. Relative to economically disadvantaged students, StudentU applicants are more likely to be female (9.0 pp), Hispanic (11.3 pp), gifted (5.7 pp), and enrolled in a charter school (4.5 pp). They are less likely to be black (9.9 pp), white (3.4 pp), designated as special education (4.7 pp), and economically disadvantaged (16.7 pp). They are also slightly younger when starting fifth grade (0.059 year) and have higher baseline test scores in math and reading (0.330 and 0.277 SD, respectively) than the population of economically disadvantaged fifth-grade students as a whole.

### Covariate Balance Tests, Data Missingness, and Attrition

Table 2 presents results from covariate balance tests designed to compare StudentU lottery winners and losers in the matched sample (N = 324). Columns 1 and 2 report summary statistics to describe lottery winners and lottery losers, respectively, while column 3 presents the difference in means between the lottery winners and lottery losers, conditional on cohort fixed-effects and cohort fixed-effects interacted with gender. Column 4 reports the p-value for the test of statistical significance for the difference in means.

Table 2.

Covariate Balance Tests

(1)(2)(3)(4)
Lottery WinnersLottery LosersDiff.: Winners vs. Losersp-value
Black 0.420 0.504 −0.085 0.166
(0.496) (0.501) (0.061)
Hispanic 0.520 0.393 0.141 0.022
(0.502) (0.489) (0.061)
White 0.030 0.022 0.005 0.823
(0.171) (0.148) (0.021)
Special education 0.110 0.156 −0.060 0.151
(0.314) (0.364) (0.042)
(0.429) (0.395) (0.051)
EL 0.180 0.161 0.023 0.618
(0.386) (0.368) (0.047)
Age 10.479 10.522 −0.036 0.495
(0.362) (0.529) (0.053)
Economically disadvantaged 0.910 0.799 0.118 0.005
(0.288) (0.402) (0.042)
Charter 0.130 0.125 −0.007 0.867
(0.338) (0.331) (0.041)
Math achievementa −0.100 −0.234 0.130 0.220
(0.820) (0.881) (0.106)
Reading achievementb −0.259 −0.375 0.087 0.445
(0.864) (0.931) (0.114)
p-value (joint F-test)c    0.178
Observations 100 224 324
(1)(2)(3)(4)
Lottery WinnersLottery LosersDiff.: Winners vs. Losersp-value
Black 0.420 0.504 −0.085 0.166
(0.496) (0.501) (0.061)
Hispanic 0.520 0.393 0.141 0.022
(0.502) (0.489) (0.061)
White 0.030 0.022 0.005 0.823
(0.171) (0.148) (0.021)
Special education 0.110 0.156 −0.060 0.151
(0.314) (0.364) (0.042)
(0.429) (0.395) (0.051)
EL 0.180 0.161 0.023 0.618
(0.386) (0.368) (0.047)
Age 10.479 10.522 −0.036 0.495
(0.362) (0.529) (0.053)
Economically disadvantaged 0.910 0.799 0.118 0.005
(0.288) (0.402) (0.042)
Charter 0.130 0.125 −0.007 0.867
(0.338) (0.331) (0.041)
Math achievementa −0.100 −0.234 0.130 0.220
(0.820) (0.881) (0.106)
Reading achievementb −0.259 −0.375 0.087 0.445
(0.864) (0.931) (0.114)
p-value (joint F-test)c    0.178
Observations 100 224 324

Notes: Column 1 reports raw means and standard deviations for StudentU lottery winners who applied for and received a lottery offer in the spring semesters of 2012—14. Column 2 reports raw means and standard deviations for StudentU lottery losers who applied for but did not receive a lottery offer in the spring semesters of 2012—2014. Column 3 reports the difference in means and associated standard errors in parentheses for StudentU lottery winners versus lottery losers, conditional on cohort fixed-effects and cohort fixed-effects interacted with gender. Column 4 reports the p-value associated with a two-tailed t-test of the difference in means reported in Column 3. The sample includes first-time, non-sibling applicants to StudentU in the 2011—12 to 2013—14 school years. EL = English learner.

aN = 311 for math achievement.

bN = 310 for reading achievement.

cN = 310 for the regression in which covariates are predictors of treatment status.

Becaues lottery offers were allocated randomly within gender subgroups, there should not be any differences between the lottery winners and lottery losers, conditional on cohort fixed-effects and cohort fixed-effects interacted with gender. This is the case with two exceptions: There is imbalance between lottery winners and lottery losers in terms of the percent of Hispanic students and share of students who are economically disadvantaged at baseline (fifth grade). Lottery winners are 14.1 percentage points more likely than lottery losers to be Hispanic and 11.8 percentage points more likely to be economically disadvantaged. I do not find any other statistically significant differences between the lottery winners and lottery losers, and I cannot reject the null hypothesis (F-test) that the covariates are jointly zero as predictors of treatment status.17 As a precaution, I control for student characteristics and covariates measured at baseline in all subsequent analysis. These controls should account for any ways in which these characteristics may have influenced student outcomes. To the extent that this measured imbalance affects my estimates, however, it is likely to bias treatment effect estimates toward zero, since the treatment group is more economically disadvantaged than the control group at baseline.

The main threats to the internal validity of my estimates come from data missingness (i.e., missing baseline covariates) and sample attrition. In this paper, sample attrition (i.e., missing ninth-grade outcome data) is the result of (1) students exiting the public/charter system in North Carolina or (2) repeating a grade following the StudentU lottery but prior to ninth grade. Sample attrition does not result from dropping out of StudentU or leaving Durham County, as outcomes are tracked so long as the student remains enrolled in a public or charter school in North Carolina.

Online appendix table A4 summarizes frequencies and rates of non-missing data by source, overall and separately by lottery group. When considered row by row, the table illustrates how the “matched sample” (N = 324) becomes the “regression sample” (N = 279). From the sequence of rows, two facts emerge: First, missing outcome data account for most of the reduction in sample size, although missing baseline covariates contribute, too. Second, both of these phenomena have larger impacts on lottery losers. The table also makes clear that (1) missing baseline demographic characteristics are not a problem (first row), (2) the issue of missing baseline test scores is small (4 percent) but different between lottery groups (lottery winners are 4.8 percentage points more likely to have non-missing baseline test scores than lottery losers), and (3) after considering both missing baseline covariates and missing ninth-grade outcomes, lottery winners are around 11.8 percentage points more likely to be in the “regression sample” than lottery losers (fourth row).

Missing covariates and missing ninth-grade outcomes could bias my treatment effect estimates if students who have missing covariates or outcomes differ systematically from those who do not. I explore this issue in detail for lottery losers, since the issue is very small for lottery winners (N < 10).18 Online appendix table A5 presents the results from balance tests among lottery losers comparing baseline characteristics for students who do and do not end up in the regression sample. Reassuringly, I do not find much evidence that these groups differ on observables, except for the percent of students who are white and student age (in the fall of fifth grade). These differences, although statistically significant, are not practically meaningful. Given the small share of students in the StudentU applicant pool who are white (2.5 percent), this 2.7 percentage point difference between groups is not unexpected. The 0.36-year age difference (4 months) is also not practically relevant. The results from these balance tests alleviate some concerns about lottery losers either (i) attriting due to lottery status (e.g., discouragement or failing to make on-time grade progress)19 or (ii) compensating for lottery status (e.g., exiting the public/charter system for private school), although it is possible that these two effects cancel each other out. As final reassurance that differentially missing outcomes are not responsible for my findings, I augment presentation of main results with results from three standard methods for dealing with missing outcome data: mean imputation, inverse probability weighting, and multiple imputation. Reassuringly, the main results are very similar to those obtained when using these methods. I present the results in online appendix tables D1–D9.

To estimate the effect of a StudentU lottery offer and StudentU enrollment on student outcomes in early high school, I implement a two-stage least-squares (2SLS) approach that exploits the program's lottery-based admissions system. Using student-level information on lottery offers combined with information on program take-up, I obtain both intent-to-treat (ITT) and treatment-on-the-treated (TOT) estimates that capture the effects of lottery offers and enrollment, respectively. Below I present my estimating equations, discuss key threats to validity, and report first-stage results.

### Estimating Equations

To estimate the reduced-form effect of a StudentU lottery offer on student outcomes, I use an equation of the following form:
$Yic=α0+α1×SUic+∑jγjdij+∑jλjdij×Femalei+Γ'Xic+ɛic.$
(1)

In equation 1, $Yic$ is an outcome (e.g., the number of course credits earned by the end of ninth grade) for student i in cohort c. $SUic$ is a binary variable equal to one if student i won the lottery in cohort c. $dij$ are cohort-specific dummies, which are included on their own and interacted with gender (j = 2012, 2013, 2014). The interacted dummies account for the fact that StudentU conducted annual admissions lotteries separately for male and female students. $Xic$ is a vector of student-level covariates obtained at baseline (fifth grade), including: gender, race/ethnicity (black, Hispanic, other), charter enrollment, economically disadvantaged, age, gifted designation, special education placement, and EL status. All baseline covariates were measured during the student's fifth-grade school year, prior to the time when the student learned about whether he or she had received a StudentU lottery offer. The error term, $ɛic$, is assumed to be uncorrelated with all other determinants of the outcome and captures random fluctuations in student-level outcomes.

To estimate the effect of StudentU enrollment on student outcomes, I implement a 2SLS approach using the following two equations:
$Enrollic=κ0+κ1×SUic+∑jσjdij+∑jιjdij×Femalei+Θ'Xic+νic.$
(2)
$Yic=β0+β1×Enroll^ic+∑jμjdij+∑jζjdij×Femalei+Δ'Xic+ηic.$
(3)

Equation 2 captures the effect of a lottery offer, $SUic$, on enrollment in StudentU, $Enrollic$, for student i in cohort c, conditional on the same covariates, cohort dummies, and cohort dummies interacted with gender as in equation 1. From this first-stage equation I obtained predicted values of the binary enrollment indicator, $Enroll^ic$, and included these on the right-hand side in the second stage, which is summarized by equation 3. Second-stage estimates of $β1$ can be interpreted as the average effect of enrollment in StudentU on student outcomes among the compliers (i.e., those students induced to enroll in StudentU by the lottery offer). This effect is equal to the ratio of $α1$, the effect of receiving a lottery offer on student outcomes (numerator), to$κ1$, the effect of receiving a lottery offer on the likelihood of enrollment in StudentU (denominator).

### First-Stage Results

Table A6 in the online appendix reports first-stage estimation results from equation 2. The results in column 1 indicate that StudentU lottery winners were around 85.7 percentage points more likely to enroll in StudentU than lottery losers. Enrollment is defined here like “take-up,” reflecting any amount of participation in formal StudentU programming (i.e., Summer Academy or the School Year Program).20 The results in columns 2 and 3 demonstrate that addition of student-level demographic covariates and baseline test scores have minimal effects on the magnitude of this point estimate, as is expected if lottery winners and losers are balanced on observables at baseline. Columns 6 and 7 present results separately by baseline achievement test scores.21 Low-achieving lottery winners are slightly more likely to take up the offer of enrollment than high achieving lottery winners (88.0 percent take-up versus 86.6 percent take-up).

### The Effect of StudentU on Credits Earned in Early High School

Panel A of table 3 presents ITT estimates representing the effect of a lottery offer on the number of credits earned by the end of ninth grade. The point estimate in column 3 indicates that lottery winners accumulated around 0.45 more course credits than lottery losers by the end of ninth grade. This translates into a 6 percent increase relative to the control group mean of 7.04 credits. This point estimate is nearly identical to the point estimates presented in columns 1 and 2, which come from specifications that exclude baseline student covariates and test scores, respectively. Panel B presents TOT results for the same credit accumulation outcome. The TOT estimate represents the effect of enrolling in StudentU on credits earned in ninth grade among the compliers. Due to high take-up rates, the TOT results are very similar to the ITT estimates.

Table 3.

The Effects of StudentU on Credits Earned in Early High School

All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. ITT
Won lottery 0.463*** 0.480*** 0.452*** 0.822*** 0.082 0.003
(0.159) (0.160) (0.161) (0.209) (0.191)
Panel B. TOT
Enrolled in SU 0.541*** 0.558*** 0.524*** 0.940*** 0.096 0.003
(0.183) (0.182) (0.181) (0.231) (0.218)
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
Control mean  7.04  6.68 7.45
Observations  279  143 136
All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. ITT
Won lottery 0.463*** 0.480*** 0.452*** 0.822*** 0.082 0.003
(0.159) (0.160) (0.161) (0.209) (0.191)
Panel B. TOT
Enrolled in SU 0.541*** 0.558*** 0.524*** 0.940*** 0.096 0.003
(0.183) (0.182) (0.181) (0.231) (0.218)
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
Control mean  7.04  6.68 7.45
Observations  279  143 136

Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01.

To investigate treatment heterogeneity on the basis of prior achievement, I next present results from a specification in which lottery status is interacted with an index of baseline student achievement. This index of baseline achievement is an average of each student's standardized math and reading test scores from fifth grade. I divided the sample into two groups: low and high baseline achievement, respectively, based on the sample median (−0.26 SD). Students in the high baseline achievement group have an average index score of 0.38 SD (median = 0.26 SD), while students in the low baseline achievement group have an average index score of −0.90 SD (median = −0.80 SD).

Columns 4 and 5 present results from the interacted model and reveal clear evidence of heterogeneous treatment effects. The point estimate for lottery winners with low baseline achievement is 0.822 credits and is statistically significant at the p = 0.05 level, while the point estimate for lottery winners with high baseline achievement is 0.082 credits and is statistically indistinguishable from zero. Column 5 reports the p-value from a statistical test of the difference in these two coefficients, which allows me to reject the null hypothesis of equality (p = 0.003). Once again, panel B presents TOT estimates that are qualitatively very similar to the ITT estimates.

### The Effect of StudentU on Grade Point Average in Early High School

Panel A of table 4 presents ITT estimates representing the effect of a lottery offer on student grade point average at the end of ninth grade. The point estimates for the full sample in columns 1–3 are small in magnitude and statistically insignificant. Columns 4 and 5 present results from a specification in which lottery status is interacted with an index of baseline student achievement and once again reveal evidence of significant treatment heterogeneity. The point estimate for lottery winners with low baseline achievement is 0.370 weighted grade points and is statistically significant at the 0.05 level. In contrast, the point estimate for lottery winners with high baseline achievement is small, slightly negative, and statistically indistinguishable from zero. The p-value from a statistical test of the difference in these two coefficients in column 5 is 0.027, which allows me to once again reject the null hypothesis of equality. Panel B presents TOT estimates, which are again qualitatively similar.

Table 4.

The Effects of StudentU on Grade Point Average (GPA) in Early High School

All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. ITT
Won lottery 0.166 0.176 0.140 0.370*** −0.090 0.027
(0.141) (0.128) (0.120) (0.142) (0.171)
Panel B. TOT
Enrolled in SU 0.194 0.205 0.162 0.423*** −0.106 0.023
(0.161) (0.144) (0.135) (0.157) (0.195)
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
Control mean  2.81  2.31 3.37
Observations  279  143 136
All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. ITT
Won lottery 0.166 0.176 0.140 0.370*** −0.090 0.027
(0.141) (0.128) (0.120) (0.142) (0.171)
Panel B. TOT
Enrolled in SU 0.194 0.205 0.162 0.423*** −0.106 0.023
(0.161) (0.144) (0.135) (0.157) (0.195)
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
Control mean  2.81  2.31 3.37
Observations  279  143 136

Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01.

### The Effect of StudentU on the Probability of Suspension in Early High School

Panel A of table 5 presents estimates of the effect of a lottery offer on the probability of suspension during ninth grade. The point estimate for the full sample in column 3 indicates that lottery winners were around 10.3 percentage points less likely to be suspended during ninth grade than their lottery loser counterparts. In relative terms, this translates into a 47 percent reduction (control mean is 23.7). The point estimates in columns 4 and 5 reveal significant evidence of treatment effect heterogeneity on the basis of baseline achievement. The point estimate in column 4 reports a statistically significant 17.1 percentage point reduction in the likelihood of suspension in ninth grade among lottery winners with low baseline achievement. In contrast, column 4 reports a 3.6 percentage point reduction that is statistically insignificant. The p-value from a statistical test of the null hypothesis of equality is 0.092 (marginally significant). The results from the interacted model once again provide strong evidence to suggest that the mean impact in the full sample of lottery winners is driven by effects on the subgroup of students with low baseline achievement.

Table 5.

The Effects of StudentU on the Probability of Suspension in Early High School

All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. ITT
Won lottery −0.099** −0.112** −0.103** −0.171*** −0.036 0.092
(0.047) (0.046) (0.046) (0.059) (0.063)
Panel B. TOT
Enrolled in SU −0.115** −0.130** −0.120** −0.196*** −0.042 0.087
(0.054) (0.052) (0.052) (0.065) (0.072)
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
Control mean  .237  .306 .160
Observations  279  143 136
All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. ITT
Won lottery −0.099** −0.112** −0.103** −0.171*** −0.036 0.092
(0.047) (0.046) (0.046) (0.059) (0.063)
Panel B. TOT
Enrolled in SU −0.115** −0.130** −0.120** −0.196*** −0.042 0.087
(0.054) (0.052) (0.052) (0.065) (0.072)
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
Control mean  .237  .306 .160
Observations  279  143 136

Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01.

### Discussion of Main Results

StudentU delivers all of the components of its programming to students simultaneously, which makes it impossible—in this setting—to identify how specific parts of the program (e.g., small group instruction, small-group mentoring, nutrition, child/family services) affected student outcomes. It also makes it impossible to identify whether there were complementarities between the program's components that made them more effective because they were delivered together rather than in isolation. These questions, although unanswered here, are important and deserve additional attention in future research. Despite these limitations, however, it is worth noting that the structure of StudentU programming shares several features identified in previous work as critical to the success of out-of-school time interventions. Among other factors, Schwartz et al. (2018) identify “high-quality enrichment experiences” and “a high level of engagement between adults and students” as factors crucial to the success of out of school time programs.

To gain insight into the mechanisms underlying increased credit accumulation, higher grade point averages, and reduced likelihood of suspension among lottery winners with low baseline achievement, I investigated several intervening variables that provide insight into the causal channels through which StudentU affects student outcomes. To distinguish between channels of on-time grade progress, course-taking and completion, achievement (cognitive skills), and behavior, I reestimated the same models for several intervening variables that serve as proxies for these channels.

Table 6 presents ITT estimates of the effect of a lottery offer on on-time grade progress following admission to StudentU (i.e., for sixth through ninth grades). The point estimate in column 3 indicates that lottery winners were around 2.1 percentage points (2 percent) more likely to make on-time grade progress than lottery losers, although the effect is only marginally statistically significant. The point estimate for lottery winners with low baseline achievement is 3.2 percentage points and statistically significant, while the point estimate for lottery winners with high baseline achievement is 0.1 percentage point and statistically indistinguishable from zero, although I cannot reject the null hypothesis of equality (p = 0.133). For completeness, I present TOT estimates for the same outcomes in online appendix table A8.

Table 6.

The Effects of StudentU on On-Time Grade Progress (Intent-to-Treat)

All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
On-time grade progression 0.027** 0.027** 0.021* 0.032** 0.010 0.133
(0.012) (0.012) (0.011) (0.016) (0.009)
Control mean 0.975 0.975 0.979 0.961 1.000
Observations 295 295 283 136 147
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
On-time grade progression 0.027** 0.027** 0.021* 0.032** 0.010 0.133
(0.012) (0.012) (0.011) (0.016) (0.009)
Control mean 0.975 0.975 0.979 0.961 1.000
Observations 295 295 283 136 147
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes

Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01.

Table 7 presents ITT estimates representing the effect of a lottery offer on credits attempted in ninth grade and a binary variable indicating whether the student failed any courses during ninth grade. The point estimates in panel A do not reveal and statistically significant effects on credits attempted, although the point estimates in panel B do reveal substantial reductions in the likelihood of any course failure. The point estimate in column 3 indicates that lottery winners in the full sample are around 8.6 percentage points less likely to fail any courses relative to their lottery loser counterparts. The point estimates in columns 4 and 5 make clear that this effect is entirely driven by lottery winners with low baseline achievement, who are 20.4 percentage points less likely to fail any courses during ninth grade. This contrasts with the small and statistically insignificant point estimate among lottery winners with high baseline achievement. The p-value from a statistical test of the difference between these coefficients is p = 0.008, which means that I can reject the null hypothesis of equality. For completeness, I present TOT estimates for the same outcomes in online appendix table A9. These results are qualitatively similar to the reduced-form effects.

Table 7.

The Effects of StudentU on Student Course-Taking and Completion (Intent-to-Treat)

All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. Credits attempted
0.132* 0.130* 0.119 0.167 0.072 0.464
(0.074) (0.074) (0.074) (0.102) (0.094)
Control mean 7.692 7.692 7.692 7.633 7.759
Observations 279 279 279 136 143
Panel B. Course failure (0/1)
−0.081 −0.096* −0.086* −0.204*** 0.031 0.008
(0.053) (0.052) (0.051) (0.069) (0.065)
Control mean 0.265 0.265 0.265 0.398 0.115
Observations 279 279 279 136 143
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. Credits attempted
0.132* 0.130* 0.119 0.167 0.072 0.464
(0.074) (0.074) (0.074) (0.102) (0.094)
Control mean 7.692 7.692 7.692 7.633 7.759
Observations 279 279 279 136 143
Panel B. Course failure (0/1)
−0.081 −0.096* −0.086* −0.204*** 0.031 0.008
(0.053) (0.052) (0.051) (0.069) (0.065)
Control mean 0.265 0.265 0.265 0.398 0.115
Observations 279 279 279 136 143
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes

Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01.

I do not find any statistically significant evidence to suggest that improvements in cognitive skills are a likely mechanism through which lottery winners improve their early high school outcomes, although I acknowledge that low statistical power is a concern for these outcomes.23 Table 8 presents ITT estimates of the effect of winning the lottery on math and reading test scores in eighth grade.24 For completeness, I present TOT estimates for the same outcomes in online appendix table A10.

Table 8.

The Effects of StudentU on Student Achievement (Eighth-Grade Test Scores) (Intent-to-Treat)

All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. Math achievement
0.029 −0.010 −0.043 0.071 −0.154 0.147
(0.124) (0.105) (0.089) (0.114) (0.121)
Control mean −0.199 −0.199 −0.199 −0.672 0.329
Observations 275 275 275 135 140
0.104 0.063 0.018 0.103 −0.065 0.187
(0.115) (0.096) (0.079) (0.105) (0.097)
Control mean −0.205 −0.205 −0.205 −0.673 0.319
Observations 275 275 275 135 140
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. Math achievement
0.029 −0.010 −0.043 0.071 −0.154 0.147
(0.124) (0.105) (0.089) (0.114) (0.121)
Control mean −0.199 −0.199 −0.199 −0.672 0.329
Observations 275 275 275 135 140
0.104 0.063 0.018 0.103 −0.065 0.187
(0.115) (0.096) (0.079) (0.105) (0.097)
Control mean −0.205 −0.205 −0.205 −0.673 0.319
Observations 275 275 275 135 140
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes

Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01.

Table 9 presents ITT estimates representing the effect of a lottery offer on the likelihood of a violent/weapons-related and other disciplinary infractions in ninth grade. Although I do not find any evidence of reductions in the likelihood of violent/weapons-related infractions, I find substantial evidence of reductions in the likelihood of other disciplinary infractions in panel B. The point estimates in column 4 indicate that lottery winners in the full sample are around 10.3 percentage points less likely to have a reported other infraction in ninth grade. The point estimates in columns 4 and 5 reveal that this main effect is likely driven by lottery winners with low baseline achievement. The point estimate for lottery winners with low baseline achievement is 16.1 percentage points and statistically significant, while the point estimate for lottery winners with high baseline achievement is 4.6 percentage points and statistically indistinguishable from zero, although I cannot reject the null hypothesis of equality (p = 0.146).

Table 9.

The Effects of StudentU on Student Behavior (Intent-to-Treat)

All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. Violent/weapons infractions
−0.038 −0.036 −0.032 −0.021 −0.042 0.692
(0.034) (0.034) (0.035) (0.047) (0.040)
Control mean 0.097 0.097 0.097 0.112 0.080
Observations 279 279 279 136 143
Panel B. Other infractions
−0.100** −0.113** −0.103** −0.161*** −0.046 0.146
(0.046) (0.045) (0.045) (0.059) (0.060)
Control mean 0.227 0.227 0.227 0.296 0.149
Observations 279 279 279 136 143
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes
All StudentsBaseline Achievement
LowHighp-value
(1)(2)(3)(4)(5)(6)
Panel A. Violent/weapons infractions
−0.038 −0.036 −0.032 −0.021 −0.042 0.692
(0.034) (0.034) (0.035) (0.047) (0.040)
Control mean 0.097 0.097 0.097 0.112 0.080
Observations 279 279 279 136 143
Panel B. Other infractions
−0.100** −0.113** −0.103** −0.161*** −0.046 0.146
(0.046) (0.045) (0.045) (0.059) (0.060)
Control mean 0.227 0.227 0.227 0.296 0.149
Observations 279 279 279 136 143
Demographic controls No Yes Yes Yes Yes
Baseline achievement No No Yes Yes Yes

Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01.

Taken together, the results suggest that StudentU improved course completion and reduced disciplinary infractions in school among lottery winners with low baseline achievement. I do not find any evidence of any improvements in student achievement. This is consistent with previous literature on the effects of tutoring outside of the regular school day (Heinrich, Meyer, and Whitten 2010; Zimmer, Hamilton, and Christina 2010; Heinrich et al. 2014). It contrasts with work that reports results from one-on-one or two-on-one, high-dosage tutoring that occurs within the context of the regular school day (see, e.g., Fryer 2014; Kraft 2015; Ander, Guryan, and Ludwig 2016; Chabrier, Cohodes, and Oreopoulos 2016).

Figure 2a plots the empirical relationship between the Early High School Outcomes Index and the likelihood of graduation from high school within five years. A one-unit change in the Early High School Outcomes Index is associated with a 29 percentage point increase in the likelihood that a student graduates from high school within five years (control mean = 0.59). Although there is some variation across cohorts, the relationship appears to be quite stable.

Figure 2b depicts the effect of winning the StudentU lottery on the likelihood that the Early High School Outcomes Index is equal to one. The pooled estimate indicates that StudentU lottery winners were around 12 percentage points more likely to have an Early High School Outcomes Index equal to one when compared with their lottery loser counterparts (control mean = 0.51). Based on these estimated relationships, I calculate an expected graduation rate of 77 percent for lottery winners and 73 percent for lottery losers, which implies that the effect of winning the lottery on the likelihood of high school graduation is a 4 percentage point increase. This predicted effect on high school graduation is considerably smaller than the actual effect realized in the Pathways program at the original site of implementation (15.3 percentage points), although it is similar in magnitude to the Pathways effect at the expansion site (5.8 percentage points) and to the (statistically insignificant) effect reported for the QOP (3.2 percentage points) (Oreopoulos, Brown, and Lavecchia 2017; Rodriguez-Planas 2012).

The predicted effect of StudentU on high school graduation is large and meaningful but should not be considered in isolation. When evaluating any educational intervention, it is important to consider not only the effects—and their social value—but also the level of investment required to obtain said impacts. To obtain estimates of per student spending for StudentU, I used three years’ worth of annual expenditure data broken down by program and one year's official tax records that reported expenditures broken down by category (e.g., salary/wages, insurance, supplies, food).25 Using these data, I estimated per student spending between $4,790 and$5,905 (2016 dollars) per year. Summing these costs over four years and assuming a real discount rate of 7 percent, I obtained a total cost estimate of $16,225 to$20,002 (2016 dollars) per student, which represents the total cost per student for four years of StudentU programming and services (this corresponds to the number of years between a student's matriculation into the program and the early high school outcomes observed in this paper). This range of cost estimates compares favorably to estimates from other comprehensive programs and related programs in the literature. Oreopoulos, Brown, and Lavecchia (2017) report a per student cost estimate of $13,400 for Pathways and Rodriguez-Planas (2012) reports a per student cost estimate of$25,000 for QOP (4-year program).

### Predicted Effect of StudentU (SU) on High School Graduation

Figure 2.
Predicted Effect of StudentU (SU) on High School Graduation

Notes: (a) Coefficient estimates for the effect of the Early High School Outcomes Index on the likelihood of high school graduation (within 5 years) for three successive cohorts of first-time ninth-grade students in Durham County, NC. (b) Coefficient estimates for the effect of winning the StudentU lottery on the probability that the Early High School Outcomes Index is equal to one.

Figure 2.
Predicted Effect of StudentU (SU) on High School Graduation

Notes: (a) Coefficient estimates for the effect of the Early High School Outcomes Index on the likelihood of high school graduation (within 5 years) for three successive cohorts of first-time ninth-grade students in Durham County, NC. (b) Coefficient estimates for the effect of winning the StudentU lottery on the probability that the Early High School Outcomes Index is equal to one.

Close modal

In this paper I investigate the effects of StudentU, a comprehensive, multiyear program designed to improve educational outcomes among disadvantaged students, on the early high school outcomes of participating students. To do this, I exploit variation induced by offers from oversubscribed admissions lotteries. Although I find some evidence of effects on early high school outcomes among the full sample of lottery winners, more detailed investigation reveals that these were driven exclusively by the subgroup of lottery winners who entered the program with low baseline achievement.

In an investigation of the mechanisms underlying these effects, I find evidence to suggest that on-time grade progress, course completion, and reduced reports of disciplinary infractions in school are the likely channels through which these effects flowed. Specifically, I find that lottery winners were around 3.2 percentage points more likely to make on-time grade progress, 20.4 percentage points less likely to fail a course during ninth grade, and around 10.3 percentage points less likely to be reported for a miscellaneous (i.e., not violent and not weapons-related) disciplinary infraction during ninth grade.

As a means to forecast longer-run impacts on students’ high school graduation outcomes, I conclude by presenting the results from a simple prediction exercise designed assess how these improvements in early high school outcomes likely translate into successes later in high school. Using an index of students’ early high school outcomes, I predict that lottery winners are around 4 percentage points more likely to graduate from high school than lottery losers (5 percent effect). These results are squarely in line with estimated effects presented in the previous literature. I also report estimates of per-student program costs, which also compare favorably to estimates reported for similar comprehensive programs. Taken together, the results from this paper suggest that comprehensive services delivered outside of the regular school day have the potential to improve the educational outcomes of disadvantaged students outside of the regular school day.

I thank the Editors, Randall Reback and Andrew McEachin, and three anonymous referees for their helpful comments and suggestions. I am grateful to Kara Bonneau and Radhika Deshmukh-McDiarmid for their assistance in matching the data for this project. Liz Ananat, Steve Bednar, Barbara Biasi, Sandy Black, Russ Bittmann, Charlie Clotfelter, Sue Dynarski, Matt Johnson, Ezra Karger, Sunny Ladd, Alicia Modestino, Nolan Pope, Seth Sanders, Elizabeth Setren, Crystal Zhan, and participants at the UNC-CH Educational Inequality Workshop, Elon University, the AEFP 44th Annual Conference, the SEA 88th Annual Meeting, the 2019 Triangle Applied Micro Conference, and the 2019 Carolina Region Empirical Economics Day provided helpful comments on this draft and on earlier versions of this paper. Graeme Peterson provided exceptional research assistance. All errors are my own.

1.

The program studied in this paper is a 21st Century Community Learning Center. The 21st Century Community Learning Center grant program is the largest competitive federal grant program devoted to funding out-of-school time programs and activities for children in the United States (Augustine and Thompson 2020).

2.

McCombs et al. (2019) provide a review of studies that focus on summer intervention and note that the majority of existing work in this area addresses the effectiveness of summer interventions that focus on reading and literacy.

3.

Three related but distinct strands of literature consider (1) the effects of programs and services provided to children and families through coordinated, community efforts but still accessed individually based on individual families’/children's needs (see, e.g., Dobbie and Fryer 2011); (2) the effects of “wraparound” or supportive services targeting students’ nonacademic needs that are delivered through targeted schools or districts and integrated with existing school/district services (see, e.g., Johnston et al. 2017; Winters 2017; Gandhi et al. 2018; Opper et al. 2019); and (3) the effects of voluntary, district-offered summer programming (Augustine et al. 2016).

4.

Students in the high baseline achievement group entered the program with an average math and reading score of 0.38 SD (median = 0.26 SD) relative to the statewide mean of zero (and unit standard deviation). In contrast, students in the low baseline achievement group entered the program with an average math and reading score of −0.90 SD (median = −0.80 SD).

5.

Despite the reference to DPS in the mission statement, StudentU is also open to students who attend charter schools and independent (private) schools in Durham County.

6.

See column 7 of table 1.

7.

Fryer (2017) defines “high-dosage” tutoring as tutoring delivered in groups containing 6 students (or fewer) for 3 or more days each week, or for a total of 50 or more hours in a 36-week period. Tutoring programs that meet for fewer hours or in larger groups are defined as “low-dosage.”

8.

Per institutional review board mandate, StudentU application and lottery records were sent directly from StudentU to staff at NCERDC, who handled the process of matching the identifiable student-level information contained in lottery records to the administrative student-level data. The matched data were then de-identified and transferred from NCERDC to the author for analysis.

9.

See Gassman-Pines and Bellows (2018) for a brief overview and discussion of match rates obtained in previous studies that have utilized the NCERDC data.

10.

See panel D of table A1.

11.

In this paper, I define students as economically disadvantaged if they are eligible for free or reduced-price lunch.

12.

I constructed a weighted version of student grade point average using letter grades and course designations (Honors, Advanced Placement, etc.) to ensure that GPA was measured consistently across schools. For more detail on this measure, see online Appendix B.

13.

Broadly speaking, the category of all other disciplinary infractions included drug-related, disruptive, sexual, and miscellaneous offenses. For a sample list of infractions, see online Appendix B.

14.

The administrative data do not provide information on parental education nor first-generation college status, so this restriction based on economic disadvantage is the closest approximation that can be implemented in the administrative data.

15.

I produce a version of this table for the subset of students from the matched sample (N = 279) with non-missing covariates and non-missing ninth-grade outcome data (the “regression sample”). See online appendix table A2.

16.

For a more detailed examination of differences between the traditional public and charter sectors in the state of North Carolina, see Ladd, Clotfelter, and Holbein (2016).

17.

I produce a version of this table for the subset of the matched sample (N = 279) with non-missing covariates and non-missing ninth grade outcome data (the “regression sample”). The differences between the lottery winners and lottery losers are similar, although the difference in the proportion of Hispanic students between groups is smaller in magnitude and only marginally statistically significant. See online appendix table A3. The difference in the proportion of economically disadvantaged students is slightly larger and remains statistically significant.

18.

My Data Use Agreement with NCERDC prohibits the reporting of statistical calculations on groups of students smaller than N = 10.

19.

I explore on-time grade progress (a potential effect of StudentU) more fully in section 6.

20.

This measure of enrollment in StudentU allows for the possibility that students did not complete the program or, in a few cases, started the program late.

21.

I construct an index of baseline achievement by taking the mean of the student's fifth-grade math and reading test scores. Subject-specific test scores are expressed in standard deviation units (normalized relative to the statewide mean and standard deviation by grade, subject, and year.

22.

I investigate the possibility of heterogeneous treatment effects by gender but do not find any evidence statistically different treatment effects for male versus female students. The results are reported in online appendix tables C1–C3.

23.

I estimate that the minimum detectable effect is 0.2 SD for test scores in both math and reading.

24.

There are no achievement test scores available in ninth grade.

25.

For a more detailed explanation of the expenditure data and my calculations, see online Appendix B.

Ander
,
Roseanna
,
Jonathan
Guryan
, and
Jens
Ludwig
.
2016
.
.
Policy Proposal No. 2016-02. The Hamilton Project: The Brookings Institution
.
Angrist
,
Joshua
,
Daniel
Lang
, and
Philip
Oreopoulos
.
2009
.
Incentives and services for college achievement: Evidence from a randomized trial
.
American Economic Journal: Applied Economics
1
(
1
):
136
163
.
Augustine
,
Catherine H.
,
Jennifer Sloan
McCombs
,
John F.
Pane
,
Heather L.
Schwartz
,
Jonathan
Schweig
,
Andrew
McEachin
, and
Kyle
Siler-Evans
.
2016
.
Learning from summer: Effects of voluntary summer learning programs on low-income urban youth
.
RAND Education
.
Santa Monica, CA
:
RAND Corporation
.
Augustine
,
Catherine H.
, and
Lindsey E.
Thompson
.
2020
.
Getting support for summer learning: How federal, state, city, and district policies affect summer learning programs
.
RAND Education and Labor
.
Santa Monica, CA
:
RAND Corporation
.
Bettinger
,
Eric P.
, and
Rachel B.
Baker
.
2014
.
The effects of student coaching: An evaluation of a randomized experiment in student advising
.
Educational Evaluation and Policy Analysis
36
(
1
):
3
19
.
Bhattacharya
,
Jayanta
,
Janet
Currie
, and
Steven J.
Haider
.
2006
.
Breakfast of champions? The school breakfast program and the nutrition of children and families
.
Journal of Human Resources
41
(
3
):
445
466
.
Carrell
,
Scott
, and
Bruce
Sacerdote
.
2017
.
Why do college-going interventions work?
American Economic Journal: Applied Economics
9
(
3
):
124
151
.
Castleman
,
Benjamin L.
,
Lindsay C.
Page
, and
Korynn
Schooley
.
2014
.
The forgotten summer: Does the offer of college counseling after high school mitigate summer melt among college-intending, low-income high school graduates?
Journal of Policy Analysis and Management
33
(
2
):
324
334
.
Chabrier
,
Julia
,
Sarah
Cohodes
, and
Philip
Oreopoulos
.
2016
.
What can we learn from charter school lotteries?
Journal of Economic Perspectives
30
(
3
):
57
84
.
Clotfelter
,
Charles T.
,
Steven W.
Hemelt
, and
Helen F.
.
2018
.
Multifaceted aid for low-income students and college outcomes: Evidence from North Carolina
.
Economic Inquiry
56
(
1
):
278
303
.
Cook
,
Philip J.
,
Kenneth A.
Dodge
,
George
Farkas
,
Roland G.
Fryer
,
Jonathan
Guryan
,
Jens
Ludwig
,
Susan
Mayer
,
Harold A.
Pollack
, and
Laurence
Steinberg
.
2014
.
The (surprising) efficacy of academic and behavioral intervention with disadvantaged youth: Results from a randomized experiment in Chicago
.
NBER Working Paper No. 19862
.
Corcoran
,
Sean P.
,
Brian
Elbel
, and
Amy Ellen
Schwartz
.
2016
.
The effect of breakfast in the classroom on obesity and academic performance: Evidence from New York City
.
Journal of Policy Analysis and Management
35
(
3
):
509
532
.
Dobbie
,
Will
, and
Roland G.
Fryer
.
2011
.
Are high-quality schools enough to increase achievement among the poor? Evidence from the Harlem children's zone
.
American Economic Journal: Applied Economics
3
(
3
):
158
187
.
Duncan
,
Greg J.
, and
Katherine
Magnuson
.
2011
.
The nature and impact of early achievement skills, attention skills, and behavior problems
. In
Whither opportunity: Rising inequality, schools, and children's life chances
,
edited by
Greg J.
Duncan
and
Richard J.
Murnane
, pp.
47
70
.
New York
:
Russell Sage Foundation
.
Farkas
,
George.
2011
.
Middle and high school skills, behaviors, attitudes, and curriculum enrollment, and their consequences
. In
Whither opportunity: Rising inequality, schools, and children's life chances
,
edited by
Greg J.
Duncan
and
Richard J.
Murnane
, pp.
71
90
.
New York
:
Russell Sage Foundation
.
Ford
,
Reuben
, and
Isaac
Kwakye
.
2016
.
Future to discover: Sixth year post-secondary impacts report
.
Ottawa, Ontario
:
Social Research and Demonstration Corporation
.
Fryer
,
Roland G.
2014
.
Injecting charter school best practices into traditional public schools: Evidence from field experiments
.
Quarterly Journal of Economics
129
(
3
):
1355
1407
.
Fryer
,
Roland G.
2017
.
The production of human capital in developed countries: Evidence from 196 randomized field experiments
. In
Handbook of economic field experiments
,
edited by
Abhijit Vinayak
Banerjee
and
Esther
Duflo
, pp.
295
322
.
Amsterdam, Netherlands
:
North-Holland
.
Gandhi
,
Allison Gruner
,
Rachel
Slama
,
So Jung
Park
,
Patrick
Russo
,
Kendra
Winner
,
Robin
Bzura
,
Wehmah
Jones
, and
Sandra
Williamson
.
2018
.
Focusing on the whole student: An evaluation of Massachusetts's wraparound zone initiative
.
Journal of Research on Educational Effectiveness
11
(
2
):
240
266
.
Gassman-Pines
,
Anna
, and
Laura
Bellows
.
2018
.
Food instability and academic achievement: A quasi-experiment using snap benefit timing
.
American Educational Research Journal
55
(
5
):
897
927
.
Grossman
,
Jean Baldwin
, and
Joseph P.
Tierney
.
1998
.
Does mentoring work? An impact study of the big brothers big sisters program
.
Evaluation Review
22
(
3
):
403
426
.
Heinrich
,
Carolyn J.
,
Patricia
Burch
,
Annalee
Good
,
Rudy
Acosta
,
Huiping
Cheng
,
Marcus
Dillender
,
Christi
Kirshbaum
,
Hiren
Nisar
, and
Mary
Stewart
.
2014
.
Improving the implementation and effectiveness of out-of-school-time tutoring
.
Journal of Policy Analysis and Management
33
(
2
):
471
494
.
Heinrich
,
Carolyn J.
,
Robert H.
Meyer
, and
Greg
Whitten
.
2010
.
Supplemental education services under no child left behind: Who signs up, and what do they gain?
Educational Evaluation and Policy Analysis
32
(
2
):
273
298
.
Heller
,
Sara B.
,
Harold A.
Pollack
,
Roseanna
Ander
, and
Jens
Ludwig
.
2013
.
Preventing youth violence and dropout: A randomized field experiment
.
NBER Working Paper No. 19014
.
Heller
,
Sara B.
,
Anuj K.
Shah
,
Jonathan
Guryan
,
Jens
Ludwig
,
Sendhil
Mullainathan
, and
Harold A.
Pollack
.
2017
.
Thinking, fast and slow? Some field experiments to reduce crime and dropout in Chicago
.
Quarterly Journal of Economics
132
(
1
):
1
54
.
Imberman
,
Scott A.
, and
Kugler
.
2014
.
The effect of providing breakfast in class on student performance
.
Journal of Policy Analysis and Management
33
(
3
):
669
699
.
Johnston
,
William R.
,
Celia J.
Gomez
,
Lisa
,
Lea
Xenakis
, and
Brent
Anderson
.
2017
.
Developing community schools at scale: Implementation of the New York City community schools initiative
.
RAND Corporation
.
Kim
,
James S.
, and
David M.
Quinn
.
2013
.
The effects of summer reading on low-income children's literacy achievement from kindergarten to grade 8: A meta-analysis of classroom and home interventions
.
Review of Educational Research
83
(
3
):
386
431
.
Kraft
,
Matthew A.
2015
.
How to make additional time matter: Integrating individualized tutorials into an extended day
.
Education Finance and Policy
10
(
1
):
81
116
.
,
Helen F.
,
Charles T.
Clotfelter
, and
John B.
Holbein
.
2016
.
The growing segmentation of the charter school sector in North Carolina
.
Education Finance and Policy
12
(
4
):
536
563
.
Leuven
,
Edwin
,
Hessel
Oosterbeek
, and
Bas van der
Klaauw
.
2010
.
The effect of financial rewards on students’ achievement: Evidence from a randomized experiment
.
Journal of the European Economic Association
8
(
6
):
1243
1265
.
McCallion
,
Gail.
2001
.
21st century community learning centers: An overview of the program and analysis of reauthorization issues
.
Order Code RL30306
.
Washington, DC
:
Congressional Research Service
.
McCombs
,
Jennifer Sloan
,
Catherine H.
Augustine
,
Fatih
Unlu
,
Kathleen M.
Ziol-Guest
,
Scott
Naftel
,
Celia J.
Gomez
,
Terry
Marsh
,
Goke
Akinniranye
, and
Ivy
Todd
.
2019
.
Investing in successful summer programs: A review of evidence under the Every Student Succeeds Act. RAND Education and Labor
.
Santa Monica, CA
:
RAND Corporation
.
Opper
,
Issac M.
,
William R.
Johnston
,
John
Engberg
, and
Lea
Xenakis
.
2019
.
Assessing the short-term impact of the New York City renewal schools program
.
Working Paper WR-1303-NYCDOE
.
RAND Education and Labor
.
Oreopoulos
,
Philip
,
Robert S.
Brown
, and
Lavecchia
.
2017
.
Pathways to education: An integrated approach to helping at-risk high school students
.
Journal of Political Economy
125
(
4
):
947
984
.
Reardon
,
Sean F.
2011
.
The widening academic achievement gap between the rich and the poor: New evidence and possible explanations
.
In
Whither opportunity: rising inequality, schools, and children's life chances
,
edited by
Greg J.
Duncan
and
Richard J.
Murnane
, pp.
91
116
.
New York: Russell Sage Foundation
.
Rodriguez-Planas
,
Nuria.
2012
.
Longer-term impacts of mentoring, educational services, and learning incentives: Evidence from a randomized trial in the United States
.
American Economic Journal: Applied Economics
4
(
4
):
121
139
.
Schueler
,
Beth E.
2020
.
Making the most of school vacation: A field experiment of small group math instruction
.
Education Finance and Policy
15
(
2
):
310
331
.
Schwartz
,
Heather L.
,
Jennifer Sloan
McCombs
,
Catherine H.
Augustine
, and
Jennifer T.
Leschitz
.
2018
.
Getting to work on summer learning: Recommended practices for success
, 2nd ed.
RAND Education and Labor
.
Santa Monica, CA
:
RAND Corporation
.
Stapleton
,
Katina R.
1998
.
21st century community learning centers: A summary of the program
.
Report Number CRS-1998-EPW-0113
.
Washington, DC
:
Congressional Research Service
.
Winters
,
Marcus A.
2017
.
Costly progress: DeBlasio's renewal school program
.
Report
.
New York
:
Manhattan Institute
.
Zimmer
,
Ron
,
Laura
Hamilton
, and
Rachel
Christina
.
2010
.
After-school tutoring in the context of no child left behind: Effectiveness of two programs in the Pittsburgh public schools
.
Economics of Education Review
29
(
1
):
18
28
.