Abstract

We examine the Los Angeles Unified School District's Public School Choice Initiative (PSCI), which sought to turnaround the district's lowest-performing schools. We ask whether school turnaround impacted student outcomes, and what explains variations in outcomes across reform cohorts. We use a Comparative Interrupted Time Series approach using administrative student-level data, following students in the first (1.0), second (2.0), and third (3.0) cohorts of PSCI schools. We find that students in 1.0 turnaround schools saw no significant improvements in outcomes, whereas students enrolled in 2.0 schools saw significant gains in English Language Arts in both years of the reform. Students in 3.0 schools experienced significant decreases in achievement. Qualitative and survey data suggest that increased support and assistance and the use of reconstitution and restart as the sole turnaround methods contributed to gains in 2.0, whereas policy changes in 3.0 caused difficulties and confusion in implementation, leading to poor student performance.

1.  Introduction

Leaders at all levels of government are searching for systemic reforms to help them improve student achievement. One policy solution, school turnaround, has been growing in popularity and has taken center stage in recent national policy debates. Former U.S. Secretary of Education Arne Duncan called for states and districts to “turn around” their lowest performing schools and “transform” them into higher performing organizations (USDOE 2010a). Accordingly, the Obama administration has highlighted school turnaround as a priority strategy for low-performing schools, including it in its School Improvement Grant (SIG) initiative, Race to the Top, and waivers to the Elementary and Secondary Education Act.

Turnaround reforms expect schools and districts to enact changes that produce significant achievement gains in a very short period of time, and to sustain that improvement over the long run (Herman et al. 2008; Villavicencio and Grayman 2012). Designed to improve conditions in consistently underperforming schools by changing staffing, governance, support, and/or instruction, school turnaround encompasses a range of improvement strategies. The particular reforms included under the “turnaround” umbrella vary widely, ranging from the relatively incremental change of adding a new professional development provider, to the more intensive implementation of a new curriculum. The most dramatic changes include “reconstitution,” which requires that districts hire a new school principal and replace at least 50 percent of the teaching staff, or “restart,” which requires that schools close and reopen under a charter or other external operator (Herman et al. 2008). Amid all of this policy attention paid to school turnarounds, research on their implementation and effects remains limited. The majority of research focuses solely on reconstitution efforts and gives far less attention to other forms of turnaround. As a result, little is understood about the impacts of school turnaround reforms on student (or other) outcomes, or about the differential impacts that may emerge from different methods of turnaround.

This paper is part of a four-year study that examined the implementation and impact of a turnaround reform in the Los Angeles Unified School District (LAUSD) called the Public School Choice Initiative (PSCI). PSCI was a complex reform that aimed to improve student achievement by turning around the district's lowest performing schools while simultaneously expanding LAUSD's portfolio of schools by allowing both internal and external stakeholders to compete to operate PSCI schools. The district's theory of change behind PSCI incorporated the turnaround concept into its larger portfolio model and held that, with intensive supports and appropriate autonomies, a range of school providers would be able to increase student achievement in low-performing “focus” schools and a set of newly opened “relief” schools (opened to relieve overcrowding).1

In this paper, we focus on the turnaround reform element of PSCI. Designed for gradual scale-up, PSCI involved annual rounds (cohorts) of schools with the intention that all low-performing schools would be transformed into high performers. We use LAUSD's PSCI as a case with which to ask two questions that remain largely unanswered by extant research: (1) Did school turnaround reform impact student outcomes in low-performing schools? and (2) What explains variations in outcomes for different cohorts of turnaround schools? To answer these questions, we use a Comparative Interrupted Time Series (CITS) estimation approach using LAUSD's administrative student-level dataset that follows students from the 2003–04 to the 2012–13 school years (seven years pre-implementation and three years of reform implementation). For three years of implementation we follow the first cohort of PSCI schools (cohort 1.0), which utilized moderate forms of turnaround including the implementation of new curriculum and school plans. The second cohort of schools (cohort 2.0), for which we have two years of outcome data, implemented more drastic turnaround reforms, solely consisting of reconstitution and restart models that called for new leadership and staff in schools alongside programmatic changes. Last, we track the third cohort of impacted schools (cohort 3.0), which utilized “softer” turnaround models and operated under substantially changed parameters (described later), for a single year of implementation (after which test scores are no longer available).

Based on these analyses, we find that students in the first cohort of PSCI focus schools, on average, saw no significant improvements in either the first year of the reform or in the following two years. Students enrolled in cohort 2.0 focus schools, however, saw significant improvements in California Standards Test (CST) scores in the first year of the reform, with continued growth in the second year. In contrast, students in cohort 3.0 schools experienced significant decreases in test scores in the first year of reform implementation. All of these findings are robust to multiple sensitivity checks. Qualitative data from interviews with district and partner organization leaders, case studies of PSCI schools, and data from surveys of PSCI school principals suggest two inter-related explanations for cohort 2.0's improved performance: the use of reconstitution and restart as the sole methods of school turnaround and increased resources provided for professional development and technical assistance. In contrast, cohort 3.0 schools may have performed poorly in the first year of the reform because of substantial changes in the PSCI policy that created confusion and difficulties for the school teams implementing these turnaround efforts.

Because school turnaround assumes rapid improvements in student achievement, a study of the early years of intervention effects is particularly relevant. In addition, given the relatively thin body of knowledge on the efficacy of turnaround reforms, results from an individual district's turnaround reform—particularly one that incorporated multiple popular models of turnaround—will help to build evidence that can be considered in larger policy discussions about their effectiveness. In the remainder of this paper we first place the LAUSD experience in a broader policy and research context by summarizing the literature on school turnarounds. We then provide further detail about LAUSD's implementation of school turnaround through PSCI. Next, we outline our student-level administrative dataset, the CITS approach taken to estimate the effects of the reform on student achievement, and the survey and qualitative data used to flesh out our results. We then present results and discuss the implications for policy, practice, and research.

2.  A Brief Review of the Literature on School Turnaround Reforms

School turnaround takes many forms, incorporating a variety of strategies that range from dramatic (e.g., school closure and reopening under a new operator or the hiring of new leadership and faculty) to relatively modest (e.g., changes in professional development or curriculum). Under the SIG program, turnaround is given a narrower definition as one of four possible interventions for improving low-performing schools in which schools must replace the principal, fire all of the school staff and rehire no more than half of them, and grant the new principal sufficient flexibility to implement a comprehensive approach to improve student outcomes. The remaining SIG models include “restart,” “transformation,” and “closure” (USDOE 2010b).2 Regardless of label, all school turnarounds focus “on the most consistently underperforming schools and involve dramatic, transformative change” quickly—within two to three years (Calkins et al. 2007, p. 10; also see Herman et al. 2008; Villavicencio and Grayman 2012).

To date, little evidence exists regarding the efficacy of school turnaround efforts. The U.S. Department of Education's Institute of Education Sciences What Works practice guide on school turnaround (Herman et al. 2008) found no empirical studies of requisite rigor demonstrating intervention effects. The one notable exception is a recent study that uses a regression discontinuity approach to isolate the impact of SIG-funded reforms on student achievement (Dee 2012). That study finds evidence that SIG-funded school reforms led to significant improvements in the performance of California's lowest-performing schools in their first year of SIG implementation. Importantly, Dee finds the SIG turnaround (often labeled “reconstitution”) model drives the positive results and other models are less effective in improving school performance.

Much of the remaining research on turnarounds focuses solely on the reconstitution model and utilizes descriptive and qualitative methods to assess the implementation and intermediate effects of the reform. Reconstitution is conceptually similar to the definition of turnarounds under the SIG reform, allowing school staff to reapply for their jobs, with only a low proportion (under 50 percent) being offered re-employment. The theory behind reconstitution holds that, by infusing new faculty and leadership into the low-performing school, the revitalized school will have the will and capacity to implement necessary reforms and improve student achievement (Doherty and Abernathy 1998; Malen et al. 2002; Rice and Malen 2003; Rice and Croninger 2005). Although research about the implementation and intermediate outcomes of school reconstitution finds that new staffs in reconstituted schools work longer hours and expend greater effort (Hansen, Kraetzer, and Mukherjee 1998), reconstitution studies on the whole identify serious challenges faced by districts in accessing an adequate supply of such capable and committed staff and providing additional valued resources and support structures to bolster the capacity of these schools (Fraga, Erlichson, and Lee 1998; Wong et al. 1999; Malen et al. 2002; Hess 2003; Rice and Malen 2003, 2010; Malen and Rice 2004; Rice and Croninger 2005; Hamilton, Heilig, and Pazey 2013). One study of six reconstituted schools in one district finds negative near-term outcomes of school restructuring, including high levels of teacher turnover, with first-year and noncertificated teachers often replacing experienced teachers, and only marginal adjustments in classroom practice (Malen et al. 2002; Rice and Malen 2003; Malen and Rice 2004, 2009). Similar findings are reported in Chicago (Hess 2003) and Texas (Hamilton, Heilig, and Pazey 2013). Recent studies of SIG-funded school turnarounds identify particular challenges in maintaining an adequate supply of effective principals and teachers and finding individuals at the state and local levels with the expertise and commitment to carry out major school improvement efforts (Scott et al. 2012; USGAO 2012; Le Floch et al. 2014).

Although much of the research about reconstitution suggests difficulties in implementation, some studies of reconstitution and other similar school reform efforts can provide lessons for districts as they embark on turnaround reforms. For example, implementation studies on reconstitution efforts posit that incentives to recruit and retain high-quality teachers, provision of time and autonomy for staff to engage in school redesign efforts, and access to financial and capacity building resources are crucial to successful implementation (Rice and Malen 2010). Similarly, research on whole-school reform (which resembles transformation efforts) identifies several factors that facilitate implementation, including: strong instructional leadership from the principal and support from districts and model developers (Berends, Bodilly, and Kirby 2002; Datnow et al. 2003; Aladjem and Borman 2006), capacity of teachers to implement the model (time and expertise) (Berends, Bodilly, and Kirby 2002; Datnow et al. 2003), strong professional development for teachers (Muncey and McQuillan 1996; Nunnery 1998), and broad support for the model (Nunnery 1998; Borman et al. 2000; Datnow and Stringfield 2000; Berends, Bodilly, and Kirby 2002). As will be described herein, the design of LAUSD's turnaround effort appears to incorporate and address some of these factors by situating school turnaround within a competitive school design process, the Public School Choice Initiative.

3.  School Turnarounds as Implemented in LAUSD's Public School Choice Initiative

Fueled in part by the district's persistent and widespread student achievement problems, the LAUSD Board of Education adopted the Public School Choice resolution in August 2009. The initiative allowed internal and external teams to submit plans to operate the district's lowest-performing (focus) schools, as well as a set of newly constructed (relief) schools (not discussed in this paper) that had been built to relieve overcrowding. Internal team applicants included groups of teachers (school-based teams) and combinations of teachers, parents, and/or administrators from the local school community (local district teams). External teams included nonprofit organizations, charter management organizations, or some combination of internal/external actors. Although much of the rhetoric surrounding the adoption of this initiative alluded to school choice and increasing the number of quality educational options for parents and students in LAUSD, the initiative as adopted was not intended to be a typical “choice” program in which parents choose the school their child will attend, but rather a process in which the community had the opportunity to participate in developing school plans. The ultimate choice in PSCI was made by the LAUSD Board and Superintendent and not directly by parents.

Similar to other turnaround reforms (see NCEE 2014), PSCI rested on a theory of action that held that school operators would write innovative and evidence-based school plans targeted at the particular communities of students they served. Armed with high-quality plans and intensive support from the district and its partners, these schools could choose to take on increased levels of autonomy over staffing, budget, governance, and curriculum. The inclusion of parent and community feedback and district oversight and accountability mechanisms would guide school operators toward success while holding them accountable for student outcomes, eventually leading to improvements in student performance.

LAUSD used a set of performance metrics to select annual cohorts of low-performing focus schools. Although the specific metrics included in the district's decision-making process changed slightly in each cohort, they generally included the following indicators: (1) school program improvement status (needs improvement under the No Child Left Behind Act [NCLB]); (2) Academic Performance Index (API, an aggregate score consisting of a weighted combination of various achievement measures); (3) percentage of students scoring proficient or advanced on the CSTs; (4) meeting Adequate Yearly Progress targets; and (5) dropout rates.3 The district intended to continue selecting cohorts of low-performing schools using variations of this selection mechanism until all struggling schools in the district became high performers.

LAUSD asked applicant teams to select a governance model from among six existing district models that varied in the levels of autonomy the school had from district and/or union policies. These models were: independent charter schools, pilot schools, Expanded Site Based Management Model schools, network partner schools, affiliated charter schools, and traditional school models. In the first two cohorts of the reform, selected operators with plans calling for charter school restarts were allowed to operate charter schools that would not be subject to LAUSD's union contracts. The third and fourth cohorts of schools, however, were required to hire district employees to staff their schools regardless of school governance model. In addition, the district's request for proposals required teams to provide detailed school plans. These proposals ranged from 100 to 300 pages and described how the school would operate, including proposed curriculum and professional development. Submitted applications underwent a multi-stage review and final recommendations were issued to the LAUSD Board (in cohorts 1.0 and 2.0) and to the Superintendent (in cohorts 3.0 and 4.0), who decided on the final set of winning applicants.4

PSCI differed from other district and state turnaround efforts in two primary ways, both of which allowed for variation in turnaround implementation across schools. First, PSCI introduced an element of competition into the selection of school operators. By encouraging competition between multiple applicants for each school site, allowing internal and external operators to compete, and incorporating a multi-tiered plan review structure into the selection process (see Marsh, Strunk, and Bush 2013 and Strunk et al. 2016 for more detail), LAUSD allowed for the possibility that every school involved in PSCI could be a restart in which an external operator would reopen the school with a new operational plan, new leadership, and new staff. In addition, LAUSD allowed for the possibility that, if school plans were not adequate or if district leadership was concerned that current school leadership and staff could not execute on a new school plan, the district could reconstitute the school, bringing in new leadership and/or new teaching staff. Again, this allowed for the possibility that targeted focus schools could be turned around via less or more dramatic methods, depending on the quality of the plan(s) submitted and the particular team(s) involved.

To date, there have been four cohorts of PSCI, which include 45 low-performing turnaround schools. We do not discuss cohort 4.0 because California ceased administering CSTs in the first year of cohort 4.0 implementation, making it impossible to track student achievement. After the fourth cohort of the reform, the district placed PSCI on indefinite hold. Table 1 outlines the numbers of focus schools included in each of the first three cohorts and highlights the variation in turnaround model type within each cohort. The first round of PSCI involved fourteen focus schools, none of which was reconstituted or restarted. Cohort 2.0 included five schools, all of which were reconstituted or restarted.5 Cohort 3.0 included nine focus schools, none of which was reconstituted or restarted. Because of differences in turnaround models and related policies and supports between the three cohorts, we view each cohort of the intervention as a separate variant of a turnaround intervention and examine separately the impacts of turnaround on student outcomes in each cohort.

Table 1. 
Sample of Focus Schools in Cohorts 1.0, 2.0, and 3.0
1.02.03.0Combined 1.0–3.0
PSCI focus schools 14 28 
Reconstitution 
Restart 
Transformation 14 23 
1.02.03.0Combined 1.0–3.0
PSCI focus schools 14 28 
Reconstitution 
Restart 
Transformation 14 23 

Notes: PSCI 4.0 schools are not included in our analyses. All treated schools in cohorts 1.0–3.0 remain in our analysis sample except for one focus school in cohort 3.0 that does not have a unique County-District-School (CDS) code and cannot be tracked in our data. Reconstituted schools were required to lay off staff and rehire fewer than 50 percent of their staff. Although 12 focus schools were identified for treatment in cohort 1.0, two of these schools, Lincoln Senior High and San Fernando Middle School, separated in 2011–12 into a main campus and a smaller pilot academy with separate CDS codes (Leadership and Entertainment Media Arts and San Fernando Institute of Applied Media, respectively). As such, our final sample of focus schools in Cohort 1.0 increases from 12 schools in 2010–11 to 14 schools in 2011–12 and 2012–13. Similarly, one focus school in Cohort 2.0 was restarted as two, smaller charter schools, increasing our treated sample from four focus schools as identified in 2010–11 to five focus schools in 2011–12. Of the nine focus schools in cohort 3.0, one of these schools had not been assigned a unique CDS code and as such, we are not able to track its students in our administrative data. Restarted schools were reopened under a new school operator, retaining the same students but not necessarily the same staff. Transformed schools were allowed to keep staff but operated under a new school plan with dramatic changes to school operations.

The literature discussed in the previous section suggests, in many ways, that PSCI may have been a particularly promising version of the turnaround reform. Specifically, PSCI was context-specific, included various methods of turnaround (from reconstitution to transformation), incorporated systemic district-level changes, and invested in capacity building. The extant literature suggests these attributes are important in successful turnarounds (Knudson, Shamburgh, and O’Day 2011). In addition, PSCI required applicant teams to highlight how they would foster many of the elements found to contribute to successful turnaround efforts, such as data-driven decision making, teacher collaboration, staff development, curricular alignment, and parental involvement (Duke 2006). That said, many of the challenges that have plagued other turnaround efforts, such as the supply of qualified educators who are committed to the turnaround plans and the provision of sufficient resources, may not have been adequately addressed by the PSCI reform. We will return to these themes in section 5.

4.  Data and Methods

In order to measure the impact of PSCI on student outcomes, we use LAUSD administrative student- and school-level data. We include in our sample all students enrolled in cohort 1.0 focus schools during the first three years of PSCI implementation (2010–11 through 2012–13), in cohort 2.0 schools during the first two years of implementation (2011–12 and 2012–13), and for a single year in cohort 3.0 focus schools (2012–13). We cannot track the impact of the reform on student achievement after the first year of implementation for the third cohort of PSCI schools and students because California ceased offering CSTs. Our main sample also includes students enrolled in a set of “near-selected” comparison schools and all low-performing schools in LAUSD (both described later). In addition, we draw from survey and qualitative data to provide context and suggest possible explanations for the quantitative findings.

This section first reviews the LAUSD administrative data we use to assess the effect on student outcomes of being enrolled in a turnaround school. Next, we define our treatment and comparison groups. We then outline our CITS estimation strategy. Last, we provide a brief overview of the survey and qualitative data used in this study.

Administrative Data

We begin with a panel dataset from LAUSD's administrative data for the 2003–04 to 2012–13 school years. This dataset includes approximately 5.72 million student-year, or 1.15 million individual student, observations from all second- through eleventh-grade students enrolled in LAUSD for one or more years from 2003–04 to 2012–13, excluding those in alternative education schools and some charter schools. Our sample also excludes 1.1 percent of students who do not demonstrate normal grade progression.6 Because we examine the impact of PSCI on student outcomes in the first three years of PSCI implementation, we limit our sample to only those students who are enrolled in PSCI or comparison schools in the 2010–11, 2011–12, and/or 2012–13 school years. This leaves us with 0.94 million student-year, or 413,044 individual student, observations from 2003–04 through 2012–13. These students are enrolled in LAUSD PSCI focus, near-selected, or an expanded set of low-performing comparison schools (those classified as in Program Improvement Year 3 [PI3+] under NCLB) during any of the three outcome years.

In our analyses, we incorporate student-level data including students’ CST scores in math and English Language Arts (ELA), information on students’ race and ethnicity, qualification for the federal free- and reduced-price lunch program (an indicator of poverty) and special education services, students’ English language learner (ELL) status, and measures of student mobility. Our outcomes of interest are math and ELA CST scores, standardized by subject-grade to have a mean of 0 and a standard deviation of 1. We also include school-level data from public data sets, including school level (elementary, middle, or high school), school enrollment, and the proportion of students in a school who are minority, socioeconomically disadvantaged, and ELLs.7

Treatment and Comparison Groups

As a starting point to our analysis, we can examine unadjusted trends in ELA and math achievement of each cohort of students enrolled in focus schools in 2010–11, 2011–12, or 2012–13. The solid line (with circles) in figure 1 shows pre–post trends in math and ELA achievement scores for students in each of the three cohorts of focus schools. We first notice that students in focus schools perform below average (0) in ELA and math throughout the timeframe of our analysis (2003–04 to 2012–13). Moreover, in all but one instance (i.e., ELA achievement in cohort 2.0 schools), these students demonstrate a decline in achievement in their first year of the PSCI intervention. In both cohorts 1.0 and 2.0, there is a slight improvement in year two of the reform. However, students in Cohort 1.0 schools exhibit a slight decrease in achievement in the third year of the reform.

Figure 1.

Trend in Unadjusted Standardized ELA and Math Achievement Scores of Students Enrolled in Cohorts 1.0, 2.0, and 3.0 Focus Schools versus Near-selected Schools (without later treated cohorts) by Timing of PSCI Intervention.

Figure 1.

Trend in Unadjusted Standardized ELA and Math Achievement Scores of Students Enrolled in Cohorts 1.0, 2.0, and 3.0 Focus Schools versus Near-selected Schools (without later treated cohorts) by Timing of PSCI Intervention.

Although this comparison of before versus after trends suggests that PSCI turnaround efforts have somewhat impacted student achievement, this analysis does not control for other social, economic, or educational factors that could have influenced students during this time period. For example, LAUSD, like other districts, was subject to increasing accountability pressures from both state and federal governments, which may have culminated in the time period right around the implementation of PSCI with the Race to the Top legislation, discussions of reauthorization of the Elementary and Secondary Education Act, and other federal legislation that stressed turning around low-performing schools and taking severe actions to hold low-performing schools accountable. In Los Angeles specifically, the early years of PSCI also coincided with the pending exit of Superintendent Ramon Cortines, uncertainty as to who would replace him, and the eventual appointment of John Deasy as new Superintendent, along with elections of a new teachers’ union president and School Board members. This combined governance shift may have caused schools to begin acting differently just as PSCI was implemented. Moreover, ongoing negotiations between the teachers’ union and district generated additional uncertainty that likely affected the behavior of all schools. Therefore, we cannot credibly identify causal inferences about PSCI from before versus after trends in student achievement.

These non-PSCI-specific trends should have impacted all schools in LAUSD, however, or at least all low-performing schools. As such, we can compare the deviation from prior achievement trends among students at treated PSCI schools with an analogous deviation for students at a comparison group of schools who should have been less affected by PSCI (since they were at schools not included in the intervention itself).8 Theoretically, the deviation from trend in the comparison group should reflect other hard-to-observe factors that may have influenced student achievement or other outcomes even in the absence of PSCI, including all of the previously mentioned factors that have impacted all, or at least all low-performing, schools in LAUSD (Bloom 1999; Bloom et al. 2001; Shadish, Cook, and Campbell 2002; Dee and Jacob 2011; Somers et al. 2013).

Given the comparative nature of our CITS research design, causal inference relies on the identification of plausible comparison groups. In our analysis, we compare students enrolled in focus schools in 2010–11 through 2012–13 to students enrolled in near-selected comparison schools during the same time period. As mentioned earlier, LAUSD used a set of indicators to select focus schools for inclusion in PSCI. To be included in PSCI, a school must have met all of the indicators. The school was excluded from PSCI if it was missing data for any criteria. We follow the way LAUSD selected PSCI focus schools and count as a near-selected school any school that missed one indicator, regardless of whether or not that data point was missing. As such, this comparison set includes schools that had nonmissing data for all indicators and met all but one of those indicators, and schools that had missing data for one indicator and met all of the indicators for which they had data. Schools with missing data for more than one indicator are not counted as near-selected in this group, even if they met all the indicators for which they had supporting data.9 In addition, we do not include near-selected schools that are selected for PSCI turnaround in the following year.10 Table 2 shows the number of near-selected schools in the full sample of comparison schools for each cohort, and then breaks down the number of those included in our analyses (for which we are not missing data). Baseline equivalence tests comparing prior achievement in ELA and math of students enrolled in focus versus near-selected schools indicate that differences between the two groups are no greater than 0.25 standard deviations of the combined variance in achievement, thus suggesting baseline equivalence between focus and near-selected schools in the outcomes of interest (available from the authors upon request).

Table 2. 
Sample of Near-selected Schools Included in our Comparison Groups for Cohorts 1.0, 2.0, and 3.0
Near-selected Comparison Schools
CohortFull SampleFull Sample w/o Alternative SchoolsDataset Sample w/o Some Charter SchoolsaDataset Sample Without Schools Treated in Following Cohort
1.0 102 81 77 71 
2.0 45 29 29 23 
3.0 43 43 42 38 
Near-selected Comparison Schools
CohortFull SampleFull Sample w/o Alternative SchoolsDataset Sample w/o Some Charter SchoolsaDataset Sample Without Schools Treated in Following Cohort
1.0 102 81 77 71 
2.0 45 29 29 23 
3.0 43 43 42 38 

aWe are missing data on four independent charter schools in cohort 1.0 and one independent charter school in cohort 3.0.

A possible threat to the causal inference of the CITS research design concerns endogenous student mobility. The introduction of PSCI could induce nonrandom mobility patterns that cause the student populations of treated and comparison schools to change at the onset of the intervention. This compositional shift in demographics and prior achievement could lead to changes in student outcomes that are confounded with our estimated effects of PSCI. To determine if there is endogenous mobility in our sample, we compare student transfer rates (within and outside of LAUSD) in focus schools and near-selected schools during each cohort's identification year (2009–10 for cohort 1.0, 2010–11 for cohort 2.0, and 2011–12 for cohort 3.0).11 We find students at near-selected schools demonstrate a slightly higher transfer rate than students at focus schools in cohorts 1.0 and 2.0 (2 to 3 percent), and comparable transfer rates in cohort 3.0. Students who transfer out of both kinds of school (focus and near-selected schools) in all cohorts demonstrate significantly lower levels of achievement than students who remain in their schools. Students who transfer out of near-selected schools demonstrate significantly higher levels of achievement in ELA (cohort 2.0) and math (cohorts 1.0 and 2.0) than students who transfer out of focus schools, however. These findings suggest that the student populations at treated and comparison schools may have changed at the onset of the PSCI intervention for cohorts 1.0 and 2.0, and might possibly contribute some upward bias to our results. To ensure this was not the case, we run our CITS models (further described subsequently) using individual student mobility as the outcome of interest. If mobility is truly endogenous, then the turnaround reform would have increased (or decreased) student mobility in impacted schools. We find no evidence of significant or substantial effects of PSCI participation on student mobility, using either a linear probability model or a logit (available from the authors upon request). Nonetheless, we control for student mobility in our analysis to account for the fact that PSCI could have induced changes in student mobility.12

Estimation Strategy

Based on the intuition of the CITS research design, we estimate the following regression model separately for each cohort of the reform:
formula
1
where represents student i's CST scores in ELA and math, normed by grade/subject/year in school s in year t. is a trend variable, which we center to begin at 1 in the 2003–04 academic year, the first year for which we have student-level administrative data. is a dichotomous indicator variable equal to 1 in all years since the implementation of PSCI for that cohort. For instance, for cohort 1.0, switches from 0 to 1 in the 2010–11 academic year. is defined as the number of years since implementation. For cohort 1.0, this takes the form of . As such, this variable takes on a value of 1 in the second year of implementation for each of cohorts 1.0 and 2.0, and a value of 2 in the third year of implementation for cohort 1.0. is a vector of school-level covariates including natural log of enrollment, school-level, and percent of students who are ELLs and socioeconomically disadvantaged. We do not include the proportion of students who are minority in the set of school variables because it is too highly correlated with the proportion of students who are eligible for free- or reduced-price lunch. is a vector of time-varying student-level covariates including eligibility for free and reduced-price lunch, special education status, an indicator of nonstructural student mobility in each year, and lagged achievement scores. is a student fixed-effect and is a random error with mean zero that is clustered at the school-level. is a time-invariant dichotomous variable that measures the treatment imposed by the PSCI turnaround reform. As such, this variable equals 1 if a student is enrolled in a focus school in the treatment years, and 0 if the student is enrolled in a comparison school in these years. Overall, this specification allows for a PSCI turnaround effect that can be reflected in both a level shift in the outcome variable () as a well as a shift in the achievement trend (). For cohort 1.0, the overall effect of PSCI turnaround reforms at the end of the third year of implementation is . For cohort 2.0, the overall effect at the end of the second year of implementation is . The cohort 3.0 effect is simply the level shift, . We do not estimate for cohort 3.0 as we only measure student outcomes after one year of implementation.13

Sensitivity Analyses

One problem that is likely to arise in our analysis concerns the identification of comparison schools. We realize that using near selected schools other than those selected into the intervention in the following year as our primary comparison puts us at risk of removing schools from our comparison set that are in fact most comparable to schools treated in each cohort. As such, we run a second set of regressions based on the full set of comparison schools, including those schools treated in later cohorts of the reform. Next, to further ensure that our selection of comparison schools does not lead to biased estimates of the PSCI intervention, we run a third set of models that compare student achievement at focus schools to that of students at all low-performing schools in LAUSD—those labeled as in PI3+ or higher under NCLB classifications (i.e., those consistently failing to make Adequate Yearly Progress under NCLB). In this analysis, we compare students in focus schools with the set of students in PI3+ schools not treated by PSCI. There are 348 schools in this comparison group, 337 of which are nonalternative education schools that are not treated in PSCI. Because of incomplete data from charter schools, we are left with a final sample of 322 PI3+ schools in our panel dataset.14

Our next robustness check concerns our use of the first official year of implementation as the first year in which PSCI might impact students in treated schools. For instance, for cohort 1.0 we use 2010–11 as treatment year 1 (in which the implementation of cohort 1.0 commences) because this is the first year selected schools operate under their new plans following the 2009–10 school year in which all competing plans were reviewed. Schools identified for treatment in each cohort knew of their treatment status in the year immediately prior to treatment, however, and in fact were in the midst of writing their PSCI school plans. Based on our qualitative data, we believe this might have caused administrators, teachers, and possibly students to change their behaviors in this “pre” year—for better or for worse—and this may have impacted performance in these schools in the pre-implementation/identification year. We determine if there is an “identification year effect” by estimating alternate models that allow the PSCI intervention to begin in the identification year. This is similar to a traditional “pre-year” robustness check in which we might look for bias in our results. This analysis is not intended to uncover bias but rather to isolate an impact of a different sort in the selection year.

Our next set of robustness checks explores whether or not our models predict significant changes in outcomes that should not be impacted by the onset or implementation of PSCI. Specifically, we run the model specified above, this time predicting students’ race (black) or ethnicity (Hispanic)—characteristics that should not be impacted by the treatment. Because students’ race and ethnicity do not change over time, we cannot include a student fixed effect in our model. We run the ELA and math achievement models with no fixed effect and find no substantive differences with our fixed effect model (available upon request).

Our last sensitivity analysis addresses concerns that we may be identifying the impact of the federal SIG turnaround intervention rather than PSCI. SIG schools were given considerable funding and support to aid in turnaround efforts. Six focus and ten near-selected schools in cohort 1.0, and three focus and three near-selected schools in cohort 2.0, were part of the federal SIG intervention (no focus schools were SIG schools in cohort 3.0). We estimate models that include an indicator for SIG schools and an interaction of that indicator with the PSCI treatment indicator to separately estimate trends for focus and near-selected schools that received the federal SIG intervention.

Qualitative and Survey Data

We also draw on qualitative data from multiple data sources: (1) we conducted twenty-six interviews with LAUSD board members, superintendents, and executive-level staff, key leaders at partner organizations such as the teachers’ and administrators’ unions, the Los Angeles School Development Institute (LASDI, a partnership between the district, the teachers’ union, and the administrators’ union created to provide technical assistance to some applicant teams), and the United Way; (2) we conducted case studies of four cohort 2.0 focus schools and two cohort 3.0 focus schools (which include interviews with five design team leaders during the identification year, twenty administrators and forty teachers, and observations at nine school accountability reviews during implementation years); (3) we observed four school accountability reviews and three technical assistance meetings that involved multiple 1.0 schools; and (4) we analyzed all relevant documents (meeting agendas, PowerPoint presentations, and print and online communication). Collectively, these data provide information on the overall design of PSCI and information on what occurred in schools during the pre-year and the first years of PSCI implementation.

We coded the observation notes, interview transcripts, and documents along dimensions upon which the intervention was based (e.g., the initiative's theory of change) and then analyzed these data within and across schools. Specifically, we created detailed case reports, with qualitative data coded along topical themes (e.g., technical assistance, hindrance to school start-up, budgeting and finance, parent engagement). We coded non-case interviews and observation notes along the same themes, and also conducted open coding to identify emergent themes. We then conducted a cross-case analysis, examining data within each theme across all case schools, to infer factors that influenced implementation. To enhance the internal validity and accuracy of findings, we triangulated data from multiple sources, comparing interview data to documents and observation notes whenever possible.

We also include results from a survey of PSCI focus school principals. We administered this survey in the spring of each study year for every cohort of treated principals, asking about their perceptions of the reform, intermediate outcomes that might have stemmed from the reform, and their implementation of reform elements. Given our study timeline, we do not capture cohort 1.0 principal responses in their first year of implementation, but we do have data from the second and third years (2011–12 and 2012–13). We have principal responses from cohort 2.0 principals in their first and second years of implementation (2011–12 and 2012–13), and cohort 3.0 principals in their first year of implementation (2012–13). Table 3 shows response rates by cohort and year. In addition, we draw upon data from a survey administered to all applicant team leaders (leaders of the teams proposing school plans to operate focus schools) in the second and third years of the initiative. In the second cohort, 37 of 46 applicant teams (80 percent) responded, and in the third cohort, 46 of 54 teams (85 percent) responded. The inclusion of these qualitative and survey data and the accompanying analyses enable us to better understand potential explanations for differences in the efficacy of the reforms, and therefore ways to improve turnaround reforms in future policy efforts.

Table 3. 
Response Rates by Cohort and Year for Principal School Survey
CohortTotalCompleted 2011–12 Principal SurveyCompleted 2012–13 Principal Survey
1.0 14 11 
(response rate)  (79%) (64%) 
2.0 
(response rate)  (60%) (60%) 
3.0 — 
(response rate)   (67%) 
CohortTotalCompleted 2011–12 Principal SurveyCompleted 2012–13 Principal Survey
1.0 14 11 
(response rate)  (79%) (64%) 
2.0 
(response rate)  (60%) (60%) 
3.0 — 
(response rate)   (67%) 

Note: Given our study timeline, we do not capture cohort 1.0 principal responses in their first year of implementation, but we do have data from the second and third years of implementation (2011–12 and 2012–13). We have responses from cohort 2.0 principals in their first and second years of implementation (2011–12 and 2012–13), and cohort 3.0 principals in their first year of implementation (2012–13).

Limitations

Although we are able to address most of the limitations to our analyses with the robustness checks specified here, it is, of course, still possible that unobserved factors in the treated schools drive our results. Nonetheless, the very purpose of the CITS design is to mitigate this possibility, and our many specification checks are intended to verify that our results are indeed capturing the impact of turnaround reforms across cohorts. Although we have taken various measures to account for unobserved factors in our analyses, there are two other limitations to our empirical methods that are worth noting. The first limitation to our analysis concerns the small sample of treated turnaround schools in each cohort of PSCI, as is shown in table 1. These small sample sizes may limit generalizability of our results. The small sample sizes impact our principal survey results, as well. Even though response rates from focus principals are relatively high (between 60 percent and 79 percent, depending on cohort and year), absolute sample sizes remain quite small given the size of the overall treated sample. This is especially the case for cohort 2.0, for which we have only five treated schools and survey responses from only three principals in each year.

Second, we are unable to address the limited availability of data. Given that we have three years of “post” data, we cannot examine achievement trajectories for more than a three-year period for cohort 1.0 schools, two years for cohort 2.0 schools, and one year for cohort 3.0 schools. Longer implementation time periods would provide a better picture of the long-term effects of the reform. We have no option but to stop data collection at this point, however, because California ceased offering the CSTs after the 2012–13 school year and did not release standardized test scores in 2013–14. Given the expectation that turnaround reforms bring about improvements in student achievement in just one or a few years, we consider three years to be a valid time frame for analysis of PSCI. Note that cohort 3.0 results, which only have one year of achievement outcomes, can only inform us about the immediate effect of the reform on cohort 3.0 turnaround schools, and should be considered suggestive given the lack of post-trend data.

Third, our qualitative data collection activities began in 2010–11 and mainly covered cohorts 2.0 and 3.0. As a result, we did not directly observe technical assistance and support provided to cohort 1.0 schools, which occurred in 2009–10. We do draw on data from observations of meetings and reviews of cohort 1.0 schools, which occurred during the first year of implementation for these schools, as well as from interviews with district and other leaders related to cohort 1.0.

5.  Results

Examination of Graphical Trends

Before presenting the results from our regressions, we look to figure 1 to compare the unadjusted trends in math and ELA achievement for students enrolled in treated PSCI schools with those of students enrolled in comparison group schools. As discussed before, the lines connecting these mean values represent the trend in student outcomes. The solid lines (with circles) show students in treated focus schools and the dashed lines (with triangles) show students in near selected schools. Overall, comparing the graphs across cohorts suggests a number of interesting possible findings. First, across all cohorts, the first year of implementation does not provide immediate gains in achievement outcomes for focus students relative to near-selected students, although the magnitudes of achievement drops are substantively small for cohorts 1.0 and 2.0 and may not deviate too far from the general trajectories of students in near-selected low-performing LAUSD schools. In fact, in cohort 2.0 there is evidence of an increase in ELA achievement in year 1 relative to near-selected schools. Second, whereas first-year drops in student achievement in the first two cohorts of the reform appear relatively small in magnitude, there are drastic decreases in achievement for cohort 3.0 in their first year of the reform relative to students in near-selected schools, and possibly even in the year of identification relative to near-selected schools (particularly in math achievement). Third, figure 1 shows consistent year 2 gains in student achievement in both math and ELA relative to comparison schools for both cohorts 1.0 and 2.0, although by year three gains appear to be somewhat washed away in math (cohort 1.0). Last, the disparate patterns evidenced across cohorts may suggest responses to changes in the intervention itself.15

Comparative Interrupted Time Series Results

Table 4 presents results from our main CITS specifications comparing the ELA and math achievement of students enrolled in focus versus near-selected schools. We find interesting variation in impacts across cohorts. Although cohort 1.0 focus schools, shown in the first and fourth columns of table 4, see no significant changes in achievement in either the level year or in overall growth in the following two years, students in cohort 2.0 schools (columns 2 and 5) experience significant and somewhat substantial gains in ELA achievement in both the first and second year of the reform. In fact, the overall change in ELA performance for these students after two years of PSCI implementation (shown in the bottom row of table 4) is 14.4 percent of a standard deviation, which is relatively large by education reform standards. Cohort 2.0 math regressions show positive but insignificant improvements in both years 1 and 2. Overall, students enrolled in cohort 2.0 experience an 8.0 percent standard deviation increase in math achievement over the first two years of the reform, although this is not significant at conventional levels.16,17

Table 4. 
ELA and Math Achievement for Students Enrolled in Focus Schools Relative to Near-selected Schools
ELAMath
1.02.03.01.02.03.0
YEARt −0.024*** −0.024*** −0.020*** −0.008 −0.018 −0.009 
 (0.004) (0.006) (0.005) (0.009) (0.012) (0.008) 
PSCIt −0.050*** −0.057*** 0.102*** −0.017 −0.043** 0.075** 
 (0.007) (0.012) (0.014) (0.014) (0.017) (0.028) 
YEARS_SINCE_PSCIt 0.023*** 0.086*** — 0.020* 0.018 — 
 (0.005) (0.018) — (0.010) (0.031) — 
Tis × YEARt −0.002 −0.019*** 0.001 −0.014** −0.010 0.003 
 (0.004) (0.003) (0.005) (0.005) (0.008) (0.010) 
Tis × PSCIt −0.014 0.079** −0.102*** 0.009 0.025 −0.162** 
 (0.013) (0.024) (0.025) (0.028) (0.046) (0.055) 
Tis × YEARS_SINCE_PSCIt 0.008 0.065** — 0.020 0.055 — 
 (0.010) (0.020) — (0.018) (0.036) — 
FRL ELIGIBLE 0.008** −0.009 0.000 0.009+ −0.014* 0.011 
 (0.003) (0.005) (0.004) (0.005) (0.007) (0.007) 
SPED ELIGIBLE −0.031*** −0.062*** 0.004 0.042*** 0.032** 0.080*** 
 (0.006) (0.009) (0.012) (0.008) (0.011) (0.013) 
MOBILITY −0.015** −0.009 −0.005 −0.035*** −0.059*** −0.023* 
 (0.005) (0.008) (0.007) (0.011) (0.016) (0.011) 
LAG ACHIEVEMENT SCORE 0.075*** 0.093*** 0.046*** 0.084*** 0.100*** 0.046*** 
 (0.005) (0.007) (0.008) (0.007) (0.009) (0.012) 
% FRL ELIGIBLE 0.027 0.087* 0.086* 0.039 0.009 0.040 
 (0.025) (0.039) (0.043) (0.043) (0.066) (0.070) 
% ENGLISH LANGUAGE LEARNER −0.222*** −0.227*** −0.305*** −0.203*** −0.128* −0.180** 
 (0.031) (0.032) (0.041) (0.048) (0.051) (0.067) 
LN(ENROLLMENT) −0.009 0.009 0.011 −0.023+ 0.001 0.002 
 (0.007) (0.009) (0.012) (0.013) (0.015) (0.020) 
MIDDLE SCHOOL 0.025 −0.013 −0.075*** 0.078* 0.098* 0.013 
 (0.016) (0.022) (0.021) (0.032) (0.039) (0.037) 
HIGH SCHOOL 0.131*** 0.108** 0.025 −0.169** −0.112 −0.296*** 
 (0.024) (0.036) (0.030) (0.053) (0.073) (0.059) 
CONSTANT 0.077 −0.083 −0.066 0.122 −0.002 −0.053 
 (0.054) (0.059) (0.082) (0.096) (0.124) (0.143) 
R2-ad 0.774 0.762 0.782 0.656 0.650 0.684 
# of students 117,541 29,750 28,469 116,151 29,113 28,263 
# of schools 85 28 46 85 28 46 
F-stat 39.456 46.466 17.035 48.819 47.675 27.492 
Total effect 0.002 0.144*** — 0.048 0.080 — 
(as of 2010–11 for 1.0, 2011–12 for 2.0) (0.020) (0.028)  (0.056) (0.049)  
ELAMath
1.02.03.01.02.03.0
YEARt −0.024*** −0.024*** −0.020*** −0.008 −0.018 −0.009 
 (0.004) (0.006) (0.005) (0.009) (0.012) (0.008) 
PSCIt −0.050*** −0.057*** 0.102*** −0.017 −0.043** 0.075** 
 (0.007) (0.012) (0.014) (0.014) (0.017) (0.028) 
YEARS_SINCE_PSCIt 0.023*** 0.086*** — 0.020* 0.018 — 
 (0.005) (0.018) — (0.010) (0.031) — 
Tis × YEARt −0.002 −0.019*** 0.001 −0.014** −0.010 0.003 
 (0.004) (0.003) (0.005) (0.005) (0.008) (0.010) 
Tis × PSCIt −0.014 0.079** −0.102*** 0.009 0.025 −0.162** 
 (0.013) (0.024) (0.025) (0.028) (0.046) (0.055) 
Tis × YEARS_SINCE_PSCIt 0.008 0.065** — 0.020 0.055 — 
 (0.010) (0.020) — (0.018) (0.036) — 
FRL ELIGIBLE 0.008** −0.009 0.000 0.009+ −0.014* 0.011 
 (0.003) (0.005) (0.004) (0.005) (0.007) (0.007) 
SPED ELIGIBLE −0.031*** −0.062*** 0.004 0.042*** 0.032** 0.080*** 
 (0.006) (0.009) (0.012) (0.008) (0.011) (0.013) 
MOBILITY −0.015** −0.009 −0.005 −0.035*** −0.059*** −0.023* 
 (0.005) (0.008) (0.007) (0.011) (0.016) (0.011) 
LAG ACHIEVEMENT SCORE 0.075*** 0.093*** 0.046*** 0.084*** 0.100*** 0.046*** 
 (0.005) (0.007) (0.008) (0.007) (0.009) (0.012) 
% FRL ELIGIBLE 0.027 0.087* 0.086* 0.039 0.009 0.040 
 (0.025) (0.039) (0.043) (0.043) (0.066) (0.070) 
% ENGLISH LANGUAGE LEARNER −0.222*** −0.227*** −0.305*** −0.203*** −0.128* −0.180** 
 (0.031) (0.032) (0.041) (0.048) (0.051) (0.067) 
LN(ENROLLMENT) −0.009 0.009 0.011 −0.023+ 0.001 0.002 
 (0.007) (0.009) (0.012) (0.013) (0.015) (0.020) 
MIDDLE SCHOOL 0.025 −0.013 −0.075*** 0.078* 0.098* 0.013 
 (0.016) (0.022) (0.021) (0.032) (0.039) (0.037) 
HIGH SCHOOL 0.131*** 0.108** 0.025 −0.169** −0.112 −0.296*** 
 (0.024) (0.036) (0.030) (0.053) (0.073) (0.059) 
CONSTANT 0.077 −0.083 −0.066 0.122 −0.002 −0.053 
 (0.054) (0.059) (0.082) (0.096) (0.124) (0.143) 
R2-ad 0.774 0.762 0.782 0.656 0.650 0.684 
# of students 117,541 29,750 28,469 116,151 29,113 28,263 
# of schools 85 28 46 85 28 46 
F-stat 39.456 46.466 17.035 48.819 47.675 27.492 
Total effect 0.002 0.144*** — 0.048 0.080 — 
(as of 2010–11 for 1.0, 2011–12 for 2.0) (0.020) (0.028)  (0.056) (0.049)  

Notes: Results are from our main CITS specification comparing the ELA achievement of students enrolled in cohorts 1.0–3.0 focus versus near-selected schools (not including those included in later cohorts of the reform). Our main variables of interest are Tis × PSCIt, which represents the effect of PSCI on achievement in the first year of the reform, and Tis × YEARS_SINCE_PSCIt, which represents the growth effect of PSCI in subsequent years. Standard errors are clustered to the school level.

+p < 0.10; *p < 0.05; **p < 0.01; ***p < 0.001.

Although the year 1 level effects of PSCI on math achievement are not significantly different from zero for cohorts 1.0 and 2.0, students in cohort 2.0 schools appeared to perform significantly better in ELA achievement than did students in near-selected schools.18 However, students in Cohort 3.0 focus schools experience a rather large and significant drop in both ELA and math achievement in the first year of the reform, relative to students in near-selected comparison schools. Again, these results are suggestive, but the magnitudes of the decreases in student achievement in Cohort 3.0 are 10.2 percent and 16.2 percent of a standard deviation in ELA and math, respectively. When we again examine figure 1, it is clear that at least part of the magnitude of the 3.0 effects is due to the slight increase in student achievement experienced by students in near-selected schools. Whereas students in focus schools had been experiencing a slight downward trajectory in years previous to PSCI, with a marked decrease during implementation, students in near-selected schools had been on a fairly level performance trajectory, with small increases and decreases throughout. In addition, figure 1 shows that in all cases but one (cohort 2.0 ELA achievement), students in focus schools saw at least a slight decrease in achievement during the first year of implementation. Their comparison near-selected schools saw similar or greater decreases in achievement in the first two cohorts relative to pre-treatment trends, however, suggesting that PSCI students in cohorts 1.0 and 2.0 performed less poorly in their first year than they would have without the intervention.

Sensitivity Analyses

We present in the Appendix the coefficients of the level and growth effects for our five sets of sensitivity checks, with the calculation of overall impact where appropriate. Table A.1 provides results for our first four sensitivity checks. The first four columns show results when we expand our comparison sets to all near-selected schools and to all PI3+ schools, which confirm the findings of our primary analysis. This indicates that our main findings are not biased by our selection of comparison schools. The next two columns provide results from a falsification test whereby we ask if the PSCI turnaround reforms predict what should be unrelated student characteristics (race and ethnicity) in patterns similar to the effects seen above on student achievement outcomes. We find, as expected, PSCI turnarounds do not predict student race or ethnicity. The last two columns of table A.1 provide results from our selection-year analysis, which allows for the possibility that PSCI treatment may have begun in the identification year. We find that, for cohorts 1.0 and 2.0, there were no significant impacts of PSCI identification and planning on student achievement outcomes. In Cohort 3.0, however, students in identified PSCI focus schools suffered a significant decrease in math and ELA achievement relative to near-selected schools. In the next section we discuss why we believe this might be the case.

Table A.2 provides results from specifications that separately model trends for non-SIG PSCI schools and SIG PSCI schools.19 We find that main effects of PSCI remain consistent in models that account for the SIG intervention.

Qualitative and Survey Analysis

In an intervention as complex as PSCI's turnaround reform, it is difficult to make sense of differences in outcomes across cohorts. In particular, we find that PSCI had null impacts for cohort 1.0 and very positive impacts on students enrolled in cohort 2.0 focus schools in both the first and ensuing implementation years, and substantively negative impacts on students enrolled in cohort 3.0 focus schools in the identification and first implementation years. In this section, we pull from our principal surveys (keeping in mind the relatively high response rate but overall small sample size of these results), our applicant team leader surveys and our qualitative data to help to explain these discrepancies in outcomes between reform cohorts.

Positive Cohort 2.0 ELA Achievement Results

Our qualitative data suggest four factors likely contributed to the improvements in cohort 2.0 achievement results: (1) LAUSD learned from and improved upon difficulties it faced in the initial cohort of the reform; (2) LAUSD and partners provided substantial professional development to cohort 2.0 schools to improve implementation; (3) all 2.0 focus schools were turned around through reconstitution and restart models, as opposed to the use of “softer” forms of turnaround in cohorts 1.0 and 3.0; and (4) cohort 2.0 respondents reported greater ease of implementation and higher fidelity to implementation plans than did other cohorts’ respondents.

Lessons Learned from Cohort 1.0.

District leaders and stakeholders reported that cohort 1.0 schools faced difficult startup processes. Although many whole school reform efforts face challenges in the first year of implementation (see, e.g., Borman et al. 2003; Booker et al. 2004; Bifulco, Duncombe, and Yinger 2005; Gill et al. 2005; Bifulco and Ladd 2006; Orland et al. 2008; De la Torre et al. 2013), study respondents noted that difficulties went beyond traditional first-year problems and were exacerbated by specific aspects of PSCI, especially by the fact that teachers employed at many PSCI schools were not familiar with or committed to the school plans intended to guide reform. According to one teacher's union leader, “The startup process in 1.0 was terrible … We held meetings … where we heard common complaints about the teachers not knowing the plan, that there wasn't any support for the plan, that the principal wasn't actually implementing the plan even if they did know the plan.”

District and partner organization leaders overwhelmingly reported that implementation-phase support and technical assistance were inadequate during cohort 1.0 schools’ first year of implementation. As a result, some schools may not have had the capacity to fully implement their plans. Although PSCI 1.0 teams received substantial support with writing their PSCI plans, one LASDI leader explained that implementation support was provided too late: “I think our biggest problem was the drop off after [the selection decision]. We didn't hit them [with implementation support] until April, after they had started, and there's just not the same level of urgency.”

LAUSD and LASDI realized relatively early on that many cohort 1.0 schools were not improving in their first year of implementation. John Deasy, who was serving on the district's leadership team but was neither the architect of PSCI nor superintendent of LAUSD in the first year of PSCI implementation (but was in year 2), explained: “If I learned anything from the first round, it was that in some places nothing changed whatsoever—possibly drifted. Well, if the school is already in peril and nothing is changing, you could make the case it probably drifted backwards from an already bad place.”

Enhanced Professional Development and Support for Cohort 2.0.

In response to year 1 complaints about the lack of professional development (PD) support and time to institute turnaround reforms, LAUSD provided all cohort 2.0 schools with two weeks of paid PD for their staffs. Offered in the summer before the start of the first school year, this PD was intended to allow school teams to work together to implement their school plans and to collaborate on the reforms that would help to turn around student performance. Staff at our cohort 2.0 case study sites noted that this PD was helpful and benefited their schools by ensuring that all staff were familiarized with the school plan, vision, and mission, and by providing training in new instructional methods and fostering teacher collaboration and respect for school leaders. One case study respondent called out the importance of this summertime PD, noting: “during the two weeks of summer, we spent a lot of time on the four cornerstone pieces of the plan, so they [teaching staff] were familiar with it.” The same respondent added, “I think that our staff is very familiar with the vision and mission. We spent a lot of time talking about it at the two weeks at the beginning of our opening year [and that] really helped us do that.” Survey data confirm that cohort 2.0 schools unanimously received support in certain areas of school operations during the first year of implementation. In particular, all cohort 2.0 principals reported receiving technical assistance for using data to guide their educational program, preparing for school reviews and walkthroughs, and understanding how PSCI schools would be held accountable for accomplishing school goals. Notably, cohort 3.0 schools did not receive the same paid summer PD opportunities, and as a result lower proportions of cohort 3.0 principals reported receiving the same degree of technical assistance upon implementation.

LAUSD also was able to devote additional resources to later cohorts of PSCI schools. In August 2010, the district and partners received nearly $5 million in federal funds through the Investing in Innovation grant competition, and $1 million in matching private funds, which were intended to support the development and implementation of school plans and to develop new accountability processes for schools that were operating PSCI schools. During the first year of the Investing in Innovation grant, much of this support was allocated to assisting cohort 2.0 teams develop PSCI plans.

The Use of Reconstitution and Restart Turnaround Models.

Of the five focus schools in cohort 2.0, three were reconstituted and two were opened by a charter school operator in a type of restart turnaround model. This means that all focus schools in cohort 2.0 were restaffed: Reconstituted schools were instructed to rehire no more than 50 percent of their staff, and the two charter schools did not rehire any teachers who had previously worked at the school sites. The former version of turnaround is similar to the SIG turnaround model that Dee (2012) found to be effective in his study of California SIG schools, and the latter restart model is consistent with SIG turnaround in that it requires whole-school restaffing. Principals in these cohort 2.0 schools generally reported appreciating the opportunity to restaff their schools. For instance, the principal of one cohort 2.0 case study school noted that reconstitution was

a good thing for our school, that we could get different people in, that we can really get a staff here who's committed to the school, who wants to be at this school, and whoever was going to be here will have to go through a process of being here. … So, it really got us a chance [to get rid of] people who weren't really on board with certain things.

This is not to say that reconstitution was not without its difficulties. In particular, staff at all cohort 2.0 case study schools reported that there was insufficient time between the time plans were selected and the next school year began to adequately restart the school. Respondents highlighted that staffing posed a particular challenge. In addition to the short time period in which principals were supposed to select new staff, principals at two cases stated they were hindered in restaffing by the limited supply of high-quality teachers still available in the hiring pool at the relatively late date at which PSCI schools could begin recruiting. This problem was augmented by the fact that many teachers who were not rehired simply shuffled between cohort 2.0 focus schools. One cohort 2.0 teacher explained that this shuffling was especially problematic because, although some teachers at reconstituted schools may change their behavior significantly in their new setting, others may continue to rely on previous (and presumably inadequate) instructional and classroom management methods. Despite these challenges, all 2.0 case principals reported in both the first and second years of implementation that they were satisfied with almost all of their staff selections.

In addition to the benefits of new staff, reconstitution may have also brought with it a host of supports and flexibilities, including some related to staffing. One principal at a cohort 2.0 reconstituted school explained,

We were able to reconstitute the whole school. … We were given a clean slate and we were also given additional funds through LAUSD for PDs two weeks before. We were also given the flexibility on how we want to create our schedule for the day … do the periodic assessments … There was a lot of curriculum freedom along with the staffing. We were able to hire every single person we wanted to hire. We weren't restricted to those people that we wanted to hire from this list … it made a huge difference.

Greater Ease of Implementation and Fidelity to Reform Plans.

Consistent with these explanations, our principal survey data indicate that cohort 2.0 schools had easier experiences implementing the PSCI plans in these early years, with a higher proportion of cohort 2.0 principals reporting the highest levels of plan implementation fidelity relative to cohort 3.0 and 1.0 schools during the first and second years of implementation, respectively. In particular, all cohort 2.0 school principals reported high levels of plan implementation fidelity in the areas of curriculum and instruction, use of assessment and school data, professional development, school culture and climate, and school leadership and staffing. In contrast, only 50 percent to 83 percent of cohort 3.0 schools, and 64 percent to 91 percent of cohort 1.0 schools, reported high levels of implementation fidelity in these same areas. (Tables showing differences in survey responses between cohorts are available from the authors upon request.)

In addition to smoother implementation of school plans, a lower proportion of cohort 2.0 schools reported hindrances in several areas of plan implementation. In the first year of implementation, between zero percent to 33 percent of cohort 2.0 schools reported high levels of hindrances in terms of not having enough time to plan for the school year, failing to obtain waivers or autonomies requested in their school plans, and having insufficient resources to implement their school plans. These proportions are much lower than the 17 percent to 83 percent of cohort 3.0 schools that reported high-levels of hindrances in these same areas. We find a similar pattern in survey data for the second year of implementation (comparing cohort 2.0 with 1.0).

Negative Cohort 3.0 Achievement Results

Although we have only one year of post-treatment data for Cohort 3.0—which causes us to interpret our results for this cohort as entirely suggestive rather than causal—the stark drops in student achievement found in both the identification and implementation year for cohort 3.0 focus schools (relative to the null outcomes in cohort 1.0 and the positive results in cohort 2.0) warrant some explanation from our qualitative data. In particular, our qualitative data indicate that the difference in outcomes can in part be explained by two factors: (1) substantial changes to the PSCI policy that impacted cohort 3.0; and (2) changes in the motivation of teams applying to turnaround PSCI schools and the resulting quality of school plans.

Changes to the PSCI Reform.

Following the appointment of Superintendent Deasy, and the election of several new Board members and a teachers’ union president, LAUSD made substantial changes to the initiative. Responding to Board and union pressure, Dr. Deasy signed a Memorandum of Understanding (MOU), which was then ratified by union members and approved by the Board in December 2011, stating that external teams of charter operators and nonprofits were only eligible to participate in PSCI if they agreed to operate the school using district employees under the current collective bargaining agreement (CBA). In effect, this removed much of the competition over focus schools, as most external groups refused to give up autonomy over hiring and freedom from the district's CBA, and thus withdrew from competition. In exchange for this policy shift, Dr. Deasy negotiated an agreement that all district schools would now have the option of adopting a newly created governance model (Local Initiative School), which would allow schools greater freedoms in the areas of curriculum and instruction, school scheduling, and staffing selection (but still requiring that all teachers would work under the district CBA).

On their own, these changes drastically changed the tenor of the reform from one that welcomed charter operators and other nonprofits to one that in effect removed them from the competition. In addition, these changes were made in the middle of cohort 3.0's identification and planning year. As a result, cohort 3.0 applicant teams actively writing plans to turn around identified schools reported increased levels of confusion as they attempted to change their plans to respond to the new governance models and competition rules. Cohort 3.0 case study sites conveyed uncertainty and apprehension during plan writing because they were aware, as teams were writing their plans, that the district and the union were negotiating an MOU to modify the PSCI process, including eligibility criteria and available governance models. Although teams were provided additional time to submit their plans following the MOU, case study staff reported misunderstandings and questions resulting from these changes. One cohort 3.0 applicant team reported:

There [were] a lot of changes like half-way through … like the union negotiations and who can be included and who cannot. It's a point where you think you are applying for something … and then half-way through it's kind of like well you might not even be able to apply. … I mean, there was confusion I think for anywhere between 30 to 60 days in terms of: Do we qualify? Can we apply?

Given the tumultuous process during the planning and application year for cohort 3.0, it is no surprise that cohort 3.0 principals were more likely to report in their survey responses that they faced difficulties in plan implementation stemming from a lack of sufficient resources and time than did cohort 2.0 principals.

Changes to Cohort 3.0 Applicant Team Motivation and Plan Quality.

Our qualitative data also indicate a potential shift in the motivation of teams to participate in the process that may have resulted in lower quality plans. Across all cohorts, several teams in our case study schools indicated “preservation of the school” as their main motivation for participation. This rationale was qualitatively different between earlier cohorts (1.0 and 2.0) and the third cohort of applicants. In cohorts 1.0 and 2.0, the preservation theme was presented as a need to “keep our school” from being taken over by a team of educators that had not already been operating the school, and especially from charter school operators. In cohort 3.0, however, teams expressing a preservation rationale stated they “had to” write a plan or “didn't have a choice.” This was likely due to the reduction in competition for schools that stemmed from the revised MOU. Applicant team leader survey responses support these case study findings, indicating a statistically significant increase in the number of teams reporting one factor motivating their participation in PSCI was that they were told to write a plan by an administrator (6 percent in cohort 2.0 to 22 percent in cohort 3.0). The shift from an intrinsic desire to “keep” or “protect” a school to external pressure to write a plan may have affected the quality of plans produced and/or implementation fidelity. In fact, our independent analysis indicates plan quality decreased between cohorts 2.0 and 3.0 of the reform (see Strunk et al. 2017). This too may have contributed to the drop in student achievement between cohorts 2.0 and 3.0.

6.  Implications for Research and Policy

Our results show that turnaround reform in LAUSD did have significant and substantial impacts on student outcomes, and both the direction and magnitude of effects vary across cohorts and over time. In particular, turnaround reforms implemented as part of LAUSD's PSCI led to significant positive impacts on ELA achievement for students in cohort 2.0 of the reform and to significant negative impacts on both ELA and math achievement for students in cohort 3.0. Our qualitative data suggest that LAUSD's willingness to learn from implementation challenges in cohort 1.0 and attempts to remedy them in cohort 2.0 (by providing substantial PD and planning time) may have led to easier plan implementation, higher fidelity to original school plans in early implementation years, and fostered positive achievement growth for cohort 2.0 schools. In addition, the use of dramatic turnaround methods (reconstitution and restart), as opposed to softer reform methods (transformation) used in cohorts 1.0 and 3.0, may have enabled positive outcomes. Substantial declines in first-year achievement for students in cohort 3.0 schools may have been caused by dramatic and confusing changes to the PSCI policy mid-reform, which led to differences in educators’ motivations for engaging in the reform. These findings substantiate Dee's (2012) findings that reconstitution may improve student performance to a greater extent than other turnaround methods. In addition, they corroborate earlier work that suggests, in order to improve student outcomes, substantial resources and supports (such as PD, planning time, and flexibility over hiring, curriculum, schedule and other operational decisions) are necessary to enable turnaround or other whole-school reforms (see, for examples, Fraga, Erlichson, and Lee 1998; Wong et al. 1999; Malen et al. 2002; Hess 2003; Rice and Malen 2003, 2010; Malen and Rice 2004; Rice and Croninger 2005; Hamilton, Heilig, and Pazey 2013).

As an increasing number of districts and states adopt turnaround reforms as a key mechanism of school improvement, it is critical that policy makers attend to results from existing turnaround reforms so they can implement these policies with the greatest chance of helping students succeed. The outcomes associated with the PSCI turnaround reform—all three iterations under study here—have implications for both local and national policy. In particular, there are three main lessons for policy makers that emerge from our study of PSCI.

First, it is important that districts and states are flexible in their policy-setting such that they can learn from early iterations of the reform and implement mid-course corrections. LAUSD made substantial changes to PSCI between each cohort of the reform, based in part on lessons learned from earlier cohorts. In particular, changes made to PSCI between cohorts 1.0 and 2.0 suggest that the district became aware of early startup difficulties stemming from turnaround reforms, and worked to provide extra technical assistance and resources to help schools generate and implement high-quality school plans. Consistent with extant research on reconstitution and whole-school reform, the provision of resources, aligned to school capacity deficits, appeared to facilitate cohort 2.0 implementation (Muncey and McQuillan 1996; Nunnery 1998; Berends, Bodilly, and Kirby 2002; Datnow et al. 2003; Aladjem and Borman 2006). School-designed and school-led professional development time prior to the beginning of the school year may have influenced educator motivation and support for changes to school organization and operations, fostered teacher collaboration, and built the necessary capacity for teachers to implement dramatic instructional changes.

Nonetheless, LAUSD's changes to PSCI also serve as a cautionary tale. The second lesson from PSCI may be that drastic changes made to core tenets of a reform mid-cycle may lead to confusion and other intermediate outcomes that can translate into achievement declines. To that end, the steep drop in math and ELA achievement of students enrolled in cohort 3.0 schools suggests the changes made to PSCI mid-year (for example, regarding what kinds of operators could apply for schools and under what conditions) may have resulted in adverse effects on the development of school plans, selection of school operators, and the staffing of schools. These effects may have influenced educator motivation and capacity to engage in organizational and instructional improvement efforts, eventually resulting in lower student achievement.

The third lesson stems from the heterogeneity of turnaround strategies used during the reform across cohorts. This variation in turnaround strategies used by LAUSD over the course of the reform provides interesting opportunities to examine differences across turnaround models. In particular, it appears that, when paired with increased supports, reconstituted schools (and schools that were restarted as charter schools) benefited from the enhanced flexibilities and new staffs that came with these turnaround strategies. This may suggest drastic turnaround reforms, such as reconstitution and restart, are more effective at improving student achievement, which is contrary to earlier implementation research on school reconstitution but in line with the most recent evidence about reconstitutions within SIG reforms (Dee 2012). Of course, the sample size in this study is too small to definitively state the superiority of one versus another. Unlike previous studies, however, our survey and qualitative data provide insights into the mechanisms that might account for these results. Future research on turnaround reforms would benefit from attention to disparate outcomes across turnaround models. In addition, attention to the context of school turnaround may offer important lessons on the relative importance of the provision of startup resources, capacity building, the supply of quality teachers, and the source of reform plans (internal versus external) for the implementation of different turnaround models. Whereas our study could not disentangle the various factors accounting for the positive cohort 2.0 outcomes, it suggests they result from the combination of restaffing as well as additional resources and support. Future studies might examine the following questions: If, in fact, dramatic staffing changes are required to spur improvement, can it occur at scale? Do certain supports matter more for reconstituted schools than schools adopting more modest turnaround approaches?

In light of this growing interest in prescriptive turnaround reforms, our study suggests that it is important to establish a firm research base about the efficacy of and implementation challenges inherent to specific reform models before instituting them in federal legislation governing multiple students, schools, and districts. Moreover, this study highlights the need to examine changes in interventions over time, and to be aware of how treatment and comparison schools may evolve over the course of a long-term intervention (Lemons et al. 2014).

Clearly, more research on the implementation and eventual efficacy of school turnaround reforms is necessary. As these reform concepts spread, it will be important to build on the successes and challenges of other districts implementing such reforms across the country to inform policy decisions at the federal, state, and local levels. Given the limited body of extant research on turnaround reforms, LAUSD's PSCI provides an excellent opportunity to study the impacts of one large urban school district's attempt to turn around struggling schools. This study provides researchers and practitioners with lessons regarding elements of turnaround that proved helpful in the LAUSD context. Research on various other districts implementing turnaround reforms can build this critical knowledge base, allowing policy makers and practitioners to make more informed decisions about which strategies to use, under what circumstances, and in what contexts.

Notes

1. 

Although the larger study on which this paper is based examines both relief schools and focus schools, this paper focuses only on the turnaround element of PSCI. Other work provides information about the implementation and outcomes associated with the relief schools (see Marsh, Strunk, and Bush 2013; Marsh et al. 2015; Strunk et al., 2016).

2. 

In this paper we refer to turnaround as the broader umbrella term, indicating a reform like PSCI that requires dramatic improvement in chronically low-performing schools. We call the SIG “turnaround” model “reconstitution.”

3. 

In addition, schools selected into PSCI could not be an iDesign school (a specific kind of partnership school within LAUSD). The indicators included in the district's selection process changed in later cohorts. In cohort 2.0, an API growth threshold and API growth points were also considered, and schools involved in cohort 1.0 or SIG were excluded from the process. In cohort 3.0, additional criteria included CST growth and subgroup proficiency.

4. 

In the first two cohorts of the reform the review included a panel of internal and external reviewers, a Superintendent's Review Panel, and parent and community advisory voting. The superintendent utilized these data, along with his own evaluation of the plans, and submitted final recommendations to the LAUSD Board, which then voted on the winner. Several of the Superintendent's recommendations were overturned by the Board in both years. Selected plans were implemented during the following academic year. In the third cohort, the advisory vote was removed and the superintendent became the final and binding vote. We provide greater detail on the proposal and award process for PSCI in earlier work (Marsh, Strunk, and Bush 2013; Strunk et al., 2016; Bush-Mecenas, Marsh, and Strunk 2016).

5. 

Cohort 2.0 included fewer focus schools because LAUSD's identification mechanism set lower performance bars for inclusion, resulting in fewer schools targeted for inclusion in PSCI.

6. 

Charter schools are not required to provide data to LAUSD. However, through work with the California Charter School Association and various individual charter operators, we were able to include charter school data for all of the treated focus schools, as well as 50 percent of the charter schools in near-selected comparison schools. Our sample excludes the 25 percent of enrolled LAUSD students who are in nontested grades (K, 1, and 12) and the approximately 89,000 students in charter schools for which we could not obtain data. Usual grade progression is defined as students who progress one grade in each year, who are ever retained in the same grade, or who skip ahead a grade once in two consecutive years. We remove students with atypical grade progression patterns from our sample because we are concerned that these students’ grades are miscoded.

7. 

In our final sample of students, we have about 12,700 student-year observations that belong to students enrolled in schools that span all grades K–12. These student-year observations are assigned a school level (elementary, middle, or high) based on the corresponding grade level of the student in that year.

8. 

We know from the district's theory of change (discussed at length in Marsh, Strunk, and Bush 2013) that low-performing schools not included in PSCI were still expected to be affected by the intervention due to spillover effects. Nonetheless, these non-PSCI low-performing schools should not have been equally affected by the intervention given that they were not subjected to the intervention itself, and any spillovers likely have a delayed time component to them. As such, spillover effects likely would be less obvious in the immediate years of the reform. In addition, these spillover effects would likely bias results downward, as they would result in increased performance among comparison schools due to PSCI.

9. 

Approximately one third of schools in LAUSD are missing data on one or more of the selection indicators and are, in essence, given a “bye” from the district on that missing indicator. The majority of these schools are charter schools (and not required to provide data to LAUSD). The most frequent missing indicator is graduation rate. In addition, 3 percent of schools are missing data on the percent proficient indicator and 3 percent are missing data on the API indicator.

10. 

For the first three cohorts of the reform, as schools in one cohort were being treated by the intervention, LAUSD was selecting schools for the next cohort of PSCI. Given this staged intervention design, at least some of our comparison schools will become treatment schools in the following cohort. Our main analysis removes schools from our comparison set if they are treated in the following round of PSCI. This yields the least biased estimates of the PSCI intervention, as it is probable that cohort 1.0 comparison schools treated in 2.0 are undergoing some form of treatment as part of PSCI's plan review process when schools treated in 1.0 are implementing the first year of their school plans and are in their first year of implementation in cohort 1.0's second year of implementation. The same logic applies for cohorts 2.0 and 3.0.

11. 

Students are identified as transferring within LAUSD if they change schools in consecutive years. We do not count students who matriculate from elementary to middle school or middle school to high school, or transfer from a feeder to a relief school for cohort 1.0 at the end of 2009–10, for cohort 2.0 at the end of 2010–11, or for cohort 3.0 at the end of 2011–12 as students who transfer within LAUSD. Students are identified as transferring out of LAUSD if they transfer to a charter school or do not reappear in our panel in the following year. We do not count students who matriculate from high school or leave the school district for other reasons (marriage, pregnancy, overage, etc.) as students transferring outside of LAUSD.

12. 

There may be a concern that PSCI also causes teachers to move across schools and out of the district in ways that might affect student achievement. In related analyses we do find that PSCI induces teacher mobility within and out of LAUSD, but there is little evidence that PSCI consistently impacts the mobility of more or less effective teachers.

13. 

Although we only have one year of outcomes for cohort 3.0, changes in PSCI policy before this cohort make these first-year results particularly interesting, so we leave them in and discuss them with the acknowledgment that these results should be considered as suggestive rather than causal.

14. 

We also could have used propensity-score modeling (PSM) to compare students enrolled at PSCI-treated schools to a similar group of students based on observable characteristics. Betts et al. (2010), however, find that PSM-derived comparison groups in CITS designs are less accurate than a comparison group of all nontreated schools, especially when analyzing data from large school districts. This suggests that comparing students enrolled in PSCI-treated schools to those enrolled in PI3+ schools is likely a more accurate pairing than deriving a matched comparison group.

15. 

Graphs of the adjusted trends in mean achievement for students enrolled at PSCI-treated schools to those of students in comparison schools (controlling for the student and school-level covariates specified in our regression analyses) more accurately reflect the results from our CITS analyses and are available from the authors upon request.

16. 

One might be concerned that particularly bad performance in the pre-year would drive year 1 results to look substantially and significantly positive. However, the graphic depictions in figure 1 show that this was not the case. In addition, our third sensitivity check, discussed subsequently, shows insignificant differences between treatment and comparison groups in the pre-year. Combined, these analyses assuage our concern that pre-year performance drove year 1 results.

17. 

Although the math achievement results are null in cohort 2.0, we note that the standard errors do not rule out comparable effects in math as in ELA.

18. 

We do not know why we find gains in ELA but not math, other than anecdotal evidence from discussions with district and school personnel that indicate treated schools focused primarily on ELA instruction in the early years of the intervention.

19. 

Sample sizes for each of these subgroups is fairly small, especially the group of SIG-focus schools, so we consider these results as solely exploratory.

Acknowledgments

The authors gratefully acknowledge support from the United States Department of Education through an Investing in Innovation grant. We also greatly appreciate the cooperation of administrators and Public School Choice Initiative participants within the Los Angeles Unified School District. We are indebted to Elizabeth Robitaille and Allison Bajracharya for their assistance in helping us to obtain student-level data from participating charter schools. In addition, we have benefited greatly from helpful feedback from Dominic Brewer, Carrie Connoway, Tom Dee, Matt Hill, Rebecca Jacobsen, Hank Levin, Betty Malen, Andrew McEachin, Donna Muncey, Macke Raymond, David Rattray, Jon Valant, and seminar participants at the University of California-Berkeley Graduate School of Education, the University of Arkansas Department of Education Reform, and the University of Michigan Ford School of Public Policy, as well as two anonymous reviewers. Any remaining errors are our own.

REFERENCES

Aladjem
,
Daniel K.
, and
Kathryn M.
Borman
.
2006
.
Summary of findings from the national longitudinal evaluation of Comprehensive School Reform
.
Paper presented at American Educational Research Association Annual Conference
,
San Francisco, CA, April
.
Berends
,
Mark
,
Susan J.
Bodilly
, and
Sheila N.
Kirby
.
2002
.
Facing the challenges of whole-school reform: New American Schools after a decade.
Santa Monica, CA
:
RAND
.
Betts
,
Julian
,
Jesse
Levin
,
Ana Paula
Miranda
,
Bruce
Christenson
,
Marion
Eaton
, and
Hans
Bos
.
2010
.
An evaluation of alternative matching techniques for use in comparative interrupted time series analyses: An application to elementary education
.
Unpublished paper
,
University of California San Diego
.
Bifulco
,
Robert
, and
Helen F.
Ladd
.
2006
.
The impacts of charter schools on student achievement: Evidence from North Carolina
.
Education Finance and Policy
1
(
1
):
50
90
. doi:10.1162/edfp.2006.1.1.50.
Bifulco
,
Robert
,
William
Duncombe
, and
Johnathan
Yinger
.
2005
.
Does whole-school reform boost student performance? The case of New York City
.
Journal of Policy Analysis and Management
24
(
1
):
47
72
. doi:10.1002/pam.20069.
Bloom
,
Howard S.
1999
.
Estimating program impacts on student achievement using short interrupted time series
.
New York
:
MDRC
.
Bloom
,
Howard S.
,
Sandra
Ham
,
Laura
Melton
, and
Julieanne
O’Brien
.
2001
.
Evaluating the accelerated schools approach: A look at early implementation and impacts on student achievement in eight elementary schools
.
New York
:
MDRC
.
Booker
,
Kevin
,
Scott M.
Gilpatric
,
Timothy
Gronberg
, and
Dennis
Jansen
.
2004
.
Charter school performance in Texas
.
Unpublished paper
,
Texas A&M University
.
Borman
,
Geoffrey
,
D. L.
Rachuba
,
Amanda
Datnow
,
M.
Alberge
,
M.
MacIver
, and
Sam
Stringfield
.
2000
.
Four models of school improvement: Successes and challenges in reforming low-performing, high-poverty Title I schools (CRESPAR Report No. 48)
.
Baltimore, MD
:
Center for Research on the Education of Students Placed At Risk, Johns Hopkins University
.
Borman
,
Geoffrey D.
,
Gina M.
Hewes
,
Laura T.
Overman
, and
Shelly
Brown
.
2003
.
Comprehensive school reform and achievement: A meta-analysis
.
Review of Educational Research
73
(
2
):
125
230
. doi:10.3102/00346543073002125.
Bush-Mecenas
,
Susan
,
Julie
Marsh
, and
Katharine O.
Strunk
.
2016
.
Portfolio reform in Los Angeles: Successes and challenges in school district implementation
. In
Thinking systemically: Improving districts under pressure
, edited by
Alan J.
Daly
and
Kara S.
Finnigan
, pp.
123
150
.
Washington, DC
:
American Educational Research Association
.
Calkins
,
Andrew
,
William
Guenther
,
Grace
Belfiore
, and
Dave
Lash
.
2007
.
The turnaround challenge: Why America's best opportunity to dramatically improve student achievement lies in our worst-performing schools
.
Boston, MA
:
Mass Insight Education & Research Institute
.
Datnow
,
Amanda
, and
Sam
Stringfield
.
2000
.
Working together for reliable school reform
.
Journal of Education for Students Placed at Risk
5
(
1–2
):
183
204
. doi:10.1080/10824669.2000.9671386.
Datnow
,
Amanda
,
Geoffrey D.
Borman
,
Sam
Stringfield
,
Laura T.
Overman
, and
M.
Castellano
.
2003
.
Comprehensive school reform in culturally and linguistically diverse contexts: Implementation and outcomes from a four-year study
.
Educational Evaluation and Policy Analysis
25
(
2
):
142
170
. doi:10.3102/01623737025002143.
Dee
,
Thomas S.
2012
.
School turnarounds: Evidence from the 2009 stimulus
.
NBER Working Paper No. w17990
.
Dee
,
Thomas S.
, and
Brian
Jacob
.
2011
.
The impact of No Child Left Behind on student achievement
.
Journal of Policy Analysis and Management
30
(
3
):
418
446
. doi:10.1002/pam.20586.
De la Torre
,
Marissa
,
Elaine M.
Allensworth
,
Sanja
Jagesic
,
James
Sebastian
,
Michael
Salmonowicz
,
Coby
Meyers
, and
R. Dean
Gerdeman
.
2013
.
Turning around low-performing schools in Chicago.
Chicago, IL
:
Consortium for Chicago School Research, University of Chicago
.
Doherty
,
Kathryn
, and
Sarah
Abernathy
.
1998
.
Turning around low-performing schools: A guide for state and local leaders
.
Washington, DC
:
U.S. Department of Education
.
Duke
,
Daniel L.
2006
.
Keys to sustaining successful school turnarounds
.
Spectrum Journal of Research and Information
24
(
4
):
21
35
.
Fraga
,
Luis R.
,
Bari A.
Erlichson
, and
Sandy
Lee
.
1998
.
Consensus building and school reform: The role of the courts in San Francisco
. In
Changing urban education
, edited by
Clarence N.
Stone
, pp.
66
92
.
Lawrence
:
University Press of Kansas
.
Gill
,
Brian
,
Laura
Hamilton
,
J. R.
Lockwood
,
Julie A.
Marsh
,
Ron
Zimmer
,
Deanna
Hill
, and
Shana
Pribesh
.
2005
.
Inspiration, perspiration, and time: Operations and achievement in Edison schools
.
Santa Monica, CA
:
RAND
.
Hamilton
,
Madlene P.
,
Julian V.
Heilig
, and
Barbara L.
Pazey
.
2013
.
A nostrum of school reform? Turning around reconstituted urban Texas high schools
.
Urban Education
20
(
10
):
1
34
.
Hansen
,
J. B.
,
A. V.
Kraetzer
, and
P.
Mukherjee
.
1998
.
Adams-Hunt achievement initiative final evaluation report
.
Paper presented at American Educational Research Association Annual Conference
,
Montreal, Canada, August
.
Herman
,
Rebecca
,
Priscilla
Dawson
,
Thomas S.
Dee
,
Jay
Greene
,
Rebecca
Maynard
,
Sam
Redding
, and
Marlene
Darwin
.
2008
.
Turning around chronically low-performing schools: A practice guide
(
NCEE 2008–4020
). Available http://ies.ed.gov/ncee/wwc/pdf/practice_guides/Turnaround_pg_04181.pdf.
Accessed 20 January 2016
.
Hess
,
G. Alfred
, Jr.
2003
.
Reconstitution three years later: Monitoring the effects of sanctions on Chicago high schools
.
Education and Urban Society
35
(
4
):
494
517
.
Knudson
,
Joel
,
Larissa
Shamburgh
, and
Jennifer
O’Day
.
2011
.
Beyond the school: Exploring a systematic approach to school turnaround
.
San Mateo, CA
:
California Collaborative on District Reform
.
Le Floch
,
Kerstin C.
,
Beatrice
Birman
,
Jennifer
O’Day
,
Steven
Hurlburt
,
Diana
Mercado-Garcia
,
Rose
Goff
,
Karen
Manship
, et al
2014
.
Case studies of schools receiving School Improvement Grants: Findings after the first year of implementation
(
NCEE 2014–4015
). Available http://ies.ed.gov/ncee/pubs/20144015/pdf/20144015.pdf.
Accessed 28 January 2016
.
Lemons
,
Christopher J.
,
Douglas
Fuchs
,
John K.
Gilbert
, and
Lynn S.
Fuchs
.
2014
.
Evidence-based practices in a changing world: Reconsidering the counterfactual in education research
.
Educational Researcher
43
(
5
):
242
252
. doi:10.3102/0013189X14539189.
Malen
,
Betty
,
Robert
Croninger
,
Donna
Muncey
, and
Donna
Redmond-Jones
.
2002
.
Reconstituting schools: ‘testing’ the ‘theory of action’
.
Educational Evaluation and Policy Analysis
24
(
2
):
113
132
. doi:10.3102/01623737024002113.
Malen
,
Betty
and
Jennifer
King Rice
.
2004
.
A framework for assessing the impact of education reforms on school capacity: Insights from studies of high-stakes accountability initiatives
.
Educational Policy
18
(
5
):
631
660
.
Malen
,
Betty
, and
Jennifer
King Rice
.
2009
.
School reconstitution and school improvement: Theory and evidence
. In
Handbook of education policy research
, edited by
Gary
Sykes
,
Barbara
Schneider
, and
David N.
Plank
, pp.
464
477
.
New York
:
Routledge
.
Marsh
,
Julie A.
,
Katharine O.
Strunk
, and
Susan
Bush
.
2013
.
Portfolio district reform meets school turnaround: Early implementation findings from the Los Angeles Public School Choice Initiative
.
Journal of Educational Administration
51
(
4
):
498
527
. doi:10.1108/09578231311325677.
Marsh
,
Julie A.
,
Katharine O.
Strunk
,
Alice
Huget
, and
Susan C.
Bush-Mecenas
.
2015
.
Democratic engagement in district reform: The evolving role of parents in the Los Angeles Public School Choice Initiative
.
Educational Policy
20
(
1
):
51
84
. doi:10.1177/0895904814563204.
Muncey
,
Donna E.
, and
Patrick J.
McQuillan
.
1996
.
Reform and resistance in schools and classrooms: An ethnographic view of the Coalition of Essential Schools
.
New Haven, CT
:
Yale University Press
.
National Center for Educational Evaluation and Regional Assistance (NCEE)
.
2014
.
Operational support, authority and monitoring of school turnarounds
(
NCEE Evaluation Brief 2014–4008
). Available https://ies.ed.gov/ncee/pubs/20144008/.
Accessed 28 January 2016
.
Nunnery
,
John A.
1998
.
Reform ideology and the locus of development problem in educational restructuring: Enduring lessons from studies of educational innovation
.
Education and Urban Society
30
(
3
):
277
295
. doi:10.1177/0013124598030003002.
Orland
,
Martin
,
Brooke
Connolly
,
Tony
Fong
,
Lauren D.
Sosenko
,
Naida C.
Tushnet
,
Robert K.
Yin
,
Januela M.
Burt
, and
Emily
Warner
.
2008
.
Evaluation of the Comprehensive School Reform program implementation and outcomes: Third-year report
.
Washington, DC
:
U.S. Department of Education, Office of Planning, Evaluation and Policy Development
.
Rice
,
Jennifer K.
, and
Robert
Croninger
.
2005
.
Resource generation, reallocation, or depletion: An analysis of the impact of reconstitution on school capacity
.
Leadership and Policy in Schools
4
(
2
):
73
104
. doi:10.1080/15700760590965569.
Rice
,
Jennifer K.
, and
Betty
Malen
.
2003
.
The human costs of education reform: The case of school reconstitution
.
Educational Administration Quarterly
39
(
5
):
635
666
. doi:10.1177/0013161X03257298.
Rice
,
Jennifer K.
, and
Betty
Malen
.
2010
.
School reconstitution as an education reform strategy: A synopsis of the evidence
.
Washington, DC
:
National Education Association
.
Scott
,
Caitlin
,
Jennifer
McMurrer
,
Shelby
McIntosh
, and
Kenne
Dibner
.
2012
.
Opportunities and obstacles: Implementing stimulus-funded school improvement grants in Maryland, Michigan, and Idaho
.
Washington, DC
:
Center on Education Policy
.
Shadish
,
William R.
,
Thomas D.
Cook
, and
Donald T.
Campbell
.
2002
.
Experimental and quasi-experimental designs for generalized causal inference.
Boston, MA
:
Houghton-Mifflin
.
Somers
,
Marie Andree
,
Pei
Zhu
,
Robin
Jacob
, and
Howard
Bloom
.
2013
.
The validity and precision of the comparative interrupted time series design and the difference-in-difference design in educational evaluation
.
New York
:
MDRC
.
Strunk
,
Katharine O.
,
Julie A.
Marsh
,
Susan
Bush-Mecenas
, and
Matthew
Duque
.
2016
.
The best laid plans: An examination of school plan quality and implementation in a school improvement initiative
.
Educational Administration Quarterly
52
(
2
):
259
309
. doi:0013161X15616864.
U.S. Department of Education (USDOE)
.
2010a
.
Secretary Arne Duncan's testimony before the House Subcommittee on Labor/HHS/ED appropriations on the President's proposals for the 2011 education budget and for reauthorization of ESEA
. Available www2.ed.gov/news/speeches/2010/03/03182010.html.
Accessed 19 January 2016
.
U.S. Department of Education (USDOE)
.
2010b
.
Guidance on fiscal year 2010 School Improvement Grants under section 1003(G) of the Elementary and Secondary Education Act of 1965
.
Washington, DC
:
USDOE, Office of Elementary and Secondary Education
.
U.S. Government Accountability Office (USGAO)
.
2012
. School Improvement Grants: Education should take additional steps to enhance accountability for schools and contractors. Washington, DC: USGAO.
U.S. Department of Education (USDOE)
.
2012
.
School Improvement Grants: Education should take additional steps to enhance accountability for schools and contractors
.
Washington, DC
:
USGAO
.
Villavicencio
,
Adrianna
, and
Justina K.
Grayman
.
2012
.
Learning from turnaround middle schools: Strategies for success.
Available https://steinhardt.nyu.edu/scmsAdmin/media/users/sg158/PDFs/turnaround_ms/TurnaroundMiddleSchools.pdf. Accessed 20 January 2016.
Wong
,
Kenneth
,
Dorothea
Anagnostopoulos
,
Stacey
Rutledge
,
Laurence
Lynn
, and
Robert
Dreeben
.
1999
.
Implementation of an educational accountability agenda: Integrated governance in the Chicago Public Schools enters its fourth year
.
Chicago, IL
:
Irving B. Harris Graduate School of Public Policy Studies, University of Chicago
.

Appendix:  Sensitivity Checks

Tables A.1 and A.2 provide the coefficients of the level and growth effects for our five sets of sensitivity checks, with the calculation of overall impact where appropriate. Table A.1 provides results for our first four sensitivity checks. As described in the main text, this table shows that our main findings are not biased by our selection of comparison schools, and that PSCI does not predict student race or ethnicity (as expected). The last two columns in table A.1 provide results from our selection year analysis, which allows for the possibility that PSCI treatment may have begun in the identification year. In these models, we only show the level effect, as it is of primary interest because it shows the effect of PSCI identification on student outcomes. Although significant results in this identification year do not invalidate our main results, they would suggest that simply planning a turnaround reform, especially in the unique high-stakes, competitive context of PSCI, impacts student outcomes. We find that, for cohorts 1.0 and 2.0, there were no significant impacts of PSCI identification and planning on student achievement outcomes. In cohort 3.0, however, students in identified PSCI focus schools suffered a significant decrease in math and ELA achievement relative to near-selected schools.

Table A.1. 
ELA and Math Achievement for Students Enrolled in Focus Schools Relative to Near-selected Schools and PI3+ Schools, with Selection Year as First-Treatment Year
Near-selected w/2.0–4.0PI3+PlaceboSelection Year
CohortTreatment EffectELAMathELAMathBlackHispanicELAMath
1.0 Level −0.010 0.012 −0.012 0.017 0.009 −0.009 0.012 0.003 
  (0.013) (0.028) (0.011) (0.026) (0.009) (0.011) (0.015) (0.017) 
 Growth 0.003 0.019 0.010 0.028 0.008 −0.008   
  (0.010) (0.018) (0.009) (0.018) (0.007) (0.007)   
 Overall −0.003 0.049 0.009 0.073 0.025 −0.025   
 (as of 2010–11) (0.020) (0.056) (0.018) (0.055) (0.017) (0.021)   
 Level 0.070** 0.023 0.054* 0.005 −0.003 0.012 −0.037 −0.015 
  (0.023) (0.044) (0.022) (0.045) (0.022) (0.023) (0.042) (0.046) 
2.0 Growth 0.055** 0.035 0.065*** 0.034 −0.002 0.002   
  (0.018) (0.034) (0.016) (0.025) (0.014) (0.018)   
 Overall 0.126*** 0.057 0.119*** 0.038 −0.004 0.014   
 (as of 2011–12) (0.027) (0.048) (0.025) (0.043) (0.030) (0.040)   
3.0 Level −0.070* −0.116* −0.051* −0.104* 0.014 −0.045 −0.041* −0.108* 
  (0.028) (0.057) (0.020) (0.047) (0.020) (0.030) (0.017) (0.044) 
Near-selected w/2.0–4.0PI3+PlaceboSelection Year
CohortTreatment EffectELAMathELAMathBlackHispanicELAMath
1.0 Level −0.010 0.012 −0.012 0.017 0.009 −0.009 0.012 0.003 
  (0.013) (0.028) (0.011) (0.026) (0.009) (0.011) (0.015) (0.017) 
 Growth 0.003 0.019 0.010 0.028 0.008 −0.008   
  (0.010) (0.018) (0.009) (0.018) (0.007) (0.007)   
 Overall −0.003 0.049 0.009 0.073 0.025 −0.025   
 (as of 2010–11) (0.020) (0.056) (0.018) (0.055) (0.017) (0.021)   
 Level 0.070** 0.023 0.054* 0.005 −0.003 0.012 −0.037 −0.015 
  (0.023) (0.044) (0.022) (0.045) (0.022) (0.023) (0.042) (0.046) 
2.0 Growth 0.055** 0.035 0.065*** 0.034 −0.002 0.002   
  (0.018) (0.034) (0.016) (0.025) (0.014) (0.018)   
 Overall 0.126*** 0.057 0.119*** 0.038 −0.004 0.014   
 (as of 2011–12) (0.027) (0.048) (0.025) (0.043) (0.030) (0.040)   
3.0 Level −0.070* −0.116* −0.051* −0.104* 0.014 −0.045 −0.041* −0.108* 
  (0.028) (0.057) (0.020) (0.047) (0.020) (0.030) (0.017) (0.044) 

Notes: Table A.1 presents the level, growth, and/or overall treatment effects for four sensitivity checks including: (1) expanding our comparison group to all near-selected schools in PSCI (columns 1 and 2); (2) expanding our comparison group to all PI3+ schools in LAUSD (columns 3 and 4); (3) falsification tests for whether PSCI predicts student characteristics such as race/ethnicity (columns 5 and 6); and (4) estimating the level effect of treatment identification in PSCI on student outcomes (columns 7 and 8). Standard errors are clustered to the school level.

*p < 0.05; **p < 0.01; ***p < 0.001.

Table A.2. 
ELA and Math Achievement for Students Enrolled in Focus Schools Relative to Near-selected Schools while Controlling for SIG Schools (2010–11 SIG grantees for Cohort 1.0 schools and 2012–13 SIG grantees for Cohort 2.0 schools)
CohortTreatment EffectELAMath
 Focus (No SIG) Level −0.010 0.024 
  (0.016) (0.040) 
1.0 Focus (No SIG) Growth 0.000 0.020 
  (0.014) (0.023) 
 Focus (SIG) Level −0.011 0.049 
  (0.023) (0.033) 
 Focus (SIG) Growth −0.017 −0.095+ 
  (0.037) (0.057) 
2.0 Focus (No SIG) Level 0.099*** −0.048 
  (0.026) (0.036) 
 Focus (No SIG) Growth 0.037* −0.012 
  (0.018) (0.046) 
 Focus (SIG) Level −0.037 0.281*** 
  (0.036) (0.080) 
 Focus (SIG) Growth 0.099** 0.156* 
  (0.032) (0.072) 
CohortTreatment EffectELAMath
 Focus (No SIG) Level −0.010 0.024 
  (0.016) (0.040) 
1.0 Focus (No SIG) Growth 0.000 0.020 
  (0.014) (0.023) 
 Focus (SIG) Level −0.011 0.049 
  (0.023) (0.033) 
 Focus (SIG) Growth −0.017 −0.095+ 
  (0.037) (0.057) 
2.0 Focus (No SIG) Level 0.099*** −0.048 
  (0.026) (0.036) 
 Focus (No SIG) Growth 0.037* −0.012 
  (0.018) (0.046) 
 Focus (SIG) Level −0.037 0.281*** 
  (0.036) (0.080) 
 Focus (SIG) Growth 0.099** 0.156* 
  (0.032) (0.072) 

Notes: Table A.2 reports the level and growth effects for focus schools that were and were not awarded SIG grants. Given that SIG grants were awarded to schools in 2010–11 and 2012–13, we control for the first-wave of SIG awards in our analysis of cohort 1.0 schools and the second-wave of SIG awards in our analysis of cohort 2.0 schools. In cohort 1.0, six focus schools (with approximately 9,800 students) were awarded SIG grants and eight focus schools (with approximately 12,700 students) were not awarded SIG grants. Three near-selected schools (with approximately 6,500 students) were awarded SIG grants and 68 near-selected schools (with approximately 88,000 students) were not awarded SIG grants. In cohort 2.0, three focus schools (with approximately 2,000 students) were awarded SIG grants and two focus schools (with approximately 3,700 students) were not awarded SIG grants. One near-selected school (with approximately 950 students) was awarded a SIG grant and 22 near-selected schools (with approximately 23,000 students) were not awarded SIG grants. We do not include analysis for cohort 3.0 because there were no treated SIG schools in this cohort.

+p < 0.10, *p < 0.05, **p < 0.01, ***p < 0.001. Standard errors are clustered to the school level.

Table A.2 provides results from specifications that separately model trends for non-SIG PSCI schools and SIG PSCI schools. We note that the sample size for each of these subgroups is fairly small, especially the group of SIG-focus schools, so we consider these results as solely exploratory. In controlling for SIG, we find PSCI still had no significant effects on student outcomes for cohort 1.0 schools, and a significant and positive effect on ELA achievement for cohort 2.0 schools. In the case of cohort 1.0, we see SIG-treated focus schools actually performed worse than other focus schools in years 2 and 3, perhaps dragging down the effects of PSCI on student outcomes. In cohort 2.0, we find SIG-treated focus schools performed better than other focus schools, although the positive impact of PSCI on non-SIG focus schools remains.