Abstract

School choice researchers are often limited to comparing one type of choice with another (e.g., charter schools vs. traditional public schools). One area researchers have not examined is the effects of different school types within the same urban region. We fill this gap by analyzing longitudinal data for students (grades 3–8) in Indianapolis, using student fixed effects models to estimate the impacts of students switching from a traditional public school to a charter, magnet, Catholic, or other private school. We find that students experience no differences in their achievement gains after transferring from a traditional public school to a charter school. However, students switching to magnet schools experience modest annual losses of −0.09 standard deviation (SD) in mathematics and −0.11 SD in English Language Arts. Students switching to Catholic schools also experience annual losses of −0.18 SD in mathematics. These findings are robust to a series of alternative model specifications. Additionally, we find some variability in the mean school type impacts by students’ race/ethnicity, English language learner status, and number of years enrolled in a choice school. We discuss our results in the context of the variability of choice school effects across an entire urban area, something future research needs to examine.

1.  Introduction

For some time, parents, educators, and policy makers have relied on a variety of efforts to address the nation's education problems in urban settings in an attempt to close racial/ethnic and socioeconomic achievement gaps. One such set of efforts has to do with school choice, which offers a striking number of policies and programs across the United States, including school scholarships or vouchers; charter, magnet, and private schools; open enrollment; homeschooling; supplemental educational services; and school transfer options under the federal No Child Left Behind Act, which has been replaced with the Every Student Succeeds Act (for an overview see Berends et al. 2009; Berends 2014). As these policies and programs have multiplied—with mixed results about achievement effects—so too have the number of bitter controversies surrounding them (Henig 2008; Berends, Cannata, and Goldring 2011). This warrants additional research.

One area that has received little attention to date is research that simultaneously examines the effects of various school choice options on student achievement. For example, few studies have been able to compare the effects of charter, magnet, private, and traditional public schools at the same time. (Most compare one school type to another, such as charter and traditional public schools.1) In Indiana, however, because students in traditional public, charter, magnet, and private schools all take the same state assessment tests, we have a unique opportunity to examine achievement effects across students and school types using longitudinal student assessment data. Thus, we can address the following research questions:

  1. What is the impact of school type (charter, magnet, Catholic, private) on student achievement gains compared with traditional public schools?

  2. Do these impacts change over time after a student has enrolled in a new school?

  3. Are these impacts different for some demographic groups of students compared with others, having an effect on the racial/ethnic or socioeconomic achievement gaps?

  4. Are these impacts different for some academic groups of students (e.g., English language learners, special education students) compared with their peers?

In what follows, we describe the context of choice in the Indianapolis urban area and briefly review the existing research on different types of school effects on student achievement before outlining our data and methods, presenting results, and discussing the implications.

2.  Background and Previous Literature

School Choice in Indianapolis

There is a growing number of charter public schools in the Indianapolis urban area, many of which the Indianapolis mayor's office authorizes.2 (This is the only mayor's office in the nation that authorizes charter schools.) Currently, the city's Office of Education Innovation manages thirty-five charter schools and oversees their mission of providing high-quality education to Indianapolis families.

In addition to charter schools, there are nearly two dozen magnet schools in the Indianapolis urban area, primarily operated by Indianapolis Public Schools (IPS). Magnet schools have a specific focus, mission, and instructional design. The magnet schools in the Indianapolis urban area provide educational options that include Montessori, science and technology, Spanish immersion, health care, and environmental education foci.

In the private sector, the Indianapolis urban area has more than 60 private schools, the majority of which are Catholic and run by the Archdiocese of Indianapolis. Over the past few years, the Indiana Department of Education (IDOE) has implemented the Indiana Choice Scholarship Program, which since 2011 provides vouchers to low- and modest-income families so that their children can attend private schools of their choice.3 This program has greatly boosted private school attendance in the Indianapolis urban area, with nearly 5,000 students using vouchers.

Most studies compare one school type to another, but we are able to examine several school types simultaneously within the same area by analyzing comparable, longitudinal student achievement data. Overall, this makes Indianapolis an important site to examine effects of different school types on student outcomes within the broader context of school organization across sectors.

What Are the Effects of School Type on Student Achievement?

Before we describe our data and analyses, we review what we know about the effects of different school types—private and Catholic schools, charter schools, and magnet schools—to provide a context for examining school choice within Indianapolis. In general, whatever the school type, the effects on student achievement are quite mixed (some positive impacts, some negative, some null). Because the majority of these studies compare one school type with traditional public schools, additional research that examines multiple school types simultaneously is necessary, particularly for urban locales where a variety of school choice policies and programs have been implemented.

Private and Catholic School Effects

Many people argue that private schools, particularly Catholic schools, outperform public schools (Coleman, Hoffer, and Kilgore 1982; Coleman and Hoffer 1987; Chubb and Moe 1990). However, the actual performance effects and their implications are often the center of heated debate (see Lee and Bryk 1989). When comparing differences between school types, for instance, there is a considerable concern about selection bias. Some argue that students who attend Catholic schools vis-à-vis public schools are different, such as social background, motivation, values and beliefs, among other characteristics, of which some are measureable and some are difficult to measure or are unobserved (Goldberger and Cain 1982).

The research to date suggests there is sizable variability in the effects of Catholic schools on student achievement by level (e.g., high school vs. elementary). When examining the effects of attending a Catholic high school on mathematics achievement, researchers analyzing nationally representative cohorts of high school students from the 1980s to the present have found a large and positive Catholic school effect—ranging from one third to one half of a standard deviation (e.g., Coleman and Hoffer 1987; Hoffer 2009; Carbonaro and Covay 2010).

Despite the consistency of these results across numerous studies and cohorts of students, other researchers have questioned the magnitude and significance of these effects. Using an approach that addresses selection bias by accounting for the ratio of selection on observables to selection on unobservables, Altonji, Elder, and Taber (2005) did not find a significant mathematics achievement effect. The authors determined that most of the positive Catholic school effect could be attributed to either selection issues or to unobserved differences between Catholic and public high school students. There is also no clear picture of the effects of Catholic high schools on student achievement in English Language Arts (ELA) or other subjects.

To date, researchers have conducted fewer studies on the effectiveness of Catholic schools serving students in grades K–8. However, this body of research suggests that Catholic school effects at the elementary school level are either null or negative. For instance, analyzing the Early Childhood Longitudinal Study, Carbonaro (2006) found that kindergarten students in public and Catholic schools experienced similar achievement gains in mathematics and general knowledge, net of other characteristics, and Reardon, Cheadle, and Robinson (2009) showed that public school students between third and fifth grades outperformed their Catholic school peers in math but experienced similar gains in reading. Elder and Jepsen (2014) conducted a more recent analysis of the Early Childhood Longitudinal Study data using ordinary least squares (OLS) regression, propensity score matching, and the Altonji, Elder, and Taber (2005) approach. The authors found no evidence of a Catholic school effect on reading test scores for elementary and middle school students, and a sizeable negative, statistically significant effect on mathematics scores (2 to 10 percentile points)—suggesting that the Catholic school advantage existing in the raw data is a result of the selection of higher-ability students into Catholic schools.4

Charter School Effects

Unlike the Catholic school evidence base, which is largely composed of studies relying on estimation strategies that may not mitigate selection issues, contemporary research on charter schools has relied on stronger research designs to estimate the effectiveness of charter schools. To date, the research shows, on average, that there are some positive effects of charter schools on mathematics achievement and educational attainment but that there is also a great deal of heterogeneity of charter school effects (Betts and Tang 2014; Berends 2015).

A collection of studies that resemble randomized experiments with individual or groups of charter schools demonstrates positive effects of charter school attendance on academic achievement outcomes (Hoxby and Murarka 2008; Hoxby, Murarka, and Kang 2009; Abdulkadiroglu et al. 2011; Dobbie and Fryer 2011; Angrist et al. 2012). These studies specifically investigate oversubscribed (i.e., more students applied to enroll than seats available) charter schools that implement a lottery to determine student enrollment. Researchers use the results of these lotteries as a natural experiment to compare the outcomes of students winning the lottery and subsequently enrolling in charter schools with those of their peers who lose the lottery and enroll elsewhere (typically a traditional public school). Results for charter school students are moderate to large and positive across both math and reading/ELA achievement, ranging from 0.12 to 0.36 standard deviation (SD) per year in middle and high schools in and around Boston (Abdulkadiroglu et al. 2011; Angrist et al. 2012), to 0.20 SD per year for students attending the Promise Academy in the Harlem Children's Zone (Dobbie and Fryer 2011), to a more modest 0.06 to 0.09 SD per year in schools across New York City (Hoxby, Murarka, and Kang 2009).

Although these studies have strong internal validity designs, it is difficult to generalize the results to the broader context of charter schools across the United States. Another set of studies relies on broader samples of charter schools and uses experimental or quasi-experimental methods to estimate charter school effects on student achievement (for a review see Bifulco and Ladd 2006; Teasley 2009; Gleason et al. 2010; Imberman 2011; CREDO 2013; Tuttle et al. 2013; Betts and Tang 2014; Berends 2015). These studies find more mixed results (some positive, some neutral, some negative).

Magnet School Effects

A collection of studies on the effect of magnet schools on student achievement shows mixed results. These studies range from using nationally representative samples of magnet school students to quasi-experimental designs that longitudinally track the achievement of students who win and lose the lottery for admission to a magnet school.

For example, in the nationally representative National Educational Longitudinal Survey, Gamoran (1996) used both OLS and multilevel regression models and found robust results, suggesting that magnet schools are more effective than regular public schools in raising the achievement of students in reading and social studies (about 0.8 to 1.0 Item Response Theory scale points). An American Institutes for Research evaluation of the federal program that provides funding to magnet schools found that the achievement progress in magnet schools was no greater than in a comparison set of regular public schools when controls were introduced for demographic composition of the schools (Christenson et al. 2003). Unlike the Gamoran study, which examined nationally representative longitudinal student-level achievement, the American Institutes for Research study was limited to examining school-level aggregate data.

Bifulco (2012) examined the effects of two interdistrict magnet middle schools close to Hartford, Connecticut. Both schools served grades 6–8 and were created to promote racial and socioeconomic integration. Families applied for their child to attend the schools, which were oversubscribed. Each school held lotteries to determine admission, which allowed for a randomized research design. Although the purpose of Bifulco's paper was a within-study comparison of the experimental estimates on eighth-grade reading scores to nonexperimental estimates, he found that the standardized experimental impacts ranged between 0.20 and 0.23 SD, depending on whether covariates and pretreatment test scores were included in the estimation. These positive reading results of the magnet schools translate into treatment-group students gaining an extra year of achievement compared with control students.

Additional studies that examine the effects of magnets by relying on lotteries in Chicago and Nashville do not reveal robust positive results on achievement. For example, Cullen, Jacob, and Levitt (2006) found no differences in achievement or attendance rates between students who won the lottery and those who lost in a study of magnet and other open enrollment schools in Chicago. In Nashville, although Ballou, Goldring, and Liu (2006) found a positive magnet school effect on mathematics achievement when comparing lottery winners and losers, these effects disappeared when controls for student demographics and prior achievement were included in the models. This finding suggests that although there was random assignment in the lotteries, treatment and control groups differed with respect to student characteristics related to achievement. The authors conclude that the most likely explanation is differential patterns of attrition among the lottery winners and losers.

In summary, there is substantial variability in impacts of school types on student achievement gains when comparing private, Catholic, charter, and magnet schools, each in isolation of one another, with traditional public schools. Questions remain about whether one school type outperforms the others within the same urban area and on the same achievement measures, and whether the variability in achievement impacts is more meaningful between school types than within. It is to these issues that we turn with longitudinal data on Indianapolis students in different types of schools.

3.  Data and Methods

Data Description and Measures

We use five years (2008–09 school year through 2012–13) of longitudinal, student demographic and test score records from the IDOE. The records contain information about students in grades 3–8 attending traditional public, charter, and private schools who participate in the annual ELA and mathematics assessments of the Indiana Statewide Testing for Educational Progress Plus (ISTEP+) program.5

Indiana is unique because a large number of private K–8 schools participate in the ISTEP+ program (52 percent statewide; 75 percent of Indianapolis urban area private schools).6 Many private schools, including most Catholic schools, participated in ISTEP+ testing over the past decade as part of their accreditation process. Additional private schools began taking the ISTEP+ as a required part of their participation in the statewide school voucher program. All students in ISTEP+ participating schools take the test, regardless of whether or not they receive a voucher. The uniquely high rate of private school participation in annual statewide testing in Indiana allows us to make apples-to-apples comparisons of student-learning outcomes across public and private school types over time.

Our primary outcomes of interest are students’ annual ELA and math ISTEP+ test scores. We standardized each student's scores by the subject/grade/year mean and standard deviation of students statewide.7 The standardized measures allow us to draw comparisons, in SD units, between individuals in different types of schools. We also created year-to-year test score gains in SD units by taking the difference of each student's prior and current year standardized scores.

We use a number of student-level demographic and academic background characteristics reported in the annual IDOE data. These characteristics include each student's gender, race/ethnicity, grade level, free or reduced-price lunch (FRPL) recipient status, English proficiency status, and special education status. We recoded each of these variables into binary indicators.

The longitudinal records also contain information about the student's school of record, including the school name and National Center for Education Statistics (NCES) unique identification number (ID). Using the NCES ID, we linked the schools to the Common Core of Data (CCD) and the Private School Universe Survey (PSS) to augment and enhance the available school-level data.8 We used these data to create binary indicators of the school type (e.g., traditional public, charter, magnet, Catholic, and other private) of each student's annual enrollment. From these, we were able to identify students who switch between school types.

We also created two binary indicators to identify all students who switch schools between years, regardless of whether or not they change school type. The first indicates whether a student made a structural move due to normal grade progression; the second indicates whether a student made a nonstructural move (switching schools for any other reason) (for the significance of these types of moves see Schwartz, Stiefel, and Cordes 2017). Both variables indicate a switch only in the school year immediately after the switch took place (t) even though the switch takes place between years t and t − 1. Although we do not observe the underlying reason why students change schools, these two variables together help to identify our estimated choice school effects.

Sample Description

Our analytical sample comprises all Indianapolis urban area (Marion County) public and private schools participating in the ISTEP+ testing program. Marion is the most populous county in the state (i.e., has the largest school-age population) with the majority of residents living within the city of Indianapolis. The county is also substantially more racially and ethnically diverse than the rest of the state.9 The sizable, diverse population combined with the variety of public and private school types makes the Indianapolis urban area prime for investigating effects of different school types.

Nine large public school districts serve the Indianapolis urban area, including IPS. In total, 158 unique traditional public schools served students in grades 3–8 across the five years of this study (51 IPS). For public choice schools, these data include 27 charter schools and 21 magnet schools (18 of which were operated by IPS). On the private side, all 26 Catholic schools in Marion County participated in ISTEP+ testing and 18 other private schools (all with religious affiliation) also participated.10

Approximately 80 percent of students participating in ISTEP+ testing attended traditional public schools in Marion County in the 2008–09 school year. This percentage fell to just below 76 percent in the 2012–13 school year. A few factors influenced these changes. First, ten new charter schools opened during the five-year period (two closed) and the total number of students in charter schools doubled. A handful of new magnet schools opened and a few others closed. In the private sector, the implementation of the Indiana Choice Scholarship (school voucher) program in the 2011–12 school year reversed the general downward enrollment trend. Table 1 contains information about the number of students and schools by school type and year across the Indianapolis urban area.

Table 1.
Number of Students and Schools in the Indianapolis Urban Area by School Type and Year
PublicCharterMagnetCatholicOther Private
StudentsSchoolsStudentsSchoolsStudentsSchoolsStudentsSchoolsStudentsSchools
2008—09 59,558 153 3,291 17 4,701 17 5,579 28 1,267 14 
2009—10 60,901 148 4,052 19 4,452 15 5,468 28 1,132 14 
2010—11 61,604 147 4,934 22 4,289 15 5,232 26 1,037 13 
2011—12 59,798 142 6,801 24 5,187 18 5,399 26 1,774 15 
2012—13 60,324 146 8,246 25 3,636 15 5,525 26 2,033 15 
PublicCharterMagnetCatholicOther Private
StudentsSchoolsStudentsSchoolsStudentsSchoolsStudentsSchoolsStudentsSchools
2008—09 59,558 153 3,291 17 4,701 17 5,579 28 1,267 14 
2009—10 60,901 148 4,052 19 4,452 15 5,468 28 1,132 14 
2010—11 61,604 147 4,934 22 4,289 15 5,232 26 1,037 13 
2011—12 59,798 142 6,801 24 5,187 18 5,399 26 1,774 15 
2012—13 60,324 146 8,246 25 3,636 15 5,525 26 2,033 15 

Notes: Number of students and schools calculated from students in grades 3—8 and participating in the ISTEP+ testing program. These numbers reflect all students in each year, not just those included as part of our analytical sample.

When creating our analytical sample, we made several decisions along the way that excluded a number of students across all school types from our final analyses. We detail these decisions in Appendix B. Based on these decisions, our final analytical sample for investigating student achievement gains contains 166,903 observations for 56,757 unique students.

Estimation Strategy

Our goal in this paper is to assess the impacts on student achievement for those students attending a type of choice school (charter, magnet, Catholic, other private) compared with their traditional public school peers in the Indianapolis urban area. We would ideally randomly assign students in the Indianapolis urban area to either a charter, magnet, Catholic, other private, or traditional public school and estimate unbiased school effects by comparing the achievement outcomes of students in each type of school. We only observe the enrollment choices for each student (i.e., there is no random assignment), so we must turn to alternative methods to account for bias resulting from the selection of schools.

One approach we could use to mitigate selection bias is to use data from lotteries that determine which students may enroll in choice schools. This is a common approach for evaluating oversubscribed charter schools.11 IDOE does not provide information about charter school oversubscription or any lottery results. Further, Catholic and other private schools are not required to hold lotteries to determine enrollment.12

When lottery and other experimental data are not available, it can be difficult to estimate causal effects of choice school attendance on student outcomes. Enrollment in a choice school depends on the active decisions made by parents and their children—choices that typically depend on student background, parental preferences, and motivation as well as available opportunities in traditional public schools. As a result, a simple comparison of student outcomes across school types could provide a biased estimate of the relative effectiveness of each school type. For example, if charter schools attract students with high academic ability or motivation, then charter school performance might appear better than that of traditional public schools because of differences in the academic composition of students attending the two types of schools.

With longitudinal student records, we propose a quasi-experimental approach to isolate the contributions of school types on student outcomes, all while controlling for the preparation and background of students attending each type of school. Researchers have used student fixed effects models in previous charter school evaluations (Buddin and Zimmer 2005; Betts and Hill 2006; Bifulco and Ladd 2006; Sass 2006; Hanushek et al. 2007; Imberman 2011; Nicotera, Mendiburo, and Berends 2011). Our longitudinal data are particularly well suited for expanding upon these previously used student fixed effects models to compare the test score gains of students switching to charter, magnet, Catholic, and other private schools to their test score gains in traditional public schools.

We develop our preferred student fixed effects model from the general form of an educational production function, where student outcomes are an additive function of the student's school and individual inputs over time.13 In building our preferred model, we begin with an achievement “levels” model, which we write as
formula
1

This OLS regression model estimates the mathematics or ELA achievement level () for each student i in grade g and year t. The model compares the average differences in achievement levels between students in charter (), magnet (), Catholic (), and other private schools () with traditional public school students. We control for a vector of fixed and time-varying individual student characteristics (), including indicators of student race/ethnicity, sex, FRPL status, English language learner (ELL) status, special education status, and whether or not the student made a structural or nonstructural school change between years t and t − 1 (indicated only in year t) (for recent research on these moves, see Schwartz, Stiefel, and Cordes 2017). Grade-by-year effects () account for systematic differences across exams and over time. The term represents school cluster–robust standard errors, used in all models.14

Because there are likely other observed and unobserved factors that may influence test scores and choice of school, we must adjust the model to account for these potential biases. We can improve the first model by using an achievement “gains” model, written as
formula
2

Here, the outcome represents the yearly achievement gain for each student—or the difference in a given student's test scores from his previous grade (g − 1) and year (y − 1) to his current grade and year.

In model 2, we estimate the average difference in mathematics or ELA achievement gains made by students in charter, magnet, Catholic, and other private schools with gains made by students in traditional public schools. This achievement gains model improves upon the levels model because it adjusts estimates of school type effects for each student's prior achievement.15 Theoretically, differencing out a student's prior achievement will capture observed aspects of a student's previous educational experiences that might influence their subsequent achievement and selection of school type. Also, the year-to-year gains mostly account for certain unobserved, time-varying differences between students that might be directly reflected in achievement outcomes, such as motivation.

Although the gains model accounts for the impacts of prior achievement, we have not accounted for any unobserved student characteristics that may influence a student's achievement gains or selection of school type. We can adjust for time-invariant, unobserved differences between students by including a student-specific fixed effect () to model 2:
formula
3

In using model 3, we retain the advantages of using an achievement gains model and now adjust for any unobserved, time-invariant differences between students attending different types of choice schools and traditional public schools. We build upon this preferred model in order to investigate differences in school type effects across subgroups of students (e.g., by race/ethnicity, years in choice school). We estimate separate models that include interaction terms between the subgroup indicators and the school type variables for each set of subgroup differences.16

The fixed-effects model 3 mitigates the degree of self-selection bias that occurs from a student's choice of schools. However, this model relies upon students who are “switchers” from a traditional public to a choice school. The effect of each of the choice school types is identified by comparing the test score gains of students observed at least once in a choice school (achievement gain from choice to public) and at least twice in a traditional public school (gain across two years within public). The estimates of the average effects of choice school types from the fixed effects model would still be biased upward (or downward) if students who are switching from traditional public schools experience substantial dips (or gains) in the year immediately prior to switching to a choice school. This pretreatment phenomenon, known in the job-training literature as “Ashenfelter's Dip” (Ashenfelter 1978), suggests that a substantial drop in student performance may be a signal to parents to have their child change school types.

In order to account for the possibility of pre-existing trends, we first describe the average achievement profiles of the switching students in the year prior to switching from a traditional public school. Then, we also follow the approaches of Hanushek et al. (2007) and Imberman (2011) and conduct a robustness check that uses interrupted panel estimates. We reestimate the fixed effects model after removing observations for each student in the year prior to switching school types. We compare these results with the baseline fixed effects model (see Appendix table A.3), and we do not find substantial changes in effects. This indicates that our results do not suffer from bias resulting from a dip (or gain) in achievement trends in the one year immediately preceding a student's switch of schools.17

Overall, the fixed effects model has three primary advantages for examining charter, magnet, Catholic, and other private school effectiveness. First, we are able to compare the impacts of those students switching to any choice type simultaneously with traditional public school students and one another. This is a unique feature of this study, particularly with regard to the existing school choice literature that uses a similar methodological approach. Second, the fixed effects approach provides a broader assessment of how school performance varies across each school type than is possible with lottery data for a subset of oversubscribed schools, which may be different from undersubscribed schools. Using this approach, we are able to compare the performance of choice school students across the entire Indianapolis urban area as opposed to only lottery schools. Third, the fixed effects model uses the longitudinal nature of these data to “difference out” the unobserved variation between students to less biased estimates than an OLS or random effects model.

The fixed effects model has important limitations, however. First, while the fixed effects model does account for unobserved, time-invariant heterogeneity, other time-varying characteristics that simultaneously explain achievement gains and choice of school type could bias our results. As we describe above, we test for the presence of pre-transfer trends and account for time-varying heterogeneity with achievement gains, but cannot completely rule out other unobserved, time-varying factors. Second, the fixed effects gains model uses at least three successive years of test score data (two periods of test score gains) to isolate a test score trend, which excludes many students from the analysis. Thus, we focus on students who have ISTEP+ achievement data from at least three consecutive years, constraining the majority of our results to students in grades 5 to 8. Third, the model identifies school effects from differences in student outcomes for students switching school types (e.g., public to charter). This focus on school “switchers” may provide an unrepresentative indication of the impact of an entire school type, since switchers might be qualitatively different from nonswitchers in ways that influence the effect that private or charter schooling has on them. To this end, we also include results from both the OLS levels and gains models for comparison with our fixed effects results.

The OLS models capture achievement differences between all students enrolled in choice schools relative to their traditional public school peers, not just students switching school types. However, the OLS results do not represent true baseline measures of school effects. Any differences in impacts between the OLS and fixed effects models may be due to either the change in the composition of students (switchers may be different from nonswitchers) or differences in the identification strategy (the OLS model identifies choice effects by comparing all choice and traditional public school students; the fixed effects model identifies choice effects only for switchers when they are enrolled in choice schools versus traditional public schools).

We also include weights in an additional sensitivity test. Schools are either underrepresented or overrepresented in the fixed effects model, depending upon the number of students switching to each school. Therefore, we calculate and include in the fixed effects model inverse probability weights by the number of students switching to each choice school. With these weights, we estimate school type effects in a manner more representative of each individual school.18

4.  Results

We first compare the achievement and demographic profiles of students switching to various school types with their peers remaining in traditional public schools. Overall, we observe that nearly 10 percent of all traditional public school students switch to a choice school between grades 3 and 8. Nearly half of these total switches from a public to a choice school occur when students make a structural change prior to beginning seventh grade, and the other half are nonstructural changes at some other point in time.

Some noteworthy descriptive statistics in table 2 include the percentages of African American, Latino, and multiracial students who move to a different school type. These racial and ethnic minorities are more likely to switch from traditional public schools. For instance, of the total number of students switching to charter schools, 62.3 percent were African American, 7.2 percent were Latino, and 7.6 percent were multiracial. In magnet schools, 57.3 percent of switchers were African American and 16.2 percent were Latino. The switchers to charter and magnet schools largely reflect the racial and ethnic composition of the schools into which they move. On the other hand, although 25.5 and 20.0 percent of switchers in Catholic schools were African American and Latino, respectively, these numbers reflect substantially greater proportions of racial and ethnic minorities than the overall Catholic school population. Outside of racial/ethnic background, 24.1 percent of the switchers in Catholic schools are ELLs, and 12.9 percent are special education students. A greater proportion of switchers to charter schools (18.1 percent) are special education students, compared with the proportion of students always enrolled in charter schools (14.8 percent).

Table 2.
Demographic and Academic Characteristics of Analytical Sample
Remain in Same School TypeTransfer from Public School to
PublicCharterMagnetCatholicOther PrivateCharterMagnetCatholicOther Private
Students 40,455 2,428 1,766 4,497 528 1,375 2,527 373 136 
Schools 158 27 21 26 18 26 20 26 14 
% Female 49.7 53.4 52.2 48.5 48.1 50.8 52.9 49.6 58.4 
% Black 32.8 53.5 54.7 5.3 2.2 62.3 57.3 25.5 18.1 
% Latino 14.9 10.6 12.2 5.3 2.2 7.2 16.2 20.0 10.1 
% Other race/ethnicity 9.2 9.2 6.5 6.1 3.9 7.6 7.2 9.9 8.0 
% Free/reduced-price lunch 72.3 84.9 81.0 16.4 19.5 89.1 98.5 64.3 69.9 
% English language learners 14.2 8.7 9.6 6.2 11.4 7.0 13.3 24.1 14.7 
% Special education 13.8 14.8 21.4 9.6 8.9 18.1 19.8 12.9 11.8 
% Ever retained 2.6 7.8 7.1 0.7 1.1 10.1 7.6 4.8 5.2 
Mean math score −0.09 −0.33 −0.41 0.41 0.26 −0.46 −0.53 −0.15 −0.15 
Mean ELA score −0.17 −0.26 −0.35 0.52 0.35 −0.47 −0.56 −0.09 −0.12 
Mean math gain −0.01 0.03 −0.03 −0.01 −0.03 −0.02 −0.10 −0.08 −0.05 
Mean ELA gain −0.01 0.02 −0.01 0.03 −0.02 −0.03 −0.06 −0.01 −0.06 
Remain in Same School TypeTransfer from Public School to
PublicCharterMagnetCatholicOther PrivateCharterMagnetCatholicOther Private
Students 40,455 2,428 1,766 4,497 528 1,375 2,527 373 136 
Schools 158 27 21 26 18 26 20 26 14 
% Female 49.7 53.4 52.2 48.5 48.1 50.8 52.9 49.6 58.4 
% Black 32.8 53.5 54.7 5.3 2.2 62.3 57.3 25.5 18.1 
% Latino 14.9 10.6 12.2 5.3 2.2 7.2 16.2 20.0 10.1 
% Other race/ethnicity 9.2 9.2 6.5 6.1 3.9 7.6 7.2 9.9 8.0 
% Free/reduced-price lunch 72.3 84.9 81.0 16.4 19.5 89.1 98.5 64.3 69.9 
% English language learners 14.2 8.7 9.6 6.2 11.4 7.0 13.3 24.1 14.7 
% Special education 13.8 14.8 21.4 9.6 8.9 18.1 19.8 12.9 11.8 
% Ever retained 2.6 7.8 7.1 0.7 1.1 10.1 7.6 4.8 5.2 
Mean math score −0.09 −0.33 −0.41 0.41 0.26 −0.46 −0.53 −0.15 −0.15 
Mean ELA score −0.17 −0.26 −0.35 0.52 0.35 −0.47 −0.56 −0.09 −0.12 
Mean math gain −0.01 0.03 −0.03 −0.01 −0.03 −0.02 −0.10 −0.08 −0.05 
Mean ELA gain −0.01 0.02 −0.01 0.03 −0.02 −0.03 −0.06 −0.01 −0.06 

Notes: We report ISTEP+ math and ELA scores in standard deviation units, standardized relative to Indiana statewide average. We calculated gains in ISTEP+ math and ELA scores in standard deviation units by taking the difference of each student's test score and previous-year test score. This table includes only those observations of students who transfer from a traditional public to a choice school and have at least three valid test score measures (two periods of test score gains), which are necessary for identification in our preferred model. ELA = English language arts.

Descriptive results suggest that students switching to charter schools in the Indianapolis urban area performed, on average, 0.47 and 0.46 SD lower than the Indiana statewide mean in math and ELA, respectively (see bottom panel of table 2). Magnet school switchers scored an average of 0.53 and 0.56 SD lower in mathematics and ELA, respectively, compared with the state average. Collectively, students switching to charter and magnet schools are lower-achieving than students remaining in traditional public schools by 0.3–0.4 SD, and have lower scores than peers in their new charter and magnet schools by 0.1–0.2 SD, on average.

Students switching to Catholic and other private schools scored at approximately the same level as their peers remaining in traditional public schools and slightly below the overall Indiana mean. However, students switching to private schools achieved at substantially lower levels than peers in their new Catholic or other private school. Those students switching to Catholic schools scored an average of 0.56 SD and 0.61 SD lower in math and ELA, respectively, and students switching to other private schools scored an average of 0.41 SD and 0.47 SD lower. The average annual achievement gains of students who switch from a public school to another school type were small and negative. Generally, the average gains across school type and subject area ranged from −0.10 to −0.01 SD per year.

In table 3 we investigate further the achievement of students switching school types in the years immediately before and after switching schools. Overall, we find that lower-achieving public school students are the ones transferring to charter and magnet schools, and average to higher-achieving public school students are the ones who transfer to Catholic and other private schools. We see that in the year before making the transfer, students switching to charter schools have low mean mathematics (−0.48 SD) and ELA (−0.50 SD) achievement. These scores remain at the same low levels in the first year after transferring from a public school. Similarly, before students transferred to magnet schools, they had low average mathematics (−0.47 SD) and ELA (−0.51 SD) achievement, and their average achievement levels dipped further (−0.65 SD in math, −0.67 in ELA) during their first year after transferring from a traditional public school.

Table 3.
Academic Performance in Year Before and After Changing Schools
CharterMagnetCatholicOther Private
% of students returning to public school 18.4 19.3 4.3 3.7 
Year Before Transferring from Public School 
Mean math score −0.48 −0.47 −0.04 −0.14 
Mean ELA score −0.50 −0.51 −0.11 −0.10 
Mean math gain −0.03 −0.10 0.01 −0.06 
Mean ELA gain −0.05 −0.05 −0.03 0.00 
Year After Transferring from Public School 
Mean math score −0.49 −0.65 −0.23 −0.29 
Mean ELA score −0.48 −0.67 −0.09 −0.14 
Mean math gain −0.01 −0.17 −0.18 −0.16 
Mean ELA gain 0.02 −0.14 0.01 −0.09 
CharterMagnetCatholicOther Private
% of students returning to public school 18.4 19.3 4.3 3.7 
Year Before Transferring from Public School 
Mean math score −0.48 −0.47 −0.04 −0.14 
Mean ELA score −0.50 −0.51 −0.11 −0.10 
Mean math gain −0.03 −0.10 0.01 −0.06 
Mean ELA gain −0.05 −0.05 −0.03 0.00 
Year After Transferring from Public School 
Mean math score −0.49 −0.65 −0.23 −0.29 
Mean ELA score −0.48 −0.67 −0.09 −0.14 
Mean math gain −0.01 −0.17 −0.18 −0.16 
Mean ELA gain 0.02 −0.14 0.01 −0.09 

Notes: We report ISTEP+ Math and ELA scores in standard deviation units, standardized relative to Indiana statewide average. We calculated gains in ISTEP+ Math and ELA scores in standard deviation units by taking the difference of each student's test score and previous year test score. This table includes only those observations of students who transfer from a traditional public to a choice school and have at least three valid test score measures (two periods of test score gains), which are necessary for identification in our preferred model. ELA = English language arts.

By contrast, students transferring to Catholic and other private schools tended to be scoring at the same average level as their traditional public school peers but below the statewide average. Before the transfer, students who moved to Catholic and other private schools had mean mathematics scores of −0.04 and −0.14 SD, respectively, and average scores of around −0.10 SD in ELA. In the year after transferring, average test scores in mathematics dropped to −0.23 SD in Catholic schools and −0.29 SD in other private schools. ELA scores remained approximately the same for students transferring to either private school type. Across all school types, we find that a much larger proportion of students transferring to charter (18.4 percent) and magnet schools (19.3 percent) return to a traditional public school at some point when observed in these data than do students transferring to Catholic (4.3 percent) and other private schools (3.7 percent).

Table 4 presents results from our achievement models for mathematics and ELA scores. We estimate three models for each subject, portraying the average school type effects based on: (1) an OLS model that examines achievement levels by school type, controlling for other relevant characteristics, (2) an OLS model estimating effects on test score gains, and (3) the preferred fixed effects model estimating achievement gains for those students who switch from a traditional public school to another school type. We focus on results from our preferred model.

Table 4.
Estimated Effects of Attending Choice Schools on Student Achievement
MathematicsELA
OLS Levels (1)OLS Gains (2)FE Gains (3)OLS Levels (1)OLS Gains (2)FE Gains (3)
Charter −0.09 0.01 −0.01a 0.00 0.01 0.04b 
 (0.07) (0.02) (0.05) (0.05) (0.01) (0.03) 
Magnet −0.14 −0.07* −0.09 −0.07 −0.04 −0.11*bc 
 (0.08) (0.03) (0.04) (0.09) (0.03) (0.05) 
Catholic 0.03 −0.02 −0.18***a 0.22*** 0.03* 0.00c 
 (0.06) (0.01) (0.04) (0.04) (0.01) (0.03) 
Other private −0.15 −0.04 −0.16 0.02 −0.03 −0.10 
 (0.10) (0.02) (0.08) (0.10) (0.02) (0.08) 
Structural change −0.03 −0.10*** −0.11*** −0.06* −0.07*** −0.07** 
 (0.03) (0.02) (0.03) (0.03) (0.02) (0.02) 
Nonstructural change −0.17*** −0.01 0.02 −0.17*** −0.01 0.01 
 (0.02) (0.01) (0.02) (0.02) (0.01) (0.01) 
Student demographic controls 
Student academic controls 
Grade–year fixed effects 
Student fixed effects 
Total observations 157,459 157,231 157,231 156,975 156,086 156,086 
Total students 54,044 54,011 54,011 53,998 53,920 53,920 
MathematicsELA
OLS Levels (1)OLS Gains (2)FE Gains (3)OLS Levels (1)OLS Gains (2)FE Gains (3)
Charter −0.09 0.01 −0.01a 0.00 0.01 0.04b 
 (0.07) (0.02) (0.05) (0.05) (0.01) (0.03) 
Magnet −0.14 −0.07* −0.09 −0.07 −0.04 −0.11*bc 
 (0.08) (0.03) (0.04) (0.09) (0.03) (0.05) 
Catholic 0.03 −0.02 −0.18***a 0.22*** 0.03* 0.00c 
 (0.06) (0.01) (0.04) (0.04) (0.01) (0.03) 
Other private −0.15 −0.04 −0.16 0.02 −0.03 −0.10 
 (0.10) (0.02) (0.08) (0.10) (0.02) (0.08) 
Structural change −0.03 −0.10*** −0.11*** −0.06* −0.07*** −0.07** 
 (0.03) (0.02) (0.03) (0.03) (0.02) (0.02) 
Nonstructural change −0.17*** −0.01 0.02 −0.17*** −0.01 0.01 
 (0.02) (0.01) (0.02) (0.02) (0.01) (0.01) 
Student demographic controls 
Student academic controls 
Grade–year fixed effects 
Student fixed effects 
Total observations 157,459 157,231 157,231 156,975 156,086 156,086 
Total students 54,044 54,011 54,011 53,998 53,920 53,920 

Notes: Dependent variables are in standard deviation units, relative to the Indiana statewide average. Student demographic controls include student sex and race/ethnicity. Student academic controls include free-reduced-price lunch status, English language learner status, special education status, whether or not the student was retained in the same grade from the prior year, and whether or not a student made a structural or nonstructural change of schools in a given year. Robust standard errors, adjusted for the clustering of students within schools, are in parentheses.

a–cSuperscript letters next to estimates represent statistically significant differences in achievement gains at α = 0.05 between groups indicated with the same letter. ELA = English language arts; FE = fixed effects; OLS = ordinary least squares.

p < 0.10; *p < 0.05; **p < 0.01; ***p < 0.001.

The overall effects of school type on mathematics and ELA, as shown in table 4, reveal that there is substantial variability across school types. For instance, Catholic school switchers in the Indianapolis urban area experience moderate and statistically significant average annual losses in mathematics (−0.18 SD, p < 0.001) compared with the gains they experienced while attending traditional public schools. However, we found null effects for the math achievement levels and gains of all Catholic school students. We estimate zero annual gains for Catholic school switchers in ELA, despite all Catholic school students having positive and statistically significant ELA achievement levels (0.22 SD, p < 0.001) and gains (0.03 SD, p = 0.030) from the OLS models. We find negative but statistically insignificant average annual gains in mathematics (−0.16 SD) and ELA (−0.10 SD) for those students switching to other private schools.19

Students who switch to charter schools do not experience significant achievement gains in mathematics or ELA. The fixed effects charter estimate in mathematics is nearly zero (−0.01 SD) and in ELA is positive (0.04 SD), but neither estimate is statistically significant. Similarly, the charter effects on mathematics and reading in the OLS-levels and gains models were small and statistically insignificant. The overall effects on achievement for magnet school switchers reveal modest losses in mathematics (−0.09 SD, p = 0.072) and statistically significant losses in ELA (−0.11 SD, p = 0.019).

In addition to estimating the effects of switching to each school type vis-à-vis public schools, we also compare school types to one another. We use a post-estimation test to determine whether any of the mean school type effects differ significantly from one another within each subject. We find three incidences of significant differences between school types. Students switching to Catholic schools have lower annual average gains in mathematics than those switching to charter schools (p = 0.004). Students switching to magnet schools have lower average gains in ELA than charter (p = 0.006) and Catholic (p = 0.043) school switchers.20

We subjected the mean choice school type estimates to six additional specifications of the student fixed effects models to check the robustness of our findings. Generally, we find similar results as our preferred fixed effects model when we: (1) include all switchers, including those switching between choice schools and making multiple switches; (2) use achievement levels as the outcome in the fixed effects model; (3) remove observations from the year prior to switching school types in an interrupted panel; (4) constrain the sample to include only those students making structural school changes; (5) change the outcome to the student's current year test score with the prior year test score as a predictor; or (6) include in the fixed effects model inverse probability weights by the number of students switching to each choice school. We find robust evidence that students switching to magnet schools experience statistically significant achievement losses ranging from −0.09 to −0.17 SD per year in mathematics and losses of −0.08 to −0.15 SD in ELA. Similarly, students switching to Catholic schools experience significant annual losses between −0.07 and −0.20 SD in mathematics.21 We display the full results of these robustness checks in Appendix table A.3.

Because the amount of time that students spend in a new school may affect their achievement gains, table 5 examines the effects of switching from a traditional public school to another school type by the year that students are enrolled in the new type (i.e., first year, second year, or third year enrolled). For students who switch to charter schools, the estimates show that the effects become increasingly negative over time, but none of the charter estimates is statistically significant. The effects of switching to a magnet school on mathematics and ELA are negative in the first year (−0.10 SD in math [not significant] and −0.13 SD in ELA, p = 0.006). These effects become less negative and statistically insignificant by year two.

Table 5.
Estimated Effects of Attending Choice Schools on Student Achievement Gains by Year
Mathematics
1st Year2nd Year3rd Year
Charter 0.00 −0.05 −0.08 
 (0.06) (0.06) (0.08) 
Magnet −0.10 −0.04 0.03 
 (0.06) (0.05) (0.12) 
Catholic −0.19*** −0.19** 0.02 
 (0.05) (0.06) (0.13) 
Other private −0.20 0.01 — 
 (0.10) (0.12) — 
 ELA 
 1st Year 2nd Year 3rd Year 
Charter 0.06 −0.04 −0.06 
 (0.04) (0.04) (0.07) 
Magnet −0.13** 0.01 −0.04 
 (0.05) (0.05) (0.11) 
Catholic 0.02 −0.04 −0.04 
 (0.04) (0.05) (0.12) 
Other private −0.10 −0.07 — 
 (0.10) (0.16) — 
Mathematics
1st Year2nd Year3rd Year
Charter 0.00 −0.05 −0.08 
 (0.06) (0.06) (0.08) 
Magnet −0.10 −0.04 0.03 
 (0.06) (0.05) (0.12) 
Catholic −0.19*** −0.19** 0.02 
 (0.05) (0.06) (0.13) 
Other private −0.20 0.01 — 
 (0.10) (0.12) — 
 ELA 
 1st Year 2nd Year 3rd Year 
Charter 0.06 −0.04 −0.06 
 (0.04) (0.04) (0.07) 
Magnet −0.13** 0.01 −0.04 
 (0.05) (0.05) (0.11) 
Catholic 0.02 −0.04 −0.04 
 (0.04) (0.05) (0.12) 
Other private −0.10 −0.07 — 
 (0.10) (0.16) — 

Notes: Dependent variables are in standard deviation units, relative to the Indiana statewide average. We estimated the yearly effects by adapting our preferred student fixed effects model 3 to include interactions between year of enrollment in choice school with the choice school type indicators. We have three years of post-switch data for only six students switching to other private schools, so we do not report these results. Robust standard errors, adjusted for the clustering of students within schools, are in parentheses. ELA = English language arts.

p < 0.10; **p < 0.01; ***p < 0.001.

Similar to the average gains, students who switch to a Catholic school experience statistically significant achievement losses in mathematics in the first and second years (−0.19 SD). However, we find less-pronounced negative effects in the third year in a Catholic school (0.01 SD). There is no significant variation in the effects on ELA gains by the amount of time students have spent in a Catholic school. For students switching to other private schools, they experience similar but statistically insignificant losses in mathematics during the first year (−0.20 SD) as their Catholic school peers. The achievement losses are near zero in subsequent years of enrollment.

In addition to our research question that examines the achievement impacts of switching from a traditional public school to a new school type, we also examine the question about whether these impacts are greater for some groups of students compared with others, thus having an effect on racial/ethnic achievement or socioeconomic gaps. Table 6 presents results from the models that examine the impact of attending different school types by students’ race/ethnicity and FRPL status.22 Generally, across subjects, there is some variation in effects of switching to another school type by race/ethnicity or FPRL status.

Table 6.
Estimated Effects of Attending Choice Schools on Student Achievement Gains by Demographics
Mathematics
WhiteBlackLatinoOther RaceFree/Red. LunchPaid Lunch
Charter −0.20**ab 0.08acfgh −0.13c −0.02b −0.01q −0.01 
 (0.07) (0.04) (0.08) (0.09) (0.05) (0.06) 
Magnet −0.11 −0.07*fi −0.13* −0.12 −0.11* −0.07 
 (0.09) (0.04) (0.05) (0.08) (0.05) (0.05) 
Catholic −0.18*** −0.12gj −0.25*** −0.15 −0.21***q −0.15*** 
 (0.04) (0.07) (0.07) (0.09) (0.04) (0.04) 
Other private −0.06d −0.47***dehij −0.35 −0.04e −0.09 −0.19* 
 (0.09) (0.11) (0.22) (0.16) (0.12) (0.08) 
 ELA 
 White Black Latino Other Race Free/Red. Lunch Paid Lunch 
Charter −0.15**kln 0.11***kmp −0.07m 0.11l 0.04r 0.06s 
 (0.05) (0.03) (0.07) (0.08) (0.03) (0.05) 
Magnet −0.14o −0.09*p −0.14*** −0.09 −0.11*r −0.12s 
 (0.07) (0.05) (0.04) (0.07) (0.05) (0.07) 
Catholic 0.04no 0.01 −0.05 −0.08 −0.01 0.01 
 (0.05) (0.06) (0.07) (0.09) (0.04) (0.04) 
Other private −0.10 0.07 −0.27 −0.22 −0.07 −0.12 
 (0.09) (0.12) (0.28) (0.22) (0.08) (0.12) 
Mathematics
WhiteBlackLatinoOther RaceFree/Red. LunchPaid Lunch
Charter −0.20**ab 0.08acfgh −0.13c −0.02b −0.01q −0.01 
 (0.07) (0.04) (0.08) (0.09) (0.05) (0.06) 
Magnet −0.11 −0.07*fi −0.13* −0.12 −0.11* −0.07 
 (0.09) (0.04) (0.05) (0.08) (0.05) (0.05) 
Catholic −0.18*** −0.12gj −0.25*** −0.15 −0.21***q −0.15*** 
 (0.04) (0.07) (0.07) (0.09) (0.04) (0.04) 
Other private −0.06d −0.47***dehij −0.35 −0.04e −0.09 −0.19* 
 (0.09) (0.11) (0.22) (0.16) (0.12) (0.08) 
 ELA 
 White Black Latino Other Race Free/Red. Lunch Paid Lunch 
Charter −0.15**kln 0.11***kmp −0.07m 0.11l 0.04r 0.06s 
 (0.05) (0.03) (0.07) (0.08) (0.03) (0.05) 
Magnet −0.14o −0.09*p −0.14*** −0.09 −0.11*r −0.12s 
 (0.07) (0.05) (0.04) (0.07) (0.05) (0.07) 
Catholic 0.04no 0.01 −0.05 −0.08 −0.01 0.01 
 (0.05) (0.06) (0.07) (0.09) (0.04) (0.04) 
Other private −0.10 0.07 −0.27 −0.22 −0.07 −0.12 
 (0.09) (0.12) (0.28) (0.22) (0.08) (0.12) 

Notes: Dependent variables are in standard deviation units, relative to the Indiana statewide average. We estimated the differential race/ethnicity effects by adapting our preferred student fixed effects model 3 to include interactions between race/ethnicity with the choice school type indicators. Robust standard errors, adjusted for the clustering of students within schools, are in parentheses. ELA = English language arts.

a–sSuperscript letters next to estimates represent statistically significant differences in achievement gains at α = 0.05 between groups indicated with the same letter.

p < 0.10, *p < 0.05, **p < 0.01, ***p < 0.001.

It appears that black students experience positive gains after switching from a traditional public school to a charter school. In charter schools, black students experience annual gains of approximately 0.11 SD in ELA (p < 0.001) and 0.08 SD in mathematics (p = 0.065). However, black students who switch to other school types (magnet, Catholic, and other private) experience statistically significant losses in mathematics. The ELA gains for black students switching to magnet schools are also negative (−0.09 SD, p = 0.044), with null ELA gains for black students who switch to either type of private school.23 We find the impacts on math achievement in charter schools are statistically significantly different from magnet schools in both subjects and different from Catholic and other private schools in mathematics (p < 0.05).

Latino students who switch from a traditional public school experience achievement losses across both subjects and across all sectors. For example, Latino switchers to Catholic schools experience a −0.25 SD loss in mathematics achievement when compared with the gains they experienced in traditional public schools (p = 0.001). Similarly, Latino students who switch to magnet schools have a −0.13 SD loss in mathematics (p = 0.015) and a −0.14 SD loss in ELA (p < 0.001) compared with the gains they experienced in traditional public schools. Latino students who switch to a charter school experience a 0.13 SD loss in mathematics, but this impact is not statistically significant at the 5 percent level. We find no evidence of significant achievement differences across school types for Latino students switching from public schools.

White students who switch to a Catholic school have significant losses in mathematics achievement (−0.18 SD, p < 0.001); there is no significant effect in ELA. White students also experience statistically significant losses in both subjects in charter schools (−0.20 SD in math, p = 0.003; −0.15 SD in ELA, p = 0.005). The annual ELA achievement gains made by white students switching to Catholic schools are significantly higher than white students switching to either charter or magnet schools.

Within school types, there are a number of statistically significant achievement differences by race/ethnicity. In charter schools, black students switching from traditional public schools have significantly higher average annual achievement gains in both mathematics and ELA than either their white or Latino peers. In contrast, black students switching to other private schools have much lower average annual gains than their white or other racial/ethnic peers.

Overall, we observe less variability in effects by socioeconomic status (proxied by receiving FRPL) than we do by race/ethnicity. In mathematics, we find similar achievement losses for students switching to all four school types and receiving FRPL as their paid-lunch peers. In magnet and Catholic schools, both groups of students experience losses that are significant, but not statistically distinguishable within each school type. However, we do find that across school types, charter school switchers receiving FRPL experience smaller annual achievement losses than their Catholic school peers who also receive FRPL. In general, the annual ELA losses and gains by socioeconomic status are more modest, with only magnet school students receiving FRPL experiencing statistically significant losses. These magnet school students have gains that are statistically different from their charter school peers.

In table 7, we report results on how annual achievement gains differ between academic groups and across school types based on two characteristics of students: whether or not students are either ELL students or special education students. English language learners who attend magnet (−0.16 SD), Catholic (−0.29 SD), and other private schools (−0.46 SD) experience statistically significant annual losses in mathematics. Non-ELL students also have significant losses in mathematics in magnet and Catholic schools. Only those ELL students attending other private schools score significantly lower than their non-ELL peers. These students also score much lower than ELL students switching to charter or magnet schools. The impacts on ELA gains for ELL students in Catholic schools are positive but statistically insignificant, and the ELA gains for ELL students switching to the other three types of schools mirror the mathematics gains. Again, ELL students switching to other private schools score significantly lower than their ELL peers switching to all other types of schools.

Table 7.
Estimated Effects of Attending Choice Schools on Student Achievement Gains by Academic Characteristics
Mathematics
English Language LearnerNon-ELLSpecial EducationNonspecial Education
Charter −0.03b −0.01d −0.02k −0.01l 
 (0.09) (0.05) (0.09) (0.05) 
Magnet −0.16*c −0.08 −0.10 −0.11* 
 (0.06) (0.05) (0.09) (0.04) 
Catholic −0.29**c −0.17***d −0.23***k −0.18***l 
 (0.11) (0.04) (0.07) (0.04) 
Other private −0.46***ab −0.11a −0.15 −0.15 
 (0.11) (0.08) (0.13) (0.09) 
 ELA 
 English Language Learner Non-ELL Special Education Nonspecial Education 
Charter −0.03f 0.04j −0.02 0.05m 
 (0.08) (0.04) (0.06) (0.03) 
Magnet −0.15**gh −0.10*j −0.10 −0.11*mn 
 (0.05) (0.05) (0.06) (0.03) 
Catholic 0.06hi 0.00 −0.04 0.00n 
 (0.07) (0.04) (0.07) (0.03) 
Other private −0.41***efgi −0.06e −0.17 −0.10 
 (0.11) (0.08) (0.20) (0.08) 
Mathematics
English Language LearnerNon-ELLSpecial EducationNonspecial Education
Charter −0.03b −0.01d −0.02k −0.01l 
 (0.09) (0.05) (0.09) (0.05) 
Magnet −0.16*c −0.08 −0.10 −0.11* 
 (0.06) (0.05) (0.09) (0.04) 
Catholic −0.29**c −0.17***d −0.23***k −0.18***l 
 (0.11) (0.04) (0.07) (0.04) 
Other private −0.46***ab −0.11a −0.15 −0.15 
 (0.11) (0.08) (0.13) (0.09) 
 ELA 
 English Language Learner Non-ELL Special Education Nonspecial Education 
Charter −0.03f 0.04j −0.02 0.05m 
 (0.08) (0.04) (0.06) (0.03) 
Magnet −0.15**gh −0.10*j −0.10 −0.11*mn 
 (0.05) (0.05) (0.06) (0.03) 
Catholic 0.06hi 0.00 −0.04 0.00n 
 (0.07) (0.04) (0.07) (0.03) 
Other private −0.41***efgi −0.06e −0.17 −0.10 
 (0.11) (0.08) (0.20) (0.08) 

Notes: Dependent variables are in standard deviation units, relative to the Indiana statewide average. We estimated the differential ELL and special education effects by adapting our preferred student fixed effects model 3 to include interactions between ELL or special education status with the choice school type indicators. Robust standard errors, adjusted for the clustering of students within schools, are in parentheses. ELA = English language arts; ELL = English language learner.

a–mSuperscript letters next to estimates represent statistically significant differences in achievement gains at α = 0.05 between groups indicated with the same letter.

p < 0.10; *p < 0.05; **p < 0.01; ***p < 0.001.

We observe less variability in results overall when looking at the achievement gains of special education students who switch schools. Special education and nonspecial education students alike who switch to Catholic schools both experience significant losses in mathematics. Meanwhile, special education and non-special education students switching to magnet schools experience similar achievement losses to one another across both subjects. Overall, we do not find evidence that special education students within any school type score differently from their nonspecial education peers in the same school type.

5.  Discussion

This study is one of the first to be able to analyze state administrative longitudinal records in an urban area to examine a wide range of school types and the effects on achievement. We focus on students who switch from a traditional public school to another school type, whether a charter, magnet, Catholic, or other private school. Generally, we find that students who transfer to charter and magnet schools are lower-achieving than their peers in both their public and new choice schools—higher-achieving public school students are the ones who transfer to Catholic and other private schools (though these students are still substantially lower-achieving than their peers in their new choice school). Students switching to Catholic and other private schools are more racially/ethnically, socioeconomically, and academically diverse than those typically enrolled.

Yet, there is variability in the impact of school type on average student achievement gains within the range of school choice options available to parents. For example, we find a significantly negative average impact on mathematics scores for students who switch from a traditional public school to a Catholic school and statistically negative average effects on mathematics and ELA gains for students who switch to a magnet school. For students who switch to a charter school we find no impact on achievement gains, indicating their average gains in student achievement are no different in the new charter school compared to those they experienced previously in traditional public schools. The mean achievement losses in math attained by Catholic school switchers and ELA losses attained by magnet school switchers are significantly different (lower) than their charter school peers in the respective subjects. Although we find that students switching to charter schools by and large do not have different average achievement gains as when they were enrolled in a traditional public school, they also do not appear to experience some of the deleterious effects as their peers who switch to different types of schools.

There is some evidence that African American students who switch to a charter school experience positive benefits in terms of their ELA achievement. However, black students who transfer to magnet, Catholic, or other private schools experience average losses in mathematics achievement. Thus, there is variability by school type for African American students who make switches, and this has implications for black–white student achievement gaps—increasing the mathematics gap when attending magnet, Catholic, or other private schools vis-à-vis attending a traditional public school, and narrowing it in ELA when attending charter schools.

The experiences of Latino students who switch from traditional public schools to a new school type are collectively negative in terms of achievement growth. Whereas most students—whatever their race/ethnicity—experience similar gains in the new school type as they did in their previous traditional public schools, Latino students who transfer to magnet and Catholic schools experience significant achievement losses in mathematics, and Latino magnet school switchers also experience significant losses in ELA. These losses are magnified for those students who are classified as ELL (most ELL students are Latino), particularly in other private schools. In part, these losses could be due to the lack of support services in the students’ new schools, particularly relevant for the Latino students switching to Catholic or other private schools who may need ELL services that are not already in place (24 percent of Latino switchers to Catholic schools are ELL students). A possible explanation for these group differences is that the counterfactuals differ for different groups of students—for example, black students may have attended previous traditional public schools that differ in quality from those attended by Latino students. Given that our descriptive results show that these groups of students come from low-achieving public schools and were low achievers in these schools, this may not be the most viable explanation.

As in similar studies that rely on fixed effects models, an important limitation of our study is that we are looking at students who switch from a traditional public school to a school of choice within the Indianapolis urban area, whether that be a charter, magnet, Catholic, or other private school. Because of this, there may be something quite different about these students who switch rather than those who stay.24 For example, many of the students who switch tend to be lower-performing students and disproportionately African American or Latino. Furthermore, we are unable to obtain causal estimates of the impacts on achievement for those students who are always enrolled in a choice school. Advocates of choice schools argue that benefits accrue the longer a student is enrolled—which may be partially supported by the reduction of negative gains in Catholic schools over time for students switching schools.

Despite these limitations, most students across our comparisons experience similar gains in their new school compared with their previous traditional public school, even though students who switch experience some achievement losses on average if they attend magnet or Catholic schools. We also capture the average achievement gains of students enrolled in these four types of schools across an entire urban area and not constrained to a select set of oversubscribed schools. As such, average gains will tend to be smaller, and we find there is substantial variability in the effects across individual schools within each school type. While we find average differences between school choice types, we believe any claims about the effectiveness of various choice policies as it pertains to the collective improvement of Indianapolis urban area schools is beyond the scope of this paper.

In the next steps of our research, we need to explore further the variability of impacts not only between school types as we have done in this paper, but also to a much greater degree within each school type. We intend to explore the variability in school type impacts by school demographic and academic composition. We also plan to examine variability within charter schools, such as by the type of operator (e.g., not-for-profit vs. for-profit). Although small sample sizes could present a challenge in these subgroup analyses, adding future years of achievement data as they become available will help address this issue.

We find average student achievement gains are relatively modest and similar across school types when aggregated across an entire urban area. In some ways, this is unsurprising. Within any type of school, there is bound to be a high degree of variability—some schools are good, some are average, and some are in dire need of improvement. We need to further examine this variability in detail and move beyond the proverbial school choice “horse race” to better understand the conditions under which choice can be effective (Berends 2015). Perhaps it is the school's mission, organization, or instructional design that is more important for student achievement gains than being a “charter,” “magnet,” “Catholic,” or another type of “private” school (see Bryk et al. 2010; Cohen et al. 2014; Renzulli, Barr, and Paino 2015). To examine these issues will require additional data collection, something we intend to do in our future research.

Acknowledgments

Author order was determined randomly; each contributed equally to this paper. This paper was supported by Notre Dame's Center for Research on Educational Opportunity, Institute of Educational Initiatives, the Spencer Foundation, and the Walton Family Foundation. We are grateful to the Indiana Department of Education for providing access to state administrative records and for supporting independent analyses. We are also grateful for the substantial feedback we received from numerous colleagues, including Jo Blacketor, Josh Cowen, John Elcesser, Phil Gleason, Helen “Sunny” Ladd, Maddy Mavrogordato, Macke Raymond, Amy Ellen Schwartz, James Shuls, Patrick Wolf, and three anonymous reviewers. All opinions expressed in this paper represent those of the authors and not necessarily the institutions with which they are affiliated. All errors in this paper are solely the responsibility of the authors.

REFERENCES

Abdulkadiroglu
,
Atila
,
Joshua
Angrist
,
Susan
Dynarski
,
Thomas J.
Kane
, and
Parag A.
Pathak
.
2011
.
Accountability and flexibility in public schools: Evidence from Boston's charters and pilots.
Quarterly Journal of Economics
126
(
2
):
699
748
. doi:10.1093/qje/qjr017.
Altonji
,
Joseph G.
,
Todd E.
Elder
, and
Christopher R.
Taber
.
2005
.
Selection on observed and unobserved variables: Assessing the effectiveness of Catholic schools
.
Journal of Political Economy
113
(
1
):
151
184
. doi:10.1086/426036.
Angrist
,
Joshua D.
,
Susan
Dynarski
,
Thomas J.
Kane
,
Parag A.
Pathak
, and
Christopher R.
Walters
.
2012
.
Who benefits from KIPP
?
Journal of Policy Analysis and Management
31
(
4
):
837
860
. doi:10.1002/pam.21647.
Ashenfelter
,
Orley
.
1978
.
Estimating the effect of training programs on earnings
.
Review of Economics and Statistics
60
(
1
):
47
57
. doi:10.2307/1924332.
Ballou
,
Dale
,
Ellen
Goldring
, and
Keke
Liu
.
2006
.
Magnet schools and student achievement
.
Unpublished paper, Columbia University
.
Berends
,
Mark
.
2014
.
The evolving landscape of school choice in the United States
. In
Handbook of urban education
,
edited by
H. R.
Milner IV
and
K.
Lomotey
, pp.
451
473
.
New York
:
Routledge
.
Berends
,
Mark
.
2015
.
Sociology and school choice: What we know after two decades of charter schools
.
Annual Review of Sociology
41
(
15
):
159
180
. doi:10.1146/annurev-soc-073014-112340.
Berends
,
Mark
,
Marissa
Cannata
, and
Ellen B.
Goldring
, eds.
2011
.
School choice and school improvement
.
Cambridge, MA
:
Harvard Education Press
.
Berends
,
Mark
,
Matthew G.
Springer
,
Dale
Ballou
, and
Herbert J.
Walberg
, eds.
2009
.
Handbook of research on school choice
.
New York
:
Routledge
.
Betts
,
Julian R.
, and
Paul
Hill
.
2006
.
Key issues in studying charter schools and achievement: A review and suggestions for national guidelines
.
Seattle, WA
:
NCSRP White Paper Series No. 2
.
Betts
,
Julian R.
, and
Emily
Tang
.
2014
.
A meta-analysis of the literature on the effect of charter schools on student achievement
.
Seattle, WA
:
Center on Reinventing Public Education Working Paper
.
Bifulco
,
Robert
.
2012
.
Can nonexperimental estimates replicate estimates based on random assignment in evaluations of school choice? A within-study comparison
.
Journal of Policy Analysis and Management
31
(
3
):
729
751
. doi:10.1002/pam.20637.
Bifulco
,
Robert
, and
Helen F.
Ladd
.
2006
.
The impacts of charter schools on student achievement: Evidence from North Carolina
.
Education Finance and Policy
1
(
1
):
50
90
. doi:10.1162/edfp.2006.1.1.50.
Bryk
,
Anthony S.
,
Valerie
Lee
, and
Paul
Holland
.
1993
.
Catholic schools and the common good
.
Cambridge, MA
:
Harvard University Press
.
Bryk
,
Anthony S.
,
Penny B.
Sebring
,
Elaine
Allensworth
,
Stuart
Luppescu
, and
John Q.
Easton
.
2010
.
Organizing schools for improvement: Lessons from Chicago
.
Chicago
:
University of Chicago Press
.
Buddin
,
Richard
, and
Ron
Zimmer
.
2005
.
Student achievement in charter schools: A complex picture
.
Journal of Policy Analysis and Management
24
(
2
):
351
371
. doi:10.1002/pam.20093.
Carbonaro
,
William
.
2006
.
Public–private differences in achievement among kindergarten students: Differences in learning opportunities and student outcomes
.
American Journal of Education
113
(
1
):
31
65
. doi:10.1086/506493.
Carbonaro
,
William
, and
Elizabeth
Covay
.
2010
.
School sector and student achievement in the era of standards based reforms
.
Sociology of Education
83
(
2
):
160
182
. doi:10.1177/0038040710367934.
Center for Research on Education Outcomes (CREDO)
.
2013
.
National charter school study
.
Stanford, CA
:
CREDO
.
Christenson
,
Bruce
,
Marian
Eaton
,
Michael S.
Garet
,
Luke C.
Miller
,
Hiroyuki
Hikawa
, and
Phyllis
DuBois
.
2003
.
Evaluation of the Magnet Schools Assistance Program 1998 grantees, final report
.
Washington, DC
:
U.S. Department of Education
.
Chubb
,
John E.
, and
Terry M.
Moe
.
1990
.
Politics, markets, and America's schools
.
Washington, DC
:
Brookings Institution Press
.
Cohen
,
David K.
,
Donald J.
Peurach
,
Joshua L.
Glazer
,
Karen E.
Gates
, and
Simona
Goldin
.
2014
.
Improvement by design: The promise of better schools
.
Chicago
:
University of Chicago Press
.
Coleman
,
James S.
, and
Thomas
Hoffer
.
1987
.
Public and private high schools: The impact of communities
.
New York
:
Basic Books
.
Coleman
,
James S.
,
Thomas
Hoffer
, and
Sally
Kilgore
.
1982
.
High school achievement: Public, Catholic, and private schools compared
.
New York
:
Basic Books
.
Cullen
,
Julie B.
,
Brian A.
Jacob
, and
Steven
Levitt
.
2006
.
The effect of school choice on participants: Evidence from randomized lotteries
.
Econometrica
74
(
5
):
1191
1230
. doi:10.1111/j.1468-0262.2006.00702.x.
Dobbie
,
Will
, and
Roland G.
Fryer
.
2011
.
Are high quality schools enough to close the achievement gap? Evidence from a social experiment in Harlem
.
American Economic Journal: Applied Economics
3
(
3
):
158
187
. doi:10.1257/app.3.3.158.
Elder
,
Todd
, and
Christopher
Jepsen
.
2014
.
Are Catholic primary schools more effective than public primary schools
?
Journal of Urban Economics
80
(
1
):
28
38
. doi:10.1016/j.jue.2013.10.001.
Gamoran
,
Adam
.
1996
.
Student achievement in public magnet, public comprehensive, and private city high schools
.
Educational Evaluation and Policy Analysis
18
(
1
):
1
18
. doi:10.3102/01623737018001001.
Gleason
,
Philip
,
Melissa
Clark
,
Christina Clark
Tuttle
, and
Emily
Dwoyer
.
2010
.
The evaluation of charter school impacts: Final report (NCEE 2010-4029)
.
Washington, DC
:
National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education
.
Goldberger
,
Arthur S.
, and
Glen C.
Cain
.
1982
.
The causal analysis of cognitive outcomes in the Coleman, Hoffer, and Kilgore Report
.
Sociology of Education
55
(
2
):
103
122
. doi:10.2307/2112291.
Grogger
,
Jeffrey
, and
Derek
Neal
.
2000
.
Further evidence on the effects of Catholic secondary schooling
.
Brookings-Wharton Papers on Urban Affairs
2000
:
151
201
. doi:10.1353/urb.2000.0006
Hanushek
,
Eric A.
,
John
Kain
,
Steven
Rivkin
, and
Gregory
Branch
.
2007
.
Charter school quality and parental decision-making with school choice
.
Journal of Public Economics
91
(
5-6
):
823
848
. doi:10.1016/j.jpubeco.2006.09.014.
Henig
,
Jeffrey R.
2008
.
Spin cycle: How research gets used in policy debates—The case of charter schools
.
New York
:
Russell Sage Foundation
.
Hoffer
,
Thomas B.
2009
.
Perspectives on private schools
. In
Handbook of research on school choice
,
edited by
Mark
Berends
,
Matthew G.
Springer
,
Dale
Ballou
, and
Herbert J.
Walberg
, pp.
429
446
.
New York
:
Routledge
.
Hoxby
,
Caroline M.
, and
Sonali
Murarka
.
2008
.
Methods of assessing achievement of students in charter schools
. In
Charter school outcomes
,
edited by
Mark
Berends
,
Matthew G.
Springer
, and
Herbert J.
Walberg
, pp.
7
37
.
Mahwah, NJ
:
Taylor & Francis Group
.
Hoxby
,
Caroline M.
,
Sonali
Murarka
, and
Jenny
Kang
.
2009
.
How New York City's charter schools affect achievement
.
Cambridge, MA
:
New York City Charter Schools Evaluation Project
.
Imberman
,
Scott
.
2011
.
Achievement and behavior in charter schools: Drawing a more complete picture
.
Review of Economics and Statistics
93
(
2
):
416
435
. doi:10.1162/REST_a_00077.
Indiana Department of Education (IDOE)
.
2011
.
2011–2012 ISTEP+ program manual: Policies and procedures for Indiana's assessment system
.
Indianapolis, IN
:
IDOE
.
Indiana Department of Education (IDOE)
.
2015
.
Choice scholarship program annual report: Participation and payment data
.
Indianapolis, IN
:
IDOE
.
Jencks
,
Christopher
.
1985
.
How much do high school students learn
?
Sociology of Education
58
(
2
):
128
135
. doi:10.2307/2112252.
Lee
,
Valerie E.
, and
Anthony S.
Bryk
.
1989
.
A multilevel model of the social distribution of high school achievement
.
Sociology of Education
62
(
3
):
172
192
. doi:10.2307/2112866.
Neal
,
Derek
.
1997
.
The effects of Catholic secondary schooling on educational achievement
.
Journal of Labor Economics
15
(
1
):
98
123
. doi:10.1086/209848.
Nicotera
,
Anna
,
Maria
Mendiburo
, and
Mark
Berends
.
2011
.
Charter school effects in an urban school district: An analysis of student achievement gains in Indianapolis
. In
School choice and school improvement
,
edited by
Mark
Berends
,
Marisa
Cannata
, and
Ellen B.
Goldring
, pp.
35
50
.
Cambridge, MA
:
Harvard Education Press
.
Reardon
,
Sean
,
Jacob
Cheadle
, and
Joseph
Robinson
.
2009
.
The effect of Catholic schooling on math and reading development in kindergarten through fifth grade
.
Journal of Research on Educational Effectiveness
2
(
1
):
45
87
. doi:10.1080/19345740802539267.
Renzulli
,
Linda
,
Ashley
Barr
, and
Maria
Paino
.
2015
.
Innovative education? A test of specialist mimicry or generalist assimilation in trends in charter school specialization over time
.
Sociology of Education
88
(
1
):
83
102
. doi:10.1177/0038040714561866.
Sander
,
William
.
1996
.
Catholic grade schools and academic achievement
.
Journal of Human Resources
31
(
3
):
540
548
. doi:10.2307/146264.
Sass
,
Tim R.
2006
.
Charter schools and student achievement in Florida
.
Education Finance and Policy
1
(
1
):
91
122
. doi:10.1162/edfp.2006.1.1.91.
Schwartz
,
Amy Ellen
,
Leanna
Stiefel
, and
Sarah A.
Cordes
.
2017
.
Moving matters: The causal effect of moving schools on student performance
.
Education Finance and Policy
12
(
4
):
419
446
. doi:10.1162/edfp_a_00198.
Teasley
,
Bettie
.
2009
.
Charter school outcomes
. In
Handbook of research on school choice
,
edited by
Mark
Berends
,
Matthew G.
Springer
,
Dale
Ballou
, and
Herbert J.
Walberg
, pp.
209
225
.
New York
:
Routledge
.
Tuttle
,
Christina C.
,
Brian
Gill
,
Philip
Gleason
,
Virginia
Knechtel
,
Ira
Nichols-Barrer
, and
Alexandra
Resch
.
2013
.
KIPP middle schools: Impacts on achievement and other outcomes
.
Washington, DC
:
Mathematica Policy Research
.
U.S. Census Bureau
.
2014
.
State and County QuickFacts for Marion County, Indiana
.
Available
https://www.census.gov/quickfacts/fact/table/marioncountyindiana/PST045216.
Accessed 23 October 2017
.

Notes

1. 

One of the few studies to examine sector effects simultaneously was Gamoran's (1996) analysis of nationally representative data. He found positive effects of attending magnet and Catholic schools on student achievement compared with traditional public school attendance, and no advantage of attending secular private schools.

2. 

The Indianapolis urban area—which we exclusively define as Marion County, Indiana—contains 11 independent public school districts with 172 schools serving students in the elementary and middle grade levels (K–8) as of the 2012–13 school year.

3. 

A family of four with an income below $43,568 (100 percent of reduced-price lunch eligibility) qualifies for a 90 percent Choice Scholarship; a family of four with an income between $43,569 and $65,352 qualifies for a 50 percent Choice Scholarship. The amount of the Choice Scholarship depends on the per-student state funding for the student's school district of residence, though the maximum scholarship for students in grades K–8 is $4,800. Approximately 7,500 scholarships were made available under the program in 2011–12, the first year of the program. The program has grown at a rapid pace since then: 15,000 scholarships were available in 2012–13, and there was no cap on the number of eligible Indiana students who could receive a scholarship in the current school year. This will make Indiana's program the largest statewide voucher program in the United States. In 2011–12, 1,066 students used Choice Scholarships in Indianapolis to attend private schools. In 2012–13, 2,285 used Choice Scholarships. In the current school year, over 5,000 students are using vouchers in the city of Indianapolis alone, more than doubling the population of voucher students in the city (see IDOE 2015).

4. 

In addition to research that investigates the average effects of Catholic schools on students, some research shows that Catholic schools primarily benefit historically disadvantaged students (Coleman, Hoffer, and Kilgore 1982; Coleman and Hoffer 1987; Lee and Bryk 1989; Bryk, Lee, and Holland 1993; Sander 1996; Neal 1997; Grogger and Neal 2000). However, other researchers argue that “the evidence that Catholic schools are especially helpful for initially disadvantaged students is quite suggestive, but not conclusive” (Jencks 1985, p. 134).

5. 

The ISTEP+ is the No Child Left Behind Act–mandated state test for Indiana students in grades 3–8 and is aligned to the Indiana Academic Standards. The ISTEP+ tests students each spring in ELA and math. Until the 2008–09 school year, IDOE administered the ISTEP+ in the fall. We only use spring ISTEP+ data from 2009 onward to avoid issues with conflating test score gains over different periods. The ISTEP+ is vertically equated across grades and consists of multiple-choice, constructed-response, and extended-response items scored using Item Response Theory methods. Reliability coefficients range from 0.88 to 0.94 in ELA and 0.88 to 0.95 in math (IDOE 2011). Schools now have the option to allow their students to take the multiple-choice section online.

6. 

There were 61 private schools serving students in grades 3–8 in the Indianapolis urban area (Marion County) between the 2008–09 and 2012–13 school years. This excludes all special education and small homeschools. Of these 61 private schools, 46 participated in ISTEP+ testing (75 percent). All 28 Catholic schools, 17 of 28 other religious schools (61 percent), and 1 of 5 nonsectarian schools (20 percent) participated. Other private schools participating in ISTEP+ may be different than those not participating.

7. 

We standardized scores statewide as opposed to only within the sample to reflect the distribution used to scale the ISTEP+. The Indianapolis urban area distribution largely reflects the statewide distribution.

8. 

The CCD reflects demographic and background information on the universe of public schools in the United States collected from an annual survey administered by NCES. Similarly, the biennial PSS contains information about the universe of private schools (over 95 percent participate). We were able to apply CCD data to all public schools in the IDOE longitudinal data for each corresponding year of IDOE data. The IDOE data for private schools from the 2008–09 school year reflects PSS information from 2007–08, and the remaining IDOE data for private schools from 2009–10 through 2011–12 reflects PSS information from 2009–10.

9. 

Marion County had just over 900,000 residents as of the 2010 Census. Nearly 91 percent of residents live within the city of Indianapolis as the city boundaries are roughly co-terminus with the county boundaries. Approximately 59 percent of the population is non-Hispanic white, 27 percent African American, and 10 percent Hispanic/Latino (U.S. Census Bureau 2014).

10. 

Our analysis spans from the 2008–09 school year through the 2012–13 school year. Some traditional public, charter, magnet, Catholic, and other private schools closed during the analytical period and others have opened. We include all schools in our analysis in which we observe student test score data for at least two years. Similarly, some schools may have closed or opened from the end of the 2012–13 school year to the present, reflecting a minor discrepancy between the current school numbers.

11. 

State law requires charter schools to use lotteries of student applicants to determine enrollment eligibility.

12. 

Private schools participating in the Indiana Choice Scholarship Program are required to hold enrollment lotteries when the number of individuals receiving a scholarship (voucher) and applying to enroll in the school exceeds the number of seats available. To date, we know of no private schools that have implemented enrollment lotteries.

13. 

See Bifulco and Ladd (2006, pp. 64–67) for a description of the production function that we adapted for our study.

14. 

We do not control for individual school characteristics in addition to student characteristics because we are interested in estimating the mean effect of each school type.

15. 

As an alternative to the gains model, we could have also included prior year achievement as an additional covariate to the OLS levels model 1. We estimate a version of this model with the inclusion of student fixed effects as part of our series of robustness checks.

16. 

The models with interaction terms differ based upon the specific student subgroups we are trying to investigate. For all time-invariant characteristics (e.g., race/ethnicity), we include interaction terms between each racial/ethnic subgroup indicator and school type with no school type main effect terms. For the year-by-year effects, we leave the school type main effect terms in the model to serve as the baseline test score gain in the year a student switches school types and the corresponding year-by-school type interaction terms differentiate the change in test score gains from the baseline year. For the model that tests for differences by voucher recipients, we include interactions between the voucher and private school indicators and leave the charter and magnet school main effects as is.

17. 

We estimated an interrupted panel by removing only the year prior to students switching school types instead of also removing the two years prior. Many students only have two years of data before switching schools. We also estimated an event study model to investigate the presence of pre-treatment trends. The interpretation and display of the results is challenging, given the differing baseline achievement gains for each school type and varying number of pretreatment observations for each student. The results are qualitatively similar to the interrupted panel estimates.

18. 

We calculated the inverse probability weights by dividing the total number of students in each choice school in our analysis by the number of students switching into that school within each year. We then applied these weights to each student switching from a public to a choice school and reestimated the fixed effects model. We only used the weight calculated in the first year after a student switched schools (due to weighting requirements in the fixed effects model). While choice schools with no switchers are automatically omitted from all fixed effects analyses (one magnet and two other private schools), we also exclude any choice schools with fewer than five students from the weighted analysis to avoid inflating individual school estimates based on a small number of students (two magnet, one charter, three Catholic, and two other private schools).

19. 

Our analysis coincides with the implementation of the Indiana Choice Scholarship Program. Therefore, some students switch to Catholic or other private schools when receiving a voucher, and some transfer without receiving a voucher. We estimated a separate model to disaggregate the average school type effects for Catholic and other private schools based on voucher and non-voucher recipients. We find no statistically significant difference in achievement gains between the two groups and within each school type (see table A.2).

20. 

In table A.2, we also examine whether achievement gains by school type for students remaining in choice schools are different from students leaving choice schools and returning to traditional public schools. We find no evidence that results differ between these groups within school type (between-school type differences remain).

21. 

When including weights in the fixed effects model, we find that Catholic school switchers experience a loss of 0.07 SD in mathematics. This is the only nonstatistically significant finding for these students within our robustness checks, in part because of the inflated standard errors resulting from weighting. We believe the unweighted fixed effects estimates underrepresent schools where students had higher than average (not necessarily positive) achievement gains. This estimate falls between the OLS and fixed effects mathematics gains estimates displayed in table 4.

22. 

Although we estimated results for all racial/ethnic groups, we combined American Indian, Asian, and multiracial switchers into the category “Other Race” because there were so few students from each of these individual racial or ethnic categories who switched into any given school type. Also, because race/ethnicity does not change over time, we estimated these results by including a full set of race/ethnicity interactions with school type (no reference group).

23. 

There are also significant differences in achievement gains between students switching to other private schools compared with those switching to either magnet or Catholic schools.

24. 

Despite the superior internal validity of their methodological approach, studies that rely upon lotteries to study charter, magnet, or private schools effects also face this same threat to external validity. Students (families) entering lotteries and/or choosing to enroll in a choice school as a result of winning the lottery may be different than those not applying or taking up the intended treatment.

25. 

There are too few students making multiple switches or switching between individual choice school types (e.g., charter to Catholic or vice versa) to include in our analysis. We compare the demographic and academic characteristics of all other switchers to the students switching school types in our analytical sample in table A.1. We find no discernable differences between switchers omitted from and kept in our analytical sample. We also include in table A.3 all switchers in a sensitivity test of our preferred model.

Appendix A:  Additional Data

Table A.1.
Demographic and Academic Characteristics of Students Changing School Types
Transfer from Public School to
CharterMagnetCatholicOther PrivateAll Other Switchers
Students 1,375 2,527 373 136 2,695 
% Female 50.8 52.9 49.6 58.4 49.3 
% Black 62.3 57.3 25.5 18.1 60.3 
% Latino 7.2 16.2 20.0 10.1 7.3 
% Other race/ethnicity 7.6 7.2 9.9 8.0 7.7 
% Free/reduced-price lunch 89.1 98.5 64.3 69.9 80.3 
% English language learner 7.0 13.3 24.1 14.7 7.9 
% Special education 18.1 19.8 12.9 11.8 18.3 
% Ever retained 10.1 7.6 4.8 5.2 8.9 
Mean math score −0.46 −0.53 −0.15 −0.15 −0.39 
Mean ELA score −0.47 −0.56 −0.09 −0.12 −0.34 
Mean math gain −0.02 −0.10 −0.08 −0.05 0.01 
Mean ELA gain −0.03 −0.06 −0.01 −0.06 0.00 
Transfer from Public School to
CharterMagnetCatholicOther PrivateAll Other Switchers
Students 1,375 2,527 373 136 2,695 
% Female 50.8 52.9 49.6 58.4 49.3 
% Black 62.3 57.3 25.5 18.1 60.3 
% Latino 7.2 16.2 20.0 10.1 7.3 
% Other race/ethnicity 7.6 7.2 9.9 8.0 7.7 
% Free/reduced-price lunch 89.1 98.5 64.3 69.9 80.3 
% English language learner 7.0 13.3 24.1 14.7 7.9 
% Special education 18.1 19.8 12.9 11.8 18.3 
% Ever retained 10.1 7.6 4.8 5.2 8.9 
Mean math score −0.46 −0.53 −0.15 −0.15 −0.39 
Mean ELA score −0.47 −0.56 −0.09 −0.12 −0.34 
Mean math gain −0.02 −0.10 −0.08 −0.05 0.01 
Mean ELA gain −0.03 −0.06 −0.01 −0.06 0.00 

Notes: We report ISTEP+ Math and ELA scores in standard deviation units, standardized relative to Indiana statewide average. We calculated gains in ISTEP+ Math and ELA scores in standard deviation units by taking the difference of each student's test score and their previous-year test score. This table includes only those observations of students who transfer from a traditional public to a choice school and have at least three valid test score measures (two periods of test score gains), which are necessary for identification in our preferred model. All other switchers represent students that include students transferring between choice schools, students transferring first from a choice to a public school, and students with three or more transfers between public and choice schools. ELA = English language arts.

Table A.2.
Estimated Effects of Attending Choice Schools on Test Scores–—Vouchers, Stayers, and Exiters
Mathematics
VoucherNon-VoucherStay in Choice SchoolReturn to Public
Charter — −0.01a −0.04c 0.10e 
  (0.05) (0.05) (0.10) 
Magnet — −0.09 −0.06d −0.20**e 
  (0.05) (0.05) (0.07) 
Catholic −0.21***a −0.14* −0.19***cd 0.09 
 (0.05) (0.06) (0.04) (0.29) 
Other private −0.18 −0.11 −0.15 −0.41 
 (0.11) (0.09) (0.08) (0.30) 
 ELA 
 Voucher Non-Voucher Stay in Choice School Return to Public 
Charter — 0.04 0.03f 0.07g 
  (0.03) (0.03) (0.06) 
Magnet — −0.11*b −0.09*f −0.16*gh 
  (0.05) (0.04) (0.08) 
Catholic −0.02 0.04b 0.01 0.24h 
 (0.04) (0.06) (0.04) (0.14) 
Other private −0.04 −0.22 −0.11 0.14 
 (0.09) (0.13) (0.08) (0.31) 
Mathematics
VoucherNon-VoucherStay in Choice SchoolReturn to Public
Charter — −0.01a −0.04c 0.10e 
  (0.05) (0.05) (0.10) 
Magnet — −0.09 −0.06d −0.20**e 
  (0.05) (0.05) (0.07) 
Catholic −0.21***a −0.14* −0.19***cd 0.09 
 (0.05) (0.06) (0.04) (0.29) 
Other private −0.18 −0.11 −0.15 −0.41 
 (0.11) (0.09) (0.08) (0.30) 
 ELA 
 Voucher Non-Voucher Stay in Choice School Return to Public 
Charter — 0.04 0.03f 0.07g 
  (0.03) (0.03) (0.06) 
Magnet — −0.11*b −0.09*f −0.16*gh 
  (0.05) (0.04) (0.08) 
Catholic −0.02 0.04b 0.01 0.24h 
 (0.04) (0.06) (0.04) (0.14) 
Other private −0.04 −0.22 −0.11 0.14 
 (0.09) (0.13) (0.08) (0.31) 

Notes: Dependent variables are in standard deviation units, relative to the Indiana statewide average. We estimated the differential voucher or exit effects by adapting our preferred student fixed effects model 3 to include interactions between voucher or exit status with the choice school type indicators. Robust standard errors, adjusted for the clustering of students within schools, are in parentheses. ELA = English language arts.

a–hSuperscript letters next to estimates represent statistically significant differences in achievement gains at α = 0.05 between groups indicated with the same letter.

p < 0.10; *p < 0.05; **p < 0.01; ***p < 0.001.

Table A.3.
Estimated Effects of Attending Choice Schools on Test Scores–—Alternative Specifications
Mathematics
Pref. FE Gains (3)All Switchers (3A)FE Levels (3B)Int. Panel (3C)Struct. Changes (3D)Prior Ach. RHS (3E)Weighted FE (3F)
Charter −0.01 −0.02 −0.01 −0.00 −0.03 −0.02 0.01 
 (0.05) (0.04) (0.05) (0.06) (0.07) (0.05) (0.07) 
Magnet −0.09 −0.09* −0.17*** −0.11* −0.13* −0.11** −0.16** 
 (0.04) (0.04) (0.04) (0.05) (0.05) (0.04) (0.06) 
Catholic −0.18*** −0.16*** −0.20*** −0.15** −0.20*** −0.18*** −0.07 
 (0.04) (0.03) (0.03) (0.05) (0.04) (0.04) (0.07) 
Other private −0.16 −0.20*** −0.18* −0.23* −0.13 −0.15 −0.23* 
 (0.08) (0.06) (0.08) (0.11) (0.08) (0.11) (0.10) 
 ELA 
 Pref. FE Gains (3) All Switchers (3A) FE Levels (3B) Int. Panel (3C) Struct. Changes (3D) Prior Ach. RHS (3E) Weighted FE (3F) 
Charter 0.04 0.04 0.03 0.06 0.02 0.02 −0.04 
 (0.03) (0.03) (0.03) (0.04) (0.05) (0.04) (0.05) 
Magnet −0.11* −0.08* −0.13*** −0.15** −0.13* −0.11*** −0.11 
 (0.05) (0.04) (0.03) (0.05) (0.06) (0.03) (0.06) 
Catholic 0.00 0.03 0.01 0.01 0.02 0.04 0.01 
 (0.03) (0.03) (0.03) (0.04) (0.04) (0.04) (0.05) 
Other private −0.10 −0.18*** −0.14** 0.05 0.03 −0.04 0.04 
 (0.08) (0.06) (0.05) (0.07) (0.08) (0.06) (0.12) 
Mathematics
Pref. FE Gains (3)All Switchers (3A)FE Levels (3B)Int. Panel (3C)Struct. Changes (3D)Prior Ach. RHS (3E)Weighted FE (3F)
Charter −0.01 −0.02 −0.01 −0.00 −0.03 −0.02 0.01 
 (0.05) (0.04) (0.05) (0.06) (0.07) (0.05) (0.07) 
Magnet −0.09 −0.09* −0.17*** −0.11* −0.13* −0.11** −0.16** 
 (0.04) (0.04) (0.04) (0.05) (0.05) (0.04) (0.06) 
Catholic −0.18*** −0.16*** −0.20*** −0.15** −0.20*** −0.18*** −0.07 
 (0.04) (0.03) (0.03) (0.05) (0.04) (0.04) (0.07) 
Other private −0.16 −0.20*** −0.18* −0.23* −0.13 −0.15 −0.23* 
 (0.08) (0.06) (0.08) (0.11) (0.08) (0.11) (0.10) 
 ELA 
 Pref. FE Gains (3) All Switchers (3A) FE Levels (3B) Int. Panel (3C) Struct. Changes (3D) Prior Ach. RHS (3E) Weighted FE (3F) 
Charter 0.04 0.04 0.03 0.06 0.02 0.02 −0.04 
 (0.03) (0.03) (0.03) (0.04) (0.05) (0.04) (0.05) 
Magnet −0.11* −0.08* −0.13*** −0.15** −0.13* −0.11*** −0.11 
 (0.05) (0.04) (0.03) (0.05) (0.06) (0.03) (0.06) 
Catholic 0.00 0.03 0.01 0.01 0.02 0.04 0.01 
 (0.03) (0.03) (0.03) (0.04) (0.04) (0.04) (0.05) 
Other private −0.10 −0.18*** −0.14** 0.05 0.03 −0.04 0.04 
 (0.08) (0.06) (0.05) (0.07) (0.08) (0.06) (0.12) 

Notes: Dependent variables are in standard deviation units, relative to the Indiana statewide average. All models are adaptations of the preferred student fixed effects gains model 3 presented in our main results. In model 3A, we include all switchers (students transferring between choice schools, those first transferring from a choice to a public school, students with three or more transfers between public and choice schools, and those in our analytical sample). In model 3B, we use achievement levels as the outcome. In model 3C, we use an interrupted panel where we remove the observation in the year prior switching school types for each student. In model 3D, we restrict the sample to only those students making structural changes. In model 3E, we change the outcome to the student's current year test score and use the prior year test score as a predictor. In model 3F, we include inverse probability weights by number of switchers to each school. Robust standard errors, adjusted for the clustering of students within schools, are in parentheses. ELA = English language arts; FE = fixed effects; RHS = right-hand side.

p < 0.10; *p < 0.05; **p < 0.01; ***p < 0.001.

Appendix B:  Creating the Analytical Sample

First, we excluded students who had more than one observation in a given school year (with different demographic or academic characteristics) because we were unable to determine which observation was correct. This includes students with multiple administrative records (188 students) or those students transferring schools within a given year (279 students). We also omitted any students missing a test score, as we cannot calculate year-to-year test score gains. As we described in our estimation strategy, we also omitted any students with only one or two observations, as we cannot identify their test score gain trends using a fixed effects model. Together, these two steps removed an additional 78,870 students from our analysis. We also excluded any students enrolled in alternative public schools, such as special education or technical schools (372 students).

We chose to exclude any students initially transferring from a charter, magnet, or private school, students making multiple transfers, and those students switching between choice schools (668 students).25 Our focus is on those students switching from traditional public schools to a choice school type. We want to compare the achievement gains that students make in a new choice school with the gains they previously made in a traditional public school in order to assess the effectiveness of types of schools in the Indianapolis urban area. We do estimate separate effects for those students who switch to a choice school and later return to a traditional public school (see table A.2 for results). We also include any students making structural school changes within the same school type.