Teaching to the Student: Charter School Effectiveness in Spite of Perverse Incentives

Charter middle schools in Boston have obtained impressive test score results and strong reputations, resulting in hundreds of children on waitlists, hoping for a chance to enter one of these schools. According to The Boston Globe (2011), two Boston middle school charters are in the top ten middle schools in Massachusetts, as ranked by proficiency on the eighth grade state exam. Causal research based on lotteries confirmed the impressive test score results by showing charter school students who won the lottery and attended school outperformed those who did not win the lottery and did not attend (Abdulkadiroglu et al. 2009, 2011). These results are particularly important because they control for selection bias, countering the frequent criticism that charter schools “cream” certain kinds of students.

The mechanisms behind this large impact are unclear, however. Case studies and non-causal quantitative research suggest that long school days and years, low student–teacher ratios, coherent mission and curriculum, and other school characteristics may contribute to charter school success. On the other hand, another potential cause of the charter school effect is score inflation caused by test preparation activities. Score inflation is defined as “increases in scores that do not signal a commensurate increase in proficiency in the domain of interest” that the test is designed to assess (Koretz 2008, p. 34). Two potential causes of score inflation are strategic coaching of predictable characteristics of tests and reallocation of teaching effort to highly tested topics. If charter schools are engaging in these types of activities, their strong results may be due to score inflation, rather than an actual increase in students’ comprehension. Currently, there is no quantitative evidence for or against the existence of score inflation at charter schools, but there is anecdotal evidence that charter schools are very test-aware.

The accountability system that charter schools face, which has additional accountability measures on top of No Child Left Behind (NCLB), incentivizes teachers to reallocate to highly tested content and to coach certain types of items in order to raise overall score, but not necessarily increase students’ human capital. Using fine-grained data from Massachusetts, I investigate the Boston charter middle school effect more deeply to see if charter students are more successful than their counterparts in other Boston schools on all aspects of the Massachusetts Comprehensive Assessment System (MCAS) and if any of the gains can be explained by score inflation. If charter school students outpace their peers on all elements of the test (rarely tested standards as well as common standards and topics) on science, math, and English/language arts (ELA), and on all types of questions (multiple choice, short answer, and open response) then I will have no evidence of charter schools using test preparation to a greater extent than other schools in Boston.

This is the first study of charter schools that uses item-level information to disaggregate the test score effects in order to determine if charter schools are using test preparation to fuel their test score results. I present results for rarely tested content, including rarely tested curriculum standards, science, and less-emphasized topics to investigate reallocation, and results by item type to investigate coaching.

Although accountability pressure from the state rating system and public competition around test score results might induce teachers to utilize test preparation, I find no evidence of this. Charter school students have large gains on almost all components of MCAS exams, leading me to suggest that their success is not due to differential test preparation, in spite of perverse incentives that might encourage it. The results are robust to adjustments made for attrition and sample matching. Additionally, charter schools do not focus on children on the “bubble” of proficiency—instead, gains are magnified for the least academically prepared.

The organization of the paper is as follows. In section 2, I provide the background and context by describing the charter school impact research, reviewing the relevant details of prior work in Boston, and discussing score inflation. In section 3, I provide a theoretical framework. Section 4 describes the outcome measures, data, and sample. In section 5, I present my identification strategy and in section 6 my results. Section 7 addresses threats to validity. Section 8 concludes.

Lottery-based studies of charter schools have generally found positive results of charter schools on academic achievement. These studies compare students who are offered a seat at a charter school through a lottery with those who are not offered a seat, meaning that the only difference between the two groups is the random offer of charter school attendance. Most of these lottery-based studies are small and city-specific, however. They are also limited to schools that are oversubscribed, which restricts their generalizability. Additionally, lottery-based results may overestimate the underlying citywide results if higher demand occurs at higher-quality schools. Hoxby, Murarka, and Kang's (2009) investigation of New York City charter schools found gains for charter school students in grades 4 through 8. Dobbie and Fryer (2011) focus on one charter school in the Harlem Children's Zone in New York City and found dramatic results, with the causal effect of charter school attendance on math achievement of around a standard deviation over the course of three years in middle school. Interestingly, a recent national lottery-based evaluation of 36 charter schools found no significant effects overall, but significant gains for attendance at urban charter schools (Gleason et al. 2010).

In Boston, the causal effect of charter school attendance on middle school math scores is 0.4 standard deviations on the MCAS, and the effect on middle school ELA scores is 0.2 standard deviations on the MCAS, for each year of charter school attendance. The results for high schools are similar, though slightly smaller, with about a 0.2 standard deviation gain in both ELA and math per year (Abdulkadiroglu et al. 2009, 2011). The middle schools that participate in the Boston research, updated with additional years and newly opened schools, form the sample for this study.

When examining charter school impacts across Massachusetts, the Boston effect was muted (Angrist et al. 2011, 2013), but when the impacts were disaggregated by urbanicity, urban charters performed at similar levels to the Boston schools.

Results from broad comparisons between charter schools and traditional public schools are more mixed (CREDO 2009; Zimmer et al. 2009). The advantage of these studies is that they include students from both highly demanded and less demanded schools. They cannot adjust for the omitted variable bias inherent in comparing attendees at charters with those who may have never applied to a charter, however. A recent report finds that matching estimators can sometimes replicate lottery-based charter effects, but also finds that regression and fixed effects approaches are less successful at replication, perhaps another reason for the divergence in the literature (Fortson et al. 2012). Results from both lottery-based studies and other comparisons are limited in scope to the general impact of charter school attendance on test outcomes, not the details on these outcomes or the mechanisms behind the effects.

Although the Boston results show large impacts for highly demanded charters, the authors cannot use the test score impacts to investigate the specific mechanisms that lead to the strong results. Quantitative research on charter schools is just beginning to investigate the mechanisms behind test score impacts. To date, charter schools have almost all been treated as a “black-box” where schools produce educational achievement by undetermined mechanisms. Hoxby, Murarka, and Kang's (2009) investigation of New York City charters attempts to peek into the black-box by associating some characteristics of charter schools with their success. They find that charter schools that have a longer school year/day, more minutes of instruction in core subjects, a “small rewards/small punishment discipline” system, a performance pay structure, and/or mission statements that emphasize academic success tend to have greater test score success than charter schools without such policies (Hoxby, Murarka, and Kang 2009, p. V-5). These associations should not be interpreted causally, because although they use lottery-based estimates, the connection to characteristics is descriptive. Abdulkadiroglu et al. (2011) observe that Boston charter schools have much lower student–teacher ratios, younger teachers, and fewer in-subject licensed teachers, but again, these are descriptive, not causal, associations. Dobbie and Fryer (2013) find that positive charter school results are associated with “frequent teacher feedback, the use of data to guide instruction, high-dosage tutoring, increased instructional time, and high expectations” (p. 28). Angrist, Pathak, and Walters (2013) suggest that the positive impacts for urban Massachusetts charters are partially due to demographics and partially due to adherence to a “no excuses” philosophy. Recent case studies of five high-performing charter schools in Massachusetts, including three schools in this study, found those successful charter schools were characterized by a strong mission and a school culture dedicated to that mission; structures “that support student learning”; a focus on getting the “right” personnel; involved parents; and “classroom procedures that maximize[d] time on task and tightly link[ed] content to the Massachusetts curriculum framework” (Merseth et al. 2009, p. 228). The factors described here may be the determinants of charters’ success on test scores.

Another factor that could influence charter schools’ MCAS success is test preparation. If test preparation is about “working more effectively, teaching more, [and] working harder” (Koretz 2008), then charter school test score gains might be due to an increase in these beneficial activities. But other, less benign, kinds of test preparation might be a factor in charters’ MCAS success. If test preparation focuses on trivial knowledge of the test or reallocates resources to tested subjects, it could lead to score inflation. Why would potential score inflation in MCAS scores matter? If we think that MCAS outcomes are a measure of future success, not just an academic signpost during school, then test preparation and score inflation impede the inferences that we can draw from MCAS scores. To illustrate, when there is score inflation, a high math MCAS score would give a false impression of future success in math because the high score reflects test preparation rather than increased understanding of the content matter. Thus, if charter school effects are due to test preparation, the inference that they prepare students well for future math courses would be false.

Score inflation can be caused by four types of test preparation: “reallocation, alignment, coaching, [and] cheating” (Koretz 2008, p. 251). Cheating clearly undermines the purpose of testing and leads to score inflation by increasing test scores with no parallel increase in learning (Jacob and Levitt 2003). Reallocation, alignment, and coaching are more ambiguous. Reallocation and alignment involve focusing resources and teaching on tested (or highly tested) topics and subjects, and cause score inflation when they draw efforts away from other parts of the curriculum that actually contribute to the underlying domain that the test is attempting to measure. Coaching occurs when teaching focuses on trivial aspects of the test, taking away time from meaningful content or focusing understanding of a topic in a specific format or organization. This causes score inflation by giving the impression that students comprehend the underlying domain of the test when actually they have become proficient in test taking methods or problems presented in a specific format.

Reallocation is likely widespread. With the implementation of NCLB, school districts across the nation are spending more time on tested subjects and less time on other subjects (McMurrer 2007; Nichols and Berliner 2007). Effects on test scores can be seen through gains on highly tested content but smaller or no gains on other content. Jacob (2005) finds that the implementation of high-stakes testing in Chicago led to gains on math items that are easy to teach or more common on the assessment, but no gains on other parts of the test, implying that reallocation to highly tested subjects caused the math gains. In Boston high school charters, Merseth (2010) sees impressive results on the MCAS but less-impressive results on college entrance exams, and she suggests that teaching at the schools may focus on material in line with the state exam but not the higher-order cognitive tasks tested on the SAT.¹

As mentioned earlier, coaching involves teaching students about test-specific aspects of the assessment, rather than content. Some familiarity with test forms is important, but techniques that teach methods of guessing or standard responses to open response questions can inflate scores. Hamilton (2003) describes case studies and nationwide studies where teachers only distribute problems that parallel the formats on the test and change their instruction to mirror the format of state exams. Koretz (2008) describes methods like the process of elimination on multiple choice exams that, if taught, would increase students’ test-taking skills but not the knowledge that tests are trying to assess.

Despite their successes, Boston charters are not immune to the accountability pressures that might induce test preparation and result in score inflation. Although widely perceived as successful schools because of their MCAS scores, NCLB's Adequate Yearly Progress (AYP) rankings identify most Boston area charters as needing improvement. In 2011, the only Boston charter middle schools not identified as in “improvement” or “corrective action” status under NCLB's standards for subgroups were Edward Brooke and Excel (Massachusetts Department of Elementary and Secondary Education 2011a). Boston charter schools, like many other schools in the nation, have the threat of NCLB sanctions as an incentive to do well on standardized exams. They are also under pressure to maintain high MCAS rankings that are widely trumpeted. Finally, charter schools must be renewed every five years in Massachusetts. Although renewals are not solely based on test scores, academic achievement is part of the renewal process. These triple pressures might encourage test preparation which would cause score inflation. In section 3, I describe in more detail how accountability systems can distort behavior to induce score inflation.

There is also evidence that Boston charter schools are very test conscious. Merseth et al.'s (2009) in-depth study of five charters, three of which are included in this study, indicates that teachers and administrators are very test-aware. Merseth et al. report that curriculum is carefully prepared to match with the Massachusetts Curriculum Frameworks. Teachers use publicly available MCAS items from prior years and they use assessments similar to the MCAS, and teachers constantly track their students’ progress on content that is tested. Nevertheless, these test-aware behaviors need not lead to score inflation if the test preparation activities involve teaching more or better, rather than reallocating time to tested subjects or coaching on trivial details.

Boston middle school charters produce large gains for their students on the MCAS. The mechanisms behind Boston charter middle schools’ success on the MCAS are unclear, however. They may be due to structural reasons, like longer school days and years, or low student–teacher ratios. They may be due to curriculum and planning efforts. Or they may be due to differential test preparation that results in score inflation. The purpose of this paper is to attempt to discover more details on this apparent success. I do so by disaggregating the MCAS scores so as to separate MCAS outcomes that are susceptible to test preparation from those that are not.

By determining if charters do not perform consistently across all measures of the test, I can look for evidence of test preparation. For instance, a particularly large effect on the multiple choice outcome, but little or no effect on the open response or short answer outcomes, might indicate coaching to item type. Similarly, a particularly large effect on standards that are tested most frequently, but little or no effect on standards tested rarely, might indicate reallocation within mathematics to highly tested topics. For an additional check for this type of reallocation, I also exploit the fact that science is less emphasized in the accountability system and investigate whether science gains are similar in size to math and ELA gains.

Several articles (Jacob and Levitt 2003; Muralidharan and Sundararaman 2011; Barlevy and Neal 2012) have framed score inflation as a principal–agent problem. Accountability systems are put into place by state education agencies and the federal government in order to improve student achievement, but individual actors in the education system have an incentive to change their behavior so as to increase measured student achievement, and not necessarily students’ underlying knowledge. Jacob and Levitt (2003) argue that accountability incentivizes cheating, and find overt cheating in 4 to 5 percent of Chicago classrooms. On the other hand, Muralidharan and Sundararaman (2011) argue that although a teacher incentive pay system in India might induce perverse responses, there is no evidence of such responses. Barlevy and Neal's (2012) theoretical incentive scheme also induces socially optimal responses.

Accountability systems may be formal, such as those prescribed by NCLB and state educational agencies. In Massachusetts, charters face an additional accountability system with five-year reviews from their authorizing agency (i.e., the state). In its reviews, the state requires charters to have “academic program success,” “organizational viability,” and be “faithful to the terms of the charter.” Student performance is accounted for by the academic program requirements, which, prior to 2013, included MCAS proficiency or growth toward proficiency and AYP. The factors are also accounted for in the charter faithfulness requirement, as many charter school missions include an explicit focus on academic success. Although there are many other aspects of the reauthorization process, student academic achievement is quite important. Charter schools have similar pressures under the authorization process, the state accountability system, and NCLB, because they all rely on MCAS and proficiency levels or progress toward proficiency. Accountability systems may also be informal, such as pressure exerted by publicity around test scores and school rankings. This might be operationalized by parents with increased pressure on school leaders and teachers, or by parents moving their children out of lower-performing schools. It could also be enforced by principals, who have greater control over teacher hiring and firing than in traditional public schools.²

To describe potential score inflation in Boston charters, I draw heavily on the model used by Muralidharan and Sundararaman (2011), with some modifications. Teachers (who may be encouraged in a particular direction by their school leaders, both of whom are agents in this context), under the various formal and informal accountability systems described here, can spend time on two topics, ⁠, frequently tested content, and ⁠, infrequently tested content. In the context of this study, math and ELA would be considered frequently tested content, whereas science is infrequently tested. Within subjects, some curriculum standards are tested frequently and others are not (for details, see section 4 on outcomes). Additional time spent on frequently tested topics is represented by and additional time on infrequently tested topics is represented by ⁠.

An increase (decrease) in test scores will increase (decrease) teacher utility. Additionally, if and ⁠, reallocating time from infrequently tested items (⁠⁠) to frequently tested items (⁠⁠) will increase utility through two channels. First, when test scores will increase. Second, when ⁠, costs will decrease. We expect if more curricular materials are provided for highly tested items and collaboration between teachers is easier for such items so that shifting time to lowers costs. Additionally, when is more emphasized on the test than ⁠, it is likely that since additional will payoff on many items whereas additional will contribute to relatively few points on an exam. The most important question is whether has the same functional form as and both have non-decreasing returns. If both content areas influence gains in students’ underlying human capital equally, it does not matter if teachers reallocate between and ⁠. But if has decreasing marginal returns or if ⁠, reallocation to incentivized by the accountability system will lower human capital gains for students.

I argue that it is possible to separate into two components, and ⁠, which in turn correspond to and ⁠. For example, measures performance on frequently tested content and measures performance on infrequently tested content. I can then observe whether teachers respond to the incentive system that encourages them to increase by focusing on ⁠, as measured by ⁠, or on ⁠, as measured by ⁠.

Similar interpretations can be made if represents test preparation activities that increase but do not increase (i.e., coaching) and represents other classroom activities that increase both and ⁠.

Each of the outcome measures attempts to highlight a different way that instruction, and thus test scores, can be manipulated or reallocated. The outcome data come from detailed information from individual-level MCAS results. Developed as a result of the 1993 Massachusetts Education Reform Act, which also allowed charters in the state, the MCAS has been the state's standardized test system since 1998. Since 2006, math and English/language arts have been tested in all of the relevant grade levels, and science is tested in eighth grade.

Using the detailed MCAS results, I added further information from the MCAS to create outcome variables that go beyond subject scores. Massachusetts makes public the question type, topic, difficulty, correct answer, and, since 2007, corresponding Massachusetts curriculum standard for each MCAS question (Massachusetts Department of Elementary and Secondary Education 2011b).³ Indeed, the state even publishes the actual MCAS question. Thus, when I merged these data with item-level responses, I was able to identify each question that an individual student answered correctly and create outcome metrics based on subsets of questions.

The outcomes are grouped in three ways: rare standards, question type, and topic. Information on standards was first available for the spring 2007 MCAS, so outcomes using rarely tested standards have a restricted time range. I refer to this as the rare standards sample. Question type and question topic outcomes are available for all MCAS administrations, so I refer to these outcomes as covering the full sample. Each of the outcome measures is a standardized raw score of points in the category by subject, grade, and year. For reference, I also report outcomes for overall standardized score in each subject (“all items”) in both the rare standards and full samples.

The MCAS exams consistently test each of the outcomes in similar proportions across years, making the frequently tested standards, question types, and topics on the test predictable. See table 1 for details. For instance, in math, multiple choice items always account for about 30 points, short answer items about 5 points, and open response items about 19 points (the test format changed slightly in 2010). Topic areas also follow a consistent pattern across years.

The MCAS outcomes used here make up about 80 percent of the MCAS exam; the other 20 percent includes items for equating and trial purposes, which are not reported or included in score calculation but are similar in type and topic to the common 80 percent of items (Massachusetts Department of Elementary and Secondary Education 2007). Thus schools and teachers can predict the format and topic of the MCAS each year. This predictability may lead to test preparation, as teachers can anticipate these features of each year's exam.

For MCAS exams from spring 2007 to 2011, I determined which standards were given the most and least weight and divided the standards into terciles of rare standards, somewhat common standards, and common standards. This outcome allows me to assess whether charter school students do better on frequently assessed standards than on standards only assessed occasionally (to return to the theoretical model, and ⁠). For instance, a question about Massachusetts standard 8.N.11:

was asked only once between 2007 and 2011. In contrast, questions about Massachusetts standard 8.M.3:

were asked twenty-one times in 2007–2011, five times in 2007, 2008, and 2009, four times in 2010, and twice in 2011 (Massachusetts Department of Elementary and Secondary Education 2000). Although the second standard likely encompasses more concepts than the first standard, it is difficult to determine whether one or the other is more important for overall understanding of mathematics.

Question type outcomes are multiple choice, short answer, or open response. Only the mathematics exams have short answer questions. Multiple choice questions and short answer questions are each worth one point on the exam and open response questions are worth four points, with students scoring zero to four on each open response. The format of question types was only changed once in the relevant period, with the math and science exams adding four multiple choice questions and subtracting one open response question in 2010. The format of the ELA exam was never changed in the relevant time period.

Question topic outcomes are specific to subject. For math they include geometry; measurement; number sense and operations; patterns, relations, and algebra; and data analysis, statistics, and probability. For ELA they include reading, and language and literature; and for science they include earth science, biology and life sciences, physical sciences, and technology and engineering. In math, number sense and operations and patterns, algebra, and relations are the most frequently tested topics, followed by data analysis, statistics, and probability. Geometry and measurement are tested the least in the middle school grades. In ELA, reading makes up the majority of the exam and language and literature items only make up a small portion of the test. Science topics are tested evenly. Across subjects, topic divisions are consistent across time. For instance, in ELA, reading accounts for 44 to 48 points on the exam and language and literature 4 to 8 points, depending on the test year.

The data for this analysis come from statewide data sets provided by the Massachusetts Department of Elementary and Secondary Education, as well as lottery records collected from individual charter schools in Boston. The state provided data on students’ demographic backgrounds, program participation, and school attendance for school years 2001–02 through 2010–11, and MCAS scores in math, ELA, writing, and science. I assigned students to their most attended school in each year, except that students who attended at least one charter school were assigned to the charter school even if it was not their most attended school. Thus, a student who attended a charter school for one month and a student who attended a charter school for one year were both assigned to the charter school for that year. Because I attribute a full year of attendance and the students’ tests scores to the charter schools, no matter how long the student attended, my results based on years of attendance can be considered a lower bound on the effect of attending a year of charter school.⁴

In addition to the state data, lottery records were collected from each charter school for the main entry grade in each school (fifth or sixth grade). Lotteries were coded to identify students offered a seat at the charter school, to identify students who were never offered admission to the charter school, and to identify students who did not receive admissions offers randomly, such as students with sibling priority. Not all of the Boston area middle schools that admitted students for middle school entry in fifth or sixth grade were able to contribute records for lottery-based analysis. Two charter schools that contained middle school grades closed, two had insufficient lottery records, and two admitted the majority of their students at the kindergarten level. Appendix table A.10 includes details on school participation. The state data were combined with the lottery data through a matching process, which was then assembled into the analytic data set.

Because my focus is on middle school outcomes, I limit my data set to students with baseline information from the grade of application to a charter (either fourth or fifth grade) who entered charter school lotteries in spring 2002 to spring 2010. The available outcome scores vary with subject and grade level and are detailed in table A.1.

I estimate the causal effect of attendance at a charter school on student achievement in the same way as Abdulkadiroglu et al. (2009, 2011). Because my intention is to disaggregate the charter school effect and determine if it is due to score inflation, however, the outcome measures are standardized components of the MCAS instead of subject scores, and are estimated separately by grade level, rather than pooled.

If all applicants who received an offer for a seat at a charter school attended that charter school and no applicants who did not receive an offer attended (i.e., if all applicants were all compliers) ordinary least squares regression using a variable representing the receipt of an offer would be sufficient to estimate the effect of charter school attendance on outcomes. In this case, however, because some applicants who received an offer to attend a charter school chose not to attend and a few students who lost the lottery ultimately attended a charter school,⁵ I use an instrumental variables approach to estimate the causal effect of charter school attendance on the outcomes of interest.

Here, is the grade-level–specific test score based outcome of interest; indicates the number of years of attendance, including repeated grades, at any charter school after the lottery at time ⁠; is a vector of student level demographic and test score control variables determined before the lottery; and is an error term. I also include a set of year-of-outcome fixed effects, ⁠, and a set of lottery fixed effects ⁠, that represent the charter school lottery risk set.⁶

Here, is estimated by ⁠, a vector of student baseline demographic and test score control variables; ⁠, a set of lottery fixed effects that represent the charter school lottery risk set; a set of year-of-outcome fixed effects; ⁠, an error term; and the instrument, ⁠, which is a dummy variable that indicates if a charter school lottery applicant has received an offer to attend at least one charter school (sometimes referred to as winning the lottery).

In summary, is the first stage effect, which in this case is the difference between the average number of years a student offered a seat at a charter school attends a charter school and the average number of years a student not offered a seat at a charter school attends a charter school. The causal effect of ⁠, a year of charter school attendance, on ⁠, the test score component, is ⁠, which I also refer to as the local average treatment effect. The treatment effect is local because it applies only to compliers, and because it is estimated using a partial compliance estimator, it can also be referred to as the average causal effect (Angrist and Imbens 1995). The associated reduced form or intent-to-treat effect, or effect of on ⁠, is found in an equation similar to equation 4 where is substituted for ⁠. The coefficient of interest is ⁠, which is the causal effect of a year of charter attendance, and is the ratio of the reduced form coefficient (difference in test based outcome between those offered a seat and those not offered a seat) to the first stage coefficient (difference in years of attendance at a charter school between those offered a seat and those not offered a seat).

I fit the two-stage least squares (2SLS) model described earlier for each of my MCAS outcomes, such as math multiple choice score and science rare standards score.⁸ I then inspect these outcomes to determine the composition of the middle school effects and its consistency or inconsistency across sections of the MCAS. I can also look for the effects of differential test preparation.

By comparing treatment effects across MCAS outcomes, I can see if the treatment effect for one or more of the outcomes has a larger response than the treatment effect on other outcome types. For the question type outcomes (multiple choice, short answer, and open response), differential success across outcomes may indicate that charter schools have coached to that question type to a greater extent than the other Boston schools attended by charter lottery losers. Likewise, by comparing treatment effects across standard frequency and subject topics, I can observe if results by standards frequency and topic are substantially different from each other. If charter school students are much more successful on common standards rather than typical standards, or certain frequently tested math topics rather than others, I would have evidence that charter schools are reallocating effort to teaching certain math standards and topics to a greater extent than other schools in Boston.

There are two important caveats. First, if charter schools are using coaching or reallocation with the same frequency as Boston Public Schools (BPS), I expect to see no difference in treatment effects due to coaching or reallocation. For instance, if both charter schools and other public schools are teaching students guessing strategies for multiple choice items, the subscore for multiple choice items would not stand out, even if test preparation occurred. Additionally, if charter schools are effective at coaching across all types of test questions, or are reallocating from untested subjects to all tested standards and topics, then I could not identify a coaching or reallocation effect, because no outcome would stand out. If charters are coaching a particular item type more than the comparison schools and more than other item types, however, I would expect to see a differential treatment effect on that item type subscale. Similarly, if charter schools are reallocating to common standards or more highly tested topics within a subject, I would expect to see higher scores on the more frequently tested items and lower scores on the less frequently tested standards and topics.

In table 2, I present the first stage results that show that the offer of a seat at a charter school does predict future attendance at charter schools. Results are similar across samples and subjects. By sixth grade, on average, students who are offered a seat through the charter lottery attend about 0.7 years more of charter school than students who did not receive an offer of a seat. By seventh grade, on average, students who are offered a seat through the charter lottery attend a charter for a full year more than students who did not receive an offer. By eighth grade, the difference is almost a year and a half.

The first stage estimate may be less than the total potential time a student could attend a charter for two reasons. First, only 70 percent of students who win the lottery at one of the oversubscribed charter schools attend a charter school. Second, a third of the students who did not win a seat through an oversubscribed lottery nonetheless attended some charter school for some time. These latter students could attend a charter by entering at a later grade, obtaining sibling preference, getting a spot off the waitlist late in the school year, or attending a charter not included in the lottery sample.

In table 3, I present the reduced form results for rare standards, science, and question type. These results show the effect of being offered a seat at an oversubscribed charter school on MCAS subscale outcomes. Recall that the outcomes are standardized subscores, so that a statistically significant reduced form effect can be interpreted as the additional standard deviations (⁠⁠) correct on the MCAS subscore that a student offered a seat at a charter school scores compared with students not offered a seat. In table A.3, I present results pooled across grade levels for outcomes that are similar across grade levels.⁹

The causal effects on MCAS outcomes of attending a year of charter school are simply the ratio of the reduced form coefficients in table 3 to the first stage coefficients in table 2. In table 4, I show the 2SLS results, or average causal effects, on MCAS subscale outcomes per year of attendance at a charter. Because the causal effects are per year of charter school attendance, the intention-to-treat effects in table 3 will be scaled up or down to be equivalent to a year of charter school. Thus the sixth grade 2SLS effects are larger than those reported in table 3. The seventh grade 2SLS effects are about the same, and the eighth grade 2SLS results are smaller than the corresponding reduced form results.

In table A.4 I also report in raw MCAS score points the mean outcome score for lottery applicants who did not win a seat in the lottery and those who did. The difference between the two means are roughly equivalent to the reduced form estimates. (The reduced form estimates also include control variables to increase statistical precision.) The mean scores give context to the causal effects that report in the tables in standard deviation units. On the overall scores, students offered a seat in the lottery tend to outscore their counterparts not offered a seat by 3.5–4 MCAS raw points in math, 0.5–2 MCAS raw score points in ELA, and 3–4 raw score points in science, depending on the particular sample. In ELA, the difference is only one multiple choice item on the test, but in math and science the difference is as large as 3–4 multiple choice items or the full score on an open response item. Because these overall gaps are spread across multiple subscales, and some subscales are only a few MCAS points themselves, differences in raw score points between offered and non-offered students will be smaller.

To examine whether charter schools are reallocating more than public schools from less frequently tested topics within each subject, table 3 presents results for the reduced form and table 4 for the 2SLS results of the charter school impact on rarely tested standards, somewhat common standards, and common standards.¹⁰ There is no conclusive pattern. Within each subject, subscores are within 0.05 to 0.2 of each other. As a whole, results by standards are positive, significant, and fairly large for all but one subscale: rare items in eighth grade ELA. This single nonsignificant result may be due to chance, given the large number of outcomes I am testing, or it may be due to some reallocation away from rare standards in eighth grade ELA. But as a whole, the pattern across the standards outcomes does not suggest a pattern of reallocation away from the least frequently tested items.

This setup assumes that each MCAS is a weighted random draw of items, with items weighted toward common standards, and that the 2007–11 exams are similar in standards distribution to past exams. Teachers observe this over time and would have the opportunity to focus on the most common standards. However, perhaps teachers only focus on last year's exam and then reallocate their time away from untested standards. To test for this, I create variables indicating items with standards not on last year's test and items with standards on last year's test. This is only possible in sixth and eighth grade math and eighth grade science, as seventh grade math and all years of ELA standards are tested on every MCAS. The sample for this analysis is also limited to MCAS 2008–11 administrations, because I need both item-level standards data (2007–11) and information about last year's exam (so 2007 cannot be included). I present results from this analysis in table A.5. Again, there is no consistent pattern across subscales, with charter school students outperforming comparison students on both standards that were not tested in the previous year and on standards that were tested in the previous year.

To return to the theoretical framework outlined in section 3, the content of commonly tested standards (or those on last year's test) would correspond to and the content related to rarely tested standards (or those not on last year's test) would correspond to ⁠. I directly observe the MCAS scores related to this content, and ⁠, respectively. Because the test score outcomes are of the same magnitude and significance level, I conclude that, in spite of incentives that may encourage differential test preparation, I do not have evidence of reallocation across standards.

As shown in the previous section, I find no evidence of reallocation within subject content on the MCAS from frequently tested standards to less frequently tested standards. Schools and teachers may not be reallocating their efforts within a subject, however, but rather away from less-tested subjects toward highly tested subjects. Nationally, the Center on Education Policy reports school districts increasing instructional time on tested subjects and decreasing time on subjects like science, social studies, foreign languages, arts, and physical education since the implementation of NCLB (McMurrer 2007). Although I cannot directly compare instructional time, I can investigate whether charter schools in Boston have a similar impact on science as on math and ELA and, for the first time, present results on science for Boston charters.

Although science is tested in Massachusetts, it is tested only once in grades 6 through 8 and results from the test do not enter the calculation of AYP during the study time period.¹¹ Similarly, they are not emphasized in the public presentation of results—each year The Boston Globe publishes proficiency MCAS rankings by district and schools. The science results are in a panel far below the math and ELA rankings (The Boston Globe2011). Because charters do not face the same accountability pressure for science results, they might reallocate their efforts away from science toward math and ELA. If so, I would expect the effect on science of winning the lottery (table 3, column 7) and the average causal response of attending a charter school (table 4, column 7) “all items” scores to be much smaller in magnitude and potentially not significant. I find, however, that results for the 8th grade science MCAS are quite similar to the results for the 8th grade math MCAS. The 2SLS effect in the full sample is about a 0.25 gain in math test scores and 0.29 gain in science test scores, per year of attendance at a charter school. These gains are of similar size and are both significant at the 0.001 level. Thus, I find no conclusive evidence of reallocation away from science. Similar to the interpretation of the standards findings, my comparison of high versus low stakes subjects is represented in the theoretical framework where corresponds to math and ELA, and corresponds to science. I find similar test scores for each test type.

This finding is somewhat analogous to the findings from a recent evaluation of teacher incentives in India (Muralidharan and Sundararaman 2011). Like the pressures in Massachusetts from NCLB, which incentivize math and ELA but not other subjects, in Muralidharan and Sundararaman's experiment teachers were explicitly rewarded for student achievement in math and reading, but not in science or social studies. They found significant gains in all subjects—suggesting that teachers increase their efforts across all topics when they are facing incentives, that academic press on students transfers across subjects, or that there is spillover from highly incentivized subjects to non-incentivized subjects.

The bottom panels of tables 3 and 4 present reduced form and 2SLS results for all question types. Investigating question type should allow me to see evidence of coaching by question type. For instance, if charter schools were coaching a particular strategy on open response questions more than traditional schools did, I would expect to see a higher relative score for open response questions than for other question types. It is not entirely clear which question type would benefit the most from coaching. Multiple choice items can be coached with test-taking techniques like the process of elimination or encouraging students to guess (there is no penalty for guessing on the MCAS). Open response items can be coached by encouraging students to write down any answer, instead of leaving the response blank, or to use key words to signal structure. If there is differential coaching across question types, however, perhaps because it is easier to coach to one item type, it could appear with different effect sizes across question type. In this case, difficult to coach items would be represented by in the model and easy to coach items are represented by ⁠.

In general, charter school students do just as well on each type of question as they do on the subject as a whole. For example, in sixth grade, the overall 2SLS effect on the math MCAS is 0.53 and scores by question type are quite similar: multiple choice, 0.55⁠; short answer, 0.54⁠; and open response, 0.40⁠. In one case (sixth grade ELA), the 2SLS effect for one question type is not significant, although there are significant results for the other question type: overall ELA gains of 0.17⁠, multiple choice gains of 0.18⁠, and a nonsignificant positive result of 0.10 for open response. This exception may be due to chance (given the large number of outcomes I am examining, it's not surprising that one would not be significant), or it may be due to a lack of emphasis on writing in sixth grade. Either way, I still conclude that, for the most part, charter schools outperform their peers in traditional public schools on all question types and see no direct evidence of coaching to question type.¹²

Reduced form results by MCAS topic are presented in table 5 and 2SLS results in table 6. Examining content topics is a similar exercise to examining rarely tested standards. Some topics are consistently tested less frequently—geometry and measurement in math and language and literature in ELA. If charter students perform less well on less frequently tested content areas, I would have evidence of reallocation within subject to more highly tested content areas.

I find, however, that unlike students in Chicago, where the introduction of high-stakes testing resulted in differential effects by question topic (Jacob 2005), charter school students do better than comparison students on all topics on the subject exams. Although there is some fluctuation in the magnitude of effects across topics and grades, all show strongly significant positive results. Therefore, although I cannot rule out reallocation within math topics to those more frequently tested on the MCAS, I have no evidence of it. If both charter schools and the schools that charter lottery losers attend are reallocating their teaching efforts within the math exam to comparable extents, I also would not be able to detect evidence of reallocation.

Students offered a seat in a charter school lottery are more likely to be matched to the state database than students not offered a seat. This is likely due to lottery losers being more likely to enter private school. If these unmatched students are substantially higher performing than the matched lottery losers, however, their omission from the results would bias my findings upward. To address this possibility, I present results in table A.7 that include only applicants from the 2002 and 2009 spring lotteries, which do not have a significant difference in match rates between the offered and non-offered groups (table A.6). I only show grade 6 results because of small sample sizes for the higher grades. Although there is some volatility in the results, as a whole they are just as large or even larger than the findings for the full sample, leading me to conclude that differential match rates are not biasing the results.

If students leave the sample at different rates based on their offer or lack of an offer of a seat at a charter school, the results may be biased if students who leave differ in unobserved ways from students who stay. Table 7 shows that there is no significant differential attrition between students offered and not offered a seat. In case there are unobserved patterns among attriters that could influence outcomes, I refit my results including attriters, by using baseline test scores as substitutes for missing middle grade outcomes (baseline math score is used for all math and science outcomes, and baseline ELA score is used for ELA outcomes). This model assumes that students with missing outcomes continue to perform at the same level as at baseline. In actuality, performance at the exact same level between baseline grade and middle school is unlikely, but it is a good proxy because test scores are strongly correlated across grades (r 0.75). With baseline scores assigned for missing outcomes, the findings are essentially the same as those presented in section 6 (for brevity, in table A.8 I present only the 2SLS results). Because there is little to no difference between the original findings and the results with baseline test scores assigned to missing outcomes, I conclude that the findings are not biased by selective attrition.

Instead of reallocating resources to highly tested areas in order to boost scores, charter schools may be reallocating resources to particular students to increase test scores. Focusing on students for whom intervention is mostly likely to influence proficiency categorization could increase test scores due to differential treatment effects by student type. Several studies have found that schools and teachers focus on students who are on the verge of proficiency (which is the test score outcome used in AYP calculations), perhaps to the detriment of other students. Neal and Schanzenbach (2010) show differential test score increases for students in Chicago in the middle of the test score distribution, the so-called “bubble kids,” and a case study from Texas demonstrates this is an explicit pattern in some schools (Booher-Jennings 2005).

In order to determine if charter schools are focusing on students on the verge of or just above proficiency to a greater degree than their traditional school counterparts, I include interaction terms in the model that estimates the effect of charter school attendance for students within four scaled score points of the baseline proficiency threshold in the baseline grade. For example, the proficiency threshold is 240, so students scoring 236 and 238 are considered near and underneath, respectively, the threshold in their baseline year, and students scoring 240 and 242 are considered near and above, respectively, the threshold in their baseline year.¹³ This baseline definition attempts to both measure prior proficiency level in the way a school or teacher would when examining the records of individual students, and also to avoid concerns about endogeneity. I present interaction results only for grade 6 outcomes, because these are the closest to when prior proficiency is determined.¹⁴

Because Massachusetts AYP determinations are based on a state calculated Composite Performance Index (CPI) that also gives credit to some scores below proficiency, I also create “near” variables for each kink in the CPI calculation. CPI points are awarded as such: proficient or above (above 240 MCAS points)—100 CPI points; needs improvement high (230–238 MCAS points)—75 CPI points; needs improvement low (220–228 MCAS points)—50 CPI points; warn/fail high (210–218 MCAS points)—25 CPI points; and warn/fail low (200–208 MCAS points)—0 CPI points. Massachusetts also allows schools to achieve AYP through improvement, which involves a specific goal set for each school and subgroup. Improvement is also calculated using the CPI, with its kinked nature, however, which would again put the focus on students near thresholds rather than throughout the achievement distribution.

I investigate the interaction between years of attendance at a charter school and prior scores (table 8 for math and table 9 for ELA). To test whether overall score or specific place in the score distribution is relevant, I do this both for overall standarized score, and in a separate model, near each prior CPI relevant threshold: proficient, needs improvement high, needs improvement low, and warn/fail high. If charter schools are focusing on students on the bubble of proficiency (or another score threshold) to a larger extent than their traditional public school counterparts, I would expect the interaction terms for students in the prior year near the threshold category to have a significant positive contribution to the test score impacts (columns 4, 6, 8, and 10). This is the case for one of the math outcomes and one of the ELA outcomes (perhaps, given the large number of coefficients tested, due to chance). Instead, it appears that the charter school effect is largest across all math outcomes and two of the ELA outcomes for students with the lowest prior test scores (column 2). Thus, I find little evidence in test score outcomes that charters are focusing on students on the verge of proficiency or another score threshold at a rate greater than the schools that their counterparts attend. The charter schools are in fact most effective, at least in math, for the many students at the very bottom of the proficiency distribution.

This paper investigates the details of the large causal impacts of attendance on MCAS outcomes at highly demanded middle school charters in Boston. Despite an incentive structure that would seem to reward teachers and charter schools for focusing on certain aspects of MCAS tests, I find no evidence of test preparation in comparison with traditional public schools. The consistent results across all elements of the test provide no discernible evidence of more reallocation between rare and common standards, low and high stakes subjects, multiple choice and open response questions, and infrequently and frequently tested topics in charter schools compared with traditional public schools. These results remain substantively the same when baseline test scores are assigned to those with missing outcomes or when limited to the sample with the same match rate by offer status. Nor is there evidence that charter schools are focusing on bubble students at a greater rate than other schools in Boston. My analysis strategy cannot conclusively rule out inappropriate test preparation, especially if it is consistent across all aspects of the test or if it is comparable to the test preparation that comparison schools conduct. Nevertheless, the evidence I show here also aligns with recent work showing that Boston charter high school¹⁵ students outperform their counterparts on SAT and advanced placement tests and are more likely to enroll in four-year colleges (Angrist et al., forthcoming). Follow-up work on the Harlem Children's Zone also finds positive outcomes on nonstate standardized test academic and social outcomes (Dobbie and Fryer, forthcoming). Combined with this recent evidence from the literature, the lack of any evidence of test preparation in these findings is suggestive that charter school gains are due to building the human capital of their students, rather than just increasing test scores, in spite of incentives that encourage teaching to the test.

I am grateful to Carrie Conaway and the staff at the Massachusetts Department of Elementary and Secondary Education, and the Boston area charter schools for their generous access to data and their time and assistance. Sandy Jencks, Daniel Koretz, Richard Murnane, Lindsay Page, John Willett, and, especially, Joshua Goodman provided helpful comments. I also thank my charter team colleagues, Joshua Angrist, Susan Dynarski, Jon Fullerton, Thomas Kane, Parag Pathak, and Christopher Walters, the Center for Education Policy Research at Harvard University, and the School Effectiveness and Inequality Institute at MIT.