June 25, 2013
The Center for Education Reform (CER) has argued – echoing highly respected researchers — that the only reports valid for understanding and comparing charter school achievement are “gold standard” randomized control trials (RCTs) such as those authored by Stanford Economist Dr. Caroline Hoxby, and University of Arkansas’ Dr. Patrick Wolf, to name just two respected creators of nearly a dozen reports. Such studies compare students who were chosen randomly from two pools – students who were chosen by lottery and attend the school of choice, and students who did not attend but were also in the lottery. Hoxby has done such studies with regard to charter schools and Wolf has conducted such studies for voucher programs.
The CREDO 2013 National Charter Study employs a completely different method of assessing student achievement, which is described in detail in the report. Because of the Center for Education Reform’s ongoing critique of their methodology, CREDO addresses the issue of randomized control “gold standard” studies and argues that RCTs are not valid for broad charter school studies. Once again, we take issue and have addressed each of CREDO’s research methodology arguments below:
CREDO concedes that, “randomized controlled trials (RCT) are considered the ‘gold standard’ in social science research. However, there are a few caveats necessary to conduct a RCT.” Points 1 through 4 in the Technical Appendix are CREDO’s points about RCTs. The subsequent notations represent CER’s response, based on input from experts and credible researchers in the field.
CREDO POINT 1. The lottery must be random. This is often not true in charter schools, as many schools permit preferences to siblings of current students, children of school founders or staff, or residential preferences for students who live near the school (See Betts, J. and Hill, P., 2006 for a summary of potential challenges to the internal validity of RCTs).
CER RESPONSE TO POINT 1. This is not a valid statement. RCTs easily handle students who are exempt from lotteries, like siblings, by excluding those students from the analysis. This is common research practice and in no way threatens the internal validity of the research. Randomization can be tested and is tested by all good charter studies.
CREDO POINT 2. There must be sufficient numbers of students participating in the lottery. In other words, there have to be large numbers of students who do not get selected into the charter school. While many charter schools have waiting lists, most charter schools do not have large enough waiting lists for a RCT.
CER RESPONSE TO POINT 2. The CREDO study does not have data on charter schools’ admissions lotteries, so it does not use a randomization-based method of evaluation. Had they had access, they may indeed have had enough subjects to conduct such a trial. According to Caroline Hoxby’s critique in 2009, “Lacking lottery data, the CREDO study depends on a matching method based on charter school students prior histories in the traditional public schools. But it does not match each charter school student to individual traditional public school students with similar demographic characteristics. Rather, it matches each charter school student to a group of students in traditional public schools. A charter school student can potentially be matched to a group that contains many students. The study then computes average achievement and other average characteristics of each group. Thereafter the study treats these group averages as though they were students.” There are numerous other problems with this approach that experts such as Hoxby have enumerated.
CREDO POINT 3. The charter schools that meet conditions 1 & 2 above must be representative of all charter schools. Violating condition 3 creates major problems in conducting a valid national charter study using RCT for several reasons.
1. Charter schools which have a long-term reputation for quality may be more likely to hold a lottery than weaker or newer charter schools.
2. Charter schools located near particularly low quality traditional public schools may be more likely to hold lotteries than charters located near higher performing TPS.
3. Charter schools in areas with fewer choice options may be more likely to have lotteries than charter schools located in areas with a higher number of choice options.
CER RESPONSE TO POINT 3. There is nothing scientific or evidentiary about the points in this argument. Presuming that highly reputable charters are more likely to hold a lottery than weaker or newer schools is not only presumptuous, but it ignores the legal requirements set forth in states and as a consideration of federal funding. Charter schools that have more applicants than seats are required to have a lottery. As such schools fill up, students who are selected by lottery have preference year after year. Most charters then have lotteries only for certain numbers of seats that are available in certain grades year after year. Some schools may not have a lottery for years after they have been open and operating because they are oversubscribed. That doesn’t mean that the students attending those schools are not able to be part of a trial because they have self-selected as they began in that school by lottery and subsequently chose to stay in that school. That is no different a condition than students who attend their neighborhood public school who choose to stay year after year because they either have no choice, were denied from a charter lottery, they are satisfied, they have no knowledge of other choices or any number of scenarios. In other words, once a charter school lottery occurs, the students who “settle” in each of the schools being studied are comparable.
Federal law defines a charter by virtue of lotteries: NCLB, Title V, Part B: Charter Schools. Section 5210: “The term ‘charter school’ means a public school that . . . (f) does not charge tuition, and ;. . . (h) admits students on the basis of a lottery if more children apply for admission than can be accommodated.”
The presumption by CREDO that charters near low-performing traditional public schools are more likely to hold lotteries again ignores requirements of law and has no basis in fact. Whether or not a state or community has many or limited choice options, every school community has different circumstances dictating how parents may view their local options. From NYC to Boston to rural Colorado, charters are having lotteries when they are new to fill seats unless they do not have enough enrollment, and if they do not have enough enrollment, experience tells us that that whatever that school offers is not sufficient in the eyes of parents in that community, or the school was compromised by local political battles or procedural delays in getting approved. There are also several other possibilities that exist for undersubscribed charters that do not have lotteries, but to attribute one possible cause relating to quality options demonstrates ignorance of the landscape CREDO is purporting to study.
Thus CREDO’s Point 3 to qualify why RCTs do not work with broad charter samples – based on assumptions about why parents choose and bad information about how lotteries are conducted – is wholly without foundation.
CREDO POINT 4. RCTs have strong internal validity but weaker external validity. While RCTs are the gold standard for estimating the effect of a single treatment (e.g. the effect of attending a specific charter school), any of the violations listed in 3 above could damage the ability to generalize results to other charter schools. CREDO’s matching method has much greater external validity because it is not limited to charter schools with random lotteries and sufficiently large waiting lists. The charter schools and students in CREDO’s data set look much more like the national charter sector than those eligible to be included in a RCT. In addition, a recent meta-analysis of the charter school literature found that, “as long as baseline test scores are controlled for, the specific method of analysis employed will not severely impact conclusions.” (Betts, J et al., 2011) In this light, RCTs and quasi-experimental methods should be considered complements, not substitutes.
CER RESPONSE TO POINT 4. External validity (i.e. extrapolation to charter schools outside the study) is a problem for ALL studies. It is not at all improved by the CREDO methodology, which is fundamentally flawed because it cannot control for selection.
Many researchers concede that there is a trade-off between internal and external validity. But rigorous research almost always prefers internal validity when a trade-off is forced. We want to know precisely and accurately the effect of an intervention, even if we can only know that for a limited population. It does us little good to know inaccurately the effect of an intervention on a broader population.
All studies are representative of the schools they include. This does not make CREDO studies better–just different.
The overwhelming problem with CREDO studies is that they have NO method of controlling for self-selection into charter schools or for charter schools disproportionately attracting disadvantaged students.