New state-level studies demonstrate more rigorous standard of research than national study
CER Press Release
Washington, D.C.
January 15, 2013
A recent report by CREDO, a Stanford University based organization, finds that Michigan charter school students are seeing two-month gains in learning over traditional schools in one year. The study also found that 35 percent of charters in Michigan have significantly more positive gains in reading and 42 percent of charters have more gains in math than their traditional school counterparts. Continuing a trend CREDO has found in other states in which they’ve conducted studies of state-level public school student achievement data.
The Center for Education Reform has been critical of the work CREDO performed in the past, specifically in its first, national study, which used incongruous state data about students to make conclusions about charter school performance in selected states. This national “study” on 16 states resulted in an inaccurate but often quoted characterization among lawmakers and in the press that charter schools in general are not performing as well as other public schools when in reality, that report did not make meaningful comparisons on students within states or nationally.
A review of CREDO’s recent state level reports shows an improved approach to handling data.
According to CER President, Jeanne Allen: “Where the approach and data used in CREDO’s national study remains flawed, we believe that the current methodology used in the Michigan and New Jersey studies is more sound, and more closely approximates the level of research expertise we should be using to judge charter school achievement. In these studies, it appears that more and better demographic and school level data were used to identify and compare individual students to their ‘traditional public school’ counterparts, providing a more realistic view of students, and therefore, more credible results.”
Many researchers, including Stanford Economist Caroline Hoxby, argued about the original CREDO study that because of flawed methodology in analyzing student achievement, there was a bias against charter schools, one that the press and opponents immediately seized upon to suggest that fewer charters are making the grade. As Hoxby points out in her analysis of that 2009 study:
“The achievement of charter school students is measured with much more error than the achievement of the controls, which are not individual students but are group averages of students in the traditional public schools. By using the achievement data as both the dependent variable and (lagged) an independent variable, the CREDO study forces the estimated effect of charter schools to be biased, and the bias is negative …This paper also notes that the CREDO study violates four rules for the empirically sound use of matching methods to evaluate charter schools’ effects.”
Among the flaws in the original study:
“The CREDO study does not match individual charter school students to individual traditional public school (TPS) students with similar demographic characteristics. Instead, it matches each charter school student to a group of students in traditional public schools. A charter school student can potentially be matched to a group that contains many students. The study then computes average achievement and other average characteristics of each group. Thereafter, the study treats these group
averages as though they were students.1 The group for each charter school student is selected according to the following procedure. Each charter school is associated with a set of traditional public schools based on which schools their students attended before they applied to the charter school. Naturally, this information is not available for many
charter school students because they applied as kindergarteners, previously attended a school outside the local area, previously attended a private school, or simply do not have this information recorded. Nevertheless, a set of traditional public schools is picked for each charter school.” In short, matching methods involve substantial judgment, and the judgments focus on unobserved variables.
Says Allen: “We have known from multiple measures over the years that the majority of charter schools, particularly those serving children least likely to succeed, outperform, on average their traditional public school counterparts. The gold standard to which researchers aspire requires that we utilize real, individual, data over time, to ensure we are making accurate judgments about their progress in their respective schools. Until then we believe that attempts to make conclusions about the effects of charter schools must be cautiously viewed We welcome any new research that uses accurate data to make its conclusions and to that, believe that the CREDO reports on charter school achievement in New Jersey and Michigan provide a strong case that most charter schools are fulfilling their intended mission.”