Vol. 15, No. 25
Special Charter Research Wars Edition
DON’T LET EM’ FOOL YOU. We’d like to caution Newswire’s readers this week about the release of another national report that uses statistical gymnastics to make spurious comparisons of student achievement in charter schools across state lines. At the stroke of midnight a report on charter school achievement started making headlines across the country. Some accounts were positive, some negative and others called it a “mixed bag.” Regardless, they are all simply wrong.
CREDO REDUX. Prior to its release of the National Charter School Study 2013, The Center for Research on Education Outcomes (CREDO) produced a series of reports – one national, the others state-based – looking at student achievement in charter schools compared to traditional public schools. CREDO’s 2009 national report, Multiple Choice: Charter School Performance in 16 States, is the basis for the statement often made across certain media, policy and education circles that only one out of five charter schools succeed.
The 2013 report is broken down into two separate analyses. First, it provides an update on the 2009 report, which reviewed charter school performance in 16 states yet made generalizations about charters nationwide. Second, it examines learning gains across states, schools and nationally using data from 27 total locations (New York City is included as separate data from New York state).
There’s a perception that this report was done based on absolute test scores using apples-to-apples comparisons. Highly criticized by leading researchers and economists for failing to use “gold standard” Randomized Control Trials (RCT) it also fails to address concerns raised by RCT standards.
THE FINE PRINT. Within this CREDO study it is said that, “not surprisingly, the performance of charter schools was found to vary significantly across states.” CREDO recognizes in fine print that there are wide variations in state tests and that they have somehow determined a way to align them for meaningful comparison. That of course begs the question – if it’s that easy to align state tests and results across state lines, why is there a national move for Common Core State Standards and aligned tests?
CREDO’s report also argues that it employs growth data for students to create a picture of student achievement gains – or losses – over time. It attributes the ability to do this with better and more consistent data collected by states. However, it’s not that simple. Page 24 of CREDO’s Supplementary Findings Report demonstrates the conundrum of analyzing groups of data and not individual student data consistently over varying periods of time. For example, CREDO acknowledges that their results include students who have only spent one or two years in charter schools, “not allowing much time for their cumulative impact to be seen.” Much more is of great concern and anyone using this report to make conclusions would be wise to read the fine print before doing so.
‘GOLD STANDARD’ IT’S NOT. CER has argued – echoing highly respected researchers — that the only studies that are valid for understanding and comparing charter school achievement are “gold standard” randomized control studies such as those done by Stanford Economist Dr. Caroline Hoxby, and University of Arkansas’ Dr. Patrick Wolf, to name just two among at least a dozen more. Such studies compare students who were chosen randomly from two pools – students who were chosen by lottery and attend the school of choice, and students who did not attend, but were also in the lottery.
The 2013 CREDO Study takes CER’s previous critiques to account in a side-by-side rebuttal, stating, “The lottery must be random. This is often not true in charter schools, as many schools permit preferences to siblings of current students, children of school founders or staff, or residential preferences for students who live near the school.” Once again we take issue with this statement.
CER has responded in a point-by-point counter response you can find here.
THE NEGATIVE ECHO. The echo from this report is damaging as the results do not accurately convey a national picture. CREDO’s policy prescriptions are even more troubling. Its plans to address what it concludes as uneven student achievement in charters is lacking in any experience in how state policies are written and how they impact actual schools and students. State-by-state and community-by-community analyses are the only true measures to date that offer validity for parents, policymakers and the media to report to make smart decisions about educational choices and outcomes for students. We still believe this is not the detailed study that the charter community needs to assess real progress or lack thereof. Check out CER’s full analysis for helpful talking points.