Sign Up For Our Newsletter

NCES Study: Friedman Foundation Responds

On July 14, the U.S. Department of Education released a study that the teachers’ unions are holding up as evidence that public schools are better than private schools. The study doesn’t actually show this, and is riddled with methodological flaws anyway. If you tell the average American that public schools are better than private schools, she’s likely to respond, “What have you been smoking?” In this case, the evidence shows that the average American is right.

The study tells us nothing whatsoever about the relative quality of public and private schools

It takes raw test scores from isolated years and applies statistical controls for demographic factors like race, income, and disabilities. While the raw scores are higher in private schools, once you apply the statistical controls, public school students actually have similar or even higher scores. The teachers’ unions are rushing to claim that this shows public schools are better than private schools. In fact, as the study itself clearly says, these data show nothing of the kind.

As every education researcher knows, single-year snapshots of test scores reflect student quality much more than school quality. The only real way to get at school quality is to examine year-to-year changes in test scores. A student whose test scores are high is probably just a good student; it’s the student whose test scores are rising who shows the quality of his school. A much more likely explanation for these data is that students who enter private schools tend to have test scores a little lower than other students of the same race and socioeconomic status. That sounds counterintuitive, because we usually think of private school students as privileged. But they are only privileged in terms of their demographic status – which this study controls for. It makes perfect sense that it’s the low performers within each racial and socioeconomic group whose parents will make the sacrifices necessary to put them in private schools. They’re the ones who need it the most.

But don’t take our word for it. The study itself says the same thing – in not one but two big sections labeled “Cautions in Interpretation,” the study forthrightly states that these data tell us nothing whatsoever about the relative quality of public and private schools. The teachers’ unions are just blowing smoke, as always.

The study is shot through with other methodological flaws

Paul Peterson of Harvard University, examining the study’s data, has discovered that the study only produces a positive finding for public schools because it uses the wrong variable to measure Limited English Proficient students. When the correct variable is substituted, the results are positive for private schools. He also points out that public schools are much more likely to classify students as disabled (about 13% versus about 3%). Peterson doesn’t say it, but the main reason for this is that public schools get bigger budgets when they slap the “disabled” label on a child, a perverse incentive that private schools don’t have. If this difference isn’t accounted for, any attempt to compare public and private schools is invalid. Peterson will release his analysis at the annual American Political Science Association meeting this fall.

Andrew Coulson of the Cato Institute has also pointed out that the study controls for variables that are “endogenous,” meaning that they are systematically related to the study’s variable of interest, school sector (that is, public versus private schools). School size is systematically related to school sector, and absentee rates probably are as well, but the study controls for them. “Don’t control for endogenous variables” is straight out of Social Science 101.

A much larger body of much better studies finds that, yes, private schools are better

Those who claim this study as evidence that public schools do better are standing against an enormous scientific consensus. If the available research shows anything, it shows that private schools provide a better education than public schools. The consensus among empirical studies on this issue is as strong as on any social policy question. If social science tells us anything at all, it tells us that private schools do better.

While this study’s method of looking at single-year snapshots can’t even examine whether there’s a causal relationship between school sector (public or private) and test scores, it just happens that there’s a large body of very high-quality research that does allow us to evaluate this connection, and it overwhelmingly finds that private schools do better. The most convincing evidence comes from seven studies using “random assignment,” the gold standard for scientific method. In all seven studies, students who won a random lottery to use a school voucher at a private school had significantly greater test score gains than similar students who lost the lottery and stayed in public schools. Numerous studies using other methods have also produced a strong consensus in favor of this finding.

Greg Forster, Ph.D. is a senior fellow and director of research with the Milton and Rose D. Friedman Foundation.