UPDATED 6/28–A Deeper Look at the Graduation Rate Debate (Dan Losen vs. Joydeep Roy, with comments by Larry Mishel)

Losen, May 24:

I’m among the experts debating this issue. I’m disappointed in Mishel whose work I usually hold in high esteem. I’ve read Mishel’s book and it misleads the reader by ignoring obvious inconsistencies in data on New York City, Florida and Chicago that he relegates to the appendix. In fact, wherever Mishel looks at actual student record data that he deems reliable, he too finds a dropout crisis. This is contrary to his own conclusions based on surveys with admitted problems of years surveyed, sample design, and undercounts. Specifically, Mishel’s survey-based estimates of the national rate graduation rates are 15% to 35% higher than the actual record data he argues are accurate in Florida, Chicago and New York City, the places he looked at more directly.

Mishel finds that Florida’s four year graduation rate for Blacks is about 55%, and Hispanics about 60%, and these rates Mishel admits are inflated by counting GED recipients as graduates. In Chicago, Mishel finds that Black 19 year old males have a graduation rate of 39% and Hispanic males 51%. In New York City Mishel points to an extended 7 year completion rate for all students of just 60%. The New York City rates he cites are actually about 44% for the 4 year graduation rate according to the State of New York. Mishel ignores the fact that only the 4 year rate meets the requirements for evaluating schools and districts under the No Child Left Behind Act.

These alarmingly low numbers are consistent with the analysis from Chris Swanson and many other researchers, besides Jay Greene, that we have relied upon at Harvard.

Mishel’s own numbers indicate a crisis. There is an urgent need to address the crisis facing minority youth. Improving the data collection should be part of these efforts rather than cause for further delay.

Finally, Mishel offers no useful recommendations and would have us wait many years until we have more accurate data before we address the problem. He acknowledges the crisis in urban schools where we find high percentages of minority youth yet appears to want to stay the course, which would continue to put minority students at a great disadvantage.

Roy, May 24:

I am one of the co-authors of the Economic Policy Institute book disputing the current conventional wisdom that minorities have only a 50-50 chance of graduating from high school with a regular diploma. Here is my quick response to two of Dr. Losen’s assertions.

Dr. Losen writes in his comments that “wherever Mishel looks at actual student record data that he deems reliable, he too finds a dropout crisis”. While this depends on what one means by the word “crisis”, it should be pointed out that we refer to the NELS:88 survey as the gold standard, and it is indeed a longitudinal tracking of actual students over time (for a period of 12 years). The results from the NELS show that minorities have close to a 75% graduation rate. Similar high-quality longitudinal surveys also indicate that the rates have been steadily rising over the last 30 or 40 years – see Tables 3 and 4 in our book, available here, which compares NELS:88 to High School & Beyond and NLSY97 to NLSY79, and also Figures C and D – except possibly for black males during the last decade.

Dr. Losen also argues that “Mishel’s survey-based estimates of the national rate graduation rates are 15% to 35% higher than the actual record data he argues are accurate in Florida, Chicago and New York City, the places he looked at more directly.” It is hardly surprising that the national graduation rates for blacks is significantly higher than that for blacks in inner cities like Chicago and New York City. White males in Chicago graduate at a rate of 58% – this does not imply that the national graduation rate for whites is 58%. (Even Greene and Winters’s estimate of white male graduation rate is 74% – see this Manhattan Institute study.)

Our aim in this study has been to create a better understanding of the true challenges we face and the progress we’ve made, and help lead the way to better targeted solutions for continuing to close the remaining gaps. Understanding where we are and how far we’ve come can help identify what has been working in American public education. There are significant problems to be addressed – the minority graduation rates are still low and there are significant gaps in completion between whites and Asians on the one hand and blacks and Hispanics on the other. In some inner cities like Chicago black males have only a 40% chance of completing high school with a regular diploma. However, we believe that unless we know the true picture we are unlikely to correctly address these problems.

Losen, May 30:

This blog debate highlights two of the fundamental problems with Mishel and Roy’s analysis: Mishel and Roy don’t have great data and they use the wrong standard for evaluating the current health of our high schools, especially with regard to Black and other minority students. 

Their strongest assertions are linked to a comparatively small number of student outcomes for the Class of 1992. Their "gold standard," the NELS 88, is a 12-year-old estimate based on a national sample of about 20,000 students. Those unfamiliar with the survey will be surprised to learn that the actual number of Black students in the sample was approximately 1,456 students. Most, but not all, of these Black survey responses were confirmed by looking at their actual student records.  That may be an excellent sample as surveys go, but does not come close t

o the accuracy that might be achieved if we looked at individual longitudinal student record data from every district and state in the nation.  As Mishel and Roy point out, the fact that most of those surveyed in the national sample were confirmed with actual student records is what makes NELS Golden.

As Dr. Roy points out, in their book, NELS shows that back in 1992 only 63.2% of Blacks students graduated "on time." Researchers who use NELS agree that this number should be adjusted downward somewhat because of bias in the sample of Blacks in particular. This is because Black youth are disproportionately incarcerated, homeless or have disabilities, and would not be accurately represented in the NELS sample which excluded those populations. The reported 63.2% does not reflect any of the adjustments for such bias.

Not only are the sampled data suspect, the extended years graduation rate is the wrong standard. We should agree that the vast majority of high schools are designed (and budget their resources) so that successful students will take four years, from grade 9-12 to earn the credits they need to graduate high school with a real diploma. If we accept that premise, then schools and districts should calculate and report the “on time” graduation rate. As Congress required in No Child Left Behind, the graduation rate is the percentage of students that graduate in the “standard number of years with a diploma.” In regulations the administration clarifies that GEDs and other alternative certificates should not be counted when calculating graduation rates.  Other measures of school completion do have important value, but an estimate of a four year rate, which is the “standard number or years,” and based only on real diplomas, should be the primary measure we use to evaluate the efficacy of our high schools.

From the start, Mishel and Roy ignore the fact that Chris Swanson’s graduation rate estimate was created purposefully to be used as an “on time” graduation rate to meet the needs of evaluating schools and the requirements of NCLB. Over Swanson’s objections, Mishel and Roy treat his estimate of a four year “on time” rate as if it were a 6 year rate creating an apples to oranges comparison. Further ignoring the fact that Swanson’s numbers are for the Class of 2001 and the survey sample they use is for the Class of 1992, they insist Swanson is seriously exaggerating the actual rate.   They rarely compare his “on time” rates to the 62.3%, or other “on time” rates.

The more important problem is that rates based on outcomes of 5, 6, 7 or more years is a poor indicator of how well a particular high school or district is performing, or whether a particular education reform is having the desired impact.  But Mishel and Roy rarely discuss the “on time” 63.2% figure for the Class of 1992, and prefer the 74% number. If we stick to comparing apples to apples, we would have narrowed the difference between the 4 year "on time" 50% estimate for Blacks that we report based on Swanson’s data, and their 4 year estimate of 63.2%, reducing the difference from 25% to 13%  percentage points. 

But there are further problems: Mishel and Roy’s survey sample of 1,456 Black students is particularly useless for understanding the recent condition of education in urban school districts, or for poor southern states, where most Black students live. Places similar to Florida, New York and Chicago.

With only 1,465 actual students from across the nation in their 1992 sample, and no state or district results in the NELS to start with, it becomes apparent why Mishel and Roy rely more on "non-gold standard" measures (surveys lacking student record confirmation) to dispute our claim that Black and Latino and Native American students, and especially males from these groups, are experiencing a severe problem in many poor and racially isolated districts all across the nation.

The best way to check the current 2006 value of Mishel and Roy’s national estimates for the Class of 1992, is to see whether their older data accurately represents individual student records today. Unfortunately, that’s impossible to do on the national level. All they can give us are these data from Florida, Chicago, and New York.

In Florida the data come from over 200,000 individual longitudinal student records in all. As a result, we have the “golden” student records of tens of thousands of Black students from 2003, not just a sample of 1,456.  Mishel calls these data “much better estimates of graduation rates.” The data show that Blacks had an “on time” graduation rate of 54 % in 2003.  I regard this data as upwardly biased, and Mishel admits as much in the footnotes, because Florida counts and reports GED recipients as graduates and also removes from the denominator those students who leave school to attend alternative education programs, when they should be counted against the graduation rate. But, despite the serious concerns that the graduation rate reported in Florida overstates the “on time” graduation rate, it is safe to say that the rate based on individual student records would unlikely be any higher than 54%. The data from Chicago and New York suggest even lower rates for Black students. With no racial breakdown, the State of New York reports an “on time” graduation rate of 44% based on tens of thousands of longitudinal student records – Roy and Mishel’s gold vein. And even this estimate is biased because New York excludes from the cohort the records of any student not in school for 5 consecutive months before the count was taken in grade 9. In Chicago, the golden measure yields a 39% graduation rate for Black male19 year olds in 1999 and 30.8 % for Black males in 1998. According to Mishel and Roy, these rates present “an accurate picture of high school completion by entering 9th graders.”

Although in each case these golden rates, based on tens of thousands of students, yield rates far below the NELS “on time” estimate based on just 1,456 Black students, it is theoretically possible that we would find very very high rates for Blacks elsewhere to balance out the reliable data from two of America’s largest urban districts and one Southern state. But if that were the case we would have heard about such unusual districts by now. Or perhaps New York, Chicago and Florida are much much worse than any other districts and states as Dr. Roy asserts.

In our series of reports, including Dropouts in California, and Book, DROPOUTS IN AMERICA, The Civil Rights Project has examined studies of individual schools by researchers Balfanz and Letgers at Johns Hopkins, recent longitudinal student record data from Los Angeles, detailed district enrollment data from every state analyzed by Swanson of the Urban Institute, and research from many other scholars. Based on the collection of research, we argue that there is a very serious problem in many schools and districts, especially poor urban schools with high levels of racial isolation. Districts in crisis can be found in every state of the nation.

Not only do a multitude of studies from independent scholars across the country yield results that are consistent with the 50% estimate, but Mishel acknowledges that Swanson’s method yielded nearly identical results as Chicago’s “golden” longitudinal record data, the same sort of data that distinguished the NELS as gold. Moreover, our most recent report on the crisis in California revealed that in Los Angeles Latinos and Blacks combined have a 48% graduation rate. Once again, this was based on “golden” longitudinal individual student identifier data analyzed and reported by
Dr. Julie Mendoza.  I have no reason to doubt that our 50% estimate would prove to be fairly accurate if we had the golden data we all desire from across the nation.  While neither The Civil Rights Project, nor Mishel and Roy, can say with absolute precision just how bad the situation currently is, we assert that we have ample evidence of an urgent problem and that this crisis is real and needs action now.

Roy, June 2:

Before getting into details it is important to clarify the issue. The NELS shows a black graduation rate of 75% while the Swanson-Urban Institute methodology that Dr. Losen favors shows a 50% rate. It seems that Dr. Losen is questioning the sampling of the NELS and saying that a failure to properly sample blacks leads to an overstated graduation rate. We have not found any scholar who claims that there is any sizeable sampling problem in the NELS (we have asked Losen in prior private communications to provide us people we can talk to about this, as we take such issues seriously) and no one has ever made an argument that sampling problems could explain a 25 percentage point difference in the black graduation rate. If there were such a problem then the vast literature based on the NELS, at least as regarding blacks, would need to be disregarded.

Now to the details. The NELS:88 sample began with 25,000 students – 24,599 to be precise. Of these, 16,321 were white, 3,011 were black and 3,177 were Hispanics (see A Profile of The American Eighth Grader: NELS:-88 Student Descriptive Summary, page 103, Appendix A, A-3). The weighted percentages were 71.4% for whites, 13.2% for blacks and 10.4% for Hispanics.

So the actual number of black students in the sample was over 3,000 rather than 1,456 as Dr. Losen claims. The features of the NELS that make it unique and the best dataset to calculate graduation rates are – a large nationally representative dataset; longitudinal tracking of 8th graders till their high school graduation and beyond; multiple rounds of interview with sample members, resulting in low levels of attrition rates from the sample; and checking of graduation status against actual school transcripts to minimize self-reporting bias.

Anyone interested in knowing more on the methodology or results from the NELS should check up the NELS:88 webpage at the Department of Education website, which lists all the different publications and products. People should also check the new report by Cliff Adelman at the U.S. Department of Education, The Toolbox Revisited: Paths to Degree Completion From High School Through College.

Dr. Losen also incorrectly asserts that the NELS sample is not representative because Black youth are disproportionately incarcerated, homeless or have disabilities. Neither of these has any significant effect on the estimates, even the minority ones. First, incarceration is not a problem because we are talking of tracking students beginning in their 8th grade. As the U.S. Department of Justice reports, overall only about 0.2% of all state prisoners were under age 18 – see Bureau of Justice Statistics bulletin for April 2005. Accounting for incarcerated youth is more important when we talk about educational attainment of people aged 25-29 years old or following people in longitudinal surveys even if they enter prison. However, our results from the 2000 census clearly show that incarceration biases the results only for black males, and even then the effect is not large, certainly not enough to believe that there’s a 50% graduation rate. We either include incarcerated populations or adjust for their presence when we examined household surveys.

A second widely used and respected survey – the National Longitudinal Survey of Youth conducted by the U.S. Department of Labor – actually includes the prison population in its sampling framework and comes to the same conclusions. The NLSY shows that the graduation rate with a high school diploma (excluding GEDs) was 82.2% for people aged 20-22 in 2002 – 85% for whites, 75% for Blacks and 76% for Hispanics (see Table 3 in our book).

As I mentioned in my earlier response, the initial round of NELS:88 had under-sampled that portion of the special education population that is most severely mentally or physically disabled. However, coverage of this population was improved in the first follow-up by the fact that in the base year ineligibles study, nine of the 23 students excluded because of physical barriers to participation, and 140 of the 322 students who had been excluded because of mental barriers to participation, were reclassified as eligible. Similarly, 49 of the previously ineligible sample members were found to be eligible in the second follow-up followback study of excluded students; of these 49 excluded students, 44 had been previously excluded due to mental disability and 5 for physical limitations (link).

A recent Department of Education study (see article in Education Week) finds that there has been a significant improvement in school completion and academic performance of youth with disabilities. Comparing the 1987 cohort with the 2003 cohort, the report shows that school completion rate of youth with disabilities increased and the dropout rate decreased by 17 percentage points during this time period. About 70% of the 2003 cohort had completed high school by the time the data were collected. The rate of postsecondary education participation by youth with disabilities more than doubled over this time.

Dr. Losen argues that “researchers who use NELS agree that this number (black on-time graduation rate of 63%) should be adjusted downward somewhat because of bias in the sample of Blacks”. However, we have not seen any researcher using the NELS argue this before – it would be useful to refer to the particular researchers and their studies. As stated above, we have previously asked Dr. Losen for articles we can read or people to talk to who have such views of NELS.

Dr. Losen also argues that because the NELS:88 graduation rates were for the Class of 1992, they are not relevant now and in particular, cannot be compared to recent estimates. However, the Greene and Swanson estimates of graduation rates for the early ‘90s are very similar to their estimates for recent years. For example, for the Class of 2001, Greene’s numbers (Tables 8-11) are as follows – Total 70%, Whites 80%, Blacks 55%, Hispanics 50%. Swanson’s numbers (page 20) for the same graduating class are 68% for Total, 75% for Whites, Blacks 50%, Hispanics 53%. So given that Greene and Swanson estimates for the early 90s were far off from the NELS estimates, they are likely to be significantly off in recent years as well.

Dr. Losen incorrectly claims that we “ignore the fact that Chris Swanson’s graduation rate estimate was created purposefully to be used as an “on time” graduation rate”. However, it is impossible for Swanson to compute an ‘on-time’ graduation rate because the available data only provides all of the diplomas granted each year- early, on-time, and late. This is something Greene readily acknowledges and Swanson never challenged us on this p

oint at a seminar at the Urban Institute. If Losen wants to believe that the graduation rates are on-time because Swanson has labeled them as such, that is his prerogative. However, we would like to know how that late diplomas are distinguished from others in the calculation.

Dr. Losen wrongly claims that we believe the state and city estimates on graduation estimates to be the “gold vein”. As I mentioned in my earlier response, while working on graduation rates based on state and city student longitudinal data records, we became aware that without a true national-level student identifier system, it is difficult to convincingly take account of the ‘leavers’ problem – that is, whether students who leave a school district or state really do enroll somewhere else, or they drop out. There are often other issues regarding these estimates – mostly related to how different states define different aspects of enrollment and graduation. Hence we decided to put the discussion in an appendix. Note that these problems are not shared by the NELS, which is a national sample.

Our purpose in bringing the state and city estimates to the front was to show that estimates based on student longitudinal records- the same data relied on for enrollment and diploma counts, organized differently and better- yield much higher (ten to fifteen percentage points) graduation rates – both overall and for individual races as in Florida and New York City – than the measures used by Dr. Greene and Dr. Swanson for Florida and New York City. Even in Chicago, where the differences in levels is not very high, Swanson’s CPI and Greene’s measure show quite different trends compared to those based on student longitudinal records, and hence are inappropriate. It is especially bizarre for Losen to claim that the low graduation rates observed in New York City and Florida support his view. In fact, the graduation rates are unacceptably low. However, if Losen relies on Swanson’s and Greene’s computations nationally, and we show that their estimates for these areas are substantially less than better measures, it is hard to understand how that supports Swanson’s and Greene’s national estimates.

Finally, Dr. Losen claims that “a multitude of studies from independent scholars across the country yield results that are consistent with the 50% estimate”. This is incorrect, in that the studies are only done by a handful of researchers – Dr. Jay Greene, Dr. Chris Swanson, Dr. John Warren at the University of Minnesota – and a few others. All these studies rely solely on the administrative data on enrollment and diplomas, and moreover, they introduce population or other adjustments which impart serious bias to their estimates, particularly for minorities. There are estimates from Walt Haney using the same underlying data that show far higher graduation rates. It is not so much that these researchers rely on the CCD enrollment and diploma data: it is the choices they make in using the data that lead them to vastly understate graduation rates, particularly for minorities. It is interesting to note that when Dr. Warren claimed the superiority of his methods to those of others including Greene and Swanson, he did so by virtue of the fact that his estimates came closest to the NELS:88 estimates (see the section titled Validating the ECR in his article at the Education Analysis Policy Archives, page 18).

Losen, June 12, 9:41 a.m.:

A third fundamental flaw of Roy’s and Mishel’s is that they are myopically concerned with only one thing, data analysis, and would wait until another generation of poor and minority youth are lost before they would consider serious action on a national scale. This paralysis by analysis is deeply disturbing to those of us whose work is dedicated to remedying the crisis faced by poor Black and Latino youth in far too many schools and districts across America.

Typical of our exchange, is this comment of Dr. Roy’s, "It is especially bizarre for Losen to claim that the low graduation rates observed in New York City and Florida support his view. In fact, the graduation rates are unacceptably low."

What Dr. Roy fails to realize is that my view, and that of The Civil Rights Project at Harvard University, is that there are unacceptably low graduation rates for Blacks, Latinos, Native Americans and other disadvantaged youth. If the "on time" rate back in 1992 was 63.2% and not 50%, that would still be a crisis. But the fact remains that where we have gained access to tens of thousands of longitudinal student record data (from the current century) those individual state and district “on time” rates have been more consistent with a 50/50 Black and Latino “on time” estimate rather than either the 1992 rate of 63.2% or the 1994 rate of 74%, that latter of which Mishel and Roy would have people believe more accurately describes the condition of education for Blacks in 2006.

Mishel and Roy don’t have data for evaluating schools, districts or states. Nor do they have much to crow about if they focused on “on time” graduation rates. So they revert to apples and oranges comparisons of “on time” enrollment based estimates to extended year survey based estimates. That leaves folks wondering as to why they treat the “on time” information, even the NELS 63.2% rate, as irrelevant to the portrayal of a crisis for Black students in our schools?

Despite Roy’s assertion,The Civil Rights Project does not rely on any one source, or just enrollment data to assert there is a crisis. I’ve already said that an “on time” rate of 63.2% would suffice. In addition, not all sampled data support their insistence that NELS 88’s rates from 1992 or 1994 are the most accurate indicators for 2006. Consider, for example, that according to the 2005 NAEP data, nearly half of all Black and Latino students in grade 8 scored “below basic” in reading. I would not be surprised if most of these non-readers fail to graduate “on time” with a diploma. Keep in mind that below basic is two steps below proficient. This is just more evidence that in 2006, our primary concern about a crisis for Black and Latino students, and the quality of schools and districts they attend, is not exaggerated.

So the most important issue, obscured by Mishel and Roy, is that our public schools, K-12 are not providing adequate educational opportunities for these minority students, while subjecting them to unjust high stakes exit exams and encouraging students to dropout rather than stay in school. The “on time” rate is more important for evaluating our schools and districts, especially in the face of these problems and the disproportionate burden they impose on poor and minority youth.

We know the NELS has great value for some purposes, but ultimately it is also a survey with sampling biases and other shortcomings. In our book Dropouts in America: Confronting the Graduation Rate Crisis, Phillip Kaufman writes, “Since NELS is a sample survey, it is subject to the same potential for bias due to non-response and undercoverage bias that the CPS has.” (p. 119-120) He also points out that that the NELS was designed, “to provide national estimates of dropout and completion rates and, except for very large states, cannot provide statistically reliable state estimates or any school district estimates.”

We join Mishel and Roy to the extent that we aren’t happy with the current lack of clear and transparent data, an

d we both call for much better tracking of individual students to get a much better understanding. Apparently they don’t believe the "on time" graduation rate is the most important for evaluating schools and education reform policies. I’m disappointed that Mishel and Roy give no significant time whatsoever to the “on time” rate of 62.3% when they boldly insist there is no crisis. Instead they submerge that piece of information and repeatedly highlight the least useful estimates for describing the state of education for Black students, that’s the 6 year, 74% rate based on a small national sample of students where fewer than 1,456 of the surveyed responders actually had their student records confirmed. I strongly urge Mishel and Roy to consider the reaction of the White middle class if they learned their "on time" graduation rate was 63.2%.

On another more technical note: Roy claims the number in NELS is actually 3,000. But after attrition, according to a reputable NELS researcher, for Blacks, the number in the sample that started at about 3,000, was reduced for a variety of reasons to 1,456 by 1992 – at least as far as the “golden” group of students for whom most had responses that were checked against actual student records. So while Roy insists I made an error, and skirts over the attrition in his blog response, his book shows how the original 25,000 in the sample shrunk to about 20,000 or less, as detailed on page 73 of Roy’s appendix. If Roy insists the gold standard is based on 3,000, I’d accept those numbers if he takes an oath that there was no attrition in the Black sample and that nearly all of the 3,000 original Black students not only remained in the sample, but had their transcripts checked to confirm their survey responses in 1992 or 1994. What I’d really like to know from Roy is just how many of the 3,000 original Blacks had their survey responses checked against their school records in 1992. That number, which would be the only “golden” data they have for Blacks, is likely even less than 1,456.

Instead of acknowledging the weaknesses of the data they use, Mishel and Roy repeatedly prefer to mislead the public by posting op-eds that call the crisis “exaggerated.” To their credit, you can find the data that undermines their assertions in their own book, but rather than insisting that honorable researchers like Swanson and Warren are exaggerating, they should give equal time to the fact that that "on time" graduation rates are unacceptably low for Blacks and Latinos according to NELS, and by any method, and that the survey data they use offer little current or specific information on the current health of increasingly racially isolated schools and districts serving predominantly poor Black and Latino youth.

In summary, Mishel and Roy don’t have great data, and give far too little attention to “on time” graduation rates. Ultimately, they miss the point. For Black, Latino and other disadvantaged youth, there is more than enough information depicting low graduation rates to call the situation a crisis and urgently demand an improvement in educational opportunities.

Larry Mishel, June 19, 8:18 a.m.:

I am taking the response to Dan Losen over from Joydeep. This is difficult to do for several reasons.

Personal Attacks
First, Losen’s tone is way off the reservation and his personal attack is shameful. Consider how he starts his last contribution, saying Joydeep and I:

“….are myopically concerned with only one thing, data analysis, and would wait until another generation of poor and minority youth are lost before they would consider serious action on a national scale. This paralysis by analysis is deeply disturbing to those of us whose work is dedicated to remedying the crisis faced by poor Black and Latino youth in far too many schools and districts across America.”

I am not sure what his basis is for saying we do not favor any serious action on a national scale, that we see no problems or that we do not care about disadvantaged students? We certainly have stated that there are too many dropouts, especially for low-income and minority students. We highlight the existing racial and ethnic gaps in graduation and show the black-white gap is even larger than what the regular CPS data show (the Hispanic-white gap, however, is smaller than previously thought, though still large). Is it because we do not scream about an undifferentiated across-the-board high school dropout ‘crisis’? As for policy implications, you can read below what we think they are. And, has Dan Losen cornered the market on concern for minority youth? Puhhhleeeze. Dan is only embarrassing the worthy organization, the Harvard Civil Rights Project, with which he’s associated. At EPI, which I lead, we speak truth to power every day of the week. We certainly do not accept any lecture by Losen on our commitments or motives. Because of Losen’s tone and personal attacks this will be our last attempt to intellectually engage him—there will be no more postings from us.

A Useful Engagement
A second difficulty is that Losen does not practice empirical social science (as far as we can tell, he does not do any data analysis) so he does not discuss measurement issues in ways that allow useful engagement. How could that be done? For starters he could identify his critique of our evidence by stating what he thinks is wrong with the data we rely on or how we analyzed the data. He could then identify the size of the error or bias result from any flawed data or methods. He might even acknowledge weaknesses in the methods he endorses and try to reconcile estimates of graduation rates. Losen, rather than engage in this manner, dismisses empirical findings and whole datasets based on the simple assertion of a bias. That is, Losen blasts out critiques without any assessment of whether they matter worth a darn. They do not.

Joydeep tried to jumpstart a useful discussion at the start of his last contribution:

‘Before getting into details it is important to clarify the issue. The NELS shows a black graduation rate of 75% while the Swanson-Urban Institute methodology that Dan Losen favors shows a 50% rate. It seems that Losen is questioning the sampling of the NELS and saying that a failure to properly sample blacks leads to an overstated graduation rate. We have not found any scholar who claims that there is any sizeable sampling problem in the NELS (we have asked Losen in prior private communications to provide us people we can talk to about this, as we take such issues seriously) and no one has ever made an argument that sampling problems could explain a 25 percentage point difference in the black graduation rate. If there were such a problem then the vast literature based on the NELS, at least as regarding blacks, would need to be disregarded.”

As far as I can tell, Losen has never made any claims that his alleged problems with the NELS can explain the difference between a roughly 75% graduation rate in NELS and the 50% rate he seems to be comfortable with. Nor has he identified any researcher or research paper that has made a serious (actually, any) critique of NELS. It is curious that Losen cites Kaufman saying the NELS can’t produce state or local estimates. First, Kaufman actually identifies the NELS as the best source of data. Second, the point Kaufman is cited for- NELS doesn’t produce local estimates- is not an answer to why Losen’s preferred method for analyzing local data (the Swanson measure) yields a 25 percentage point lower graduation rate for blacks at the national level than that of the NELS—if Swanson’s methods are so far off at the national leve

l they can’t possibly be worth anything at the local level.

What we have offered
We have compiled or analyzed all of the data we could find, assessed it, and tried to correct biases that we could identify (accounting for high and rising incarceration of black men; including the military and institutional populations into the analysis; develop estimates of GED receipt from ACE data; examine sampling coverage issues and more). We have presented the best estimates from household survey data, including the decennial census, and from national longitudinal data such as the NELS and the NLSY. We find that all of these other data show graduation rates for minorities that are far higher than those found by Greene and Swanson—the conventional wisdom.

We have specifically examined the methods used by Greene and Swanson in their analysis of enrollment and diploma data and have shown how specific choices they make in using the CCD lead them to seriously understate graduation rates, especially for minorities (in our report and in a Q&A).

Swanson’s measure is the one Losen likes the best. We have shown how Swanson’s use of ninth grade enrollment as his estimate of those entering ninth grade is wrong: around 20% of minorities in ninth grade have been held back and are not ‘entering’ ninth graders. The consequence of using ninth grade enrollment instead of eighth grade enrollment (to avoid what is called the ‘ninth grade bulge’) is to artificially lower minority graduation rates by 12-13 percentage points. We have heard no rebuttals of this critique from Losen or anyone else. We think this is because there is no reasonable response.

So What?
Does it really matter whether the overall graduation rate is 83%, as we estimate, or the now conventional number of 67-70%, as Greene and Swanson (and the National Governors Association) claim? After all, estimates of lower dropout rates still represent a greater than desired population of dropouts.

Our findings do not negate that there are definitely places and populations where dropping out is far too frequent – in some, horrifyingly frequent. There are definitely dropout problems that require comprehensive action so as to improve students’ life chances and to address class and race/ethnic gaps in graduation. But there is also an important value in getting the facts correct and especially in recognizing improvements over time. Touting a 50% graduation rate for minorities is not only factually incorrect but can also too easily encourage mischaracterizing the black student population as too hard to reach or as being disinterested in education. Identifying that three of every four black youths get a diploma and another 13% receive a GED is both factually correct and appropriately acknowledges the striving and persistence of black students.

Artificially low graduation rates also lend themselves to supporting a misdirected across-the-board indictment of schools. Inaccurate characterization of success in high schools can lead to misguided or wrongfully-targeted reform efforts. It can lead to efforts similar to sending homeland security funds to Montana and Idaho rather than to New York and Washington. Recognition that there has been progress in improving graduation rates might lead to an examination of what some schools must be doing right, practices that might be supported and extended to other schools. A misguided across-the-board indictment of schools might also lead to equally misguided radical reforms that could be harmful.

In truth, the dropout problem is concentrated in about 20% of our high schools. If we examine graduation rates by socio-economic status, we see that there is only a 3% dropout rate for the upper three-fifths of students, by socio-economic status. (This calculation includes GED recipients as high school completers). Yet, the bottom fifth had 27% who failed to complete high school in any way, and only 62% who obtained a regular diploma. The next higher fifth had 13% who failed to complete high school. So, there are definitely populations that are not successfully completing high school and getting diplomas, which is bad for those children and bad for the country.

Interestingly, among students in the lowest socio-economic fifth, black students have the highest probability of completing high school, greater than that of low-income whites and Hispanics. This is another reminder of how intertwined race and class are in our society.

To the extent we have a dropout crisis; it is primarily a crisis of youths at the bottom of the socio-economic scale, regardless of race (although race is clearly the key factor in allocating students to the bottom fifth and plays a far-too important role in our society). To address their problems we need comprehensive efforts to improve these children’s schools, alternative programs and the building up of second-chance systems. But we also have to think about the lives of these students outside of school and the disadvantages they faced even before they ever got to school (there are huge education disparities by income and race when students start kindergarten!). This means fighting poverty through better jobs and wages, providing early childhood development programs, creating stable housing, providing health care and fighting crime. That is certainly a national program we think is worth fighting for every day.

Losen, June 28, 1:48 p.m.:

Apples v. Oranges

I’m happy to end this debate, but here is my response to Mishel: I find this debate intensely frustrating in tone and substance because I admire Mishel’s other work. I take serious issue with their responses to me in this blog, treatment of the wide range of work in our book Dropouts in America, and their misstatements and omissions of relevant facts in their published work on this particular topic.

1. Mishel says, “For starters he could identify his critique of our evidence by stating what he thinks is wrong with the data we rely on or how we analyzed the data. We have not found any scholar who claims that there is any sizeable sampling problem in the NELS (we have asked Losen in prior private communications to provide us people we can talk to about this, as we take such issues seriously).”

Social Scientists do criticize NELS. Kaufman, now deceased, raised serious concerns about the NELS and all methods. In our book, Dropouts in America: Confronting the Graduation Rate Crisis, Kaufman goes into detail to unpack numerous concerns regarding the bias in both the CPS and the NELS, emphasizing the racial bias. He states: " NELS and CPS may both have serious coverage problems in their realized samples. Minority students and schools may be so seriously underrepresented in their sampling frames that it leads to great systematic bias in both surveys." (p.125). I think most readers would regard “GREAT SYSTEMIC BIAS” as a “sizeable” reason to proceed with caution.

Besides the work of the late Kaufman, Gary Orfield, Harvard Professor at the Graduate School of Education and the Director of The Civil Rights Project, has asserted a significant problem with racial bias in the NELS but has no interest in this wrestling match. Chris Swanson, one of the most prominent researchers using an enrollment based estimate, actually cut his teeth as a researcher on the NELS data set working with this data for five years at the National Opinion Research Center, the organization most closely associated with the survey development and use. He has raised numerous issues with the data, and I a

nd others have made Mishel aware of his objections. Robert Balfanz, Senior researcher at Johns Hopkins specializing in high school reform has also told Mishel of NELS bias.

But Mishel erroneously asserts: “As far as I can tell, Losen has never made any claims that his alleged problems with the NELS can explain the difference between a roughly 75% graduation rate in NELS and the 50% rate he seems to be comfortable with. Nor has he identified any researcher or research paper that has made a serious (actually, any) critique of NELS.”

So in addition to Mishel’s failure to see what Kaufman had written, or acknowledge what I and others have told him, part of the problem lies in Mishel’s contingency, that “nothing can explain the difference between a roughly 75% graduation rate in NELS and the 50% rate he seems to be comfortable with.” In other words, for Mishel, nothing short of explaining away the entire difference between a 50% "on time" rate, and a 75% extended years estimate will do. This is a very weak defense dependent on the very issue that is in dispute. Mishel and Roy assume the validity and accuracy of their 75% benchmark, an extended years estimate, not an “on time” estimate. That is part of what we are debating about. I think that there is likely consensus among education researchers that the 4 year enrollment based estimates should not be compared to an extended years estimate. I’ve yet to read any researcher support Mishel’s comparison of a four year estimate with an extended year estimate. In a recent New York Times piece, Claudia Golden, a Harvard economics professor who does her own education research weighed in on the debate between Mishel’s assertions and those based on enrollment data. “They’re using two different types of data, and each has its own problems,” she said. “The truth lies somewhere in between.” So Mishel and Roy can ask for more names, but if they insist on an erroneous method of comparison, I doubt we can resolve the misleadingly large gap that Mishel has constructed.

2. Mishel says, “I am not sure what his basis is for saying we do not favor any serious action on a national scale…”

My basis for saying this comes from the conclusion to their book which reads as follows:

"Unfortunately, the only data available at the school district level are the CCD data that we judge to provide inaccurate estimates of high school completion. This suggests that measures of high school completion at the school-district level WILL PROBABLY HAVE TO WAIT until data systems that track individuals are available…. Doing so will require a national system and sufficient resources for schools to track ‘leavers.’ WE ARE SKEPTICAL THAT THIS WILL HAPPEN ANYTIME IN THE NEAR FUTURE." (Emphasis added). The rest of the conclusion calls for more research, better assessment of bias in the CPS and better understanding of the role of GEDs. Laudable goals, but from my perspective these do not qualify as "serious action on a national scale." To their credit, in other places they do say, in reference to their estimate of a 25% dropout rate that "we must fix this glaring social problem." However, I don’t think that my critique of their underwhelming conclusion was unwarranted, nor cause to feel ashamed.

As a civil rights advocate who works with civil rights leaders at the national, state and local level, I can guarantee Mishel that folks are concerned when the state of education for poor Blacks gets masked over by national statistics or suggestions that we can’t do much without more research and better data. For the record, the “paralysis by analysis” is a line from a speech about graduation rates given recently by John Jackson, the National Policy Director for the NAACP.

3. I do not doubt the good intentions of Mishel and Roy. I do accept that they care. I’ll choose not to escalate the personal attacks. I would instead agree with Mishel’s concluding blog paragraph. I too see the need to address poverty and all of the related problems. These are critical pursuits. I also do not consider schools as the only arc to navigate these troubled waters. Unfortunately, Mishel’s published report doesn’t delve into the kind of solutions that would help in the near future and gives no specific guidance on how to address the crisis in poor urban school districts in a serious way. I’d add that the fact that we urgently need to solve numerous problems in many aspects of society need not distract educators from addressing a crisis in our public schools, especially for minority students. Our work at The Civil Rights Project has highlighted how schools, districts and states have failed to acknowledge the graduation rate crisis for many years, choosing to focus on test scores alone. Research suggests that if we can improve graduation rates there would be substantial benefits to all involved.

4. Mishel and Roy have yet to explain why it is when schools are evaluated on a “four year” rate, they focus only on the extended years graduation rates? On “The News Hour,” I was glad to see Mishel provide a range for Black graduation rates, yet he still emphasized the 75% figure. I’ve repeatedly asked Roy and Mishel to justify the emphasis on the extended years survey rate when the primary issue is the health of schools and districts. Over and over I ask the question, but they have yet to provide an answer. The law requires an “on time” or four year estimate. We should know about other rates, too, and I am all for second chances, but we should provide the supports and resources to schools and districts to improve the four year rate, and that rate should be central to evaluating the health of our schools and districts.

5. Mishel and Roy err in implying that The Civil Rights Project relies on just one data set. Our book looks at the issue from many viewpoints and has data from Swanson, Balfanz, Hauser, Kaufman, Rumsberger, Allensworth, Haney, McPartland and more. The book is far more comprehensive than a discussion of reporting accuracy. For arguments sake, I have agreed to assume the accuracy in 2006 of the NELS 88 survey, which provided a 63.2% "on time" graduation rate for Blacks, nationally, for the Class of 1992. The central point I make is that by many measures, including NELS 88’s "on time" estimate there is a serious crisis.

6. I raise the Florida, New York and Chicago longitudinal data deemed "much better" by Mishel and Roy, as evidence that the crisis is not “seriously exaggerated,” an assertion that Mishel and Roy have failed to prove. I point out that the "on time" rates in these locations cry out for urgent action. While they try to use these data, unsuccessfully, to cast doubt on one particular set of estimates, they ignore the fact that Black "on time" graduation rates of between 30.8% and 55% derived from these data suggest a true crisis. While these limited data don’t prove that either their 63.2% "on time" estimate or preferred "74%" estimate is wrong, it should make Mishel and Roy think that there could be many many schools, districts and states closer to 50%. Given the Florida, New York City and Chicago data, if the 74% rate they assert describes reality for the nation, then their national rate, at best, masks over the far harsher reality in too many districts across the nation. The other possibility is that their sample of 1,041 Black respondents for the Class of 1992 doesn’t accurately describe the Black graduation rate in 2006. Given that these three very significant sources of data do not come even close to their estimate, I’m surprised that they refuse to acknowledge the possibility that their estimates are way off. The Civil Rights Project has been fighting this battle for over five years. We didn’t focus on this issue to advance a particular estimate, instead w

e believe in using the best evidence available from multiple sources to show that this urgent problem has gotten far too little attention until very recently.

7. Contrary to Mishel’s assertion, I do raise fair doubts about Roy and Mishel’s methods and conclusions. Swanson and Greene have also criticized his methods in their published response to Mishel’s opinion piece in Education Week. I have checked my assertions with senior education researchers, and don’t need a Ph.D to raise these important questions. They assert that the NELS estimates from 1992 and 1994 gives them an unimpeachable gold standard, and that therefore they had the real truth on their side. Roy has also asserted that my numbers were in error. But, according to a highly regarded researcher, there were 3,000 Black students in the original sample, but the 1988-2000 cohort is much smaller. Of the roughly 1,456 in the sub-sample, there were only 1,041 black respondents who actually had their transcripts checked. This 1,041 constitutes the “golden” cohort that made up the "on time" graduating Class of 1992. Some of those in the sample were in private school. While I can’t say with precision how far off the numbers are, I can say that there are serious problems with using this sample of public and private students for the Class of 1992 to assert factual knowledge of the public school graduation rate for Blacks in 2002. Mishel’s is far from an ideal method. The highly esteemed education researchers I have checked in with think that the questions I raise about the representation of the verified students, concerns about bias in the sample, the contrary evidence from Florida, New York and Chicago, and aligning years for comparing estimates are all legitimate issues and reason for doubting the accuracy of NELS.

8. What is the affect of the bubble? I have challenged Mishel on this in private. As far as asserting there is an urgent crisis, high grade retention is more evidence of problems with high schools. While the bubble has the potential to deflate reported rates, it also has the potential to inflate graduation rates, or it can have no impact.

Simply put, if the retained 9th grade students, and non-retained 9th grade students both had a 50% diploma rate, the “on time” rate would have no significant impact from the bubble. If we accept for arguments sake Mishel’s assertion that about 25% of Blacks are retained in grade 9, we can derive their success rate by comparing the on time NELS with the extended years NELS diploma rates of 63% and 74%, a difference of 11%. In other words 11/25 retained Blacks earned a diploma if NELS is correct. That is a 45% success rate for the Black students in NELS that were retained at grade.

So if we assume that Mishel is right about the retention rate for Blacks, and that based on NELs there is a 45% success rate, then the bubble impact would be close to 0 where the four year enrollment estimate of graduation rates was 45%. If we use the enrollment based 4 year estimate of 50% the bubble would have a only a minor deflationary impact. The only way the bubble has a significant impact is if a much larger percent of the retained students fail to graduate. But that would call into question Mishel’s extended year estimate of 74%.

The point is, most researchers agree that in the real world, where retention rates are high, and dropouts among retained students are also high, the bubble would likely deflate graduation rates, but only between 1% and 5% under most circumstances. Mishel makes unrealistic assumptions, not supported by his NELS 74% estimate, to exaggerate the impact of the bubble. Folks like Robert Balfanz, of Johns Hopkins, who do both empirical work with national data sets and qualitative analysis of struggling urban schools, have tried to explain to Mishel and Roy that the bubble can inflate or deflate the graduation rate, and that the enrollment based estimates are far closer to the reality they see in the data as well as in their experience, first hand. In any event, The Civil Rights Project is one of the leading advocates for more accurate graduation rates and has worked with numerous others to develop model legislation on this point.

Conclusion: Ultimately, the technical debate returns to whether Mishel’s assertion of exaggeration is true, or whether he is the one exaggerating the difference between estimates. I assert that the estimates based on enrollment are not precise but very informative. Roy and Mishel arrive at a very different understanding, but mostly because they compare the apples of school enrollment data with Mishelian oranges, that take many more years to ripen.

Check back for a response from Roy or Mishel.