Over at This Week in Education, John Thompson pens a post about his perceptions of bias in Washington Monthly’s recent report about New Orleans schools. As there’s a big conference going on in NOLA right now on this very topic (which I sadly could not attend (though perhaps not so sadly given it’s June in New Orleans), but you should watch the live streams tomorrow–link above), I figured I’d give this one a thorough read and respond in a measured way. So that’s what I’m doing here. Again, I don’t think of myself as much of a choice fan–it’s far from my main interest in either policy or research–but the logical and rhetorical problems with this kind of writing are, to my eye, so manifest that they really need to be addressed.
John starts by praising Caitlin Emma’s reporting on New Orleans, quoting her saying there’s no proof the New Orleans model works . He contrasts Emma to Osborne’s report, which concludes “the Crescent City’s schools have produced what some experts believe to be the most rapid academic improvement in American history,” which he goes on to trash in extended detail.
His critique begins: “Osborne starts with the dubious claim by the pro-charter CREDO that charters receive less per student funding…”
I’ll stop you right there and remind you that the first CREDO report found that charter schools underperformed traditional publics. Only the recent reports have found charter improvement. So unless by “pro-charter” he means “uses advanced statistical methods and concludes that charters marginally outperform traditional public schools in recent reports but not in earlier reports,” this characterization of CREDO is absurd .
Next, Thompson says:
He cites the objective researcher, Douglas Harris, who says that NOLA undertook “the most radical overhaul of any type in any school district in at least a century.” But, Osborne cites no evidence by Harris or anyone else that the New Orleans radicalism can work in a sustainable manner or that it could be scaled up. Instead, he devotes almost all of his article to praising true believers in unproven theories on school improvement.
I guess this is dancing around what the article actually said. What the article said was things like the following:
Before Katrina, most public schools were terrible. In 2005 the city ranked sixty-seventh out of sixty-eight districts in Louisiana, itself a low performer compared to other states. Last year, New Orleans was forty-first out of sixty-nine school districts in Louisiana.
Before Katrina, some 62 percent of students attended schools rated “failing” by the state. Though the standard for failure has been raised, only 7 percent of students attend “failing” schools today.
Before Katrina, only 35 percent of students scored at grade level or above on state standardized tests. Last year 62 percent did.
Before Katrina, almost half of New Orleans students dropped out, and less than one in five went on to college. Last year, 73 percent graduated from high school in four years, two points below the state average, and 59 percent of graduates entered college, equaling the state average.
And according to a 2015 CREDO study, between 2006 and 2012 New Orleans’s charter students gained nearly half a year of additional learning in math and a third of a year in reading, every year, compared to similar students in the city’s non-chartered public schools.
Because the OPSB was only allowed to keep schools that scored above the state average, the failing schools were all in the RSD. In the spring of 2007, the first full school year after Katrina, only 23 percent of RSD students tested at or above grade level. Seven years later, fully 57 percent did. As Figure 1 shows\, RSD students in New Orleans have improved almost four times faster than the state average.
Now, you might argue with those statistics–that they’re based on creaming, or that the poorest of the poor have been driven out of NOLA, or some other critique (though my read of the evidence on this is pretty clear). But they’re not no evidence. They’re actually quite a bit of evidence. And other work by folks like Josh Angrist (arguably one of the strongest methodologists around) finds big effects of takeovers in New Orleans, too. That looks relatively “sustainable” to me, a decade after Katrina. Perhaps it wouldn’t work elsewhere, but it’s not nothing.
Or maybe, John says, it is nothing! Maybe the test score gains identified by CREDO and mentioned in Osborne’s study are meaningless.
Improved test scores in such schools might or might not be meaningful. In a situation like that, is there any reason to believe that increased test scores mean that more learning occurs when all stops are pulled from test prep in a C school, as opposed to a D or F school? Rounds of such remediation are bound to improve metrics important to adults, but do they help or hurt the children who endure them?
Rhetorical questions are often a sign that there’s not a good argument being made, and this is no exception. Because, again, if the charter schools had underperformed the district schools on the tests, John would be using it as fodder (I’ve seen enough of these pieces to know). And at least the serious ones among us know that, while test scores do not measure everything, they also do not measure nothing. And even if they did measure nothing, the article says graduation rates are way up, too (and I’ve heard attendance as well)!
Thompson says that he, for one, is withholding judgment on NOLA’s reforms until Doug Harris comes out with ERA’s report on the city. I, too, think Doug is an excellent researcher who does not have an agenda other than getting the facts right. If the facts come back that charters are outperforming traditional public schools in New Orleans, you can bet your bottom dollar there won’t be a followup post about how the reforms were right all along. [EDITED TO ADD: Doug gave his keynote presentation at the ERA conference today. The conclusion was positive impacts on test scores of .2 to .4 standard deviations.]
I’ll conclude by quoting my two favorite paragraphs from the piece. I’m not going to tear them apart, because they do that to themselves. But I will note something that has bothered me for quite a while. John’s blog post is on This Week in Education, a site run by Alexander Russo. Russo also runs The Grade, where he critiques education media (he also does this on Twitter, and I think it’s quite valuable in both contexts). I have never understood, and I never will, how these two things can be reconciled. To my eye, Thompson’s posts betray an agenda that will not change with any amount of research evidence. How this kind of writing ends up in a prominent position on the blog of a media critic boggles my reformy mind:
Osborne doesn’t acknowledge the much more likely scenario. Under such a Social Darwinian system, survival will go to the best of the test score fabricators. Market-driven reformers will do what they have done best since NCLB imposed primitive bubble-in accountability. They will treat children as test scores. Or worse, they will treat them as dollar signs. Either way, competition-driven reform will likely continue to damage the poorest children of color.
More selective charters might or might not try to offer a holistic education to the more motivated students. Those that do will be showcased to spin corporate reform with a more human face. Charters with the most challenging students will continue to do what it takes to survive, and twist the facts to non-education reporters and politicians.
 Editor’s note: if Emma were to write something a year from now that concluded the reforms had produced improvements, he’d undoubtedly cast her aside like day-old bagels.
 And again, this is a case where if CREDO had concluded that charters were underperforming, he’d be gloating and touting those findings.