Anecdotal Evidence, True or False, Can’t Help Us

It comes as little surprise to me that one of the leaders of the school reform movement is a serial fabulist. The entire movement is built on an edifice of plausible-sounding narratives that are always revealed to be empty when you give them a skeptical reading. On issue after issue within education, the education reform movement writ large has embraced the pleasant unreality of convenient stories over the messy work of gathering evidence responsibly and using it to improve public policy. So that Joel Klein lied about his background doesn’t surprise me; he’s just acting out personally the tendency of the movement writ large. (Perhaps this will shame The Atlantic out of letting him write advertising copy for his privatized reform efforts, which is like letting Ronald McDonald write pieces in your magazine about how McDonald’s hamburgers are delicious and affordable.)

What I am a little bit more surprised about is the fact that so many people are acting as though it’s the lies in his story that render it an improper point of discussion, and not the fact that it is a single piece of anecdotal evidence and thus useless for directing public policy. At the very best, Klein’s story would be a single data point. Arrayed against it is a large and growing body of empirical data that demonstrates that many of the reform movement’s favored policies– charter schools, merit pay, ease in firing teachers– has negligible effect on education outcomes, despite the promises of that movement.

Check this out from theProspectarticle:

As proof, Klein—and others for him—cites his life story in what has become a stump speech for his brand of school reform. Again and again, Klein recounts his own deprived childhood and how it was a public-school teacher who plucked him from a path to mediocrity or worse. He offers his autobiography as evidence that poverty is no bar to success and that today’s disadvantaged children fail only because they are not rescued by inspiring teachers like those from whom Klein himself had benefitted.

OK. Even if Klein’s story were true, and he was a poor kid who lived in a bad neighborhood like he says, this single story couldn’t tell us anything of responsible use when it comes to our public policy. If this story is really being used as evidence, the people taking it seriously probably shouldn’t be making important decisions about education. And if Richard Rothstein is right (and his piece is really a wonderful work of meticulous journalism), and the conventional wisdom has been deeply impacted by Klein’s story… well, that’s insane and scary.

There are multiple reasons to oppose charter schools, merit pay, teacher union busting, and private school vouchers, many related to fair labor practices, accountability when using public money, and local control of local schools. But the most obvious and salient reason is that none of them work to improve educational outcomes in the way that school reformers claim. That’s what any responsible assessment of the extant social science– rigorously conducted, appropriately sampled– tells us. That would be true whether or not one of the people growing rich off of the reform movement personally lived the plot of a Dickens novel. What happened to one guy is irrelevant. What matters is the crushing weight of all the numbers.

50 replies
  1. 1
    srv says:

    Clearly, you have not been to Texas and seen what wonders twenty years of Republican control and reform of education have wrought.

    Texas Monthly has a write up of the heroic Governor finally taking on all those pointy head liberals at UT and TAMU.

  2. 2
    Handy says:

    Anecdotal is very convincing. It is how you sell snake oil.

  3. 3
    jwb says:

    @srv: And if he has his way Perry is going to run them into the ground, too. It will be interesting how the alumni respond. The alumni beat him back last time, but I don’t know if they’ll be able to do it again. On the other hand, this may well be the issue that rips the Texas GOP apart.

  4. 4
    Matthew Reid Krell says:

    Relationship between Freddie and B-J commentariat: schizophrenic.

  5. 5
    Snowwy says:

    Alger Hiss, not Dickens. But I’m nitpicking, don’t mind me.

  6. 6
    Peregrinus says:

    I think the reason Klein being a liar (to put it bluntly) is that the ed reform movement sells itself as being more accountable and therefore more honest than the “union thugs” on the other side. Their shit is always about the shady “special interests” that control your children’s education but are not responsive to you, versus the new, clean charter school whose corporate board is obviously going to be far quicker on the draw when it comes to complaints and performance.

    The fact that Klein can’t even tell the truth about his own life, then, becomes pretty serious as it means he’s not holding himself accountable.

    I mean, Freddie, I get what you’re saying and agree with it, but the problem is that public opinion on policy isn’t really swayed by data points or lack thereof, it’s swayed by visceral feeling. Where I live I can’t go two steps without someone telling me how the evil teacher’s union controls all the jobs in the city school district and how public schooling is just “an entitlement program.” They have no real information to base that on other than personal anecdotes, either.

  7. 7
    geg6 says:


    I think Tom Corbett is following Governor Goodhair’s lead here in PA. Terrible what he’s done and is doing to education in this state.

  8. 8
    Soonergrunt says:

    So you’re saying that one of the thought leaders of the education reform movement is provably full of shit.

    The sun rises in the east. Water is wet…

  9. 9
    PeakVT says:

    @Snowwy: Horatio Alger, not Alger Hiss.

  10. 10
    Peregrinus says:


    I was going to say – I didn’t think Klein had lived the life of a Soviet spy . .

  11. 11
    geg6 says:


    I don’t know why I’m “undefined” in comment #7.


  12. 12
    jwb says:

    @geg6: Perry has gotten extensive pushback from places where it matters: GOP donors and money handlers. That’s why it has the potential to cause a major rift in the party. On the other hand, for whatever reason, he is feeling emboldened and the lesson Dewhurst has taken from his primary defeat is don’t ever let anyone get to the right of you. It’s looking to be a lethal combination, but it’s also running up against the money machine. The legislative session will be very interesting.

  13. 13
    Omnes Omnibus says:

    @Peregrinus: He wasn’t a spy!

    Sorry, I thought it was time for a fight on some issue.

  14. 14
    Soonergrunt says:

    @geg6: Nothing in the dashboard on that.

  15. 15
    PeakVT says:

    @geg6: Sometimes when I edit a comment “undefined” shows up when I save. I think it’s a local Ajax/Javascript/whatever error.

  16. 16
    Mike G says:

    The MSM propagated a bullshit ideology that coincidentally increases the wealth and power of the wealthy and powerful.
    Color me surprised.

  17. 17
    efroh says:

    Hi Freddie,

    This was a good post. Glad you are still here.

  18. 18
    Yutsano says:

    @Omnes Omnibus: We can always find things to fight about around here. Fuck the Pack. See?

  19. 19
    YoohooCthulhu says:

    I think the problem people here have with Freddie is pretty much entirely explained by his self-important writing style.

  20. 20
    Ben Franklin says:



  21. 21
    Raven says:

    @Yutsano: I like the cut of your jib.

  22. 22
    Yutsano says:

    @Raven: You want a fight? I am perfectly capablle of obliging. I’m a Scorpio, I excel at chaos.

  23. 23
    22over7 says:

    Education reform, ultimately, has the same roots as the War on Drugs: money. Lotsa lotsa money.

    It’s flowing into the coffers of private corporations and charter school companies. Plenty of greased hands, lots of happy words to sell to the parents and voters. Eventually, we will be where we were a couple of centuries ago, that is, those with means will send their children to the highest-quality private schools (Phillips Andover, Choate Rosemary Hall, and the like), where they will learn to rule, while everyone else gets somewhere between a basic vocational education and none at all. Feature, not bug. Makes it easier to bamboozle them. Education really does have a liberal bias.

    Welcome to America Inc., soon to become a wholly-owned subidiary of Bain Capital.

  24. 24
    epar says:

    You can’t paint the school reform movement with a broad brush. Charter Schools in Massachusetts are very successful. 6 of the 11 schools that got a 100% pass rate on the state standardized test for 10th grade english are charters (out of 355 total schools taking the test). 4 of the top 6 high schools for the 10th grade math test were also charters. My wife is a teacher at one of those charter schools and I know many of her colleagues and administrators. Maybe in Texas the charter movement is just a front for union busting, but in MA its about getting a better education for poor urban kids.

  25. 25
    Raven says:

    @Yutsano: Me too, Nov 10.

  26. 26
    Raven says:

    @Yutsano: Me too, Nov 10.

  27. 27
    Omnes Omnibus says:

    @Yutsano: You realize that thith means war.

  28. 28
    Ben Franklin says:


  29. 29
    Ben Franklin says:


  30. 30


    I think you’re on to something. A lot of people beginn with the conclusion and work their way backwards, grabbing every loose little scrap of a story here, a dubious claim there, as long as it fits with what they believe. And they’ll throw away any proof, however conclusive and systematic, as long as it doesn’t fit with what they believe.

    And Republicans have been telling people for 30 or 40 years that unions are corrupt and they steal from hard-working decent people and that The Market™ knows all and understands all and heals all that’s out of whack. Some of us have grown up never knowing a time when this tide of shit wasn’t washing over our society. So, when people hear about “charter schools” and “vouchers” and “school choice”, it fits in with what they’ve been hearing forever.

    We all know the drill: Government can do nothing right. Government workers are lazy, incompetent losers who couldn’t get a “real job”. Teachers are government workers, so they’re lazy bums who only want to gouge hard working taxpayers and get rich while only working 7 hours a day for nine months a year. Ah, but businessmen know all and have that golden touch. They’re lean and mean, and don’t take any bullshit, and they give you the best for less. They look only for profit, so they have to churn out something worth buying or they’ll lose money. The good schools will thrive and the bad schools will fade, and die off, which will let the good schools take over and everybody wins.

    Now, what makes things even worse is that schools are complicated. It’s hard to run a good school. I’ve seen this myself. You need the best people and a lot of money, and even with all that, a school can still do badly. There are no quick answers to setting a bad school right. But we have grown far too fond of easy answers and getting what we want now. If I say, “Well, this mess took a long time to get where it is, and it’ll be hard to mend things, and it’ll take some time and a lot of money, and even with all that, it still might not get much better. Things aren’t what they were when we were little; so many children coming to school don’t speak English, and don’t have parents at home who can help them often…” And then I go on, listing other hurdles and hurdles and still more hurdles. And then some slick asswipe comes along and tells everybody, “Hey, don’t listen to that egghead. I can turn this school right around in a year, and it won’t cost half of what he says it will. Just give me that money and I’ll have things running like a Swiss watch in no time! I’m a businessman, I’ve done this all my life. All you have to do is shitcan the unions and let The Market™ decide! It’s magic! Trust me. Trusssssst me!”

    And which one do you think most people will believe? Which one do you think most people want to believe? It’s easy to be cynical and think there’s no way to shake people out of believing the crap the Republicans have shoveled down their throats all these years. I know there must be a way. We just have to find it. I don’t know what it’ll be, though. How do you reach 300,000,000 people who’ve been more thoroughly brainwashed than George Romney was in 1968? I’ll be damned if I know, but there has to be a way.

  31. 31
    Ben Franklin says:


  32. 32
    Corner Stone says:


    Texas Monthly has a write up of the heroic Governor finally taking on all those pointy head liberals at UT and TAMU.

    If Kay Bailey had actually given one iota of a shit about beating Perry she had plenty of backers waiting to donate. You’ve never seen more red faced, royally pissed of Republican voting Aggie alums than when Perry tried his best to put the death roll on TAMU.

  33. 33
    Corner Stone says:


    and the lesson Dewhurst has taken from his primary defeat is don’t ever let anyone get to the right of you.

    It was a lesson to them all. Doesn’t bode well for Texas Republicans.
    One hopes.

  34. 34
    SiubhanDuinne says:


    Wait, I thought you just had a birthday.

  35. 35
    TheMightyTrowel says:

    @Raven: Me too Nov 16. Scorpios for the win!

  36. 36
    JadedOptimist says:

    @Anatoliĭ Lъudьvigovich Bzyp (formerly Horrendo Slapp, Jimperson Zibb, Duncan Dönitz, Otto Graf von Pfmidtnöchtler-Pízsmőgy, Mumphrey, et al.): You allude to another piece of the puzzle that works in favor of the snake oil sellers: Improving schools takes time. If you’re running an elementary school, and tell the parent of a 3rd-grader that it’s going to take 4 years to turn things around, their thought is “That’s too long! My precious baby will be on to middle school by then!” Enter Mr RapidTurnaroundMagicBusinessGuy, who says he can fix it all by NEXT YEAR (but to show good faith, the district needs to give him a 10 year contract to run the new charter). He comes in, kicks out the second language kids and anyone with an IEP, and PRESTO! Test scores go UP. But they go down at the next (not Charter) school down the street that had to take those kids. Better turn it into a charter, too!

    Freakin charlatans.

  37. 37
    Yutsano says:

    @Raven: 40th revolution on the 30th. A Dawg is never gonna let me live this down…

    @Omnes Omnibus: Ehh, what’s up duck?

  38. 38
    TheMightyTrowel says:

    @Yutsano: It’s a big year for me too – 30 in a month. Crazy. I still feel like I’m 18 half the time.

  39. 39
    Elliecat says:

    @Anatoliĭ Lъudьvigovich Bzyp (formerly Horrendo Slapp, Jimperson Zibb, Duncan Dönitz, Otto Graf von Pfmidtnöchtler-Pízsmőgy, Mumphrey, et al.):

    You need the best people and a lot of money, and even with all that, a school can still do badly.

    I just have to say I am so effing sick of hearing about “bad schools.” No doubt they are out there somewhere but the school reform bs’ers have convinced people that they are EVERYWHERE! ALL the public schools are BAD!

    I have people tell me to my face how “terrible” our local public schools are, even though they do not have children in the schools and I do. Even when I tell them that I think our schools are good, they cling to this belief. In most cases it means that there are Black kids in the schools and Latino ones and Poor ones and Special Needs ones – at any rate the schools considered the “worst schools” locally are the ones with the most of those kids, despite the teachers being as good as teachers at any other school.

    (Edited because the stupid thing posted before I was done.)

  40. 40
    A moocher says:

    @Anatoliĭ Lъudьvigovich Bzyp (formerly Horrendo Slapp, Jimperson Zibb, Duncan Dönitz, Otto Graf von Pfmidtnöchtler-Pízsmőgy, Mumphrey, et al.):

    How do you reach 300,000,000 people who’ve been more thoroughly brainwashed than George Romney was in 1968? I’ll be damned if I know, but there has to be a way.

    No, there really doesn’t have to be a way. Not while the systematic propaganda continues. Not while the system of internal communications becomes ever more corrupt an impenetrable to the truth, or even to contrary argument. Not while the lower levels of government fall piece by piece into the hands of the enemy (ALEC).

    The oxygen for all this comes from a small number of very wealthy individuals and families who have conducted a systematic campaign for decades to achieve the current situation. We know who some of the are: the Kochs, that Macao gambling impressario, the Mellons, some wallstreet dudes. There are sure to be others less visible, but easily identifiable by the resources of a national government.

    These people have to be taken out of play.This means taxing their wealth away, preventing dynastic accumulation, campaign finance reform, dissolution of the media conglomerates, revoking the tax-free status of organised religion. I do not believe that these ends can any longer be achieved by political means.

  41. 41
    Ben Franklin says:

    @A moocher:

    Not while the systematic propaganda continues

    I could be wrong, but I think you meant systemic..

  42. 42
    Ben Franklin says:


    It doesn’t matter. They’re on a roll.

  43. 43


    You’re right; I could have put that better, or as Mitt Romney might say, more artfully. I guess a better way of saying it is to talk about schools having problems rather than being bad. Either way, I think the same idea holds: It’s easy for schools to have problems, since all the people who run them and oversee them, and all the people who teach there, and all the people who learn there are, well, people. In some places, like poor districts, there will be more problems to overcome. So, you’re right; it isn’t that some schools are “good” and others “bad” and still others somewhere between “good” and “bad”, it’s that some schools have more problems to deal with, more hardships and hurdles to overcome.

    But the bigger point I was making still holds true: It’s easy to sell people on an easy, cheap, quick, painless fix. It’s harder to get people to go along with some plan that will really help overcome all those hurdles, but that will take some time, cost some tax money and have far more working parts, any of which can go off the rails. People want things done now. And Republicans have made a thriving business of conning people into believeing that they (the Republicans) can give it to them, and that whenever aything goes wrong, it’s the Democrats who are fucking their genius plans up.

  44. 44

    I’ll also say that one thing I sure as hell don’t understand is how Republicans think that paying teachers less and dumping more work on them is going to make things better. They sure don’t seem to think things work that way with C.E.O.s… How anybody can buy that shit is beyond me.

    I’ll also note that, as far as I know, the geniuses who sell these fly-by-night schemes to turn schools around overnight by shitting on the teachers have never set foot inside a classroom since they were students. I worked as a teacher for two years in Honduras. I’m not a natural at it, to put it mildly, and it was a daily struggle. And my monthly pay of $100 wasn’t going to make me rich (I wonder how many of these school privatization assholes think paying American teachers $100 a month would be coddling them).

    But I was 26, and I had no real job and no family, and I got to live in a Caribbean beach town in a house that was no more than 1000 from the sea. I fell asleep each night to the crashing of the waves. And that was why I did it for two years. A few retirees down there have been teaching since I first went down, but since they get Social Security, they can afford to do it and live pretty well down there

    Anyway, the point is that I know what goess into teaching. It isn’t some boondoggle that lazy losers slide into because they get three months off every year and can stick the taxpayers with an unreasonable bill. They do this because they want to do something worthwhile with their lives, which is a lot more than I cans ay for the dickwads hawking privatization. but then, those creeps think that anybody who works with any end in mind other than getting filthy rich is a pitiful loser. But I’d like to see Joe Klein or Bobby Jindal or any of these other fucks make it through one week of teaching in a public school and then come out on the other end and say with a straight face that teachers make too much money. Assholes…

  45. 45
    Peregrinus says:

    @Omnes Omnibus:

    Fair point, yes. Must’ve been the Cuban momentarily coming out in me.

  46. 46
    Bob says:

    Ha ha. This is a silly lie, a very silly lie. I too grew up in public housing in Queens, my father was a government worker, neither parent finished college. And I too tell people I’m a product of public housing, that I grew up on the streets. And then I laugh, because: It’s. A. Joke. I went to school (NYC neighborhood public schools, of course) with the smartest, most ambitious bunch of people I’ve ever been around (and I hate to name drop but I am an MIT graduate). Everybody knew they were going to college, and beyond. Klein is from that postwar upwardly mobile Jewish community in New York. Nobody had money, but the city colleges were free at the time and many educational, medical, science, and business leaders came from that environment. Klein is not fooling anybody with that silly lie.

    Education “reform” is full of frauds and crooks and idiots. The “tell” right at the beginning was the standardized exams, the fake “proficiency”, and the 100% proficient requirement. Setting the system up to fail. And knowing it full well. Sure there were well meaning people on board at the beginning but now it’s being driven by politics and greed. And this Klein clown, with the cheating on the NY state exams – why aren’t there people in jail over that – and absolutely no consequence for him, would be exhibit number one were it not for even worse frauds and clowns out there.

  47. 47
    Brachiator says:

    RE: Arrayed against it is a large and growing body of empirical data that demonstrates that many of the reform movement’s favored policies—charter schools, merit pay, ease in firing teachers—has negligible effect on education outcomes, despite the promises of that movement.

    Well, no. Charter schools do not appear to be doing the job promised. Merit pay programs as currently devised are problematic. I don’t know whether it is still happening, but at one point in some California school districts, and elsewhere, there were some master teachers who were paid more. This is a form of merit pay.

    Firing teachers: again, issues in specific states and districts vary. In California, the Los Angeles Unified School District finds itself in the uncomfortable position of defending the right of pedophile teachers to keep their jobs. This is fanaticism.

    There have been cases in which students have filmed teachers sleeping or deliberately evading their teaching responsibilities. What was the response of teachers? How dare you film us! Restrictions placed on students. To hell with teachers like this. To hell with them.

    Aggregate “education outcomes” don’t mean shit to me. I agree with some of Freddie’s overall view of the problems. But I don’t think he has as firm a handle on data analysis as he pretends.

    As for solutions? I don’t know that anyone has a best answer. Yet.

  48. 48
    mclaren says:

    @Anatoliĭ Lъudьvigovich Bzyp (formerly Horrendo Slapp, Jimperson Zibb, Duncan Dönitz, Otto Graf von Pfmidtnöchtler-Pízsmőgy, Mumphrey, et al.):

    I’ll also say that one thing I sure as hell don’t understand is how Republicans think that paying teachers less and dumping more work on them is going to make things better.

    Republicans don’t give a shit whether paying teachers less and dumping more work on them will make things better. The goal here is to nuke the public sector and privatize it all. So they can then loot it all.

    That’s the whole Republican goal. It’s all they want. It’s nothing but a program of systematic looting. Privatize everything, then smash and grab and run away with the loot.

    The drunk-driving C student tried it with social security. Reagan got away with it for the military (the move to military contractors as a “cost saving” device started under Reagan, and of course costs skyrocketed) and with the financial sector (under Reagan, S&Ls were deregulated, resulting in the predictable theft and scams and wholesale fraud).

    The orgy of looting really got underway under the drunk-driving C student from Texas and his torturer sidekick, though. All made possible by privatization. Whenever you hear the word “privatization,” substitute the word “looting” — then you’ll have an accurate translation of the sentence.

  49. 49
    Miss Waterlow says:

    What’s up with hating on Freddie? I get that there’s a Balloon-Juice tone (you wouldn’t know it, but I’ve been around here for a long time) and he’s just a little to the serious side of that. But (and “but” only if you think that’s a problem, which I don’t) he’s smart as fuck, he cares about and goes deeply into important issues that nobody else seems here seems to notice, he writes well, and he DOES have “a firm handle on data analysis.” Brachiator, by contrast, seems the armchair pundit here. Being in the public education advocacy world for years, I’m pleasantly familiar with y’all. Unlike, say, exactly how many troops should be left in Eastern Afghanistan in the Spring, ed reform is a topic on which everyone feels like just a little bit of an expert. Freddie rocks. He gives BJ some needed ballast. Boot him off and you might fly away in cloud of stanky old cat fur and sarcasm.

  50. 50
    RSR says:

    I realize I’m late to comment on this post, but one note about Rothstein’s Prospect article. The “compressed” three year junior high school program caught my eye, as my nephew in NYC, who attended what is essentially a magnet high school on Staten Island, has often mentioned that four years of high school was too long for him. He excelled and wanted to get done and get out and get on with his life. That option, had it existed, would of course not be the right choice for every student. But for some, it could be reasonable.

    I recall one young man at my high school in the late eighties left after junior year to start a college program that would award him his high school diploma (or equivalent) as well as a college degree, saving at least one year over the traditional eight year span of high school and undergrad college. (Our high school was (and still is) expensive, and it probably cost less to attend public university than senior year of high school.)

Comments are closed.