How Good Was 538?

Again, for those of you who have never worked with stats, I am not sure you can appreciate how good Nate Silver and company were at predicting the outcome of the election. They were this good:

Our model projects that Obama will win all states won by John Kerry in 2004, in addition to Iowa, New Mexico, Colorado, Ohio, Virginia, Nevada, Florida and North Carolina, while narrowly losing Missouri and Indiana. These states total 353 electoral votes. Our official projection, which looks at these outcomes probabilistically — for instance, assigns North Carolina’s 15 electoral votes to Obama 59 percent of the time — comes up with an incrementally more conservative projection of 348.6 electoral votes.

We also project Obama to win the popular vote by 6.1 points; his lead is slightly larger than that in the polls now, but our model accounts for the fact that candidates with large leads in the polls typically underperform their numbers by a small margin on Election Day.

Currently, Obama has 349 electoral college votes, and leads the popular vote 52.4-46.3, or, as Nate called it, by a margin of 6.1 points.

This uncanny accuracy is the equivalent of dropping a penny from the top of a 50 story building and landing it in a shot glass. And not one of those double-shot sized ones, either.

This is sick accurate.

Share On Facebook
Share On Twitter
Share On Google Plus
Share On Pinterest
Share On Reddit






39 replies
  1. 1
    dr. bloor says:

    A lot of folks were close this time. The most impressive thing about Nate is that the model he’s developed is so sophisticated it allows him to drop that penny into the shot glass over and over and over again.

  2. 2
    Jeff says:

    And to think, it all started with baseball statistics:

    The one constant through all the years, Ray, has been baseball. America has rolled by like an army of steamrollers. It has been erased like a blackboard, rebuilt and erased again. But baseball has marked the time. This field, this game: it’s a part of our past, Ray. It reminds of us of all that once was good and it could be again. Oh… people will come Ray. People will most definitely come.

  3. 3
    demimondian says:

    I do work with stats, and, yes, it’s an amazing performance. (Although his famous prediction of the White Sox losing 93 games is better, to my mind.)

  4. 4
    arbitrista says:

    Only Obama is going to win Indiana, which means that his projected electoral college vote is 364. Which is still pretty good, but not quite Cassandra-like prognostication.

  5. 5
    zbw says:

    I predicted that 538 would be that accurate.

  6. 6
    Montysano (All Hail Marx & Lennon) says:

    Great job, Nate. As a mathematics major in college, I’m suitably impressed.

    However, based on your appearances on Olbermann, let me recommend taking some time off and allowing some sunshine to fall on your skin.

  7. 7
    Cris v.3.1 says:

    What I really love about 538 is that it exemplifies one of the virtues of the web: it’s an aggregator, not a primary source (i.e. it’s not a polling outfit, and wouldn’t exist without the myriad of independent polling outfits) but its aggregation produces a result that is superior to the sum of its parts.

    That, and Silver’s algorithm is clearly excellent. Pollster.com and electoral-vote.com are also aggregators, but Silver apparently has a more sophisticated — and apparently, based on this one data point, more accurate — model.

  8. 8
    tonycpsu says:

    Looks like he could be off by 15 votes if NC comes through for the Dems after the inevitable recount (and assuming MO and IN stay where they are.)

    Regardless, sticking the landing on the popular vote total is truly remarkable. Nate is the bomb.

  9. 9
    dr. bloor says:

    However, based on your appearances on Olbermann, let me recommend taking some time off and allowing some sunshine to fall on your skin.

    Screw that. I want him to keep his pale ass in the basement and tell me who I should put my $ on for next year’s pennant races.

  10. 10
    Comrade Stuck says:

    Rasmussen also pretty much nailed it on popular vote %.

  11. 11
    Comrade Tax Analyst says:

    Yes, John, I started following Nate’s 538 site several months ago and found myself completely fascinated. I have to say that there were times in September and October when McCain seemed to be gaining fast that it truly helped me maintain some semblance of sanity. I printed out a list of his state-by-state projections and had it at hand last night as I watched the numbers as they totalled up on the TV screen. I noticed that some early numbers that might have alarmed me in the past instead just caused me to think about the "what and why" of those numbers…where within that particular state were they coming from? So when I saw the IN numbers that showed McCain leading early on, but saw that Marion County, where Indianapolis is located, had not been significantly reported yet, I felt that Obama was going to be competitvely in that state…no assurance of winning, of course, but I felt like Nate’s numbers were going to be solid there – Indiana would be close, but it clearly would be in play. While many other pollster’s were fairly accurate in the race, I thought 538 was clearly at the top of the heap. He did a fantastic job, and it will be interesting to read what Nate has to say about the race in retrospect.

  12. 12
    Face says:

    This is sick accurate.

    He missed Indiana, ergo he suxxors.

  13. 13
    Jeff says:

    @dr. bloor: Unfortunately, Vegas probably has him on speed dial with a wad of cash to make sure they get the news first.

  14. 14
    gbear says:

    However, based on your appearances on Olbermann, let me recommend taking some time off and allowing some sunshine to fall on your skin.

    I hear you can get a makeover artist pretty cheap right now.

  15. 15
    Laura W says:

    SO, yes.
    SO, why were so many of you so freakin’ panicked and sick to your stomachs with nerves and stress yesterday?
    Is Nate reassuringly accurate in hindsight only?
    I am SO new to even caring about this, but even I knew the polls were so absurdly tilted that there was no way this was not going to be.

    No matter.
    All Hail Nate & Crew!
    (and thank you for being the stop I made over coffee in the morning before coming to BJ because I knew Nate’s succinct sanity would fortify me for whatever crap the day brought.)

  16. 16
    Laura W says:

    @gbear: HA!!!!!!!!!!

  17. 17
    Xecky Gilchrist says:

    I will add my voice to the chorus: Nate Silver &co. = EPIC WIN

  18. 18
    Cris v.3.1 says:

    Is Nate reassuringly accurate in hindsight only?

    Yes.

    And for that matter, hindsight is the only thing that reassures me in election years.

  19. 19
    cleek says:

    Looks like he could be off by 15 votes if NC comes through for the Dems after the inevitable recount

    and once they’re done counting the provisional ballots, it’ll be over. there won’t be a recount since nothing hinges on it. it’s not like McCain still has a shot with our 15 EVs. Obama wins the enchilada with or without NC.

    SO, why were so many of you so freakin’ panicked and sick to your stomachs with nerves and stress yesterday?

    it’s part of being a Democrat… ?

  20. 20
    Frank Sobotka says:

    This has to be a good year for Nate. He was the only one who predicted the Rays to be a competitive 90 win team, and now he’s nailed the election almost square. As someone who’s been snooping around the fringes of Baseball Prospectus for years now, it’s nice to see him get some recognition.

    So what does 538 do next? Start aggregating assorted mayor, city council, alderman, and dog catcher polls?

  21. 21
    Zifnab says:

    it’s part of being a Democrat… ?

    Snatching defeat from the jaws of victory since 1964.

  22. 22
    Dylan says:

    I can’t help thinking that with the number of pollsters and aggregators around, someone is going to be uncannily accurate.

    It takes more than a couple of lucky breaks to convince me.

  23. 23
    ThatLeftTurnInABQ says:

    What I really love about 538 is that it exemplifies one of the virtues of the web: it’s an aggregator, not a primary source (i.e. it’s not a polling outfit, and wouldn’t exist without the myriad of independent polling outfits) but its aggregation produces a result that is superior to the sum of its parts.

    This is absolutely correct with regard to Nate’s statistical work at 538, but don’t overlook the outstanding in-the-field reporting by Sean Quinn and Brett Marty. For my money the best primary reporting on the mechanics of the Obama ground game was at 538 (and a close 2nd to Al Giordano’s TheField).

    The MSM never even came close to providing the level of detail and analysis that Sean and Marty did, and there is no way to understand what happened in Indiana for example, without knowing what was going on in the respective field offices of the two campaigns – Marty’s photos of McCain battleground state field offices which were empty or locked in the later stages of the contest were devastating evidence of what was going on behind the scenes.

  24. 24
    Shaggy says:

    Sounds like a waste of a perfectly good shot glass. Now, if only you could pour, say, an appropriate beverage into said glass…wait, never mind, that reminds me of last night.

    Nate did a decent job with the popular vote and I really liked Sean’s election journal type thingy where they visited (or tried to visit, as was often the case with the McCain campaign) offices of both campaigns in different states.

  25. 25
  26. 26
    ex-duchess says:

    538 and Nate rocked my world for this election. Unbelievably accurate, always level-headed–one of my sanity go-to sites for this entire election. His job offers are going to be myriad; if not people are stoopid.

  27. 27
    tavella says:

    What I thought was especially impressive was that he was very accurate as to what was dark blue and dark red — they were being called quickly, and and he very accurately picked out the marginal states. Electoral-vote, for example, had Texas as only barely GOP.

  28. 28
    KRK says:

    So what does 538 do next?

    They’ve already announced they’re converting to pr0n: 538 After Dark.

    And folks should really check out tonycpsu’s link above. Nate Silver’s Internet celebrity cred cannot be doubted.

  29. 29
    Steko says:

    Sam Wang at Princeton Election Consortium and the guys at Stochastic Democracy did comparably as did many other people.

    Nate’s bigger contributions though include busting bad methodology of several pollsters in addition to his other content and user friendly data loaded site. 538 is like pollster.com, PEC, and the ground game diaries from DKos mixed together.

  30. 30
    Bob Barbaque says:

    Toggling back and forth between 538 and Google election kept Me in the catbird seat up till McCains concession call
    over at Taegan Goddards news aggregator.
    I came here to feel some love as Obama gave his wonderful acceptance speech.
    I have a bottle of Moet ’77 that I have hoarded for 31 years,
    it will be time to savor the flavor on January 20 2009,the end of an error and the dawn of an historic administration.
    Again,congratulations Everyone!

  31. 31

    What struck me is that Nate Silver dramatically improved as a TV guest over the last few weeks. His appearance on Colbert about four weeks ago was almost painful. By last night, he was pretty good.

    His original weakness didn’t surprise me. Silver has done some great work at Baseball Prospectus, but he has never seemed that interested in defending it. He seems more comfortable cranking out the numbers than explaining them. Unfortunately, that left the defense of his work up to some of the folks at BP that I think are jackasses. (I’m looking at you, Will Carroll.)

  32. 32
    terry chay says:

    If you read his analysis, his instinct on the model mispredicting IN flipping was correct because he mentions that the model cannot account for “up to a 2 point swing due to the ground game.” The same thing occurs in MO if it flips.

    He was off on the how red VA would be an how blue MT would be. Still within the MOE on both states. NV was also bluer than estimated. WV was redder.

    He missed the call on the senator race in AK (over 3x outside MOE!). He also refrained from calling Prop 8, but his instinct on the numbers was way off wrong (basically he missed a true Bradley effect going on there that those of us who live in CA knew to be true).

    Still not bad. It seems the more data he had the more accurate he was (popular vote was spot on, the weights on that adjusted the numbers in the model to make the state predictions more accurate than any single pollster missing just the strength of the ground game in some states and the amount of bigotry in appalachia), the less data there was (a senatorial race with a black swan effect, prop 8) the farther off he was.

    Still I told someone last week that if you bet the futures markets on 538 you’d have made a killing. And I stand by that judgement.

  33. 33
    bayville says:

    Comrade John-

    I believe Nate did get Indiana wrong by about seven votes.
    And let’s be honest he also got Alaska completely wrong, which of course, doesn’t count — because it’s Alaska.

  34. 34

    No doubt about it, 538 was the authority on this year’s polling.

    And where their pointers were a little off, it looks to me like it was a lack of good polling in those areas. Thin or crappy polling just doesn’t provide the data needed to do what Silver does, which is to brilliantly analyze data. The more data, and the better it is, the more he can do, and the better the results will be. That’s clear.

  35. 35
    shortstop says:

    Okay, but I gotta brag that if MO goes to McCain and NC to Obama, I will have predicted every single state correctly and have gotten the electoral vote exactly right.

  36. 36

    […] Cole looked at the final tally last night and his jaw hit the floor. Currently, Obama has 349 electoral college votes, and leads […]

  37. 37
    Shinobi says:

    I totally appreciate it. In fact I appreciate it so much it makes me realize just how big of a nerd I am. /wistful sigh

    I wish they were hiring.

  38. 38
    Hopeful says:

    Hmmm

    He won Indiana. Missouri is still out. He so far has 364 electoral votes but Nate is still a genius. He kept me sane.

  39. 39
    Stephen says:

    @Dylan: and that’s why you don’t do statistics for a living.

Trackbacks & Pingbacks

  1. […] Cole looked at the final tally last night and his jaw hit the floor. Currently, Obama has 349 electoral college votes, and leads […]

Comments are closed.