R2K Fraud

Kos has just posted a report by a couple of independent researchers that shows that some Research 2000 polling was probably a fabrication. He’s also written up an explanatory post describing what happened and the (lack of) response from R2K.

Kos’ transparency about polling was the reason the researchers were able to identify the fraud, and the impetus for the investigation was 538’s pollster rankings.






71 replies
  1. 1
    geg6 says:

    I just posted this in the previous thread so I’ll just cut and paste it here:

    And this is why I simply can’t lump Markos in the same dungheap as so many others in the emo-left:

    I can’t express enough my gratitude to Mark, Michael, and Jonathan for helping bring this to light. Sure, our friends on the Right will get to take some cheap shots, and they should take advantage of the opportunity. But ultimately, this episode validates the reason why we released the internal numbers from Research 2000—and why every media outlet should do the same from their pollster; without full transparency of results, this fraud would not have been uncovered. As difficult as it has been to learn that we were victims of that fraud, our commitment to accuracy and the truth is far more important than shielding ourselves from cheap shots from the Right.

    The diaries over at GOS are the very definition of the suck, but Markos is the kind of guy I can admire even though I don’t always agree with him.

  2. 2
    dmsilev says:

    I read through that report, and it’s pretty damning. Kudos to Markos both for taking the initial allegations seriously and for publishing everything in a prominent location (unlike the habit that a lot of newspapers have of putting their mea culpas on page B45).

    dms

  3. 3
    Randy P says:

    I read that. A lot of meat for us statistics geeks to chew on. It’s funny what can slip by you. I’ve been reading those polls on and off for months.

    One of the anomalies they point out is that for male-female breakdowns on different questions, the two numbers are either both odd or both even in all but 2 out of 778 cases. Normally you’d expect one to be even and one to be odd about half the time, just from normal statistical variation.

  4. 4
    dmsilev says:

    Also worth noting: The odd-even correlation that was the first red flag was, if memory serves, the same statistical test that Nate Silver used when he busted Strategic Vision Polling for fraud last year.

    Moral of the story: Humans are pretty bad at faking random data.

    dms

  5. 5
    MattR says:

    You think a firm called Research2000 would have done a little research and figured out that Markos would not tolerate this kind of BS and that someone in the GOS community (or elsewhere) would ferret it out. I have to believe this was laziness (or possibly being overextended and unable to do the polling) and that R2K was not stupid enough to try and actively manipulate the results. Of course I have been repeatedly surprised by the stupidity of people recently.

  6. 6
    cleek says:

    i feel sorry for the poor guy who has to spend his days typing pseudo-random nonsense into R2K’s spreadsheets.

  7. 7
    Randy P says:

    @dmsilev: They should have used a lot more 7’s. Isn’t 7 the most random number?

  8. 8
    Dave says:

    What I found most interesting is how they were able to use the lack of 0% changes from week-to-week to help identify falsely-created data. As fascinating as this is from the obvious news perspective, the insight into human behavior when it comes to deception, for non-math guys like myself, is really something.

  9. 9
    dmsilev says:

    @Randy P: Heh. Actually, going back and looking at Nate’s original post, it seems I misremembered slightly. He looked at the distribution of trailing digits, and found a significant deficit of 1s, 2s, and 3s.

    dms

  10. 10
    beltane says:

    @geg6: Thank you. Markos should not be lumped together with Jane Hamsher, Cenk Uyger and the rest of the manic progressive crowd. While he may often be wrong, he is far more professional (and sane) than the rest of them. The only time I’ve heard him promote third-party bullshit is when he’s egging on the teabaggers.

  11. 11
    sukabi says:

    @MattR: you’d probably find that there are a lot of “polling” organizations that use the same “model” that Research2000 used, as most “news organizations” just “report” what’s put in front of them without question as long as it fits their narrative… and if it doesn’t fit, they’ll pull out something from one of the older polls to “support” it.

  12. 12
    Brachiator says:

    A recent Megan McArdle bullcrap column about the “limitations” of peer review, which a Balloon Juice poster recently referred to, easily comes to mind here.

    Kos shouldn’t waste time worrying about possible cheap shots from the Right here. They deny any analysis that doesn’t fit their delusions, without regard to accuracy.

  13. 13
    catclub says:

    @cleek:
    If said person cannot figure out a way to write a computer program to randomize the fiddled numbers, that person will have to just suffer.

    It takes a lazy worker to get a lot of work done with less effort.

  14. 14
    LarsThorwald says:

    I’m sorry. But I went to law school because it was my understanding that there would be no math.

  15. 15
    somethingblue says:

    So does this mean John McCain gets to be president after all?

  16. 16
    LarsThorwald says:

    The tell-tale heart in this little caper was the repeated use of the digits “8008” as trailing digits, and the fact that every time the the Research 2000 team delivered the results, they were heard to giggle “boob”.

  17. 17
    LarsThorwald says:

    As Mark Twain said, “There are lies, damn lies, and Research 2000 results.”

    I could go on and on and on. Try the veal.

  18. 18
    Jon H says:

    @dmsilev: ” Kudos to Markos both for taking the initial allegations seriously and for publishing everything in a prominent location (unlike the habit that a lot of newspapers have of putting their mea culpas on page B45).”

    On the other hand, it might be easier because of the arm’s length relationship with R2K.

    If the polling had been done in-house by dailykos peeps, it would probably be more tempting to stonewall or cover it up or otherwise be defensive.

    Instead, R2K defrauded Kos, didn’t provide what the contract required, and thus their failing doesn’t really reflect badly on Kos. So it’s a lot easier to do this sort of public R2K culpa.

  19. 19
    scarshapedstar says:

    @Dave:

    What I found most interesting is how they were able to use the lack of 0% changes from week-to-week to help identify falsely-created data. As fascinating as this is from the obvious news perspective, the insight into human behavior when it comes to deception, for non-math guys like myself, is really something.

    During WWII, the Germans had to come up with new Enigma settings each day. The machine had 4 interchangeable cogs inside of it, lined up in a row, and part of generating a new setting was that you would shuffle the cogs around. Well, the Germans had an official policy that to make things as random as possible they would never keep 2 cogs next to each other for two days in a row.

    Sounds all well and good, but the Allies knew about this, and it DRAMATICALLY reduced the number of possible combinations for each day. Once they cracked the code on one day they could rule out about half of the possibilities for the next day. There were other kinds of hanky-panky with the way that the numbers were randomly generated; codes like 1111 were thrown out by the secretary picking numbered balls out of a lottery-style tumbler.

    As a general rule, if you think you have some great idea for making something more random, you’re probably dead wrong and doing the exact opposite. It’s almost like a law of thermodynamics.

  20. 20
    catclub says:

    @scarshapedstar:
    I think Knuth wrote something like: “If all the scientific papers based on a faulty pseudo-random number generators (PRNGs)
    were withdrawn, there would be a large gap in science library bookshelves.”

    That was probably 30 years ago. I hope that things have improved.

    Of course, if Research 2000 had even used the most transparently bad PRNG they would never have been caught.
    Instead they were MUCH lazier.

    The tests that bad PRNGs fail take massive sample sizes before they can decide betwen non-random/random.

  21. 21
    Tonal Crow says:

    @scarshapedstar:

    Well, the Germans had an official policy that to make things as random as possible they would never keep 2 cogs next to each other for two days in a row.

    Do you have a citation for this? I’m skeptical about it because it shows remarkably poor cryptographic judgement. They’d have done orders of magnitude better by simply flipping coins to determine the wheel order.

    As a general rule, if you think you have some great idea for making something more random, you’re probably dead wrong and doing the exact opposite. It’s almost like a law of thermodynamics.

    Yep. Randomness is a much more interesting and involved topic than most people realize.

  22. 22
    Jay C says:

    Whatever else you might think of Markos Moulitsas and the GOS frontpagers: at least they have the professionalism (when it’s their shit on the line anyway, not just ramblings from some of their 235,000+ members) to call out bogus polling when they find it, and post the conclusions for all to see and read. Which not all bloggers will do, unforced.

    Of course, math fiddles aside, some signs of trouble with R2K’s data might have been more obvious than others:

    That was two weeks ago, and despite repeated promises to provide us that data, Research 2000 ultimately refused to do so. At one point, they claimed they couldn’t deliver them because their computers were down and they had to work out of a Kinkos office.

    No confirmation, though, whether any interventions by the family dog were involved…..

  23. 23
    Randy P says:

    @Jon H:

    Instead, R2K defrauded Kos, didn’t provide what the contract required, and thus their failing doesn’t really reflect badly on Kos.

    You wouldn’t conclude that based on the comments at the FiveThirtyEight report that started suspicions about Research 2000.

    Over at Daily Kos they’re always making fun of Rasmussen, and they use Research 2000, which according to this is MUCH, MUCH, MUCH worse?
    __
    Looks like Rasmussen doesn’t suck nearly as much as that bastion of liberal “intelligence” the Daily Kos Research 2000 poll. I guess polling is another example of something liberals don’t do well. I don’t see any of you Rasmussen bashers talking about how far Research 2000 has fallen since hooking up with the Daily Kos.

    Can’t wait to hear comment from Markos Moulitsas…. He keeps on bashing Rasmussen, while Research 2000 is the fifth worst pollster in America….

  24. 24
    sukabi says:

    @Jay C: yeah, it sounds kinda like massive cheetos crumbs were their downfall….

  25. 25
    someguy says:

    I figured something was up after that poll a couple months ago showing that 20% of the right are birthers, and 30% think Obama is a secret communist muslim and not an American citizen.

    Those numbers are *waaaayyyy* too low to be plausible…

  26. 26
    Cranky Old Washington Post Reporter says:

    This is why I shall never trust these odious computer bloggers. Sally, David, and I shall have such a laugh at this at this weekend’s cotillion.

  27. 27
    Calouste says:

    @Randy P:

    Research2000 was just making shit up, without a bias one way or another. Rasmussen on the other hand seems to always have a measurable Republican bias compared to other pollsters, not necessarily due to making shit up as much as it is through leading questioning. One is trying to make a buck without actually doing work, the other is about influencing public opinion. The second one is actually a better business model in the US political climate.

  28. 28
    Origuy says:

    @Tonal Crow: The Germans had a number of flaws in their encryption procedures. I don’t know about this one, but the Bletchley Park crew, and before them Polish Intelligence, were able to exploit several of them.

    The Wikipedia articles on Enigma and Bletchley Park are interesting reading. Different parts of the German military had different procedures and different equipment.

  29. 29
    Randy P says:

    @catclub: George Marsaglia has written extensively on this subject. He’s the guy who originally pointed out the issues back in the 60s I think. Does Knuth cite his work?

    Matlab at least uses Marsaglia’s ideas in their PRNGs. I know that there are notes in Unix man pages about the built-in PRNGs that some are better than others, without specifying why.

    It’s an esoteric subject and people doing simulation haven’t necessarily heard of it. So I’m not sure how much the situation has improved. I wouldn’t be surprised to learn that Excel, for instance, still has 60s-style PRNGs built in. But then I’m a Microsoft hater.

    Incidentally, Marsaglia is also the author of the DIEHARD software for testing PRNGs.

  30. 30
    Corner Stone says:

    @Tonal Crow:

    Randomness is a much more interesting and involved topic than most people realize.

    The human brain seeks to derive order out of chaos. Pattern recognition was one of the things that kept us alive.
    Show any room full of people two photographs of “rainfall on a sidewalk” and ask them which is more random.

    And I agree with you, it’s fascinating.

  31. 31
    Martin says:

    @Calouste: Actually, Ras seems to have two distinct modes of operation which even Nate has noticed – the narrative setting mode where they boost the republican well ahead of the race, and the actual polling where they give a more accurate assessment.

    So, according to Ras, the GOP will win every race in the fall, and in head-to-heads pre-primary the GOP will seem invincible, and then when the polling for the general actually starts in earnest, suddenly the race tightens and Ras falls in line – and this late polling is what Nate tested in his rankings.

  32. 32
    That's Master of Accountancy to You, Pal (JMN) says:

    @Tonal Crow:

    Do you have a citation for this? I’m skeptical about it because it shows remarkably poor cryptographic judgement. They’d have done orders of magnitude better by simply flipping coins to determine the wheel order.

    The Germans were remarkably poor about code security throughout the war. They delayed changing from the three rotor to the four rotor Enigma machines for a lot longer than they should have because they thought they were unbreakable. They consistently transmitted the same message in multiple codes (such as naval Enigma and Luftwaffe Enigma, which were different); when you do that, you take the chance that the other side has broken one of the two codes. If they have, then you pretty much just handed them the second one, too.

    It wasn ‘t Enigma, but the Soviets had an agent, Lieutenant General Fritz Thiele, who was in charge of the Wehrmacht’s communications office. From this position, he simply put one more person on the distribution list for important messages. This person was Rudolf Rossler, living in Switzerland, who simply passed them on to Moscow. The Germans never caught Thiele’s espionage activities. He was a part of the July 20 plot, and was arrested and executed for that, but not for his espionage.

    As a general rule, all cryptographic breakthroughs require security breaches on the part the communicating parties. The computers keep getting better, but they still depend upon cracks in the system.

  33. 33
    giltay says:

    @Tonal Crow: I can’t see a reference specifically to that rule, but the Germans showed remarkably poor cryptographic judgement, anyway. There’s a list of examples on Wikipedia. (See also Barbara W. Tuchman’s The Zimmerman Telegram for a famous example from the First World War.)

  34. 34
    MikeJ says:

    @Randy P:

    So I’m not sure how much the situation has improved. I wouldn’t be surprised to learn that Excel, for instance, still has 60s-style PRNGs built in. But then I’m a Microsoft hater.

    Most libraries used by programming languages, on whatever platform, use pretty crappy prng by default, usually for speed. Unless you’re doing crypto or faking statistics they’re fine. When you get to the point that you need better random numbers, it’s easy enough to import a better library.

  35. 35
    Tonal Crow says:

    They consistently transmitted the same message in multiple codes (such as naval Enigma and Luftwaffe Enigma, which were different); when you do that, you take the chance that the other side has broken one of the two codes. If they have, then you pretty much just handed them the second one, too.

    Not quite. You have made it easier to crack the second code by permitting the attacker to use a known-plaintext attack, as opposed to the unknown-plaintext attack that she had to use before cracking the first code. How much easier a known-plaintext attack is depends upon the underlying cipher(s), the amount of plaintext captured, the plaintext’s relative entropy, etc.

    As a general rule, all cryptographic breakthroughs require security breaches on the part the communicating parties. The computers keep getting better, but they still depend upon cracks in the system.

    Usually the cipher (encryption algorithm) is the cryptosystem’s most-secure element. The downfalls usually come in generating good keys — and in keeping them secret.

  36. 36

    @dmsilev: I suspect that R2K knew, just as SV knew, what its clients wanted to see, and adjusted poll numbers accordingly.

  37. 37
    Punchy says:

    You’d think a firm called Research2000 would realize they’re 10 years behind the times.

  38. 38
    Joel says:

    What’s interesting about the R2K stuff is that they were a reliably middle-of-the-road, uncontroversial pollster. Their incentive for fabricating data could be nothing less than pure laziness. It’s no surprise that they’d be bad at faking it, if they weren’t even going to go through the effort of actually polling people.

  39. 39
    MikeJ says:

    @Joel:

    Their incentive for fabricating data could be nothing less than pure laziness.

    Not lazyness, greed. It’s expensive to actually poll people.

    Last time I actually worked with a pollster [1] was back when people were just starting to get cell phones, and it was a hugely expensive pain in the ass back then. These days it’s probably damn near impossible to get a decent sample.

    [1] Not involved with stats, just with infrastructure and back end programming. I have no special knowledge of polling, other than watching other people do it at fairly close range.

  40. 40
    dmsilev says:

    FWIW, TPM talked with R2K’s lawyer about DK’s allegations (acronyms FTW…). As you might imagine, R2K denies everything:

    Ali’s attorney, Richard Beckler of Howrey LLP in Washington, told TPMmuckraker in an interview, “This guy is completely all wet. This allegation of fraud is absurd.” He added, “These guys are basically ruining Mr. Ali’s business.”

    Beckler claimed that Kos “wont even pay his goddamn bill. He owes [Ali] $50, $60, $70,000 dollars, something in that neighborhood.”

    (Ali is President of Research 2000).

    dms

  41. 41
    joe from Lowell says:

    I find the comparison to the Strategic Vision scandal illuminating.

    Markos busted his own polling firm. It took Nate Silver to bust the GOP’s favorite courtiers.

    Once the case came out, Republican die-hards lined up to defend SV. Democrats want nothing to do with Research2000 now.

    There really is one reality-based side in this country.

  42. 42
    Dave Fud says:

    @Tonal Crow: Bauer (2007) p. 441 is listed in Wikipedia for this, see note 98: http://en.wikipedia.org/wiki/C.....ortcomings

  43. 43
    Randy P says:

    @MikeJ: Zogby Interactive was supposed to be the wave of the future in the post-landline world and I had high hopes for them. But they are so far down on the FiveThirtyEight pollster reliability ranking that they’re an outlier of the bottom. They’re not only in the cellar of the ranking, they burrowed through the cellar floor and into bedrock.

  44. 44
    Chyron HR says:

    Beckler claimed that Kos “wont even pay his goddamn bill. He owes [Ali] $50, $60, $70,000 dollars, something in that neighborhood.”

    Nothing says “I am a reputible pollster owed legitimate debts” like a 20 grand margin of error.

  45. 45
    Joel says:

    @Randy P: The YouGov/Economist polls have done reliably better, although they’re still not as good as the SurveyUSA/Gallup/Pew polls.

    The disparity is closing fast, however. Pretty soon, there won’t be very many reliable polls at all.

  46. 46
    kay says:

    @Chyron HR:

    Nothing says “I am a reputible pollster owed legitimate debts” like a 20 grand margin of error.

    Hah! Good one.

  47. 47
    Corner Stone says:

    @dmsilev:

    This guy is completely all wet.

    He actually said that? Did he later go on to say Ah Applesauce! Or if Kos had a beef with them he might be bumped off?

  48. 48
    matoko_chan says:

    @Randy P: Rasmussen is just using selective poll representation. they don’t show as dirty as R2K because R2K polled all contests.
    they only poll contests where they are confident they will get right beneficial results. Selective polling is right out of How to Lie with Statistics.

  49. 49
    kay says:

    We get polling calls a lot in national elections because we live in Ohio and are (I think) are some median, middle demographic. Plus, our phone number is listed.
    I tell the truth, but I must say, my husband just makes stuff up. He’ll say anything: “Constitution Party, ma’am, but leaning Bush”.
    My favorite poll question (to date) was “do you own a US flag?” Last question.
    I do, but I wondered about that. That was a “patriotism screen”.

  50. 50
    Cacti says:

    So, who will GOS use for their new pollster?

  51. 51
    Agoraphobic Kleptomaniac says:

    Didn’t Kos have a poll a few months back that showed how crazy the republicans really were and how much the tea-party had taken over.

    Was that a R2K poll?

  52. 52
    MikeJ says:

    @Agoraphobic Kleptomaniac: Without seeing the story we have no way of knowing. Why don’t you go find it for us if you would like for us to see it?

  53. 53
    bartkid says:

    @Randy P:
    According to Dilbert, it is 9. Always.

    At this point, let me point out that 72.548% of all statistics are just made up on the spot.

  54. 54
  55. 55
    Dave Fud says:

    @Cacti: To be decided, per his post.

    Soon, we’ll have a new pollster (or pollsters) to work with, helping us to fulfill our vision of surveying races and issues that are often overlooked by the traditional media and polling outfits.

  56. 56
    Randy P says:

    @MikeJ:
    @Agoraphobic Kleptomaniac:

    Afraid so.
    http://www.dailykos.com/statepoll/2010/1/31/US/437
    “DailyKos/Research 2000 Poll, Jan 31, 2010”

    And that one got a lot more press than the weekly tracking polls.

    Now that we know to check for anomalies, look at those results. For each question, the “not sure” column is the same number within a percentage point for virtually every slice of the population. Looks like the spreadsheet operator chose one category each time to have an outlier, but the others are almost identical.

  57. 57
    MikeJ says:

    @Randy P: That would be an interesting poll for somebody honest to conduct.

  58. 58
    Gunner Billy K says:

    What did we do before Nate Silver?

  59. 59
  60. 60
    Sentient Puddle says:

    The letter to Nate in full.

    Yup, this is very popcorn-worthy.

  61. 61
    Marc says:

    Heh…looks like R2K is gonna go down with guns a’ blazing. This will be entertaining to watch.

  62. 62
    MattR says:

    @Marc: All they need to do now is hire Norm Coleman’s lawyers.

  63. 63
    Peter J says:

    The problem with both R2K and SV seems to be that they never actually did all the polling they said they had done.

    The problem with Ras is not that they don’t do the polling, I’ve never heard any accusations like that, but that they afterwards apply their likely voter filter and through that they are then able to skew the results. Most pollsters don’t do LV filters until late in the election cycle, since it’s hard to predict who will and will not vote early, Ras never seems to have had a problem with that. Also, Ras uses these LV filters on almost every poll they release.

  64. 64

    @Sentient Puddle:
    Interesting. R2K might not know that Moulitsas is a lawyer as well as a blogger. I don’t think he’d file a suit without having his ducks in a row.

  65. 65
    malraux says:

    @arguingwithsignposts: More likely, R2K is doing whatever it can to introduce enough FUD to the debate. They are probably hoping for a quiet out of court settlement with closed records on the subject. I’m guessing Kos is going to push for an open result to destroy the company. If Kos has the facts on his side, it’ll be a pretty simple case.

    That and some of the R2K claims really strike me as fly-by-night company claims. The idea that the company would have enough problems to be forced to be working out of a kinkos? That’s a company so small as to be unable to handle massive data gathering. I’m sure they outsource the call center stuff, but still.

  66. 66

    @malraux:

    The idea that the company would have enough problems to be forced to be working out of a kinkos?

    Well, that idea is something out of 10 years ago.

    Also, apparently the lawyer for R2K apparently sent a copypasta to Nate Silver meant for someone else. Stay classy, R2K.

  67. 67
    asiangrrlMN says:

    @Sentient Puddle: Oh man. This is gonna be GOOD. I have always had more respect for Markos than any of the other far-lefty bloggers (I did like Uygur until he went completely off the rails), and I am glad that he is pushing forward on this. I didn’t know he’s a lawyer; it makes the whole situation that much tastier. And, I don’t think I would want to mess with Nate Silver AND Markos at the same time.

  68. 68
    Bill Murray says:

    @Randy P: I guess if you’re too lazy, you won’t even cheat correctly

  69. 69
    The Other Chuck says:

    @Randy P:

    Zogby isn’t so much crooked as that their internet polling is about as rigorous as the phone-in polls on FOX. In other words, their methodology is too worthless to be crooked, manipulating it might actually improve the quality.

  70. 70
    Tonal Crow says:

    @asiangrrlMN: Markos is “far-lefty”? Or is that meant sarcastically?

  71. 71
    xian says:

    not sure why “consistently left” or “Dem-supporting” = “far left” – kos is hardly a socialist or communist (=”far left” to me), and I (briefly) worked for him back in ’04. He is extremely pragmatic, just has a progressive world view. Is that considered extreme or “far” something from the middle ground here?

Comments are closed.