He Lost on Jeopardy, Baby

Here’s Ken Jenning’s take on his experience playing Jeopardy against Watson, an IBM research project programmed to play the game.

I’m not a big Jeopardy geek, but my understanding is that players are surprised at how big a role button management plays in winning or losing a round. In the few minutes of the Watson game that I watched, it was pretty clear that Watson was excellent at pressing the button at exactly the right moment if it knew the answer, which is more a measure of electromechanical reflex than human-like intelligence.

To the credit of IBM engineers, Watson almost always did know the right answer. Still, there were a few bloopers, such as the final Jeopardy question from yesterday (paraphrasing): “This city has two airports, one named after a World War II hero, and the other named after a World War II battle.” Watson’s guess, “Toronto”, was just laughably bad — Lester Pearson and Billy Bishop fought in World War I, and neither person is a battle. The right answer, “Chicago”, was pretty obvious, but apparently Watson couldn’t connect Midway or O’Hare with WW II.






80 replies
  1. 1
    Thunderlizard says:

    Even better, the final jeopardy Mistermix is talking about? The category was US CITIES!

  2. 2
    MikeJ says:

    From the Nova special I saw last week I thought they had fixed the bug where Watson gave the same wrong answer that another player had already given, but he did it on the show too.

  3. 3

    My father-in-law Tivo’s Jeopardy, so I might have the chance to watch them with him at some point. It’s interesting about that bad answer, though, that if it hadn’t been a Final Jeopardy question, Watson wouldn’t have forced an answer, And what’s even more curious is that it “knew” that it had a weakness in that category, because it only wagered $974 (which I imagine will be a Jeopardy clue one day).

  4. 4
    Thunderlizard says:

    It wagered 974 because it had a runaway. (No player could possibly catch it)

  5. 5
    Cat Lady says:

    “An away game for humanity”. Nice.

  6. 6
    Egypt Steve says:

    I love Jeopardy and I expected this to be more interesting than it was — I was pretty annoyed that the first two nights were more infomercial for IBM than the game itself. Agreed, it’s an impressive computer, but after a while, the novelty wears off. They definitely should have given it Hal’s voice, though.

  7. 7
    Commenting at Balloon Juice since 1937 says:

    I thought Watson made the show boring. After the first couple of minutes it was apparent that timing the buzzer correctly is fifty percent of the competition.

  8. 8
    jon says:

    The Blue Screen of Death will soon read “What was your computer?”

  9. 9
    ploeg says:

    @Thunderlizard:

    The official IBM Research explanation is that, in practice, the category name often has limited usefulness for finding the correct question. For example, a question for “Potent Potables” could be “What is Lynchburg, TN?”. Lynchburg, TN is not a potent potable, it’s just where potent potables are made. They therefore tuned the algorithm to ignore category name so that they could hone in on other clues that have proven to be more useful. It’s effective in most cases, but you’re going to get whoppers when the category name is an important clue.

  10. 10
    Dave says:

    Buzzer timing is the key to winning the game. I have a little experience because my ex was a contestant on the game (even won a match)so I got to watch it up close. If you can master that, and the control to NOT buzz in when you don’t (which is also key and something Watson was good at), your odds of winning go way up. And while we can reason and guess better than a computer…timing a reflex action seems to be in a computer’s wheelhouse.

  11. 11
    chmatl says:

    My husband was on Jeopardy about 10 years ago. Unfortunately, he failed miserably because he couldn’t get the hang of the buzzer. By the time he finally mastered buzzing in, he was so far behind his competitors there was no way to catch up. It was humiliating for him because it seemed like such a simple thing to press that stupid little button, but it completely undid him. We have a recording of his appearance but we’re not allowed to watch it, ever.

  12. 12
    Fwiffo says:

    It seemed to have a surprising amount of trouble with categories. As for getting fed the opponents’ wrong answers – Ken answered “20s” and Watson said “1920s” and probably didn’t realize that “20s” was the same thing. Its backup answers were not even decades, however.

  13. 13
    Hyd says:

    I was on Jeopardy, and managed to win twice. Speed on the button (for humans, anyway) is _the_ difference between the winners and losers on Jeopardy. My informed guesstimate is that about 20% or so of the questions are known by a single player.

    There are indicator lights that tell you as a player when you can buzz in. The _really_ good players don’t rely on the indicator lights, and time their buzzer on Alex’s voice. Instead of watching the lights they’re mimicking the production assistant that controls the lockout button. I was fast on the button, but never got the hang of timing the lights.

    In a straight-up 50-question test, KenJen would beat Watson. I think most players would give Watson a run for its money, or even win. But that edge on the buzzer is too tough to beat.

  14. 14
    MikeJ says:

    @Fwiffo: The backup answers were really awful. My favorite was “A poor workman blames Yogi Berra.”

  15. 15
    Hyd says:

    From a cynical perspective, they weren’t going to trot Watson out there to get his ass beat. It was going to be close, at the least.

  16. 16
    Comrade Javamanphil says:

    @Dave:

    If you can master that, and the control to NOT buzz in when you don’t

    This really highlights a huge, unfair advantage that a passionless computer has in this competition. It is never stressed that it is falling behind and so it never gets the itchy button finger (and there are a host of other reasons a human might get stressed about the button pushing.) In fact, it never feels stress at all. I would think that is a huge help in any competition.

  17. 17
    Just Some Fuckhead says:

    Watson wouldn’t last the first night on Survivor or Top Shot.

  18. 18
    gnomedad says:

    Watson plays ‘Jeopardy!’ well, but what else can it do?

    But even amid Watson’s triumph, some answers it blew point to problems, say experts such as machine-learning pioneer Douglas Lenat of Cycorp in Austin. “Like a human idiot savant, it would get wrong a large fraction of things that almost all sane adult humans would get right,” Lenat says. “For instance, if the category were ‘Uphill vs. Downhill,’ and the clue was ‘The direction that beer flows.’ ” (Watson doesn’t understand gravity as a concept, and few documents it searches are likely to discuss the propensity of beer to flow downhill.)

  19. 19

    There was an AP article that mentioned that Watson doesn’t listen to the opponent’s answer. AFAIK they didn’t fix anything in this regard – it would require voice-recognition and be prone to error, and it’s not the kind of thing you would want to patch in hastily.

    Also, re the secondary answers, whenever you saw an answer with a confidence level below 50%, that’s Watson “knowing” that it’s almost certainly wrong. You just get to see it because the engineers have exposed its “thought process” a bit – it would never answer that. It’s kind of like you doing free-association when you hear an answer, and you get something in your head but you know it’s not right. Like the amusing thing about Moisture Milks, where Watson thought of “butter” (a sort of free association from milk) but the actual answer was the manufacturer, Alberto.

  20. 20
    Chyron HR says:

    Watson didn’t win. I mean, it may have “won” in the literal technical sense, but it didn’t REALLY win. It fake won, like when Toto got all those Grammys.

    And I bet those grapes were sour, anyways.

  21. 21
    evap says:

    One of the creators of Watson gave a talk where I work yesterday (I’m a professor in a math and computer science dept.), it was fascinating. He explained a bit about the software behind Watson. He showed the “Toronto” answer and explained why Watson chose “Toronto”. Watson’s second choice was actually the correct answer, but he had slightly more confidence in “Toronto” so chose that. Every answer Watson comes up with comes with a “confidence” level based on a variety of factors. Watson buzzes in based on how confident it is in the answer, but it also depends on the state of the game. At the beginning, Watson buzzes if confidence is above 50%, but once the game is going there is an algorithm to determine when to buzz in, based on who is ahead, if it’s the end of the game, etc. The confidence for the “Toronto” answer was low and if it hadn’t been final jeopardy, Watson would not have buzzed in.

    A good human can beat Watson to the buzzer, because humans can anticipate the end of the question and will start to buzz in before the “end of question” light goes on. It takes Watson’s mechanical finger 3 seconds to buzz and it only happens after the “end” light goes on (Watson gets an electronic signal). Fascinating stuff!! They are working on adapting Watson to help doctors diagnose and things like that.

  22. 22
    Walker says:

    When I was in high school, I was part of a scholar’s bowl team. The really amazing people on the team had these incredible buzzer strategies. For example, there were quite a few questions of the form “What is the capital of X?” I had a teammate who was really good at these questions and would always buzz in at the word “of”; the announcer was guaranteed to spill over into the next word before he had to answer.

  23. 23
    evap says:

    @Thunderlizard: To be fair to Watson, apparently there are several “Toronto”‘s in the U.S. And Watson had Chicago as its second answer, with confidence close to (but less than) the confidence for “Toronto”. Categories are tricky, because the name of the category does not necessarily tell you the type of the answer. “Right type” is one measure that Watson uses to measure confidence.

  24. 24
    DPirate says:

    My uncle was on the show, or taped an episode at least. He said the same thing, that timing the button was key.

  25. 25
    Steve says:

    I’ve never been on Jeopardy (unlike everyone else, apparently) but I did college bowl which is a similar concept. The buzzer was definitely a big part of it, and what I found is that it sort of depends on getting into a groove where you time it right, nail a question, and then you’re able to maintain the good timing for future questions. On the flip side, if someone else has the momentum then it’s hard to cut in at the perfect moment and steal their mojo. I think this is why you often see people run off streaks of correct answers on Jeopardy, as opposed to just randomly bouncing around with each question.

  26. 26
    shortstop says:

    It wasn’t so much that Watson almost always knew the answer–although it did a fascinatingly large amount of the time; it was that unlike its human opponents, it wouldn’t guess if the probability of being correct were too low. Yesterday’s show had a long run of low probabilities toward the end, and Watson was silent throughout. Conversely, a few times the right answer was low on, or completely absent from, Watson’s list of top matches.

    After watching for a couple of days, it became easier for me to predict which questions Watson would excel at: things that required straight data dumps like song lyrics, authors, geography and so forth always had super-high probabilities. The whole experiment, and what it tells us about natural language and the complexities of inference and association, has been really fun to watch.

  27. 27
    shortstop says:

    @evap: Didn’t see your post until now. I wish I’d been there for that talk — sounds great. And I somehow didn’t pick up that the algorithm for confidence/probability changes as the game advances.

    I can’t get enough of this stuff.

  28. 28
    Dave Ruddell says:

    I’ll just chime in with the other former contestants; the signaling device rules all in the game. I lost to an eventual 5 time winner, and it does mostly come down to the buzzer. The thing is, I thought I had the timing down pretty good in the practice, but during my actual game, not so much.

  29. 29
    joes527 says:

    I think the buzzer thing was an unfair advantage. I’m not sure how Watson was getting signalled that Alex was done reading the question. Was it listening to Alex speak and determining on its own that he was done? Or was it getting an external “all clear” signal? If the later, then this is a huge advantage. (even if the humans got the same signal, there is no way that they could match the computer’s reaction time)

    I was enormously impressed that it got the final jeopardy question. It was a particularly tricky path. “An account of the principalities of Wallachia and Moldavia” isn’t the answer. “Inspired by” is one of the less likely hard facts to be in the database, but even then, Dracula wasn’t the answer. Watson had to go from “An account …” -> Dracula -> Bram Stoker without stopping too soon or heading down a wrong path because of some stray word in the question.

    I too welcome out new machine overlords.

  30. 30
    shortstop says:

    Someone explain the “mastermix” reference to me, huh?

  31. 31
    gnomedad says:

    @MikeJ:

    From the Nova special I saw last week I thought they had fixed the bug where Watson gave the same wrong answer that another player had already given, but he did it on the show too.

    From what I’ve read, Watson got the questions via digital text and had no way of knowing what the humans were answering.

  32. 32
    joes527 says:

    @evap:

    It takes Watson’s mechanical finger 3 seconds to buzz and it only happens after the “end” light goes on

    Then they must have done some fancy editing. When you watch the show you can see Ken trying to buzz in <1 second after Alex finishes and getting smoked by Watson. Again and again.

  33. 33
    jayackroyd says:

    It’s funny that the more computers achieve human accomplishments, the clearer it becomes that AI will never come to pass. The solutions are always brute force, always narrowly focused and also clearly not the same thing as what people (and dogs and cockroaches FTM) do.

    The Atlantic has an article this month by a guy who was one of the participants in the annual Turing test event. The same problem is apparent–it’s really just a matter of time before the algorithm craps out.

    Also, Watson didn’t really work quite right.

    Does anyone know whether the machine was processing the audio of the questions being read? That would be a great deal more impressive than a direct interface to a digital question feed, which I infer from this quote in the second link: the system that was the interface between Watson and the Jeopardy computer.

  34. 34
    jayjaybear says:

    The buzzers on Jeopardy don’t even begin to work until Alex is done with the question and a set of lights on either side of the clue board go out. If you buzz in before those lights go out, you get locked out of the buzzer for something like 1/2 second, which can be devastating. My guess is that Watson was linked into the buzzer lights circuit so that his button algorithm started when they went out.

    Anticipating (like someone mentioned in their college bowl experience above) therefore doesn’t work on Jeopardy.

  35. 35
    jayackroyd says:

    @jayjaybear:

    This is, by the way, a change from the way the game worked back in the Art Fleming days. The buzzers activated as soon as the question was revealed.

    I actually think this is a more interesting challenge, because you have to buzz in speculatively based on the category and the amount. But it’s probably worse teevee, because you get a fair number of stumped contestants, especially late in the program.

  36. 36
    me says:

    @joes527: He probably meant 3 milliseconds becuase there’s no way it took 3 seconds for him to buzz in.

  37. 37
    shortstop says:

    @jayackroyd: No audio.

  38. 38
    Corner Stone says:

    @jayackroyd:

    The solutions are always brute force, always narrowly focused and also clearly not the same thing as what people (and dogs and cockroaches FTM) do.

    I’m not so sure humans are any different, IMO. I think our wiring just leads some of us to brute force an answer faster than others.

  39. 39
    Rick Massimo says:

    This all reminds me of the mid-’70s, when people were laughing about how awful those stupid computers were at trying to play chess. Give it time. (Although I do understand that the breakthroughs in computer chess were different.)

  40. 40
    Brachiator says:

    @joes527:

    I think the buzzer thing was an unfair advantage. I’m not sure how Watson was getting signalled that Alex was done reading the question. Was it listening to Alex speak and determining on its own that he was done? Or was it getting an external “all clear” signal? If the later, then this is a huge advantage.

    From Ars Technica:

    Though Watson seemed to be running the round and beating Jennings and Rutter to the punch with its answers many times, Welty insisted that Watson had no particular advantage in terms of buzzer speed. Players can’t buzz in to give their questions until a light turns on after the answer is read, but Welty says that humans have the advantage of timing and rhythm.
    __
    “They’re not waiting for the light to come on,” Welty said; rather, the human players try to time their buzzer presses so that they’re coming in as close as possible to the light. Though Watson’s reaction times are faster than a human, Welty noted that Watson has to wait for the light. Dr. Adam Lally, another member of Watson’s team, noted that “Ken and Brad are really fast. They have to be.”

    Also, as some others have noted, what made this contest most interesting was not that Watson could come up with answers, but that he could “understand” so many of the questions.

    And we should began to worry if, after conquering everyone in a rematch, Watson gloats, “Take that, human bitchez!”

  41. 41
    Evan says:

    On the Final Jeopardy question from the first game, the phrasing was more like, “Its largest airport is named after a WWII hero; its second largest, a WWII battle.” The phrasing of the second clause is ambiguous, and Watson didn’t know how to transfer over “airport” from the first clause.

    This seems like the most difficult thing to teach a computer: how to use context to fill in the blanks. We see a paragraph full of demonstrative pronouns and dangling modifiers and participles, and we know what everything is pointing at almost instantly. Do we use a consistent algorithm that could be implemented in software? Do we assemble a list of the possibilities and eliminate them until we only have one left? Weigh probabilities? Some combination of these? Or do we have a set of strategies, from which we choose the most appropriate for the situation?

  42. 42
    RobertB says:

    @jayackroyd – they stopped doing it that way because it became a speed reading contest.

    Also @jayackroyd – ‘never’ is a pretty big word in discussions of what may and may not come to pass in the field of AI. Even in your example, the best machine candidate failed their implementation of the Turing Test by one vote. Given the history of computing, that room full of hardware that plays a halfway-decent game of Jeopardy will in ten years be 32 times faster and fit on a box that will sit on your desk.

  43. 43
    Observer says:

    @evap:

    It takes Watson’s mechanical finger 3 seconds to buzz and it only happens after the “end” light goes on (Watson gets an electronic signal).

    What mechanical finger? didn’t see one. If I had an electronic stent hooked directly to my brain that would buzz the Jeopardy buzzer, I’d beat Ken Jennings too.

    Probably so would you.

  44. 44
    New Yorker says:

    Buzzer timing makes a huge difference, especially at a high level. Watching Jennings or Rutter, I would often come up with the right answers as well, but by the time my mind had processed the information, they had already buzzed in.

    I’m close friends with Brad Rutter’s younger brother, and he told me that Brad would often buzz in before he knew the answer, and would figure it out in the time between buzzing and answering. Pretty crazy.

  45. 45
    Steve M. says:

    I’m completely failing to grasp why everyone’s so worked up about this. I’m a fairly good Jeopardy player, or I used to be (I once made it to the second round in a tryout in Atlantic City), but I’m no more intimidated by Watson than I am by Google’s ability to find stuff online faster than I can in a library card catalog, or a car’s ability to go faster than I can run, or a telephone’s ability to transmit a message farther than I can yell. That’s what machines do. So what? They’re not thinking — they’re just acting on our commands.

  46. 46
    joes527 says:

    @Brachiator:

    but Welty says that humans have the advantage of timing and rhythm.

    Watson pwned Ken (a player particularly known for his buzzer-fu) on the buzzer.

    Watson was amazing in its ability to answer, but it absolutely destroyed the humans on the buzzer.

    Welty need to reconsider his assumptions about who has the advantage.

  47. 47
    RSA says:

    @Brachiator:

    Welty insisted that Watson had no particular advantage in terms of buzzer speed.

    Interesting. I happen to know Chris Welty, though it’s been years since we’ve talked. (I also haven’t really followed the Watson story until this morning.)

    I think Welty’s basically right about the issue not being input or output. Watson gets questions through a direct text feed, but we could imagine hooking it up to a machine vision system. Watson has a mechanical finger that presses the button but doesn’t anticipate end of the question, when it’s legal to press the button. But all this is just timing, differences of tens or hundreds of milliseconds–change the parameters, and the contest could go either way. The interesting things have to do with Watson’s strategies for choosing categories and with the kinds of questions it’s able to answer.

  48. 48
    Corner Stone says:

    @New Yorker:

    I’m close friends with Brad Rutter’s younger brother, and he told me that Brad would often buzz in before he knew the answer, and would figure it out in the time between buzzing and answering. Pretty crazy.

    I would argue this is how most of us would play the game as well. Using a pretty basic risk assessment strategy it doesn’t make sense to wait except in certain situations.

  49. 49
    Herbal Infusion Bagger says:

    Mistermix, you got a shout-out on Savage Love:

    Writing about Representative Christopher “Craigslist Congressman” Lee last week on Balloon Juice—a blog I read daily—BJ contributor “Mastermix” said: “If we want to do something about the high divorce rate, we might want to get real [about] the role of a discreet, mutually-agreed-upon affair as a safety valve.”

    http://www.avclub.com/articles.....011,51890/

    Congrats.

  50. 50
    Gina says:

    Rensselaer Polytechnic Institute had a three-day screening of the Watson Jeopardy Challenge, with a panel that included IBM team members, that talked about all of these issues. The buzzer thing is hashed out, and the Toronto thing is too. The panel’s discussions and audience questions are online now.

    I was able to take my kids, 9 & 12, and while it was geared toward adults, they got a lot out of it and had a good time getting an inside look from the panel members. Oh, and they got a huge kick out of Watson pwning Jennings.

  51. 51
    gogol's wife says:

    My husband pointed out that this clue is bogus. Midway was called Midway before WWII and was not named after the battle.

  52. 52
    evap says:

    @Evan: Exactly. The speaker yesterday talked about “elided” pieces of a sentence and how difficult it is to code this into Watson. The two pieces of the sentence are independent clauses, but part of the second piece is missing (implied). They are working on how to get Watson to recognize that the second clause was “it’s second largest AIRPORT is named…”

    Watson has an easier time if the question has “this”, as in “this city” … or “this author”… Then it better knows what “type” the answer should be.

    I may have the “3 seconds” wrong, that is how I remember it from the talk. The speaker claimed that if a human anticipates the end of a sentence, i.e., begins sending a single from the brain to the finger before the light goes on, then the human will buzz faster than Watson. The best players do this, although it’s tricky because you can get locked out if you signal too soon.

    @Steve M.: The point is that Watson can’t just “look things up”. We can figure out from context and other very subtle clues what is being asked. Watson’s “brain” is much cruder, in some sense.

    The speaker from yesterday is giving a series of lectures at Georgia Tech today and tomorrow. I bet IBM will eventually put them up on their website for all to see.

  53. 53

    […] Mastermix notes that, “players are surprised at how big a role button management plays in winning or losing a round. In the few minutes of the Watson game that I watched, it was pretty clear that Watson was excellent at pressing the button at exactly the right moment if it knew the answer, which is more a measure of electromechanical reflex than human-like intelligence.” […]

  54. 54
    shortstop says:

    @gogol’s wife: Don’t know if it was named after the battle or not, but it was Chicago Municipal Airport until 1949, when it was renamed Midway.

  55. 55
    Brachiator says:

    @RSA:

    The interesting things have to do with Watson’s strategies for choosing categories and with the kinds of questions it’s able to answer.

    Yep. I didn’t see the full Jeopardy episodes, but now I want to go back and catch up. I’d also like to know more about the issue of Watson not being able to “hear” other contestants give an incorrect answer.

  56. 56
    shortstop says:

    @evap:

    The speaker from yesterday is giving a series of lectures at Georgia Tech today and tomorrow. I bet IBM will eventually put them up on their website for all to see.

    Hope so — I’ll be watching for them.

  57. 57
    FreeAtLast says:

    Pragmatically speaking, I think focusing on the buzzer advantage is irrelevant to Watson’s real future. What I’d like to see, at various confidence thresholds and various time limits, is the number of correct/no answer/incorrect that Watson comes up with on a large number of categories/”answers”.

  58. 58
    swbarnes2 says:

    I Won Ben Stein’s Money, and that was all about the buzzer. Before they start, they give you a bit of practice with the light, and one of my competitors didn’t quite get the hang of it. And I beat him to the buzzer enough times.

    Ben Stein’s strategy was to hold the buzzer behind his back, and pound on it, and it did work, he held me to only one question that round.

    I did this kind of thing in high school, and a bit in college, and yes, ringing in before the question was quite done was the key, either anticipating the answer, or expecting an extra syllable. You also had a few seconds after buzzing in to figure out the answer, so you counted on solving the question then. Except for math, math questions had to be answered immediately.

    For instance if the question is “Name the author of ‘Redburn’, ‘White Jacket’ and “, that ‘and’ is the clue that the giveaway title is just to follow. So you buzz in then, and “Mo-” comes out, and then you know.

  59. 59
    jayackroyd says:

    @Corner Stone:

    Biological systems are fundamentally different. You cannot represent neurotransmitter flows on a paper strip consisting of a sequence of ones and zeroes, as you can with any von Neumann machine, including Watson. Nor can you represent synapse firing patterns a binary logic gate.

    The expectation was that with enough raw processing power, learning systems would emerge, because the heart of consciousness and intelligence is rooted in learning. That’s proven not to be the case.

    Even with an enormous amount of processing power, we are still limited to very narrowly defined areas of general intelligence where computers perform at human levels. And we’re closer to fusion than we are to machine consciousness.

    I don’t think either machine intelligence or consciousness is possible using anything based on current technology.

  60. 60
    Bubblegum Tate says:

    @Hyd:

    In a straight-up 50-question test, KenJen would beat Watson.

    Ken Jennings is the Michael Jordan of Jeopardy. His run was absolutely riveting.

    Also: I am envious that you got to be on Jeopardy! It’s something I aspire to someday…probably well in the future when I’ve had time to get English kings, U.S. VPs and the wide, weird world of art down cold.

  61. 61
    jayackroyd says:

    @RobertB:

    speed reading wouldn’t help. You had to buzz in before you even started reading. You couldn’t answer until after Fleming had read the entire question.

    I don’t know why they changed it. It’s speculation that it sucked to have the last half of double jeopardy taken up with the trailing players buzzing in, and then silent.

  62. 62
    Corner Stone says:

    @jayackroyd: Hmmm. I think associative thinking is the next step, and I think it’s inevitable.
    As a biological wetworks, I’m not convinced it’s the ability of our brain to crunch X-times per second as it is to skip past all the data and associations that are not helpful or irrelevant. I think we brute force a lot of solutions, but our contextual ability help winnow down the focus.
    IMO, Watson et al are right on the cusp of this.

  63. 63
    Triassic Sands says:

    @evap:

    They are working on adapting Watson to help doctors diagnose and things like that.

    Is that going to lower the cost of medical care?

    I read somewhere that when Jennings was on Jeopardy he supported giving other contestants more practice time with the buzzer, because he felt his longer experience gave him an unfair advantage. If true, then Jennings is a very good sport.

    The New York Times gives people the opportunity to play against Watson (just search for “Jeopardy Watson”), although it isn’t under Jeopardy rules, since the human contestant (you) always gets the opportunity to answer every question, i.e., there is no buzzer. I was surprised by how little Watson knew. I know that I would never beat Watson in a game of Jeopardy that involved a buzzer, but beating him in the Times game was easy. Of course, the categories matter and Watson may have knowledge that I don’t have about a lot of things, but the categories in the Times game are not ones that I would generally consider strong ones for me.

    After the game was over, I started it again, and this time I passed on every question, so I could see how many Watson would get correct if he won every buzzer race. The result was surprising. Given the opportunity to answer every question Watson only got 17 out of 30 correct. He was wrong or didn’t know the answer 13 times. It doesn’t take a great intellect to top that.

  64. 64
    Kilkee says:

    @chmatl: Been there, done that. About 20 years ago. Only recently can I watch the video. And yes, the buzzer is a killer. I don’t know how they hav enot come up with an improved method after all these years. Perhaps they like it the way it is.

  65. 65
    shortstop says:

    @Triassic Sands:

    Is that going to lower the cost of medical care?

    After Big Pharma gets done inserting product placements all over the “indicated treatments”?

    (Too cynical?)

  66. 66
    Bill Arnold says:

    @me:
    For whatever it’s worth, top Jeopardy contestants are often able to time the buzz within something like 10 milliseconds (I’m told). This is not simple reaction time which is more like 150 milliseconds.
    If one wanted to change the rules of the game, one approach might be for the game to give the buzz to everyone who buzzed in within 20 milliseconds. One would need to tune it against actual timings from top contestants.

  67. 67
    Triassic Sands says:

    @shortstop:

    (Too cynical?)

    It isn’t possible to be too cynical when discussing Big Pharma. In fact, any level of cynicism attainable by a human being is still going to be inadequate concerning these worst of all corporations (tied for worst, anyway, with many others).

  68. 68
    gogol's wife says:

    @shortstop:

    God, my husband was wrong about something having to do with Chicago. I’ve never known that to happen. Thanks!

  69. 69
    Quiddity says:

    What they should have done is group the buzzer responses into buckets of, say, .1 seconds. If more than one contestant is in that bucket, then it’s randomly assigned. That way you don’t over-privilege buzzer-skill (especially Watson’s), while still retaining an ordering of response times (now with the granularity of, in this case .1 seconds).

  70. 70
    shortstop says:

    @gogol’s wife: If you play your cards right, you might get a free deep-dish pizza, Bears jersey or Empire Carpet installation out of this! Work it, baby.

  71. 71
    Bill Arnold says:

    @Quiddity:
    Seriously, the top (human) Jeopardy players are able to time the buzz to more like 0.01 second. (I’m sure there was discussion with Sony about the buzzer rules, and they decided to stick with the normal rules.)
    The bucket idea is what I personally would have done, though with much smaller buckets.
    I would have liked to see a real meat arm though, or maybe a tentacle extending all the way from the server room.

  72. 72
    Amir_Khalid says:

    The post headline is Weird Al, not Greg Kihn, right?

  73. 73
    Rita R. says:

    Gonna chime in with the other former contestants here and agree the buzzer factor is huge. I was surprised by that and my thought at the time was that it favors video game geeks (who tend to be male and younger) who’ve spent many, many hours fighting virtual battles in their basements.
    Don’t know about the “bucket” idea though, since it’s just replacing one non-knowledge based factor — buzzer timing — with another — random assignment.

    I did manage to pull out the win one day, despite my mediocre at best buzzer skills. Which led to my other big surprise: how much of your winnings you have to pay in taxes. My liberal self felt a brief moment of affinity for anti-tax Republicans at that. But it passed.

  74. 74
    jayackroyd says:

    @Corner Stone:

    An associative word search algorithm is not thinking. Yes, we do think associatively. But, you know, smells, audio snippets other non digitizable elements are part of that process.

    In an otherwise deeply flawed book, The Emperor’s New Mind, Penrose said that you could have a computer that could predict the weather with perfect precision, and it would still not be raining in there.

  75. 75
    RSA says:

    @jayackroyd:

    You cannot represent neurotransmitter flows on a paper strip consisting of a sequence of ones and zeroes, as you can with any von Neumann machine, including Watson. Nor can you represent synapse firing patterns a binary logic gate.

    Actually, you can, just as we now represent in digital form what used to be stored on vinyl, for example. Of course, there’s error, as with a digital representation of almost any real-world phenomenon. The deep questions, I think, have to do with organization, processing, and and efficiency. As you point out, we don’t know how or whether our current techniques could lead to general intelligence.

    Also, AI aside, I’ve never been impressed with the generality of the argument that weather simulations don’t get you wet. (I think it originates with Searle.) Hook up a weather simulator to a special effects machine, for example. I could say that it’s not really raining, but I’d get just as wet.

  76. 76
    jayackroyd says:

    Just to note that what we really mean by an intelligent computer is something we all recognize. The best example, IMO, is Heinlein’s Mike Holmes/Adam Selene in Moon is a Harsh Mistress.

    That’s what we mean.

    We aren’t close. We aren’t three human generations close to that.

  77. 77
    jayackroyd says:

    @RSA:

    It may well originate with Searle, and in many ways, Searle is an asshole in these discussions. But he raises real issues.

    Look, I’m biased at this point. I’ve been skeptical forever–I still have a colleague who owes me an ounce of gold for a bet made in 1981.

    I try not to be biases as new stuff happens, but the new stuff consistently supports my view that machine consciousness/intelligence is just not gonna work inside this architecture. You can say that you can simulate analog neurotransmitter processes, but, sheesh, can you really simulate the scent of …..well, you can fill in….Human (and animal “Mayr: “What is the source of human consciousness? Animal consciousness”) cognitive processing is deeply tied to a sensorium that itself defies digitization. I don’t deny that it is really instructive, useful, a great idea, to investigate this stuff. But we are very, very far away from anything like AI. I won’t claim it’s impossible. I will claim it is unattainable by von Neumann machines.

    I also think manned missions to Mars are a crock.

    So there.

  78. 78
    jayackroyd says:

    And it should fucking go without saying, but I’ve been in troll spaces in the last couple of days, and I wanna say how much I appreciate a thoughtful response to a point I’ve made.

  79. 79
    jayackroyd says:

    @RSA:

    Sorry.I’ve rushed my first response to this.

    It’s worth noting that the vinyl representation of the live performance sucks. The fact that its analog doesn’t give it special weight. There’s much to be said for th

  80. 80
    RSA says:

    @jayackroyd:

    I don’t deny that it is really instructive, useful, a great idea, to investigate this stuff. But we are very, very far away from anything like AI. I won’t claim it’s impossible. I will claim it is unattainable by von Neumann machines.

    I’m on board, except for the last sentence. I think it’s an empirical question, but of course reasonable people can have different opinions about the eventual outcome. What gives me some hope is progress that people continue to make on harder and more general problems. Not Jeopardy, per se, but things like probabilistic reasoning and unsupervised learning.

    Coincidentally, I’m also a fan of Mayr’s philosophical work, though I actually haven’t come across his thinking on consciousness. I’ll have to look it up. For me, consciousness (p-consciousness rather than a-consciousness, if you’re familiar with Ned Block’s work) is a philosophical issue that computer science won’t crack.

Trackbacks & Pingbacks

  1. […] Mastermix notes that, “players are surprised at how big a role button management plays in winning or losing a round. In the few minutes of the Watson game that I watched, it was pretty clear that Watson was excellent at pressing the button at exactly the right moment if it knew the answer, which is more a measure of electromechanical reflex than human-like intelligence.” […]

Comments are closed.