I am really fascinated by the Watson series on Jeopardy thus far. What stuck out to me were not the answers he got right, but the answers that the computer eliminated or the answers he got wrong. IBM said it best: “Watson not only knows what he knows, but he knows what he doesn’t know.” This is definitely more of a challenge than getting a question right, as we have seen many computers do “well” at things.

Watson’s quirks were due to limitations in hardware as well as simply guessing incorrectly. Watson receives the questions via a txt file from the Jeopardy board, and cannot hear any audio input. As such, if a contestent guesses an incorrect answer, Watson cannot use the information to move to his second or third guess. Furthermore, Watson will only guess if the answer is past a “guessing threshold”, indicated by a line on the screen. This threshold changes depending on the confidence of the¬†question¬†itself. So, there were a couple of guesses he didn’t make that were actually correct, but the threshold wasn’t high enough. Finally, if Watson guesses incorrectly, it was more likely grossly incorrect than close at the right answer. This seems to suggest that Watson didn’t understand the question, rather than just having guessed wrong. You can make that assumption because the top three guesses he would have would be not even close for a couple of mistakes Watson made.

Right now Watson is tied with Rutter at $5000, and Jennings is at $2000. Jennings looks determined to make a comeback, so it will be interesting to see how the computer does tomorrow. The series ends on Wednesday.

Advertisements