Video of the full presentation referenced in this article is available here.
During the three-day Jeopardy challenge that saw IBM’s Watson emerge victorious over mere mortals Ken Jennings and Brad Rutter, viewers learned a bit about the future IBM envisioned for its Deep QA system outside the game show circuit. At SemTech in San Francisco, IBM research staff member Aditya Kalyanpur provided a little more insight into the technology’s capabilities and its suitability for other domains where structured and unstructured data abound.
“What the technology really enables is better decision-making over vast amounts of structured and unstructured data,” said Kalyanpur, who joined the Watson project a couple years back after working on other research at IBM. (The Semantic Web Blog featured Kalyanpur in his pre-Watson days as one of the young guns moving the technology forward in this article.) “It’s applicable in so many domains. The bottom line is any domain where you need to make sense of vast amounts of information, this can work,” he told the crowd assembled to hear more about the inner workings of the charismatic computer system.
Such domains run the gamut from corporate enterprises, where decision-makers spend hours, days, or weeks searching the web and reading documents and analyzing evidence within and across them to draw conclusions with confidence, all the way through to fields such as tech support, legal and health-care. “With an expert QA system a lot of that hard work [of reading and analyzing] is done by the system,” he said. “You can ask a question in natural language and let the machine produce answers along with evidence and give you a nice summary report with the answers, evidence and confidence.”
The first vertical frontier poised to benefit will be the medical sector, for which an application using Deep QA is to be built. Think of the diagnostics requirements for doctors, who have to consider and cull through a wealth of information in assessing patient conditions, from symptoms to individual history to family history, all the way to data from medical journal reports or databases. “They all can become evidence dimensions in the system and the machine can learn in the medical domain how to combine these pieces of evidence,” Kalyanpur explained as an onscreen illustration demonstrated how confidence in possible diagnoses changed in real-time as the technology went about evaluating the various dimensions.
“Because of the evidence profile the doctor can interact with the system to see what is going on,” he said. “There may be multiple potentially correct answers, so producing a hypothesis and associated confidence with each of them is crucial. And [producing] evidence, too, so we understand what produced that confidence.”
Recall from the Jeopardy challenge that viewers were able to see how confident Watson was in the answer that it provided in each instance. Digging into the “brains” of Watson shows that its evidence profiles create the conditions for its confidence in one answer over another. As an example, Kalyanpur demonstrated how Watson determined that it was Argentina that shares the largest land border with Chile rather than Bolivia, another border-sharing country that it considered. “Watson prefers Argentina because it wins out on location, passage support, popularity, source reliability and classification. Bolivia wins on popularity, because apparently there is a commonly discussed border dispute between it and Chile, but other evidence far outweighs popularity,” he explained.
Behind the scenes Watson and its Deep QA is generating multiple interpretations of a question and multiple candidate answers, each of them a hypothesis that must be tested and validated. Scoring of these is accomplished using a lot of different algorithms on more than 500 dimensions, and then the information is handed off to a machine-learning model trained on previous questions to do the work of determining confidence levels. It’s all happening in parallel, too, exploiting IBM Power 7 core processors to get everything done in just a few seconds.
The greatest impact comes by way of unstructured text analytics, learning patterns in data and extracting knowledge to create, for example, semantic frames of relationships –such as that ‘inventors patent inventions’ or ‘people earn degrees at schools.’ But “there’s also a wealth of freely available structured information, like Linked Open Data, so it would be silly not to use it too,” Kalyanpur said, adding that IBM wanted to complement the results from unstructured text analytics. Additional value comes from the fact that spatial and temporal constraints, for example, are easier to check against validated structured knowledge bases. Structured data also comes in handy for reconciling entities, too, such as when an entity-answer is already in a clue. For instance, if a Jeopardy clue mentions that Big Blue acquired a company, Watson would know that the answer can’t be IBM because Big Blue is IBM.
Kalyanpur also discussed capabilities in the areas of missing links, where Watson can figure out the entity that is not expressed in a clue but is implicitly related to it, and in decomposition and synthesis. The latter made it possible for Watson to figure out in the Jeopardy rhyme time category that “a long tiresome speech delivered by a frothy pie topping” could be broken apart and recombined to deliver the answer “harangue meringue.”
But, as smart as Watson is, it can – and did – make mistakes. In one of the Rhyme Time challenges, Kalyanpur recounted, the clue was related to a rhyme about boxing and hitting below the belt. The correct answer: Low blow. The one Watson came up with: Wang Bang.
Kalyanpur spoke in more detail with The Semantic Web Blog after the presentation about DeepQA and the Watson project. Click on the video below to discover more.