MindMeld – you may know the term best from StarTrek and those fun-loving Vulcan practices. But it lives too at Expect Labs, as an app that listens to and understands conversations and finds relevant information within them, and as an API that lets developers create apps that leverage contextually-driven search and discovery – and may even find the information users need before they explicitly look for it.
Anticipatory computing is the term Expect Labs uses for that. “This is truly a shift in the way that search occurs,” says director of research Marsal Gavaldà. “Anticipatory computing is the most general term in the sense that we have so much information about what users are doing online that we can create accurate models to predict what a user might need based on long-ranging history of that user profile, but also about the context.”
The more specific set of functionality that contributes to the overarching theme of anticipatory computing, he explains, “means that you can create intelligent assistants that have contextual search capabilities, because our API makes it very easy to provide a very continuous stream of updates about what a user is doing or where a user is.”
Also critical to supporting anticipatory computing and intelligent assistants is the power of its Knowledge Graph – the tying of a reference to an entity out there, whether person, location, product, movie or book, in order to anchor them together and find all the properties about those objects. “That is the meaning of the Knowledge Graph,” says Gavaldà, noting Expect Labs has a baseline of public general knowledge of the world based on Wikipedia but augments that with custom data, such as vertical-specific information. Think of deep knowledge bases like IMDB for movies “but in a very clean way,” he says. “So we have a baseline Knowledge Graph and enhance that based on the specific app and also on more dynamics. As we get more contextual updates we can extract on-the-fly entities.”
The App And The API
The MindMeld intelligent assistant app for the iPad showcases the API, which is the company’s main focus, the goal being to encourage developers to build similar apps tailored to their own document collections or data sources. The MindMeld API leverages schema.org and Open Graph tags as part of the document ingestion process for creating an app. “The API is fairly clean and well-designed where you can just post documents to a document collection or crawl a website so you don’t have to individually post every document,” he says. “But the key thing is if your documents contain schema.org or Open Graph tags we make use of those to speed up the creation of a custom Knowledge Graph because it’s easier if you provide us with structured data.”
You can instruct MindMeld to listen to voice conversations with friends, show you related information from across the web and your social graph as you talk, and analyze what was recently said to summarize the key points of a conversation. “One of the emphasis for Expect Labs is speech recognition and voice-driven apps because the most important context is what you say when you talk about something that’s obviously very meaningful, and it’s the highest indicator of intent,” says Gavaldà. With a voice-driven intelligent assistant, a user talking about Oscars and Tom Hanks to a movie database would have an easy way of getting back appropriate, in-context search results, for example.
Advances in speech recognition from deep neural and recurring neural networks have added up to a 30 percent reduction in word error rate, he notes. Additionally, MindMeld supports on-the-fly speaker adaptation, getting better at recognizing a speaker’s speech patterns the more it hears him talk. Equally important, though, is that “for these types of apps you don’t need 100 percent accuracy to get 100 percent results. If you are discussing movies and get the title or actor correct, the rest doesn’t matter so much. It’s the extraction of entities or the key phrase that matters.”
A location-centric intelligent assistant built on MindMeld’s API can grasp information about a user’s location from her mobile device, and infer, for example, that she’s going to work. Expect Labs can’t name specific companies leveraging this capability but Gavaldà says there is a lot of inbound interest from auto companies. “They all want to develop the next-generation assistant in the car and the key thing about the car is location,” he says – the context that matters here is not only where they are now but also where they are going and where they might be in the next half-hour.
“The timing is right,” he says. “The notion of a large number of mobile devices being able to provide contextual updates — the raw data of how location changes from second to second, and more aggregate analysis that the person seems to be walking or riding a bike or driving. Some operating systems like Android provides you with an entire level of abstraction that makes this all easier for us.”
Current State and Next Steps
As a scientist at his core, Gavaldà says he always sees opportunity for improvements. “There are two dimensions – engineering to scale the backend to hundreds of thousands or millions of users and the other is making sure we also improve accuracy from speech recognition to the Knowledge Graph entities and to the accuracy of results.” Also in the works is growing language support. Currently it supports English and half a dozen other Western languages, but it’s also actively working on Asian languages.
The company continues to focus on its tools such as its API Explorer for users to see all end points against their own data. And, says Gavaldà, because Expect Labs is taking into account initial feedback and extending its technology to areas like covering more languages, “we also are going to have a more granular definition of what we call the Activity Stream,” he says. Right now the most crucial contextual updates are text-based and others fall into a catch-all category. “So there are a lot of other activities not modeled in a granular way and we are adding some of those,” he explains. Also be on the lookout for Expect Labs to go beyond providing developers with same code to offer some full-fledged apps for different verticals, to make it easier for them to take advantage of the backend of the system.