IBM Watson is a platform that represents a new era of computing based on its ability to interact in natural language, process vast amounts of disparate forms of big data and learn from each interaction. It is able to sift through and understand massive amounts of big data at unprecedented speeds to assist professionals in understanding data quickly and easily, while increasing knowledge and gaining value over time. IBM Watson is not only solution in this space, Digital Reasoning and several other players also have compelling offerings. IBM Watson just happens to have the most name recognition.
What is IBM Watson?
Watson is a cognitive system that sifts through massive libraries of data to discover insights that can help its users answer simple to the most complex of questions. Watson understands the nuances of human language. It is able to bring back relevant answers in context of the question. It also gets smarter, learning from each interaction with its users, and each piece of data it ingests. Watson is not a ‘search engine’ or deep learning that requires data scientists to interpret results. It does much more.
Watson can “think” or “reason” in a way very similar to the human brain. It processes information, draws conclusions, and learns from its experiences. Watson does not use predefined rules and structured queries to uncover answers, it instead generates hypotheses based on a wide variety of potentially relevant information and connections. Answers are expressed as recommendations along with confidence rankings. Unlike a traditional analytic tool, the more data it receives the more it can learn and provide higher quality insights.
IBM Watson Solutions
IBM Watson is an ecosystem of cognitive computing capabilities
Navigating the IBM Watson ecosystem can be a bit daunting at first. Be sure to read through the Cognitive Business Solutions to get an initial overview.
Right now there are several key areas of innovation.
- Watson Explorer Platform: the core expandable cognitive indexing and natural language search framework
- Watson Developer Cloud: a collection of Watson REST APIs that can be used in apps
- Watson Industry Solutions: IoT, Marketing, Supply Chain, Health, Education, Financial Services, Regulatory Compliance, Surveillance, Banking, Insurance and many more.
IBM Watson Explorer
One of the core IBM Watson offerings is Watson Explorer content analytics engine, platform and framework.
The IBM Watson Explorer platform ingests and analyzes all types of data and media files. The platform enables the following:
- Powerful indexing and natural language search
- Hybrid cloud connectors for information access
- Advanced multi-media content analytics to aggregate and visualize unstructured content to reveal hidden insights and patterns
- App Builder framework for quickly creating custom cognitive applications
- Integration of cognitive results into analytics applications
IBM Watson Explorer foundation components are the Watson Explorer Engine, Watson Explorer Application Builder, and Watson Explorer Results Module. For business users, analytical components such as Watson Explorer Content Analytics and Knowledge Studio are used to mine information and review hidden insights.
Content Analytics Miner, included with Watson Explorer Advanced Edition, is an interactive content mining tool that helps business users mine large amounts of text for new business insights using a series of views that can show trends, patterns, and anomalies in information.
Watson Knowledge Studio enables business analysts to create advanced, rule-based annotators without writing code. Through a simple drag-and-drop interface, a subject matter expert can build domain- and language-specific resources into dictionaries and create parsing rules to identify facets, entities and relationships for Watson Explorer.
The cloud-based application that enables developers and non-technical domain experts to collaborate using one common tool.
The best way to understand what these tools do is to watch a short video demo.
How it Works
Watson Explorer is entirely built around XML for unique flexibility and integration. Search systems collect, parse and process all types of structured and unstructured data, and store information in an index to facilitate rapid retrieval and relevance calculations.
A powerful aspect of the Watson Explorer language is that it enables the creation of unstructured document conversion, images, video, audio, translation between languages and parsing agents that can be run in parallel. Historically many of these data sources were considered dark data – impossible to easily analyze. The capability to extract information from those sources is invaluable. Any resource that is accessible locally or in the cloud via HTTP(S), LDAP, REST, SOAP or other framework connectors can be processed.
After information is ingested, it gets indexed. Index design determines the cognitive features and capabilities that can be offered to developers, administrators and end users.
Behind the scenes, Waston uses a position-based index that offers numerous advantages over the traditional vector-based index approach used almost universally in commercial and open-source search systems. A compact position-based structure provides the foundation for intelligent query processing, content refreshes and even security. Watson Explorer’s use of positional indices helps liberate the information from the conceptual limitations of monolithic document models.
Linguistic processing is performed in two stages. First when a text document is processed to be added into the index, and again when a user enters a query. Linguistic support is available for semantic search and dictionary-based segmentation. There are over 330 domain specific libraries that can be used. Hybrid segmentation combines the high precision benefits of dictionary-based segmentation with the high recall benefits of non-dictionary based, n-gram segmentation, stop word removal and character normalization.
Another feature of Watson Explorer Engine is alerts. Today there are three types of alerts that can be used to notify users via email of interesting results that may required action.
Integrating IBM Watson into Apps
After information is processed with the cognitive framework, addition output destinations and integration with analytics applications is commonly done to vastly improve decision making context in apps the business already uses.
Here’s an example of a Watson Explorer unified information application. Clients can use this powerful, easy-to-deploy application framework to create apps to support specific activities such as customer care, research, marketing, sales and more. Virtually any activity that requires information can be enhanced by a Watson Explorer unified information app.
The benefits of cognitive apps include the powerful combination of structured and unstructured information for contextually relevant insights. The flexible architecture based on search indices is adaptable and less “brittle” than structured approaches. Also integrated cognitive services within business applications help scale human expertise.
For More Information
I have barely touched the surface on this topic. If you are interested in getting started with IBM Watson and exploring that ecosystem offerings, the following are a list of starting points that I found in my studies of this vendor.
- Cognitive Business Solutions
- IBM Watson Explorer Home
- Demos and White Papers
- IBM Developer Works
- IBM Bluemix Cloud