The AI platform of IBM to support Hindi language to expand India presence
Bengaluru, NFAPost: IBM announced innovative new capabilities planned for IBM Watson designed to help scale the use of AI by businesses, especially by bringing on board NLP technologies for Hindi language.
Developed by IBM Research, the new capabilities are designed to improve the automation of AI, provide a higher degree of precision in natural language processing, (NLP) and foster greater trust in outcomes derived from AI predictions.
New Capabilities
* Reading Comprehension is based on an innovative question-answering (QA) system from IBM Research. Currently in beta in IBM Watson Discovery, it is planned as a new feature that can help identify more precise answers in response to natural language queries from vast troves of complex enterprise documents. It also provides scores that indicate how confident the system is in each answer.
* FAQ Extraction uses a novel NLP technique from IBM Research to automate the extraction of Q&A pairs from FAQ documents. Currently in beta in IBM Watson Assistant’s search skill, it is planned as a new feature to help businesses keep virtual assistants up-to date with the latest answers and reduce the time-consuming process of manual updates.
* A new intent classification model is now available in IBM Watson Assistant. It is designed to more accurately understand an end-user’s goal or intent when engaging with a virtual assistant and to enable administrators to train the system faster. The model provides more accurate results from less data versus compared commercial systems1. This can help businesses go live with virtual assistants in a few days while achieving high accuracy.
* Watson Discovery now includes support for 10 new languages including Hindi. With this capability, it can now natively understand the meaning of Hindi text written in Devanagari script and extract useful information from it such as business keywords, named entities, relationships, perform sentiment analysis, and other such NLP features. IBM Research – India collaborated with its AI Horizon Network partner – IIT Bombay to lead this effort.
AI Factsheets
The move is widely seen as IBM efforts to advance AI governance capabilities to help build trust and transparency in business outcomes. IBM also announced plans to commercialise IBM Research-developed ‘AI Factsheets’ in Watson Studio in Cloud Pak for Data throughout the next year.
Like nutrition labels for foods or information sheets for appliances, AI Factsheets are designed to provide information about a product’s important characteristics. Standardising and publicizing this information will help build trust in AI services across the industry.
To complement this, IBM Services for AI at Scale, a new consulting offering, provides businesses with a framework, methodology, and underlying technology to guide organizations on their journey to trustworthy and ethical AI. IBM Cloud Pak for Data also has new capabilities to provide a complete foundation for AI that can run on any cloud.
Innovations for clients
Commenting on the development, IBM Research India Director Gargi Dasgupta said the company’s announcement reiterates IBM’s focus on helping clients innovate across all three foundations of AI – natural language processing, building trust, and automation.
“IBM Research India has been key partner and leader in IBM’s AI advancements with a focus on Trustworthy AI initiatives and along with US team (IBM Research Yorktown) conceptualized the core idea of AI FactSheets. In the area of natural language processing – by using a combination of AI-deep learning, shallow semantic understanding models we have enabled Watson to understand Hindi language natively,” said Gargi Dasgupta.
For instance, Gargi Dasgupta said Watson Discovery can natively understand documents in Hindi and extract meaning, without having to resort to any translation to English — an outcome of the company’s collaboration with IIT Bombay’s Center for Indian Language Technology as a part of IBM’s AI Horizon Network.
AI Horizon Network
IIT- Bombay – Department of Computer Science and Engineering Prof Pushpak Bhattacharyya said India has taken positive steps to establish a leadership role in AI and IBM-IITB AIHN (AI Horizon Network) project is an important step – both for science and technology and for India’s progress.
“The collaboration is focused on automation of cognition and perception, with emphasis on cutting edge research, high-quality publications and open and widely usable AI resources. There are social and commercial needs, whose servicing requires user interaction and information dissemination in Indian languages (IL), however, the complexity of Indian languages, low Indian language corpora and other constraints makes adoption and adaptation of English centric NLP very difficult for IL-NLP,” said Prof Pushpak Bhattacharyya
Through the partnership with IBM, IIT- Bombay – Department of Computer Science and Engineering has been able to use machine learning for IL-NLP and address challenges related to low resource, understanding of Hindi language sense, intent, sentiment and more.
“In two years, the endeavour has seen stellar publications coming out of the collaboration and creation of quality software (in NLP, Speech and Multimodal AI),” said Prof Pushpak Bhattacharyya.
Infuse intelligence
Leading global brands are using IBM Watson to infuse intelligence into their workflows. For instance, a global leader in assurance, tax, strategy and transactions and consulting services, EY is transforming the M&A due diligence process using IBM Watson Discovery.
Following the global expansion of its alliance with IBM earlier this summer, EY launched Diligence Edge, which includes a custom NLP model trained on EY proprietary M&A language, to help dealmakers identify and leverage valuable insights throughout the transaction’s lifecycle, as well as make informed decisions on deal value opportunities to gain an advantage in competitive deal processes.
Today’s news is another step from IBM to advance its AI for business strategy: from commercialising advanced NLP capabilities from Project Debater; expanding its AI-powered automation capabilities with the launch of IBM Watson AIOps, acquisitions of WDG Automation and Instana2 and expanded ecosystem partnerships including ServiceNow; and helping businesses, healthcare organizations, governments and academic institutions use IBM Watson Assistant to get information they deem trustworthy on the pandemic to stakeholders.
