Chat Bot Challenges(Azure Services)

Chat Bots, Recommendation Engine etc. have become popular in large number of business domains such as Finance, Banking, Tour & Travel etc. Companies are building chatbots for booking hotel, movies, suggestion etc., customer support, enquiring bus flights, tax saving advice, stock advice etc and many more.

They are required in such a large number because they reduce time , efforts and cost required to get task done( and if designed well , improve user experience as well)

Chatbots can be divided into two broader picture:

1. Generic Chatbots: also known as virtual assistants, such as Google assistant, Amazon Alexa, Cortana, Siri etc, can be used for calling, typing messages, booking calendar slot, fetching result from web. This system have been trained on massive amount of user’s data , encyclopedias, conversational dialogues and stories with human etc.
2. Domain Specific : This are generally used as per particular domain specific chatbot. Just think about some bank Bot answering information related to corresponding Bank , let us say ICICI bank or a weather bot can only tell weather related information.It cannot book a table in some restaurant or set a morning alarm for you.

Some of the popular paid services such as IBM Watson , Microsoft Azure where you can build up Chatbots using NodeJS, C#, Python and services provided by them. For Example Microsoft Azure have two most important services for building a Bot(QNA Maker and LUIS).

We will try to find out some of challenges when you are working with Microsoft Azure Service like LUIS( Language Understanding Intelligent Service ): A machine learning-based service to build natural language into apps, bots, and IoT devices. Let us discuss them step by step and how to solve the problem:

LUIS.ai:

LUIS.ai is custom machine learning intelligence to a user’s conversational , natural language(NLU) to predict overall meaning and extracting relevant details such as Entity extraction and other important and relevant details

Luis Consider two types of model as listed below. We are using model similar to Prebuilt model

• Prebuilt model include prebuilt entities and can be said without ML/AI specific technique
• Custom model have custom entities where LUIS gives you to identify your own custom intents and entities including ML driven entities or a combination of both

Some of the Challenges due to either of two model are:

To much Intent can confuse the LUIS model

In order to get the same top intent between all the apps, make sure the intent prediction between the first and second intent is wide enough that LUIS is not confused, giving different results between apps for minor variations in utterances.

Entity are optional but highly recommend :

You do not need to create entities for every concept in your app, but only for those required for the app to act. Entity represent parameters or data for an intent: Entities are data you want to pull from the utterance. This can be a name, date, product name, or any group of words. For example, in the utterance “Buy a ticket from New York to London on March 5”, three entities are used. Location, Origin, Location. Destination and Prebuilt datetimeV2 which would contain the value “New York”, “London” and “March 5” respectively.

Entities are shared across intent :

Entities are shared among intents. They don’t belong to any single intent. Intents and entities can be semantically associated but it is not an exclusive relationship. In the utterance “Book  a meeting with Kamal”, “Kamal” is an entity of type Person. By recognizing the entities that are mentioned in the user’s input, LUIS helps you choose the specific actions to take to fulfill an intent.

But what happen sometimes, in case you are using too many Intent in your model, same types of utterances in two entity or Intent consisting of more then one entity , your LUIS Model can make wrong prediction. For example:

Solution for above challenges:

• Use Composite Entity: A composite entity is made up of other entities, that form part of whole.
• Pattern.any : Entity where end of entity is difficult to determine. It is also uses ML capability
• Regular Expression: Use particular pattern from text
• Machine-learned(ML) entities work best when tested via endpoint queries and reviewing endpoint utterances.

As per our current scenario we are facing the wrong intent identification when more than one Entity are used or wrong Intent identification.

1. It’s all about finding a balance between the number of intents, and the number of options or actions within an intent. To try to explain, let’s take an example where a user wants to ask questions about a company and questions about an aggregation Initially the question’s are simple like, “Contact No of Kamal”, “Who is Chetan”. But when you start expanding or allow more option such as “Who is Kamal Naithani and what are his skills set” it may either respond to wrong intent or no answer.
2. We can use Pattern in such cases : Patterns are tools you can use to generalize common utterances, wording, or ordering that signals a particular intent. LUIS first recognizes entities, and then can use matching to identifying patterns within the rest of the utterance. Patterns can be used to improve prediction accuracy of utterances by using entities and their roles to extract data using a specific pattern. This reduces the number of utterance example you would need to provide to teach LUIS about the common utterances for an intent, saving the time it would take to train your LUIS app while improving its’ accuracy.
3. Similarly Phrase lists can be used which provides hints that certain words and phrases are part of a category. If LUIS learns how to recognize one member of the category, it can treat the others similarly. . This improve the accuracy of intent scores and identify entities for words that have the same meaning (synonyms) by adding an interchangeable phrase list

When LUIS makes a prediction, it gets results from all models and picks the top scoring model. Example utterances assigned to one intent act as negative examples for all other intents.

Intents with significantly more positive examples are more likely to receive positive predictions. This is called data imbalance. We have to work on this with proper analysis

Make sure the vocabulary for each intent is used only in that intent.
Intents with overlapping vocabulary can confuse LUIS and cause the scores for the top intents to be very close. This is called Unclear Prediction. We have to identify if there exist case like this

Apply Active Learning to add user utterances.

LUIS is meant to learn quickly with few examples. User utterances will help you find examples with a variety of formats. So add Phrase List and pattern

Use Dispatch services in case you have more number of Domain or Intent to increase the accuracy.

Dispatch Service: If your app is meant to predict a wide variety of user utterances, consider implementing the dispatcher model. If a bot uses multiple LUIS models ,knowledge bases (knowledge bases), you can use Dispatch tool to determine which LUIS model or QnA Maker knowledge base best matches the user input.

1. The parent app indicates top-level categories of questions.
2. Create a child app for each subcategory.
3. The child app breaks up the subcategory into relevant intents. Breaking up a monolithic app allows LUIS to focus detection between intents successfully instead of getting confused between intents across the top level and intents between the top level and sub levels.
4. Schedule a periodic review of endpoint utterances for active learning, such as every two weeks, then retrain and republish.

Batch Testing: You can use batch testing to understand the problem and improve results. Batch testing validates your active trained model to measure its prediction accuracy. A batch test helps you view the accuracy of each intent and entity in your current trained model, displaying results with a chart

1. Review the batch test results to take appropriate action to improve accuracy, such as adding more example utterances to an intent if your app frequently fails to identify the correct intent
2. It generally divide the Utterance/Intent or Utterance/Entity result in 4 Quadrant i.e.. Data points on the False Positive and False Negative sections indicate errors, which should be investigated. If all data points are on the True Positive and True Negative sections, then your app’s accuracy is perfect on this data set.

Some of the best practices while working with LUIS.ai

1. We will also validate the current Luis Model design as per LUIS.ai best practice. Some of them are listed below
2. Intents:Create an intent when this intent would trigger an action.
3. Intents should be specific.
4. If intents are semantically close, consider merging them.
5. Entities:Create when bot needs some parameters or data from the utterance.
6. Use ML related entities
7. Utterances:Begin with 10-15 utterances per intent.
8. Each utterance should be contextually different.
9. The None intent should have between 10-20% of the total utterances.

Please provide your comments and suggestion.

Regards

Team Kite4Sky