Conversational AI with Rasa: Entities

Summary

In this video from the Conversational AI with Rasa series, Yusta explains the importance of entities in enhancing your assistant's understanding and response accuracy. The video covers different methods for entity extraction, including pre-built models like Duckling and SpaCy, regex patterns, and machine learning. It also delves into advanced features like synonyms, lookup tables, and entity roles and groups, providing practical insights on how to implement and utilize these techniques in your Rasa assistant for improved performance.

Takeaways

• 🧠 Entities are critical for AI assistants to better understand user inputs and respond appropriately, often extracted from user messages.
• 📝 Common entities include numbers, dates, country names, and other relevant information that can be contextualized in responses or actions.
• 📚 The training data for entity extraction is stored in the 'nlu.yaml' file, with entities labeled using a simple square bracket and label convention.
• 🚀 Rasa offers pre-built models like Duckling and Spacey for extracting entities without extensive labeled data, useful for dates, numbers, and more.
• 🔍 Regex is a method in Rasa for extracting entities that follow a deterministic pattern, such as user IDs, enhancing the assistant's ability to identify specific details.
• 🤖 Machine learning is employed for custom entities that don't have pre-built models or don't follow a regex pattern, requiring more training data for better results.
• 📊 Rasa's DIET classifier is a powerful machine learning model for entity extraction, part of the NLU model pipeline configurations.
• 📈 The output of entity extraction by Rasa is in JSON format, providing detailed information about the entity, its value, location in the sentence, and the extraction method used.
• 🔄 Synonyms in Rasa allow normalization of entity values, useful for mapping different user terms to a single value for consistent data usage.
• 📋 Lookup tables in Rasa are used to enhance extraction for entities with known possible values, like country names, by generating case-sensitive regex patterns.
• 🎯 Entity roles and groups in Rasa add additional context to entities, helping distinguish between different types of information, such as origin and destination in flight booking.
• 📘 Entity roles can influence the conversation flow and are configured in training data stories and referenced in the domain file for more natural and context-aware responses.

Q & A

What are entities in the context of Rasa?

Entities are pieces of information that your assistant can extract from user inputs to better understand what is being asked and use those details in a specific context.

What are some common examples of entities?

Common examples of entities include numbers, dates, country names, and any kind of relevant information, such as destinations in a flight booking assistant.

Where should the training data for entity extraction be stored in Rasa?

The training data for entity extraction should be stored inside the nlu.yaml file.

How should entities be labeled in the training data?

Entities should be surrounded by square brackets and the label should be included inside parentheses next to the word.

What are the different methods for entity extraction in Rasa?

The different methods for entity extraction in Rasa include using pre-built models, regex, and machine learning.

What is Duckling and how is it used in Rasa?

Duckling is a tool for extracting entities like numbers, dates, URLs, email addresses, etc. It requires no training data for your assistant to extract these details.

What is SpaCy and how does it enhance entity extraction in Rasa?

SpaCy is a powerful library that enables the use of pre-built models to enhance entity extraction for details like person names and locations.

How does regex help in entity extraction in Rasa?

Regex allows defining a specific pattern that the entities should follow, making it suitable for extracting entities that follow a deterministic pattern, such as user IDs.

When should machine learning be used for entity extraction in Rasa?

Machine learning should be used for extracting custom entities that don't have pre-built models or don't follow specific patterns.

What is the Rasa DIET classifier?

The Rasa DIET classifier is a powerful machine learning model used for entity extraction that requires a lot of good quality training data to achieve the best results.

What additional features does Rasa offer for enhancing entity extraction?

Rasa offers features such as synonyms, lookup tables, and entity roles and groups to enhance entity extraction.

How do synonyms work in Rasa?

Synonyms allow mapping extracted entity values to a different value to normalize the data, which is useful when users refer to the same thing using different terms.

What are lookup tables in Rasa and how are they used?

Lookup tables are lists of words used to generate case-sensitive regex patterns, enhancing entity extraction for details with a set of known possible values.

What are entity roles and groups in Rasa?

Entity roles and groups allow assigning additional details to entities, such as distinguishing between origin and destination in a flight booking assistant.

Outlines

00:00

🔍 Understanding Entities in Rasa AI

This paragraph introduces the concept of entities in the context of conversational AI with Rasa. Entities are vital pieces of information extracted from user inputs that assist the AI in comprehending the query and responding appropriately. The paragraph explains that entities can range from simple details like numbers and dates to more complex and context-specific information. It also touches on the training data requirements for entity extraction, which should be formatted within the 'nlu.yaml' file using a specific labeling convention. The paragraph further discusses various methods for entity extraction in Rasa, including the use of pre-built models like Duckling for common entities and Spacey for names and locations, regex for deterministic patterns, and machine learning for custom entities. The importance of quality training data for machine learning models is emphasized, along with the inner workings and considerations when building these models.

05:03

🛠️ Enhancing Entity Extraction with Rasa Features

The second paragraph delves into additional features of Rasa that can enhance entity extraction. It begins by discussing synonyms, which help in mapping different user expressions to a single normalized value, facilitating consistent data usage across the system. The paragraph explains how synonyms can be added either in the 'nlu.yaml' file or inline with training examples. It also introduces lookup tables, which are used to create regular expression patterns for known sets of values, such as country names. Furthermore, the paragraph covers entity roles and groups, which allow for the differentiation of entities within a context, such as distinguishing between an origin and a destination in a flight booking scenario. The importance of providing varied examples for roles and groups is highlighted to enable the AI to learn effectively. The paragraph concludes by noting that entity roles can influence conversation flow and must be integrated into training data stories and the domain file.

Keywords

💡Entities

Entities in the context of conversational AI refer to the specific pieces of information that an assistant extracts from user inputs to better understand the request and respond appropriately. They are crucial for context understanding and action execution. In the script, entities such as 'destination' are extracted from user statements like 'book a ticket to Sydney', where 'Sydney' is identified as an entity of the type 'destination'.

💡Conversational AI

Conversational AI, also known as chatbots, leverages natural language processing to simulate human-like conversations with users. The video script discusses how entities play a vital role in enhancing the functionality of these AI systems. For example, the assistant's ability to understand and act upon a user's request to book a flight is facilitated by the extraction of entities like 'Sydney'.

💡Rasa

Rasa is an open-source framework used for building conversational AI. It is mentioned in the script as the platform that developers use to implement entity extraction and other NLU (Natural Language Understanding) functionalities. The script explains various methods of entity extraction available in Rasa, such as using pre-built models, regex, and machine learning.

💡NLU (Natural Language Understanding)

NLU is a subfield of AI that focuses on enabling computers to understand human language. In the video, NLU is integral to the process of entity extraction, where the assistant interprets user inputs to identify and utilize entities. The script describes how training data for entity extraction should be formatted within the 'nlu.yaml' file in Rasa.

💡Duckling

Duckling is a tool integrated within the Rasa framework that specializes in extracting entities such as numbers, dates, URLs, and email addresses without the need for extensive training data. The script highlights Duckling as a powerful pre-built model that simplifies the process of entity extraction for common data types.

💡Regex

Regex, short for regular expressions, is a sequence of characters that defines a search pattern. In the context of the video, Regex is used to extract entities that follow a deterministic pattern, such as user IDs. The script explains how to define and include Regex patterns in the 'nlu.yaml' file for custom entity extraction.

💡Machine Learning

Machine learning is an AI technique that allows computers to learn from and make decisions based on data. The script discusses using machine learning for extracting custom entities that do not have pre-built models or do not follow a specific pattern. It emphasizes the need for substantial, high-quality training data to achieve accurate results.

💡DIET Classifier

The DIET (Distant Supervision for Entity Recognition) Classifier is a machine learning model within Rasa that is particularly powerful for entity extraction. The script mentions it as one of the models that can be trained on custom data to extract entities that do not fit standard patterns or have no pre-built models available.

💡Synonyms

Synonyms in the context of conversational AI allow the system to map different user expressions to a single, normalized value. This is important for consistency when querying databases or making API calls. The script describes how synonyms can be added in the 'nlu.yaml' file or inline with training examples to ensure entity values are standardized.

💡Lookup Tables

Lookup tables are lists of words used to create regular expression patterns for entity extraction. They are particularly useful for extracting entities with a known set of possible values, such as country names. The script explains how to include a lookup table in the 'nlu.yaml' file to enhance entity extraction for specific types of entities.

💡Entity Roles and Groups

Entity roles and groups are features that allow developers to add context to entities, helping the assistant understand the relationship between different entities in a user's request. For instance, in a flight booking scenario, the script mentions distinguishing between 'origin' and 'destination' by using roles. Groups can also be used to categorize entities, such as grouping topics related to a specific order in a pizza booking assistant.

Highlights

Entities are vital for assistants to understand user inputs and respond contextually.

Entities can be any relevant piece of information, such as numbers, dates, or country names.

Entity extraction training data should be stored in the nlu.yaml file.

Entities are labeled in training data by surrounding the word with square brackets and adding the label in parentheses.

Rasa offers pre-built models like Duckling for extracting numbers, dates, URLs, and email addresses without extensive training data.

Spacey is another library for enhancing entity extraction with pre-built models for names and locations.

To use pre-built models, reference them in the NLU model configuration file.

Regex is useful for extracting entities that follow a deterministic pattern, like user IDs.

Machine learning is employed for custom entities without pre-built models or specific patterns.

Rasa's diet classifier is a powerful model for entity extraction using machine learning.

High-quality training data is crucial for better machine learning model results.

Rasa's output for entity extraction is in JSON format, detailing the entity, value, location, and extraction method.

Synonyms help normalize extracted entity values for consistent database querying or API calls.

Synonyms can be added via the nlu.yaml file or inline with NLU training examples.

Lookup tables enhance extraction for entities with known possible values, like country names.

Entity roles and groups allow distinguishing between different types of entities, such as origins and destinations in flight booking.

Entity roles can influence the conversation flow and must be included in training data stories and the domain file.

Entities are essential for assistants to comprehend user requests and utilize them contextually in later interactions.

Transcripts

[Music]

hi everyone and welcome back to the

conversational ai with rasa series my

name is yusta and i'm extremely excited

to continue your developer education

journey with you

in this video we will talk about

entities

entities are the pieces of information

that your assistant can extract from

user inputs those details can help your

assistant to better understand what is

being asked and later on use those

details in a specific context when

responding back to the user or running

specific actions

the most common examples of entities are

details like numbers dates country names

and similar but an entity can really be

any kind of piece of information that is

relevant and important to your assistant

for example for a flight booking

assistant it would be really useful to

know which detail in users input is a

destination

that's why in this example where the

user says that they would like to book a

ticket to sydney

sydney is being extract as an entity of

a type destination

now a quick note on how the training

data for entity extraction should look

like and where should it be stored

the training data for entity extraction

should be stored inside of your nlu.yaml

file now how this data should be labeled

it's a very simple convention

the word that should be extracted as an

entity should be surrounded by square

brackets and then next to it you should

include the label to this entity inside

of the parentheses

there are a few ways of how entities can

be extracted using rasa

different methods work best for specific

entity types but the most common

approach that we see being used by

developers is using a mix of different

methods to achieve the best results

let's talk about those methods one by

one

the first approach is using pre-built

models

rasa enables you to use pre-built models

to enhance the entity extraction results

without requiring you to have loads of

labeled data for example

a tool called duckling is an extremely

powerful approach for extracting

entities like numbers dates urls email

addresses and similar

the best part about duckling is that it

requires no training data for your

assistant to extract those details

another really powerful library spacey

enables you to use pre-built models to

enhance the entity extraction for

details like person names locations and

similar to enable your assa assistant to

use pre-built models all you have to do

is reference those models in your nlu

model configuration file we will talk

about those details in the later

episodes of the series

another approach for extracting entities

with fasa is using regex

regex allows you to define a specific

pattern that the entities you would like

to extract should follow

it means that regex is the best approach

for extracting entities that follow a

specific deterministic pattern for

example user ids and similar details

to enable your rasa assistant to use

regex you will have to define the regex

pattern and include it in your nlu.yaml

file in addition to a few training

examples for that specific entity

don't forget to name the entity you

would like your assistant to extract the

same way as you named your regex pattern

and last but not least the third

approach for extracting entities with

rasa is using machine learning

if some of the entities that you would

like your assistant to extract don't

have pre-built models or if they don't

follow a specific pattern so using regex

is not really an option

you should use machine learning to

extract those

entities machine learning models are

very powerful for extracting what we

call custom entities

to extract entities using machine

learning models you will need some

training data in fact you will need a

lot more training data than for previous

approaches

but the more good quality training data

you have the better results you will

achieve

raza comes with a few machine learning

models that you can use and train on

your own training data

for example one of the most powerful

models for entity extraction is rasa's

diet classifier when extracting entities

using machine learning models there are

lots of things that are happening under

the hood and there are quite a few

things you have to take into account

when building those models

we will talk about this topic a lot more

in the later episodes of the series when

we will dive deeper into the nlu model

pipeline configurations

when entities are extracted under the

hood rasa produces an output in a json

format

that output is rich in detail about what

kind of entity has been extracted what

is the value of that entity and where in

a sentence that detail was found and

also what method or a model has been

used for extracting this detail so for

example here you can see an entity city

being extract with a value new york city

and the method that has been used for

extracting this entity is diet

classifier

in addition to just extracting entities

rasa comes with a few additional

features that can help you enhance the

entity extraction even more let's talk

about them

the first one is called synonyms

synonyms allow you to map extracted

entity value to a value different than

the one extracted

now where this is useful in some cases

users will refer to the same thing using

lots of different terms

but you as a developer you will want to

use the values of the entities for let's

say querying the database or making api

calls or using those details for

something else

this means that you will need the

extracted entity values to be normalized

and may be mapped under one specific

value and this exactly what you can

achieve with synonyms

there are two ways of how synonyms can

be added to your rasa assistant

one of them is by adding a new section

to your nlu.yaml file called synonym you

have to define the actual value that

extracted values will have to be mapped

to and then you have to provide examples

of how users might refer to that

specific synonym

the second approach of adding synonyms

to your aza assistant is by adding them

in line with your nlu training examples

all you have to do is add another

parameter called value which will

reference the value that extracted

entities will have to be mapped to

a very important note about synonyms

synonym mapping happens after entities

are extracted which means that you will

need some training data to enable your

assistant to extract entities first

another very powerful feature of rasa is

lookup tables

lookup tables are lists of words that

can be used to generate case sensitive

regular expression patterns

with lookup tables you can enhance

entity extraction for details that have

a set of known possible values

for example you can use lookup table to

enhance the entity extraction for

country names to achieve that include

the list of all countries in the world

in your nlu.yaml file under the section

lookup

and the last very powerful feature of

rasa for entity extraction

is entity roles and groups

if you are building an assistant for

flight booking or something similar you

will quickly realize that you will want

to add additional details for specific

entities for example

if the user says that they would like to

book a ticket from new york to boston

in a very simple scenario your assistant

will extract new york and boston as an

entity location but for your assistant

to be natural in how it responds and in

general to be able to run necessary

actions to book a ticket your assistant

will have to distinguish which detail is

the origin which one is the destination

you can enable your assistant to do that

by defining entity roles

to do that you will have to include

additional parameter called role to your

entity labeled data and define which

role a specific entity should correspond

to

entity groups allow you to group

specific entities under a specific

category

so for example in a pizza booking

assistant you can enable your assistant

to group specific topics corresponding

to a specific order

an important thing about entity roles

and groups is that you have to include

quite a few different examples for your

assistant to really learn so make sure

to include different examples of

different variations of roles and groups

entity roles can also be configured to

influence the flow of the conversation

to do that you will have to include

roles in your training data stories just

like you see in the example here and

also reference them in your domain file

entities are extremely important pieces

of information that can enable your

assistant to better understand what is

being asked and enable your assistant to

use them later on in a specific context

we will dive even deeper into the topic

of collecting the information and using

it in a specific context in the later

episodes of the conversational ai with

rasa series i hope i'll see you there