Building an Enterprise Chatbot - Work with Protected Enterprise Data Using Open Source Frameworks

Table of Contents

About the Authors

About the Technical Reviewer

Acknowledgments

Introduction

Chapter 1: Processes in the Banking and Insurance Industries

Banking and Insurance Industries

A Customer-Centric Approach in Financial Services

Benefits from Chatbots for a Business

Chatbots in the Insurance Industry

Automated Underwriting

Instant Quotations

AI-Based Personalized Experience

Simplification of the Insurance Buying Process

Registering a Claim

Finding an Advisor

Answering General Queries

Policy Status

Instant Notifications

New Policy or Plan Suggestions

Conversational Chatbot Landscape

Summary

Chapter 2: Identifying the Sources of Data

Chatbot Conversations

General Conversations

Specific Conversations

Training Chatbots for Conversations

Self-Generated Data

Customer Interactions

Phone

Emails

Chat

Social Media

Customer Self-Service

Mobile

Customer Service Experts

Open Source Data

Crowdsourcing

Personal Data in Chatbots

Introduction to the General Data Protection Regulation (GDPR)

Data Protected Under the GDPR

Data Protection Stakeholders

Customer Rights Under the GDPR

Chatbot Compliance to GDPR

Summary

Chapter 3: Chatbot Development Essentials

Customer Service-Centric Chatbots

Business Context

Policy Compliance

Security, Authentication, and Authorization

Accuracy of User Input Translation to Systems

Chatbot Development Approaches

Rules-Based Approach

Advantages of the Menu-Based Approach

Disadvantages of the Menu-Based Approach

AI-Based Approach

Advantages of the AI-Based Approach

Disadvantages of the AI-Based Approach

Conversational Flow

Key Terms in Chatbots

Utterance

Intent

Entity

Channel

Human Takeover

Use Case: 24x7 Insurance Agent

Summary

Chapter 4: Building a Chatbot Solution

Business Considerations

Chatbots vs. Apps

Growth of Messenger Applications

Direct Contact vs. Chat

Business Benefits of Chatbots

Cost Savings

Customer Experience

Success Metrics

Customer Satisfaction Index

Completion Rate

Bounce Rate

Managing Risks in Chatbots Service

Third-Party Channels

Impersonation

Personal Information

Confirmation Check

Generic Solution Architecture for Private Chatbots

Workflow Description

Key Features

Technology Stack

Maintenance

Summary

Chapter 5: Natural Language Processing, Understanding, and Generation

Chatbot Architecture

Popular Open Source NLP and NLU Tools

NLTK

spaCy

CoreNLP

gensim

TextBlob

fastText

Natural Language Processing

Processing Textual Data

Reading the CSV File

Sampling

Tokenization Using NLTK

Word Search Using Regex

Word Search Using the Exact Word

NLTK

Normalization Using NLTK

Noun Phrase Chunking Using Regular Expressions

Named Entity Recognition

spaCy

POS Tagging

Dependency Parsing

Dependency Tree

Chunking

Named Entity Recognition

Pattern-Based Search

Searching for Entity

Training a Custom NLP Model

CoreNLP

Tokenizing

Part-of-Speech Tagging

Named Entity Recognition

Constituency Parsing

Dependency Parsing

TextBlob

POS Tags and Noun Phrase

Spelling Correction

Machine Translation

Multilingual Text Processing

TextBlob for Translation

POS and Dependency Relations

Named Entity Recognition

Noun Phrases

Natural Language Understanding

Sentiment Analysis

Polarity

Subjectivity

Language Models

Word2Vec

Neural Network Architecture

Using the Word2Vec Pretrained Model

Performing Out-of-the-Box Tasks Using a Pretrained Model

Word Pair Similarity

Sentence Similarity

Arithmetic Operations

Odd Word Out

fastText Word Representation Model

Information Extraction Using OpenIE

Topic Modeling Using Latent Dirichlet Allocation

Collection of Documents

Loading Libraries and Defining Stopwords

Removing Common Words and Tokenizing

Removing Words That Appear Infrequently

Saving the Training Data as a Dictionary

Generating the Bag of Words

Training the Model Using LDA

Natural Language Generation

Markov Chain-Based Headline Generator

Loading the Library

Loading the File and Printing the Headlines

Building a Text Model Using Markovify

Generating Random Headlines

SimpleNLG

Loading the Library

Tense

Negation

Interrogative

Complements

Modifiers

Prepositional Phrases

Coordinated Clauses

Subordinate Clauses

Main Method

Printing the Output

Deep Learning Model for Text Generation

Loading the Library

Defining the Training Data

Data Preparation

Creating an RNN Architecture Using a LSTM Network

Defining the Generate Text Method

Training the RNN Model

Generating Text

Applications

Topic Modeling Using spaCy, NLTK, and gensim Libraries

Tokenizing and Cleaning the Text

Lemmatization

Preprocessing the Text Method for LDA

Reading the Training Data

Bag of Words

Training and Saving the Model

Predictions

Gender Identification

Loading the NLTK Library and Downloading the Names Corpus

Loading the Male and Female Names

Common Names

Extract Features

Randomly Splitting into Train and Test

Training the Model

Model Prediction

Model Accuracy

Most Informative Features

Document Classification

Loading Libraries

Reading the Dataset into the Categorized Corpus

Computing Word Frequency

Checking the Presence of Frequent Words

Training the Model

Most Informative Features

Intent Classification and Question Answering

Intent Classification

Setting tensorflow as the Back End

Building the Model

Classifying the Intent

Question Answering

Building the Model

Context and Question

Serving the DeepPavlov Model

Summary

Chapter 6: A Novel In-House Implementation of a Chatbot Framework

Introduction to IRIS

Intents, Slots, and Matchers

Intent Class

IntentMatcherService Class

The getIntent Method of the IntentMatcherService class

Intent Classification Service

General Query Intent

Matched Intent Class

Slot Class

IRIS Memory

Long- and Short-Term Sessions

Long-Term Attributes

Short-Term Attributes

The Session Class

Dialogues as Finite State Machines

State

Shields

Transition

State Machine

Building a Custom Chatbot for an Insurance Use Case

Creating the Intents

CustomNumericSlot

BooleanLiteralSlot

AccTypeSlot

IPinSlot

AlphaNumericSlot

IrisConfiguration

Adding States

Shields

DontHaveAccTypeShield

DontHaveQuoteDetailsShield

HaveAccTypeShield

HaveClaimIdShield

HaveQuoteDetailShield

Adding Execute Methods

Exit State

FindAdvisorState

GetAccountBalanceState

GetAccTypeState

GetClaimIdState

AskForQuote State

GetQuote State

Start State

GeneralQuery State

Adding State Transitions

Managing State

Exposing a REST Service

ConversationRequest

ConversationResponse

ConversationService

ConversationController

Adding a Service Endpoint

Summary

Chapter 7: Introduction to Microsoft Bot, RASA, and Google Dialogflow

Microsoft Bot Framework

Introduction to QnA Maker

Introduction to LUIS

Introduction to RASA

RASA Core

RASA NLU

Introduction to Dialogflow

Summary

Chapter 8: Chatbot Integration Mechanism

Integration with Third-Party APIs

Market Trends

Stock Prices

Weather Information

Connecting to an Enterprise Data Store

Integration Module

Demonstration of AskIris Chatbot in Facebook Messenger

Account Balance

Claim Status

Weather Today

Frequently Asked Questions

Context Switch Example

Summary

Chapter 9: Deployment and a Continuous Improvement Framework

Deployment to the Cloud

As a Stand-Alone Spring Boot JAR on AWS EC2

As a Docker Container on AWS EC2

As an ECS Service

Smart IRIS Alexa Skill Creation in Less Than 5 Minutes

Continuous Improvement Framework

Intent Confirmation (Double-Check)

Predict Next Intent

A Human in the Loop

Summary

Index