Our Experiment to See How Well ChatGPT Might Work if Incorporated into Our Customer Engagement & Communication Automation Tool.
ChatGPT’s Generative AI
OpenAI’s ChatGPT has generated a tremendous amount of interest, opinions and expectations about its potential use cases.
The recently published ChatGPT is a chatbot model, based on GPT-3, designed to carry on conversations and dialogues that appear natural. (Such technologies are also known as “Generative AI”). GPT-3 is an AI model trained on 45TB of web text producing human-like responses when given a prompt. While GPT-3 was out of beta in July 2020, followed by GPT3.5 in Jan 2022, ChatGPT is still in the experimental stage and is not yet available for commercial use.
While we already use GPT3 in our platform, we view ChatGPT’s potential applicability for further automating customer engagement and customer service for SMBs of particular interest.
The Increasing Importance of AI to SMBs
Inspired by 24/7 patient service challenges at my wife’s dental practice, we founded and developed the Agentz automated assistant, a customer engagement & communication automation tool (ECAT), to offer AI powered technology that could automate many front desk functions of small to medium sized businesses. Our design goals were very targeted:
- It should provide a very simple out-of-the-box experience for many common industries
- It should enhance the consumer experience, not detract from it
- It should work with the communication channels consumers prefer – phone, text, social, and web
- It should be unbelievably affordable for SMBs
Since starting Agentz four years ago, technologies have continued to evolve and consumer acceptance of automated assistants has increased. In fact, 74% of consumers now prefer to use automation over live conversations.
As consumers express satisfaction and even preference for using automated help, SMBs have been more willing to adopt those technologies.
NLP/NLU vs. Generative AI
Traditionally, Natural Language Processing & Understanding (NLP & NLU) technologies like RASA, Google Dialogflow, and Microsoft Luis have been used to build automated conversational assistants for common customer support use cases. The primary goal of these technologies is to understand the intent of a consumer inquiry and help guide them to the right answer or resolution. For these technologies to be successful, they require a structured knowledgebase of information about the business and their industry.
Generative AI like ChatGPT differs from this, structurally, in that it is supposed to help generate a response or offer a resolution based on training data, reducing or even eliminating the need for a curated knowledgebase.
How Might ChatGPT Benefit the Agentz Automated Assistant?
While we are currently using OpenAI for some functionality in our product, along with other NLP/NLU technologies, we decided to run deeper experiments using ChatGPT to see how and where we might use it to further improve the Agentz AI.
The goals of our experiment were to learn if using ChatGPT could:
- Enable SMBs to fully rely on AI-generated answers
- Increase the accuracy of understanding a consumer inquiry and providing the right answer or resolution
- Improve the customer experience
- Reduce costs associated with AI training
- Simplify SMB onboarding
Running Our ChatGPT Experiment
We took a sample of actual consumer queries, across various industries, from our platform and ran them through ChaptGPT/GPT-3 to see how it would respond. Would ChatGPT respond to consumer questions as well or better than our curated knowledgebase?
Overall, we were impressed with some of the results but found some aspects lacking. Here are our findings:
1. Relying 100% on AI-generated answers has negative aspects
- ChatGPT can provide great answers to general industry-related questions such as “What is a root canal?” or “Why should I consider a root canal vs. tooth extraction?”
- ChatGPT is not able to answer business-specific questions such as “I have a problem with my insurance bill” or “I need a root canal and need to know the cost and options”
- ChatGPT lacks consistency in its answers offering vastly different responses to minor question variations. This means that the same question and intent, but asked slightly differently, can provide a variety of answers giving consumers varying experiences
Verdict: There is a high risk of negative consumer experience if relying 100% on ChatGPT generated answers without intent-based curation.
2. ChatGPT can not provide action-based responses
- ChatGPT seems to do a good job understanding the intent of a consumer query
- The model requires additional curated data and technology to provide a good action-based response. ChatGPT alone could not handle the inquiry “Do you have any appointments available tomorrow”
Verdict: ChatGPT is not equipped to hold a conversation that knows when to offer an action-based response that would result in a new customer lead capture or customer service resolution
3. ChatGPT is more expensive to train than other traditional NLP technologies
- OpenAI makes it easier to train the model using APIs and allows the use of their core Generative AI technology
- The time it took to train business-specific data was longer, increasing the operational cost to do so. In fact, we experienced a four-fold increase in training time and a significant increase in cost
- Training costs are incurred every time business data is updated so platforms built for SMBs and using OpenAI APIs may require different pricing models, resulting in more expensive solutions
Verdict: We found the cost of training the AI to be significantly higher than with other traditional NLP technologies. For enterprise clients this may not be an issue, but for SMBs that rely on more affordable technologies, this could rule out the use of ChatGPT. We also predict that the cost to use ChatGPT may increase further as it becomes commercially available.
4. ChatGPT does/does not improve the customer experience
- ChatGPT provides a great experience when generating detailed answers to informational, industry-related queries
- ChatGPT can not enhance the customer experience for business specific customer support inquiries by providing a resolution
Verdict: The success of automated assistants depends largely on customer experience. Favorable experiences include design, usability, as well as helpfulness of the responses that AI gives to specific questions. For ChatGPT, this was a mixed bag.
5. OpenAI’s large language model can improve the SMB onboarding experience
- OpenAI is useful in generating industry specific data that can be used for pre-training models and can help improve the onboarding experience
- The automatically generated data still requires augmentation, confirmation & curation to make sure there are no negative effects
Verdict: ChatGPT has the ability to shorten the onboarding experience for the small business owner, giving them a better out-of-box experience but needs human oversight.
The Final Verdict
The text creation capability of ChatGPT-like generative AI technologies is admittedly very impressive. We can see the potential for technologies such as ChatGPT to enhance the Agentz automated assistant when it comes to certain types of question and answer creation. However, that is just one piece of an overall conversational automation solution.
We think of ChatGPT as the engine that goes into a car on the assembly line. The engine is great for generating some text based responses, but the car needs many more parts working together seamlessly so that consumers will enjoy driving it from point A to point B.
Those other parts are seamless coverage of multiple communication channels, channel specific customer experiences, pre-training and curation of data specific to a business, elegant guidance to the desired outcome, access to live humans, and integration with other systems – all highly important parts that make sure that automation is well designed and provides an exceptional customer experience.
Where Do We Go From Here?
At Agentz, we LOVE the idea of simply feeding raw information into a large language model like ChatGPT and not having to be concerned with creating our own structured knowledgebase of curated training data to answer customer inquiries.
While ChatGPT can’t handle business specific inquiries out-of-the-box with full accuracy, we will be implementing a hybrid process to minimize the need for knowledge base creation from scratch and use various models depending on the type of customer inquiry.
We continue to remain agile in our adoption of new generative AI technologies that can support automating customer engagement and communication for SMBs.
About Agentz
Agentz patent-pending technology is designed and built from scratch for SMBs with a focus on simplicity.
Agentz harnesses the power of AI without compromising the consumer experience with a human touch. Agentz technology learns over time and gets smarter. The technology is tested by millions of conversations and many happy customers.
Learn more about Agentz or contact us today.