Instructors, students, and staff can access the chatbot by clicking on the Help icon in the main left navigation in Canvas and selecting the link to Chat with Canvas Support Chatbot.
My name is Patricia Goldweic and I am a software engineer with the Teaching and Learning Technologies group (TLT) in Northwestern IT. I recently completed work on an AI chatbot that is now available to all Canvas users which provides just-in-time answers to some of the common questions asked of the Canvas support staff. It was a fascinating project and I learned a few things along the way that I think you’ll find interesting.
First, Some Basic Terminology
A chatbot is piece of software that interacts with users using natural language, enabling simple conversations with a limited amount of task-related content.
A chatbot can only effectively help someone if it correctly recognizes the user’s intentions. All conversational AI (artificial intelligence) tools available today require an intent model—a kind of language model—for recognizing the user’s goal (intent) based on his/her input. These tools use machine learning to classify users’ utterances into a set of ‘buckets’ representing the different intents that are part of this model. An action model—a ‘dialog’ in IBM Watson Assistant—then guides the bot through a conversation in the selection of appropriate responses. Finally, a software application supports the user interface and orchestrates the bot’s actions while enlisting other services.
Lessons Learned
This project began in the latter part of 2017 as our software development group was experimenting with the new IBM Watson cognitive services. Along with two TLT content experts—Erin Green and Jonathan Diehl—we started by carefully analyzing different sources of data, such as service records from both Northwestern and Canvas, as well as existing IT Knowledge Base and Canvas support articles. After around a year of data analysis and preparation, expert data categorization, model building and initial application development, we began testing in the winter of 2018. That initial test was crucial as it highlighted the need for a search fallback feature and some dynamic model improvement, both of which became important building blocks of the current release.
Some demo videos of the Canvas Support Chatbot in action can be viewed below:
Aside from learning a lot about AI conversational technology and associated tools for building the software applications controlling chatbots, I learned some valuable lessons applicable to other bot-building projects.
On coming up with an initial intent model
Automated extraction of data from existing records was a worthy goal. And while it provided us a first pass without too much content-expert participation, we quickly learned that we would still require manual intervention to make sense of all the data. Also, in many cases, the language needed to be simplified to provide greater clarity by the content experts.
On maintaining the intent model over time
As new intents and language for them are added to the model, care needs to be taken to ensure that the existing model continues to perform as expected (e.g. users sometimes employ simple words that do not help the bot infer the user’s correct intent). This suggests that testing needs be incorporated into the maintenance workflow at regular intervals.
On setting up our expectations properly
We learned to abandon some of our initial expectations with respect to:
- The type of language that users will employ when talking to the bot. It is important that the bot can explicitly request what it needs. In some cases, the bot will insist on follow-up questions before moving on.
- Whether users would be compelled to reply to the Chatbot’s questions. We originally expected users would do so; however, we quickly learned that users tend to abandon a conversation the more they are required to type. This drove us to add simple buttons to the Chatbot’s user interface so that users could quickly reply to some of the bot’s questions without too much effort on their part.
- Whether users will approach the Canvas Chatbot solely to ask Canvas-related questions. We found that users come to the bot with questions that are not always Canvas-related. We decided to tackle a few explicitly within the bot, but mostly depend on the escalation feature for the rest.
Final Thoughts and Roadmap
We plan to augment a bit the Canvas Support Chatbot’s responsibilities as time goes on by allowing it to take some simple actions in the Canvas domain (via programmatic access to Canvas). This coupled with the bot’s self-improvement feature should improve coverage of our most-frequent questions over time. We hope to see usage of the bot grow so that it becomes an integral part of the Canvas support network and paves the way for this technology to transfer to other areas of Northwestern IT.