As part of the equivalent of a qualifying exam at the MIT Media Lab, I had to do a study on multi-turn interactions in conversational agents. I focused on what was at the moment (this was in 2020) a state of the art agent which was the Domino's pizza ordering assistant using the Google assistant. This assistant was built with conversationality in mind. I ran a user study to identify how the agent's workflow was and any issues that were present. I found the following issues:
•Problem with contextual edits
•“Redo” keyword clears the order
•No undo
•“Clear order” to exit doesn’t work, it doesn’t even clear the order
•Timeouts/misunderstandings are repeats
•Great exiting/better than Google's default exiting
•Help is a global help that is not very useful
To address these, I built a conversational agent in DialogFlow, which had the following improvements (particularly following conversational patterns):
•Starts with an Open Request pattern for adding items to an order rather than waterfall of slot filling
•Repairs in much later turns for editing items
•Undo functionality for errors operations
•Extended telling pattern for menu, ingredients
•Simplified menu with only 4 primary items
•Recently placed order unlockable only if you have previous orders
Below are some audio examples of the Domino's agent vs. my agent.
Original Domino's Agent
Dom 2.0 - Improved Conversational Domino's Agent
To test this Dominos agent, I ran a user study (remotely over the phone) in which participants would interact (in a random order) with the Dominos agent and with my agent. I ran 7 participants through the study, with the following questions:
•Will the inclusion of certain conversational patterns (repair, extended telling) make the pizza ordering experience more pleasant?
•Will the inclusion of certain conversational patterns (repair, extended telling) make the pizza ordering experience more conversational?
•Will decreasing the verbosity of certain prompts make the experience more conversational?
•Will including contextual edits/questions make the experience more pleasant and more conversational?
•Will adding variations into the prompts make the experience more conversational?
I got the following results; overall the speed at which people ordered improved, people felt it was more conversational. All of this because of the use of conversational patterns
Below is a full analysis of the agent and the work that was done
Lastly below are the conversational flows for both interactions
Dom 2.0 Agent
