Communication Design for an Educational AI Chatbot : Analyzing Cipherbot's Communication Style and Challenges

Abstract

We analyzed the communication patterns of Cipherbot, an educational AI chatbot that addresses validation and transparency problems in AI-student interaction, with 44 undergraduate business students using the system. Findings show that Cipherbot delivers information grounded in learning materials, primarily focusing on statements of fact, with a 99% occurrence rate. Also, about a third (31%) of Cipherbot's messages to students contain an example, which adheres to pedagogical best practices. However, more than a third (36%) of Cipherbot's communication with students also contained traces of opinion, particularly normative statements about how companies or individuals should behave. Cipherbot also demonstrates some level of social etiquette, using thank yous and apologies (7%), but it rarely engages in requests for information, or clarifications and comprehension checks–as these categories might be useful for student engagement, future exploration into diversifying Cipherbot's communication style to support learning is required. Student feedback suggests that usability issues in educational AI chatbots comprise both communicational and technical issues, e.g., incompleteness and redundancy–iterative testing of prompts with student feedback could address many of these challenges. Based on our findings, we make six propositions about students’ interaction with educational AI chatbots.