User research and AI Chatbot design to elevate student engagement in collaborative environments

Designing an AI-enabled real-time intervention to improve student engagement in an online group discussion environment.

Challenge

With the recent pandemic, online education has become a significant part of the curriculum for students of all age groups and thus exploration of solutions for improving student interactions in an online group discussion is critical for academic and personal development.

The use of collaborative apps surged by 141% as over 1.2 million children worldwide are studying outside the traditional in-person classroom.

45% of elementary school students use digital tools daily for online classes.

Of the schools that offered at least one online course, 81.9% were primary schools, and 64% were middle school students.

71% of professors are concerned about increasing engagement in online classes whereas 31% are concerned with improving student collaboration.

How might we design an AI-enabled real-time intervention that improves student engagement and bridges the gap between online and in-person group discussion experiences?

Solution

We designed an AI-enabled chatbot named, AI Mate, that assists individuals during group discussions by identifying the reason for their lack of participation(motivation, knowledge, confidence, self-reflection) and intervention(Validate participation, provide feedback, suggestion prompts) in the most relevant way. Through user testing, we explored the following research questions:

RQ1: What is the effect of the three types of intervention (validate participation, feedback on participation, suggestion prompts) by AI chatbot on student participation and satisfaction in online group discussion?
 
RQ2: What information on student participation is preferred by the instructor from online group discussions?

Understanding the overarching themes of research

Artificial Intelligence in Education (AIED)
AIED aims to use a combination of AI, learning theory, and educational practice to improve learning outcomes. Most research focuses on the personalization of learning for individuals and lacks focus on how AI technology can enhance student-student collaborative learning and compensation for the adverse effects of the absence of peers.

Improvement in social interaction skills
The impact of real-time feedback considerably improved performance since it allowed participants to reflect while taking part in the conversations. However, traditional feedback approaches to expedite behavior change can make one feel uncomfortable having their group status evident to peers for fear evaluated or targeted based on societal expectations.

Chatbots in group discussions
Multi-party-based chatbots are virtual agents that communicate with multiple users in a group. Researchers on multi-party-based chatbots have regarded chatbots as tools that support group interaction while including them as group members by focusing on their social roles. 

Comparing existing solutions

Most AI-based adaptive products in the market do not focus on enhancing or supporting an online discussion environment, and none of them include chatbots. As concluded from our literature study, users prefer chatbots over voice assistants in an online learning environment because the latter can be distracting.

The presence of AI in the social development and education of children was clear in the initial market analysis. However, it lacks crucial learning aspects such as interactive and individualized feedback, as well as a lack of connection between instructor and students, which they believe are barriers to a successful online learning experience. There is a potential void that conversational agents may cover by giving interactive and tailored feedback and keeping students engaged throughout the online discussion process.

Understanding User Needs

We interviewed five children with their parents, and we comprehended the children’s perspectives and sentiments regarding their group discussion experiences. We questioned parents for their thoughts on their child’s social interactions, as well as the strategies they use to address any difficulties that arise and their views on privacy and safety concerning the applications that children use.

Difficulties in online group discussions: The major challenge for kids was speaking alone among a group of students who were hesitant to speak. During conversations, it would be important to stimulate reserved or unmotivated students without pressuring them.

Desirable feedback: Instead of extensive feedback, most parents said a summary of performance would be sufficient as reporting a student’s participation rate might be hyper-competitive.

Privacy concerns: The chief concern about privacy was that the lessons were recorded and saved as data. It would be helpful if AI could inform teachers or parents that it is observing or monitoring their interactions and not using the data in explicit form.

Redefining needs and goals

User Needs
Inadequate information, motivation, confidence, nervousness, and a lack of self-reflection cause a lack of student engagement in an online group discussion.

Goals
To investigate the impact of AI-enabled real-time intervention and feedback on team members by providing immediate cues to participants with limited intermediation on group dynamics and individual behavior in an online group discussion.

The solution- AI Mate

Persona

A narrative and persona will help propel chatbots’ character, conversations, and actions. According to research, personality plays a significant role in how users view and communicate with chatbots. The chatbot will have an identity and will embody the personality and characteristics in its phrasing and tonality.

Role and characteristics

The fundamental question was what characteristics our AI will inherit in order to enhance the trust and experience of the students. We incorporated the following characteristics into our chatbot based on the defined interventions to provide the most natural and trustful experience:

• Sentiment and Personality
• Anthropomorphism
• Humanness
• Gender

Scenarios and Interventions

Based on the preliminary research and inspection of various pain points identified through the user interviews, three types of interventions that are perceived primarily to enable individual engagement are defined in our solution. The three interventions are:

Validate participation by encouraging contributing participants and reaching out for others’ opinions. AI Mate will empathize with students and encourage them to share their opinion with the group with continued participation.

Feedback on participation is given by indicating their relative level of participation in the group discussion based on time spoken and turns taken where self-reflection will be a key component.

Suggestion prompts will provide specific and general prompts to help initiate the conversation. The AI takes the role of the instructor, who assists lagging individuals through careful observation and expertise.

Implementation

AI Mate chatbot for students

The chatbot will exchange direct messages with each individual present in the group discussion through the integrated chat feature of a particular group discussion platform. We created sample chats for each scenario, which will convey the flow the users will experience. The purpose of the conversations would be to suggest prompts, motivate, and provide self-reflection.

Mate dashboard for teachers

The dashboard’s information will aid instructors or caregivers in getting a better understanding of student engagement and AI Mate’s role in it.

Although, as with any new technology, there is no optimal approach to developing metrics to evaluate bot performance, it is critical to identify KPIs and ways to analyze them, both statistically and qualitatively. Even though there is no comparable data for bots, some crucial standard metrics still hold true and apply here as well.

Study Design

Screening & Recruitment

Due to logistic constraints on recruitment, we conducted our evaluation with undergraduate students. Additionally, we also interviewed a middle school instructor to get an expert evaluation of the AI Mate concept and dashboard evaluation.

Method- Wizard of OZ

We introduced the AI Mate in one of the two sessions using the Wizard of Oz user testing technique, where one researcher embodied the persona and interventions defined for the system and carried on conversations with the three participants. After each session, participants answer the questionnaires aimed at understanding motivation, satisfaction, contribution, metacognition, and knowledge perception of themself during the discussion.

After the two sessions, participants answered a post-test questionnaire to understand the usefulness and perception of the AI Mate by participants, including satisfaction, usability, and human factors. Finally, a quick one-to-one debriefing was conducted with individuals to ask any follow-up questions. 

To keep the context for both discussions as similar as possible, 

1. The two group discussions were conducted on common topics with a similar format [talking about advantages and disadvantages] to keep the flow of the conversations similar.
2. The order of with and without AI discussions was reversed in one of the group
3. With-AI study for both the groups had the same topic

Zoom video conferencing platform was used for the evaluation, where AI Mate joins as a user and communicates with the participants through direct messaging. Sample conversation dialogues were created for each intervention that portrays the personality of the AI Mate.

Analysis and Result

Quantitative: We used descriptive statistics, and a paired t-test to compare with AI and without AI discussion sessions. Further, the time spoken and turns taken by each participant in a group discussion were considered to interpret overall individual engagement.

Qualitative: The analysis was conducted on data obtained through open-ended questions, video recording observations, and debriefing. The interview with the instructor majorly focused on understanding their mental model regarding group discussions and AI integrations, understandability and use cases of dashboard features, and their effect on teaching.

Improved Engagement in conversations

• On average, individual engagement measured using time spoken and turns taken increased for each participant when the AI Mate was involved in the conversation.
• 25% increase in speaking time on average for an individual.
• The time spoken improved for all the participants, while the turns taken did not have a visible difference.

Feedback on AI Mate personality (post-survey)

• Most participants agree AI Mate was “reliable and truthful” “similar to human response” and “realistic and engaging”. Some participants also mentioned the AI Mate as obtrusive.
• Most participants agree AI Mate enhances their motivation and helps to understand how they are doing. However, some participants did not agree that AI Mate gave them appropriate knowledge during the group discussion.

Participant suggestions for AI Mate

“I only receive 2 messages from it, so I think I did not have a lot of interaction with the AI chatbot like I expected. It would be better if I can receive more messages, or interact with it more.”

“Sometimes the comments from the AI were a bit late. The discussion had already moved on to the next topic”

User Testing with a Teacher

At last, we conducted an hour-long 1:1 interview with one middle school teacher (female in her 20s, currently teaching Earth Science at a middle school since 2019) on Zoom. The purpose of this study was to receive feedback on our analytical dashboard, which aids instructors or caregivers in getting a better understanding of student engagement and AI Mate’s role in it. Major feedback from her can be summarized:

Modified Dashboard

Achievements

Motivation is a major component of AI-Mate’s influence on the participants

AI Mate’s motivated prompts encouraged participants to contribute better and provide higher satisfaction ratings. With trust as the foundation for validation and feedback, these interventions can help improve individual engagement in online group discussions. Further, participants who consider themselves active contributors in group discussions showed a similar level of engagement in both discussions (with and without AI). However, they showed higher satisfaction ratings in the discussion where AI Mate was involved.

Suggestion prompts were found to be the most useful

Timely and relevant suggestion prompts to participants were perceived to be the most useful intervention by the AI Mate. Part of the participants who interpret themselves as passive speakers in group discussions indicated their persistent struggle with getting involved in conversations. They asserted such prompts would be helpful for them to easily get involved, reducing the barriers.

Sentiment Analysis and individual engagement within a group

The primary attributes of interest that would help the instructor analyze and keep track of students are identified to be understanding the emotional state and needs of students and understanding their progress over time. Providing this information through a secure medium and allowing the analysis of group dynamics for individual students would provide instructors with valuable insights into how they can improve their study methodologies.