Targeting Student Engagement: Identifying Key Chatbot Features and Student Segments for Effective AI-based Interventions
Solo Project
Tools
-
Qualtrics
-
R, Excel
-
mTurk
-
Google Scholar
Date
June 2023 - Dec. 2023
UXR Skill
-
Research Planning
-
Survey Design
-
Data Collection
-
Conjoint survey analysis
-
K-Means Clustering
-
Report generation
Key Results
01
Identified the ideal combination of chatbot features and target student group to improve AI intervention engagement.
02
Delivered 4 workshops on conjoint and segmentation analysis to 250 undergraduate and 55 graduate students.
03
Contributed to a $1.5M funding application by serving as an expert on conjoint survey and segmentation analysis.
RESEARCH GOALS
Chatbot-based growth mindset interventions are on the rise in higher education.
But, 80% of them fail to engage students due to missing the mark on key chatbot features and target audiences.
What are the must-have features for a successful chatbot? Which student groups should be the focus?
Answering these questions is crucial for boosting student engagement, enhancing growth mindsets, and ultimately improving academic well-being.
RESEARCH PROCESS
01
Desktop Research & Literature Review
-
Identified 3 key chatbot features in interventions.
-
Identified 4 student demographic factors influencing intervention engagement.
03
Data Analysis & Synthesis
-
Conducted conjoint and k-means clustering using R.
-
Generated a report with actionable design and implementation strategies for higher ed personnel.
02
Survey Design & Data Collection
-
Designed a survey suitable for conjoint and k-means clustering analyses.
-
Recruited 200 undergraduates via mTurk.
04
Report Dissemination
-
Submitted a research report to a HCI conference.
-
Delivered 2 workshops on data analysis techniques for undergraduates and graduates.
LITERATURE REVIEW
Chatbot-based growth mindset interventions leverage some combinations of the following 3 chatbot features: appearance (how does it look?), modality (how does it communicate with users?), and identity (how does it introduces itself to users?).
Each attribute occurs at some levels. For example, the feature 'identity' has two levels; a chatbot can introduce itself as a therapist or as a peer companion.
Chatbot Features in Growth Mindset Interventions
Appearance
Levels
Machine-like Appearance
Virtual character Appearance
Fully Human
Modality
Levels
Text Only
Voice Only
Voice and Text
Identity
Levels
Therapist
Peer Companion
Also, in general intervention contexts, there are 4 key demographic factors that influence students' engagement: grade point average (GPA), honours program enrolment, academic year, and gender.
Student Demographics Important in Intervention Engagement
GPA
Honors program
Academic year
Gender
I used prior research as a guideline and examined the 3 chatbot features and 4 student demographic factors in a growth mindset intervention.
SURVEY DESIGN
My research questions were best answered with conjoint and segmentation analysis. I considered multiple factors to design a survey suitable for the analysis.
1. Random balanced design
2. Total number of questions shown to participants
3. Number of chatbot profiles shown per question
Participants evaluated 45 chatbot profiles across 15 tasks, with 3 profiles per task. Each profile featured a unique combination of 3 randomly assigned feature levels and a photo matched to the chatbot’s appearance level. The presentation of the photos for each appearance level was randomized (e.g., 'machinelike' appearance randomized the presentation of 6 machinelike photos).
Which chatbot therapist would motivate you to attend and actively engage in a growth mindset intervention?
Physical Appearance
Machinelike
Virtual Character
Human
Identity
Therapist
Peer
Therapist
Modality
Text
Voice & Text
Voice
Next, participants answered demographic questions, essential for segmentation analysis.
Please indicate your information on the following questions:
Are you enrolled in honours program?
GPA
Academic Year
Gender
Most importantly, participants were asked:
Would you subscribe and attend weekly growth mindset intervention sessions if they were available to you?
Segmentation Analysis
The data
One popular method to segmentation is k-means clustering, and this analysis requires the data to be structured differently than the data for conjoint analysis. Here, gender for male is coded 0 and female 1; participants who are in honours were coded 1; participants' GPA ranged from 0 to 4 and their academic year ranged from 1 to 4; importantly, participants who indicated that they would subscribe to weekly intervention sessions were coded 1 and those who did not were coded 0.
The model
One drawback of K-means clustering is that it doesn’t automatically determine the ideal number of groups. To address this, you can estimate multiple models—say with 3, 4, or 5 groups—and then assess which model produces the most distinct and meaningful clusters based on key factors.
Find about the detailed R script for both analyses on Github.
DATA ANALYSIS
Conjoint Analysis
The data
The data for a conjoint survey has one row for each possible chatbot profile (alt) within each set of 15 questions (ques), thus creating 3 × 15 = 45 rows per participants (id). There were 200 participants, with 45 rows each, so there are 200 × 45 = 9,000 rows.
Here, participant 1 chose a text-based chatbot, whose identity is a peer companion and appearance is machinelike, as indicated by choice = 1.
The model
I used multinomial logistic regression {mlogit} to estimate which chatbot features were important predictors influencing participants' choice of a chatbot profile.
The predictions
The {mlogit} model itself does not tell us the optimal combination of chatbot features, so we need to use this function to get at the question!
RESULTS
Top 3 Chatbot Profiles for Optimal Student Engagement
Participants would choose a chatbot with these features 25% of the time.
-
Communicates in text
-
Has machine appearance
-
Is a peer to students
Participants would choose a chatbot with these features 19% of the time.
-
Communicates in text
-
Is a virtual character
-
Is a peer to students
Participants would choose a chatbot with these features 8% of the time.
-
Communicates in text
-
Has human appearance
-
Is a peer to students
Target Student Group for Optimal Student Engagement
Here are the student personas a growth mindset intervention should or should not target. The status of honours program enrolment did not differentiate the participants.
Priority student group
Has an average GPA around 2
Entering sophomore
Plans on going to graduate school
Non-priority Student Groups
Has a poor GPA under 2
Is a freshmen
Does not plan to go to graduate school
Has an average GPA around 2
Is a senior
Does not plan to go to graduate school
RECOMMENDATION
Based on the findings, I recommend these design and implementation strategies that can boost student engagement for a growth mindset intervention.
01
Prioritize chatbots that communicates via text and acts peers to undergraduates
-
Among the top three profiles most effective for student engagement, a common preference emerged: students favor chatbots that communicate via text and act as peers. Therefore, if resources are limited, prioritize these two features, with a focus on giving the chatbot a machine-like appearance.
02
Target the priority student group for optimal engagement
-
Higher education should focus on the priority student group identified in the study. These are the ‘low-hanging fruit’—the students most likely to engage with the intervention as it stands.
-
For non-priority student groups, additional research is needed to understand what would motivate their engagement with the intervention.