top of page
Targeting Student Engagement: Identifying Key Chatbot Features and Student Segments for Effective AI-based Interventions
Key Results

Solo Project

Tools

  • Qualtrics

  • R, Excel

  • mTurk

  • Google Scholar

Date

June 2023 - Dec. 2023

UXR Skill

  • Research Planning

  • Survey Design

  • Data Collection

  • Conjoint survey analysis

  • K-Means Clustering

  • Report generation

Key Results

01

Identified the ideal combination of chatbot features and target student group to improve AI intervention engagement.

02

Delivered 4 workshops on conjoint and segmentation analysis to 250 undergraduate and 55 graduate students. 

03

Contributed to a $1.5M funding application by serving as an expert on conjoint survey and segmentation analysis. 

RESEARCH GOALS

Chatbot-based growth mindset interventions are on the rise in higher education.

But, 80% of them fail to engage students due to missing the mark on key chatbot features and target audiences.

What are the must-have features for a successful chatbot? Which student groups should be the focus?

 

Answering these questions is crucial for boosting student engagement, enhancing growth mindsets, and ultimately improving academic well-being. 

Research Process

RESEARCH PROCESS

01

Desktop Research & Literature Review

  • Identified 3 key chatbot features in interventions.

  • Identified 4 student demographic factors influencing intervention engagement.

03

Data Analysis & Synthesis

  • Conducted conjoint and k-means clustering using R.

  • Generated a report with actionable design and implementation strategies for higher ed personnel.

02

Survey Design & Data Collection

  • Designed a survey suitable for conjoint and k-means clustering analyses.

  • Recruited 200 undergraduates via mTurk.

04

Report Dissemination

  • Submitted a research report to a HCI conference.

  • Delivered 2 workshops on data analysis techniques for undergraduates and graduates. 

LITERATURE REVIEW

Chatbot-based growth mindset interventions leverage some combinations of the following 3 chatbot features: appearance (how does it look?), modality (how does it communicate with users?), and identity (how does it introduces itself to users?).

 

Each attribute occurs at some levels. For example, the feature 'identity' has two levels; a chatbot can introduce itself as a therapist or as a peer companion. 

Chatbot Features in Growth Mindset Interventions

Appearance

Levels

Machine-like Appearance

Virtual character Appearance

Fully Human

Modality

Levels

Text Only

Voice Only

Voice and Text

Identity

Levels

Therapist

Peer Companion

Also, in general intervention contexts, there are 4 key demographic factors that influence students' engagement: grade point average (GPA), honours program enrolment, academic year, and gender

Student Demographics Important in Intervention Engagement

GPA

Honors program

Academic year

Gender

pngtree-vector-black-line-glowing-idea-bulb-clipart-design-png-image_6352152.png

I used prior research as a guideline and examined the 3 chatbot features and 4 student demographic factors in a growth mindset intervention. 

Literature Review
Survey Design

SURVEY DESIGN

My research questions were best answered with conjoint and segmentation analysis. I considered multiple factors to design a survey suitable for the analysis. 

1. Random balanced design

2. Total number of questions shown to participants 

3. Number of chatbot profiles shown per question

Participants evaluated 45 chatbot profiles across 15 tasks, with 3 profiles per task. Each profile featured a unique combination of 3 randomly assigned feature levels and a photo matched to the chatbot’s appearance level. The presentation of the photos for each appearance level was randomized (e.g., 'machinelike' appearance randomized the presentation of 6 machinelike photos).

Which chatbot therapist would motivate you to attend and actively engage in a growth mindset intervention?

Physical Appearance

Machinelike

Virtual Character

Human

Identity

Therapist

Peer

Therapist

Modality

Text

Voice & Text

Voice

Next, participants answered demographic questions, essential for segmentation analysis.

Please indicate your information on the following questions: 

Are you enrolled in honours program?

GPA

Academic Year

Gender

Most importantly, participants were asked: 

Would you subscribe and attend weekly growth mindset intervention sessions if they were available to you?

Data Analysis
Segmentation Analysis

The data

One popular method to segmentation is k-means clustering, and this analysis requires the data to be structured differently than the data for conjoint analysis. Here, gender for male is coded 0 and female 1; participants who are in honours were coded 1; participants' GPA ranged from 0 to 4 and their academic year ranged from 1 to 4; importantly, participants who indicated that they would subscribe to weekly intervention sessions were coded 1 and those who did not were coded 0. 

Screenshot 2024-08-27 at 10.30.46 PM.png

The model

One drawback of K-means clustering is that it doesn’t automatically determine the ideal number of groups. To address this, you can estimate multiple models—say with 3, 4, or 5 groups—and then assess which model produces the most distinct and meaningful clusters based on key factors.

Screenshot 2024-08-27 at 10.28.48 PM.png

Find about the detailed R script for both analyses on Github

DATA ANALYSIS

Conjoint Analysis

The data

Screenshot 2024-08-30 at 2.23.47 PM.png

The data for a conjoint survey has one row for each possible chatbot profile (alt) within each set of 15 questions (ques), thus creating 3 × 15 = 45 rows per participants (id). There were 200 participants, with 45 rows each, so there are 200 × 45 = 9,000 rows.

 

Here, participant 1 chose a text-based chatbot, whose identity is a peer companion and appearance is machinelike, as indicated by choice = 1.  

The model

Screenshot 2024-08-27 at 10.15.57 PM.png

I used multinomial logistic regression {mlogit} to estimate which chatbot features were important predictors influencing participants' choice of a chatbot profile.  

The predictions

Screenshot 2024-08-27 at 10.17.55 PM.png

The {mlogit} model itself does not tell us the optimal combination of chatbot features, so we need to use this function to get at the question!

Results

RESULTS

Top 3 Chatbot Profiles for Optimal Student  Engagement

Participants would choose a chatbot with these features 25% of the time.

  • Communicates in text

  • Has machine appearance

  • Is a peer to students

Participants would choose a chatbot with these features 19% of the time.

  • Communicates in text

  • Is a virtual character

  • Is a peer to students

Participants would choose a chatbot with these features 8% of the time.

  • Communicates in text

  • Has human appearance

  • Is a peer to students

Target Student Group for Optimal Student Engagement

Here are the student personas a growth mindset intervention should or should not target. The status of honours program enrolment did not differentiate the participants.

Priority student group

Has an average GPA around 2

Entering sophomore

Plans on going to graduate school

Non-priority Student Groups

emoji3_edited.jpg
37c1989408e7741490e6da359380f3ef.png

Has a poor GPA under 2

Is a freshmen

Does not plan to go to graduate school

cq78l32dwsjtmda9g14lrqrgcv2d_edited.png

Has an average GPA around 2

Is a senior

Does not plan to go to graduate school

Recommendation

RECOMMENDATION

Based on the findings, I recommend these design and implementation strategies that can boost student engagement for a growth mindset intervention. 

01

Prioritize chatbots that communicates via text and acts peers to undergraduates

  • Among the top three profiles most effective for student engagement, a common preference emerged: students favor chatbots that communicate via text and act as peers. Therefore, if resources are limited, prioritize these two features, with a focus on giving the chatbot a machine-like appearance.

02

Target the priority student group for optimal engagement

  • Higher education should focus on the priority student group identified in the study. These are the ‘low-hanging fruit’—the students most likely to engage with the intervention as it stands.

  • For non-priority student groups, additional research is needed to understand what would motivate their engagement with the intervention.

bottom of page