Evaluating user needs and learning outcomes from the accessibility training program
Institution
Carleton University
Role
Program Evaluation Lead
Team
Faculty Supervisor and 3 Program Directors
Date
Dec 2020 - June 2022
UXR Skill
-
1:1 Interviews
-
Thematic Analysis
-
Longitudinal Survey
-
Survey Design
-
Quantitative Data Analysis
-
Data Visualization
Tools
-
Excel, NVivo
-
R
-
Zoom, Google Docs
CONTEXT
READi is Canada's first accessibility training program for graduate students
READi has 2 objectives:
-
Objective 1. To equip graduate students with accessibility knowledge.
-
Objective 2. To deliver a satisfactory program experience to graduate students and not-for-profit community partners.
Graduate students and community partners are the main program stakeholders. Community partners are involved in student learning by collaborating with them on a research project called the Action Team Project (ATP).
Graduate Students
Community Partners
Key Results
01
Built an internal reposity housing a protocol for a mixed-methods program evaluation, from R script to interview and longitudinal survey guideline.
02
Increased research impact by 160% through publishing 11 mixed-methods research reports with actional program recommendations
03
Presented the importance of UXR for program evaluation with educators at the internal university conference.
04
Achieved a 100% participant retention rate across an 8-month longitudinal survey study via impactful participant relationship-building.
05
Presented the importance of UXR for program evaluation with educators at the internal university conference.
06
Increased student satisfaction by12% through influencing the program structural change with a support system to address the student pain points.
Fall Semester
Winter Semester
Summer Semester
Fall Semester
Workshops
Elements: field trips, guest lectures, and case studies with accessibility leaders.
Graduate Course
Elements: final research report, monthly progress reports, and meetings with community partners, Ph.D. mentor, and program faculty.
ATP (8 months)
Elements: ATP mid-progress presentation and networking.
Retreat
Elements: ATP final presentation, keynote speaker presentation, and networking.
Symposium
The timeline of READi's 5 training components.
PROBLEM
Is READI doing what it claims to do?
I presented 3 studies to the team to evaluate whether READi was delivering its objectives to the program stakeholders.
Study 1 Secondary Research to investigate the overall impact of READi on graduate students by analyzing internal data.
-
Research Question 1: What is the impact of READi on graduate students’ learning?
Study 2 Longitudinal Survey to investigate the impact of ATP on graduate students.
-
Research Question 2: How does ATP impact graduate students' learning?
Study 3 User Interviews to assess stakeholders' program satisfaction and thoughts on how the program can improve.
-
Research Question 3: How can READi improve to support program stakeholders' needs?
The table shows my proposed research methods for each study. The team was on board to implement 3 studies.
WHICH Research Method?
WHY Chosen Research Method
WHO to Recruit?
Secondary Research
To leverage internal data and understand READi's overall impact.
Graduate students who submitted their program reflections.
Longitudinal Survey
To understand the impact of ATP and pinpoint the exact time the students experience changes in their learning.
Graduate students who are currently enrolled in READi.
User Interviews with
Community Partners
To collect in-depth responses on partners' satisfaction with ATP participation and identify areas of program improvement.
Community partners involved in the ATP currently and in the past
User Interviews with
Graduate Students
To collect in-depth responses on their program satisfaction and identify areas of program improvement.
Graduate students enrolled in READi currently and in the past.
OVERALL PROCESS
I led all processes involved in 3 studies.
Here is an overview of a 2-year program evaluation.
Step 1. Scoping
I conducted a literature review and examined existing data in READi: 1) to fine-tune research questions and 2) to identify mostly used measurements to track students' learning.
Microsoft Word
Google Scholar
Step 3. Conducting: Longitudinal Survey (Study 2)
I tracked graduate students' learning over 8 months using Qualtrics.
Jan
Survey 1
May
Survey 2
Oct
Survey 3
Step 5. Conducting: Stakeholder Interviews (Study 3)
I recruited cohorts and community partners from 2017 to 2020. I interviewed 15 graduate students and 5 community partners.
Zoom
Trint
Step 2. Conducting: Secondary Research
(Study 1)
I analyzed 22 student reflections collected from 2017 to 2020. Secondary research is a great first way to answer some research questions.
Microsoft Word
NVivo 12
Step 4. Analyzing & Synthesizing: Quantitative Data Analysis
I conducted the Skillings Mack test to see changes in students' learning over 8 months.
R Studio
Microsoft Excel
Step 6. Analyzing & Synthesizing: Thematic Analysis
Theme 1.
MAXQDA
Theme 2.
Microsoft Word
Step 7. Program Refinement
I ideated with the team on crucial structural changes that could be implemented.
Original
A bi-weekly meeting to support students.
New
A weekly meeting to support students.
SECONDARY RESEARCH: KEY INSIGHTS
Evaluating the overall impact of READi on student learning
I found 4 themes that highlighted the impact of READi.
Theme 1. Expanded perspective in viewing accessibility
Students viewed accessibility from cultural, social, and legislative perspectives and understood that creating accessible products, services, or environments was a multidisciplinary effort.
"The program helped me to understand working towards accessibility is bringing people from different industries who can come together to contribute to these challenges in different ways.”
-U
Theme 3. Empathy for persons with disabilities
Students showed empathy towards the challenges that persons with disabilities faced and became aware of accessibility barriers in their daily life.
“I looked at how some updated metro stations in Ottawa hadn’t carefully looked at how people with visual disabilities could navigate them. I find myself thinking about case studies like this while I travel around in my day-to-day life.”
-N
Theme 2. Positive attitude for persons with disabilities
Students showed a changed mindset towards disability and focused on the strengths of persons with disabilities.
“Working with people with disabilities for the ATP and seeing how they use computers faster than I could if they had the right technology made me realize how accessibility is about changing mindsets and technology, not changing people.”
-J
Theme 4. Improved professional skills
Students improved their professional skills, including team work, communication, and project management skills.
“I gained interpersonal skills and communication skills from working with the community partners on the ATP. By speaking with persons with disabilities, I gained confidence and understanding of how people can use various abilities to communicate."
-R
LONGITUDINAL SURVEY: RESEARCH SETUP
Evaluating the impact of ATP on student learning
1. Survey Design
This is the design process I follow in creating a survey.
1.
Identify what you want to measure. Use prior work and PsycTests to locate measurements.
2.
Change the wording in the original measurements to make them fit the study context.
3.
Randomize the order of measurements to avoid order effect.
4.
Pilot test the survey and revise items that are confusing and long.
5.
Check whether responses in dataset from a pilot test have been recorded in the way they should.
I had 4 measurements that tapped into graduate students’ cognitive, affective, and behavioural learning. All measurements used a 7-point Likert scale (1 = Strongly disagree to 7 = Strongly agree).
Measurement 1. Cognitive Learning
-
Students indicated their factual understanding of accessibility.
-
Sample items: “I can identify benefits of accessibility" and “I can define the purpose of the Accessible Canada Act.”
Measurement 2. Empathy
-
Students indicated the degree to which they empathized with the specific community their ATP will directly impact.
-
Sample items: "compassionate," “moved” and “soft-hearted” on a sematic scale.
Measurement 3. Career Interests
-
Students indicated the degree to which they were interested in pursuing a career where they could apply accessibility knowledge.
-
Sample items: “I want to explore career options where I can apply accessibility-related knowledge” and “I am not interested in learning what accessibility-related career opportunities are available.”
Measurement 4. Student Engagement
-
Emotional engagement with READi and sample items are “I feel energetic being part of this program” and “I am interested in the material I learn in this program.”
-
Physical engagement with the READi (i.e., the degree to which a student puts effort in working on assignments.), and sample items are “I exert my full efforts towards this program” and “I devote a lot of energy towards this program.”
-
Cognitive engagement with READi and sample items are “my mind is always focused on project discussions” and “I pay a lot of attention to project discussions and tasks.”
2. Participant Recruitment
I recruited 14 graduate students from the 2020-21 cohort (Cohort 1) and the 2021-22 cohort (Cohort 2).
-
Cohort 1 completed the survey in May and October (Mean age = 30.7 years old).
-
Cohort 2 completed the survey in January, May, and October (Mean age = 24.25 years old).
FOOD FOR THOUGHT
Tailor your measurement to the construct of interest. If I were to repeat the study, I would use the cognitive learning measurement that reflects students’ learning from participating in the ATP, and such measurement would better reflect research question 2. The ATP focuses on improving students’ accessibility knowledge about the specific disability community, and the measurement that I used reflects students’ general knowledge about accessibility.
Maintain participants by being empathetic and personable. Across two cohorts, I successfully retained all students over 5 or 8 months. I gave students ample time to complete the survey considering their hectic schedules, gave them continuous study updates, and built personal connections with students by sending them emails and Slack messages.
LONGITUDINAL SURVEY: KEY INSIGHTS
I conducted the Skillings-Mack test to account for a small sample size and non-independence of error.
-
For Cohort 1, there was a marginal significance; students reported higher empathy and were more physically engaged in October than in May (Skillings-Mack Statistic = 3.60, p = 0.06).
-
For Cohort 2, there was no statistical difference between times (Skillings-Mack Statistic = 0.23, p = 0.89).
Figures for cognitive and emotional engagement are not shown and report medians.
Cognitive Learning
7
6
5
4
5.88
5.85
May
October
Career Interests in Accessibility
7
6
5
4
4.60
4.60
May
October
Empathy
* marginal sig.
7
6
5
4
May
October
5.45
5.77
Physical Engagement
7
6
5
4
5.33
6.17
* marginal sig.
May
October
STAKEHOLDER INTERVIEW: RESEARCH SETUP
Understanding stakeholders' satisfaction and pain points
Participant Recruitment
Graduate Students
Age:
21-29 yrs old
Gender:
10 Female
5 male
Cohort Year:
6 from 2017-18
1 from 2018-19
4 from 2019-20
4 from 2020-21
Community Partners
Age:
25-51 yrs old
Gender:
4 Female
1 male
Cohort Year:
1 from 2017-18
2 from 2019-20
2 from 2020-21
STAKEHOLDER INTERVIEWS: KEY INSIGHTS
ALL program stakeholders were satisfied with READi.
I am so happy that I learned about accessibility through READi. I would not have learned about accessibility at this depth otherwise.
.png)
Graduate Students
It is amazing to be part of students' education . Receiving their recommendations to solve our accessibility issues is a bonus.
Community Partners
Graduate students' satisfaction came from the good quality of education that they received. Community partners' satisfaction came from being purposeful by helping students learn about accessibility.
Participants also indicated areas where the program could improve. I describe 2 important themes from each stakeholder.
Theme 1. Support is there, but some students are afraid of reaching out
Graduate students expressed that faculty support was always there. Faculty gave students emotional and instrumental support when they felt uneasy after interacting with critical care patients for the ATP.
But, students wanted the faculty to reach out to them first (vs. the other way around) because they are reluctant to reach out for help for various reasons.
One reason was that students had a close relationship with the faculty and this closeness made them afraid to ask for help.
"I am close to everyone in READi faculty...I don't want to look bad for appearing incompetent in needing help."
-P
Some students also did not reach out for support because they perceived their relationship with the faculty to be formal.
"I felt like I have formal topics to discuss in our meetings with the faculty. ‘It was more like, ‘OK, we have to have something to discuss or show.’ "
Theme 4. Students can respect community partners' personal and organizational values
Partners were frustrated when the graduate student team proposed project ideas that did not align with their organizational and personal values.
They wanted the program to educate the students about respecting their values.
One partner was initially frustrated by the student team’s project findings, which portrayed the organization as if it did not care about accessibility.
"Graduate students need to be very conscious that if their project is supporting that organization and saying that the organization is doing good."
-L
Similarly, another partner said that the student team was going in a direction that portrayed certain user groups of the organization's website in a negative light.
"When the project direction started to get dialectical about opposing this user group to another user group, I panicked."
-T
-Y
I also found 5 themes (click!) describing two program stakeholders' needs from the program and from one another and what they thought to be effective elements in READi.
SUMMARY OF FINDINGS
Across 3 studies, I found evidence that READi was meeting its two objectives. It has increased students’ accessibility knowledge and satisfied the students and community partners. I also identified how the program can improve to increase their satisfaction.
Overall READi Impact
Specific ATP Impact
Affective Learning Outcomes
•Increased interest in accessibility
•Positive attitude towards people with disabilities
•Increased empathy for people with disabilities
Note. The overlap denotes the finding specific to ATP's impact was also found for READi's impact.
Cognitive Learning Outcomes
•Expanded perspective on accessibility by understanding different models of disabilities.
Behavioural Learning Outcomes
•Improved professional skills
•Increased physical engagement
REFLECTION
FUTURE WORK
TAKEAWAY
-
My findings revealed 2 important needs of the community partners. Program changes have not been made to support those needs yet. READi is in the process of transitioning from an independent training program to a degree-equivalent program. Once this status change happens, the team and I will be in conversation on how to refine the program to support the partners' needs.
-
Strategize, strategize, strategize. This was a large-scale project! I was lost and questioned what I should be doing or why I was doing this particular task. But my research plan helped me stay focused and reminded me of my high-level goals for each study. I again saw the value in a detailed research plan in helping me achieve all the goals.
-
Different research methods answer different research questions. I used 3 research methods to get at three research questions. Conducting an interview to track long-term student learning outcomes could be better. Similarly, using a survey to understand stakeholder needs is not appropriate. Through this program evaluation, I once again realized that being a good researcher means developing the capacity to weigh the pros and cons of each research method and assess which method is suitable to answer a particular research question.