Evaluating Accessibility Program to Understand Program Stakeholders' Learning Outcomes and Needs

Team

Evaluation lead (me)
Four faculty from information technology, industrial design, and systems and computer engineering

Skills

Secondary research
User interviews
Thematic analysis
Longitudinal survey
Quantitative data analysis
Survey design

Timeframe

Dec 2021–Mar 2023
This project consisted of a 2-year longitudinal survey study

CONTEXT

READi is Canada's first accessibility training program for graduate students

It is a 1-year program funded by the Natural Sciences and Engineering Research Council (NSERC), Canada's major federal agency. The program director, program faculty, and program committee direct the program.

READi has 2 objectives:

Objective 1. To equip graduate students of various backgrounds (e.g., HCI, computer science) with accessibility knowledge.
Objective 2. To involve not-for-profit community partners (hereafter, community partners) in graduate students’ learning and build good partnerships with community partners.

Given these objectives, there are 2 main program stakeholders.

Graduate Students

Community Partners

READi offers 5 training components to meet the objectives.

Component 1. Graduate Course on Accessibility and Inclusive Design.

Component 2. Action Team Project (ATP). This is the flagship component and project-based learning. Graduate students collaborate with community partners to solve partners’ accessibility issues in their organizations, and they research to understand the issues involving the specific community (e.g., people with disability, seniors).

Components 3-5. Retreat, Symposium, & Workshops. Graduate students present their progress on the ATP at the Retreat and present their final ATP at the Symposium. They learn about diverse accessibility topics at Workshops.

Fall Semester

Winter Semester

Summer Semester

Fall Semester

Workshops

Elements: field trips, guest lectures, and case studies with accessibility leaders.

Graduate Course

Elements: final research report, monthly progress reports, and meetings with community partners, Ph.D. mentor, and program faculty.

ATP (8 months)

Elements: ATP mid-progress presentation and networking.

Retreat

Elements: ATP final presentation, keynote speaker presentation, and networking.

Symposium

The timeline of READi's 5 training components.

PROBLEM

Is READI doing what it claims to do?

In the past, the program director and faculty had informal conversations with a few graduate students, and they broadly knew the program was meeting its objectives. However, they wanted a data-driven program evaluation, and I took up this exciting task.

Based on what I knew from the program objectives, I developed two higher-level research questions:

initial

Research Question 1: What is the impact of READi on graduate students’ learning?
Research Question 2: How can READi improve to support program stakeholders’ needs?

READi had a lot of internal data collected over 5 years:

Behavioural data: publication records of graduate students, student attendance to workshops
Student submissions that described what they did for the ATP
Students’ reflections on what they learned from participating in each program component

After assessing the data, I modified the original research questions and presented 3 studies to the team.

Study 1 Secondary Research to investigate the overall impact of READi on graduate students by analyzing students' reflections. It was best to analyze the reflections; existing behavioural data demonstrated students' non-accessibility achievement, and other student submissions merely described students' tasks.

revised

Research Question 1: What is the impact of READi on graduate students’ learning?

Study 2 Longitudinal Survey to investigate the impact of ATP on graduate students. The study had two benefits: 1) emphasizing ATP's (potentially positive) impact in the program's annual progress report to the federal agency could bring extra funding, and 2) we can pinpoint the exact time in which graduate students showed improvement or decrement in their learning.

revised

Research Question 2: How does ATP impact graduate students' learning?

Study 3 User Interviews to assess community partners' satisfaction from participating in the ATP and their thoughts on how program elements can improve to support their needs. I also proposed conducting user interviews with graduate students to collect in-depth responses on their satisfaction with the program and their thoughts on improving it.

Research Question 3: How can READi improve to support program stakeholders' needs?

The table shows my proposed research methods for each study. The team was on board to implement 3 studies.

WHICH Research Method?

WHY Chosen Research Method

WHO to Recruit?

Secondary Research

To leverage internal data and understand READi's overall impact.

Graduate students who submitted their reflections.

Longitudinal Survey

To understand the impact of ATP and pinpoint the exact time the students experience changes in their learning.

Graduate students who are currently enrolled in READi.

User Interviews with

Community Partners

To collect in-depth responses on partners' satisfaction with ATP participation and identify areas of program improvement.

Community partners involved in the ATP currently and in the past

User Interviews with

Graduate Students

To collect in-depth responses on their program satisfaction and identify areas of program improvement.

Graduate students enrolled in READi currently and in the past.

OVERALL PROCESS

I led all processes involved in 3 studies.

Here is an overview of a 2-year long program evaluation that I independently led, from planning and implementing each study to data analysis to final reporting.

Step 1. Scoping

I conducted a literature review and examined existing data in READi: 1) to fine-tune research questions and 2) to identify mostly used measurements to track students' learning.

Microsoft Word

Google Scholar

Step 3. Conducting: Longitudinal Survey (Study 2)

I tracked graduate students' learning over 8 months using Qualtrics.

Jan

Survey 1

May

Survey 2

Oct

Survey 3

Step 5. Conducting: Stakeholder Interviews (Study 3)

I recruited cohorts and community partners from 2017 to 2020. I interviewed 15 graduate students and 5 community partners.

Zoom

Trint

Step 2. Conducting: Secondary Research

(Study 1)

I analyzed 22 student reflections collected from 2017 to 2020. Secondary research is a great first way to answer some research questions.

Microsoft Word

NVivo 12

Step 4. Analyzing & Synthesizing: Quantitative Data Analysis

I conducted the Skillings Mack test to see changes in students' learning over 8 months.

R Studio

Microsoft Excel

Step 6. Analyzing & Synthesizing: Thematic Analysis

Theme 1.

MAXQDA

Theme 2.

Microsoft Word

Step 6. Program Refinement

I ideated with the team on crucial structural changes that could be implemented.

Original

A bi-weekly meeting to support students.

New

A weekly meeting to support students.

SECONDARY RESEARCH: KEY INSIGHTS

Evaluating the overall impact of READi on student learning

Graduate students posted reflections on what they learned from each program component on the university’s e-portfolio. I thoroughly evaluated the quality of each student's reflection and excluded the one that had poor quality (e.g., one sentence).

I had 22 reflections of good quality, conducted a thematic analysis, and found 4 themes that highlighted the impact of READi.

Theme 1. Expanded perspective in viewing accessibility

Students viewed accessibility from cultural, social, and legislative perspectives and understood that creating accessible products, services, or environments was a multidisciplinary effort.

"The program helped me to understand working towards accessibility is bringing people from different industries who can come together to contribute to these challenges in different ways.”

-U

Theme 3. Empathy for persons with disabilities

Students showed empathy towards the challenges that persons with disabilities faced and became aware of accessibility barriers in their daily life.

“I looked at how some updated metro stations in Ottawa hadn’t carefully looked at how people with visual disabilities could navigate them. I find myself thinking about case studies like this while I travel around in my day-to-day life.”

-N

Theme 2. Positive attitude for persons with disabilities

Students showed a changed mindset towards disability and focused on the strengths of persons with disabilities.

“Working with people with disabilities for the ATP and seeing how they use computers faster than I could if they had the right technology made me realize how accessibility is about changing mindsets and technology, not changing people.”

-J

Theme 4. Improved professional skills

Students improved their professional skills, including team work, communication, and project management skills.

“I gained interpersonal skills and communication skills from working with the community partners on the ATP. By speaking with persons with disabilities, I gained confidence and understanding of how people can use various abilities to communicate."

-R

LONGITUDINAL SURVEY: RESEARCH SETUP

Evaluating the impact of ATP on student learning

1. Survey Design

This is the design process I follow in creating a survey.

1.

Identify what you want to measure. Use prior work and PsycTests to locate measurements.

2.

Change the wording in the original measurements to make them fit the study context.

3.

Randomize the order of measurements to avoid order effect.

4.

Pilot test the survey and revise items that are confusing and long.

5.

Check whether responses in dataset from a pilot test have been recorded in the way they should.

I had 4 measurements that tapped into graduate students’ cognitive, affective, and behavioural learning. I chose these measurements because they are mostly studied learning outcomes for university students.

All measurements used a 7-point Likert scale (1 = Strongly disagree to 7 = Strongly agree).

Measurement 1. Cognitive Learning

Students indicated their factual understanding of accessibility.
Sample items: “I can identify benefits of accessibility" and “I can define the purpose of the Accessible Canada Act.”

Measurement 2. Empathy

Students indicated the degree to which they empathized with the specific community their ATP will directly impact.
Sample items: "compassionate," “moved” and “soft-hearted” on a sematic scale.

Measurement 3. Career Interests

Students indicated the degree to which they were interested in pursuing a career where they could apply accessibility knowledge.
Sample items: “I want to explore career options where I can apply accessibility-related knowledge” and “I am not interested in learning what accessibility-related career opportunities are available.”

Measurement 4. Student Engagement

Emotional engagement with READi and sample items are “I feel energetic being part of this program” and “I am interested in the material I learn in this program.”
Physical engagement with the READi (i.e., the degree to which a student puts effort in working on assignments.), and sample items are “I exert my full efforts towards this program” and “I devote a lot of energy towards this program.”
Cognitive engagement with READi and sample items are “my mind is always focused on project discussions” and “I pay a lot of attention to project discussions and tasks.”

2. Participant Recruitment

I recruited 14 graduate students from the 2020-21 cohort (cohort 1) and the 2021-22 cohort (cohort 2).

Cohort 1 completed the survey in May and October vs. cohort 2 completed the survey in January, May, and October.
10 participants from cohort 1 (Mean age = 30.7 years old) vs. 4 participants from cohort 2 (Mean age = 24.25 years old).
I gave participants a 2-week time frame to complete the survey.

FOOD FOR THOUGHT

Tailor your measurement to the construct of interest. If I were to repeat the study, I would use the cognitive learning measurement that reflects students’ learning from participating in the ATP, and such measurement would better reflect research question 2. The ATP focuses on improving students’ accessibility knowledge about the specific disability community, and the measurement that I used reflects students’ general knowledge about accessibility.

Maintain participants by being empathetic and personable. Across two cohorts, I successfully retained all students over 5 or 8 months. I gave students ample time to complete the survey considering their hectic schedules, gave them continuous study updates, and built personal connections with students by sending them emails and Slack messages.

LONGITUDINAL SURVEY: KEY INSIGHTS

I conducted the Skillings-Mack test to account for a small sample size and non-independence of error.

For cohort 1, there was a marginal significance; students reported higher empathy and were more physically engaged in October than in May (Skillings-Mack Statistic = 3.60, p = 0.06).
For cohort 2, there was no statistical difference between times (Skillings-Mack Statistic = 0.23, p = 0.89).

Figures for cognitive and emotional engagement are not shown and report medians.

Cognitive Learning

5.88

5.85

May

October

Career Interests in Accessibility

4.60

May

October

Empathy

* marginal sig.

May

October

5.45

5.77

Physical Engagement

5.33

6.17

* marginal sig.

May

October

STAKEHOLDER INTERVIEW: RESEARCH SETUP

Understanding stakeholders' satisfaction and needs

Participant Recruitment

I recruited graduate students and community partners who participated in the program over a year ago. I also recruited past graduate students and community partners who were still involved in the program in some ways so that they could contextualize their responses based on what they know of the program’s current elements, which have been slightly modified over the years.

Graduate Students

Age:

21-29 yrs old

Gender:

10 Female

5 male

Cohort Year:

6 from 2017-18

1 from 2018-19

4 from 2019-20

4 from 2020-21

Community Partners

Age:

25-51 yrs old

Gender:

4 Female

1 male

Cohort Year:

1 from 2017-18

2 from 2019-20

2 from 2020-21

Click for Interview Questions

STAKEHOLDER INTERVIEWS: KEY INSIGHTS

100% of all program stakeholders were satisfied with READi.

I am so happy that I learned about accessibility through READi. I would not have learned about accessibility at this depth otherwise.

Graduate Students

It is amazing to be part of students' education . Receiving their recommendations to solve our accessibility issues is a bonus.

Community Partners

Graduate students' satisfaction came from the good quality of education that they received. Community partners' satisfaction came from being purposeful by helping students learn about accessibility.

Participants also indicated areas where the program could improve. I describe 2 important themes from each stakeholder.

Theme 1. Support is there, but some students are afraid of reaching out

Graduate students expressed that faculty support was always there. Faculty gave students emotional and instrumental support when they felt uneasy after interacting with critical care patients for the ATP.

But, students wanted the faculty to reach out to them first (vs. the other way around) because they are reluctant to reach out for help for various reasons.

One reason was that students had a close relationship with the faculty and this closeness made them afraid to ask for help.

"I am close to everyone in READi faculty...I don't want to look bad for appearing incompetent in needing help."

-P

Some students also did not reach out for support because they perceived their relationship with the faculty to be formal.

"I felt like I have formal topics to discuss in our meetings with the faculty. ‘It was more like, ‘OK, we have to have something to discuss or show.’ "

Theme 4. Students can respect community partners' personal and organizational values

Partners were frustrated when the graduate student team proposed project ideas that did not align with their organizational and personal values.

They wanted the program to educate the students about respecting their values.

One partner was initially frustrated by the student team’s project findings, which portrayed the organization as if it did not care about accessibility.

"Graduate students need to be very conscious that if their project is supporting that organization and saying that the organization is doing good."

-L

Similarly, another partner said that the student team was going in a direction that portrayed certain user groups of the organization's website in a negative light.

"When the project direction started to get dialectical about opposing this user group to another user group, I panicked."

-T

-Y

I also found 5 themes (click to see them!) describing two program stakeholders' needs from the program and from one another and what they thought to be effective elements in READi.

SUMMARY OF FINDINGS

Making sense of all results across 3 studies, READi and ATP positively impacted students’ cognition, affect, and behaviours.

Overall READi Impact

Specific ATP Impact

Affective Learning Outcomes

•Increased interest in accessibility

•Positive attitude towards people with disabilities

•Increased empathy for people with disabilities

Note. The overlap denotes the finding specific to ATP's impact was also found for READi's impact.

Cognitive Learning Outcomes

•Expanded perspective on accessibility by understanding different models of disabilities.

Behavioural Learning Outcomes

•Improved professional skills

•Increased physical engagement

While program stakeholders were overall satisfied with the program, they also shared how it can improve, and some of their needs were targeted at each other.

OUTCOMES & LESSONS

A change was made to implement a weekly meeting with a senior Ph.D. student.

IMPACT

TAKEAWAY

The program has changed graduate students' bi-weekly meeting with a senior Ph.D. mentor to a weekly meeting. The mentor completed the program one year before the current cohort and they support the cohort's ATP. The idea is to make graduate students feel comfortable asking for support from a non-authoritative figure.

Built awareness in the program personnel of the needs of community partners and graduate students.

Built an internal research repository for the faculty, so that in future they can evaluate the program again or other similar training programs that they are involved in.

Opened new research collaborations with the faculty at the University of Toronto as the result of my presentation of the program evaluation research report at a Human-Computer Interaction conference.

Strategize, strategize, strategize. This was a large-scale project! I was lost and questioned what I should be doing or why I was doing this particular task. But my research plan helped me stay focused and reminded me of my high-level goals for each study. I again saw the value in a detailed research plan in helping me achieve all the goals.

Different research methods answer different research questions. I used 3 research methods to get at three research questions. Conducting an interview to track long-term student learning outcomes could be better. Similarly, using a survey to understand stakeholder needs is not appropriate. Through this program evaluation, I once again realized that being a good researcher means developing the capacity to weigh the pros and cons of each research method and assess which method is suitable to answer a particular research question.

Three academic publications to computer engineering education conferences.

Jin Kang, Adrian D. C. Chan, Chantal M. J. Trudel, Boris Vukovic, and Audrey Girouard. 2021. A Longitudinal Evaluation of the Impact of a Graduate Student Accessibility Training on Student Learning Outcomes. In Western Canadian Conference on Computing Education (WCCCE'23).

Jin Kang, Adrian D. C. Chan, Chantal M. J. Trudel, Boris Vukovic, and Audrey Girouard. 2021. Diversifying Accessibility Education: Presenting and Evaluating an Interdisciplinary Accessibility Training Program. In 21st Koli Calling International Conference on Computing Education Research (Koli Calling ’21).

Jin Kang, Chantal M. J. Trudel, Audrey Girouard, and Adrian D. C. Chan. 2021. Research and Education in Accessibility, Design, and Innovation (READi) Training Program: Preparing Graduate Students for Careers in Accessibility Research and Design. In Proceedings of 3rd Annual EduCHI’21 Symposium.

NEXT STEP

Continuing Redesign

My findings revealed 2 important needs of the community partners. Program changes have not been made to support those needs yet. READi is in the process of transitioning from an independent training program to a degree-equivalent program. Once this status change happens, the team and I will be in conversation on how to refine the program to support the partners' needs.