Evaluating Accessibility Program to Understand Program Stakeholders' Needs.

Team

Lead investigator (me)
Four senior faculty

Skills

Secondary research
Stakeholder interviews
Thematic analysis
Longitudinal survey
Quantitative data analysis
Survey design

Timeframe

Dec 2021 - Mar 2023*

*The project took 2 years due to a longitudinal survey study.

Deliverables

Program recommendations and refinement
Academic publications

PROBLEM

Are program stakeholders' needs being met?

READi is a graduate training program designed to enhance students’ accessibility competencies. It has 5 program components and the community partner research project is the flagship component. In these projects, students in the team collaborate with local community partners to research partners’ accessibility issues. Each year, READi trains a cohort of graduate students and invites new community partners.

One year training

1.

Community Partner Research Project

2.

Workshops

3.

Retreat

4.

Symposium

READi's 5 program components.

5.

Graduate Course on Inclusive Design

There are two main program stakeholders: 1) a graduate student cohort and 2) community partners. Prior to my involvement, no program evaluation had been done to understand students’ learning outcomes and each stakeholder’s needs. I put forth the following research questions:

Research Question 1 (RQ1): What are short-term student learning outcomes from program participation?

Research Question 2 (RQ2): What are long-term student learning outcomes from program participation?

Research Question 3 (RQ3): What are the needs of students and community partners during their program involvement?

STRATEGIZE

READi had accumulated a lot of data from program stakeholders since its birth 5 years ago. Yet no one had meaningfully analyzed the data. As soon as I joined the team, I went on a scavenger hunt to dig out existing data and reflected on which data would give me meaningful insights in relation to research questions. After I had a full grasp of the data, I knew what additional data I needed and which research methods would be the best to collect the new data.

I created a 4-page research plan that proposed 3 studies. Study 1 was secondary research to get at RQ1, Study 2 was a longitudinal survey study to get at RQ2, and Study 3 was a stakeholder interview study to get at RQ3. For Study 2 and 3, I proposed possible research methods, how each method will address the research question, and estimated sample size.

I shared the plan with the team, so that we can decide on one research method and the team was clear on the project's objectives, timelines, and deliverables from each study.

This section specifies all research questions.

note: these questions are more broken down than those presented in this portfolio.

This section specifies a plan for Study 1. I indicated which existing data sources I can analyze.

OVERALL PROCESS

I independently led a 2-year project from research conception to data analysis to final reporting. Below, I will show you a snippet of secondary research (Study 1) and a longitudinal survey study (Study 2).

Analyze

Step 1. Secondary Research (Study 1)

Analyzed 22 student program reflections from 2017 to 2020.

NVivo 12

Analyze

Step 3. Thematic Analysis

Theme 1.

NVivo 12

Project Overview

Secondary Research

Stakeholder Interviews

Thematic Analysis

Longitudinal Survey

Quantitative Data Analysis

Ideation & Implementation

Step 2. Stakeholder Interviews

(Study 3)

Interviewed 15 graduate students and 5 community partners in READi.

Understand

Step 4. Longitudinal Survey

(Study 2)

Tracked students' learning over 8 months using Qualtrics.

Jan

Survey 1

May

Survey 2

Oct

Survey 3

Analyze

Step 5. Quantitative Data Analysis

Conducted the Skillings-Mack test to see students' learning changes over 8 months.

Ideate

Step 6. Ideate & Implementation

We revised one program element to support the student group's needs.

RESEARCH OVERVIEW: SECONDARY RESEARCH (STUDY 1)

I analyzed 22 students’ program reflections collected from 2017 to 2020. I conducted a thematic analysis using NVivo and found students’ short-term learning outcomes and their views on improving the program.

Focusing on Theme 1. Learning Outcomes.

I found two main learning outcomes: (a) increased accessibility awareness and (b) enhanced professional skills. Students showed increased accessibility awareness, which I defined at multiple levels. First, they showed empathy towards the challenges that persons with disabilities faced and became aware of accessibility barriers in their daily life.

“We looked at how some updated metro stations in [a city name] hadn’t carefully looked at how people with visual disabilities could navigate them. I find myself thinking about case studies like this while I travel around in my day-to-day life.”

-A

Second, students showed a changed mindset towards disability and focused on the strengths of persons with disabilities.

“Doing the community research project at [the organization] and seeing how [they] could

use computers faster than I could in some cases if they had the right technology made me realize how accessibility is about changing mindsets and technology, not changing people.”

-J

Lastly, students improved their professional skills, including team work, communication, and project management skills.

“I also gained interpersonal skills and communication skills from working with the clients at [the organization name] by speaking with the clients I gained confidence and understanding of how people can use their various abilities to communicate."

-P

Thematic analysis is complete once you draw out themes and their respective codes. Data visualization is an important part of thematic data analysis.

Student Experience of READi

Program

Improvement

Specify

learning goals

Help students stay on track

Networking opportunities

Increased accessibility awareness

Learning Outcomes

Future Endeavours

Ongoing knowledge application

Enhanced professional

skills

Empathetic

Changed

mindset

Multiple

angles

The figure shows 3 themes that outline graduate students’ view on program improvement and their reported learning outcomes and future endeavours.

Theme 2.

Theme 1.

Theme 3.

FOOD FOR THOUGHT

In this study, I had a LOT of existing data on my hand: a) student publication record, b) student recognitions and awards, c) a number of workshops delivered and student attendance, d) student monthly progress reports, e) student evaluation of certain program elements, f) student community project final reports, and f) student program reflections.

The first thing I did was to make sense of all existing data. I had to understand what those data would tell me in relation to research questions. Since one of our research questions asked about students’ short-term learning outcomes, it became obvious to me to analyze students' program reflections because these reflections captured students’ view of how various program components had impacted their knowledge on accessibility and attitudes towards the disability community.

Based on my experience, I recommend others, who are about to take on secondary research, to first make sense of data, so that you can accurately assess data’s relationship to a given research question!

RESEARCH SETUP: LONGITUDINAL SURVEY (STUDY 2)

Now that I understood students' short-term learning outcomes, I conducted a longitudinal survey study to understand students' long-term learning outcome from program participation. I also tracked students' level of program engagement, empathy towards the disability community, and interests in the field of accessibility over 8 months.

1. PARTICIPANT RECUIRTMENT

I recruited 14 graduate students from the 2020-21 cohort (cohort 1) and the 2021-22 cohort (cohort 2). Cohort 1 completed a survey in May and October, whereas cohort 2 completed a survey in January, May, and October. Ten participants came from cohort 1 (Mage = 30.7 years old). Four participants came from cohort 2 (Mage = 24.25 years old). I gave participants a 2-week time frame to complete the survey.

2. SURVEY DESIGN

I designed a survey using Qualtrics and followed these steps.

1. I used PsycTests and prior research to find relevant measurements. Tip. How do you know what constructs to measure? I can give two answers to that: 1) what are your research questions? and 2) what are your team's priorities? You form research questions by reading prior work; from this initial process, you would have an idea of which constructs are important and relevant for your own study. Also, check in with your teams about what they wish to know. Do they want to know about students' behavioural change, emotional change, or both?

Cognitive Learning

Students indicated their general understanding of accessibility. Sample items: “I can identify benefits of accessibility," and “I can define the purpose of the Accessible Canada Act” (1 - Strongly disagree; 7 - Strongly agree).

Career Interests

Students indicated the degree to which they were interested in pursuing a career where they could apply accessibility knowledge. Sample items are “I want to explore career options where I can apply accessibility-related knowledge” and “I am not interested in learning what accessibility-related career opportunities are available” (1 - Strongly disagree; 7 - Strongly agree).

Empathy

Students indicated the degree to which they felt empathy towards the specific community for whom their community research project will directly impact. The stem of the question was “When I think about the specific community of my ATP project …” and the exact items were “compassionate,” “moved,” “soft-hearted,” “sympathetic,” “tender,” and “warm” (1 - Strongly disagree; 7 - Strongly agree).

Student Engagement

Students indicated their emotional engagement with READi and sample items are “I feel energetic being part of this program” and “I am interested in the material I learn in this program”. They indicated their physical engagement with the READi (i.e., the degree to which a student puts effort in working on assignments.), and sample items are “I exert my full efforts towards this program” and “I devote a lot of energy towards this program.” Lastly, they indicated their cognitive engagement with READi and sample items are “my mind is always focused on project discussions” and “I pay a lot of attention to project discussions and tasks” (1 - Strongly disagree; 7 - Strongly agree).

2. I randomized the presentation of measurements, so that each participant see a different order of measurements (this step is necessary to avoid order effect!).

3. I modified the wording of original measurements to fit into the study’s context.

4. I pretested the survey with my colleagues and the team. This step is crucial; others can tell you which items were confusing and to get an estimated time duration.

5. I alway check for a survey flow. This is a great way to check if you missed some measurements!

KEY INSIGHTS

I conducted the Skillings-Mack test to account for a small sample size and non-independence of error due to the nature of the repeated measures design. I did not find significant differences between time periods on all measurements, except on empathy towards the disability community. On empathy, for cohort 1, there was a marginal significance, such that participants reported higher empathy in October than in May (Skillings-Mack Statistic = 3.60, p = 0.06). For cohort 2, there was no statistical difference between different times (Skillings-Mack Statistic = 0.23, p = 0.89).

Empathy

6.08

January

May

October

Cognitive Learning

January

May

October

5.92

6.08

5.60

Career Interests in Accessibility

January

May

October

4.60

4.30

4.60

These figures show the effect of the program on students' empathy toward the disability community, cognitive learning, and career interests. Figures for student engagement are not shown. Medians are reported.

While we did not find significant changes in most dependent variables, it is essential to note that participants’ medians on these variables were consistently high over eight months for both cohorts, which were above the midpoint on a 7-point Likert scale.

One interpretive lens of these non-significant findings is that READi’s program elements successfully maintained students’ knowledge and engagement.

OUTCOMES & LESSONS

A change was made to implement a weekly meeting with a senior doctoral student.

IMPACT

From the stakeholder interview study (Study 3), I found some students were afraid to reach out to program faculty and ask for support. Originally, graduate students had a bi-weekly meeting with a senior doctoral student who had finished the program in a subsequent year. As a result of my study result, this bi-weekly meeting has changed to a weekly meeting, so that graduate students has more opportunities to ask for support with a non-authoritative future.

Two academic publications to computer engineering education conferences.

Two academic publications submitted to computer engineering education conference and journal.

Yourself: An impact repository builds your professional confidence and can be referenced during performance evaluations.
Your team: Understanding what research is used and how it’s used can improve your research strategy and roadmap.

Elevated the importance or visibility of UX research in your company.
Opened up new avenues for collaboration across teams.

TAKEAWAY

Strategize, strategize, strategize. This was a large-scale project! I was lost and questioned what I should be doing or why I was doing this particular task. But my research plan helped me stay focused and reminded me of my high-level goals for each study. I again saw the value in a detailed research plan in helping me achieve all the goals.
Different research methods answer different research questions. I used three research methods to get at three research questions. Conducting an interview to track long-term student learning outcomes could be better. Similarly, using a survey to understand stakeholder needs is not appropriate. Through this program evaluation, I once again realized that being a good researcher means developing the capacity to weigh the pros and cons of each research method and assess which method is suitable to answer a particular research question.

Jin Kang, Adrian D. C. Chan, Chantal M. J. Trudel, Boris Vukovic, and Au- drey Girouard. 2021. Diversifying Accessibility Education: Presenting and Evaluating an Interdisciplinary Accessibility Training Program. In 21st Koli Calling International Conference on Computing Education Research (Koli Calling ’21), November 18–21, 2021, Joensuu, Finland. ACM, New York, NY, USA, 6 pages. https://doi.org/10.1145/3488042.3490021

Jin Kang, Chantal M. J. Trudel, Audrey Girouard, and Adrian D. C. Chan. 2021. Research and Education in Accessibility, Design, and Innovation (READi) Training Program: Preparing Graduate Students for Careers in Accessibility Research and Design. In Proceedings of 3rd Annual EduCHI’21 Symposium. May 15, 2021, Virtual Conference, 9 pages.