Designing musical wearable haptic for users with vision impairments
Research Problem
-
Popular musical wearable haptic products fail to attract users with vision impairments:
-
Limited functionality: the users need to feel music information conveyed through hand gestures. But, the products mainly communicate tempo.
-
Unclear material preference: We don't know which materials the users prefer in the products.
-
Research Impact
Strategic Impact:
-
Published actionable recommendations on functionality and materiality of wearable haptic products for users with vision impairments.
-
Competitive analysis shows new product opportunities for the leading business in the market.
Solution Explored
-
Ideated with users with vision impairments & identified 2 hand gestures that the products can communicate and 4 material preferences in the products.
-
Team: Research associate (me), Graduate researcher (IT), Project supervisor (IT) • Duration: 5 months
PROBLEM
Users with vision impairments lack access to hand gestures when learning music
Much musical information is conveyed through hand gestures. But, users who are blind or have low vision (hereafter, BLV) have limited access to these cues and miss out on critical musical information.
A music conductor uses so many hand gestures for you to follow through. But I can't see them, and I am lost for information.
- A BLV user with 12 yrs of music experience from an online community.
Different hand gestures let players know about their next steps while playing.
SOLUTION
Could wearable haptic products convey hand gestures to BLV users?
We saw the potential in the products to convey hand gestures to BLV users for two reasons.
-
They look like regular accessory and do not draw unwanted social attention to a BLV user.
-
BLV users can receive information about hand gestures without taking their hands off their instruments.
We asked the following research question to identify user requirements of the products:
-
Research Question 1: Which hand gestures can be communicated by wearable haptic products to BLV users?
One must look beyond a product's functionality and consider its materials (or broadly aesthetics) to create a positive user experience. This is why we asked the second question:
-
Research Question 2: What materials do BLV users want to feel in wearable haptic products?
Accessibility is good for business.
Buying power
People with disabilities have a buying power of $50 billion in Canada.
Revenues
Companies that champion accessibility attract people with disabilities and their networks.
Canadians with disabilities
Together with their friends and families, this group is the third-largest market segment in North America.
Thus, one business outcome for following our product recommendations is:
-
Increased revenues by attracting customers with vision impairments and their networks through innovative products.
OVERALL PROCESS
We led a 4-month project from research conception to final research report preparation.
Step 1. Scoping
We conducted a literature review, and our goals were: 1) to formulate research questions and determine what needs to be investigated and 2) to quickly get up to speed on co-design techniques.
Doctoral researcher and I equally contributed here.
Microsoft Word
Google Scholar
Step 3. Conducting: Subject Matter Expert Interview
We conducted a 2-hour hybrid subject matter expert interview with a blind music teacher to improve the workshop procedure and refine discussion questions.
Doctoral researcher and I equally contributed here.
Zoom
Google Doc
Step 5. Analyzing & Synthesizing: Thematic Analysis
We conducted a collaborative thematic analysis and found 2 themes that answered to our research questions.
Doctoral researcher and I equally contributed here.
MAXQDA
Miro
Step 2. Research Planning
1. Developed a wearable haptic prototype.
2. Wrote and revised a co-design workshop script based on dry rehearsals.
3. Sorted out study logistics (e.g., participant recruitment, budget, timelines).
I contributed to #2-3. Doctoral researcher to all.
Zoom
Google Doc
Step 4. Conducting: Co-design Workshops
We conducted 5 in-person workshops with 10 BLV music teachers and learners. We recruited participants from a music school for BLV people located in New York.
Google Doc
Doctoral researcher ran the workshops.
Step 6. Dissemination
Final report and presentation: we submitted a polished research report to the HCI conference so that our work could reach out to the wider population.
Doctoral researcher and I equally contributed here.
PowerPoint
Google Doc
SUBJECT MATTER EXPERT INTERVIEW
After we prepared the first draft of a co-design script and developed a wearable haptic prototype, we interviewed a BLV subject matter expert (SME), who is a music teacher in the US, and achieved the following goals:
1. Included new discussion questions that are relevant to BLV music learners' lived experiences.
2. Refined the workshop procedure to be accessible to BLV participants.
3. Finalized which ideation materials to include.
4. Tested out whether the wearable haptic prototype and its preset vibration pattern made sense in relation to music learning.
The doctoral researcher (top panel, left) and I (bottom panel) are conversing with an SME and rehearsing a co-design workshop procedure. That is my "I am serious" face as I make important notes in the script.
-
We included these 12 ideation materials for in the co-design workshops, including soft felt, leather, metal, and wood.
-
We chose these materials because they are generally found in musical instruments and music classrooms.
-
The doctoral researcher developed a wearable haptic prototype with one preset vibration pattern called an on-and-off pattern.
vibrates
pauses
vibrates
pauses
pauses
vibrates
vibrates
-
The pattern can play at speeds ranging from 60 to 100 beats per minute (BPM).
-
Here is a visualization showing the prototype playing the pattern at 60 BPM.
RESEARCH SETUP
1. Participant Recruitment
We recruited 10 BLV music students and teachers from a music school in New York, USA (one woman & nine men).
Each co-design workshop had a pair of participants and took place at the school. All sessions were video-recorded for accurate transcripts and reliable thematic data analysis.
2. Co-design Script
We had a co-design script to standardize the data collection procedure across five co-design sessions.
The script specified:
1. Greeting statements to participants.
2. Instructions for three activities.
3. Discussion questions in order.
CO-DESIGN WORKSHOPS
Activity 1.
Participants brainstormed which materials were suitable for wearable haptic products.
In this photo, 2 participants are given some time to feel each material.
Activity 2.
Participants ideated what hand gestures can be communicated through vibrations.
Participants played their musical instruments, placed the prototype on different body parts, and generated new vibration patterns.
Purple circles highlight the placement of the prototype tried on by BLV participants.
Personal Learning
Our BLV participants did not explore the ideation probe to the extent that we had expected. Their brainstorming centred on applying two preset vibration patterns, and they came up with one new vibration pattern, which is crescendo and decrescendo. This limited exploration with the ideation probe could be due to the individual differences in people's on-the-spot brainstorming skills. Some individuals are naturally good with on-the-stop verbal brainstorming, and others are not. To address this individual difference variable, I'd revise the workshop such that BLV participants are given enough time to individually explore the ideation probe and reflect on discussion questions before they join the workshop.
KEY INSIGHTS
Theme 1. Communicate Basic
Hand Gestures
Participants agreed that wearable haptic products can convey basic hand gestures using straightforward vibration patterns.
1. Use the "on and off" vibration pattern to tell a BLV user when to start playing their instruments.
"In my jazz band, people yell at me to say it is my turn to play a drum. But I can use this technology to tell me when it is my turn to play and turn to end."
-Y
Participants played the piano upon receiving the on-and-off vibration pattern on their arms.
2. Use the "on and off" vibration pattern to indicate fermata, signalling a BLV user to take a break for an unspecified time.
"Sometimes our conductor would have a fermata, and he would indicate holding his hand out and that close when it is time to pause. We don't see it. The vibration could help". Tell us to hold music."
-K
3. Use the vibration pattern of increasing or decreasing intensity to convey crescendo and decrescendo.
"Definitely dynamics! If I was playing a phrase that needed to be quieter or I needed to transition to a different dynamic, [vibration] intensity would convey that."
-W
Theme 2. Material Preferences
There were 4 desired material properties in wearable haptic products. Materials must be ...
Distinguishable
"Technology should be made of hard materials to easy to identify different components of the device. I feel in sense of control when I can identify things."
Durable
"The device will take damage. Music [playing] can be very rigorous. Metals and woods are good."
Dirtiness
"No rough and abrasive materials. Also, no materials like silicon which collects dust and make it feel dirty whenever I touch the technology."
Enjoyable
"I have to enjoy touching the textures as I rub through my fingers. Soft fabrics felt nice."
REFLECTION
FUTURE WORK
TAKEAWAY
-
Iterate on the prototype based on user feedback and the prototype in the real world. Have BLV musicians use the prototype over months and validate the prototype.
-
Make the prototype available to the larger population based on request.
-
Learn the team's lingo for collaboration. The graduate researcher and I had different expertise, and I proactively learned about his prototyping techniques and past projects to speak his lingo. My proactive approach contributed to making our collaboration effective.
-
Delegate tasks smartly. I contributed to tasks that I am good at, including data analysis. The graduate researcher brought their expertise in prototype development. Delegating tasks based on each other's skills maximize the team's productivity, and I learned about other members' skills.
-
Be open to new approaches. We conducted most of the analysis using MAXQDA. The graduate researcher suggested that we approach theme formation using Miro. At first, I hesitated but respected his suggestion and tried Miro. Guess what? I loved developing themes with Miro. Using Miro, I found forming themes more fun and efficient.