Guide
Demonstrate the movement and isolate the active facial region.
Break the mask barrier
HYPO
MIMIA
Overview
An AU-based rehabilitation training system for Parkinson’s hypomimia, combining guided facial exercises with immediate visual feedback.
Parkinson’s hypomimia can reduce facial expressiveness and make everyday communication harder. This project reframes rehabilitation as a clear, repeatable interaction rather than an occasional clinical assessment.
ScopeInteraction designComputer visionRehabilitation HCIClinical study
Extended Abstracts of the CHI Conference on Human Factors in Computing Systems
“BREAK THE MASK BARRIER”: AN AU-BASED REHABILITATION TRAINING SYSTEM FOR PARKINSON’S HYPOMIMIA
Clinical assessment is often subjective, in-person and difficult to repeat frequently.
HOW MIGHT FACIAL TRAINING BECOME MEASURABLE, ACCESSIBLE AND MOTIVATING?
SIX CLINICIANS DEFINED THE REAL BARRIERS.
Semi-structured interviews with four neurologists and two rehabilitation physicians—each with more than ten years of experience—translated clinical practice into five design requirements.
- 01
Reduce cognitive load
Make the interface clear, singular and easy to follow.
- 02
Support emotional readiness
Use enjoyable content to lower resistance and raise motivation.
- 03
Enable active rehabilitation
Replace passive prompting with targeted guidance and immediate feedback.
- 04
Make change visible
Quantify the process and present understandable progress over time.
- 05
Scale difficulty
Train individual facial regions before coordinated full-face movements.
DETECT THE MOVEMENT.
RETURN THE FEEDBACK.
A browser-based prototype combines recorded demonstrations with OpenGraphAU recognition. The model selects the highest-confidence face, measures relevant Action Unit activation against a neutral baseline and returns encouraging feedback instead of punitive error alerts.
Recognise
Read Action Unit movement through the camera in real time.
Respond
Return an immediate correction, confirmation or composite score.
RecognitionOpenGraphAU
PrototypeVue + Python server
Training mediaClinically grounded videos recorded with an acting teacher
BUILD CONTROL.
THEN BUILD FLUENCY.
Basic training
Learn one region at a time.
Guided movements focus on the eyebrows, eyes, nose, lips, swallowing and pronunciation before combining them into a full expression.
Advanced training
Coordinate movement to rhythm.
Users choose a duration and opera track, practise integrated movements and receive “perfect,” “well done” or “come on” feedback before a composite score.
05 / Motivation layer
TURN REPETITION INTO A SMALL PERFORMANCE.
Three Chinese opera pieces match different facial tasks: “Rolling Lantern” for lip movement, “Celestial Beauty Scattering Flowers” for eyebrow, eye and nose movement, and “The Case of Chen Shimei” for pronunciation. Translucent face-changing filters appear between segments.
TESTED IN A PUBLIC HOSPITAL IN CHINA.
Seven people with Parkinson’s hypomimia, their caregivers and ten physicians took part. Participants averaged 69.57 years of age and a 2.29 MDS-UPDRS hypomimia severity score. Each supervised session lasted 20–30 minutes and ended with questionnaires and interviews.
7 patients / 6 male, 1 female / ages 62–80
7 neurology physicians / 3 rehabilitation physicians
Engagement, autonomy, usability and interactive performance
EARLY SIGNALS FROM A SMALL-SCALE STUDY.
Engagement
Desire for frequent use
Ease of learning
6 / 7reported facial relief
5 / 7reported improved articulation
7 / 7felt emotionally relaxed
Physician rating3.88 / 5Fault tolerance
Physician rating4.00 / 5Clinical suitability
Physician rating4.29 / 5Perceived effectiveness
THE STUDY ALSO SHOWED WHERE THE SYSTEM MUST IMPROVE.
Recognition bias
AU recognition was less reliable across patients with different symptom severity, with occasional delay or missed detection.
Clearer instruction
Complex tongue and coordinated movements need voice explanation so gestures are not misinterpreted.
Personalisation
Training duration, difficulty and music should adapt to individual ability, progression and preference.
Clinical validation
A larger longitudinal study, control group and randomized trial are required before making efficacy claims.
Privacy by design
Future deployment must anonymize facial data and protect stored and transmitted information.
Outcome
The system turns facial Action Unit recognition into structured basic and adaptive training, supporting repeatable practice at home and in clinical settings.
The project demonstrates how computer vision can make a subtle physical action legible enough to practise. The pilot supports usability and engagement, while a larger longitudinal RCT is still required to establish clinical efficacy.
Nature’s Embrace
Responsive Installation / Biophilic Interaction / Workplace Wellness