Doe — Augmented Speech Therapy Mirror
For my Human-Computer Interaction course, we were briefed to 'design for one' through solutions that utilize Ai enabled assistive technology.
My team focused on communication. Easing the way young people with articulation disorders can improve their expressive language. Doe is an augmented mirror with daily adaptive and visualized lessons to reach an individual's goal.
For my Human Computer Interaction course, we were briefed to 'design for one' through solutions that utilize Ai enabled assistive technology.
My team focused on communication. Easing the way young people with articulation disorders can improve their expressive language. Doe is an augmented mirror with daily adaptive and visualized lessons to reach an individual's goal.
Language is a way of expressing yourself.
Learning while young. In a child's early years, they are learning to convert their thoughts and emotions into words. Development takes time. Speech gradually becomes easier to understand and language becomes more fluent. When these language development milestones are not met, it affects their reading, writing, and social relationships. (American Speech-Language Development Association)
Learning while young. In a child's early years, they are learning to convert their thoughts and emotions into words. Development takes time. Speech gradually becomes easier to understand and language becomes more fluent. When these language development milestones are not met, it affects their reading, writing, and social relationships. (American Speech-Language Development Association)
This is Maddie
9 years old
from Chicago
“I worked with my speech pathologist from age 3 —11 to help my lisp.”
“I worked with my speech pathologist from age 3 —11 to help my lisp.”
Problem Space
How effective speech therapy depends on how well the patient can integrate what they learn from their sessions with their speech pathologist in their everyday life.
How effective speech therapy is depends on the how well the patient can integrate what they learn from their sessions with their speech pathologist in everyday life.
Infrequent Visits
For years patients go see their speech pathologist once a month for a 2 hour lesson. At home and at school is where most of the practice takes place.
For years patients go see their speech pathologist once a month for a 2 hour lesson. At home and at school is where most of the practice takes place.
Interactions with Speech Therapist
Techniques like these are used to manipulate tongue movement to then alter how we create sounds. Often a bit uncomfortable.
Techniques like these are used to manipulate tongue movement to then alter how we create sounds. Often a bit uncomfortable.
Tiers of Connections
Who does Maddie interact with? Her support. We wanted to value these people because they are her motivation.
Who does Maddie interact with? Her support. We wanted to value these people because they are her motivation.
Opportunities
How might we create an engaging, adaptive at home tool for children with articulation disorders that's easier to understand, while also giving consistent progress communication to speech therapists.
How might we create an engaging, adaptive at home tool for children with articulation disorders that's easier to understand, while also giving consistent progress communication to speech therapists.
“We create individualized goals because each person is different."
Eileen Sander, Audiologist + Speech Pathologist
Therapists qualitatively assess their patient and create new exercises each session to meet goals.
Speech Pathologists are reliant on evidence based decision making processes which allow other therapists to adapt their future lessons with tried and successful approaches.
Building Sounds
Language is our vocabulary and speech is how we articulate these words. Phonetics break down language to the fundamental sounds humans can create — there are about 40 of them in the English language. Exercising the tongue aids in the development of these sounds.
“I would use all different tricks + techniques that could help them meet their goals.”
Beth Aseere, Speech Pathologist + Evaluation Specialist
Speech Pathologists are reliant on evidence based decision making processes which allow other therapists to adapt their future lessons with tried and successful approaches.
Speech Pathologists are reliant on evidence based decision making processes which allow other therapists to adapt their future lessons with tried and successful approaches.
Learning by Mimicry
Learning by Mimicry
Children depend on our eyes and our ears when developing their language and speech. They naturally repeat these actions and sounds to learn — it's innate.
Mirrors are a tool speech pathologists use to allow their patient to watch themselves make the movements. We imagined the mirror being an artifact for how we could allow mimicry to be analyzed and adjusted.
Children depend on our eyes and our ears when developing their language and speech. They naturally repeat these actions and sounds to learn — it's innate.
Mirrors are a tool speech pathologists use to allow their patient to watch themselves make the movements. We imagined the mirror being an artifact for how we could allow mimicry to be analyzed and adjusted.
Setting up her space
App + Mirror work in tandem
Identifying Maddie's Speech Disorder
During a speech pathologist's evaluation, their patients often read this paragraph to understand what their difficulty is to create their individualized education plan.
When on boarding with Doe, Ai identifies speech errors. This is how the system orients Maddie's set of goals once approved with her speech pathologist.
During a speech pathologist's evaluation, their patients often read this paragraph to understand what their difficulty is to create their individualized education plan.
When on boarding with Doe, Ai identifies speech errors. This is how the system orients Maddie's set of goals once approved with her speech pathologist.
Building Habits
Speech therapy relies heavily on building good practice habits and having a good teacher. For Maddie, she starts her lessons with her mirror in the comfort of her own home.
Some lessons are designed to include the parent's participation. I wanted families to be present and active to continue their loved ones development to meet their communication goals.
Speech therapy relies heavily on building good practice habits and having a good teacher. For Maddie, she starts her lessons with her mirror in the comfort of her own home.
Some lessons are designed to include the parents participation. I wanted families to be present and active to continue their loved ones development to meet their communication goals.
Learning through Play
For everyday lessons, we needed to keep the lessons quick and the interest level high. Aiming to be fun, interesting, and motivating. We adopted our lessons from current speech pathologist techniques by visualizing articulatory movements.
For everyday lessons, we needed to keep the lessons quick and the interest level high. Aiming to be fun, interesting, and motivating. We adopted our lessons from current speech pathologist techniques by visualizing articulatory movements.
Lessons
After every lesson, Doe then analyzes Maddie's optimal learning patterns and prepares the next day's lesson. By readjusting the course of her lessons everyday, she can reach her goals quicker and effectively.
After every lesson, Doe then analyzes Maddie's optimal learning patterns and prepares the next day's lesson. By readjusting the course of her lessons everyday, she can reach her goals quicker and effectively.
Understanding Goals
Having a clear set of goals as a patient is valuable. Maddie is able to understand where she has come and what is left to learn. Her progress is also given to her speech pathologist and her family.
The lessons are organized by language and speech to reflect the natural way speech pathologists analyze. (asha.org)
Language — Sound Goals
Speech — Movement Goals
Sharing Beats
One of our concepted lessons was to mimic sets of sounds that create a sick beat. We wanted Maddie to be able to record these sounds and share it amongst her friends.
Incorporating the sharing element allows patients to build confidence and track their own progress.
Data Collection
When creating lessons the system incorporates the variety of unique techniques from speech pathologists as well as adapting sounds formed in dissimilar languages to find different ways of producing sounds to alter Maddie's lessons.
The mirror analyzes tongue and mouth movements. Then identifies problems through our Ai's perception in the form of contextual adaptation.
Takeaways
This was my introduction to Interaction Design. I learned a lot about system thinking and organizing for prototyping. Moving forward, I would like to prototype a few lessons to understand a plausible structure for the lesson's feedback and affordances.
This project was nostalgic. My grandmother is one of the experts we interviewed. Growing up we would often people watch together. She would tell me about common behaviors that are indicative of a certain characteristic. Since then I was always interested in nonverbal communication and the power of observation. I was most excited to create a meaningful tool for observers + doers, like my grandma, to help their patients.
Andre Physical Product Prototyping, Research
Taylor StoryBoarding, Generative Research, Video
Brogan System Diagram, Wire-framing, Research
Satyn Film Producer, Director
Me Research, Art Direction, Interaction Design, Product Strategy