Doe

Doe

Augmented Speech Therapy Mirror

For my Human Computer Interaction course, we were briefed to 'design for one' through solutions that utilize Ai enabled assistive technology.

My team focused on communication. Easing the way young people with articulation disorders can improve their expressive language. Doe is an augmented mirror with daily adaptive and visualized lessons to reach an individual's goal.

For my Human Computer Interaction course, we were briefed to 'design for one' through solutions that utilize Ai enabled assistive technology.

My team focused on communication. Easing the way young people with articulation disorders can improve their expressive language. Doe is an augmented mirror with daily adaptive and visualized lessons to reach an individual's goal.

Language is a way of expressing yourself.

Learning while young. In a child's early years, they are learning to convert their thoughts and emotions into words. Development takes time. Speech gradually becomes easier to understand and language becomes more fluent. When these language development milestones are not met, it affects their reading, writing, and social relationships. (American Speech-Language Development Association)

Learning while young. In a child's early years, they are learning to convert their thoughts and emotions into words. Development takes time. Speech gradually becomes easier to understand and language becomes more fluent. When these language development milestones are not met, it affects their reading, writing, and social relationships. (American Speech-Language Development Association)

This is Maddie

9 years old
from Chicago 

“I worked with my speech pathologist from age 3 —11 to help my lisp.” 

“I worked with my speech pathologist from age 3 —11 to help my lisp.” 

maddie 2

Problem Space

Problem Space

How effective speech therapy is depends on the how well the patient can integrate what they learn from their sessions with their speech pathologist in their everyday life. 

How effective speech therapy is depends on the how well the patient can integrate what they learn from their sessions with their speech pathologist in everyday life. 

graphs2

Infrequent Visits

Infrequent Visits

For years patients go see their speech pathologist once a month for a 2 hour lesson. At home and at school is where most of the practice takes place.

For years patients go see their speech pathologist once a month for a 2 hour lesson. At home and at school is where most of the practice takes place.

Screen Shot 2017-11-23 at 8.12.51 PM

Interactions with Speech Therapist

Techniques like these are used to manipulate tongue movement to then alter how we create sounds. Often a bit uncomfortable.

Techniques like these are used to manipulate tongue movement to then alter how we create sounds. Often a bit uncomfortable.

Screen Shot 2017-12-02 at 12.53.35 AM

Tiers of Connections

Who does Maddie interact with? Her support. We wanted to value these people because they are her motivation. 

Who does Maddie interact with? Her support. We wanted to value these people because they are her motivation. 

Opportunities

Opportunities

How might we create an engaging, adaptive at home tool for children with articulation disorders that's easier to understand, while also giving consistent progress communication to speech therapists.

How might we create an engaging, adaptive at home tool for children with articulation disorders that's easier to understand, while also giving consistent progress communication to speech therapists.

Screen Shot 2017-11-22 at 9.46.12 PM

Eileen Sander, Audiologist + Speech Pathologist

Eileen Sander, Audiologist + Speech Pathologist

“We create individualized goals because each person is different."

Therapists qualitatively assess their patient and create new exercises each session to meet goals. 

Therapists qualitatively assess their patient and create new exercises each session to meet goals. 

Building Sounds

Building Sounds

Language is our vocabulary and speech is how we articulate these words. Phonetics break down language to the fundamental sounds humans can create — there are about 40 of them in the English language. Exercising the tongue aids in the development of these sounds. 

Screen Shot 2017-11-22 at 10.08.44 PM
Screen Shot 2017-11-22 at 9.46.55 PM

Beth Aseere, Speech Pathologist + Evaluation Specialist

Beth Aseere, Speech Pathologist + Evaluation Specialist

“I would use all different tricks + techniques that could help them meet their goals.”

Speech Pathologists are reliant on evidence based decision making processes which allow other therapists to adapt their future lessons with tried and successful approaches. 

Speech Pathologists are reliant on evidence based decision making processes which allow other therapists to adapt their future lessons with tried and successful approaches. 

Learning by Mimicry

Learning by Mimicry

Children depend on our eyes and our ears when developing their language and speech. They naturally repeat these actions and sounds to learn — it's innate.

Mirrors are a tool speech pathologists use to allow their patient to watch themselves make the movements. We imagined the mirror being an artifact for how we could allow mimicry to be analyzed and adjusted.

Children depend on our eyes and our ears when developing their language and speech. They naturally repeat these actions and sounds to learn — it's innate.

Mirrors are a tool speech pathologists use to allow their patient to watch themselves make the movements. We imagined the mirror being an artifact for how we could allow mimicry to be analyzed and adjusted.

iphone screen 3

 

Setting up her space

App + Mirror work in tandem

mirror desk shot
reading 1 copy

Identifying Maddie's Speech Disorder

During a speech pathologist's evaluation, their patients often read this paragraph to understand what their difficulty is to create their individualized education plan. 

When on boarding with Doe, Ai identifies speech errors. This is how the system orients Maddie's set of goals once approved with her speech pathologist. 

During a speech pathologist's evaluation, their patients often read this paragraph to understand what their difficulty is to create their individualized education plan. 

When on boarding with Doe, Ai identifies speech errors. This is how the system orients Maddie's set of goals once approved with her speech pathologist. 

Building Habits

Speech therapy relies heavily on building good practice habits and having a good teacher. For Maddie, she starts her lessons with her mirror in the comfort of her own home.

Some lessons are designed to include the parents participation. I wanted families to be present and active to continue their loved ones development to meet their communication goals. 

Speech therapy relies heavily on building good practice habits and having a good teacher. For Maddie, she starts her lessons with her mirror in the comfort of her own home.

Some lessons are designed to include the parents participation. I wanted families to be present and active to continue their loved ones development to meet their communication goals. 

Learning through Play

Learning through Play

For everyday lessons, we needed to keep the lessons quick and the interest level high. Aiming to be fun, interesting, and motivating. We adopted our lessons from current speech pathologist techniques by visualizing articulatory movements.

For everyday lessons, we needed to keep the lessons quick and the interest level high. Aiming to be fun, interesting, and motivating. We adopted our lessons from current speech pathologist techniques by visualizing articulatory movements.

th sound

Lessons

After every lesson, Doe then analyzes Maddie's optimal learning patterns and prepares the next day's lesson. By readjusting the course of her lessons everyday, she can reach her goals quicker and effectively. 

After every lesson, Doe then analyzes Maddie's optimal learning patterns and prepares the next day's lesson. By readjusting the course of her lessons everyday, she can reach her goals quicker and effectively. 

Understanding Goals

Having a clear set of goals as a patient is valuable. Maddie is able to understand where she has come and what is left to learn. Her progress is also given to her speech pathologist and her family. 

The lessons are organized by language and speech to reflect the natural way speech pathologists analyze. (asha.org)

Language — Sound Goals
Speech — Movement Goals

Sharing your Beats 

One of our concepted lessons was to mimic sets of sounds that create a sick beat. We wanted Maddie to be able to record these sounds and share it amongst her friends. 

Incorporating the sharing element allows patients to build confidence and track their own progress. 

reading 3 copy 4

Data Collection

When creating lessons the system incorporates the variety of unique techniques from speech pathologists as well as adapting sounds formed in dissimilar languages to find different ways of producing sounds to alter Maddie's lessons.

The mirror analyzes tongue and mouth movements. Then identifies problems through our Ai's perception in the form of contextual adaptation.

Takeaway

This was my introduction to Interaction Design. I learned a lot about system thinking and organizing for prototyping. Moving forward, I would like to prototype a few lessons to understand plausible structure for the lesson's feedback and affordances. 

This project was nostalgic. My grandmother is one of the experts we interviewed. Growing up we would often people watch together. She would tell me about common behaviors that are indicative of a certain characteristic. Since then I was always interested in nonverbal communication and the power of observation. I was most excited to create a meaningful tool for observers + doers, like my grandma, to help their patients. 

Andre Physical Product Prototyping, Research
Taylor StoryBoarding, Generative Research, Video
Brogan System Diagram, Wire-framing, Research
Satyn Film Producer, Director
Me Research, Art Direction, Interaction Design, Product Strategy

Next Project

Waitstaff Review