Online Game Offers Children with Autism the Chance to Express Emotions

Aug. 25, 2011

AUSTIN, Texas — Yan Zhang, assistant professor in the School of Information at The University of Texas at Austin, is creating an interactive online game to help children with autism spectrum disorders (ASDs) communicate their emotions.

The game will help the State of Texas achieve a high-quality intervention program for children with ASDs and the research can be used to aid autism communities across the nation.

For families around the world, this new technology will, for the first time, allow much more meaningful communications with their autistic child.

ASDs are a group of neuro-developmental and psychological disorders affecting one in 110 children in the United States.

Children with the disorder often have impaired social and communication skills and are unable to recognize and understand the cognitive and emotional state of others, including those conveyed through non-verbal clues such as facial expressions.

The goal of the research is to develop an early intervention tool, an interactive and adaptive game system, to help children with ASDs recognize and understand emotions expressed through facial expressions.

“Over the past year I’ve worked closely with numerous families who have a child with autism,” said Zhang. “They are so caught up in the difficulties of daily routines that they can’t even fathom the chance of getting to have a meaningful discourse with their child. This would be life-changing for them.”

Existing computer-based interventions to help children with autism use static response-based software. The game being created by Zhang and colleague J.K. Aggarwal in the Cockrell School of Engineering, integrates adaptive and responsive components, along with real-time image capture (via webcam trained on the child's face).

These components help children with autism identify their own emotional responses onscreen, helping them to then communicate those emotions to others.

Games that are available now also do not offer context, which is crucial for emotional communication. Zhang is incorporating meaningful, recognizable social scenarios into the game and embedding the child's facial expressions onto the avatar, giving the child a chance to immediately recognize his or her own facial reaction to a situation.

With this knowledge, children can see not only their own emotional response to a situation, but can begin to recognize the emotional responses of others.

Children will directly interact with the game using both a keyboard and a mouse, while a webcam will capture their facial expressions. An expression analysis component will simultaneously track the facial movements captured by the webcam and analyze the expressions.

Based on the results of this analysis, a virtual character synthesis process will create an avatar, so that the child can see his or her facial expressions reflected on the avatar.

The results of the expression analysis can also be directly rendered to the interface component, so that the child can tell whether he or she is correctly mimicking the virtual avatar’s facial expressions and whether he or she is making a proper facial expression to match a particular social scenario.

“Due to the wide spectrum of symptoms and behaviors within ASDs, effective solutions need to be able to adapt to the characteristics of each child,” said Zhang.

“Thus, we will also make the game adaptive to each child’s behavior. For example, if a child is playing the same game mode repeatedly, in a manner that is not increasing learning, the game system will adapt the level or mode accordingly to encourage learning.”

The project, entitled LIFEisGAME, is being conducted in collaboration with Verónica Orvalho from the University of Porto, Portugal.

For more information, contact: Amy Crossette, Office of the President, 512-573-1078.

28 Comments to "Online Game Offers Children with Autism the Chance to Express Emotions"

1.  Mary McKinney said on Aug. 26, 2011

"She gets IT" Wow, she really does. With this phrase alone...

"have a meaningful discourse with their child"

2.  Lisa Gruen said on Aug. 26, 2011

When will it be available? I would love for my 10 year old ASD son who loves the app on his ipad to test it out!

3.  Amy Crossette said on Aug. 29, 2011

Lisa: The game is still under development but two modes are currently being tested. If you would like additional information, please contact Yan Zhang at yanz@ischool.utexas.edu.

4.  Angie said on Aug. 29, 2011

Sounds awesome! This is the second article in the last few months I have read about this, when will it be out? Do you need any testers? I have 2 boys with Autism, and a daughter with a speech delay...email us if you need any testers and/or when it will be out/how to buy/etc! Thanks!

5.  Kristin said on Sept. 1, 2011

How incredibly exciting! Please send out updates as the process advances. Is there a mailing list or a website that we could check? Do you need more testers? Do you need volunteers for the office, or supplies, or anything at all?

6.  Julie said on Sept. 1, 2011

I wonder if this will help with diminished empathy in non-autism spectrum children?

7.  Rossana said on Sept. 1, 2011

what a great idea! i would like my son to try/test it too. Thanks.

8.  Rhonda Fay said on Sept. 1, 2011

My Asperger's son loves the computer. This will motivate hime in all areas. Can't wait till it comes out!

9.  Carolyn said on Sept. 1, 2011

What's the name of the app, Lisa?

10.  Laura said on Sept. 1, 2011

I would love for my 18-year-old Asperger's son to be part of beta testing. Please keep us posted on testing and release dates. Thank you for dedicating your research to this growing and very misunderstood segment of the general population!

11.  Teresa Vieira said on Sept. 1, 2011

You should Know, that in Faculdade de Ciências da Universidade do Porto with the collaboration of UT-Austin is being developed a similar project:

LIFEisGAME overall objective is to deploy a low cost real time facial animation system embedded in an experimental game, which will allow to further study the symptomatic problems of facial emotion recognition.

LIFEisGAME advances the synthesis of realistic virtual characters and markerless motion capture technology, and creates a non-stressful game to help individuals recognize facial emotions in an interactive way.

http://www.portointeractivecenter.org/lifeisgame/

http://www.publico.pt/Tecnologia/um-jogo-que-leva-as-emocoes-aos-dedos-das-criancas-autistas_1508649

This project has several papers published and interviews in the news.

Best,
Teresa Vieira

12.  Susan Crowell said on Sept. 1, 2011

A UT friend Alum forwarded this article to me. I am a speech pathologist who specializes in early intervention with children who have ASD. Please let us know when this may be available for viewing and use!

13.  Elman said on Sept. 1, 2011

Based upon what I read here, this game may be useful for most "normal" children too. I still often don't know what emotion I am feeling and I am over 50 years old.

14.  Cherie S said on Sept. 1, 2011

This sounds great. I have a 15 yr old Asperger son, who loves VGs. Send me updates and we will be interested in tested the Beta.

15.  Amy Crossette said on Sept. 1, 2011

Our researcher, Yan, will be testing 2 modes of the game in the upcoming months. We hope to have a Web site available soon to disseminate information and updates. If you have a child that could participate in the testing, please contact Yan at the email above. However, please be patient for a response. Yan also teaches and our fall semester has just begun.

16.  Alvaro J. Liendo R.Ph. said on Sept. 1, 2011

Please let us know when the modes will be tested, I would like to have my 8yr old grandson to be part of your beta testing.
Thank you for all you are doing!

17.  Diane said on Sept. 2, 2011

Will watch for it - I have an 18-year old who would benefit any help in expressing emotions.

18.  Ana Dultra said on Sept. 2, 2011

I have a 10 y old boy, who is pre-verbal and loves to play Mario. Now he wears a hat just like Mario all day long.
I would love to have him participating on this study:)

19.  nora cavazos said on Sept. 2, 2011

let me know when this game comes out. very interested.

20.  Teresa Vieira said on Sept. 2, 2011

LIFEisGAME Wins Best Paper Award at ACM Conference: Verónica Orvalho won best paper for her work on facial modeling for the project.
LIFEisGAME is led by a team of researchers from U.Porto and UT Austin. Verónica Orvalho is the principal investigator from Porto, while co-PIs J.K. Aggarwal and Yan Zhan are based in Austin. All told, 30 researchers are involved with the project.

21.  Pedro Castaneda Jr MD said on Sept. 4, 2011

I am a pediatrician and have several patients/kids with Austism spectrum disorder. Please send me updates and keep me posted. Thanks.

22.  Nancy Carter said on Sept. 5, 2011

Sounds exciting. I have a 12 year old with Asperger's and look forward to hearing more about the game.

23.  Beverly Zapalac said on Sept. 5, 2011

Congratulations. I believe this game will fulfill a very great need. I have a 21-year-old Asperger granddaughter and a 13-year-old high functioning autistic grandson. Would this game be appropriate for them?

24.  mellat beyene said on Sept. 9, 2011

Sounds good, I have 11 year old autistic boy , I like to see him expressing his emotions. please keep me updated thanks.

25.  JoAlice Hollfelder said on Sept. 11, 2011

Wow, this sounds interesting. My son is 24 and has Asperger's. He and I would like to participate in any studies/ testing of the app. Please keep me on your email list for updates. Thanks, JO

26.  Holly Oden McAfee said on Sept. 15, 2011

Amazing work! Best of luck moving forward. Like many other commenters, I have a child in my family who could benefit greatly from this game. If possible, please add me to a list to be notified of the release.

27.  Julie Cruickshank said on Sept. 17, 2011

hi,

like the others above I have a four year old boy who has recently been diagnosed with AS, and would be interested in any trials / studies. We live in Ireland.

28.  Cheri said on March 26, 2012

have a non verbal 29 year old daughter. would like info on when this will be available for her to use. thank you