National Science Foundation awards CSD researchers with funding to revolutionize AI interaction for disfluent speakers

The continual enhancement of artificial intelligence and its overarching presence in various aspects of people’s day-to-day lives has impacted the way people communicate with each other through technology — but that communication is not always a seamless process.

J. Scott Yaruss and Caryn Herring of the Department of Communicative Sciences and Disorders together with PI Nihar Mahapatra, from MSU's College of Engineering, Ann Marie Ryan from MSU's Department of Psychology, and Hope Gerlach-Houck, from Western Michigan Speech, Language and Hearing Sciences are investigating the experiences of people who stutter when it comes to interacting with automatic speech recognition systems (ASRs) and voice activated artificial intelligence (AI) systems.

With generous funding from the Convergence Accelerator Grant provided by the National Science Foundation, Yaruss, Herring and their team of colleagues with support from the nonprofit organization Friends: The National Association of Young People Who Stutter will be conducting research aimed at developing an AI system that can better understand stuttered speech. The Convergence Accelerator Grant aims to fuel collaboration amongst research teams by placing them within various cohorts that deal with similar concepts in order to help create long-lasting societal impact. Yaruss and Herring’s team are a part of Track H: Enhancing opportunities for people with disabilities.

“Securing this convergence accelerator award is a particularly exciting achievement because of the nature of the program,” said Yaruss. “The convergence accelerator is designed to help scientists create products to help end-users with real-world issues, and I am eager to go through the NSF’s curriculum to improve my own research and development skills.”

Yaruss joined the Department of Communicative Science and Disorders in 2017 as a professor. He is also a practicing speech-language pathologist and a board-certified specialist in fluency disorders. Yaruss’s core focus throughout his research and teaching practices is to further the understanding of stuttering and related communication issues, with the ultimate goal of minimizing the adverse impact that stuttering causes on people’s lives.

Herring is the chair of the board of directors for Friends: The National Association of Young People Who Stutter, a non-profit support organization dedicated to empowering young people who stutter and their families. She is also doctoral candidate within the Department of Communicative Science and Disorders and is the president of the MSU student support organization, Spartan Stuttering Group. Herring aims to reduce the adverse cognitive and affecting components of stuttering in her research. Both Yaruss and Herring are researchers at the Spartan Stuttering Lab.

The team’s research project is titled, “Convergent, Human-Centered Design for Making Voice-Activated AI Accessible and Fair to People Who Stutter.” They seek to discover how ASRs and AI systems are unable to understand speech that contains speech disfluencies or stuttering and why this can lead to potential discrimination and serious disadvantages for people with speech differences. This kind of technology includes personal assistants like Siri or Alexa, phone messaging systems, and AI-based job interview processors like Google’s Interview Warmup.

Herring expresses a personal connection to the study and how excited she is to make improvements to a system that has caused issues throughout her life.

“I feel so incredibly fortunate. As a person who stutters, using ASRs is often a frustrating experience. I know all too well how demoralizing it feels to be on hold with customer service for an hour just to be hung up on when ASRs don’t understand me. So, it's exciting to be a member of the team and play a role in how ASRs can be made more inclusive for people who stutter.”

In order to develop a new AI system that can understand speech differences, the team will have to first conduct surveys with people who stutter to learn how these systems have affected their lives and the possible solutions that can be implemented to improve these interactions. They will then speak with the software and hardware engineers who develop these kinds of AI systems, in addition to hiring personnel and human resource managers, in order to learn how they make employment decisions based on the judgements of AI.

Finally, they will conduct data collection by acquiring speech samples from people who stutter so engineers on the team can use that data to prepare an accessible and fair AI system.

“This will lead to the development of a set of products that can be provided to various end-users (e.g., tech companies that use AI-based ASRs, people who stutter) to overcome the burdens identified through the project,” said Yaruss.

Both Yaruss and Herring are hopeful of the positive impact this study will have on so many people’s lives.

“I am hoping to give people who stutter equal access to ASRs,” said Herring. “It also seems like a great opportunity to educate more people about stuttering.”

“My goal in my work has always been to reduce the burdens that people who stutter experience in their lives, and with the growing use of AI systems for understanding speech, this issue has become even more important,” said Yaruss.

By Casey Halas