ImageTalk: Designing a Multimodal AAC Text Generation System Driven by Image Recognition and Natural Language Generation

arXiv ID: 2512.09610v1

Published: 2025-12-10

Authors: Boyin Yang, Puming Jiang, Per Ola Kristensson

Categories: cs.HC, cs.AI, cs.CV

Relevance Score: 1.00 / 1.00

View on arXiv Download PDF

Summary

This paper introduces ImageTalk, a novel multimodal Augmentative and Alternative Communication (AAC) system designed for people living with Motor Neuron Disease (plwMND) to overcome the limited vocabulary of traditional symbol-based AACs and the low communication rates of text entry solutions. By integrating image recognition and natural language generation, ImageTalk significantly enhances communication efficiency, demonstrating 95.6% keystroke savings, consistent performance, and high user satisfaction.

Medical Relevance

This research is highly relevant to medicine as it directly addresses a critical need for effective communication tools for plwMND, a population severely impacted by speech and motor impairments. Improving AAC systems can significantly enhance quality of life, patient autonomy, and the ability to articulate medical needs and preferences.

AI Health Application

The paper describes an AI-assisted text generation system leveraging image recognition and natural language generation. This AI is applied in a medical context to create a multimodal augmentative and alternative communication (AAC) system specifically designed for people with Motor Neuron Disease, thereby serving as a medical AI application to enhance patient communication and quality of life.

Key Points

Addresses critical communication challenges (limited vocabulary, low communication rates) for people living with Motor Neuron Disease (plwMND).
Presents ImageTalk, a novel multimodal text generation system driven by image recognition and Natural Language Generation (NLG).
System development involved an iterative design process, including tailored proxy-user-based and end-user-based phases.
Demonstrated pronounced keystroke savings of 95.6%, indicating a significant improvement in communication efficiency.
Achieved consistent performance and high user satisfaction among participants.
Distilled three specific design guidelines for the development of AI-assisted text generation systems.
Outlined four distinct user requirement levels tailored for AAC purposes, intended to guide future research and development in the field.

Methodology

The ImageTalk system was designed and developed through an iterative process that involved a tailored proxy-user-based design phase followed by an end-user-based design phase. This user-centered approach aimed to efficiently and effectively articulate the needs of plwMND within the system's development.

Key Findings

The ImageTalk system achieved a pronounced keystroke saving of 95.6%, demonstrating a significant increase in communication efficiency. Users reported high satisfaction with the system, which also exhibited consistent performance. Beyond the system's performance, the research yielded three design guidelines for AI-assisted text generation systems and defined four user requirement levels specifically for AAC purposes.

Clinical Impact

ImageTalk has the potential to significantly improve the daily lives of plwMND by providing a more efficient and effective means of communication than currently available AAC options. This can lead to better patient-provider interactions, enhanced participation in decision-making, reduced frustration, and improved mental well-being for individuals experiencing severe speech impairments.

Limitations

The abstract does not explicitly state any limitations of the ImageTalk system or the study conducted.

Future Directions

The paper suggests that the distilled three design guidelines for AI-assisted text generation systems and the outlined four user requirement levels tailored for AAC purposes will serve as foundational knowledge to guide future research and development in this field, implying ongoing refinement and expansion of such systems.

Medical Domains

Neurology Speech-Language Pathology Rehabilitation Medicine Assistive Technology

Keywords

Motor Neuron Disease Augmentative and Alternative Communication Multimodal Communication Image Recognition Natural Language Generation Assistive Technology Speech Impairment Human-Computer Interaction

Abstract

People living with Motor Neuron Disease (plwMND) frequently encounter speech and motor impairments that necessitate a reliance on augmentative and alternative communication (AAC) systems. This paper tackles the main challenge that traditional symbol-based AAC systems offer a limited vocabulary, while text entry solutions tend to exhibit low communication rates. To help plwMND articulate their needs about the system efficiently and effectively, we iteratively design and develop a novel multimodal text generation system called ImageTalk through a tailored proxy-user-based and an end-user-based design phase. The system demonstrates pronounced keystroke savings of 95.6%, coupled with consistent performance and high user satisfaction. We distill three design guidelines for AI-assisted text generation systems design and outline four user requirement levels tailored for AAC purposes, guiding future research in this field.

Comments

24 pages, 10 figures