Empathic Prompting: Non-Verbal Context Integration for Multimodal LLM Conversations

Summary

Empathic Prompting is a novel multimodal framework that enriches Large Language Model (LLM) conversations by unobtrusively integrating implicit non-verbal emotional cues from users. It utilizes a facial expression recognition service to embed affective information as contextual signals during prompting, aiming to improve conversational smoothness and alignment. A preliminary evaluation demonstrated consistent integration of this non-verbal input into coherent LLM outputs, enhancing conversational fluidity.

Medical Relevance

This framework is highly relevant for medicine as it enables AI systems to understand and respond to the critical emotional signals of patients, which are often subtle and non-verbal. This capability can significantly enhance patient-AI interactions in domains like mental health support, telemedicine, and patient education, where emotional context is paramount for effective care and communication.

AI Health Application

Empathic Prompting can be applied in healthcare to create more empathetic and effective AI-powered communication tools. For instance, it could enable virtual health assistants or chatbots to better understand a patient's emotional state during a consultation (e.g., distress, anxiety, confusion) by analyzing facial expressions. This could lead to more nuanced responses, improved patient support in mental health contexts, enhanced patient engagement in chronic disease management, or more sensitive communication in telemedicine settings where non-verbal cues are often lost.

Key Points

• **Novel Framework**: Introduces 'Empathic Prompting' for multimodal human-AI interaction, augmenting LLM conversations with implicit non-verbal context.
• **Non-Verbal Context Integration**: Captures users' emotional cues via a commercial facial expression recognition service and embeds them as contextual signals within LLM prompts.
• **Unobtrusive Operation**: The system requires no explicit user control, seamlessly integrating affective information to enhance conversational flow and alignment.
• **Modular and Scalable Architecture**: Designed with a modular architecture, enabling future integration of additional non-verbal input modalities beyond facial expressions.
• **Implementation & Preliminary Evaluation**: Implemented with a locally deployed DeepSeek LLM instance and evaluated through a preliminary service and usability study (N=5).
• **Key Finding**: Demonstrated consistent integration of non-verbal input into coherent LLM outputs, with participants reporting improved conversational fluidity.
• **High Relevance for Sensitive Domains**: Highlights significant potential for applications in chatbot-mediated communication, particularly in healthcare and education, where emotional signals are critical yet often implicit.

Clinical Impact

This technology holds potential to transform patient-facing AI applications by adding an empathetic dimension. In clinical settings, it could lead to more nuanced and effective therapeutic chatbots for mental health, improved patient engagement in chronic disease management via emotionally intelligent virtual assistants, and better detection of patient distress or non-compliance in telemedicine consultations. The ability to implicitly gauge emotional states could also aid in triaging or tailoring information delivery in patient education platforms.

Limitations

The primary limitation noted is the preliminary nature of the evaluation, conducted with a very small sample size (N=5). This limits the generalizability of the findings and positions the work as a 'proof of concept' rather than a robust, large-scale validation.

Future Directions

Suggested future directions include scaling the evaluation to a larger and more diverse user population to validate findings more broadly. Additionally, the modular architecture invites integration of further non-verbal modules (e.g., voice intonation, body language) to create an even richer contextual understanding for LLM conversations.

Medical Domains

Keywords

Abstract

We present Empathic Prompting, a novel framework for multimodal human-AI interaction that enriches Large Language Model (LLM) conversations with implicit non-verbal context. The system integrates a commercial facial expression recognition service to capture users' emotional cues and embeds them as contextual signals during prompting. Unlike traditional multimodal interfaces, empathic prompting requires no explicit user control; instead, it unobtrusively augments textual input with affective information for conversational and smoothness alignment. The architecture is modular and scalable, allowing integration of additional non-verbal modules. We describe the system design, implemented through a locally deployed DeepSeek instance, and report a preliminary service and usability evaluation (N=5). Results show consistent integration of non-verbal input into coherent LLM outputs, with participants highlighting conversational fluidity. Beyond this proof of concept, empathic prompting points to applications in chatbot-mediated communication, particularly in domains like healthcare or education, where users' emotional signals are critical yet often opaque in verbal exchanges.