MedForget: Hierarchy-Aware Multimodal Unlearning Testbed for Medical AI

Summary

MedForget is a novel hierarchy-aware multimodal unlearning testbed designed for medical AI, specifically addressing critical privacy challenges like the "right to be forgotten" for sensitive patient data under regulations such as HIPAA and GDPR. The research demonstrates that current state-of-the-art unlearning methods struggle to achieve complete, fine-grained hierarchical forgetting without compromising diagnostic performance, and can leave models vulnerable to reconstruction attacks via contextual cues.

Medical Relevance

This research is critical for developing HIPAA and GDPR-compliant medical AI systems by providing a framework to rigorously test and improve unlearning capabilities for sensitive patient data. It ensures the ethical and legal deployment of MLLMs in clinical settings for tasks such as diagnosis support and report generation, safeguarding patient privacy.

AI Health Application

The AI application is the development and ethical deployment of privacy-preserving medical AI systems. Specifically, it enables the study and improvement of machine unlearning techniques for Multimodal Large Language Models (MLLMs) used in clinical reasoning, diagnosis support, and report generation, ensuring compliance with patient data privacy regulations like HIPAA and GDPR while maintaining diagnostic performance.

Key Points

• **Problem Addressed:** MLLMs in medical AI face significant privacy and compliance issues (HIPAA, GDPR) due to training on sensitive patient data, necessitating robust unlearning capabilities to enforce the "right to be forgotten."
• **Novel Testbed:** Introduction of MedForget, a Hierarchy-Aware Multimodal Unlearning Testbed with explicit retain/forget data splits and evaluation sets containing rephrased variants to thoroughly assess forgetting.
• **Hierarchical Data Modeling:** MedForget models hospital data as a nested hierarchy (Institution -> Patient -> Study -> Section), enabling fine-grained assessment of unlearning effectiveness across eight distinct organizational levels.
• **Benchmark Composition:** The testbed comprises 3840 multimodal instances (image, question, answer), with each hierarchy level having a dedicated unlearning target to reflect diverse unlearning challenges.
• **SOTA Unlearning Limitations:** Experiments using four state-of-the-art unlearning methods on three tasks (generation, classification, cloze) reveal that existing techniques struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance.
• **Reconstruction Attack:** A novel reconstruction attack is introduced, which progressively adds hierarchical level context to prompts to test the true deletion of forgotten information.
• **Vulnerability of Fine-Grained Unlearning:** The reconstruction attack showed that models unlearned at a coarse granularity exhibit strong resistance, while fine-grained unlearning leaves models vulnerable, indicating incomplete erasure of hierarchical pathways and potential for information recovery.

Clinical Impact

MedForget serves as an essential, HIPAA-aligned testbed for advancing the development of privacy-preserving medical AI systems. It provides the tools to rigorously evaluate unlearning techniques, paving the way for trustworthy MLLMs that respect patient rights under strict regulations. The findings highlight the urgent need for more robust unlearning algorithms to enable the safe and compliant deployment of AI in sensitive clinical environments.

Limitations

The abstract explicitly states that existing state-of-the-art unlearning methods struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance. Additionally, a significant limitation identified is that fine-grained unlearning leaves models vulnerable to reconstruction attacks, indicating current methods are not robust enough for complete data deletion at granular levels.

Future Directions

While not explicitly stated as 'future directions,' the paper implicitly calls for the development of new, more effective unlearning methods that can achieve complete, fine-grained, and robust hierarchical forgetting without performance degradation. MedForget itself is presented as a practical testbed to guide future research in building truly compliant medical AI systems.

Medical Domains

Keywords

Abstract

Pretrained Multimodal Large Language Models (MLLMs) are increasingly deployed in medical AI systems for clinical reasoning, diagnosis support, and report generation. However, their training on sensitive patient data raises critical privacy and compliance challenges under regulations such as HIPAA and GDPR, which enforce the "right to be forgotten". Unlearning, the process of tuning models to selectively remove the influence of specific training data points, offers a potential solution, yet its effectiveness in complex medical settings remains underexplored. To systematically study this, we introduce MedForget, a Hierarchy-Aware Multimodal Unlearning Testbed with explicit retain and forget splits and evaluation sets containing rephrased variants. MedForget models hospital data as a nested hierarchy (Institution -> Patient -> Study -> Section), enabling fine-grained assessment across eight organizational levels. The benchmark contains 3840 multimodal (image, question, answer) instances, each hierarchy level having a dedicated unlearning target, reflecting distinct unlearning challenges. Experiments with four SOTA unlearning methods on three tasks (generation, classification, cloze) show that existing methods struggle to achieve complete, hierarchy-aware forgetting without reducing diagnostic performance. To test whether unlearning truly deletes hierarchical pathways, we introduce a reconstruction attack that progressively adds hierarchical level context to prompts. Models unlearned at a coarse granularity show strong resistance, while fine-grained unlearning leaves models vulnerable to such reconstruction. MedForget provides a practical, HIPAA-aligned testbed for building compliant medical AI systems.

Comments

Dataset and Code: https://github.com/fengli-wu/MedForget