Evaluation of Oncotimia: An LLM based system for supporting tumour boards
Summary
ONCOTIMIA is a novel GenAI-powered clinical tool designed to automate the completion of lung cancer tumour board forms, addressing the significant documentation burden in oncology. Leveraging a multi-layer data lake, RAG, and adaptive forms, the system transformed unstructured clinical data into standardized records. The evaluation demonstrated high performance, achieving 80% correct field completion with clinically acceptable latency, proving its technical feasibility and operational viability for multidisciplinary lung cancer workflows.
Medical Relevance
This research is highly relevant as it directly tackles a critical workflow inefficiency in oncology: the laborious and time-consuming manual documentation process in tumour boards. By automating this, it can free up valuable clinician time, enhance data standardization, potentially reduce human errors, and accelerate access to structured patient information, thereby improving the efficiency and quality of clinical decision-making in cancer care.
AI Health Application
The AI application is an LLM-based system (ONCOTIMIA) that uses generative AI, retrieval-augmented generation (RAG), and a rule-driven adaptive form model to automatically complete structured tumour board records from unstructured clinical documentation. Its primary goal is to support oncology decision-making by reducing the documentation burden for multidisciplinary tumour boards, specifically for lung cancer cases, while maintaining data quality.
Key Points
- Addresses the substantial documentation burden and manual data structuring challenges in Multidisciplinary Tumour Boards (MDTBs) through automation.
- Introduces ONCOTIMIA, a modular, secure clinical tool integrating Generative AI (GenAI) for automatic completion of lung cancer tumour board forms.
- System architecture comprises a multi-layer data lake, hybrid relational and vector storage, Retrieval-Augmented Generation (RAG), and a rule-driven adaptive form model.
- Evaluated the performance of six different Large Language Models (LLMs) deployed via AWS Bedrock on ten real lung cancer cases.
- Performance metrics included both completion form accuracy and end-to-end latency.
- Achieved a high accuracy of 80% for correct field completion with the best performing configuration.
- Demonstrated clinically acceptable response times for most LLMs, indicating operational viability.
- Found that larger and more recent LLMs consistently delivered the best accuracies without incurring prohibitive latency, suggesting optimal performance gains with advanced models.
Methodology
The ONCOTIMIA system integrates a multi-layer data lake for raw clinical data, hybrid relational and vector storage for efficient indexing and retrieval, and Retrieval-Augmented Generation (RAG) to provide LLMs with context from patient records. A rule-driven adaptive form model standardizes extracted information. Evaluation involved assessing six distinct LLMs (via AWS Bedrock) on ten real-world lung cancer cases. Performance was quantified by measuring the accuracy of automatically completed form fields and the end-to-end latency of the system.
Key Findings
The study found that ONCOTIMIA achieved high performance, with the best performing LLM configuration successfully completing 80% of form fields correctly. End-to-end latency was clinically acceptable for the majority of LLMs tested. Crucially, larger and more recent LLM architectures consistently yielded superior accuracies without introducing prohibitive increases in response time, indicating that advanced models are practically beneficial for this application.
Clinical Impact
The successful implementation and evaluation of ONCOTIMIA demonstrate its potential to significantly alleviate the substantial documentation burden on clinicians involved in multidisciplinary tumour boards, particularly in lung cancer. By automating the structuring of heterogeneous clinical information, it can enhance workflow efficiency, reduce administrative overhead, and ensure greater standardization and data quality, allowing oncology teams to dedicate more time to complex clinical discussions and patient care decisions.
Limitations
The abstract does not explicitly state any limitations of the study.
Future Directions
The abstract does not explicitly state any future research directions.
Medical Domains
Keywords
Abstract
Multidisciplinary tumour boards (MDTBs) play a central role in oncology decision-making but require manual processes and structuring large volumes of heterogeneous clinical information, resulting in a substantial documentation burden. In this work, we present ONCOTIMIA, a modular and secure clinical tool designed to integrate generative artificial intelligence (GenAI) into oncology workflows and evaluate its application to the automatic completion of lung cancer tumour board forms using large language models (LLMs). The system combines a multi-layer data lake, hybrid relational and vector storage, retrieval-augmented generation (RAG) and a rule-driven adaptive form model to transform unstructured clinical documentation into structured and standardised tumour board records. We assess the performance of six LLMs deployed through AWS Bedrock on ten lung cancer cases, measuring both completion form accuracy and end-to-end latency. The results demonstrate high performance across models, with the best performing configuration achieving an 80% of correct field completion and clinically acceptable response time for most LLMs. Larger and more recent models exhibit best accuracies without incurring prohibitive latency. These findings provide empirical evidence that LLM- assisted autocompletion form is technically feasible and operationally viable in multidisciplinary lung cancer workflows and support its potential to significantly reduce documentation burden while preserving data quality.
Comments
9 pages, 2 figures