Why explanations matter

Artificial Intelligence is increasingly used to support decisions in high-risk domains such as healthcare. In clinical settings, AI systems can help analyse patient data, identify patterns, and support diagnosis. However, a prediction alone is not enough: clinicians also need to understand why a system reached a certain conclusion.

This is the goal of Explainable AI, or XAI. XAI methods help make machine learning models more transparent by showing which factors influenced a prediction. For example, in a diabetes prediction task, an explanation may highlight the role of glucose level, body mass index, age, or blood pressure.

From technical outputs to human-centred explanations

Although XAI methods are useful, their outputs are often technical and difficult to interpret for non-computer-science experts. A list of feature importance scores may be informative, but it does not always provide the clinical context needed to support decision-making.

Large Language Models can help by transforming technical explanations into natural language. They can make explanations easier to read, more intuitive, and closer to the way clinicians reason. However, they also introduce a major risk: they may generate fluent but incorrect information.

Introducing TRIAGE

To address this challenge, we developed TRIAGE: Trustworthy Retrieval-based Interpretability with Abstention for Grounded Explanations in Clinical Decision Support System.

TRIAGE combines predictive models, XAI methods, and Retrieval-Augmented Generation. Instead of relying only on the internal knowledge of a language model, the system retrieves information from trusted medical sources and uses it to produce grounded explanations. This helps reduce hallucinations and makes the explanation more reliable and traceable.

How TRIAGE works

TRIAGE follows a human-centred pipeline. First, a predictive model analyses patient data and produces a classification. Then, a XAI method identifies the most relevant features behind the prediction. Finally, a retrieval-based enhancer reformulates the explanation in plain language and enriches it with medical context.

The system is also designed to support interaction. Clinicians can ask follow-up questions, request clarifications, and explore the relationship between the patient’s profile, the model’s prediction, and the generated explanation.

Knowing when not to answer

A key feature of TRIAGE is abstention. In healthcare, an AI system should not always be forced to provide an explanation. When the available information is inconsistent or uncertain, it may be safer not to present a potentially misleading answer.

For this reason, TRIAGE includes a governance layer that checks the consistency between different parts of the system. If the predictive model and the retrieval-based component disagree, the system can abstain from showing an explanation as fully reliable.

A step toward trustworthy clinical AI

Preliminary experiments on a diabetes prediction task show the value of combining traditional machine learning with retrieval-based explanations. The predictive model remains central for classification, while the retrieval-based component helps make explanations clearer, more contextualised, and easier to inspect.

This approach reflects the broader mission of TANGO: developing AI systems that are transparent, responsible, and aligned with human needs. In healthcare, this means creating tools that support clinicians, preserve human oversight, and help build trust through meaningful interaction.

Reference

Mannocci, L., Naretto, F., Passaro, L., & Monreale, A. (2025, October). RAG-Enhanced LLMs for Interactive Explainability in Clinical Decision Support Systems. In International Joint Workshop on Artificial Intelligence for Healthcare, and Hybrid Models for Coupling Deductive and Inductive Reasoning (pp. 50-63). Cham: Springer Nature Switzerland.

Written by: Lorenzo Mannocci, Postdoctoral Researcher at University of Pisa (UNIPI)