Context driven multi-query resolution using LLM-RAG to support the revision of explainability needs.

Jayawardena, Lasal; Liret, Anne; Wiratunga, Nirmalie; Nkisi-Orji, Ikechukwu; Fleisch, Bruno

Abstract

The revision step in the Case-Based Reasoning (CBR) cycle ensures that cases are adaptable and that updates can be integrated meaningfully based on evaluation metrics. However, the effectiveness of this step heavily depends on how new knowledge is acquired to support revision. In the iSee project, where CBR is used for explanation strategy recommendations, revision knowledge is typically derived from end-user feedback following an interactive, user-centric explanation experience. This raises the research question: how can we discover this knowledge from user interactions, and how can Large Language Models (LLMs) with Retrieval-Augmented Generation (RAG) enhance the explanation experience while capturing useful knowledge for the discovery? In this paper, we propose a methodology for detecting the evolution of user explanation intent (an indicator for the need for revision) through LLM-RAG enhanced interactions. Finally, experimental results evaluate the success of the proposed methodology, assessing its validity and, in particular, the reasonability of the LLM-RAG approach. Experimental results across multiple real-world use cases demonstrate that our methodology produces highly coherent and context-aware explanations, improving overall explanation clarity, and effectively identifies when explanation strategies require revision.

Citation

JAYAWARDENA, L., LIRET, A., WIRATUNGA, N., NKISI-ORJI, I. and FLEISCH, B. [2025]. Context driven multi-query resolution using LLM-RAG to support the revision of explainability needs. In Proceedings of the 33rd International conference on case-based reasoning (ICCBR 2025), 30 June - 3 July 2025, Biarritz, France. Lecture notes in computer science, [volume to be confirmed]. Cham: Springer [online], (accepted).

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 33rd International conference on case-based reasoning (ICCBR 2025)
• Start Date: Jun 30, 2025
• End Date: Jul 3, 2025
• Acceptance Date: Apr 11, 2025
• Deposit Date: Apr 16, 2025
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Series Title: Lecture notes in computer science
• Series ISSN: 0302-9743; 1611-3349
• Keywords: Interactive explanations; Large language models (LLMs); Reasonability; Explainability intent; Explanation experience; Case-based reasoning (CBR)
• Public URL: https://rgu-repository.worktribe.com/output/2795345