CBR-RAG: case-based reasoning for retrieval augmented generation in LLMs for legal question answering.

Wiratunga, Nirmalie; Abeyratne, Ramitha; Jayawardena, Lasal; Martin, Kyle; Massie, Stewart; Nkisi-Orji, Ikechukwu; Weerasinghe, Ruvan; Liret, Anne; Fleisch, Bruno

Abstract

Retrieval-Augmented Generation (RAG) enhances Large Language Model (LLM) output by providing prior knowledge as context to input. This is beneficial for knowledge-intensive and expert reliant tasks, including legal question-answering, which require evidence to validate generated text outputs. We highlight that Case-Based Reasoning (CBR) presents key opportunities to structure retrieval as part of the RAG process in an LLM. We introduce CBR-RAG, where CBR cycle’s initial retrieval stage, its indexing vocabulary, and similarity knowledge containers are used to enhance LLM queries with contextually relevant cases. This integration augments the original LLM query, providing a richer prompt. We present an evaluation of CBR-RAG, and examine different representations (i.e. general and domain-specific embeddings) and methods of comparison (i.e. inter, intra and hybrid similarity) on the task of legal question-answering. Our results indicate that the context provided by CBR’s case reuse enforces similarity between relevant components of the questions and the evidence base leading to significant improvements in the quality of generated answers.

Citation

WIRATUNGA, N., ABEYRATNE, R., JAYAWARDENA, L., MARTIN, K., MASSIE, S., NKISI-ORJI, I., WEERASINGHE, R., LIRET, A. and FLEISCH, B. 2024. CBR-RAG: case-based reasoning for retrieval augmented generation in LLMs for legal question answering. In Recio-Garcia, J.A., Orozco-del-Castillo, M.G. and Bridge, D (eds.) Case-based reasoning research and development: proceedings of the 32nd International conference of case-based reasoning research and development 2024 (ICCBR 2024), 1-4 July 2024, Merida, Mexico. Lecture notes in computer science, 14775. Cham: Springer [online], pages 445-460. Available from: https://doi.org/10.1007/978-3-031-63646-2_29

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 32nd International conference of case-based reasoning research 2024 (ICCBR 2024)
• Start Date: Jul 1, 2024
• End Date: Jul 4, 2024
• Acceptance Date: Apr 19, 2024
• Online Publication Date: Jun 24, 2024
• Publication Date: Dec 31, 2024
• Deposit Date: Jul 30, 2024
• Publicly Available Date: Jun 25, 2025
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Pages: 445-460
• Series Title: Lecture notes in computer science (LNCS)
• Series Number: 14775
• Series ISSN: 0302-9743; 1611-3349
• Book Title: Case-based reasoning research and development: proceedings of the 32nd International conference of case-based reasoning research 2024 (ICCBR 2024)
• ISBN: 9783031636455
• DOI: https://doi.org/10.1007/978-3-031-63646-2_29
• Keywords: CBR; RAG; LLMs; Text embedding; Indexing; Retrieval
• Public URL: https://rgu-repository.worktribe.com/output/2383595
• Additional Information: This paper received the 'Best Paper Award' at the ICCBR 2024 conference.