Design and Validation of Retrieval-Augmented Generation Techniques for a Trustworthy Conversational Assistant

Large Language Models have shown remarkable capabilities in natural language understanding and generation. However, their utilization in sensitive domains such as medicine or law raises concerns regarding reliability, transparency, and trustworthiness. In response to these challenges, Retrieval-Augmented Generation (RAG) techniques have emerged to ground generated outputs in reliable documentary source data. This work proposes extending the RAG base pipeline by integrating multiple safeguard mechanisms, including a query verification module to detect out-of-scope questions, a multi-source retrieval system with knowledge-refinement mechanisms to optimize information extraction from the knowledge base, and a chain-of-thought module to enhance the reasoning capabilities of the generator model. Across three experiments, the results demonstrate that including each safeguard mechanism improves system performance across different settings. Specifically, the proposed architecture more accurately rejects out-of-scope questions, enhancing retrieval effectiveness compared to widely used retrievers such as BM25. It also achieves improved performance on multiple-choice question answering and fact-checking tasks through its dual vector-store design and chain-of-thought module.

Palabras clave: Trustworthy AI, Large Language Models, Retrieval-Augmented Language Models, Text Generation