Original Paper: https://arxiv.org/abs/2407.21712
By: Xi Wang, Procheta Sen, Ruizhe Li, Emine Yilmaz
Abstract:
Despite the success of integrating large language models into the development of conversational systems, many studies have shown the effectiveness of retrieving and augmenting external knowledge for informative responses. Hence, many existing studies commonly assume the always need for Retrieval Augmented Generation (RAG) in a conversational system without explicit control. This raises a research question about such a necessity. In this study, we propose to investigate the need for each turn of system response to be augmented with external knowledge. In particular, by leveraging human judgements on the binary choice of adaptive augmentation, we develop RAGate, a gating model, which models conversation context and relevant inputs to predict if a conversational system requires RAG for improved responses. We conduct extensive experiments on devising and applying RAGate to conversational models and well-rounded analyses of different conversational scenarios. Our experimental results and analysis indicate the effective application of RAGate in RAG-based conversational systems in identifying system responses for appropriate RAG with high-quality responses and a high generation confidence. This study also identifies the correlation between the generation's confidence level and the relevance of the augmented knowledge.
Adaptive Retrieval-Augmented Generation for Conversational Systems: A Smarter Approach
Introduction
In the realm of conversational AI, the integration of Large Language Models (LLMs) has led to impressive advancements. These models can generate natural, high-quality responses, transforming the way we interact with machines. However, recent studies have highlighted several limitations of LLMs, such as generating outdated or hallucinated content. This brings us to a critical question: Is it always necessary to augment every conversational turn with external knowledge? In a recent study, researchers tackled this question by proposing an adaptive approach for retrieval-augmented generation (RAG) in conversational systems. Let's dive into the methodology, findings, and implications of their research.
Key Methodologies
To address the necessity of knowledge augmentation for each conversational turn, the researchers developed a gating model named RAGate. This model leverages human judgments to predict whether a system response requires external knowledge for improvement. The study involved several steps: