Original Paper: https://arxiv.org/abs/2408.01262
By: Kunlun Zhu, Yifan Luo, Dingling Xu, Ruobing Wang, Shi Yu, Shuo Wang, Yukun Yan, Zhenghao Liu, Xu Han, Zhiyuan Liu, Maosong Sun
Abstract:
Retrieval-Augmented Generation (RAG) systems have demonstrated their advantages in alleviating the hallucination of Large Language Models (LLMs). Existing RAG benchmarks mainly focus on evaluating whether LLMs can correctly answer the general knowledge. However, they are unable to evaluate the effectiveness of the RAG system in dealing with the data from different vertical domains. This paper introduces RAGEval, a framework for automatically generating evaluation datasets to evaluate the knowledge usage ability of different LLMs in different scenarios. Specifically, RAGEval summarizes a schema from seed documents, applies the configurations to generate diverse documents, and constructs question-answering pairs according to both articles and configurations. We propose three novel metrics, Completeness, Hallucination, and Irrelevance, to carefully evaluate the responses generated by LLMs. By benchmarking RAG models in vertical domains, RAGEval has the ability to better evaluate the knowledge usage ability of LLMs, which avoids the confusion regarding the source of knowledge in answering question in existing QA datasets--whether it comes from parameterized memory or retrieval. The code and dataset will be released.
In the dynamic landscape of Natural Language Processing (NLP), Large Language Models (LLMs) have achieved remarkable feats. However, these models often suffer from the hallucination problem, where they generate responses with factual errors. To mitigate this, Retrieval-Augmented Generation (RAG) systems have emerged, leveraging external data to enhance the factual accuracy of responses. Yet, existing benchmarks fall short in evaluating RAG systems across diverse vertical domains. Enter RAGEval, a groundbreaking framework designed to generate scenario-specific evaluation datasets, meticulously assessing the knowledge usage capabilities of different LLMs in varied contexts.
The RAGEval framework initiates by summarizing a schema from a small set of domain-specific documents. This schema encapsulates essential domain-specific knowledge, ensuring the generated content is both professional and reliable. For instance, in the financial domain, the schema may include key elements such as organizations, events, and dates. This schema is then used to guide the generation of diverse and contextually accurate documents.