Original Paper: https://arxiv.org/abs/2406.19292
By: Zheyang Xiong, Vasilis Papageorgiou, Kangwook Lee, Dimitris Papailiopoulos
Abstract:
Recent studies have shown that Large Language Models (LLMs) struggle to accurately retrieve information and maintain reasoning capabilities when processing long-context inputs. To address these limitations, we propose a finetuning approach utilizing a carefully designed synthetic dataset comprising numerical key-value retrieval tasks. Our experiments on models like GPT-3.5 Turbo and Mistral 7B demonstrate that finetuning LLMs on this dataset significantly improves LLMs' information retrieval and reasoning capabilities in longer-context settings. We present an analysis of the finetuned models, illustrating the transfer of skills from synthetic to real task evaluations (e.g., 10.5% improvement on 20 documents MDQA at position 10 for GPT-3.5 Turbo). We also find that finetuned LLMs' performance on general benchmarks remains almost constant while LLMs finetuned on other baseline long-context augmentation data can encourage hallucination (e.g., on TriviaQA, Mistral 7B finetuned on our synthetic data cause no performance drop while other baseline data can cause a drop that ranges from 2.33% to 6.19%). Our study highlights the potential of finetuning on synthetic data for improving the performance of LLMs on longer-context tasks.
From Artificial Needles to Real Haystacks: Enhancing LLMs' Retrieval with Synthetic Data
Introduction
In the ever-expanding universe of Large Language Models (LLMs), the quest for precise information retrieval and robust reasoning capabilities amidst vast text contexts is a formidable challenge. These capabilities are pivotal for tasks such as summarization and answering questions over long passages. Traditional approaches have struggled to maintain accuracy when dealing with extensive inputs. However, a groundbreaking study from the University of Wisconsin-Madison presents a novel solution: finetuning LLMs on synthetic data to enhance their long-context capabilities.
Synthetic Data as a Finetuning Tool
The core of this research revolves around a synthetic dataset meticulously designed for key-value dictionary retrieval tasks. This dataset is devoid of factual content, significantly reducing the risk of hallucinations—a common issue when models are finetuned on real-world data. The researchers focused on two primary tasks: