Original Paper: https://arxiv.org/abs/2408.02666
By: Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, Xian Li
Abstract:
Model-based evaluation is at the heart of successful model development -- as a reward model for training, and as a replacement for human evaluation. To train such evaluators, the standard approach is to collect a large amount of human preference judgments over model responses, which is costly and the data becomes stale as models improve. In this work, we present an approach that aims to im-prove evaluators without human annotations, using synthetic training data only. Starting from unlabeled instructions, our iterative self-improvement scheme generates contrasting model outputs and trains an LLM-as-a-Judge to produce reasoning traces and final judgments, repeating this training at each new iteration using the improved predictions. Without any labeled preference data, our Self-Taught Evaluator can improve a strong LLM (Llama3-70B-Instruct) from 75.4 to 88.3 (88.7 with majority vote) on RewardBench. This outperforms commonly used LLM judges such as GPT-4 and matches the performance of the top-performing reward models trained with labeled examples.
In the fast-paced world of AI and machine learning, evaluating the performance of large language models (LLMs) is crucial for their development and refinement. Traditional methods rely heavily on human-generated data, which is both costly and time-consuming. But what if we could bypass this bottleneck? Enter the world of Self-Taught Evaluators – a novel approach that uses synthetic data to train evaluators without human annotations. This method not only promises to cut down costs but also aims to keep up with the rapid advancements in LLMs.
LLMs have made remarkable strides in recent years, but their evaluation remains a significant hurdle. The standard approach involves collecting human preference judgments over model responses, which becomes outdated as models improve. Moreover, this dependency on human-generated data poses scaling challenges for new tasks or evaluation criteria.