Original Paper: https://arxiv.org/abs/2406.08414
By: Chris Lu, Samuel Holt, Claudio Fanconi, Alex J. Chan, Jakob Foerster, Mihaela van der Schaar, Robert Tjarko Lange
Abstract:
Offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. Typically, preference optimization is approached as an offline supervised learning task using manually-crafted convex loss functions. While these methods are based on theoretical insights, they are inherently constrained by human creativity, so the large search space of possible loss functions remains under explored. We address this by performing LLM-driven objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Specifically, we iteratively prompt an LLM to propose and implement new preference optimization loss functions based on previously-evaluated performance metrics. This process leads to the discovery of previously-unknown and performant preference optimization algorithms. The best performing of these we call Discovered Preference Optimization (DiscoPOP), a novel algorithm that adaptively blends logistic and exponential losses. Experiments demonstrate the state-of-the-art performance of DiscoPOP and its successful transfer to held-out tasks.
In the realm of machine learning, optimizing the performance of large language models (LLMs) to align with human preferences is a critical task. As LLMs become increasingly integrated into applications ranging from chatbots to content generation tools, ensuring that these models produce high-quality, ethical, and value-aligned outputs is paramount. Traditionally, preference optimization relies on manually-crafted loss functions designed by human experts, but this approach is inherently limited by human creativity and expertise. A recent study proposes a groundbreaking method to automate the discovery of new preference optimization algorithms using LLMs themselves.
Training LLMs typically involves fine-tuning pre-trained models on large text corpora to better align with human preferences. However, even instruction-fine-tuned LLMs can generate harmful or unethical outputs. To mitigate this, techniques like reinforcement learning with human feedback (RLHF) have been employed, and more recently, offline preference optimization algorithms have been developed. These algorithms cast the problem as a supervised learning objective, aiming to enhance model outputs based on human values without extensive real-time human intervention.
The novel approach introduced in the study involves using LLMs to autonomously discover new preference optimization algorithms. This is achieved through an iterative process where an LLM proposes and evaluates new loss functions based on previously assessed performance metrics.