Original Paper: https://arxiv.org/abs/2406.17744
By: Weizhe Yuan, Ilia Kulikov, Ping Yu, Kyunghyun Cho, Sainbayar Sukhbaatar, Jason Weston, Jing Xu
Abstract:
Aligned instruction following models can better fulfill user requests than their unaligned counterparts. However, it has been shown that there is a length bias in evaluation of such models, and that training algorithms tend to exploit this bias by learning longer responses. In this work we show how to train models that can be controlled at inference time with instructions containing desired length constraints. Such models are superior in length instructed evaluations, outperforming standard instruction following models such as GPT4, Llama 3 and Mixtral.
In the realm of artificial intelligence, instruction-following models are pivotal in delivering precise and user-centric responses. However, a persistent challenge has been the models' tendency to produce length-biased responses, often favoring longer and more verbose answers. This article delves into a novel approach, dubbed Length-Instruction Fine-Tuning (LIFT), which aims to address this issue by training models to adhere strictly to length constraints during inference.
Instruction-following models, such as GPT-4 and Llama, are designed to respond to user prompts effectively. However, these models often exhibit a "length bias," where both human evaluators and the models themselves tend to prefer longer responses. This bias not only skews evaluations but also complicates training algorithms that rely on these evaluations for optimization. A simple instruction like "Give me information about Coco Gauff" can be answered in a few sentences, paragraphs, or even a multi-page document, depending on the context. This ambiguity in expected response length poses a significant challenge in both training and evaluation.
To tackle this challenge, the researchers introduced the concept of Length-Instruction Fine-Tuning (LIFT). The methodology involves: