Original Paper: https://arxiv.org/abs/2407.19825
By: Sania Nayab, Giulio Rossolini, Giorgio Buttazzo, Nicolamaria Manes, Fabrizio Giacomelli
Abstract:
Today's large language models (LLMs) can solve challenging question-answering tasks, and prompt engineering techniques, such as chain-of-thought (CoT), have gained attention for enhancing the explanation and correctness of outputs. Nevertheless, models require significant time to generate answers augmented with lengthy reasoning details. To address this issue, this paper analyzes the impact of output lengths on LLM inference pipelines and proposes novel metrics to evaluate them in terms of \textit{correct conciseness}. It also examines the impact of controlling output length through a refined prompt engineering strategy, Constrained-CoT (CCoT), which encourages the model to limit output length. Experiments on pre-trained LLMs demonstrated the benefit of the proposed metrics and the effectiveness of CCoT across different models. For instance, constraining the reasoning of LLaMA2-70b to 100 words improves the accuracy from 36.01\% (CoT) to 41.07\% (CCoT) on the GSM8K dataset, while reducing the average output length by 28 words.
In the world of Large Language Models (LLMs), such as GPT-3 and LLaMA, efficiency and accuracy are paramount. However, techniques that improve the correctness of responses often come with trade-offs, particularly in terms of output length and inference time. This blog post explores a fascinating research study that delves into these trade-offs and introduces a novel method to balance them: Constrained Chain-of-Thought (CCoT) prompting.
Large Language Models (LLMs) have demonstrated remarkable capabilities in handling complex question-answering tasks, thanks to advanced techniques like Chain-of-Thought (CoT) prompting. CoT enhances the explanation and correctness of outputs by encouraging the model to articulate its reasoning step-by-step. However, this also leads to longer outputs, significantly increasing the time required for the model to generate a response. This delay is particularly undesirable in interactive applications where response time is critical.
The research in focus analyzes the impact of output lengths on LLM inference pipelines and proposes novel metrics to evaluate their performance in terms of conciseness and correctness. Additionally, it introduces Constrained Chain-of-Thought (CCoT) prompting, a refined prompt engineering strategy designed to limit output length without sacrificing accuracy.