Original Paper: https://arxiv.org/abs/2406.12430

By: Myeonghwa Lee, Seonho An, Min-Soo Kim

Abstract:

In this paper, we conduct a study to utilize LLMs as a solution for decision making that requires complex data analysis. We define Decision QA as the task of answering the best decision, dbest, for a decision-making question Q, business rules R and a database D. Since there is no benchmark that can examine Decision QA, we propose Decision QA benchmark, DQA. It has two scenarios, Locating and Building, constructed from two video games (Europa Universalis IV and Victoria 3) that have almost the same goal as Decision QA. To address Decision QA effectively, we also propose a new RAG technique called the iterative plan-then-retrieval augmented generation (PlanRAG). Our PlanRAG-based LM generates the plan for decision making as the first step, and the retriever generates the queries for data analysis as the second step. The proposed method outperforms the state-of-the-art iterative RAG method by 15.8% in the Locating scenario and by 7.4% in the Building scenario, respectively. We release our code and benchmark at this https URL.

Summary Notes

Unleashing the Power of PlanRAG: Revolutionizing Decision Making with Large Language Models

Introduction

In an era where data drives business decisions, the role of decision-making systems is more crucial than ever. Traditional decision support systems have evolved, but they often fall short when it comes to handling complex decisions requiring extensive data analysis. Enter Large Language Models (LLMs) and a novel approach called PlanRAG (Plan-then-Retrieval Augmented Generation). This innovative technique promises to transform LLMs into proficient decision-makers, capable of tackling even the most intricate business scenarios. Let’s delve into how PlanRAG works and its potential impact on decision-making tasks.

The Research Question

The main question this research addresses is: Can LLMs be effectively utilized for decision-making tasks that require complex data analysis? The study introduces "Decision QA" (Decision Question Answering), a task where the goal is to determine the best decision based on a given question, business rules, and a database.

Methodology: How PlanRAG Works