A recent research paper surveyed hallucination mitigation techniques implemented by other researchers, and came up with 32 hallucination mitigation techniques.

Some of these are theoretical, and may be difficult to apply in practice, but they contain common patterns and ideas that can help you build a more reliable LLM inference pipeline.

Here are 5 advanced techniques from the paper:

1. LLM-Augmenter

LLM-Augmenter uses multiple sequential LLM calls to detect hallucinations in the generated output, and repeatedly revise the output until there are no more hallucinations detected.

Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback - Microsoft ResearchLarge language models (LLMs), such as ChatGPT, are able to generate human-like, fluent responses for many downstream tasks, e.g., task-oriented dialog and question answering. However, applying LLMs to real-world, mission-critical applications remains challenging mainly due to their tendency to generate hallucinations and their inability to use external knowledge. This blog introduces our work on LLM-Augmenter, […]Microsoft Research

2. FreshPrompt

"FreshPrompt is a few-shot prompting method that leverages a search engine to incorporate relevant and up-to-date information into prompts"

Basic Idea: Plug in Google results into your RAG.

This won't be useful if your data is silo-ed from the internet, but if your LLM application is using data that can be found on the internet, this technique can be used to add an additional chunk of relevant, up-to-date context to your prompt.

FreshLLMs: Refreshing Large Language Models with Search Engine AugmentationMost large language models (LLMs) are trained once and never updated; thus, they lack the ability to dynamically adapt to our ever-changing world. In this work, we perform a detailed study of the factuality of LLM-generated text in the context of answering questions that test current world knowledge. Specifically, we introduce FreshQA, a novel dynamic QA benchmark encompassing a diverse range of question and answer types, including questions that require fast-changing world knowledge as well as questions with false premises that need to be debunked. We benchmark a diverse array of both closed and open-source LLMs under a two-mode evaluation procedure that allows us to measure both correctness and hallucination. Through human evaluations involving more than 50K judgments, we shed light on limitations of these models and demonstrate significant room for improvement: for instance, all models (regardless of model size) struggle on questions that involve fast-changing knowledge and false premises. Motivated by these results, we present FreshPrompt, a simple few-shot prompting method that substantially boosts the performance of an LLM on FreshQA by incorporating relevant and up-to-date information retrieved from a search engine into the prompt. Our experiments show that FreshPrompt outperforms both competing search engine-augmented prompting methods such as Self-Ask (Press et al., 2022) as well as commercial systems such as Perplexity.AI. Further analysis of FreshPrompt reveals that both the number of retrieved evidences and their order play a key role in influencing the correctness of LLM-generated answers. Additionally, instructing the LLM to generate concise and direct answers helps reduce hallucination compared to encouraging more verbose answers. To facilitate future work, we release FreshQA at github.com/freshllms/freshqa and commit to updating it at regular intervals.arXiv.orgTu Vu

3. Real-Time Verification and Rectification (EVER)

"Instead of waiting until the end of the generation process to rectify hallucinations, EVER employs a real-time, stepwise generation and hallucination rectification strategy. The primary objective is to detect and rectify hallucinations as they occur during the text generation process"

This real-time verification and rectification is done through a multi-step process