Summary accuracies can be tricky to evaluate.

Techniques like ROUGE scores are useful, but they rely on comparisons against a reference summary.

But in production, you will not have a reference summary. Even your development dataset may not have reference summaries.

One of the most effective techniques for evaluating summaries (without a reference) has been to use an LLM to generate 2 sets of answers for a set of closed-ended (Y/N) questions.

One set of answers would be generated based on information in the summary.

The other set of answers would be generated based on information in the source document.

We can then compare the answers to find contradictions, and hallucinations.

The flow of the eval looks something like this:

→ generate 10 questions from summary

→ ask question_answerer LLM to answer the questions using information from the summary

→ ask question_answerer LLM to answer the questions using information from the source document

→ compare the 2 sets of answers

Here's how you can try this out using Athina SDK: