Automatic Pairwise Evaluation based on YOUR grading criteria

There are 2 very common problems developers face when making changes to their AI:

1. Regressions
2. Which is actually better?

The workflow that many teams use to address this:

When you're trying a new model, prompt or retrieval strategy, a lot of teams follow a process that looks something like this:

1. Run scripts to generate 2 sets of responses: old responses as a baseline, and new responses with the changed prompt / model / retrieval.
2. Export and upload to Google sheets
3. Juggle columns around
4. Compare the old and the new responses one at a time, and mark the better one according to their grading criteria(s).
5. Gather metrics and make a judgement call: Are the new responses good enough for production or not?

This is better than "vibes-based" prompt engineering, but it doesn't sound like a process the best teams in the world use.

= many hours

The Athina Workflow