A unique challenge of working with LLMs is figuring out if a new prompt / model has improved your responses or made them worse.

So how do teams solve this right now?

The Old Way

• Run prompt A, model A on input set X
• Run prompt A, model B on input set X
• Run evaluations to assess which set has a higher score
• Export results to a spreadsheet and compare responses side-by-side

And how can Athina help?

What we built

A workflow that does the same steps automatically so you can compare prompts / models 10x faster.

The fundamental idea here is: same inputs, new outputs.

Inputs can be queries from an existing dataset from your production logs.

With Athina, you can now "re-run" the same set of input variables with a new prompt template, or a new model.