According to the latest and greatest research about state-of-the-art eval techniques, the best automatic evaluators will use LLMs.

(It's a little meta, but it works pretty well, and better than anything else except a human)

But sometimes, they are too expensive to run in production. Here's how Athina solves this.

How to run LLM-graded evals in production

• Without blowing up your OpenAI budget.

Here's how we help you solve this so you can still get model performance insights in production!

• Set a sampling percentage: You don't need to run evals on every single production log to get good insights on model performance. Set a sampling percentage of 20% to reduce your eval cost by 80%, and still benefit from the insights.
• Set a hard limit on the number of evals per month: Set a maximum number of evaluations to run per month as a hard limit. This ensures your costs are always capped.
• Set Filters: When you configure Evals on Athina, you can set filters. These filters will ensure that evals ONLY run on inferences that match the filters. For example, you could set a filter to only run evals on inferences where:
  • user query contains "refund"
  • language model is gpt-4
  • prompt slug is summarization/v3
• Use a cheaper model: Many evals will work great with GPT-3.5 (though some will not). That shaves the cost down to 1/10 of GPT-4 Turbo cost.

If you want to run evals on even cheaper models (Llama, Mistral, etc), then reply to this email. We're working on it :)

Function Evals

We just shipped over a dozen new function evals!