Do prompt / model regressions keep you up at night?

🔴 The problem

Regressions in your AI app can happen in a number of ways:

• Prompt changes: You just added 1 sentence to the prompt. What could go wrong?
• Model updates: Maybe you switched models, or maybe OpenAI updated theirs.
• You generated new embeddings, or modified your retriever.

Boom. Regressions.

This slows down your ability to push changes to production rapidly, and with confidence.

🟢 What we built

1. An open source SDK with dozens of preset evaluations for RAG pipelines, as well as support for custom evaluations.

   Athina can measure retrieval quality, as well as response quality on a number of criteria, including support for custom evaluation criteria.

   You can specify a custom failure_threshold for any eval to specify when it should pass or fail.

2. An example repository describing how you can add tests to your CI/CD pipeline to prevent LLM regressions from getting to production.

= No more regressions.

🧑‍💻 How you can use it