LLMs are still not great at producing structured JSON reliably!

Getting them to consistently produce good structured JSON outputs can be a big battle, which has all kinds of downstream consequences.

But it's a lot easier to solve this problem if you have good evals to validate JSON outputs while developing and to monitor them in production!

Which is why we're excited to share that:

Athina now has multiple JSON validation evals that will work plug and play.

1. JSON Schema: This validates that the LLM is responding with a JSON structure that matches what you expect. It's highly configurable, so you can verify lots of things:
   • whether the response is valid JSON
   • whether the structure is what you expected
   • whether fields are the correct data types (string / number / array, etc)
   • whether required fields are present

2. JSON Validation: This eval allows you to compare the LLMs JSON response to make sure that certain keys match the values in a ground truth response.

You can also do fuzzy matching on certain keys (yes, even nested keys) by cosine similarity, or even by LLM based similarity.

3. Contains JSON: If you just want to check that the LLM response contains some JSON output

For everything else, there is always the Custom Code evaluation, which allows you to define any python function as your eval.