Evaluating AI chatbots is really hard.
Most prebuilt evaluation frameworks only evaluate individual messages, but they don't look at the rest of the conversation history.
This makes evaluations unreliable and inaccurate.
What we built
We built some new evaluation metrics that give a score to the entire conversation. For example, Conversation Coherence eval measures the % of messages in a chat that are coherent given the previous chat history.
Our production eval orchestration system will now automatically pass the chat history into the evaluation prompt as well.
This is configurable with a setting, and it improves the evaluation accuracy for chatbot conversations where the meaning of the nth message depends on a previous message.