The Challenge of Testing Non-Deterministic Responses

One of the significant hurdles inAI development is dealing with non-deterministic responses.

These are responses that can vary even with the same input, making it challenging to ensure consistency and quality.

Additionally, there can be infinite combinations of user input, such as the variety of user messages to a chatbot, further complicating the testing process.

So, how do leading AI teams tackle this issue?

Strategies for Effective Prompt Testing

To identify the best prompt for your needs, several approaches can be employed:

1. Test with a Large Dataset

• Manual Inspection: Carefully review LLM generated responses to ensure they meet your quality standards. Use a spreadsheet or a dedicated tool like Athina to quickly review responses.
• Automated and Human Evaluation: Use a combination of human-eval and auto-eval to measure correctness and relevance.

2. Iterate on the Prompt or Model

• Modify Instructions: Refine your instructions or examples to guide the model more effectively.
• Experiment with Techniques: Try different prompt engineering techniques like Chain of Thought (CoT), Reflexion, and Tree of Thought (ToT).
• Compare Models: Use the same prompt with different models to see which one performs better.

3. Evaluate Against Criteria

Set clear evaluation criteria and measure how well the prompts meet these benchmarks.