Quality assurance · Production

Anaconda's Evaluations Driven Development improves Anaconda Assistant error-handling accuracy from 0–13% to 87–100%

The problem

Data scientists frequently encountered code errors without clear, reliable guidance, and the Anaconda Assistant's underlying language models correctly identified and fixed errors in at most 13% of test cases before prompt engineering.

First attempt

Initial evaluations found success rates as low as 0% for Mistral 7B and 12% for GPT-3.5-Turbo when diagnosing and fixing Python errors, before any prompt engineering was applied.

Workflow diagram · grounded in source
1
User describes error to Assistant
trigger
“Just describe your error to the Assistant”
2
AI generates explanation and fix
ai_action
“within seconds, you'll get a plain-English explanation of the problem and suggestions for fixing it”
3
Llm-eval tests across scenarios
validation
“It's a comprehensive testing framework that allows us to simulate thousands of realistic user interactions and assess the Assistant's responses across a wide range of scenarios, from debugging complex errors to generating readable, well-…”
4
Agentic Feedback Iteration
ai_action
“The model analyzes this data and provides specific suggestions for how to modify the prompts and queries to address the identified weaknesses and edge cases. For example, it might recommend adding more context to help the Assistant bette…”
5
Prompt refinement and re-evaluation
feedback_loop
“We incorporate the model's suggestions into the prompts and queries, and re-run the evaluation process to measure the impact on accuracy. We repeat this iterative process, with the model providing further rounds of feedback and refinemen…”
Reported outcome

After applying few-shot learning, chain-of-thought prompting, and Agentic Feedback Iteration, success rates rose to 87% for GPT-3.5-Turbo at temperature 0 and 100% for Mistral 7B at temperature 1.

Reported metrics
User interactions involving error help60%
GPT-3.5-Turbo (temp 0) initial success rate12%
GPT-3.5-Turbo (temp 1) initial success rate13%
Mistral 7B (temp 0) initial success rate0%
Show all 9 reported metrics
user interactions involving error help60%
GPT-3.5-Turbo (temp 0) initial success rate12%
GPT-3.5-Turbo (temp 1) initial success rate13%
Mistral 7B (temp 0) initial success rate0%
Mistral 7B (temp 1) initial success rate2%
GPT-3.5-Turbo (temp 0) post-optimization success rate87%
GPT-3.5-Turbo (temp 1) post-optimization success rate63%
Mistral 7B (temp 0.1) post-optimization success rate87%
Mistral 7B (temp 1) post-optimization success rate100%
Reported stack
Anaconda Assistantllm-evalGPT-3.5-TurboMistral 7B Instruct v0.2
Source
https://www.anaconda.com/blog/introducing-evaluations-driven-development?utm_source=chatgpt.com
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After applying few-shot learning, chain-of-thought prompting, and Agentic Feedback Iteration, success rates rose to 87% for GPT-3.5-Turbo at temperature 0 and 100% for Mistral 7B at temperature 1.

What tools did this team use?

Anaconda Assistant, llm-eval, GPT-3.5-Turbo, Mistral 7B Instruct v0.2.

What results were reported?

User interactions involving error help: 60%; GPT-3.5-Turbo (temp 0) initial success rate: 12%; GPT-3.5-Turbo (temp 1) initial success rate: 13%; Mistral 7B (temp 0) initial success rate: 0% (source-reported, not independently verified).

What failed first in this deployment?

Initial evaluations found success rates as low as 0% for Mistral 7B and 12% for GPT-3.5-Turbo when diagnosing and fixing Python errors, before any prompt engineering was applied.

How is this quality assurance AI workflow structured?

User describes error to Assistant → AI generates explanation and fix → Llm-eval tests across scenarios → Agentic Feedback Iteration → Prompt refinement and re-evaluation.

Related quality assurance cases