Quality assurance · Production

Improving Taralli food calorie tracking accuracy from 17% to 76% with DSPy evals

The problem

The initial zero-shot calorie tracking system produced wildly inaccurate outputs — including tens of thousands of kcal for common foods — because the model misinterpreted quantity fields, resulting in only 17% accuracy.

First attempt

The naive zero-shot GPT-4o mini approach with structured outputs achieved only 17% accuracy, with common failure modes including wrong calorie totals and missing food groups.

Workflow diagram · grounded in source
1
User submits food description
trigger
“take a user's food description as a string”
2
LLM generates structured nutrition output
ai_action
“pass it through gpt-4o-mini with structured outputs, and generate a JSON response”
3
Log inputs and outputs via Weave
integration
“Using W&B Weave and a simple decorator, I was able to log every input-output of the system”
4
Build golden dataset
human_review
“I started collecting a golden dataset: a collection of food descriptions and their corresponding nutritional analysis. Using OpenAI's o3 and Google's Gemini 2.5 Pro - I was able to make sure all the quantities, descriptions, and food gro…”
5
Evaluate predictions with metric
validation
“Is the total calories of the prediction within 10% of the total calories of the golden example? - Is there an overlap between the food groups of the golden example vs. the predicted one?”
6
DSPy few-shot prompt optimization
ai_action
“BootstrapFewShotWithRandomSearch. This approach finds optimal examples from the training dataset to include in the prompt, which can dramatically improve performance”
7
Deploy optimized API endpoint
output
“I integrated this new API endpoint, and now all users of the app benefit from our improved food prediction model”
8
Data flywheel for continuous improvement
feedback_loop
“As more users interact with the app, I collect more data, I can correct/review that data and add it to my golden dataset. This flywheel allows me to continuously update the prompt as we go.”
Reported outcome

After applying DSPy's BootstrapFewShotWithRandomSearch optimization with Gemini 2.5 Flash, tracking accuracy improved from 17% to 76%, and the optimized model was integrated into the production app for all users.

Reported metrics
Initial tracking accuracy (gpt-4o-mini zero-shot)17.24%
Gemini 2.5 Flash zero-shot accuracy24.1%
optimized tracking accuracy after DSPy75.9%
Tracking accuracy improvement end-to-endimprove tracking accuracy from 17% to 76%
Reported stack
gpt-4o-miniDSPyW&B WeavePydanticFastAPIGemini 2.5 Flasho3Gemini 2.5 Proopenrouter
Source
https://duarteocarmo.com/blog/evals-are-all-you-need
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After applying DSPy's BootstrapFewShotWithRandomSearch optimization with Gemini 2.5 Flash, tracking accuracy improved from 17% to 76%, and the optimized model was integrated into the production app for all users.

What tools did this team use?

gpt-4o-mini, DSPy, W&B Weave, Pydantic, FastAPI, Gemini 2.5 Flash, o3, Gemini 2.5 Pro, openrouter.

What results were reported?

Initial tracking accuracy (gpt-4o-mini zero-shot): 17.24%; Gemini 2.5 Flash zero-shot accuracy: 24.1%; optimized tracking accuracy after DSPy: 75.9%; Tracking accuracy improvement end-to-end: improve tracking accuracy from 17% to 76% (source-reported, not independently verified).

What failed first in this deployment?

The naive zero-shot GPT-4o mini approach with structured outputs achieved only 17% accuracy, with common failure modes including wrong calorie totals and missing food groups.

How is this quality assurance AI workflow structured?

User submits food description → LLM generates structured nutrition output → Log inputs and outputs via Weave → Build golden dataset → Evaluate predictions with metric → DSPy few-shot prompt optimization → Deploy optimized API endpoint → Data flywheel for continuous improvement.

Related quality assurance cases