Quality assurance · Production

PerfInsights: GenAI-powered detection of Go performance antipatterns at Uber

The problem

Optimizing Go services at Uber required deep expertise and significant manual effort, with profiling and analysis taking days to weeks; in March 2024, the top 10 Go services alone accounted for more than multi-million dollars in compute spend, making performance tuning prohibitively expensive and non-trivial for most teams.

First attempt

Initial single-shot LLM-based antipattern detection produced inconsistent and unreliable results—responses varied between runs, included hallucinations, and often generated non-runnable code—with false positives exceeding 80%.

Workflow diagram · grounded in source
1
Fleet-wide production profiling
Trigger
PerfInsights collects CPU and memory profiles from production services using Uber's daily fleet-wide profiler during peak traffic periods.
▾ source quote
“PerfInsights leverages CPU and memory profiles from production services using Uber's daily fleet-wide profiler during peak traffic periods”
2
Hotpath function filtering
Validation
For each service, PerfInsights identifies the top 30 most expensive functions based on flat CPU usage.
▾ source quote
“For each service, it identifies the top 30 most expensive functions based on flat CPU usage. This is based on the observation that the top 30 most expensive functions account for the majority of CPU usage. Additionally, if runtime.mallocgc—the Go …”
3
Static noise exclusion
Validation
A static filter excludes open-source dependencies and internal runtime functions, ensuring downstream analysis focuses only on service-owned code.
▾ source quote
“PerfInsights applies a static filter that excludes open-source dependencies and internal runtime functions. This step trims noise from the candidate set, ensuring downstream analysis focuses only on service-owned code that's most likely to benefit from optimization”
4
LLM antipattern detection
Ai action
Hotpath function source code and an antipattern catalog are passed to a large language model (LLM) that pinpoints inefficient constructs with high precision.
▾ source quote
“PerfInsights passes their full source codes and a list of antipatterns to a large language model (LLM) for analysis. By combining profiling context with pattern awareness, the model can pinpoint inefficient constructs with high precision. For example, if a function …”
5
LLM jury validation
Validation
A jury of large language models independently validates each detected antipattern and its suggested optimization.
▾ source quote
“PerfInsights leverages a jury of large language models to validate each detected antipattern. These models independently assess whether an antipattern is present and whether the suggested optimization is valid. This ensemble approach mitigates common hallucinations such as incorrectly detecting loop …”
6
LLMCheck rule-based validation
Validation
LLMCheck runs domain-specific rule-based validators to catch false positives and track detection accuracy.
▾ source quote
“PerfInsights employs a second layer of verification via LLMCheck by running through several domain-specific rule-based validators to evaluate LLM responses”
7
Confidence-scored output
Output
The final output includes a confidence score of the function's optimisability and suggested improvements, fed into downstream tools or manual developer review.
▾ source quote
“The final output includes a confidence score of the function's optimisability and suggested improvements. These insights are fed into downstream tools for code transformation or manual review by developers”
8
Integration with Optix
Integration
Validated suggestions flow directly into Optix, Uber's continuous code optimization tool.
▾ source quote
“validated suggestions flowing directly into Optix, Uber's continuous code optimization tool. This has already produced hundreds of merged diffs, measurably improving performance and generating meaningful cost savings”
9
Accuracy tracking feedback loop
Feedback loop
LLMCheck logs metrics on detection accuracy, tracking failure rates and signaling potential model drift.
▾ source quote
“LLMCheck also logs metrics on detection accuracy, tracking failure rates and signaling potential model drift”
Reported outcome

PerfInsights reduced performance analysis from days to hours, cut engineering time per issue from 14.5 hours to almost 1 hour (93.10% savings), reduced false positives from over 80% to the low teens, and produced hundreds of merged diffs driving compute cost reductions across Uber's Go services.

Reported metrics
compute spend (top 10 Go services, March 2024)more than multi-million dollars
Analysis time reductiontasks that once required days now take hours
False positive rate (after validation pipeline)over 80% to the low teens
Diffs generated and mergedhundreds
Show all 21 reported metrics
compute spend (top 10 Go services, March 2024)more than multi-million dollars
analysis time reductiontasks that once required days now take hours
false positive rate (after validation pipeline)over 80% to the low teens
diffs generated and mergedhundreds
average validated detections per day (February)265
single-day detection high (February)500
validated detections by June176
antipattern count reduction over 4 months33.5%
sustained antipattern reduction count over 4 months89 antipatterns
manual effort for 5 critical antipatterns∼320 hours (two engineers, one month)
manual effort for 11 unique antipatterns∼160 hours (four-person team, one week)
dedicated expert optimization effort∼960 hours (six months full-time)
total historical manual effort (sample cases)over 1,400 hours
projected annual antipattern reduction267 antipatterns
manual hours equivalent for projected annual volumeapproximately 3,800 hours
engineering time per issue (before vs after)from 14.5 hours to almost 1 hour of tool runtime
engineering time savings93.10%
hallucination rate reductionmore than 80%
false positive reduction (prompt engineering)80%
first high-value diff savings5 digits saving
compute cost reductionsscalable compute cost reductions
Reported stack
PerfInsightsLLMLLMCheckOptix
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://www.uber.com/en-NL/blog/perfinsights/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

PerfInsights reduced performance analysis from days to hours, cut engineering time per issue from 14.5 hours to almost 1 hour (93.10% savings), reduced false positives from over 80% to the low teens, and produced hund…

What tools did this team use?

PerfInsights, LLM, LLMCheck, Optix.

What results were reported?

compute spend (top 10 Go services, March 2024): more than multi-million dollars; Analysis time reduction: tasks that once required days now take hours; False positive rate (after validation pipeline): over 80% to the low teens; Diffs generated and merged: hundreds (source-reported, not independently verified).

What failed first in this deployment?

Initial single-shot LLM-based antipattern detection produced inconsistent and unreliable results—responses varied between runs, included hallucinations, and often generated non-runnable code—with false positives excee…

How is this quality assurance AI workflow structured?

Fleet-wide production profiling → Hotpath function filtering → Static noise exclusion → LLM antipattern detection → LLM jury validation → LLMCheck rule-based validation → Confidence-scored output → Integration with Optix → Accuracy tracking feedback loop.

Related quality assurance cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Quality assurance cases →