Quality assurance · Production

PerfInsights: Uber's GenAI system detects Go performance antipatterns and cuts optimization engineering time by 93%

The problem

Uber's top 10 Go services cost more than multi-million dollars in compute in March 2024 alone, yet optimizing Go services required deep expertise and days or weeks of manual profiling and analysis—making systematic performance tuning prohibitively expensive for most teams.

First attempt

Initial single-shot LLM-based antipattern detection produced inconsistent and unreliable results—responses varied between runs, included hallucinations, and often generated non-runnable code, with false positives exceeding 80%.

Workflow diagram · grounded in source
1
Production profile collection
Trigger
CPU and memory profiles are collected from production services during peak traffic periods using Uber's daily fleet-wide profiler.
▾ source quote
“PerfInsights leverages CPU and memory profiles from production services using Uber's daily fleet-wide profiler during peak traffic periods”
2
Top-30 function selection
Validation
PerfInsights identifies the top 30 most expensive functions per service based on flat CPU usage.
▾ source quote
“it identifies the top 30 most expensive functions based on flat CPU usage”
3
Static noise filtering
Validation
A static filter excludes open-source dependencies and internal runtime functions, trimming noise from the candidate set.
▾ source quote
“PerfInsights applies a static filter that excludes open-source dependencies and internal runtime functions. This step trims noise from the candidate set”
4
LLM antipattern detection
Ai action
Hotpath function source code and a list of antipatterns are passed to a large language model (LLM) for analysis.
▾ source quote
“PerfInsights passes their full source codes and a list of antipatterns to a large language model (LLM) for analysis. By combining profiling context with pattern awareness, the model can pinpoint inefficient constructs with high precision. For example, if a function …”
5
LLM jury validation
Validation
A jury of large language models independently assesses whether each detected antipattern is present and whether the suggested optimization is valid.
▾ source quote
“PerfInsights leverages a jury of large language models to validate each detected antipattern. These models independently assess whether an antipattern is present and whether the suggested optimization is valid”
6
LLMCheck rule-based verification
Validation
LLMCheck runs domain-specific rule-based validators as a second layer of verification to evaluate LLM responses.
▾ source quote
“PerfInsights employs a second layer of verification via LLMCheck by running through several domain-specific rule-based validators to evaluate LLM responses”
7
Output to Optix and developers
Output
Validated suggestions with confidence scores flow into Optix, Uber's continuous code optimization tool, for code transformation or manual developer review.
▾ source quote
“validated suggestions flowing directly into Optix, Uber's continuous code optimization tool”
8
Accuracy metrics and drift detection
Feedback loop
LLMCheck logs metrics on detection accuracy, tracking failure rates and signaling potential model drift.
▾ source quote
“LLMCheck also logs metrics on detection accuracy, tracking failure rates and signaling potential model drift”
Reported outcome

PerfInsights reduced engineering time per performance issue from 14.5 hours to almost 1 hour—a 93.10% time savings—and cut false positives from over 80% to the low teens, with hundreds of diffs already merged into Uber's Go monorepo.

Reported metrics
Go services compute spend (top 10, March 2024)multi-million dollars
Task durationdays now take hours
False positive rate (before)over 80%
False positive rate (after)low teens
Show all 17 reported metrics
Go services compute spend (top 10, March 2024)multi-million dollars
task durationdays now take hours
false positive rate (before)over 80%
false positive rate (after)low teens
validated detections per day average (February)265
validated detections single-day peak500
validated detections per day (June)176
reduction in validated detections over 4 months33.5%
manual engineering hours for sample antipattern setover 1,400 hours
projected manual hours to address annual antipattern volumeapproximately 3,800 hours
engineering time savings per issue93.10%
engineering time per issue (before)14.5 hours
engineering time per issue (after)almost 1 hour
hallucination rate reductionmore than 80%
merged optimization diffshundreds
false positive reduction via prompt engineering80%
cost saving milestone per optimization diff5 digits saving
Reported stack
PerfInsightsLLMLLMCheckOptix
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://www.uber.com/en-GB/blog/perfinsights/?uclick_id=0a73d271-32e7-4b77-9697-a587a4c8d9fe
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

PerfInsights reduced engineering time per performance issue from 14.5 hours to almost 1 hour—a 93.10% time savings—and cut false positives from over 80% to the low teens, with hundreds of diffs already merged into Ube…

What tools did this team use?

PerfInsights, LLM, LLMCheck, Optix.

What results were reported?

Go services compute spend (top 10, March 2024): multi-million dollars; Task duration: days now take hours; False positive rate (before): over 80%; False positive rate (after): low teens (source-reported, not independently verified).

What failed first in this deployment?

Initial single-shot LLM-based antipattern detection produced inconsistent and unreliable results—responses varied between runs, included hallucinations, and often generated non-runnable code, with false positives exce…

How is this quality assurance AI workflow structured?

Production profile collection → Top-30 function selection → Static noise filtering → LLM antipattern detection → LLM jury validation → LLMCheck rule-based verification → Output to Optix and developers → Accuracy metrics and drift detection.

Related quality assurance cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Quality assurance cases →