Incident management · Production

Anatomy of an AI agent incident: Gradient Labs resolves memory and latency issues in production

The problem

Gradient Labs' production AI agent experienced unexplained high memory usage followed by elevated latency, both difficult to diagnose because daytime redeployments masked the memory growth and variable traffic obscured the latency root cause.

First attempt

The Temporal Workflow cache was filling containers beyond their memory limit causing crashes, and Cloud Run's crash-recovery had been auto-scaling to compensate—masking the root cause. Fixing the cache inadvertently stopped the crash-driven scaling, causing the agent to scale down and introducing a latency bottleneck.

Workflow diagram · grounded in source
1
Memory usage alert fires
trigger
“Google Cloud platform alerts fired: the memory usage across our agent's containers was abnormally high”
2
Incident raised; memory increased
human_review
“we bought ourselves some time by redeploying the agent with more memory. The alert was resolved within minutes, but we knew that a deep investigation was required to get to a root cause”
3
Flame graph root-cause analysis
validation
“The Google Cloud Profiler flame graphs deltas for our agent's memory usage finally shed some light. Temporal's top-level execution functions had the biggest growth over time in exclusive memory”
4
Cache hypothesis validated
validation
“The first change was to boost the agent's memory to 5x what it previously had. Memory usage continued to grow but eventually plateaued. There was some kind of limit that we had reached. We then deployed a change to decrease the worker's …”
5
Worker cache size tuned
output
“We closed this out by tuning our worker cache size. This was a trade-off between: how much memory we need to provision for our instances (infrastructure cost) vs how large we could make the cache (which reduces network calls to Temporal …”
6
Latency increase detected
trigger
“we spotted that our AI agent's mean latency seemed to have increased”
7
LLM failover adjusted
routing
“we adjusted our LLM fail-over system and the mean latency started going back down”
8
Minimum instance count fixed
output
“By manually editing the minimum instance count, we saw an immediate uptick in activity executions and everything returned to normal”
Reported outcome

Tuning the Temporal worker cache size resolved the memory issue.
The subsequent latency bottleneck was resolved in less than an hour by manually setting a minimum instance count.

Reported metrics
Memory boost during validation test5x
Cache size reduction during validation test10x
Temporal activity scheduling delaysometimes more than 10x
Time to identify and fix latency issueless than an hour
Reported stack
GoCloud RunTemporalGoogle Cloud ProfilerTemporal Cloud
Source
https://blog.gradient-labs.ai/p/anatomy-of-an-ai-agent-incident
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Tuning the Temporal worker cache size resolved the memory issue.

What tools did this team use?

Go, Cloud Run, Temporal, Google Cloud Profiler, Temporal Cloud.

What results were reported?

Memory boost during validation test: 5x; Cache size reduction during validation test: 10x; Temporal activity scheduling delay: sometimes more than 10x; Time to identify and fix latency issue: less than an hour (source-reported, not independently verified).

What failed first in this deployment?

The Temporal Workflow cache was filling containers beyond their memory limit causing crashes, and Cloud Run's crash-recovery had been auto-scaling to compensate—masking the root cause.

How is this incident management AI workflow structured?

Memory usage alert fires → Incident raised; memory increased → Flame graph root-cause analysis → Cache hypothesis validated → Worker cache size tuned → Latency increase detected → LLM failover adjusted → Minimum instance count fixed.

Related incident management cases