Customer support · Production

Building resilient agentic systems: provider and model failover at Gradient Labs

The problem

AI agents make chains of LLM calls where each step costs latency and money, so a single failure could force the entire chain to restart; for a customer-facing financial services agent, high reliability is non-negotiable.

First attempt

A provider latency spike shifted the entire latency distribution upward without triggering the existing per-request timeout-based failover mechanism, requiring manual intervention.

Workflow diagram · grounded in source
1
Customer request received
trigger
“our AI agent interacts with the customers of financial services companies”
2
LLM chain constructs answer
ai_action
“we use a blend of different large language models (LLMs) to construct the highest quality answers”
3
Temporal checkpoints progress
integration
“we use Temporal, a durable execution system, that provides us with a way to effectively checkpoint progress out of the box”
4
Provider preference routing
routing
“each completion request starts with an ordered list of API provider preferences”
5
Provider failover evaluation
validation
“fail over from one provider to another when we encounter certain errors, rate limits or latency spikes”
6
Model-level failover
routing
“we can also activate model failover: for each LLM API request, we can configure a different model to use in the event of failure”
7
Latency alert feedback loop
feedback_loop
“through our latency-based alerts, we were able to identify this fairly quickly and manually invoke the failover mechanism”
8
Agent reply delivered
output
“their customers continue to receive replies from our AI agent”
Reported outcome

Gradient Labs built a layered resilience system using Temporal for durable execution plus provider and model failover, ensuring customers continue to receive replies even when entire LLM provider groups are down.

Reported metrics
Observed p75+ latency spikewell over 10s
Reported stack
TemporalOpenAIAnthropicGoogleAzure
Source
https://blog.gradient-labs.ai/p/building-resilient-agentic-systems
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Gradient Labs built a layered resilience system using Temporal for durable execution plus provider and model failover, ensuring customers continue to receive replies even when entire LLM provider groups are down.

What tools did this team use?

Temporal, OpenAI, Anthropic, Google, Azure.

What results were reported?

Observed p75+ latency spike: well over 10s (source-reported, not independently verified).

What failed first in this deployment?

A provider latency spike shifted the entire latency distribution upward without triggering the existing per-request timeout-based failover mechanism, requiring manual intervention.

How is this customer support AI workflow structured?

Customer request received → LLM chain constructs answer → Temporal checkpoints progress → Provider preference routing → Provider failover evaluation → Model-level failover → Latency alert feedback loop → Agent reply delivered.

Related customer support cases