Customer support · Production

How Assembled shipped GPT-5 support within two hours of launch

The problem

Early in Assembled's development, routing all inference through a single model created a single point of failure that caused outages; new model releases required downstream service code changes, making rapid integration impractical.

First attempt

Routing all inference to a single model caused production outages, forcing the team to rethink the architecture.

Workflow diagram · grounded in source
1
Monitor release signals
trigger
“By monitoring prior OpenAI announcement patterns and community chatter, we had an extremely accurate guess a week or two before GPT-5 was scheduled to launched”
2
Pre-stage model config
integration
“When the model type appeared in the API documentation, it was relatively easy for us to set our model upgrade process in place”
3
Domain-specific eval suites
validation
“We separate our evaluations into different tasks and maintain separate datatsets for voice (focused on latency and instruction following), chat (focused on tone of voice and response accuracy), and long-horizon email workflows (focused o…”
4
LLM-as-a-judge scoring
ai_action
“We make heavy use of LLM-as-a-judge to speed up evaluation”
5
Human evaluation check
human_review
“for all model upgrades we also do a set of human evaluations to get a sense of how the model behaves”
6
Provider-agnostic router swap
routing
“We now funnel every inference call through a provider-agnostic interface. Vendor, model name, and temperature are just environment variables—no downstream service code changes are required”
7
Customer dashboard toggle
output
“appeared as a "GPT-5" toggle in every customer dashboard”
Reported outcome

OpenAI launched GPT-5 at 10 AM PT; by 12 PM it had cleared Assembled's evaluation harness and appeared as a toggle in every customer dashboard—a two-hour turnaround.

Reported metrics
Model integration turnaroundtwo-hour turnaround
Time from launch to customer availability10 AM PT to 12 PM
Reported stack
GPT-5LLM-as-a-judgeOpenAI
Source
https://www.assembled.com/blog/how-we-shipped-gpt-5-support-before-lunch
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

OpenAI launched GPT-5 at 10 AM PT; by 12 PM it had cleared Assembled's evaluation harness and appeared as a toggle in every customer dashboard—a two-hour turnaround.

What tools did this team use?

GPT-5, LLM-as-a-judge, OpenAI.

What results were reported?

Model integration turnaround: two-hour turnaround; Time from launch to customer availability: 10 AM PT to 12 PM (source-reported, not independently verified).

What failed first in this deployment?

Routing all inference to a single model caused production outages, forcing the team to rethink the architecture.

How is this customer support AI workflow structured?

Monitor release signals → Pre-stage model config → Domain-specific eval suites → LLM-as-a-judge scoring → Human evaluation check → Provider-agnostic router swap → Customer dashboard toggle.

Related customer support cases