Customer support · Production

Migrating to GPT-4o: How Assembled achieved rapid transition

The problem

Migrating Assembled's entire customer base to GPT-4o within a single day posed substantial risk, requiring a comprehensive overhaul of integration processes to meet quality and reliability standards without disrupting customers across tens of thousands of support tickets.

Workflow diagram · grounded in source
1
A/B test concurrent models
validation
“We used an A/B test in parallel with our existing models to manage the migration efficiently. By running both GPT-4 and GPT-4o models concurrently, we tracked various metrics to ensure a smooth transition. These metrics included response…”
2
Golden dataset benchmark run
validation
“we ran our new model on the golden dataset and did a manual review of the quality of responses and that GPT-4o performed significantly better on most benchmarks”
3
Blind human evaluation
human_review
“Human reviewers compare responses from the current and updated systems without knowing which is which, reducing bias and ensuring objective assessments”
4
Metrics-based quality gate
validation
“Text replies are evaluated against set metrics, such as inclusion of URLs, sentiment, length, and use of bullet points”
5
Fallback routing on issue detection
routing
“If any issues were detected during the migration, customers were seamlessly switched back to GPT-4, ensuring uninterrupted service”
6
Full migration to GPT-4o
output
“we successfully migrated our entire customer base to GPT-4o in under 24 hours”
Reported outcome

Assembled successfully migrated its entire customer base from GPT-4 to GPT-4o in under 24 hours, with GPT-4o performing significantly better on most benchmarks.

Reported metrics
Migration timeunder 24 hours
GPT-4o benchmark performance vs GPT-4significantly better on most benchmarks
Reported stack
GPT-4GPT-4oLLM Comparator
Source
https://www.assembled.com/blog/migrating-to-gpt-4o-how-assembled-achieved-rapid-transition
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Assembled successfully migrated its entire customer base from GPT-4 to GPT-4o in under 24 hours, with GPT-4o performing significantly better on most benchmarks.

What tools did this team use?

GPT-4, GPT-4o, LLM Comparator.

What results were reported?

Migration time: under 24 hours; GPT-4o benchmark performance vs GPT-4: significantly better on most benchmarks (source-reported, not independently verified).

How is this customer support AI workflow structured?

A/B test concurrent models → Golden dataset benchmark run → Blind human evaluation → Metrics-based quality gate → Fallback routing on issue detection → Full migration to GPT-4o.

Related customer support cases