Customer support · Production

How ElevenLabs engineered RAG to be 50% faster with model racing

The problem

ElevenLabs built RAG directly into every query for consistent accuracy, but the query rewriting step relied on a single externally-hosted LLM, creating a hard latency dependency that accounted for more than 80% of total RAG latency.

First attempt

The prior architecture's single externally-hosted LLM for query rewriting was vulnerable to peak-demand slowdowns and provider outages, making the system fragile.

Workflow diagram · grounded in source
1
Query enters pipeline
trigger
“we've built it directly into the request pipeline so it runs on every query”
2
Dialogue history collapsed
ai_action
“the system needs to collapse dialogue history into a precise, self-contained query”
3
Parallel model race
ai_action
“Each query is sent to multiple models at once, including our self-hosted Qwen 3-4B and 3-30B-A3B models. The first valid response wins.”
4
Fallback on timeout
routing
“If no model responds within one second, we fall back to the user's raw message”
5
Relevant context retrieved
ai_action
“RAG embeds the query, retrieves the most relevant information, and passes it as context to the model”
Reported outcome

Model racing cut median RAG latency from 326ms to 155ms, with p75 dropping from 436ms to 250ms and p95 from 629ms to 426ms, while provider outages no longer interrupted conversations.

Reported metrics
median RAG latency326ms → 155ms
p75 RAG latency436ms → 250ms
p95 RAG latency629ms → 426ms
query rewriting share of RAG latency (pre-fix)more than 80% of RAG latency
Show all 5 reported metrics
median RAG latency326ms → 155ms
p75 RAG latency436ms → 250ms
p95 RAG latency629ms → 426ms
query rewriting share of RAG latency (pre-fix)more than 80% of RAG latency
overall RAG speed improvement50% faster
Reported stack
RAGQwen 3-4B
Source
https://elevenlabs.io/blog/engineering-rag
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Model racing cut median RAG latency from 326ms to 155ms, with p75 dropping from 436ms to 250ms and p95 from 629ms to 426ms, while provider outages no longer interrupted conversations.

What tools did this team use?

RAG, Qwen 3-4B.

What results were reported?

median RAG latency: 326ms → 155ms; p75 RAG latency: 436ms → 250ms; p95 RAG latency: 629ms → 426ms; query rewriting share of RAG latency (pre-fix): more than 80% of RAG latency (source-reported, not independently verified).

What failed first in this deployment?

The prior architecture's single externally-hosted LLM for query rewriting was vulnerable to peak-demand slowdowns and provider outages, making the system fragile.

How is this customer support AI workflow structured?

Query enters pipeline → Dialogue history collapsed → Parallel model race → Fallback on timeout → Relevant context retrieved.

Related customer support cases