Workflow · Production

Meta's LLM Serving Infrastructure: Four Stages of Production Challenges

The problem

Since 2023, Meta has faced unprecedented demand for LLM compute driven by large models and longer context windows, requiring a production serving infrastructure that handles fitting, latency, reliability, and scaling challenges simultaneously.

Workflow diagram · grounded in source
1
Request arrives via streaming
trigger
“almost all LLM applications are using streaming interface”
2
Prefill generates first token
ai_action
“Prefill generates the first token. It reads a huge model weight, and it does tons of computations across all of these tokens within your prompt to find all the relation between each pair of tokens, and it then outputs one token.”
3
KV cache stores attention state
integration
“The K and V tensors for the same token at the same position basically stay the same across the single generation request”
4
Continuous batching schedules requests
routing
“At every bus stop, which is the end of each decoding step, it will try to pick up new passengers if there's an empty spot”
5
Disaggregation separates prefill from decode
routing
“We replicate the already partitioned weights, and we run prefill and decode in different services. This way, we can scale their system resources separately, and get rid of this annoying 10x P99 latency for decode, because that P99 is bas…”
6
Consistent hashing routes to cached host
routing
“To route these requests to the host with the cache, we use consistent hashing to sticky route the request from the same session to the same host, and we also have to ensure that retries on host failure are also sticky.”
7
CI/CD benchmarks validate every diff
feedback_loop
“our approach is no different than traditional CI/CD. Just make small benchmark runs every single one of your diffs, and run more comprehensive ones on every release for inference engine”
Reported outcome

Meta built hierarchical KV caching and disaggregated prefill/decode infrastructure, seeing over 50% reduction in both latency and capacity for caching-eligible workloads, while supporting Meta AI, smart glasses, and massive RLHF pipelines.

Reported metrics
latency and capacity reduction from hierarchical KV cachingover 50%
Performance boost from distributed inference, smaller input/model, and caching2x to 4x boost
Performance foundation from correct basics10x foundation
Compute unlocked by quantizationtwice or even more compute
Show all 8 reported metrics
latency and capacity reduction from hierarchical KV cachingover 50%
performance boost from distributed inference, smaller input/model, and caching2x to 4x boost
performance foundation from correct basics10x foundation
compute unlocked by quantizationtwice or even more compute
machines needed while maintaining latency SLOs with disaggregationmuch fewer machines
RLHF examples processed per busy weekhundreds of millions of examples
additional TTFT latency added by disaggregation50 to 100 milliseconds
effective FLOPS loss in production environment50%
Reported stack
Meta AILlamaH100A100MI300
Source
https://www.infoq.com/presentations/llm-meta/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Meta built hierarchical KV caching and disaggregated prefill/decode infrastructure, seeing over 50% reduction in both latency and capacity for caching-eligible workloads, while supporting Meta AI, smart glasses, and m…

What tools did this team use?

Meta AI, Llama, H100, A100, MI300.

What results were reported?

latency and capacity reduction from hierarchical KV caching: over 50%; Performance boost from distributed inference, smaller input/model, and caching: 2x to 4x boost; Performance foundation from correct basics: 10x foundation; Compute unlocked by quantization: twice or even more compute (source-reported, not independently verified).

How is this workflow AI workflow structured?

Request arrives via streaming → Prefill generates first token → KV cache stores attention state → Continuous batching schedules requests → Disaggregation separates prefill from decode → Consistent hashing routes to cached host → CI/CD benchmarks validate every diff.