Workflow · Production

Cost Effective Deployment of DeepSeek R1 with Intel® Xeon® 6 CPU on SGLang

The problem

DeepSeek R1's massive model size and unique MoE architecture normally requires many high-end AI accelerators to deploy; Intel PyTorch Team proposed a CPU-only solution at fractional cost as an alternative.

First attempt

Existing CPU tools like llama.cpp processed MoE experts sequentially rather than in parallel, leading to substantially slower inference for large MoE models.

Workflow diagram · grounded in source
1
Prompt split into prefix and extend
trigger
“SGLang divides query sequence into two parts: prefix – historical sequence in which attention is a rectangle; extend – newly added prompt in which attention is a lower triangle”
2
Prefill via Extend Attention
ai_action
“The CPU kernel exactly maps to Flash Attention V2 algorithm”
3
Decode via Flash Decoding
ai_action
“We implemented Flash Decoding algorithm that chunks KV sequence into multiple splits to increase the degree of parallelism”
4
MoE expert parallel computation
ai_action
“run argsort on topk_ids and keep indices of activations in sorted_ids according to expert ids”
5
Multi-NUMA all-reduce synchronization
integration
“We also implemented communication primitives, e.g. all reduce, all gather, based on a shared memory approach”
Reported outcome

The optimized SGLang CPU backend achieves substantially faster LLM inference on Intel Xeon CPUs, with memory bandwidth efficiency of 85% for INT8 MoE and significantly reduced time-to-first-token and time-per-output-token compared to llama.cpp; the work has been upstreamed into the SGLang main branch.

Reported metrics
TTFT speedup vs llama.cpp (range across models)6-14x
TPOT speedup vs llama.cpp (range across models)2-4x
INT8 MoE memory bandwidth efficiency85%
effective memory bandwidth on MRDIMMs1.45TB/s
Show all 13 reported metrics
TTFT speedup vs llama.cpp (range across models)6-14x
TPOT speedup vs llama.cpp (range across models)2-4x
INT8 MoE memory bandwidth efficiency85%
effective memory bandwidth on MRDIMMs1.45TB/s
MLA kernel optimization speedup vs vanillaapproximately 1.9x
TPOT improvement from KV buffer fusion12%
communication overhead share of end-to-end time3%
TTFT speedup — DeepSeek-R1-671B INT813.0x
TPOT speedup — DeepSeek-R1-671B INT82.5x
TTFT speedup — Qwen3-235B-A22B INT814.4x
TPOT speedup — Qwen3-235B-A22B INT84.1x
FP8 inference efficiency relative to INT8approximately 80% to 90% of INT8 implementation
TPOT improvement from graph mode (preliminary)10%
Reported stack
SGLangIntel® Advanced Matrix Extensions (AMX)llama.cppPyTorchAVX512KTransformers
Source
https://lmsys.org/blog/2025-07-14-intel-xeon-optimization/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The optimized SGLang CPU backend achieves substantially faster LLM inference on Intel Xeon CPUs, with memory bandwidth efficiency of 85% for INT8 MoE and significantly reduced time-to-first-token and time-per-output-t…

What tools did this team use?

SGLang, Intel® Advanced Matrix Extensions (AMX), llama.cpp, PyTorch, AVX512, KTransformers.

What results were reported?

TTFT speedup vs llama.cpp (range across models): 6-14x; TPOT speedup vs llama.cpp (range across models): 2-4x; INT8 MoE memory bandwidth efficiency: 85%; effective memory bandwidth on MRDIMMs: 1.45TB/s (source-reported, not independently verified).

What failed first in this deployment?

Existing CPU tools like llama.cpp processed MoE experts sequentially rather than in parallel, leading to substantially slower inference for large MoE models.

How is this workflow AI workflow structured?

Prompt split into prefix and extend → Prefill via Extend Attention → Decode via Flash Decoding → MoE expert parallel computation → Multi-NUMA all-reduce synchronization.