Back office ops · Production

Snowflake achieves 16x embedding inference throughput improvement with Arctic Inference optimizations

The problem

When Snowflake profiled embedding models running on vLLM, GPU utilization was far worse than a PyTorch-native implementation could achieve, with the embed function accounting for only 10% of compute time while 90% was spent on CPU tasks from tokenization and serialization bottlenecks.

First attempt

vLLM's sequential tokenization-then-inference design left the GPU idle during tokenization, and Python Protobuf serialization over gRPC lacked SIMD vectorization and suffered from GIL contention, together consuming 90% of total processing time.

Workflow diagram · grounded in source
1
Embedding request via gRPC
trigger
“When vLLM receives the embedding requests with prompts as strings”
2
Disaggregated pre-tokenization
ai_action
“we disaggregated tokenization and inference into a two-stage pipeline. Instead of sending raw text, we pretokenize inputs and pass token IDs directly to vLLM. This enables pipeline parallelism; tokenization and inference can run in paral…”
3
Multi-replica GPU inference
ai_action
“we can run multiple replicas of the same model on a single GPU. These replicas can serve inference requests concurrently and thus increase inference throughput given the same number of GPUs”
4
Little-endian byte serialization
output
“We significantly reduced gRPC response latency by encoding the output list of floats (embedding) as raw bytes in little-endian”
Reported outcome

After three optimizations—little-endian byte serialization, disaggregated tokenization, and multi-replica GPU execution—Snowflake achieved 16x higher throughput for short sequences and 4.2x for long sequences versus vLLM, a 3x improvement in Snowflake Cortex AI delivering 230,000 tokens per second, and 16x cost savings versus vLLM on H200 hardware.

Reported metrics
embedding throughput improvement vs vLLM (short sequences, 50 tokens)16x
embedding throughput improvement vs vLLM (long sequences, 512 tokens)4.2x
embedding throughput improvement vs TEI (short sequences)2.4x
throughput improvement in Snowflake Cortex AI3x
Show all 9 reported metrics
embedding throughput improvement vs vLLM (short sequences, 50 tokens)16x
embedding throughput improvement vs vLLM (long sequences, 512 tokens)4.2x
embedding throughput improvement vs TEI (short sequences)2.4x
throughput improvement in Snowflake Cortex AI3x
sustained throughput in Snowflake Cortex AI230,000 tokens per second
cost savings vs vLLM (short sequences, open source benchmark)16x
GPU inference share of total compute time (baseline vLLM)10%
CPU overhead share of total compute time (baseline vLLM)90%
monthly token processing volumetrillions of tokens per month
Reported stack
vLLMArctic InferencegRPCNumPysnowflake-arctic-embed-m-v1.5Text Embeddings Inference (TEI)Cortex AICortex Search
Source
https://www.snowflake.com/en/engineering-blog/embedding-inference-arctic-16x-faster/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After three optimizations—little-endian byte serialization, disaggregated tokenization, and multi-replica GPU execution—Snowflake achieved 16x higher throughput for short sequences and 4.2x for long sequences versus v…

What tools did this team use?

vLLM, Arctic Inference, gRPC, NumPy, snowflake-arctic-embed-m-v1.5, Text Embeddings Inference (TEI), Cortex AI, Cortex Search.

What results were reported?

embedding throughput improvement vs vLLM (short sequences, 50 tokens): 16x; embedding throughput improvement vs vLLM (long sequences, 512 tokens): 4.2x; embedding throughput improvement vs TEI (short sequences): 2.4x; throughput improvement in Snowflake Cortex AI: 3x (source-reported, not independently verified).

What failed first in this deployment?

vLLM's sequential tokenization-then-inference design left the GPU idle during tokenization, and Python Protobuf serialization over gRPC lacked SIMD vectorization and suffered from GIL contention, together consuming 90…

How is this back office ops AI workflow structured?

Embedding request via gRPC → Disaggregated pre-tokenization → Multi-replica GPU inference → Little-endian byte serialization.

Related back office ops cases