Back office ops · Production

Snowflake achieves 16x embedding inference throughput improvement with Arctic Inference optimizations

The problem

When Snowflake profiled embedding models running on vLLM, GPU utilization was far worse than a PyTorch-native implementation could achieve, with the embed function accounting for only 10% of compute time while 90% was spent on CPU tasks from tokenization and serialization bottlenecks.

First attempt

vLLM's sequential tokenization-then-inference design left the GPU idle during tokenization, and Python Protobuf serialization over gRPC lacked SIMD vectorization and suffered from GIL contention, together consuming 90% of total processing time.

Workflow diagram · grounded in source
1
Embedding request via gRPC
Trigger
Embedding inference requests with prompts as strings arrive at the vLLM service behind a gRPC frontend.
▾ source quote
“When vLLM receives the embedding requests with prompts as strings”
2
Disaggregated pre-tokenization
Ai action
Inputs are pre-tokenized and token IDs are passed directly to vLLM, enabling tokenization to run in parallel with GPU inference for other requests.
▾ source quote
“we disaggregated tokenization and inference into a two-stage pipeline. Instead of sending raw text, we pretokenize inputs and pass token IDs directly to vLLM. This enables pipeline parallelism; tokenization and inference can run in parallel across different requests, even though …”
3
Multi-replica GPU inference
Ai action
Multiple replicas of the same embedding model run concurrently on a single GPU to increase inference throughput by better utilizing available GPU resources.
▾ source quote
“we can run multiple replicas of the same model on a single GPU. These replicas can serve inference requests concurrently and thus increase inference throughput given the same number of GPUs”
4
Little-endian byte serialization
Output
Embedding output vectors are encoded as raw little-endian bytes using NumPy vectorization, significantly reducing gRPC response latency.
▾ source quote
“We significantly reduced gRPC response latency by encoding the output list of floats (embedding) as raw bytes in little-endian”
Reported outcome

After three optimizations—little-endian byte serialization, disaggregated tokenization, and multi-replica GPU execution—Snowflake achieved 16x higher throughput for short sequences and 4.2x for long sequences versus vLLM, a 3x improvement in Snowflake Cortex AI delivering 230,000 tokens per second, and 16x cost savings versus vLLM on H200 hardware.

Reported metrics
embedding throughput improvement vs vLLM (short sequences, 50 tokens)16x
embedding throughput improvement vs vLLM (long sequences, 512 tokens)4.2x
embedding throughput improvement vs TEI (short sequences)2.4x
throughput improvement in Snowflake Cortex AI3x
Show all 9 reported metrics
embedding throughput improvement vs vLLM (short sequences, 50 tokens)16x
embedding throughput improvement vs vLLM (long sequences, 512 tokens)4.2x
embedding throughput improvement vs TEI (short sequences)2.4x
throughput improvement in Snowflake Cortex AI3x
sustained throughput in Snowflake Cortex AI230,000 tokens per second
cost savings vs vLLM (short sequences, open source benchmark)16x
GPU inference share of total compute time (baseline vLLM)10%
CPU overhead share of total compute time (baseline vLLM)90%
monthly token processing volumetrillions of tokens per month
Reported stack
vLLMArctic InferencegRPCNumPysnowflake-arctic-embed-m-v1.5Text Embeddings Inference (TEI)Cortex AICortex Search
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://www.snowflake.com/en/engineering-blog/embedding-inference-arctic-16x-faster/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After three optimizations—little-endian byte serialization, disaggregated tokenization, and multi-replica GPU execution—Snowflake achieved 16x higher throughput for short sequences and 4.2x for long sequences versus v…

What tools did this team use?

vLLM, Arctic Inference, gRPC, NumPy, snowflake-arctic-embed-m-v1.5, Text Embeddings Inference (TEI), Cortex AI, Cortex Search.

What results were reported?

embedding throughput improvement vs vLLM (short sequences, 50 tokens): 16x; embedding throughput improvement vs vLLM (long sequences, 512 tokens): 4.2x; embedding throughput improvement vs TEI (short sequences): 2.4x; throughput improvement in Snowflake Cortex AI: 3x (source-reported, not independently verified).

What failed first in this deployment?

vLLM's sequential tokenization-then-inference design left the GPU idle during tokenization, and Python Protobuf serialization over gRPC lacked SIMD vectorization and suffered from GIL contention, together consuming 90…

How is this back office ops AI workflow structured?

Embedding request via gRPC → Disaggregated pre-tokenization → Multi-replica GPU inference → Little-endian byte serialization.

Related back office ops cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Back office ops cases →