Recruiting · Production

Scaling LLM-based prefill-only ranking systems with SGLang at LinkedIn

The problem

LinkedIn's LLM-based ranking workloads for AI Job Search and AI People Search faced high latency and low throughput because existing serving infrastructure was optimized for generative LLMs rather than prefill-only scoring, causing sequential tokenization, fragmented batch execution, unnecessary decode loops, and strict SLA pressure.

First attempt

Multiple specific failure modes existed in the default SGLang serving path: batch boundaries were lost in ZMQ socket transmission causing fragmented GPU execution; the full decode and sampling loop ran unnecessarily for ranking; per-query prefix KV was recomputed for every candidate item; and Python GC stalls caused 100–300 ms pauses under sustained load.

Workflow diagram · grounded in source
1
Member query triggers ranking
trigger
“we evaluate every member query and rank hundreds of items by jointly reasoning over the query, the member's profile, and job or profile content”
2
Batch tokenization of prompts
ai_action
“allowing all prompts within a single request to be tokenized together using fast tokenizer batch APIs and parallel CPU execution”
3
Scoring-only prefill execution
ai_action
“The model processes the entire prompt once and returns only the final token's logits. There is no iterative token generation, no sampling, and no beam search.”
4
In-batch prefix KV reuse
ai_action
“We compute the prefix KV once using the first prompt in the batch, then intercept the forward pass between KV computation and attention so the remaining items can reuse that KV directly”
5
Ranked results returned
output
“these advancements power AI Job Search and AI People Search to deliver state-of-the-art LLM ranking to millions of members”
Reported outcome

Through staged optimizations to SGLang, text-based ranking throughput increased approximately 3x from 750 to 2,200 items/s/GPU and P99 latency for the scoring path dropped from 6220 ms to 454 ms (13.7x), with the system now powering AI Job Search and AI People Search for millions of LinkedIn members.

Reported metrics
P99 latency — embedding model batch tokenization4583 ms to 464 ms
Average latency reduction from batch send41.5%
Average latency — batch send (absolute)70.39 ms to 41.12 ms
P99 latency — scoring path optimization6220 ms to 454 ms
Show all 8 reported metrics
P99 latency — embedding model batch tokenization4583 ms to 464 ms
average latency reduction from batch send41.5%
average latency — batch send (absolute)70.39 ms to 41.12 ms
P99 latency — scoring path optimization6220 ms to 454 ms
throughput increase — scoring path~25%
throughput increase — multiprocessing optimization~40%
text-based ranking throughput (items/s/GPU)750 -> 2,200
GC stall duration — pre-fix100–300 ms pauses every few seconds
Reported stack
SGLangZMQgRPCH100 GPUs
Source
https://www.linkedin.com/blog/engineering/ai/scaling-llm-based-ranking-systems-with-sglang-at-linkedin
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Through staged optimizations to SGLang, text-based ranking throughput increased approximately 3x from 750 to 2,200 items/s/GPU and P99 latency for the scoring path dropped from 6220 ms to 454 ms (13.7x), with the syst…

What tools did this team use?

SGLang, ZMQ, gRPC, H100 GPUs.

What results were reported?

P99 latency — embedding model batch tokenization: 4583 ms to 464 ms; Average latency reduction from batch send: 41.5%; Average latency — batch send (absolute): 70.39 ms to 41.12 ms; P99 latency — scoring path optimization: 6220 ms to 454 ms (source-reported, not independently verified).

What failed first in this deployment?

Multiple specific failure modes existed in the default SGLang serving path: batch boundaries were lost in ZMQ socket transmission causing fragmented GPU execution; the full decode and sampling loop ran unnecessarily f…

How is this recruiting AI workflow structured?

Member query triggers ranking → Batch tokenization of prompts → Scoring-only prefill execution → In-batch prefix KV reuse → Ranked results returned.

Related recruiting cases