Recruiting · Production

How LinkedIn leveraged vLLM to power GenAI applications at scale

The problem

LinkedIn needed to deploy LLMs at scale across diverse real-time and batch GenAI use cases while meeting strict latency requirements, managing GPU efficiency, and giving internal engineering teams control over performance tuning without modifying engine code.

First attempt

LinkedIn's initial serving stack tightly coupled the server and engine, limiting flexibility. The first offline deployment had limited concurrency suitable only for low-QPS workloads. Traditional NER models for job search were brittle, costly to maintain, and slow to adapt to evolving language and job taxonomies.

Workflow diagram · grounded in source
1
Hirer specifies qualifications
trigger
“take a set of qualifications the hirer has specified, and assist in identifying potential candidates who meet each of those qualifications”
2
vLLM prefix-cached batched inference
ai_action
“over 50% of requests share prefix tokens, making them highly amenable to prefix caching optimizations. vLLM reuses computation for the shared portion of the input instead of recalculating it for every request. This dramatically cuts down…”
3
LLM classifies and explains per candidate
ai_action
“An LLM is able to tackle both of these areas at once, providing both explanations and classifications with high accuracy”
4
Qualified candidates filtered for hirer
output
“the most qualified candidates can be filtered for the hirers”
5
User submits job search query
trigger
“interpret the underlying intent from user inputs, which are often short, ambiguous, and context-dependent”
6
LLM interprets intent and extracts facets
ai_action
“translate this free-form text into structured interpretations and facet suggestions by cohesively analyzing the user's query along with contextual cues, such as profile attributes”
7
Personalized job recommendations returned
output
“deliver more accurate, personalized, and context-aware job recommendations within its dynamic marketplace”
Reported outcome

vLLM now supports more than 50 GenAI use cases at LinkedIn and runs on thousands of hosts.
The v1 engine upgrade saved over 60 GPUs for one workload. Open-source contributions yielded a 7% improvement in Time Per Output Token and an 8% improvement in decoding speed for smaller models.

Reported metrics
GenAI use cases supportedmore than 50
hosts running vLLMthousands of hosts
TPS improvement from scheduler tuning~10%
Throughput under saturation (v1 engine)~1245 tokens/sec
Show all 10 reported metrics
GenAI use cases supportedmore than 50
hosts running vLLMthousands of hosts
TPS improvement from scheduler tuning~10%
throughput under saturation (v1 engine)~1245 tokens/sec
GPU savings from v1 engine upgradeover 60 GPUs saved
Time Per Output Token improvement7%
decoding speed improvement for smaller models8%
Hiring Agent requests sharing prefix tokensover 50%
average tokens per candidate evaluationnearly 1000 on average
AI Job Search p95 latency target<600ms p95
Reported stack
vLLMAsyncLLMEnginePagedAttentionCUDA graphsgRPCOpenAI-compatible APIRed HatUC Berkeley Sky ComputingNVIDIALMCache
Source
https://www.linkedin.com/blog/engineering/ai/how-we-leveraged-vllm-to-power-our-genai-applications?utm_source=substack&utm_medium=email
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

vLLM now supports more than 50 GenAI use cases at LinkedIn and runs on thousands of hosts.

What tools did this team use?

vLLM, AsyncLLMEngine, PagedAttention, CUDA graphs, gRPC, OpenAI-compatible API, Red Hat, UC Berkeley Sky Computing, NVIDIA, LMCache.

What results were reported?

GenAI use cases supported: more than 50; hosts running vLLM: thousands of hosts; TPS improvement from scheduler tuning: ~10%; Throughput under saturation (v1 engine): ~1245 tokens/sec (source-reported, not independently verified).

What failed first in this deployment?

LinkedIn's initial serving stack tightly coupled the server and engine, limiting flexibility.

How is this recruiting AI workflow structured?

Hirer specifies qualifications → vLLM prefix-cached batched inference → LLM classifies and explains per candidate → Qualified candidates filtered for hirer → User submits job search query → LLM interprets intent and extracts facets → Personalized job recommendations returned.

Related recruiting cases