Recruiting · Production
Accelerating LLM inference with speculative decoding: Lessons from LinkedIn's Hiring Assistant
The problem
LinkedIn's Hiring Assistant required low-latency conversational responses for recruiters, but LLM generation was slow when processing thousands of tokens from long job descriptions and candidate profiles.
Workflow diagram · grounded in source
1
Recruiter submits request
trigger
“recruiters expect conversational responses in seconds, not minutes”
2
Ingest structured input tokens
ai_action
“Hiring Assistant routinely ingests thousands of input tokens (such as job descriptions and candidate profiles) and generates a 1,000+ tokens of structured analysis”
3
N-gram speculation drafts tokens
ai_action
“this is a model-agnostic and purely statistical approach that uses patterns from the existing input to predict the next few tokens. Its drafting cost is low, and works best when outputs contain rephrasings or structured text”
4
Parallel token verification
validation
“the verification step uses the target model's probabilities to accept or reject proposed tokens, preserving the original distribution”
5
Structured match analysis output
output
“Hiring Assistant classifies the strength of the match (e.g. skills, seniority, domain experience) and explains its reasoning with grounded evidence”
Reported outcome
Applying n-gram speculative decoding achieved nearly 4× higher throughput and an average 66% reduction in P90 end-to-end latency without any quality degradation.
Reported metrics
Throughput increasenearly 4×
P90 end-to-end latency reduction66%
Quality degradationwithout any quality degradation
Reported stack
vLLM
Source
https://www.linkedin.com/blog/engineering/ai/accelerating-llm-inference-with-speculative-decoding-lessons-from-linkedins-hiring-assistant
Read source ↗Frequently asked questions
What did this team achieve with this AI workflow?
Applying n-gram speculative decoding achieved nearly 4× higher throughput and an average 66% reduction in P90 end-to-end latency without any quality degradation.
What tools did this team use?
vLLM.
What results were reported?
Throughput increase: nearly 4×; P90 end-to-end latency reduction: 66%; Quality degradation: without any quality degradation (source-reported, not independently verified).
How is this recruiting AI workflow structured?
Recruiter submits request → Ingest structured input tokens → N-gram speculation drafts tokens → Parallel token verification → Structured match analysis output.