Recruiting · Production

LinkedIn builds a GenAI application tech stack: LangChain framework, prompt management, skill inversion, and conversational memory

The problem

LinkedIn lacked a shared, scalable foundation for GenAI development. Early products relied on fragmented Java (online) and Python (offline) stacks that required substantial ongoing effort to stay in sync, manual string interpolation for prompts that was error-prone and unscalable, and per-product re-implementation of common skills.

First attempt

A single shared Java midtier for all GenAI products became an operational bottleneck and had to be split into multiple use-case-specific services, each requiring mirrored Python logic. Staying on Java for online serving proved a suboptimal long-term choice as the GenAI ecosystem evolved primarily in Python.

Workflow diagram · grounded in source

User query enters framework

trigger

“each incoming query is processed independently of other interactions”

Conversational memory retrieval

integration

“Since LLM context windows are limited and increasing input tokens has cost/latency implications, it was important to retrieve only the relevant parts of the conversation rather than the entire conversation history. To enable this, we nee…”

Prompt resolution via Jinja templates

integration

“We standardized the use of the Jinja template language for authoring prompts, and built a Java prompt resolution library to avoid common string interpolation bugs”

Trust and Responsible AI check

validation

“a centralized GenAI proxy which offered functionalities like Trust and Responsible AI checks, seamless support for new models and model versions”

LLM inference

ai_action

“Our GenAI applications have increasingly started depending on our AI platform, which is built atop open source frameworks like PyTorch, DeepSpeed, vLLM and provides a robust and highly scalable fine-tuning and serving infrastructure. In …”

Skill invocation via registry

ai_action

“A dynamic LangChain tool that retrieves skill definitions from the skill registry and invokes the actual skill with the supplied arguments, eliminating developer specified skills in prompts and giving significantly larger agency to LLMs”

Response streamed to user

output

“incremental response streaming to reduce user-perceived latency”

Reported outcome

LinkedIn now has a standardized GenAI application framework built as a thin LangChain wrapper with integrated prompt management, conversational memory on the LinkedIn messaging stack, and a centralized skill registry.
Fine-tuned Llama models achieve comparable quality to commercial models at much lower costs and latencies, and many existing applications have been migrated to the new stack.

Reported metrics

Model cost and latency vs commercial foundational modelsmuch lower costs and latencies

Reported stack

LangChainJinjaAzure OpenAI servicePyTorchDeepSpeedvLLMLlamaCouchbaseEspressoBingOpenAI Chat Completions API

Source

https://www.linkedin.com/blog/engineering/generative-ai/behind-the-platform-the-journey-to-create-the-linkedin-genai-application-tech-stack

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

LinkedIn now has a standardized GenAI application framework built as a thin LangChain wrapper with integrated prompt management, conversational memory on the LinkedIn messaging stack, and a centralized skill registry.

What tools did this team use?

LangChain, Jinja, Azure OpenAI service, PyTorch, DeepSpeed, vLLM, Llama, Couchbase, Espresso, Bing.

What results were reported?

Model cost and latency vs commercial foundational models: much lower costs and latencies (source-reported, not independently verified).

What failed first in this deployment?

A single shared Java midtier for all GenAI products became an operational bottleneck and had to be split into multiple use-case-specific services, each requiring mirrored Python logic.

How is this recruiting AI workflow structured?

User query enters framework → Conversational memory retrieval → Prompt resolution via Jinja templates → Trust and Responsible AI check → LLM inference → Skill invocation via registry → Response streamed to user.