Ecommerce ops · Production

Building the Intent Engine: How Instacart Revamped Query Understanding with LLMs

The problem

Instacart's traditional ML-based Query Understanding system struggled with long-tail searches, noisy labeled data, and a fragmented architecture of multiple independent bespoke models that introduced inconsistencies and slowed development.

First attempt

The legacy query rewrite system covered only 50% of search traffic, and the category classification model produced irrelevant suggestions due to noisy conversion data while lacking contextual understanding for nuanced queries.

Workflow diagram · grounded in source
1
User search query submitted
Trigger
When people search for items on Instacart, the Query Understanding pipeline processes their typed phrases.
▾ source quote
“When people search for items on Instacart, they don't always type perfectly worded phrases”
2
Cache hit/miss routing
Routing
The system routes each query based on whether it results in a cache hit or miss.
▾ source quote
“The system's flow, shown in the diagram below, is determined simply by a cache-hit”
3
Offline RAG context injection
Ai action
Data pipelines retrieve and inject Instacart-specific context such as conversion history and catalog data directly into the LLM prompt.
▾ source quote
“We build data pipelines that retrieve and inject Instacart-specific context, such as conversion history and catalog data, directly into the prompt”
4
Post-processing guardrails
Validation
A guardrail computes semantic similarity scores between query embeddings and predicted category paths, discarding pairs below a relevance threshold.
▾ source quote
“This filter computes a semantic similarity score between the embeddings of the original query and the LLM's predicted category path, discarding any pair that falls below our relevance threshold”
5
Real-time fine-tuned model inference
Ai action
Cache-miss tail queries are routed to a fine-tuned Llama3-8B model that is fast and cost-effective for live inference.
▾ source quote
“When a user's query results in a cache miss (indicating a long-tail query), it is routed to our real-time model. This is a language model with a much smaller backbone (like Llama3–8B) that is fast and cost-effective for live inference”
6
Enhanced search results delivered
Output
The system delivers a consistent, high-quality experience for virtually any query a user types.
▾ source quote
“This allows us to deliver a consistent, high-quality experience for virtually any query a user types”
Reported outcome

The LLM-powered system increased query rewrite coverage to over 95% with 90%+ precision, reduced average scroll depth by 6%, and cut user complaints for tail query search results by 50%, with the system now serving millions of cold-start queries weekly.

Reported metrics
Query rewrite coverageover 95%
Query rewrite precision90%+
Legacy query rewrite coverage50%
Average scroll depth reduction6%
Show all 12 reported metrics
query rewrite coverageover 95%
query rewrite precision90%+
legacy query rewrite coverage50%
average scroll depth reduction6%
user complaints for tail query search results50%
out-of-the-box inference latencynearly 700ms
optimized inference latency target300ms
latency reduction from quantization10%
latency reduction from adapter merging30%
queries requiring real-time inference2%
search quality improvement scope (bottom percentile)bottom 2% of queries
cold-start queries served weeklymillions of cold-start queries weekly
Reported stack
LLMsRAGLoRAA100 GPUH100 GPUs
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://tech.instacart.com/building-the-intent-engine-how-instacart-is-revamping-query-understanding-with-llms-3ac8051ae7ac
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The LLM-powered system increased query rewrite coverage to over 95% with 90%+ precision, reduced average scroll depth by 6%, and cut user complaints for tail query search results by 50%, with the system now serving mi…

What tools did this team use?

LLMs, RAG, LoRA, A100 GPU, H100 GPUs.

What results were reported?

Query rewrite coverage: over 95%; Query rewrite precision: 90%+; Legacy query rewrite coverage: 50%; Average scroll depth reduction: 6% (source-reported, not independently verified).

What failed first in this deployment?

The legacy query rewrite system covered only 50% of search traffic, and the category classification model produced irrelevant suggestions due to noisy conversion data while lacking contextual understanding for nuanced…

How is this ecommerce ops AI workflow structured?

User search query submitted → Cache hit/miss routing → Offline RAG context injection → Post-processing guardrails → Real-time fine-tuned model inference → Enhanced search results delivered.

Related ecommerce ops cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Ecommerce ops cases →