Ecommerce ops · Production

Building the Intent Engine: How Instacart Revamped Query Understanding with LLMs

The problem

Instacart's traditional ML-based Query Understanding system struggled with long-tail searches, noisy labeled data, and a fragmented architecture of multiple independent bespoke models that introduced inconsistencies and slowed development.

First attempt

The legacy query rewrite system covered only 50% of search traffic, and the category classification model produced irrelevant suggestions due to noisy conversion data while lacking contextual understanding for nuanced queries.

Workflow diagram · grounded in source
1
User search query submitted
trigger
“When people search for items on Instacart, they don't always type perfectly worded phrases”
2
Cache hit/miss routing
routing
“The system's flow, shown in the diagram below, is determined simply by a cache-hit”
3
Offline RAG context injection
ai_action
“We build data pipelines that retrieve and inject Instacart-specific context, such as conversion history and catalog data, directly into the prompt”
4
Post-processing guardrails
validation
“This filter computes a semantic similarity score between the embeddings of the original query and the LLM's predicted category path, discarding any pair that falls below our relevance threshold”
5
Real-time fine-tuned model inference
ai_action
“When a user's query results in a cache miss (indicating a long-tail query), it is routed to our real-time model. This is a language model with a much smaller backbone (like Llama3–8B) that is fast and cost-effective for live inference”
6
Enhanced search results delivered
output
“This allows us to deliver a consistent, high-quality experience for virtually any query a user types”
Reported outcome

The LLM-powered system increased query rewrite coverage to over 95% with 90%+ precision, reduced average scroll depth by 6%, and cut user complaints for tail query search results by 50%, with the system now serving millions of cold-start queries weekly.

Reported metrics
Query rewrite coverageover 95%
Query rewrite precision90%+
Legacy query rewrite coverage50%
Average scroll depth reduction6%
Show all 12 reported metrics
query rewrite coverageover 95%
query rewrite precision90%+
legacy query rewrite coverage50%
average scroll depth reduction6%
user complaints for tail query search results50%
out-of-the-box inference latencynearly 700ms
optimized inference latency target300ms
latency reduction from quantization10%
latency reduction from adapter merging30%
queries requiring real-time inference2%
search quality improvement scope (bottom percentile)bottom 2% of queries
cold-start queries served weeklymillions of cold-start queries weekly
Reported stack
LLMsRAGLoRAA100 GPUH100 GPUs
Source
https://tech.instacart.com/building-the-intent-engine-how-instacart-is-revamping-query-understanding-with-llms-3ac8051ae7ac
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The LLM-powered system increased query rewrite coverage to over 95% with 90%+ precision, reduced average scroll depth by 6%, and cut user complaints for tail query search results by 50%, with the system now serving mi…

What tools did this team use?

LLMs, RAG, LoRA, A100 GPU, H100 GPUs.

What results were reported?

Query rewrite coverage: over 95%; Query rewrite precision: 90%+; Legacy query rewrite coverage: 50%; Average scroll depth reduction: 6% (source-reported, not independently verified).

What failed first in this deployment?

The legacy query rewrite system covered only 50% of search traffic, and the category classification model produced irrelevant suggestions due to noisy conversion data while lacking contextual understanding for nuanced…

How is this ecommerce ops AI workflow structured?

User search query submitted → Cache hit/miss routing → Offline RAG context injection → Post-processing guardrails → Real-time fine-tuned model inference → Enhanced search results delivered.

Related ecommerce ops cases