Back office ops · Production

Yelp scales LLM-based search query understanding to millions of daily searches in production

The problem

Yelp's pre-LLM query understanding systems were fragmented — several different systems stitched together — and often lacked the intelligence to handle nuanced user intent, spelling errors, ambiguous locations, and complex semantic phrase expansions.

First attempt

Traditional Named Entity Recognition and text similarity models could not handle the nuances of Yelp's query understanding needs, including multi-concept queries, ambiguous locations, and creative semantic expansions.

Workflow diagram · grounded in source
1
User search query submitted
trigger
“From the moment a user enters a search query to when we present a list of results, understanding the user's intent is crucial for meeting their needs.”
2
RAG context augmentation
ai_action
“we augment the input query text with the names of businesses that have been viewed for that query. This helps the model learn and distinguish the many facets of business names from common topics, locations, and misspellings.”
3
LLM segmentation and spell correction
ai_action
“spell-correction and segmentation can be done together by a sufficiently powerful model, so we added a meta tag to mark spell corrected sections and decided to combine these two tasks into a single prompt”
4
LLM phrase expansion for highlights
ai_action
“we used the LLM to generate terms that are suitable to be highlighted”
5
Offline and A/B evaluation
validation
“We then integrated the cached LLM responses to the existing system and performed offline and online (A/B) evaluations.”
6
Fine-tune GPT4o-mini for offline scale
ai_action
“Fine tune a smaller model (GPT4o-mini) that we can run offline at the scale of tens of millions, and utilize this as a pre-computed cache to support that vast bulk of all traffic”
7
BERT and T5 for real-time long-tail queries
ai_action
“at Yelp, we have used BERT and T5 to serve as our real time LLM model. These models are optimized for speed and efficiency”
8
Pre-computed results served from cache
output
“we scaled to 95% of traffic by pre-computing snippet expansions for those queries using OpenAI's batch calls. The generated outputs were quality checked and uploaded to our query understanding datastores”
9
CTR signals refine ranking
feedback_loop
“we used CTR signals for the relevant expanded phrases to further refine our ranking models”
Reported outcome

Yelp successfully deployed LLM-based query understanding to production, scaling review highlights to 95% of traffic through pre-computed results, achieving up to 100x cost savings compared to direct GPT-4 usage, and increasing Session / Search CTR across platforms.

Reported metrics
cost savings vs direct GPT-4 usageup to a 100x savings in cost
Traffic coverage via pre-computation95%
Session / Search CTRincreased Session / Search CTR
Remaining traffic not pre-computed5%
Show all 5 reported metrics
cost savings vs direct GPT-4 usageup to a 100x savings in cost
traffic coverage via pre-computation95%
Session / Search CTRincreased Session / Search CTR
remaining traffic not pre-computed5%
impact on tail querieshigher for less common queries in the tail query range
Reported stack
LLMsGPT-4GPT3RAGo1-minio1-previewGPT4o-miniBERTT5OpenAI
Source
https://engineeringblog.yelp.com/2025/02/search-query-understanding-with-LLMs.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Yelp successfully deployed LLM-based query understanding to production, scaling review highlights to 95% of traffic through pre-computed results, achieving up to 100x cost savings compared to direct GPT-4 usage, and i…

What tools did this team use?

LLMs, GPT-4, GPT3, RAG, o1-mini, o1-preview, GPT4o-mini, BERT, T5, OpenAI.

What results were reported?

cost savings vs direct GPT-4 usage: up to a 100x savings in cost; Traffic coverage via pre-computation: 95%; Session / Search CTR: increased Session / Search CTR; Remaining traffic not pre-computed: 5% (source-reported, not independently verified).

What failed first in this deployment?

Traditional Named Entity Recognition and text similarity models could not handle the nuances of Yelp's query understanding needs, including multi-concept queries, ambiguous locations, and creative semantic expansions.

How is this back office ops AI workflow structured?

User search query submitted → RAG context augmentation → LLM segmentation and spell correction → LLM phrase expansion for highlights → Offline and A/B evaluation → Fine-tune GPT4o-mini for offline scale → BERT and T5 for real-time long-tail queries → Pre-computed results served from cache → CTR signals refine ranking.

Related back office ops cases