Ecommerce ops · Production

Walmart achieves ~50% tail query cache hit rate with semantic caching and generative AI in e-commerce search

The problem

Traditional exact-match caching fell short for the nuances of human language in e-commerce search queries, and integrating generative AI introduced significant latency and cost challenges.

First attempt

Traditional exact-match caching could not handle the varied ways customers phrase search queries.

Workflow diagram · grounded in source
1
Customer submits search query
trigger
“A query like 'football watch party' returns not just snacks and chips but also party drinks, Super Bowl apparel, and televisions”
2
Semantic cache interprets query meaning
ai_action
“Semantic caching, however, thrives on understanding and interpreting the meaning behind queries, enabling it to handle variations in phrasing with finesse.”
3
Generative AI identifies product groupings
ai_action
“By understanding the intent behind customer queries, the system can present highly relevant product groupings.”
4
Relevant product results returned
output
“returns not just snacks and chips but also party drinks, Super Bowl apparel, and televisions, demonstrating our system's ability to grasp and respond to the nuanced needs of our customers”
Reported outcome

Semantic caching achieved a tail query cache hit rate close to 50%, far exceeding the anticipated 10-20%, and the system can return contextually relevant product groupings for intent-based queries.

Reported metrics
Tail query cache hit rate (actual)close to 50%
Tail query cache hit rate (anticipated)10-20%
Reported stack
LLMs
Source
https://portkey.ai/blog/transforming-e-commerce-search-with-generative-ai-insights-from-walmarts-journey/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Semantic caching achieved a tail query cache hit rate close to 50%, far exceeding the anticipated 10-20%, and the system can return contextually relevant product groupings for intent-based queries.

What tools did this team use?

LLMs.

What results were reported?

Tail query cache hit rate (actual): close to 50%; Tail query cache hit rate (anticipated): 10-20% (source-reported, not independently verified).

What failed first in this deployment?

Traditional exact-match caching could not handle the varied ways customers phrase search queries.

How is this ecommerce ops AI workflow structured?

Customer submits search query → Semantic cache interprets query meaning → Generative AI identifies product groupings → Relevant product results returned.

Related ecommerce ops cases