Ecommerce ops · Production

From Print to Digital: Making Weekly Flyers Shoppable at Instacart Through Computer Vision and LLMs

The problem

Instacart's manual flyer digitization process required 3–4 hours per flyer and became unsustainable as dozens of retailers adopted weekly flyers, creating hundreds of hours of work each week and requiring retailers to submit flyers well ahead of their go-live date.

First attempt

Off-the-shelf ML solutions each had critical limitations: multimodal LLMs produced imprecise bounding boxes for complex flyers, traditional segmentation and contour detection models generated excessive noise, and existing solutions like FoodSAM fell short of handling the breadth and variety of retail flyer products.

Workflow diagram · grounded in source
1
Retailer uploads flyer
trigger
“enabling retailers to upload their weekly and monthly promotions”
2
Phase 1: Image segmentation
ai_action
“Phase 1: Image Segmentation — Identifying and extracting bounding boxes around each product or deal on the flyer. This phase uses a custom algorithm built on Meta's Segment Anything Model (SAM), enhanced with techniques to handle the uni…”
3
OCR text extraction
ai_action
“We use PaddleOCR, a state-of-the-art OCR model, to identify text on images and extract text from the image.”
4
LLM product query generation
ai_action
“We pass this output along with the original image to an LLM model to separate deal images into queries and attributes for each product, as some deals might contain multiple products. We have noticed that adding the output of the OCR mode…”
5
ANN search and LLM ranking
ai_action
“using our Search Approximate Nearest Neighbors Algorithm (ANN), we are able to find the top 10–15 products that match the product query embedding. To complete our matches, we retrieve product attributes stored in our Instacart feature st…”
6
Human review and finalization
human_review
“What previously required 3–4 hours of manual effort per flyer, drawing bounding boxes and matching products can now be reviewed and finalized in just 30 minutes.”
7
Interactive shoppable flyer published
output
“we've transformed static flyer images into interactive shopping experiences”
Reported outcome

The two-phase pipeline reduced flyer processing from 3–4 hours to under 30 minutes — a 10x reduction — while achieving 75–90% bounding box accuracy and 95% product recall in the top position, enabling Instacart to scale digitization across its entire retail network.

Reported metrics
Flyer processing time (pipeline)less than 30 minutes
Flyer processing time (manual, previous)3–4 hours
Processing time reduction10x
Bounding box extraction accuracy75–90%
Show all 7 reported metrics
flyer processing time (pipeline)less than 30 minutes
flyer processing time (manual, previous)3–4 hours
processing time reduction10x
bounding box extraction accuracy75–90%
product identification recall (top position)95%
find rate increase from OCR output15%
simple flyer accuracy (multimodal LLM method)~90%
Reported stack
SAMLLM
Source
https://tech.instacart.com/from-print-to-digital-making-weekly-flyers-shoppable-at-instacart-through-computer-vision-and-llms-739cae1f5629
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The two-phase pipeline reduced flyer processing from 3–4 hours to under 30 minutes — a 10x reduction — while achieving 75–90% bounding box accuracy and 95% product recall in the top position, enabling Instacart to sca…

What tools did this team use?

SAM, LLM.

What results were reported?

Flyer processing time (pipeline): less than 30 minutes; Flyer processing time (manual, previous): 3–4 hours; Processing time reduction: 10x; Bounding box extraction accuracy: 75–90% (source-reported, not independently verified).

What failed first in this deployment?

Off-the-shelf ML solutions each had critical limitations: multimodal LLMs produced imprecise bounding boxes for complex flyers, traditional segmentation and contour detection models generated excessive noise, and exis…

How is this ecommerce ops AI workflow structured?

Retailer uploads flyer → Phase 1: Image segmentation → OCR text extraction → LLM product query generation → ANN search and LLM ranking → Human review and finalization → Interactive shoppable flyer published.

Related ecommerce ops cases