Ecommerce ops · Production

Instacart builds Maple to enable large-scale LLM batch processing across engineering teams

The problem

Instacart's engineering teams needed to run millions of LLM calls for catalog enrichment, fulfillment routing, and search relevance workflows, but real-time LLM provider APIs were not designed for that scale, causing rate limiting, duplicated infrastructure code across teams, and growing cost pressure.

First attempt

Before Maple, rate-limited real-time LLM calls introduced delays, each team independently wrote batch processing code from scratch, and pipelines lacked reusability requiring code modifications for every new use case.

Workflow diagram · grounded in source
1
Team submits file and prompt
trigger
“Maple accepts a CSV or Parquet file and a prompt as input”
2
Maple batches and encodes input
integration
“Batching: Splits large input files into smaller batches. Encoding/Decoding: Automates conversions to and from the LLM batch file format”
3
Route through AI Gateway
routing
“Maple proxies requests through our AI Gateway, another internal Instacart service which acts as a centralized abstraction layer for communicating with multiple LLM providers”
4
LLM processes prompt batches
ai_action
“prompt batches are dispatched to external LLM providers, and results flow back through the same path”
5
Retry failed tasks
feedback_loop
“Maple will retry these infinitely by default by constructing a new batch with the failed tasks”
6
Merge results into output file
output
“It then matches responses to inputs, creating per-batch Parquet result files. Finally, all batch results are combined into a single output file, mirroring the input format”
Reported outcome

Maple saves up to 50% on LLM costs compared to real-time calls, reduced many processes from hundreds of thousands of dollars per year to just thousands, and has become part of the backbone of Instacart's AI infrastructure supporting 10M+ prompt jobs.

Reported metrics
LLM cost savings vs real-time callsup to 50%
Annual process cost reductionreduced from hundreds of thousands of dollars per year to just thousands of dollars per year
Maximum prompt job scale10M+
Average LLM batch processing speed2.6 tasks per second
Show all 6 reported metrics
LLM cost savings vs real-time callsup to 50%
Annual process cost reductionreduced from hundreds of thousands of dollars per year to just thousands of dollars per year
Maximum prompt job scale10M+
Average LLM batch processing speed2.6 tasks per second
Batch completion timemost batches complete in under 12 hours
Parquet file size compression vs CSVup to 25x
Reported stack
MapleTemporalS3PythonPyArroworjsonAI GatewayCost Tracker
Source
https://tech.instacart.com/simplifying-large-scale-llm-processing-across-instacart-with-maple-63df4508d5be
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Maple saves up to 50% on LLM costs compared to real-time calls, reduced many processes from hundreds of thousands of dollars per year to just thousands, and has become part of the backbone of Instacart's AI infrastruc…

What tools did this team use?

Maple, Temporal, S3, Python, PyArrow, orjson, AI Gateway, Cost Tracker.

What results were reported?

LLM cost savings vs real-time calls: up to 50%; Annual process cost reduction: reduced from hundreds of thousands of dollars per year to just thousands of dollars per year; Maximum prompt job scale: 10M+; Average LLM batch processing speed: 2.6 tasks per second (source-reported, not independently verified).

What failed first in this deployment?

Before Maple, rate-limited real-time LLM calls introduced delays, each team independently wrote batch processing code from scratch, and pipelines lacked reusability requiring code modifications for every new use case.

How is this ecommerce ops AI workflow structured?

Team submits file and prompt → Maple batches and encodes input → Route through AI Gateway → LLM processes prompt batches → Retry failed tasks → Merge results into output file.

Related ecommerce ops cases