Back office ops · Production

Distributed Machine Learning at Instacart: training thousands of models with Ray

The problem

Instacart's legacy Celery-based distributed task queue could not scale efficiently as ML use cases grew: worker CPU utilization was only 10–15%, queues accumulated 300–1k+ tasks leading to long wait times, and managing Python dependencies across model types was difficult.

First attempt

The Celery-based legacy system over-provisioned worker nodes (which still sat idle over 60% of the day), could not increase concurrency without upscaling hardware, and was too complex to replicate locally for testing.

Workflow diagram · grounded in source
1
ML application submitted
trigger
“users can package prototyping code snippets on their development environment and launch them on remote AWS EKS hosts through an internal launcher API that abstracts Ray Cluster access through Ray Job API”
2
Dedicated Ray Cluster provisioned
integration
“each Fulfillment ML application is launched as an independent Ray job associated with a dedicated Ray Cluster to handle all its zone level training jobs”
3
Parallel zone-level jobs launched
integration
“it's typical to launch as many as thousands of parallel training jobs to cover all zones in a single model experiment”
4
Concurrent model training
ai_action
“allocating just 2 CPUs per zonal training job is sufficient. As a result, we are able to create more concurrent Ray workers on the same instance”
5
Production pipeline automation
integration
“when users are ready to automate their code in production pipelines (i.e. Airflow), they can use the same set of APIs to launch their containerized application to production Ray Clusters”
Reported outcome

After migrating to Ray-based distributed ML, CPU utilization rose from 10–15% to up to 80%, and end-to-end completion time for a production fulfillment model dropped from approximately 4 hours to 20 minutes, with concurrent workers rising from 10 to 70+.

Reported metrics
CPU utilization — legacy system10% to 15%
CPU utilization — Ray systemup to 80%
End-to-end completion time — legacy~4 hours
end-to-end completion time — Ray system20 minutes
Show all 10 reported metrics
CPU utilization — legacy system10% to 15%
CPU utilization — Ray systemup to 80%
end-to-end completion time — legacy~4 hours
end-to-end completion time — Ray system20 minutes
concurrent workers — legacy system10
concurrent workers — Ray system70+
queue idle time — legacy systemover 60% of the day
tasks in queue — legacy system peak300 to 1k+
parallel training tasks in case study1.5k
compute resource costreduces costs on computation resources
Reported stack
RayGriffinCeleryAirflowRay Job APIRay Core APIsRay AIRRay ServeKubernetesAWS EKS
Source
https://tech.instacart.com/distributed-machine-learning-at-instacart-4b11d7569423
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After migrating to Ray-based distributed ML, CPU utilization rose from 10–15% to up to 80%, and end-to-end completion time for a production fulfillment model dropped from approximately 4 hours to 20 minutes, with conc…

What tools did this team use?

Ray, Griffin, Celery, Airflow, Ray Job API, Ray Core APIs, Ray AIR, Ray Serve, Kubernetes, AWS EKS.

What results were reported?

CPU utilization — legacy system: 10% to 15%; CPU utilization — Ray system: up to 80%; End-to-end completion time — legacy: ~4 hours; end-to-end completion time — Ray system: 20 minutes (source-reported, not independently verified).

What failed first in this deployment?

The Celery-based legacy system over-provisioned worker nodes (which still sat idle over 60% of the day), could not increase concurrency without upscaling hardware, and was too complex to replicate locally for testing.

How is this back office ops AI workflow structured?

ML application submitted → Dedicated Ray Cluster provisioned → Parallel zone-level jobs launched → Concurrent model training → Production pipeline automation.

Related back office ops cases