Back office ops · Production

Introducing Griffin 2.0: Instacart's Next-Gen ML Platform

The problem

Griffin 1.0's CLI and GitHub PR-based interfaces imposed a steep learning curve, lacked standardization, were only vertically scalable, and created a fragmented user experience by forcing engineers to switch between multiple platforms.

First attempt

The MLFlow-based model registry in Griffin 1.0 could not handle the required query scalability, and the fire-and-forget CLI approach made it difficult to retrieve metadata or manage training-serving lineage for production deployments.

Workflow diagram · grounded in source
1
Feature definition in UI
trigger
“MLEs start by defining their features in the "Feature Sources" part of the UI. We support both "Batch Feature Sources" that use SQL queries and "Real Time features" that support Flink SQL/Flink Scala Code to create features.”
2
Workflow submission
trigger
“MLEs navigate to the "Workflows" section in the UI to submit a new workflow for training, evaluation, and scoring of their ML Models. Griffin then executes this workflow based on the specified cadence in the definition.”
3
Distributed ML training
ai_action
“we leveraged Ray for our ML Training Platform to enable horizontally scaled ML training in a distributed computing environment”
4
Validation gates
validation
“Griffin also incorporates validation at different stages, allowing us to identify and rectify errors earlier in the process. This feature aids in cost savings by preventing the execution of actual jobs on our computing infrastructure whe…”
5
Model registry storage
output
“If model training and evaluation are successful, the model is stored in the Griffin Model Registry”
6
Real-time inference endpoint
output
“Griffin users can create a new "endpoint" to host a real-time machine learning service within our service infrastructure”
Reported outcome

Griffin 2.0 replaced CLI and PR-based workflows with a unified web UI and REST API, enabled distributed training and LLM fine-tuning via Ray on Kubernetes, drastically reduced inference service setup time, and achieved substantial latency optimization for real-time inference.

Reported metrics
ML application growthtripling the number of ML applications within a year
Inference service setup timedrastically reduced
Real-time inference latencysubstantial latency optimization
Cost savings from early error detectionaids in cost savings
Reported stack
DockerRayTensorFlowLightGBMMLFlowDatadogAirflowTerraformBentoMLFlink SQLTwirpAWS ECS
Source
https://tech.instacart.com/introducing-griffin-2-0-instacarts-next-gen-ml-platform-b7331e73b8d7
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Griffin 2.0 replaced CLI and PR-based workflows with a unified web UI and REST API, enabled distributed training and LLM fine-tuning via Ray on Kubernetes, drastically reduced inference service setup time, and achieve…

What tools did this team use?

Docker, Ray, TensorFlow, LightGBM, MLFlow, Datadog, Airflow, Terraform, BentoML, Flink SQL.

What results were reported?

ML application growth: tripling the number of ML applications within a year; Inference service setup time: drastically reduced; Real-time inference latency: substantial latency optimization; Cost savings from early error detection: aids in cost savings (source-reported, not independently verified).

What failed first in this deployment?

The MLFlow-based model registry in Griffin 1.0 could not handle the required query scalability, and the fire-and-forget CLI approach made it difficult to retrieve metadata or manage training-serving lineage for produc…

How is this back office ops AI workflow structured?

Feature definition in UI → Workflow submission → Distributed ML training → Validation gates → Model registry storage → Real-time inference endpoint.

Related back office ops cases