Back office ops · Production

Recursion operationalizes petabyte-scale deep learning for drug discovery with a custom MLOps pipeline

The problem

Drug discovery is exceptionally time-consuming, costly, and has a 90% clinical trial failure rate. Recursion needed to build target-agnostic models that generalize across diseases while managing petabytes of imaging data and overcoming the lack of adequately-labeled biological data.

Workflow diagram · grounded in source

Cell image data collection

trigger

“Recursion's labs create huge amounts of high-content + high-throughput screening assays. One of the datasets we've used these robotic labs to assemble is a constantly growing dataset of over 19 PB of images of cells”

DL model training via perturbation classification

ai_action

“Our DL models use weakly supervised learning to classify which perturbation was applied to the cells based on the input image. The classification objective is purely a proxy task – the output of each model that is used in downstream appl…”

Large-scale inference for embeddings

ai_action

“Our model evaluation benchmarks require running inference on hundreds of millions of images into the DL model to evaluate the biological soundness of their resulting embeddings”

Benchmark evaluation of embeddings

validation

“Our benchmark suite produces a final report summarizing the results quantifying the biological integrity of the DL model's image embeddings”

Production model selection

human_review

“The report is used by ML scientists and decision makers to determine whether or not a DL model should be selected for production and helps our researchers understand how choices they're making impact model performance, making it easier t…”

Reported outcome

Recursion built a scalable MLOps pipeline enabling ML scientists to train hundreds of DL model variants per month and run inference on hundreds of millions of images, supporting three drug candidates in Phase 2 clinical trials and dozens more in earlier stages.

Reported metrics

Clinical trial failure rate90%

drug candidates in Phase 2 clinical trialsthree

Drug candidates enabled in earlier stagesdozens more candidates in earlier stages

Imaging dataset sizeover 19 PB of images of cells

Show all 8 reported metrics

clinical trial failure rate90%

drug candidates in Phase 2 clinical trialsthree

drug candidates enabled in earlier stagesdozens more candidates in earlier stages

imaging dataset sizeover 19 PB of images of cells

average drug development timeline14 years

inference benchmark imageshundreds of millions of images

DL models trained per monthhundreds of DL models in a month

publicly released dataset~1 TB of our data to the public

Reported stack

CellProfilerDetermined AICodefreshMLFlowGoogle Container RepositoryGoogle Cloud StorageGoogle Kubernetes EngineDockerCell Painting

Source

https://mlops.community/blog/drug-discovery-with-deep-learning-at-recursion

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Recursion built a scalable MLOps pipeline enabling ML scientists to train hundreds of DL model variants per month and run inference on hundreds of millions of images, supporting three drug candidates in Phase 2 clinic…

What tools did this team use?

CellProfiler, Determined AI, Codefresh, MLFlow, Google Container Repository, Google Cloud Storage, Google Kubernetes Engine, Docker, Cell Painting.

What results were reported?

Clinical trial failure rate: 90%; drug candidates in Phase 2 clinical trials: three; Drug candidates enabled in earlier stages: dozens more candidates in earlier stages; Imaging dataset size: over 19 PB of images of cells (source-reported, not independently verified).

How is this back office ops AI workflow structured?

Cell image data collection → DL model training via perturbation classification → Large-scale inference for embeddings → Benchmark evaluation of embeddings → Production model selection.