Finance ops · Production

Ntropy optimizes ML training pipelines with GCP, Flyte, embedding pruning, and self-supervised pretraining

The problem

Ntropy needed faster ML iteration cycles while keeping GPU compute costs reasonable when training models on large financial transaction datasets, with pipelines initially drafted quickly but not optimized for speed.

Workflow diagram · grounded in source
1
Training job initiated with checkpoint recovery
trigger
“Most of Ntropy's heavy training pipelines support checkpoint recovery through simple command line arguments, such as the — from_checkpoint flag”
2
Multi-cloud orchestration via Flyte
integration
“taking data from AWS S3, training on GCP and finally putting training artifacts back to S3. We use Flyte to orchestrate this process”
3
Profiling and bottleneck detection
validation
“We periodically use a profiler to detect bottlenecks in the training pipeline to avoid this”
4
Shared backbone training
ai_action
“it makes sense to reuse the backbone of the first model and only train the output layer of the second model. This is a very straightforward technique, but it can significantly reduce the number of parameters and training time”
5
Embedding pruning
ai_action
“We can reduce the number of parameters by retraining the tokenizer and pruning the embedding layer, removing tokens that are not used in our datasets”
6
Self-supervised pretraining on unlabeled data
ai_action
“we recently started to use our big unlabeled datasets for self-supervised pretraining. No need to reinvent the wheel, we use the masked language model approach”
7
Fine-tuning on labeled dataset
ai_action
“public pretrained model => self-supervised pretraining on unlabeled dataset => fine-tune on custom labeled dataset”
Reported outcome

One pipeline was accelerated by 3x through incremental optimizations; the transaction-labeling model's parameters were reduced to 70% of their original count via embedding pruning, cutting inference time by 20% with no quality degradation.

Reported metrics
Preemptible instance cost savings vs regular instances60–91%
Training pipeline speedup from incremental optimizations3x
Model parameter count after embedding pruning70% of original parameters
Inference time reduction after embedding pruning20%
Reported stack
GCPAWSAWS S3FlyteDockerGitHub ActionsGCRKubeflowPandasHuggingFacelmdbSlackT4A100
Source
https://mlops.community/blog/optimizing-machine-learning-training-pipelines
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

One pipeline was accelerated by 3x through incremental optimizations; the transaction-labeling model's parameters were reduced to 70% of their original count via embedding pruning, cutting inference time by 20% with n…

What tools did this team use?

GCP, AWS, AWS S3, Flyte, Docker, GitHub Actions, GCR, Kubeflow, Pandas, HuggingFace.

What results were reported?

Preemptible instance cost savings vs regular instances: 60–91%; Training pipeline speedup from incremental optimizations: 3x; Model parameter count after embedding pruning: 70% of original parameters; Inference time reduction after embedding pruning: 20% (source-reported, not independently verified).

How is this finance ops AI workflow structured?

Training job initiated with checkpoint recovery → Multi-cloud orchestration via Flyte → Profiling and bottleneck detection → Shared backbone training → Embedding pruning → Self-supervised pretraining on unlabeled data → Fine-tuning on labeled dataset.

Related finance ops cases