Workflow · Production

Training a Deep Learning Speech Synthesis Model on AWS Spot Instances Using Spotty

The problem

Training deep learning models on AWS requires manually managing GPU instances, Docker environments, volumes, and spot instance lifecycles — a complex and expensive setup without dedicated tooling.

Workflow diagram · grounded in source
1
Create Docker environment
integration
“Spotty trains models inside a Docker container. So we need to either find a publicly available Docker image that satisfies the model's requirements, or create a new Dockerfile with a proper environment.”
2
Write Spotty configuration
integration
“Create the spotty.yaml file in the root directory of the project. It consists of 3 sections: project, instance and scripts.”
3
Launch Spot Instance
trigger
“Start a Spot Instance with the Docker container: $ spotty start Once the instance is up and running, you will see its IP address.”
4
Preprocess training data
integration
“preprocess: downloads the dataset and prepares it for a training”
5
Train deep learning model
ai_action
“Once the preprocessing is done, train the model. Run the "train" script: $ spotty run train On a "p2.xlarge" instance it will probably take around 8–9 days to reach 120 thousand steps.”
6
Monitor training with TensorBoard
validation
“Start the TensorBoard using the "tensorboard" script: $ spotty run tensorboard TensorBoard will be running on the port 6006.”
7
Stop instance and snapshot volumes
output
“When you're stopping the instance, Spotty automatically creates snapshots of the volumes. When you start an instance next time, it will restore the snapshots automatically.”
Reported outcome

Using Spotty on AWS Spot Instances can save up to 70% of compute costs and significantly reduces environment setup time, making GPU-based model training accessible to everyone on the team with a few commands.

Reported metrics
compute cost savings via Spot Instancesup to 70%
estimated training duration on p2.xlarge to 120K stepsaround 8–9 days
Reported stack
SpottyTensorFlowDockerTacotron 2TensorBoardJupyter NotebooktmuxAWS CLIS3AWS
Source
https://engineering.zalando.com/posts/2018/11/train-deep-learning-models-aws.html
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Using Spotty on AWS Spot Instances can save up to 70% of compute costs and significantly reduces environment setup time, making GPU-based model training accessible to everyone on the team with a few commands.

What tools did this team use?

Spotty, TensorFlow, Docker, Tacotron 2, TensorBoard, Jupyter Notebook, tmux, AWS CLI, S3, AWS.

What results were reported?

compute cost savings via Spot Instances: up to 70%; estimated training duration on p2.xlarge to 120K steps: around 8–9 days (source-reported, not independently verified).

How is this workflow AI workflow structured?

Create Docker environment → Write Spotty configuration → Launch Spot Instance → Preprocess training data → Train deep learning model → Monitor training with TensorBoard → Stop instance and snapshot volumes.