Back office ops · Production

How DeepL built next-generation LLMs with FP8 for training and inference

The problem

BF16 training constrained DeepL's ability to scale LLMs to larger parameter counts within practical memory and latency budgets; moving to 8-bit computation was needed to increase throughput and enable more sophisticated models without sacrificing production inference latency.

Workflow diagram · grounded in source
1
LLM pre-training initiation
trigger
“The journey we take with FP8 training and inference starts with the pre-training of our LLMs.”
2
FP8 mixed precision training
ai_action
“We transitioned our existing training code from BF16 to FP8 using NVIDIA Transformer Engine: a training library provided by NVIDIA that accelerates transformer models, and includes support for FP8. Transformer Engine provides essential c…”
3
E4M3 forward / E5M2 backward pass
ai_action
“using E4M3 in the forward pass for training, and then using E5M2 in the backward pass. This effectively means that we're using the format with higher precision for predicting the probability distribution of the next token, and then the f…”
4
Fine-tuning and distillation
ai_action
“After pre-training, we fine-tune our models on certain tasks, distil large models into smaller models, do reinforcement learning, and deploy a set of parallelization strategies so we can make use of the huge number of GPUs that we have.”
5
TensorRT-LLM inference optimization
output
“the heavy lifting on support is carried out by NVIDIA TensorRT-LLM, which is NVIDIA's solution for scalable LLM inference, and which supports FP8. It takes the weights of your model from training and builds an engine to optimize the oper…”
Reported outcome

FP8 accelerated model training by 50% in MFU, ultimately reaching 80% MFU after further optimization, doubled inference throughput at the same latency budget, and enabled translation quality that outperforms previous models by 1.4x for European languages and 1.7x for complex language pairs.

Reported metrics
model training MFU with FP8 (initial)67% MFU
model training acceleration vs BF1650%
Incremental training performance increase over 15 months25%
maximum MFU achieved80% MFU
Show all 7 reported metrics
model training MFU with FP8 (initial)67% MFU
model training acceleration vs BF1650%
incremental training performance increase over 15 months25%
maximum MFU achieved80% MFU
inference throughput improvement vs BF16double the throughput for the same degree of latency
translation quality improvement (European languages)1.4x
translation quality improvement (English-Japanese)1.7x
Reported stack
NVIDIA DGX SuperPODNVIDIA H100 Tensor Core GPUsNVIDIA Transformer EngineNVIDIA TensorRT-LLM
Source
https://www.deepl.com/en/blog/tech/next-generation-llm-fp8-training
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

FP8 accelerated model training by 50% in MFU, ultimately reaching 80% MFU after further optimization, doubled inference throughput at the same latency budget, and enabled translation quality that outperforms previous…

What tools did this team use?

NVIDIA DGX SuperPOD, NVIDIA H100 Tensor Core GPUs, NVIDIA Transformer Engine, NVIDIA TensorRT-LLM.

What results were reported?

model training MFU with FP8 (initial): 67% MFU; model training acceleration vs BF16: 50%; Incremental training performance increase over 15 months: 25%; maximum MFU achieved: 80% MFU (source-reported, not independently verified).

How is this back office ops AI workflow structured?

LLM pre-training initiation → FP8 mixed precision training → E4M3 forward / E5M2 backward pass → Fine-tuning and distillation → TensorRT-LLM inference optimization.

Related back office ops cases