Finance ops · Production

Nubank fine-tunes customer transaction foundation models with joint fusion for improved benchmark AUC

The problem

Nubank needed to tailor pre-trained transaction-based foundation models to specific downstream tasks, and to incorporate tabular features such as bureau information alongside sequential transaction embeddings in a jointly-optimised way rather than training them separately.

First attempt

Late fusion trained embeddings separately from tabular features, yielding suboptimal performance. Initial DNN-based DCNv2 models showed -0.40% performance versus GBT baselines, and GBTs are not differentiable and therefore incompatible with joint fusion.

Workflow diagram · grounded in source
1
Pre-train on transaction data
ai_action
“self-supervised learning (pre-training) can produce general (unsupervised) embeddings that represent a customer's behavior from transaction data”
2
Supervised fine-tuning with prediction head
ai_action
“we take a pre-trained model and add a linear layer, called the prediction head, to predict the given label, such as a binary classification, multi-class, or regression target. The input to this prediction head is the final token embeddin…”
3
Embed tabular features via DCNv2
ai_action
“we use the DCNv2 to process the embedded tabular features and project the result into a low-dimensional embedding”
4
Joint fusion and final prediction
output
“This feature embedding is concatenated with the transformer-based embedding, and a multi-layer perceptron makes the final prediction”
Reported outcome

Supervised fine-tuning achieved a 1.68% relative improvement in AUC across benchmark tasks, and joint fusion demonstrated further advantage over late fusion, with the DCNv2-based model consistently and reliably beating GBT baselines after incorporating numerical embeddings and regularisation.

Reported metrics
AUC improvement from fine-tuning1.68% relative improvement in AUC
Initial DNN vs GBT performance gap-0.40%
Reported stack
DCNv2XGBoostLightGBM
Source
https://building.nubank.com/fine-tuning-transaction-user-models/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Supervised fine-tuning achieved a 1.68% relative improvement in AUC across benchmark tasks, and joint fusion demonstrated further advantage over late fusion, with the DCNv2-based model consistently and reliably beatin…

What tools did this team use?

DCNv2, XGBoost, LightGBM.

What results were reported?

AUC improvement from fine-tuning: 1.68% relative improvement in AUC; Initial DNN vs GBT performance gap: -0.40% (source-reported, not independently verified).

What failed first in this deployment?

Late fusion trained embeddings separately from tabular features, yielding suboptimal performance.

How is this finance ops AI workflow structured?

Pre-train on transaction data → Supervised fine-tuning with prediction head → Embed tabular features via DCNv2 → Joint fusion and final prediction.

Related finance ops cases