Incident management · Production

How Meta detects and mitigates silent data corruptions across its AI hardware fleet

The problem

Silent Data Corruptions (SDCs) — hardware errors that cause miscomputation without leaving detectable traces — significantly threaten AI training and inference reliability at Meta. Over 66% of training interruptions stem from hardware failures, and SDC rates have risen to about one fault per thousand devices as silicon density in accelerators has increased.

Workflow diagram · grounded in source
1
SDC fault occurs silently
trigger
“Silent errors or silent data corruptions (SDCs) occur when hardware miscomputes without leaving detectable traces, leading applications to consume incorrect results.”
2
Fleetscanner periodic benchmarks
validation
“Fleetscanner captures performance outliers at scale with targeted micro-benchmarks for identifying hardware defects. Tests are scheduled periodically, covering the entire fleet every 45 to 60 days.”
3
Ripple co-located workload detection
validation
“Ripple co-locates with workloads, executing tests in milliseconds to seconds, allowing fleet-wide coverage in days.”
4
Hardware Sentinel analytical detection
validation
“This novel, test-and-architecture-agnostic approach evaluates application exceptions in kernel space. It identifies core-based anomalies as silent data corruption without requiring test allocations, operating solely in the analytical pla…”
5
Reductive triage for fault isolation
validation
“This strategy involves conducting a binary search with mini-training iterations on progressively smaller cluster sizes to isolate NaN propagation. The goal is to identify a small cluster that replicates the NaN issue, allowing the offend…”
6
Quarantine and resume from checkpoint
output
“A reconstituted cluster with new nodes can then resume training from a saved checkpoint.”
Reported outcome

Meta deployed three complementary SDC detection mechanisms — Fleetscanner, Ripple, and Hardware Sentinel — fully productionized at scale across AI and non-AI infrastructure.
Hardware Sentinel outperforms testing-based methods by 41% across architectures, applications, and data centers.

Reported metrics
Training interruptions due to hardware failuresover 66%
current SDC fault rate in acceleratorsone fault per thousand devices
Hardware Sentinel performance improvement over testing-based methods41%
Fleetscanner full fleet coverage frequencyevery 45 to 60 days
Show all 5 reported metrics
training interruptions due to hardware failuresover 66%
current SDC fault rate in acceleratorsone fault per thousand devices
Hardware Sentinel performance improvement over testing-based methods41%
Fleetscanner full fleet coverage frequencyevery 45 to 60 days
Ripple fleet-wide detection speedfleet-wide coverage in days
Reported stack
FleetscannerRippleHardware SentinelPyTorchServiceLab
Source
https://engineering.fb.com/2025/07/22/data-infrastructure/how-meta-keeps-its-ai-hardware-reliable/?utm_source=substack&utm_medium=email
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Meta deployed three complementary SDC detection mechanisms — Fleetscanner, Ripple, and Hardware Sentinel — fully productionized at scale across AI and non-AI infrastructure.

What tools did this team use?

Fleetscanner, Ripple, Hardware Sentinel, PyTorch, ServiceLab.

What results were reported?

Training interruptions due to hardware failures: over 66%; current SDC fault rate in accelerators: one fault per thousand devices; Hardware Sentinel performance improvement over testing-based methods: 41%; Fleetscanner full fleet coverage frequency: every 45 to 60 days (source-reported, not independently verified).

How is this incident management AI workflow structured?

SDC fault occurs silently → Fleetscanner periodic benchmarks → Ripple co-located workload detection → Hardware Sentinel analytical detection → Reductive triage for fault isolation → Quarantine and resume from checkpoint.

Related incident management cases