Quality assurance · Production

Meta's Automated Compliance Hardening (ACH) tool uses LLMs to enable scalable mutation testing for compliance

The problem

Compliance at Meta relied on manual processes that were error-prone and hard to scale, while mutation testing — described as the most powerful form of software testing — faced five major barriers (scalability, unrealistic mutants, equivalent mutants, computational cost, and overstretching) that prevented its deployment at scale in large industrial codebases.

Workflow diagram · grounded in source
1
Engineer describes mutant
trigger
“Through simple, plain-text prompts where engineers describe the mutant to test, ACH makes this process intuitive and reliable”
2
LLM generates targeted mutants
ai_action
“ACH system uses LLMs to generate fewer, more realistic, and highly specific mutants targeted at particular fault classes (e.g., privacy faults)”
3
Equivalence Detector filters mutants
validation
“ACH features an LLM-based Equivalence Detector agent that is often capable of judging whether a mutant is equivalent to the original code. In our own research and testing with ACH we found that when combined with simple static analysis p…”
4
ACH generates unit tests
ai_action
“ACH also automatically generates unit tests that kill the mutants, so engineers only ever need to look at tests and, if they wish, mutants that are guaranteed to be non-equivalent”
5
Engineers review generated tests
human_review
“privacy engineers at Meta accepted 73% of the generated tests, with 36% judged as privacy relevant”
6
Tests catch compliance faults
output
“preventing them from reaching production, even if they're reintroduced in future code changes”
Reported outcome

ACH was deployed for privacy testing across Facebook, Instagram, WhatsApp, and Meta's wearables platforms.
Privacy engineers accepted 73% of generated tests. The equivalence detector achieved precision of 0.95 and recall of 0.96 with simple preprocessing, transforming historically time-consuming compliance processes into systems that save engineer and developer time.

Reported metrics
Generated tests accepted by engineers73%
Generated tests judged as privacy relevant36%
Equivalence detection precision (base)0.79
Equivalence detection recall (base)0.47
Show all 8 reported metrics
generated tests accepted by engineers73%
generated tests judged as privacy relevant36%
equivalence detection precision (base)0.79
equivalence detection recall (base)0.47
equivalence detection precision (with preprocessing)0.95
equivalence detection recall (with preprocessing)0.96
compliance process complexitygreatly simplify risk assessments, reduce cognitive load for developers
engineer time on compliancesave engineer and developer time while also enhancing compliance
Reported stack
ACHLLMsKotlin
Source
https://engineering.fb.com/2025/09/30/security/llms-are-the-key-to-mutation-testing-and-better-compliance/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

ACH was deployed for privacy testing across Facebook, Instagram, WhatsApp, and Meta's wearables platforms.

What tools did this team use?

ACH, LLMs, Kotlin.

What results were reported?

Generated tests accepted by engineers: 73%; Generated tests judged as privacy relevant: 36%; Equivalence detection precision (base): 0.79; Equivalence detection recall (base): 0.47 (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Engineer describes mutant → LLM generates targeted mutants → Equivalence Detector filters mutants → ACH generates unit tests → Engineers review generated tests → Tests catch compliance faults.

Related quality assurance cases