Meta's ACH uses LLM-based mutation testing to harden platforms against privacy and compliance regressions
Traditional automated test generation only sought to increase code coverage rather than targeting specific faults, and mutation testing required engineers to manually write tests with no guarantee those tests would catch the generated mutants, making the process painstaking and difficult to scale.
Earlier rule-based mutant generation produced mutants that were not realistic in terms of representing actual concerns, and engineers still had to manually write the tests with no guarantee those tests would catch the automatically-generated mutants.
ACH has been applied to Facebook Feed, Instagram, Messenger, and WhatsApp; engineers found it useful for hardening code against specific concerns; the approach generates mutants and tests very efficiently and with a high level of accuracy.
Frequently asked questions
What did this team achieve with this AI workflow?
ACH has been applied to Facebook Feed, Instagram, Messenger, and WhatsApp; engineers found it useful for hardening code against specific concerns; the approach generates mutants and tests very efficiently and with a h…
What tools did this team use?
ACH, LLMs.
What results were reported?
Test and mutant generation efficiency: very efficiently and with a high level of accuracy; Impact on regression hardening: significant impact on hardening against future regressions; Developer cognitive load: reduce cognitive load for developers (source-reported, not independently verified).
What failed first in this deployment?
Earlier rule-based mutant generation produced mutants that were not realistic in terms of representing actual concerns, and engineers still had to manually write the tests with no guarantee those tests would catch the…
How is this quality assurance AI workflow structured?
Engineer describes bug concerns → LLM generates relevant mutants → LLM generates catching tests → Verifiable assurance validation.