Quality assurance · Production

Lessons From Red Teaming 100 Generative AI Products at Microsoft

The problem

As Microsoft's AI product portfolio expanded rapidly, the volume and scope of AI red teaming grew beyond what fully manual testing could handle, requiring automation to assess safety and security risks across an increasing number of GenAI systems.

First attempt

Fully manual red teaming became impractical at scale, and gradient-based attack methods proved computationally expensive while typically requiring full model access that commercial AI systems do not provide.

Workflow diagram · grounded in source
1
Impact-first operation scoping
trigger
“The first step in an AI red teaming operation is to determine which vulnerabilities to target. While the Impact component of the AIRT ontology is depicted at the end of our ontology, it serves as an excellent starting point for this deci…”
2
AIRT ontology threat modeling
validation
“As attacks and failure modes increase in complexity, it is helpful to model their key components. Based on our experience red teaming over 100 GenAI products for a wide range of risks, we developed an ontology to do exactly that.”
3
PyRIT automated attack execution
ai_action
“PyRIT provides an array of powerful components including prompt datasets, prompt converters (e.g., various encodings), automated attack strategies (including TAP, PAIR, Crescendo, etc.), and even scorers for multimodal outputs”
4
System-level multi-technique attack
ai_action
“many of our operations develop attacks that target end-to-end systems by leveraging multiple techniques”
5
Subject matter expert review
human_review
“it is less reliable in the context of highly specialized domains like medicine, cybersecurity, and CBRN, which can be accurately evaluated only by subject matter experts (SMEs). In multiple operations, we have relied on SMEs to help us a…”
6
Red teaming insights feed benchmarks
feedback_loop
“safety concerns identified by AI red teaming can inform the development of new benchmarks”
Reported outcome

Microsoft's AIRT has red teamed over 100 GenAI products using PyRIT automation combined with human expertise, enabling the team to identify impactful vulnerabilities more quickly and cover more of the risk landscape than a fully manual approach.

Reported metrics
GenAI products red teamedover 100
Reported stack
PyRITMITRE ATT&CK®GPT-4
Source
https://arxiv.org/html/2501.07238v1
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Microsoft's AIRT has red teamed over 100 GenAI products using PyRIT automation combined with human expertise, enabling the team to identify impactful vulnerabilities more quickly and cover more of the risk landscape t…

What tools did this team use?

PyRIT, MITRE ATT&CK®, GPT-4.

What results were reported?

GenAI products red teamed: over 100 (source-reported, not independently verified).

What failed first in this deployment?

Fully manual red teaming became impractical at scale, and gradient-based attack methods proved computationally expensive while typically requiring full model access that commercial AI systems do not provide.

How is this quality assurance AI workflow structured?

Impact-first operation scoping → AIRT ontology threat modeling → PyRIT automated attack execution → System-level multi-technique attack → Subject matter expert review → Red teaming insights feed benchmarks.

Related quality assurance cases