Quality assurance · Production

A Practical Approach to Verifying Code at Scale

The problem

As autonomous coding systems generate increasing volumes of code, thorough human review becomes impractical, raising the risk that AI-written code introduces severe bugs and vulnerabilities — whether accidentally or intentionally.

First attempt

Earlier code review approaches provided only a diff with limited surrounding context, causing them to miss important codebase-wide interactions. CriticGPT was designed for simpler tasks and was not suitable for production deployment.

Workflow diagram · grounded in source

PR triggers automated review

trigger

“within OpenAI every PR is automatically reviewed, many engineers run /review in the Codex CLI before pushing”

Repo-wide context and execution

ai_action

“providing repository access and code execution abilities to a GPT-5 model and found that it results in a stronger reviewer, catching more critical issues and raising fewer false alarms”

Reviewer surfaces actionable issues

output

“compare how often the reviewer surfaces actionable issues on Codex-generated PRs versus human-written PRs”

Author addresses reviewer findings

human_review

“authors address it with a code change in 52.7% of cases (often assisted by Codex)”

Real-world deployment feedback

feedback_loop

“Deploying the reviewer both internally and across external GitHub traffic has been crucial for turning a strong lab prototype into a dependable safety layer under real engineering constraints”

Reported outcome

The agentic code reviewer is now a core part of OpenAI's engineering workflow, handling over 100k external PRs per day as of October 2025, with authors making code changes in response to 52.7% of comments and over 80% of comment reactions being positive.

Reported metrics

PRs commented on by reviewer (Codex-generated)36%

Comments on Codex-generated PRs resulting in code changes46%

Comments on human-generated PRs resulting in code changes53%

Reviewer comments addressed by authors52.7%

Show all 7 reported metrics

PRs commented on by reviewer (Codex-generated)36%

Comments on Codex-generated PRs resulting in code changes46%

Comments on human-generated PRs resulting in code changes53%

Reviewer comments addressed by authors52.7%

External PRs handled per daymore than 100k

Positive comment reactionsover 80%

Escaped defectsfewer cases where merged PRs later need bug-fix follow-up work

Reported stack

gpt-5-codexgpt-5.1-codex-maxCodex CLICodexCriticGPTGitHub

Source

https://alignment.openai.com/scaling-code-verification/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The agentic code reviewer is now a core part of OpenAI's engineering workflow, handling over 100k external PRs per day as of October 2025, with authors making code changes in response to 52.7% of comments and over 80%…

What tools did this team use?

gpt-5-codex, gpt-5.1-codex-max, Codex CLI, Codex, CriticGPT, GitHub.

What results were reported?

PRs commented on by reviewer (Codex-generated): 36%; Comments on Codex-generated PRs resulting in code changes: 46%; Comments on human-generated PRs resulting in code changes: 53%; Reviewer comments addressed by authors: 52.7% (source-reported, not independently verified).

What failed first in this deployment?

Earlier code review approaches provided only a diff with limited surrounding context, causing them to miss important codebase-wide interactions.

How is this quality assurance AI workflow structured?

PR triggers automated review → Repo-wide context and execution → Reviewer surfaces actionable issues → Author addresses reviewer findings → Real-world deployment feedback.