Quality assurance · Production

GitHub Copilot Autofix uses LLM to suggest security vulnerability fixes directly in pull requests

The problem

Security vulnerabilities detected by code scanning required manual developer effort to understand and fix, with no automated mechanism to suggest fixes alongside alerts.

First attempt

The initial approach of asking the LLM to output a standard diff patch directly failed because it exacerbated the model's known arithmetic difficulties, producing incorrect line number computations.

Workflow diagram · grounded in source
1
PR opened or commit pushed
trigger
“The user opens a pull request or pushes a commit.”
2
CodeQL scans repository
validation
“Code scanning runs as usual, as part of an actions workflow or workflow in a third-party CI system, uploading the results in the SARIF format to the code scanning API”
3
Fix generator builds LLM prompt
integration
“We extract information from the alert to construct an LLM prompt consisting of: General information about this type of vulnerability, typically including a general example of the vulnerability and how to fix it, extracted from the CodeQL…”
4
LLM generates code fix
ai_action
“we send the affected code and a description of the problem to a large language model (LLM), asking it to suggest code edits that will fix the problem without changing the functionality of the code”
5
Harmful content filtering
validation
“The LLM response is run through a filtering system which helps prevent certain classes of harmful responses.”
6
Post-processing and error correction
validation
“We employ a variety of post-processing heuristics to detect and correct small errors in the model output. For example, "before" code blocks might not exactly match the original source-code, and line numbers may be slightly off. We implem…”
7
Fix suggestion rendered in PR
output
“The code scanning backend stores the resulting suggestion, making it available for rendering alongside the alert in pull request views”
8
Developer reviews and applies fix
human_review
“Users can commit the suggested fixes directly to the pull request, or edit the suggestions in their local IDE or in a GitHub Codespace”
Reported outcome

Iterative prompt and heuristic refinement tripled the fix success rate while reducing LLM compute requirements by a factor of six; Copilot Autofix is now generally available.

Reported metrics
Fix success ratetripled
LLM compute requirementsa factor of six reduction
Evaluation dataset sizeover 1,400 alerts
Reported stack
CodeQLLLMAzureGitHub CodespaceGitHub CLI
Source
https://github.blog/engineering/platform-security/fixing-security-vulnerabilities-with-ai/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Iterative prompt and heuristic refinement tripled the fix success rate while reducing LLM compute requirements by a factor of six; Copilot Autofix is now generally available.

What tools did this team use?

CodeQL, LLM, Azure, GitHub Codespace, GitHub CLI.

What results were reported?

Fix success rate: tripled; LLM compute requirements: a factor of six reduction; Evaluation dataset size: over 1,400 alerts (source-reported, not independently verified).

What failed first in this deployment?

The initial approach of asking the LLM to output a standard diff patch directly failed because it exacerbated the model's known arithmetic difficulties, producing incorrect line number computations.

How is this quality assurance AI workflow structured?

PR opened or commit pushed → CodeQL scans repository → Fix generator builds LLM prompt → LLM generates code fix → Harmful content filtering → Post-processing and error correction → Fix suggestion rendered in PR → Developer reviews and applies fix.

Related quality assurance cases