Quality assurance · Production

GitHub Security Lab uses LLM taskflows to triage CodeQL security alerts and discover real-world vulnerabilities

The problem

Triaging security alerts is highly repetitive because false positives are caused by patterns obvious to human auditors but too fuzzy for traditional static analysis tools to encode as formal code patterns.

First attempt

Attempting complex multi-step tasks within a single prompt context caused tasks to be skipped and instructions to go unfollowed; placing information-gathering logic directly in LLM prompts (rather than MCP server tools) produced inconsistent results due to LLM non-determinism.

Workflow diagram · grounded in source

CodeQL alert generation

trigger

“The GitHub Security Lab periodically runs a set of CodeQL queries against a selected set of open source repositories”

Information collection

ai_action

“we instruct the LLM (examples are provided in the Triage examples section below) to collect relevant information about the alert, which takes into account the threat model and human knowledge of the alert in general”

Audit for false positives

ai_action

“the LLM goes through the information gathered and performs a list of specific checks to reject alert results that turned out to be false positives”

Bug report generation

output

“the next step is to create a bug report using the information gathered, as well as the reasoning for the decision at the audit stage”

Report validation

validation

“After the report is written, we instruct the LLM to check the report to ensure that all the relevant information is contained in the report, as well as the consistency of the information”

GitHub Issue creation

integration

“then we open a GitHub Issue to track the alert”

Human issue review

human_review

“After reviewing and checking the issues, we often find that there are causes for false positives that we missed during the auditing process”

Dismissal reasons feedback

feedback_loop

“When the agent analyzes similar cases in the future, it will be aware of all the past analysis stored in those issues and alert dismissal reasons, incorporate this new intelligence in its knowledge base, and be more effective at detectin…”

Reported outcome

Using LLM taskflows, the GitHub Security Lab quickly triaged a large number of CodeQL alerts and discovered approximately 30 real-world vulnerabilities since August, with results remaining fairly accurate even without automated validation.

Reported metrics

Real-world vulnerabilities discovered~30

Triage result accuracyfairly accurate

Reported stack

GitHub Security Lab Taskflow AgentCodeQLClaude Sonnet 3.5MCP serversGitHub Issue

Source

https://github.blog/security/ai-supported-vulnerability-triage-with-the-github-security-lab-taskflow-agent/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Using LLM taskflows, the GitHub Security Lab quickly triaged a large number of CodeQL alerts and discovered approximately 30 real-world vulnerabilities since August, with results remaining fairly accurate even without…

What tools did this team use?

GitHub Security Lab Taskflow Agent, CodeQL, Claude Sonnet 3.5, MCP servers, GitHub Issue.

What results were reported?

Real-world vulnerabilities discovered: ~30; Triage result accuracy: fairly accurate (source-reported, not independently verified).

What failed first in this deployment?

How is this quality assurance AI workflow structured?

CodeQL alert generation → Information collection → Audit for false positives → Bug report generation → Report validation → GitHub Issue creation → Human issue review → Dismissal reasons feedback.