quality_assurance · saas · workflow
GitHub Security Lab uses LLM taskflows to triage CodeQL security alerts and discover real-world vulnerabilities
Triaging security alerts is highly repetitive because false positives are caused by patterns obvious to human auditors but too fuzzy for traditional static analysis tools to encode as formal code patterns.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · CodeQL alert generation
The GitHub Security Lab periodically runs CodeQL queries against selected open source repositories to generate code scanning alerts.
Tools used
GitHub Security Lab Taskflow AgentCodeQLClaude Sonnet 3.5MCP serversGitHub Issue
Outcome
Using LLM taskflows, the GitHub Security Lab quickly triaged a large number of CodeQL alerts and discovered approximately 30 real-world vulnerabilities since August, with results remaining fairly accurate even without automated validation.
What failed first
Attempting complex multi-step tasks within a single prompt context caused tasks to be skipped and instructions to go unfollowed; placing information-gathering logic directly in LLM prompts (rather than MCP server tools) produced inconsistent results due to LLM non-determinism.
Results
Volume~30
Running sincesince August
Grounding & classification
Source type: technical build writeup
26 fields verified against source quotes.
agentic workflowai agentsummarizationcode diff prfailure mode describedhuman review describedmetric backednamed customerproduction runtime claimedtools describedworkflow describedsoftwareaccuracy improvementtime savedtechnical build writeupquality assuranceticket triageagentic task executionextract classify route