Compliance monitoring · Production

GitHub builds Copilot secret scanning to detect leaked passwords with AI

The problem

Regular expressions, while effective for detecting provider-formatted secrets, could not handle the nuanced and varied structures of generic passwords, generating excessive noise for security teams and developers.

First attempt

An early iteration using GPT-3.5-Turbo with few-shot prompting worked in offline evaluation but failed for unconventional file types and structures encountered in actual customer repositories.

Workflow diagram · grounded in source
1
Git push or history scan
trigger
“Secret scanning not only scans incoming Git pushes, but also your entire Git history on all branches.”
2
LLM prompt construction
ai_action
“a request to a large language model (LLM), expressed through an LLM prompt consisting of: General information about the type of vulnerability, in this case passwords. The source code location and contents of the file where we believe the…”
3
AI context analysis
ai_action
“We now detect generic passwords with GitHub Copilot, using AI to analyze context—such as the usage and location of a potential secret—to limit noise and deliver relevant alerts that are critical to the health and security of your reposit…”
4
Workload-aware capacity routing
routing
“the most impactful change came from creating a workload-aware request management system that allowed us to maximize and equitably share LLM capacity against the variety of different workloads we run during scans”
5
Mirror testing validation
validation
“We implemented a mirror testing framework that ran our prompt and filtering changes against a subset of repositories that participated in our public preview. Rescanning these repositories with our latest improvements allowed us to assess…”
6
Security alert delivery
output
“deliver relevant alerts that are critical to the health and security of your repositories”
Reported outcome

After iterative improvements, Copilot secret scanning reached general availability in October 2024, achieving up to a 94% reduction in false positives in some organizations, and is now detecting passwords on nearly 35% of all GitHub Secret Protection repositories.

Reported metrics
False positive reduction94%
GitHub Secret Protection repositories with password detectionnearly 35%
Detections and false positiveshuge drop in detections and false positives
Reported stack
GitHub CopilotGPT-3.5-TurboGPT-4GPT-4-TurboGPT-4o-miniMetaReflection
Source
https://github.blog/engineering/platform-security/finding-leaked-passwords-with-ai-how-we-built-copilot-secret-scanning/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After iterative improvements, Copilot secret scanning reached general availability in October 2024, achieving up to a 94% reduction in false positives in some organizations, and is now detecting passwords on nearly 35…

What tools did this team use?

GitHub Copilot, GPT-3.5-Turbo, GPT-4, GPT-4-Turbo, GPT-4o-mini, MetaReflection.

What results were reported?

False positive reduction: 94%; GitHub Secret Protection repositories with password detection: nearly 35%; Detections and false positives: huge drop in detections and false positives (source-reported, not independently verified).

What failed first in this deployment?

An early iteration using GPT-3.5-Turbo with few-shot prompting worked in offline evaluation but failed for unconventional file types and structures encountered in actual customer repositories.

How is this compliance monitoring AI workflow structured?

Git push or history scan → LLM prompt construction → AI context analysis → Workload-aware capacity routing → Mirror testing validation → Security alert delivery.

Related compliance monitoring cases