Quality assurance · Production

How GitHub evaluates AI models for GitHub Copilot: offline evaluation methodology

The problem

With many AI models available from proprietary and open-source providers, GitHub needed a rigorous evaluation process to determine which models to support in GitHub Copilot, since newer models do not always perform better for specific use cases.

Workflow diagram · grounded in source

Pre-production evaluation trigger

trigger

“the tests we run before making any change to our production environment”

Containerized repo code tests

validation

“We have a collection of around 100 containerized repositories that have passed a battery of CI tests. We modify those repositories to fail the tests, and then see whether the model can modify the codebase to once again pass the failing t…”

LLM-as-judge chat evaluation

ai_action

“We have a collection of more than 1,000 technical questions we use to evaluate the quality of a model's chat capabilities. Some of these are simple true-or-false questions that we can easily evaluate automatically. But for more complex q…”

LLM evaluator auditing

human_review

“We also routinely audit the outputs of this LLM in evaluation scenarios to make sure it's working correctly”

Results pipeline

integration

“Results are then piped in and out of systems like Apache Kafka and Microsoft Azure, and we leverage a variety of dashboards to explore the data”

Daily production model monitoring

feedback_loop

“We also run these tests against our production models every day. If we see degradation, we do some auditing to find out why the models aren't performing as well as they used to. Sometimes we need to make changes, for example modify some …”

Model support decision

output

“GitHub's goal is to create the best quality, most responsible AI coding assistant possible, and that guides the decision we have to make about which models to support within the product”

Reported outcome

GitHub built an offline evaluation system with more than 4,000 automated tests, around 100 containerized repositories, and more than 1,000 technical chat questions, enabling rapid model iteration without product code changes.

Reported metrics

Offline automated testsmore than 4,000

Containerized test repositoriesaround 100

Technical chat evaluation questionsmore than 1,000

Reported stack

GitHub ActionsApache KafkaMicrosoft Azure

Source

https://github.blog/ai-and-ml/generative-ai/how-we-evaluate-models-for-github-copilot/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

GitHub built an offline evaluation system with more than 4,000 automated tests, around 100 containerized repositories, and more than 1,000 technical chat questions, enabling rapid model iteration without product code…

What tools did this team use?

GitHub Actions, Apache Kafka, Microsoft Azure.

What results were reported?

Offline automated tests: more than 4,000; Containerized test repositories: around 100; Technical chat evaluation questions: more than 1,000 (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

Pre-production evaluation trigger → Containerized repo code tests → LLM-as-judge chat evaluation → LLM evaluator auditing → Results pipeline → Daily production model monitoring → Model support decision.