Quality assurance · Production

How GitHub is experimenting with LLMs to evolve GitHub Copilot

The problem

Developers spend significant time searching documentation for answers and struggle with pull request descriptions and CLI commands—areas GitHub identified as underexplored opportunities for AI assistance.

First attempt

An internal study of the early Copilot for Pull Requests feature did not go well because presenting AI output as a comment rather than an editable suggestion caused developers to distrust and reject the outputs.

Workflow diagram · grounded in source
1
Developer submits pull request
trigger
“a developer could submit their pull request and the AI model would generate a description and walkthrough of the code”
2
LLM generates PR description
ai_action
“the AI model would generate a description and walkthrough of the code in the first comment to provide important context for the reviewer”
3
Suggestion presented for editing
human_review
“Instead of a comment, we put it as a suggestion to the developer that let them get a preview of what the description of their pull request could look like that they could then edit”
4
Documentation retrieval and answer composition
ai_action
“we could use the retrieval engine to search a large corpus of documentation, and then compose those search results into a prompt that elicits better, more topical answers based on the documentation”
5
Answer with references delivered
output
“we built in the capability for our answers to provide references or links to other documentation”
6
Developer describes CLI task
trigger
“natural language prompts to describe what you wanted to do in the command line”
7
LLM generates command with explanation
ai_action
“quickly ask for and receive their desired shell commands, including a breakdown that explains each part of the command”
8
Developer verifies via explanation
validation
“It's also a security feature because you can read in natural language whether the command will change files you didn't expect to change”
Reported outcome

After pivoting to a suggestion-based UX, developer feedback shifted positively.
For Copilot for Docs, developers tolerated imperfect answers when references allowed evaluation. Copilot for CLI brought AI-powered command generation with structured explanations to the terminal.

Reported metrics
developer tolerance for imperfect AI docs outputdevelopers didn't mind if the output wasn't always perfectly correct
Reported stack
GPT-4GitHub CopilotGitHub Copilot ChatCopilot for Pull RequestsCopilot for DocsCopilot for CLILLMsvector database
Source
https://github.blog/ai-and-ml/llms/how-were-experimenting-with-llms-to-evolve-github-copilot/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

After pivoting to a suggestion-based UX, developer feedback shifted positively.

What tools did this team use?

GPT-4, GitHub Copilot, GitHub Copilot Chat, Copilot for Pull Requests, Copilot for Docs, Copilot for CLI, LLMs, vector database.

What results were reported?

developer tolerance for imperfect AI docs output: developers didn't mind if the output wasn't always perfectly correct (source-reported, not independently verified).

What failed first in this deployment?

An internal study of the early Copilot for Pull Requests feature did not go well because presenting AI output as a comment rather than an editable suggestion caused developers to distrust and reject the outputs.

How is this quality assurance AI workflow structured?

Developer submits pull request → LLM generates PR description → Suggestion presented for editing → Documentation retrieval and answer composition → Answer with references delivered → Developer describes CLI task → LLM generates command with explanation → Developer verifies via explanation.

Related quality assurance cases