Quality assurance · Production

How GitHub Next is experimenting with LLMs to evolve GitHub Copilot

The problem

Developers spent enormous time searching through documentation and discovering CLI commands, with no intelligent tooling to help them find answers or understand pull request context quickly.

First attempt

An initial internal study of the pull request description feature did not go well because developers were concerned the AI would be wrong; serving AI content as a comment rather than a suggestion created a poor UX that undermined adoption even when the content itself was correct.

Workflow diagram · grounded in source
1
Developer submits pull request
trigger
“a developer could submit their pull request and the AI model would generate a description and walkthrough of the code in the first comment to provide important context for the reviewer”
2
AI generates PR description as suggestion
ai_action
“we put it as a suggestion to the developer that let them get a preview of what the description of their pull request could look like that they could then edit”
3
Developer reviews and edits suggestion
human_review
“we moved it to a suggestion system, and all of a sudden the feedback changed to 'wow, these are helpful suggestions.'”
4
Retrieval engine searches documentation corpus
ai_action
“we could use the retrieval engine to search a large corpus of documentation, and then compose those search results into a prompt that elicits better, more topical answers based on the documentation”
5
Answer with references returned
output
“we built in the capability for our answers to provide references or links to other documentation. We found that when we deployed it, the feedback we received was that developers didn't mind if the output wasn't always perfectly correct i…”
6
Developer describes CLI command in natural language
trigger
“use natural language prompts to describe what you wanted to do in the command line”
7
LLM generates shell command with breakdown
ai_action
“quickly ask for and receive their desired shell commands, including a breakdown that explains each part of the command—without ever needing to search the web for answers”
Reported outcome

Pivoting to a suggestion-based UX for pull request descriptions transformed negative internal feedback into positive reception; adding reference links in Copilot for Docs made developers tolerant of imperfect AI outputs; and shipping early for real human feedback was established as a core development principle.

Reported metrics
developer reception of PR description featurefeedback changed to 'wow, these are helpful suggestions'
developer tolerance of imperfect Copilot for Docs outputsdevelopers didn't mind if the output wasn't always perfectly correct
Reported stack
GPT-4GitHub CopilotCopilot for Pull RequestsCopilot for DocsCopilot for CLIvector database
Source
https://github.blog/2023-12-06-how-were-experimenting-with-llms-to-evolve-github-copilot/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Pivoting to a suggestion-based UX for pull request descriptions transformed negative internal feedback into positive reception; adding reference links in Copilot for Docs made developers tolerant of imperfect AI outpu…

What tools did this team use?

GPT-4, GitHub Copilot, Copilot for Pull Requests, Copilot for Docs, Copilot for CLI, vector database.

What results were reported?

developer reception of PR description feature: feedback changed to 'wow, these are helpful suggestions'; developer tolerance of imperfect Copilot for Docs outputs: developers didn't mind if the output wasn't always perfectly correct (source-reported, not independently verified).

What failed first in this deployment?

An initial internal study of the pull request description feature did not go well because developers were concerned the AI would be wrong; serving AI content as a comment rather than a suggestion created a poor UX tha…

How is this quality assurance AI workflow structured?

Developer submits pull request → AI generates PR description as suggestion → Developer reviews and edits suggestion → Retrieval engine searches documentation corpus → Answer with references returned → Developer describes CLI command in natural language → LLM generates shell command with breakdown.

Related quality assurance cases