Workflow · Production

Inside GitHub: Working with the LLMs behind GitHub Copilot

The problem

General-purpose code generation was considered too difficult because existing models could not reliably produce useful completions, and an early chatbot prototype proved to be an inferior modality compared to in-IDE code completion.

First attempt

A static chatbot prototype for answering coding questions was abandoned in favour of in-IDE completion, and early model versions frequently suggested code in the wrong programming language.

Workflow diagram · grounded in source
1
Developer edits code in IDE
trigger
“The moment we did that and saw how well it worked, the whole static question-and-answer modality was forgotten. This new approach was interactive and it was useful in almost every situation.”
2
Context retrieved from neighboring tabs
ai_action
“we pulled similar texts in from the user's neighboring editor tabs. That was a huge lift in our acceptance rate and characters retained”
3
File path added to prompt
ai_action
“it was much more powerful to put the path of the file at the top of the document”
4
LLM generates code completion
ai_action
“the art of prompt crafting is really all about creating a 'pseudo-document' that will lead the model to a completion that benefits the customer”
5
Codebase fine-tuning personalizes output
ai_action
“we're training the underlying Codex model on a user's specific codebase to provide more focused, customized completions”
6
Acceptance signal feeds model improvement
feedback_loop
“Our greatest challenge right now is to consider why the user rejects or accepts a suggestion”
Reported outcome

By incorporating neighboring-tab context retrieval and file path headers into prompts, and fine-tuning the Codex model on users' codebases, GitHub Copilot achieved a large lift in acceptance rate and characters retained, with the underlying model improving to solve upwards of 90 percent of benchmark coding problems.

Reported metrics
Model benchmark problem-solving rateupwards of 90 percent
Initial model problem-solving rateabout half of the problems
Code acceptance rate and characters retainedhuge lift
Quality and user experience improvement from file path headerimproved by a surprising margin
Reported stack
GitHub CopilotGPT-3CodexOpenAI
Source
https://github.blog/ai-and-ml/github-copilot/inside-github-working-with-the-llms-behind-github-copilot/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By incorporating neighboring-tab context retrieval and file path headers into prompts, and fine-tuning the Codex model on users' codebases, GitHub Copilot achieved a large lift in acceptance rate and characters retain…

What tools did this team use?

GitHub Copilot, GPT-3, Codex, OpenAI.

What results were reported?

Model benchmark problem-solving rate: upwards of 90 percent; Initial model problem-solving rate: about half of the problems; Code acceptance rate and characters retained: huge lift; Quality and user experience improvement from file path header: improved by a surprising margin (source-reported, not independently verified).

What failed first in this deployment?

A static chatbot prototype for answering coding questions was abandoned in favour of in-IDE completion, and early model versions frequently suggested code in the wrong programming language.

How is this workflow AI workflow structured?

Developer edits code in IDE → Context retrieved from neighboring tabs → File path added to prompt → LLM generates code completion → Codebase fine-tuning personalizes output → Acceptance signal feeds model improvement.