Workflow · Production

Inside GitHub: Working with the LLMs behind GitHub Copilot

The problem

Code generation was considered too difficult for existing models before GPT-3, and early Copilot versions suffered from wrong-language suggestions and had no awareness of files beyond the one currently being edited.

First attempt

An initial chatbot prototype was abandoned in favour of the IDE modality, and early models defaulted to the most popular programming languages instead of detecting the correct one from context.

Workflow diagram · grounded in source

Developer edits code in IDE

trigger

“we don't just have to provide the model with the original file that the GitHub Copilot user is currently editing”

Neighboring-tab context retrieval

ai_action

“we pulled similar texts in from the user's neighboring editor tabs. That was a huge lift in our acceptance rate and characters retained”

File path context injection

ai_action

“it was much more powerful to put the path of the file at the top of the document”

Pseudo-document prompt construction

ai_action

“the art of prompt crafting is really all about creating a 'pseudo-document' that will lead the model to a completion that benefits the customer”

LLM generates code suggestion

ai_action

“the LLM is, at its core, just a document completion model. For training it was given partial documents and it learned how to complete them one token at a time”

Developer accepts or rejects

human_review

“Our greatest challenge right now is to consider why the user rejects or accepts a suggestion”

Fine-tuning on user codebase

feedback_loop

“we're training the underlying Codex model on a user's specific codebase to provide more focused, customized completions”

Reported outcome

Model accuracy on evaluation problems rose from about half to upwards of 90 percent; prompt techniques including neighboring-tab context retrieval and file-path injection produced large lifts in code acceptance; GitHub Copilot gained new capabilities and GitHub Copilot X was announced as the next evolution.

Reported metrics

Evaluation problem-solving rate (initial)about half of the problems

Evaluation problem-solving rate (improved)upwards of 90 percent

Code acceptance rate (neighboring tabs)huge lift in our acceptance rate and characters retained

Code acceptance (cross-file context)huge boost in code acceptance

Show all 5 reported metrics

evaluation problem-solving rate (initial)about half of the problems

evaluation problem-solving rate (improved)upwards of 90 percent

code acceptance rate (neighboring tabs)huge lift in our acceptance rate and characters retained

code acceptance (cross-file context)huge boost in code acceptance

quality and user experience (file path injection)improved the quality and user experience by a surprising margin

Reported stack

GitHub CopilotGPT-3Codex modelGitHub Copilot XOpenAI

Source

https://github.blog/2023-05-17-inside-github-working-with-the-llms-behind-github-copilot/

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Model accuracy on evaluation problems rose from about half to upwards of 90 percent; prompt techniques including neighboring-tab context retrieval and file-path injection produced large lifts in code acceptance; GitHu…

What tools did this team use?

GitHub Copilot, GPT-3, Codex model, GitHub Copilot X, OpenAI.

What results were reported?

Evaluation problem-solving rate (initial): about half of the problems; Evaluation problem-solving rate (improved): upwards of 90 percent; Code acceptance rate (neighboring tabs): huge lift in our acceptance rate and characters retained; Code acceptance (cross-file context): huge boost in code acceptance (source-reported, not independently verified).

What failed first in this deployment?

An initial chatbot prototype was abandoned in favour of the IDE modality, and early models defaulted to the most popular programming languages instead of detecting the correct one from context.

How is this workflow AI workflow structured?

Developer edits code in IDE → Neighboring-tab context retrieval → File path context injection → Pseudo-document prompt construction → LLM generates code suggestion → Developer accepts or rejects → Fine-tuning on user codebase.