Workflow · Production

How GitHub Copilot improves contextual understanding with Fill-in-the-Middle and neighboring tabs

The problem

Transformer models fast enough to power GitHub Copilot can process only about 6,000 characters at a time, meaning not all of a developer's code can be used as context, and selecting and ordering the right information for the model is the core challenge.

First attempt

Prior to Fill-in-the-Middle, only code before the cursor was included in the prompt and the suffix was ignored; the first version also considered only the single file being worked on in the IDE.

Workflow diagram · grounded in source
1
Developer triggers suggestion
trigger
“GitHub Copilot will generate coding suggestions whether you're currently writing or just finished a comment, or in the middle of some gnarly code”
2
Prompt assembly via algorithms
ai_action
“a series of algorithms first select relevant code snippets or comments from your current file and other sources. These snippets and comments are then prioritized, filtered, and assembled into the final prompt”
3
Neighboring tabs scan
ai_action
“GitHub Copilot to comb through all of the data and find matching pieces of code between their open files and the code around their cursor—and add those matches to the prompt. When developing neighboring tabs, the GitHub Next team and in-…”
4
Fill-in-the-Middle (FIM)
ai_action
“With FIM, we can tell the model which part of the prompt is the prefix, and which part is the suffix. Based on A/B testing, FIM gave a 10% relative boost in performance, meaning developers accepted 10% more of the completions that were s…”
5
Semantic vector retrieval (experimental)
ai_action
“Algorithms would create embeddings for all snippets in the repository (potentially billions of them), and keep them stored in the vector database. Then, as you're coding, algorithms would embed the snippets in your IDE. Algorithms would …”
6
Real-time suggestion delivery
output
“GitHub Copilot is responding to your writing and edits in real time by generating prompts–or, in other words, prioritizing and sending relevant information to the model based on your actions in your IDE—to keep giving you the best coding…”
Reported outcome

Neighboring tabs increased user acceptance of suggestions by 5% relative, FIM gave a 10% relative boost in completion acceptance, and research shows developers code up to 55% faster with GitHub Copilot.

Reported metrics
User acceptance increase from neighboring tabs5%
completion acceptance boost from FIM10%
Developer coding speedup to 55% faster
Reported stack
GitHub CopilotCodexvector databasesMicrosoft Azure AI-Platform
Source
https://github.blog/2023-05-17-how-github-copilot-is-getting-better-at-understanding-your-code/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Neighboring tabs increased user acceptance of suggestions by 5% relative, FIM gave a 10% relative boost in completion acceptance, and research shows developers code up to 55% faster with GitHub Copilot.

What tools did this team use?

GitHub Copilot, Codex, vector databases, Microsoft Azure AI-Platform.

What results were reported?

User acceptance increase from neighboring tabs: 5%; completion acceptance boost from FIM: 10%; Developer coding speed: up to 55% faster (source-reported, not independently verified).

What failed first in this deployment?

Prior to Fill-in-the-Middle, only code before the cursor was included in the prompt and the suffix was ignored; the first version also considered only the single file being worked on in the IDE.

How is this workflow AI workflow structured?

Developer triggers suggestion → Prompt assembly via algorithms → Neighboring tabs scan → Fill-in-the-Middle (FIM) → Semantic vector retrieval (experimental) → Real-time suggestion delivery.