Workflow · saas · workflow
Inside GitHub: Working with the LLMs behind GitHub Copilot
Code generation was considered too difficult for existing models before GPT-3, and early Copilot versions suffered from wrong-language suggestions and had no awareness of files beyond the one currently being edited.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · Developer edits code in IDE
The workflow starts when a GitHub Copilot user is actively editing a file in their IDE.
Tools used
GitHub CopilotGPT-3Codex modelGitHub Copilot X
Outcome
Model accuracy on evaluation problems rose from about half to upwards of 90 percent; prompt techniques including neighboring-tab context retrieval and file-path injection produced large lifts in code acceptance; GitHub Copilot gained new capabilities and GitHub Copilot X was announced as the next evolution.
What failed first
An initial chatbot prototype was abandoned in favour of the IDE modality, and early models defaulted to the most popular programming languages instead of detecting the correct one from context.
Results
Volumeabout half of the problems
Grounding & classification
Source type: technical build writeup
24 fields verified against source quotes.
code generationbuilder submittedfailure mode describedhuman review describedmetric backednamed customerproduction runtime claimedsource backedtools describedworkflow describedsoftwareaccuracy improvementemployee productivitytechnical build writeupai draft human approval