Workflow · Production

How to build an enterprise LLM application: Lessons from GitHub Copilot

The problem

Developers were consistently crunched for time and needed to write code faster with less context switching, while most AI coding assistants at the time could only complete a single line of code.

First attempt

An early web interface required developers to switch back and forth between their editor and the browser, making it an unsuitable canvas. An initial assumption that every coding language would require its own fine-tuned AI model also proved incorrect as LLMs rapidly improved.

Workflow diagram · grounded in source
1
Developer codes in IDE
trigger
“enabling developers to receive code suggestions without changing how they work”
2
Neighboring tabs context
ai_action
“GitHub Copilot processes multiple files open in a developer's IDE instead of just the single one the developer is working on. Neighboring tabs helped to increase the acceptance rates of GitHub Copilot's suggestions by 5%”
3
Whole-function suggestion generation
ai_action
“the team applied two strategies: changing the parameters to reduce the randomness of outputs and caching responses”
4
Security and content filtering
validation
“the team integrated code security capabilities to filter out suggestions that could contain security vulnerabilities (e.g., SQL injections and hard coded credentials) and used natural language filters from Azure OpenAI Service to filter …”
5
Ghost text suggestion displayed
output
“ghost text—the gray text that flashes one coding suggestion while you type”
6
Acceptance feedback loop
feedback_loop
“code acceptance rate and, eventually, code retention rate (which measures how much of the original code suggestion is kept or edited by a developer)”
Reported outcome

GitHub Copilot reached general availability with a neighboring tabs feature that increased suggestion acceptance rates by 5%, and the team scaled infrastructure on Microsoft Azure for enterprise-grade reliability.

Reported metrics
Suggestion acceptance rate improvement from neighboring tabs5%
Reported stack
GitHub CopilotOpenAI APIMicrosoft AzureMicrosoft Experimentation PlatformAzure OpenAI ServiceGitHub Copilot Chat
Source
https://github.blog/ai-and-ml/github-copilot/how-to-build-an-enterprise-llm-application-lessons-from-github-copilot/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

GitHub Copilot reached general availability with a neighboring tabs feature that increased suggestion acceptance rates by 5%, and the team scaled infrastructure on Microsoft Azure for enterprise-grade reliability.

What tools did this team use?

GitHub Copilot, OpenAI API, Microsoft Azure, Microsoft Experimentation Platform, Azure OpenAI Service, GitHub Copilot Chat.

What results were reported?

Suggestion acceptance rate improvement from neighboring tabs: 5% (source-reported, not independently verified).

What failed first in this deployment?

An early web interface required developers to switch back and forth between their editor and the browser, making it an unsuitable canvas.

How is this workflow AI workflow structured?

Developer codes in IDE → Neighboring tabs context → Whole-function suggestion generation → Security and content filtering → Ghost text suggestion displayed → Acceptance feedback loop.