Workflow · Production

How GitHub built Copilot: a globally-distributed LLM code completion service serving 400M+ requests at under 200ms

The problem

GitHub needed to serve LLM-based code completions with latency competitive against locally-run IDE autocomplete, despite the overhead of network latency, shared server resources, and cloud outages. Authentication at scale and efficient request cancellation were also unsolved challenges.

First attempt

The alpha required users to supply their own OpenAI API keys and scaled to only dozens of users. Standard HTTP/1-based cancellation forced costly TCP reconnections after every cancelled request. A point-of-presence model caused traffic tromboning and high operational burden. Most cloud load balancers downgraded HTTP/2 to HTTP/1 on the backend, undermining stream-level cancellation.

Workflow diagram · grounded in source
1
User pauses typing in IDE
trigger
“Whenever I stop typing, Copilot takes over”
2
Completion timing prediction
ai_action
“use a tiny prediction model to look at the stream of characters as they're typed and predict, are they approaching the end of the word or are they in the middle of a word”
3
Short-lived token validation
validation
“exchange that for what we call a short-lived code completion token...all we have to do is just check that that signature is valid. If it's good, we swap the key out for the actual API service key”
4
Regional proxy routing
routing
“we want users, therefore, to be routed to their 'closest' proxy region. If that region is unhealthy, we want them to automatically be routed somewhere else”
5
LLM code completion generation
ai_action
“write a function, a comment describing what you want, and Copilot will do its best to write it for you”
6
Streaming response to IDE
output
“we immediately start streaming it as soon as it starts”
7
HTTP/2 stream cancellation
integration
“Instead of resetting the TCP connection itself, you just reset the individual stream representing the request you made. The underlying connection stays open”
Reported outcome

GitHub Copilot serves more than 400 million completion requests with a mean response time under 200 milliseconds and peaks at 8,000 requests per second, achieving global resilience through regional proxy colocation and self-healing DNS routing.

Reported metrics
Completion requests servedmore than 400 million
Peak requests per second8,000
Mean response timeless than 200 milliseconds
Typed-through (cancelled) request ratearound half
Show all 6 reported metrics
completion requests servedmore than 400 million
peak requests per second8,000
mean response timeless than 200 milliseconds
typed-through (cancelled) request ratearound half
inference wasted on cancelled requests45%
client population updated within 24-36 hours of release80%
Reported stack
GitHub Copilotcopilot-proxyGoHTTP/2GLBHAProxyoctoDNSAzureOpenAIVS Code
Source
https://www.infoq.com/presentations/github-copilot/
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

GitHub Copilot serves more than 400 million completion requests with a mean response time under 200 milliseconds and peaks at 8,000 requests per second, achieving global resilience through regional proxy colocation an…

What tools did this team use?

GitHub Copilot, copilot-proxy, Go, HTTP/2, GLB, HAProxy, octoDNS, Azure, OpenAI, VS Code.

What results were reported?

Completion requests served: more than 400 million; Peak requests per second: 8,000; Mean response time: less than 200 milliseconds; Typed-through (cancelled) request rate: around half (source-reported, not independently verified).

What failed first in this deployment?

The alpha required users to supply their own OpenAI API keys and scaled to only dozens of users.

How is this workflow AI workflow structured?

User pauses typing in IDE → Completion timing prediction → Short-lived token validation → Regional proxy routing → LLM code completion generation → Streaming response to IDE → HTTP/2 stream cancellation.