Customer support · Production

Assembled scales LLM serving to millions of monthly requests using Go

The problem

Serving LLMs in production requires handling unstructured model outputs, managing concurrent API calls with additive latency, and building multi-step response transformations that remain maintainable and testable.

Workflow diagram · grounded in source
1
Structured LLM output schema
ai_action
“we can leverage Go's struct tags and reflection to generate well defined schemas”
2
Parallel RAG hybrid search
ai_action
“suppose we're running a Retrieval Augmented Generation (RAG) pipeline and want to perform hybrid search across three different search backends”
3
Goroutine result collection
integration
“This pattern reduces our total latency to that of the slowest backend, with a configurable timeout to prevent any single slow backend from bottlenecking the entire system. The results are collected via a Go channel and combined after all…”
4
Composable response pipeline
output
“We've built a composable pipeline that makes these transformations both maintainable and testable”
5
Citation parsing output
output
“The cleaner will parse the sources and pass them to the frontend as response details”
6
Python ML service bridge
integration
“we maintain a lightweight Python service that our Go infrastructure calls. This service handles computationally intensive ML tasks (like generating embeddings or clustering)”
Reported outcome

Assembled's Go-based infrastructure handles millions of monthly LLM requests with minimal performance tuning, using a composable pipeline that makes response transformations maintainable and testable.

Reported metrics
monthly LLM requests servedmillions
Infrastructure performance overheadminimal performance tuning
Search latency reductionreduces our total latency to that of the slowest backend
Reported stack
GoOpenAIgo-openaiPythonscikit-learnsentence-transformerstransformersClaude 3.5 SonnetLlama
Source
https://www.assembled.com/blog/scaling-llms-with-golang-how-we-serve-millions-of-llm-requests
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Assembled's Go-based infrastructure handles millions of monthly LLM requests with minimal performance tuning, using a composable pipeline that makes response transformations maintainable and testable.

What tools did this team use?

Go, OpenAI, go-openai, Python, scikit-learn, sentence-transformers, transformers, Claude 3.5 Sonnet, Llama.

What results were reported?

monthly LLM requests served: millions; Infrastructure performance overhead: minimal performance tuning; Search latency reduction: reduces our total latency to that of the slowest backend (source-reported, not independently verified).

How is this customer support AI workflow structured?

Structured LLM output schema → Parallel RAG hybrid search → Goroutine result collection → Composable response pipeline → Citation parsing output → Python ML service bridge.

Related customer support cases