Customer support · Production
Architecture of today's LLM applications: five steps and emerging component patterns
The problem
Building software with LLMs is fundamentally different from traditional software development, requiring developers to navigate datasets, embeddings, and parameter weights instead of compiling source code, with outputs that are probabilistic rather than predictable.
Workflow diagram · grounded in source
1
User contacts LLM assistant
trigger
“Dave calls his internet service provider (ISP) and is directed to an LLM-powered assistant”
2
Speech-to-text translation
ai_action
“Dave's verbal query then needs to be fed through a speech-to-text translation tool that works in the background”
3
LLM classifies IT complaint
ai_action
“The LLM can analyze the sequence of words in Dave's transcript, classify it as an IT complaint, and provide a contextually relevant response”
4
Vector database context retrieval
integration
“the LLM assistant has access to the company's complaints search engine, and those complaints and solutions are stored as embeddings in a vector database”
5
Prompt construction and optimization
ai_action
“the tool will help to prioritize which context embeddings are most relevant, and in which order those embeddings should be organized in order for the LLM to produce the most contextually relevant response”
6
Cached response delivery
output
“instead of generating new responses to the same query (because Dave isn't the first person whose internet has gone down), the LLM can retrieve outputs from the cache that have been used for similar queries. Caching outputs can reduce lat…”
7
Telemetry evaluation
feedback_loop
“A telemetry service will allow you to evaluate how well your app is working with actual users. A service that responsibly and transparently monitors user activity (like how often they accept or change a suggestion) can share useful data …”
Reported outcome
(not stated)
Reported stack
GitHub Copilotlangchain-ai/langchainMongoDBQdrantPineconeMilvusGPTCacheOpenTelemetry
Frequently asked questions
What did this team achieve with this AI workflow?
(not stated)
What tools did this team use?
GitHub Copilot, langchain-ai/langchain, MongoDB, Qdrant, Pinecone, Milvus, GPTCache, OpenTelemetry.
How is this customer support AI workflow structured?
User contacts LLM assistant → Speech-to-text translation → LLM classifies IT complaint → Vector database context retrieval → Prompt construction and optimization → Cached response delivery → Telemetry evaluation.