Quality assurance · Production
Discord's staged approach to rapidly developing generative AI features
The problem
Building with LLMs is challenging because the technology is relatively new in practical application, making it hard to identify the right use cases and know how to start.
Workflow diagram · grounded in source
1
Identify generative AI use cases
trigger
“we dig into challenges that commonly: Involve analysis, interpretation, or review of unstructured content (e.g. text) at scale”
2
Define product requirements
validation
“This phase requires a thoughtful analysis to select the best-suited LLM and to frame our problem as a prompt to an LLM”
3
Prototype with commercial LLM
ai_action
“We generally lean towards picking more advanced commercial LLMs to quickly validate our ideas and obtain early feedback from users”
4
AI-assisted prompt evaluation
ai_action
“AI-assisted evaluation uses best-in-class LLMs (like GPT-4) to automatically critique how well the AI's outputs match what we expected or how they score against a set of criteria”
5
Limited release and observation
feedback_loop
“we roll out a limited release (e.g. A/B test) of our product and observe the system's performance in situ”
6
Deploy at scale with safety filters
output
“we chose to incorporate content safety filters to the output of the inference server to identify undesired material before it reaches the user”
Reported outcome
Discord describes a repeatable multi-stage development process — use-case identification, requirements definition, prototyping, AI-assisted prompt evaluation, limited release, and scaled deployment — for rapidly shipping generative AI features while managing cost, quality, and safety.
Reported stack
GPT-4ChatGPTLlamaMistralTritonvLLM
Frequently asked questions
What did this team achieve with this AI workflow?
Discord describes a repeatable multi-stage development process — use-case identification, requirements definition, prototyping, AI-assisted prompt evaluation, limited release, and scaled deployment — for rapidly shipp…
What tools did this team use?
GPT-4, ChatGPT, Llama, Mistral, Triton, vLLM.
How is this quality assurance AI workflow structured?
Identify generative AI use cases → Define product requirements → Prototype with commercial LLM → AI-assisted prompt evaluation → Limited release and observation → Deploy at scale with safety filters.