Building AI Products Part I: Back-end Architecture for an Engineering Leader AI Assistant
Engineering leaders lacked a unified way to track information across team tools and critical project developments. As the AI assistant scaled, inference pipelines grew complex and brittle, and the microservices architecture conflicted with the stateful, non-deterministic nature of AI agents.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · User activity ingestion
Content from productivity tools—Slack messages, GitHub reviews, Google Doc comments, emails, and calendar event descriptions—serves as input to the system.
The team attracted 10,000 users within a year, growing from 8 to 2,000 users in about two months, outperforming incumbents such as Salesforce and Slack AI. The technical learnings drove a pivot to Outropy, a developer platform enabling engineers to build AI products.
What failed first
The initial microservices architecture was a poor fit because agents require rich stateful context, non-deterministic behavior, and data-intensive processing, conflicting with stateless minimal-context microservices design. Attempts at feature extraction using simpler ML models were also unsuccessful.