Back office ops · Production

Building AI Products Part I: Back-end Architecture for an Engineering Leader AI Assistant

The problem

Engineering leaders lacked a unified way to track information across team tools and critical project developments. As the AI assistant scaled, inference pipelines grew complex and brittle, and the microservices architecture conflicted with the stateful, non-deterministic nature of AI agents.

First attempt

The initial microservices architecture was a poor fit because agents require rich stateful context, non-deterministic behavior, and data-intensive processing, conflicting with stateless minimal-context microservices design. Attempts at feature extraction using simpler ML models were also unsuccessful.

Workflow diagram · grounded in source

User activity ingestion

trigger

“each piece of content—Slack messages, GitHub reviews, Google Doc comments, emails, calendar event descriptions…—to every agent for processing”

LLM proposition generation

ai_action

“Our solution was to use an LLM to generate propositions for every message, structured according to a format inspired by Abstract Meaning Representation, a technique from natural language processing.”

Event bus agent routing

routing

“all state-change events were posted to an event bus that agents could subscribe to. Each agent filtered out irrelevant events independently, removing the need for external systems to know what they cared about.”

CQRS event sourcing

integration

“every state change—whether creating a meeting or updating team members—was represented as a Command, a discrete event recorded chronologically—much like a database transaction log”

Temporal agent computation

ai_action

“We migrated our agents to run all their computations through Temporal workflows. We implemented our agents' main logic as Workflows, while tool and API interactions—like AI model calls—became Activities. This structure let Temporal's run…”

Personalized daily briefing

output

“We'd trained our assistant to learn each user's login time and generate reports an hour before, ensuring fresh updates.”

Reported outcome

The team attracted 10,000 users within a year, growing from 8 to 2,000 users in about two months, outperforming incumbents such as Salesforce and Slack AI.
The technical learnings drove a pivot to Outropy, a developer platform enabling engineers to build AI products.

Reported metrics

Total users within one year10,000

User growth over approximately two months8 to 2,000 users in about two months

Daily briefing generation time (degraded at scale)went from taking minutes to hours per user

Reported stack

SlackSQLAlchemyPydanticPythonFastAPIasyncioTemporalPostgresAWS Elastic Container ServiceOpenAIAzureGitHubGoogle Doc

Source

https://philcalcado.com/2024/12/14/building-ai-products-part-i.html

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The team attracted 10,000 users within a year, growing from 8 to 2,000 users in about two months, outperforming incumbents such as Salesforce and Slack AI.

What tools did this team use?

Slack, SQLAlchemy, Pydantic, Python, FastAPI, asyncio, Temporal, Postgres, AWS Elastic Container Service, OpenAI.

What results were reported?

Total users within one year: 10,000; User growth over approximately two months: 8 to 2,000 users in about two months; Daily briefing generation time (degraded at scale): went from taking minutes to hours per user (source-reported, not independently verified).

What failed first in this deployment?

How is this back office ops AI workflow structured?

User activity ingestion → LLM proposition generation → Event bus agent routing → CQRS event sourcing → Temporal agent computation → Personalized daily briefing.

The team attracted 10,000 users within a year, growing from 8 to 2,000 users in about two months, outperforming incumbents such as Salesforce and Slack AI.The technical learnings drove a pivot to Outropy, a developer platform enabling engineers to build AI products.