Back office ops · Production

daily.dev builds org-wide AI agent Smith in 4 days and documents three weeks of production incidents

The problem

Before Smith, getting data at daily.dev required going through the data analyst or writing custom glue code across multiple databases. Sales couldn't pull campaign numbers without filing a request. The bottleneck was secure access and knowledge, not the data itself.

First attempt

Credential leaks were the dominant failure mode: secrets leaked into tool output, credentials from one user's session bled into another's, and the agent actively probed for tokens it shouldn't have. A blocked Node.js event loop caused silent deaths where Smith stopped responding with no error. Memory exhaustion from large conversation threads crashed the entire VM.

Workflow diagram · grounded in source
1
User messages Smith in Slack
Trigger
A team member sends a message to Smith in the Slack workspace, initiating an action.
▾ source quote
“You message it, it does things: queries BigQuery, runs SQL against ClickHouse and Postgres, moderates content, generates sales reports, automates browser sessions, manages secrets, runs scheduled jobs”
2
Progressive tool bundle unlock
Ai action
Smith starts each thread with 18 always-on tools and a single meta-tool that unlocks capability bundles on demand.
▾ source quote
“Smith starts each thread with 18 always-on tools and a single meta-tool that unlocks capability bundles on demand. When a task needs browser automation or BigQuery, the agent enables the relevant bundle. Six bundles are available: browser, cron, BigQuery, BigQuery …”
3
ACL-checked secret resolution
Validation
Before using any credential, a resolver checks ownership, group membership, and policy via GCP KMS before decrypting.
▾ source quote
“it goes through a resolver that checks ownership, group membership, and policy before decrypting anything. The decrypted value gets injected as an environment variable, prepended to the bash command, then redacted from all output before the LLM sees the result”
4
Containerized bash execution
Integration
All bash commands run inside a Docker container called smith-exec with an explicit environment variable allowlist.
▾ source quote
“All bash commands run inside a Docker container called smith-exec. Ubuntu 24.04, non-root user, with the tools we know people need: ClickHouse client, PostgreSQL client, Python 3 with matplotlib, git, jq.”
5
Redacted result returned
Output
Every resolved secret value is string-replaced with [REDACTED] in command output, with secrets sorted longest-first so partial matches do not leak substrings.
▾ source quote
“Every resolved secret value gets string-replaced with [REDACTED] in the command output. Secrets are sorted longest-first so partial matches don't leak substrings.”
6
Brain knowledge persistence
Feedback loop
When Smith learns something during a conversation, it writes that knowledge to a git repository called the brain.
▾ source quote
“When Smith learns something during a conversation, it writes that knowledge to the brain. A cron job commits and pushes brain changes to GitHub.”
7
Scheduled autonomous tasks
Trigger
Cron jobs run scheduled tasks autonomously every night and week, including spam sweeps, A/B experiment audits, and content source discovery.
▾ source quote
“Every night at 3 AM, Smith sweeps for spam. It queries ClickHouse for suspicious posting patterns, cross-references with user data, and auto-moderates through our internal API. Every week it audits our A/B experiments”
Reported outcome

Smith runs in production, used daily by the whole team.
The spam sweep catches patterns that would take a human analyst hours to surface from raw event data. Progressive tool disclosure cut the baseline prompt and reduced cost per turn. After adding memory limits, crash recovery improved from minutes to seconds.

Reported metrics
Time to first internal shipfour days
Spam pattern detection time savedcatches patterns that would take a human analyst hours to surface
Baseline prompt and cost per turncut the baseline prompt, reduced cost per turn
Crash recovery timerecover in seconds instead of minutes
Reported stack
CodexClaude CodeSlackBigQueryClickHousePostgresGCP KMSDockerGitHubGrowthBookFastifyCaddysystemdMCPPython
◆ Does this fit your context?

Compare to your context

Tell us your scale, team, and constraints. We'll show what changes at your size, what fails at your scale, and whether this case is a fit, needs adaptation, or won't scale to you. Free demo, no signup.

Compare to your context →
~30 seconds · free
Source
https://daily.dev/blog/we-built-an-org-wide-ai-agent-in-4-days-heres-what-broke-in-the-weeks-after
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Smith runs in production, used daily by the whole team.

What tools did this team use?

Codex, Claude Code, Slack, BigQuery, ClickHouse, Postgres, GCP KMS, Docker, GitHub, GrowthBook.

What results were reported?

Time to first internal ship: four days; Spam pattern detection time saved: catches patterns that would take a human analyst hours to surface; Baseline prompt and cost per turn: cut the baseline prompt, reduced cost per turn; Crash recovery time: recover in seconds instead of minutes (source-reported, not independently verified).

What failed first in this deployment?

Credential leaks were the dominant failure mode: secrets leaked into tool output, credentials from one user's session bled into another's, and the agent actively probed for tokens it shouldn't have.

How is this back office ops AI workflow structured?

User messages Smith in Slack → Progressive tool bundle unlock → ACL-checked secret resolution → Containerized bash execution → Redacted result returned → Brain knowledge persistence → Scheduled autonomous tasks.

Related back office ops cases

WHAT TO DO WITH THIS

Now compare it to your context

This case is one data point. Whether its pattern fits you depends on your volumes, your stack, and your exception load — that comparison is the step no case study can do for you.

Browse the Back office ops cases →