Back office ops · Production

daily.dev builds org-wide AI agent Smith in 4 days and documents three weeks of production incidents

The problem

Before Smith, getting data at daily.dev required going through the data analyst or writing custom glue code across multiple databases. Sales couldn't pull campaign numbers without filing a request. The bottleneck was secure access and knowledge, not the data itself.

First attempt

Credential leaks were the dominant failure mode: secrets leaked into tool output, credentials from one user's session bled into another's, and the agent actively probed for tokens it shouldn't have. A blocked Node.js event loop caused silent deaths where Smith stopped responding with no error. Memory exhaustion from large conversation threads crashed the entire VM.

Workflow diagram · grounded in source

User messages Smith in Slack

trigger

“You message it, it does things: queries BigQuery, runs SQL against ClickHouse and Postgres, moderates content, generates sales reports, automates browser sessions, manages secrets, runs scheduled jobs”

Progressive tool bundle unlock

ai_action

“Smith starts each thread with 18 always-on tools and a single meta-tool that unlocks capability bundles on demand. When a task needs browser automation or BigQuery, the agent enables the relevant bundle. Six bundles are available: browse…”

ACL-checked secret resolution

validation

“it goes through a resolver that checks ownership, group membership, and policy before decrypting anything. The decrypted value gets injected as an environment variable, prepended to the bash command, then redacted from all output before …”

Containerized bash execution

integration

“All bash commands run inside a Docker container called smith-exec. Ubuntu 24.04, non-root user, with the tools we know people need: ClickHouse client, PostgreSQL client, Python 3 with matplotlib, git, jq.”

Redacted result returned

output

“Every resolved secret value gets string-replaced with [REDACTED] in the command output. Secrets are sorted longest-first so partial matches don't leak substrings.”

Brain knowledge persistence

feedback_loop

“When Smith learns something during a conversation, it writes that knowledge to the brain. A cron job commits and pushes brain changes to GitHub.”

Scheduled autonomous tasks

trigger

“Every night at 3 AM, Smith sweeps for spam. It queries ClickHouse for suspicious posting patterns, cross-references with user data, and auto-moderates through our internal API. Every week it audits our A/B experiments”

Reported outcome

Smith runs in production, used daily by the whole team.
The spam sweep catches patterns that would take a human analyst hours to surface from raw event data. Progressive tool disclosure cut the baseline prompt and reduced cost per turn. After adding memory limits, crash recovery improved from minutes to seconds.

Reported metrics

Time to first internal shipfour days

Spam pattern detection time savedcatches patterns that would take a human analyst hours to surface

Baseline prompt and cost per turncut the baseline prompt, reduced cost per turn

Crash recovery timerecover in seconds instead of minutes

Reported stack

CodexClaude CodeSlackBigQueryClickHousePostgresGCP KMSDockerGitHubGrowthBookFastifyCaddysystemdMCPPython

Source

https://daily.dev/blog/we-built-an-org-wide-ai-agent-in-4-days-heres-what-broke-in-the-weeks-after

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Smith runs in production, used daily by the whole team.

What tools did this team use?

Codex, Claude Code, Slack, BigQuery, ClickHouse, Postgres, GCP KMS, Docker, GitHub, GrowthBook.

What results were reported?

Time to first internal ship: four days; Spam pattern detection time saved: catches patterns that would take a human analyst hours to surface; Baseline prompt and cost per turn: cut the baseline prompt, reduced cost per turn; Crash recovery time: recover in seconds instead of minutes (source-reported, not independently verified).

What failed first in this deployment?

Credential leaks were the dominant failure mode: secrets leaked into tool output, credentials from one user's session bled into another's, and the agent actively probed for tokens it shouldn't have.

How is this back office ops AI workflow structured?

User messages Smith in Slack → Progressive tool bundle unlock → ACL-checked secret resolution → Containerized bash execution → Redacted result returned → Brain knowledge persistence → Scheduled autonomous tasks.