Back office ops · Production

Vercel removes 80% of agent tools and achieves 100% success rate with file system agent

The problem

Vercel's internal text-to-SQL agent (d0) was built with many specialized tools, heavy prompt engineering, and complex context management that made it fragile, slow, and expensive to maintain, achieving only an 80% success rate.

First attempt

The original multi-tool architecture constrained the model's reasoning by pre-filtering context and wrapping every interaction in validation logic. Its worst-case query took 724 seconds, 100 steps, and 145,463 tokens before failing.

Workflow diagram · grounded in source
1
Natural language question submitted
trigger
“d0 translates natural language questions into SQL queries against our analytics infrastructure, letting anyone on the team get answers without writing code or waiting on the data team”
2
Semantic layer file exploration
ai_action
“The file system agent now browses our semantic layer the way a human analyst would. It reads files, greps for patterns, builds mental models, and writes SQL using standard Unix tools like grep, cat, find, and ls.”
3
SQL query execution
output
“The file system agent completed the same query in 141 seconds with 19 steps and 67,483 tokens, and it actually succeeded.”
Reported outcome

The file system agent achieved 100% success rate (up from 80%), ran 3.5x faster, and used 37% fewer tokens, with the same previously failing query completing in 141 seconds with 19 steps and 67,483 tokens.

Reported metrics
Query success rate (new architecture)100%
Query success rate (old architecture)80%
Response speed improvement3.5x faster
Token reduction37% fewer tokens
Show all 10 reported metrics
query success rate (new architecture)100%
query success rate (old architecture)80%
response speed improvement3.5x faster
token reduction37% fewer tokens
old architecture worst-case query time724 seconds
old architecture worst-case steps100 steps
old architecture worst-case tokens145,463 tokens
new architecture benchmark query time141 seconds
new architecture benchmark steps19 steps
new architecture benchmark tokens67,483 tokens
Reported stack
Claude Opus 4.5AI SDKNext.jsVercel Slack BoltCubeSlack
Source
https://vercel.com/blog/we-removed-80-percent-of-our-agents-tools
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The file system agent achieved 100% success rate (up from 80%), ran 3.5x faster, and used 37% fewer tokens, with the same previously failing query completing in 141 seconds with 19 steps and 67,483 tokens.

What tools did this team use?

Claude Opus 4.5, AI SDK, Next.js, Vercel Slack Bolt, Cube, Slack.

What results were reported?

Query success rate (new architecture): 100%; Query success rate (old architecture): 80%; Response speed improvement: 3.5x faster; Token reduction: 37% fewer tokens (source-reported, not independently verified).

What failed first in this deployment?

The original multi-tool architecture constrained the model's reasoning by pre-filtering context and wrapping every interaction in validation logic.

How is this back office ops AI workflow structured?

Natural language question submitted → Semantic layer file exploration → SQL query execution.

Related back office ops cases