Workflow · Production

Wix runs 250 AI agent evals comparing docs vs skills for developer task completion

The problem

As AI agents became an increasingly important audience for developer documentation, Wix faced an unexamined assumption that purpose-built skills are superior to docs for guiding agents. Independent teams were creating skills without coordination with the underlying documentation, creating a parallel layer at risk of drifting from the actual product.

First attempt

Skills with small mistakes—misaligned project scaffolding, errors in code snippets, and best-practice bloat—eroded their advantage over docs and in some cases dramatically increased token usage.

Workflow diagram · grounded in source
1
Task assigned to sandboxed agent
trigger
“we ran sandboxed AI agents with different access to the docs. Each condition ran 3 times per task to account for variance”
2
Agent fetches docs or skills
integration
“The agent used our docs portal's llms.txt service via web-fetch”
3
Agent performs development task
ai_action
“CLI extensions: Building Wix CLI app extensions: dashboard pages, backend APIs, site widgets, event handlers, embedded scripts, modals, and plugins”
4
Agent self-evaluates output
ai_action
“we asked it to change hats and evaluate its own work. Did it complete the task as described? If not, why? What issues with the product and docs caused problems along the way?”
5
Metrics collection
output
“We also collected deterministic data on token count, turn count, and wall-clock time for each run.”
6
Framework and eval loop
feedback_loop
“Regular evaluations maintain skill freshness. Evaluations should compare skill performance against docs-optimized performance for a range of tasks. Any time a skill underperforms the docs, it's a signal that something drifted or was wron…”
Reported outcome

Agent-optimized docs improved CLI task completion from 67% to 87% while cutting token usage by 35%.
Well-aligned skills reduced tokens by 30-50% and time by 30%. The team adopted a framework treating optimized docs as the backbone and skills as a caching layer for common tasks.

Reported metrics
CLI task completion rate (baseline)67%
CLI task completion rate (optimized docs)87%
Token usage reduction (CLI docs optimization)35%
Wall-clock time reduction (CLI docs optimization)9%
Show all 18 reported metrics
CLI task completion rate (baseline)67%
CLI task completion rate (optimized docs)87%
Token usage reduction (CLI docs optimization)35%
Wall-clock time reduction (CLI docs optimization)9%
CLI completion rate docs-optimized85%
CLI completion rate skills-only78%
Token savings docs-optimized vs skills (CLI)10% fewer tokens
Speed advantage docs-optimized vs skills (CLI)8% faster
Turn count reduction docs-optimized vs skills (CLI)14% fewer turns
Token increase (misaligned skill - scaffolding case)94% more tokens
Token increase (code snippet error case)39% token increase
Token increase (best-practice bloat case)52% more token usage
Token reduction (well-aligned skills)30-50% reduction in tokens
Time reduction (well-aligned skills)30% reduction in time
API task completion (both conditions)80%
Speed advantage docs-optimized (API tasks)31% faster
Turn count reduction docs-optimized (API tasks)33% fewer turns
Token savings skills-only (API tasks)29% fewer tokens
Reported stack
Wix MCP
Source
https://www.wix.engineering/post/we-ran-250-ai-agent-evals-to-find-out-if-skills-beat-docs-the-answer-is-more-complicated-than-we-ex
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Agent-optimized docs improved CLI task completion from 67% to 87% while cutting token usage by 35%.

What tools did this team use?

Wix MCP.

What results were reported?

CLI task completion rate (baseline): 67%; CLI task completion rate (optimized docs): 87%; Token usage reduction (CLI docs optimization): 35%; Wall-clock time reduction (CLI docs optimization): 9% (source-reported, not independently verified).

What failed first in this deployment?

Skills with small mistakes—misaligned project scaffolding, errors in code snippets, and best-practice bloat—eroded their advantage over docs and in some cases dramatically increased token usage.

How is this workflow AI workflow structured?

Task assigned to sandboxed agent → Agent fetches docs or skills → Agent performs development task → Agent self-evaluates output → Metrics collection → Framework and eval loop.