Quality assurance · Production

Pinterest engineers build a test harness to optimize AI agent skill invocation rates

The problem

Pinterest engineers found that AI agents inconsistently invoked a custom iOS architecture skill (rx-mvvm), with baseline overall accuracy of only 73% for Codex and 62% for Claude Code—deemed unacceptable for critical engineering workflows.

First attempt

Initial 'vanilla' testing showed neither agent could guarantee 100% skill invocation, especially with terse or ambiguous prompts.

Workflow diagram · grounded in source

Engineer submits prompt

trigger

“neither agent could guarantee 100% skill invocation, particularly when engineers used terse or ambiguous prompts”

Agent attempts skill invocation

ai_action

“we conducted a series of tests on Pin-agent (an internal fork of OpenAI's Codex) and Claude Code to quantify the reliability of skill invocation”

Bash harness captures logs

validation

“We then use log parsing heuristics on the json output logfiles to detect skill invocation by searching for telltale patterns in the JSON-streamed debug output”

Metrics computed

output

“The script finally tallies successes across both categories and computes three key metrics with clear formulas”

Optimization techniques applied

feedback_loop

“by applying different techniques we could track and drastically improve skill invocation rates on both tested agents”

Reported outcome

By applying optimized frontmatter descriptions, aggressive directive language, and AGENTS.md skill tables, the team dramatically improved skill invocation rates on both agents, with gains much greater for Codex than for Claude Code.

Reported metrics

baseline overall accuracy — Codex73%

baseline overall accuracy — Claude62%

Skill invocation rate improvementdramatically improved

Reported stack

Claude CodePin-agentGPT 5.2-codexBash

Source

https://medium.com/pinterest-engineering/an-engineers-guide-to-better-ai-skills-implementing-a-testing-process-to-optimize-agent-a000c9c9abcd

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By applying optimized frontmatter descriptions, aggressive directive language, and AGENTS.md skill tables, the team dramatically improved skill invocation rates on both agents, with gains much greater for Codex than f…

What tools did this team use?

Claude Code, Pin-agent, GPT 5.2-codex, Bash.

What results were reported?

baseline overall accuracy — Codex: 73%; baseline overall accuracy — Claude: 62%; Skill invocation rate improvement: dramatically improved (source-reported, not independently verified).

What failed first in this deployment?