Quality assurance · Production

Neon improves MCP server tool-call eval pass rate from 60% to 100% with prompt tuning

The problem

LLMs struggle to select the correct tool from a large list and could misuse a generic SQL-execution tool instead of the required two-step stateful migration workflow, making correctness hard to guarantee without automated testing.

First attempt

With the initial basic tool descriptions, the eval pass rate was only around 60%.

Workflow diagram · grounded in source
1
Eval task submitted
trigger
“evaluations that we can use to make sure that an LLM is able to use our "prepare" and "complete" migrations, in the right order, when asked to complete a database migration task”
2
LLM prepares migration on temp branch
ai_action
“The "prepare_database_migration" tool starts the process of creating a database migration. It takes the input (SQL code) and applies it on a temporary Neon branch (an instantly created Postgres branch with all of the same data that exist…”
3
Main branch integrity check
validation
“The mainBranchIntegrityCheck is just making sure that the main branch is un-modified by the first tool call from the LLM we're testing.”
4
LLM completes migration on main branch
ai_action
“The "complete_database_migration" tool completes the migration, which means that it will actually run it on the "main" branch and then it will delete the temporary branch which was created by the "prepare_database_migration" tool.”
5
LLM-as-a-judge scoring
validation
“we're using the "LLM-as-a-judge" scorer to evaluate what happens to the task we're sending in via our MCP server. For now, we're using Claude for the LLM which is acting as a judge”
6
Prompt iteration from eval results
feedback_loop
“we've since tweaked our prompts and gotten to 100% of pass rate. In fact, we didn't have to write any "code" in order to go from 60->100 – the only thing we changed were the descriptions ("prompts") for the two MCP tools we're testing”
Reported outcome

By tweaking tool description prompts without any code changes, the eval pass rate rose from 60% to 100%.

Reported metrics
Initial eval pass rate60%
Eval pass rate after prompt tuning100%
Reported stack
MCP serverBraintrustClaude
Source
https://neon.com/blog/test-evals-for-mcp
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

By tweaking tool description prompts without any code changes, the eval pass rate rose from 60% to 100%.

What tools did this team use?

MCP server, Braintrust, Claude.

What results were reported?

Initial eval pass rate: 60%; Eval pass rate after prompt tuning: 100% (source-reported, not independently verified).

What failed first in this deployment?

With the initial basic tool descriptions, the eval pass rate was only around 60%.

How is this quality assurance AI workflow structured?

Eval task submitted → LLM prepares migration on temp branch → Main branch integrity check → LLM completes migration on main branch → LLM-as-a-judge scoring → Prompt iteration from eval results.

Related quality assurance cases