Quality assurance · Production

Pushpay builds production-ready agentic AI search on Amazon Bedrock, improving accuracy from 60–70% to 95%

The problem

Ministry leaders at Pushpay's church customers needed fast access to community insights without technical expertise, but the initial AI search agent plateaued at 60–70% accuracy because evaluation was manual and tedious, creating critical blockers to production deployment.

First attempt

The first AI search agent iteration relied on a single statically tuned system prompt and had no automated evaluation mechanism, causing it to stall at a 60–70% accuracy ceiling with no clear path to improvement.

Workflow diagram · grounded in source
1
User submits natural language query
trigger
“Pushpay users submitting natural language queries through the existing Pushpay application interface”
2
Dynamic Prompt Constructor builds system prompt
ai_action
“Dynamic prompt constructor (DPC): automatically constructs additional customized system prompts based on the user specific information, such as church context, sample queries, and application filter inventory. They also use semantic sear…”
3
LLM generates structured JSON output
ai_action
“Uses Claude Sonnet 4.5 to process prompts and generate JSON output required by the application to display the desired query results as insights to users.”
4
LLM-as-judge evaluates agent output
validation
“The evaluator component processes user input queries and compares the agent-generated output against the golden dataset using the LLM as a judge pattern This approach generates core accuracy metrics while capturing detailed logs and perf…”
5
Domain dashboard surfaces weaknesses
output
“The dashboard serves as the mission control for Pushpay's product and engineering teams, displaying domain category-level metrics to assess performance and latency and guide decisions.”
6
Strategic domain-level rollout
routing
“By temporarily suppressing underperforming categories—such as activity queries—while undergoing optimization, the system achieved 95% overall accuracy.”
7
Evaluation drives iterative improvements
feedback_loop
“The evaluation results feed into a dashboard for product and engineering teams to analyze and drive iterative improvements to the AI search agent.”
Reported outcome

Pushpay's generative AI evaluation framework raised agent accuracy from 60–70% to 95% through domain-level dashboards and strategic rollout, while reducing time-to-insight from approximately 120 seconds to under 4 seconds—a 15-fold acceleration.

Reported metrics
Initial agent accuracy60-70%
Achieved overall accuracy95%
Time-to-insightfrom approximately 120 seconds to under 4 seconds
Time-to-insight acceleration factor15-fold
Show all 5 reported metrics
initial agent accuracy60-70%
achieved overall accuracy95%
time-to-insightfrom approximately 120 seconds to under 4 seconds
time-to-insight acceleration factor15-fold
golden dataset sizeover 300 representative queries
Reported stack
Amazon BedrockClaude Sonnet 4.5Amazon Bedrock prompt cachingDynamic prompt constructor
Source
https://aws.amazon.com/blogs/machine-learning/build-reliable-agentic-ai-solution-with-amazon-bedrock-learn-from-pushpays-journey-on-genai-evaluation?tag=soumet-20
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Pushpay's generative AI evaluation framework raised agent accuracy from 60–70% to 95% through domain-level dashboards and strategic rollout, while reducing time-to-insight from approximately 120 seconds to under 4 sec…

What tools did this team use?

Amazon Bedrock, Claude Sonnet 4.5, Amazon Bedrock prompt caching, Dynamic prompt constructor.

What results were reported?

Initial agent accuracy: 60-70%; Achieved overall accuracy: 95%; Time-to-insight: from approximately 120 seconds to under 4 seconds; Time-to-insight acceleration factor: 15-fold (source-reported, not independently verified).

What failed first in this deployment?

The first AI search agent iteration relied on a single statically tuned system prompt and had no automated evaluation mechanism, causing it to stall at a 60–70% accuracy ceiling with no clear path to improvement.

How is this quality assurance AI workflow structured?

User submits natural language query → Dynamic Prompt Constructor builds system prompt → LLM generates structured JSON output → LLM-as-judge evaluates agent output → Domain dashboard surfaces weaknesses → Strategic domain-level rollout → Evaluation drives iterative improvements.

Related quality assurance cases