Quality assurance · Production

Coinbase builds a QA AI agent to 10x testing effort at 1/10 the cost

The problem

Coinbase's manual QA testing was slow and expensive, and traditional end-to-end integration tests were prone to flakiness, causing hours of debugging from minor layout changes.

First attempt

Traditional end-to-end integration tests were prone to flakiness, with minor layout adjustments causing failures that required hours of debugging.

Workflow diagram · grounded in source
1
Natural language test request
trigger
“a prompt such as "log into coinbase test account in Brazil, and buy 10 BRL worth of BTC" is sufficient to initiate a test”
2
Agent browses and acts
ai_action
“the agent directly uses visual and textual data from coinbase.com to determine the next logical action to complete the task”
3
LLM identifies issues
ai_action
“it leverages the LLM's reasoning capabilities to identify issues intelligently”
4
LLM-as-judge validation
validation
“Based on the artifacts (screenshot, issue description, etc.), we ask another LLM to evaluate if the issue is genuine or potentially a false positive. A confidence score is then produced, which is later used for filtering out low-confiden…”
5
Issue reported to developer workflow
output
“The test execution is fully integrated into the developer's workflow including Slack and JIRA integration”
Reported outcome

The qa-ai-agent detects 300% more bugs in the same timeframe at 86% lower cost than manual testing, with new tests integrable in as little as 15 minutes, and now executes 40 test scenarios identifying 10 issues weekly.

Reported metrics
AI tester accuracy vs manual75% (AI) vs. 80% (Manual)
Bug detection efficiency vs manual300% more bugs in the same timeframe
New test integration time (prompt already tested)15 minutes
New test integration time (prompt testing needed)approximately 1.5 hour
Show all 10 reported metrics
AI tester accuracy vs manual75% (AI) vs. 80% (Manual)
bug detection efficiency vs manual300% more bugs in the same timeframe
new test integration time (prompt already tested)15 minutes
new test integration time (prompt testing needed)approximately 1.5 hour
cost reduction vs manual testing86% reduction
test scenarios currently executed40
issues identified weekly10
testing time reductionWhat used to take a human tester a week to complete, now can be done in minutes
manual testing replacement target75%
productivity gain from eliminating flakinesshuge productivity gain
Reported stack
qa-ai-agentbrowser-useMongoDBBrowserStackgRPCWebSocketSlackJIRA
Source
https://www.coinbase.com/en-nl/blog/How-We-are-Improving-Product-Quality-at-Coinbase-with-AI-agents
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The qa-ai-agent detects 300% more bugs in the same timeframe at 86% lower cost than manual testing, with new tests integrable in as little as 15 minutes, and now executes 40 test scenarios identifying 10 issues weekly.

What tools did this team use?

qa-ai-agent, browser-use, MongoDB, BrowserStack, gRPC, WebSocket, Slack, JIRA.

What results were reported?

AI tester accuracy vs manual: 75% (AI) vs. 80% (Manual); Bug detection efficiency vs manual: 300% more bugs in the same timeframe; New test integration time (prompt already tested): 15 minutes; New test integration time (prompt testing needed): approximately 1.5 hour (source-reported, not independently verified).

What failed first in this deployment?

Traditional end-to-end integration tests were prone to flakiness, with minor layout adjustments causing failures that required hours of debugging.

How is this quality assurance AI workflow structured?

Natural language test request → Agent browses and acts → LLM identifies issues → LLM-as-judge validation → Issue reported to developer workflow.

Related quality assurance cases