Quality assurance · Production

Coinbase's QA AI agent detects 300% more bugs at 86% lower cost than manual testing

The problem

Coinbase's manual QA process was slow and expensive, and traditional end-to-end integration tests were flaky — minor layout changes caused failures requiring hours of debugging, with no scalable path to expanding coverage.

First attempt

Traditional end-to-end integration tests were prone to flakiness; minor layout changes caused test failures that required hours of debugging.

Workflow diagram · grounded in source
1
Natural language test request
trigger
“a prompt such as "log into coinbase test account in Brazil, and buy 10 BRL worth of BTC" is sufficient to initiate a test”
2
Agent browses and acts
ai_action
“the agent directly uses visual and textual data from coinbase.com to determine the next logical action to complete the task”
3
LLM identifies issues
ai_action
“it leverages the LLM's reasoning capabilities to identify issues intelligently”
4
LLM judges bug validity
validation
“we ask another LLM to evaluate if the issue is genuine or potentially a false positive. A confidence score is then produced, which is later used for filtering out low-confidence issues”
5
Developer workflow integration
integration
“The test execution is fully integrated into the developer's workflow including Slack and JIRA integration”
Reported outcome

The qa-ai-agent detects 300% more bugs in the same timeframe at 86% lower cost than manual testing, currently runs 40 test scenarios, identifies 10 issues weekly, and is on track to eventually replace at least 75% of current manual testing.

Reported metrics
Testing effort goal10x our testing effort at 1/10 the cost
Test completion timea week to complete, now can be done in minutes
AI accuracy vs manual75% (AI) vs. 80% (Manual)
Additional bugs detected vs manual300%
Show all 10 reported metrics
testing effort goal10x our testing effort at 1/10 the cost
test completion timea week to complete, now can be done in minutes
AI accuracy vs manual75% (AI) vs. 80% (Manual)
additional bugs detected vs manual300%
new test integration time (pre-tested prompt)15 minutes
new test integration time (new prompt)approximately 1.5 hour
cost reduction vs manual testing86%
active test scenarios40
issues identified weekly10
manual testing to be replaced by AI75%
Reported stack
browser-useMongoDBBrowserStackgRPCWebSocketLLMSlackJIRA
Source
https://www.coinbase.com/en-it/blog/How-We-are-Improving-Product-Quality-at-Coinbase-with-AI-agents
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The qa-ai-agent detects 300% more bugs in the same timeframe at 86% lower cost than manual testing, currently runs 40 test scenarios, identifies 10 issues weekly, and is on track to eventually replace at least 75% of…

What tools did this team use?

browser-use, MongoDB, BrowserStack, gRPC, WebSocket, LLM, Slack, JIRA.

What results were reported?

Testing effort goal: 10x our testing effort at 1/10 the cost; Test completion time: a week to complete, now can be done in minutes; AI accuracy vs manual: 75% (AI) vs. 80% (Manual); Additional bugs detected vs manual: 300% (source-reported, not independently verified).

What failed first in this deployment?

Traditional end-to-end integration tests were prone to flakiness; minor layout changes caused test failures that required hours of debugging.

How is this quality assurance AI workflow structured?

Natural language test request → Agent browses and acts → LLM identifies issues → LLM judges bug validity → Developer workflow integration.

Related quality assurance cases