quality_assurance · saas · workflow
PayPay builds GBB RiskBot: RAG-enhanced LLM code review system using historical incident data
PayPay's code review relied entirely on individual reviewer knowledge and ad-hoc knowledge sharing, with no automated system to systematically prevent recurring incidents across services. Knowledge silos, team turnover, and varying reviewer experience led to inconsistent risk assessment.
How it works
Common implementation structure
How this type of workflow is generally built, generalized across documented cases — not tied to any one vendor's stack. Click any stage to read what happens there. Specific products that implement these stages appear in “Tools commonly seen” below.
Stage 1 · PR opened triggers bot
When a developer opens a pull request, GBB RiskBot automatically analyzes the code changes against the historical incident database.
Tools used
GBB RiskBotGitHub ActionsOpenAI embeddingsLangChainChromaDBChatGPTgpt-4o-minitext-embedding-ada-002RAG
Outcome
GBB RiskBot operates across 12 repositories with 380+ total bot runs, at a total cost of $0.59 USD for the measured month, described as very cost-effective compared to the potential cost of production incidents. The system educates developers and democratizes knowledge across the organization.
Results
Time saved$0.59 USD
Volume380+
Cost replaced$0.001852
Grounding & classification
Source type: technical build writeup
33 fields verified against source quotes.
knowledge searchragsummarizationcode diff prknowledge basebuilder submittedmetric backednamed customerproduction runtime claimedtools describedworkflow describedfinancial servicessoftwarecost reductionemployee productivitytechnical build writeupincident managementquality assurancerag answering