Quality assurance · Production

PeterCat: Building an AI assistant for GitHub with RAG, LangChain, and Supabase

The problem

Developers waste hours searching GitHub issues and search engines for answers, then wait days for maintainer responses; general-purpose LLMs like ChatGPT hallucinate when asked about niche open-source repositories or specific framework quirks.

First attempt

The initial prototype simply relocated the language model service without addressing its inherent weaknesses for repository-specific knowledge.

Workflow diagram · grounded in source
1
Repository content vectorized
integration
“I needed to vectorize valuable repository content, including code and historical issues, storing them in a vector database. Second, I had to develop knowledge retrieval tools that could use vector-based search to retrieve relevant content”
2
GitHub webhook trigger
trigger
“GitHub notifies the GitHub Assistant through WebHooks with relevant information”
3
Agent routing
routing
“the Assistant determines if the PR review agent needs to be activated”
4
RAG knowledge retrieval
ai_action
“we first vectorize the user's input and then match it against our vector database”
5
LLM response refinement
ai_action
“we pass the retrieved text content through the language model for comprehension and refinement, ultimately producing responses that better match user needs”
6
Post comments or issue reply
output
“it actively participates in both Issue discussions and PR reviews”
Reported outcome

Within three months of its September 2024 open source release, PeterCat garnered over 850 stars and was adopted by 178 open source projects; in one documented case it saved a developer hours of frustration resolving Ant Design table component issues.

Reported metrics
GitHub stars within 3 monthsover 850
Open source projects adopted178
Developer time savedsaving them hours of frustration
Reported stack
LangChainFastAPISupabaseAWS LambdaRAGChatGPTOpenAIGitHub AppGitHub
Source
https://medium.com/@petercat.assistant/from-2-a-m-frustrations-to-smarter-repositories-how-i-built-an-ai-assistant-for-github-b7d6e641aa1d
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Within three months of its September 2024 open source release, PeterCat garnered over 850 stars and was adopted by 178 open source projects; in one documented case it saved a developer hours of frustration resolving A…

What tools did this team use?

LangChain, FastAPI, Supabase, AWS Lambda, RAG, ChatGPT, OpenAI, GitHub App, GitHub.

What results were reported?

GitHub stars within 3 months: over 850; Open source projects adopted: 178; Developer time saved: saving them hours of frustration (source-reported, not independently verified).

What failed first in this deployment?

The initial prototype simply relocated the language model service without addressing its inherent weaknesses for repository-specific knowledge.

How is this quality assurance AI workflow structured?

Repository content vectorized → GitHub webhook trigger → Agent routing → RAG knowledge retrieval → LLM response refinement → Post comments or issue reply.

Related quality assurance cases