Quality assurance · Production

Faire builds Fairey: an LLM-powered automated code review pipeline using RAG

The problem

Generic code review requirements — style guide compliance, title and description quality, test coverage, and detection of backward-incompatible changes — consume reviewer time despite requiring no deep project context.

Workflow diagram · grounded in source

GitHub webhook fires

trigger

“Fairey is also wired into Github's webhooks, which deliver event payloads whenever interesting things happen on pull requests”

Criteria-based review routing

routing

“For pull requests that meet a review's criteria (e.g. programming language, or specific diff contents), Fairey will then interact with OpenAI to perform the reviews”

RAG function-call review generation

ai_action

“Functions give assistants additional capabilities, such as fetching additional information — a technique known as RAG (Retrieval Augmented Generation)”

Output usefulness check

validation

“After interacting with OpenAI, we check the output to see if Fairey has something useful to say”

Review posted to pull request

output

“When it does, we emit the review on the pull request. Reviews usually include comments and hints, and may even include specific change suggestions for the code”

LLM evaluation and survey feedback

feedback_loop

“we assess the quality of the reviews using two signals: one quantitative, and the other qualitative. Our quantitative evaluation involves using an LLM evaluation framework, while qualitative assessment involves surveying the end user (ou…”

Reported outcome

Automated reviews have achieved positive user satisfaction and high accuracy, streamlining the review process, reducing review latency for simpler problems, and freeing engineers to focus on the most impactful and complex parts of reviews.

Reported metrics

Automated review qualitypositive user satisfaction and high accuracy

Review latency for simpler problemsreducing review latency for simpler problems

Engineer focus freed for complex review workfreeing up our talent to focus on the most impactful and complex parts of the review

Reported stack

FaireyChatGPTRAGJestGentraceCometLLMLangsmithDXlcovGitHub

Source

https://craft.faire.com/automated-code-reviews-with-llms-cf2cc51bb6d3

Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

Automated reviews have achieved positive user satisfaction and high accuracy, streamlining the review process, reducing review latency for simpler problems, and freeing engineers to focus on the most impactful and com…

What tools did this team use?

Fairey, ChatGPT, RAG, Jest, Gentrace, CometLLM, Langsmith, DX, lcov, GitHub.

What results were reported?

Automated review quality: positive user satisfaction and high accuracy; Review latency for simpler problems: reducing review latency for simpler problems; Engineer focus freed for complex review work: freeing up our talent to focus on the most impactful and complex parts of the review (source-reported, not independently verified).

How is this quality assurance AI workflow structured?

GitHub webhook fires → Criteria-based review routing → RAG function-call review generation → Output usefulness check → Review posted to pull request → LLM evaluation and survey feedback.