Quality assurance · Production

Atlassian ML comment ranker reduces PR cycle time by 30% for Rovo Dev code reviewer agent

The problem

LLM-generated code review comments without filtering were noisy, nit-picky, or factually wrong, directly causing negative user feedback. Heuristic-based filters improved precision but sacrificed recall and were too rigid to adapt to foundation model changes or product rollouts.

First attempt

Heuristic-based filters, including LLM-based comment categorization, could not fully leverage ML and transformer architectures and lacked adaptability to upstream LLM changes and new code patterns, requiring replacement by a more holistic and scientific approach.

Workflow diagram · grounded in source
1
PR creation triggers agent
trigger
“The code reviewer agent gets triggered when a PR is created”
2
LLM generates review comments
ai_action
“it first calls a LLM to generate original code review comments based on the code diff, PR title/description, and Jira work item/description”
3
Comment ranker scores each comment
ai_action
“Those originally generated comments now become the input for the comment ranker to perform its task of selecting useful comments”
4
Threshold gate filters comments
validation
“any comment that we want to post has to pass a threshold score, chosen and optimized through A/B testing”
5
Selected comments posted to PR
output
“the comment will then show "outdated" status in Bitbucket Cloud”
6
Code resolution feeds model retraining
feedback_loop
“We used code resolution binary outcome from internally-sourced dogfooding data as the ground truth to train our model”
Reported outcome

The comment ranker drove CRR to 40–45% (near the human benchmark of ~45%), reduced PR cycle time by 30%, and remained stable when the generation model switched from GPT-4o to Claude Sonnet 3.5.

Reported metrics
PR cycle time reduction30%
code resolution rate (CRR) achieved40% ~ 45%
human benchmark CRR~45%
monthly active users (MAUs)10K+
Show all 8 reported metrics
PR cycle time reduction30%
code resolution rate (CRR) achieved40% ~ 45%
human benchmark CRR~45%
monthly active users (MAUs)10K+
PRs reviewed monthly43K+
external active beta customers400+
training data comments captured53K+
training data growth after retrainingreview comments in training data increased from ~10K to ~53K
Reported stack
Rovo DevModernBERTHuggingFaceGPT-4oClaude Sonnet 3.5Sonnet 4Bitbucket CloudJira
Source
https://www.atlassian.com/blog/atlassian-engineering/ml-classifier-improving-quality?utm_source=substack&utm_medium=email
Read source ↗

Frequently asked questions

What did this team achieve with this AI workflow?

The comment ranker drove CRR to 40–45% (near the human benchmark of ~45%), reduced PR cycle time by 30%, and remained stable when the generation model switched from GPT-4o to Claude Sonnet 3.5.

What tools did this team use?

Rovo Dev, ModernBERT, HuggingFace, GPT-4o, Claude Sonnet 3.5, Sonnet 4, Bitbucket Cloud, Jira.

What results were reported?

PR cycle time reduction: 30%; code resolution rate (CRR) achieved: 40% ~ 45%; human benchmark CRR: ~45%; monthly active users (MAUs): 10K+ (source-reported, not independently verified).

What failed first in this deployment?

Heuristic-based filters, including LLM-based comment categorization, could not fully leverage ML and transformer architectures and lacked adaptability to upstream LLM changes and new code patterns, requiring replaceme…

How is this quality assurance AI workflow structured?

PR creation triggers agent → LLM generates review comments → Comment ranker scores each comment → Threshold gate filters comments → Selected comments posted to PR → Code resolution feeds model retraining.

Related quality assurance cases