- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents Lego-Review, a pipeline for constructing large-scale and verifiable code review training data from GitHub Pull Request histories. The key idea is to use review discussions, subsequent code revisions, and PR outcomes as implicit verification signals for natural-language review findings, which otherwise lack a general executable verifier. The authors reconstruct the code snapshot seen by reviewers, use a curation agent to consolidate findings and label them as supported, opposed, or uncertain, and then apply a finding-level verifier to filter teacher-generated review trajectories.
From approximately 357K merged PRs, the authors construct 112K review tasks and 61K review trajectories, of which 24K pass verification. Training Lego-Review-35B-A3B on these validated trajectories improves F1 from 5.7 for the base model to 52.9 on Lego-Review-Bench. Performance continues to improve as the number of validated trajectories increases from 1K to 16K. The experiments also show that finding-level verification is more effective than verifying only the final review decision, while adding clean examples substantially reduces false positives. Overall, the paper demonstrates that PR histories can serve as a scalable verification signal for training code review agents.
Critical strengths
-
Important and practically relevant problem. Reliable code review is an important bottleneck for AI coding agents, while obtaining large-scale, high-quality review trajectories is difficult. Using existing PR histories as a scalable verification signal is a natural and promising approach.
-
Comprehensive data construction and evaluation. The paper provides a large-scale dataset and systematically evaluates reference faithfulness, verifier reliability, finding-level versus decision-level verification, scaling with training data, and the effect of clean examples. These experiments provide fairly strong evidence for the proposed pipeline.
Critical weaknesses
-
PR history is not equivalent to technical ground truth. The audit shows that around 5.1% of supported high-priority findings do not actually hold in the code. Moreover, a finding being fixed or accepted may reflect engineering trade-offs or maintainer preferences rather than technical correctness. Thus, PR history should be viewed as a noisy proxy rather than a correctness oracle.
-
The training objective may favor reproducing historical reviewer findings rather than discovering novel issues. Findings that are technically valid but absent from the PR history are neither rewarded nor penalized. While this avoids introducing explicit false negatives, it may underestimate novel bug discovery and encourage the model to imitate the distribution of historical reviews.
Questions for authors
-
How does the paper evaluate novel bug discovery? If a reviewer identifies a technically valid high-priority issue that is not recorded in the PR history, how is this capability measured? Could independent expert review or additional code-level verification be used to estimate the model's ability to discover novel findings?
-
How sensitive is the approach to the curation and verifier models? If the references were constructed using an independent model or a substantially different reference-generation strategy, would the reported scaling trend and 52.9 F1 still hold?
3 comments
Response to authors
Thank you for your reply. My concerns have been addressed, and I will update my rating to a higher score.
Thank you for your prompt feedback. We’re glad our response addressed your concerns. Please let us know if you have any further questions.
Thank you for recognizing PR history as a promising, scalable verification signal and our comprehensive data construction and evaluation. Below, we address your concerns about PR-history signals, discovery beyond PR history, and model sensitivity. We have also revised the manuscript accordingly and highlighted the changes in blue.
W1. PR history as a supervision signal
Scalable verification with acceptable noise. We prioritize scaling the data while maintaining efficiency and controlling cost. In our audit, 94.9% of supported high-priority findings are technically valid, a level of noise we consider acceptable for scaling. For stricter quality control, when additional budget is available, multiple stronger models could cross-check the references against the code, filter out invalid findings, and further improve trajectory quality.
Learning repository preferences. We agree that PR outcomes can reflect engineering trade-offs and maintainer preferences. Repository context, including available Git history, provides clues to a project’s conventions and past decisions. We expect reviewers trained on these trajectories to explore this context and make judgments aligned with the repository’s practices.
W2 / Q1. Discovery beyond PR history
Learning beyond PR-history references. Like other LLMs trained on historical data, our reviewer draws on GitHub data. We select merged PRs from high-star repositories with multiple maintainers, whose histories already document important review findings and subsequent fixes. Yet our training data go beyond these historical references: 87.4% of defect training trajectories contain further findings, providing additional review content for SFT.
Evaluating findings beyond PR history. Beyond its 209 in-history findings, Lego-Review-Bench also contains 221 beyond-history findings, which were independently identified by multiple strong models and checked against the code. Beyond-history finding recall broadly follows the model ranking observed on overall Recall and F1. Our SFT reviewer reaches 44.7 recall, surpassing the instruction-tuned model’s 16.1 and matching DeepSeek-V4-Flash. These results show that Lego-Review supports the discovery of defects beyond PR history. With sufficient resources, training-task construction could incorporate additional verified reference findings from multiple stronger models, potentially strengthening beyond-history discovery.
Q2. Sensitivity to curation and verifier models
Reference curation and verification are consistent across models. We measure finding coverage F1 by matching each model’s reference findings against those constructed by GLM-5.3-Flash. For verification, we compare their selection decisions on the same trajectories. GPT-6-Astra and GPT-6.1-Sol achieve over 98% finding coverage F1 and agree with GLM-5.3-Flash on over 95% of selection decisions, close to its self-agreement. Our audit further confirms that 99.2% of supported high-priority references preserve the original review claims.
Our PR-history reference strategy outperforms alternative strategies. At the 8k scale, our strategy reaches 49.7 F1, surpassing 44.8 for independent review without PR-history verification and 46.5 for a PR-history decision reference. With our finding-level references, reviewers trained on Low, Medium, and High-scoring trajectories reach 44.7, 47.7, and 49.7 F1, respectively, showing that our strategy clearly distinguishes trajectories by their training value. This distinction is further supported by defect recognition, which rises from 42.0% to 50.5% from Low to High when reviewing the same defect-containing code (Section 5.3).