acceptodds
Under review as a conference paper at ICLR 2027

Lego-Review: Scaling Code Review Data with Verification from Pull Request Histories

Abstract

AI coding agents produce a growing share of software changes, making automated code review increasingly important. A promising way to train capable reviewers is to scale the tasks they practice on and the verified trajectories used for training. However, code review yields open-ended, natural-language findings, for which there is no general test-based verifier comparable to the tests used for feature implementation or issue resolution. We observe that pull request (PR) histories offer an implicit verification signal: discussions, code revisions, and other actions can show which findings were accepted or fixed and which were withdrawn or rejected. We introduce Lego-Review, a pipeline to turn this implicit evidence into verifiable review tasks and filtered trajectories. We collect merged GitHub PRs, reconstruct the code snapshots reviewers examined, and use a curation agent to consolidate findings across each PR's discussions and revisions. The agent labels each finding as supported, opposed, or uncertain and binds it to the reviewed snapshot. A finding-level verifier then filters teacher trajectories by crediting recovery of supported high-priority findings and penalizing repetition of opposed ones. From 357k PRs, we build 112k review tasks in eight languages and 24k filtered trajectories. On our proposed Lego-Review-Bench F1 rises from 5.7 for the base model to 43.1 with 1k filtered trajectories and to 52.9 with 16k, showing effective scaling with verification from PR histories. We release the data, the benchmark, and the trained reviewer to support open research on scalable code review.

Primary area:
datasets and benchmarks
Keywords:
Code Review, Software Engineering, Data Pipeline

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rSHoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Lego-Review, a pipeline for constructing large-scale and verifiable code review training data from GitHub Pull Request histories. The key idea is to use review discussions, subsequent code revisions, and PR outcomes as implicit verification signals for natural-language review findings, which otherwise lack a general executable verifier. The authors reconstruct the code snapshot seen by reviewers, use a curation agent to consolidate findings and label them as supported, opposed, or uncertain, and then apply a finding-level verifier to filter teacher-generated review trajectories.

From approximately 357K merged PRs, the authors construct 112K review tasks and 61K review trajectories, of which 24K pass verification. Training Lego-Review-35B-A3B on these validated trajectories improves F1 from 5.7 for the base model to 52.9 on Lego-Review-Bench. Performance continues to improve as the number of validated trajectories increases from 1K to 16K. The experiments also show that finding-level verification is more effective than verifying only the final review decision, while adding clean examples substantially reduces false positives. Overall, the paper demonstrates that PR histories can serve as a scalable verification signal for training code review agents.

Critical strengths
  1. Important and practically relevant problem. Reliable code review is an important bottleneck for AI coding agents, while obtaining large-scale, high-quality review trajectories is difficult. Using existing PR histories as a scalable verification signal is a natural and promising approach.

  2. Comprehensive data construction and evaluation. The paper provides a large-scale dataset and systematically evaluates reference faithfulness, verifier reliability, finding-level versus decision-level verification, scaling with training data, and the effect of clean examples. These experiments provide fairly strong evidence for the proposed pipeline.

Critical weaknesses
  1. PR history is not equivalent to technical ground truth. The audit shows that around 5.1% of supported high-priority findings do not actually hold in the code. Moreover, a finding being fixed or accepted may reflect engineering trade-offs or maintainer preferences rather than technical correctness. Thus, PR history should be viewed as a noisy proxy rather than a correctness oracle.

  2. The training objective may favor reproducing historical reviewer findings rather than discovering novel issues. Findings that are technically valid but absent from the PR history are neither rewarded nor penalized. While this avoids introducing explicit false negatives, it may underestimate novel bug discovery and encourage the model to imitate the distribution of historical reviews.

Questions for authors
  1. How does the paper evaluate novel bug discovery? If a reviewer identifies a technically valid high-priority issue that is not recorded in the PR history, how is this capability measured? Could independent expert review or additional code-level verification be used to estimate the model's ability to discover novel findings?

  2. How sensitive is the approach to the curation and verifier models? If the references were constructed using an independent model or a substantially different reference-generation strategy, would the reported scaling trend and 52.9 F1 still hold?

3 comments

Thank you for recognizing PR history as a promising, scalable verification signal and our comprehensive data construction and evaluation. Below, we address your concerns about PR-history signals, discovery beyond PR history, and model sensitivity. We have also revised the manuscript accordingly and highlighted the changes in blue.

W1. PR history as a supervision signal

Scalable verification with acceptable noise. We prioritize scaling the data while maintaining efficiency and controlling cost. In our audit, 94.9% of supported high-priority findings are technically valid, a level of noise we consider acceptable for scaling. For stricter quality control, when additional budget is available, multiple stronger models could cross-check the references against the code, filter out invalid findings, and further improve trajectory quality.

Learning repository preferences. We agree that PR outcomes can reflect engineering trade-offs and maintainer preferences. Repository context, including available Git history, provides clues to a project’s conventions and past decisions. We expect reviewers trained on these trajectories to explore this context and make judgments aligned with the repository’s practices.

W2 / Q1. Discovery beyond PR history

Learning beyond PR-history references. Like other LLMs trained on historical data, our reviewer draws on GitHub data. We select merged PRs from high-star repositories with multiple maintainers, whose histories already document important review findings and subsequent fixes. Yet our training data go beyond these historical references: 87.4% of defect training trajectories contain further findings, providing additional review content for SFT.

Evaluating findings beyond PR history. Beyond its 209 in-history findings, Lego-Review-Bench also contains 221 beyond-history findings, which were independently identified by multiple strong models and checked against the code. Beyond-history finding recall broadly follows the model ranking observed on overall Recall and F1. Our SFT reviewer reaches 44.7 recall, surpassing the instruction-tuned model’s 16.1 and matching DeepSeek-V4-Flash. These results show that Lego-Review supports the discovery of defects beyond PR history. With sufficient resources, training-task construction could incorporate additional verified reference findings from multiple stronger models, potentially strengthening beyond-history discovery.

ReviewerPrecisionRecallF1Beyond Recall
GLM-5.364.172.668.171.3
GLM-5.3-Flash63.965.864.965.8
DeepSeek-V4-Flash78.249.160.444.7
Qwen3.5-35B-A3B (Instruct)41.417.324.416.1
Gemma 4-31B-it62.410.317.77.8
Qwen3.5-35B-A3B-Base23.53.35.72.6
Lego-Review-35B-A3B (Ours, SFT on 16k data)59.647.652.944.7

Q2. Sensitivity to curation and verifier models

Reference curation and verification are consistent across models. We measure finding coverage F1 by matching each model’s reference findings against those constructed by GLM-5.3-Flash. For verification, we compare their selection decisions on the same trajectories. GPT-6-Astra and GPT-6.1-Sol achieve over 98% finding coverage F1 and agree with GLM-5.3-Flash on over 95% of selection decisions, close to its self-agreement. Our audit further confirms that 99.2% of supported high-priority references preserve the original review claims.

ModelCuration: Finding Coverage F1Verifier: Selection Agreement
GLM-5.3-Flash (self-comparison)99.18%96.81%
GPT-6-Astra98.41%95.21%
GPT-6.1-Sol98.44%95.26%

Our PR-history reference strategy outperforms alternative strategies. At the 8k scale, our strategy reaches 49.7 F1, surpassing 44.8 for independent review without PR-history verification and 46.5 for a PR-history decision reference. With our finding-level references, reviewers trained on Low, Medium, and High-scoring trajectories reach 44.7, 47.7, and 49.7 F1, respectively, showing that our strategy clearly distinguishes trajectories by their training value. This distinction is further supported by defect recognition, which rises from 42.0% to 50.5% from Low to High when reviewing the same defect-containing code (Section 5.3).

Reference and selection strategyPrecisionRecallF1Clean Rate
Alternative strategies
Independent review (without PR-history verification)54.238.144.887.9
PR-history decision reference56.339.646.590.9
Lego-Review: score-based selection
Low score54.737.844.790.9
Medium score55.441.947.790.9
High score57.743.649.790.9
Reviewer_rSHoOpenReview ↗

Response to authors

Thank you for your reply. My concerns have been addressed, and I will update my rating to a higher score.

Thank you for your prompt feedback. We’re glad our response addressed your concerns. Please let us know if you have any further questions.

Reviewer_ePUBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper constructs a dataset for code reviewing and a training trajectory for SFT using sampling from a teacher model and rejective sampling (by filtering out some trajectories that are not consistent with the reference findings). The paper proposed to use PR discussions to establish the ground-truth clean and defect labels for code reviews, of the detective ones, the findings generated will be further labeled a consolidation agent to support, opposed, or uncertain. The paper then use a teacher model to generate review trajectories for SFT. The trajectories are validated by a validator by comparing the findings generated by the teacher model with the referenced findings generated by the consolidate agent from the PR discussions. Experiments show the nice scaling of the SFT-ed model and some ablation studies show the importance of the validation of the high quality trajectories. Human reviews were conducted to validate the effectiveness of the consolidate agent and LLM verifiers.

Critical strengths

++ The paper presents a benchmark Lego-Review-Bench and high quality training trajectories of LLM code reviews. The training trajectories show nice scaling effects.

  • The paper uses human validation to quantify the effectiveness and faithfulness of LLM judges (LLM consolidate agent and LLM verifiers)
  • The paper conduct ablation studies to discuss the effectiveness of LLM verifier and justify why using score=8 to do the filtering.
Critical weaknesses
  • Some presentations in the paper are not clear as indicated in some points in the Questions for Authors block.
  • The F1 and clean rate of the SFT-ed model still has a gap between its teacher model. The paper can benefit if it maximize the trajectories generated and show the convergence point of Figure 3.
Questions for authors
  1. "repetition of opposed findings" is confusing. It can mean a) the verifier generate a finding that is the same as an referenced opposed finding or b) the verifier generate two findings that are similar to an referenced opposed findings.
  2. page 6, lines 282, "the gap justifies the verifiers asymmetry". However, the crediting and penalizing of supported and opposed findings look exactly symmetric to me.
  3. What are the failure reasons that 35.0% of the failure in the opposed findings in section 4.1?
  4. The third paragraph in introduction contains lots of details that should not be in introduction and some of them are not even supported by the main paper, e.g.,"the absense of a historical comment also cannot establish that an unmentioned finding is wrong"/
  5. The third and four paragraphs read confusing in the first sight, I recommend we need to discuss the two purpose of these two paragraph, one is to construct the benchmark (and establish it reference label), and the other is the constructing the trajectories for training.
  6. What about a PR that has a supported finding but is fixed in the follow up in the same PR discussion. is this PR considered as clean or defect?
  7. What LLM is used for the verifier? is it the same, weaker, or stronger than the teacher model?
  8. Why not fine-tune from Qwen3.5-35B-A3B instead and why SFT from the base model, given the instruction model has better performance than Base model as the very beginning?
3 comments

Thank you for recognizing our high-quality review trajectories, their strong scaling behavior, and the validation of our curation agent and verifier. Below, we address your questions about the presentation, methodology, and scaling. We have also revised the manuscript accordingly and highlighted the changes in blue.

W1. Presentation and method clarification

Q1. Removed ambiguous “repetition” wording. We mean the interpretation (a): the teacher’s report raises a finding corresponding to an opposed reference. We have replaced “repetition” with clearer wording like "inclusion" throughout the manuscript.

Q2. Removed confusing “asymmetry” wording. We have removed this confusing term from the manuscript.

Q3. Most valid opposed findings are nits. The reported 35.0% is the technical validity rate of opposed findings, rather than a failure rate. We expanded the audit to 100 opposed findings. Of the 31 technically valid findings, 27 are nits, largely concerning test coverage, documentation, diagnostics, or other improvements. Only 4% (4/100) are valid defects, which maintainers deferred, considered outside the PR’s scope, or opposed over their triggering conditions.

AuditFindingsInvalidValid defectsOther valid nits
Original201307
Extended10069427

Q4. Simplified the introduction and moved the unmatched audit to the main text. We have removed detailed snapshot constraints from the third paragraph. To support our treatment of unmatched findings, we have moved their audit to Section 4.2.

Q5. Clarified the purpose of the two paragraphs. The third paragraph motivates constructing references from PR history, and the fourth describes the construction pipeline. We have revised both paragraphs to make this distinction clear.

Q6. Defect and clean snapshots. Defect and clean tasks refer to PR snapshots: different snapshots of the same PR can form defect or clean tasks. For this PR, the first reviewed snapshot can form a defect task, while the approved final head after the concerns are resolved can form a clean task.

Q7. The verifier uses the same model as the teacher: GLM-5.3-Flash. While the teacher reviews each snapshot independently, the verifier only needs to assess its report against reference findings derived from PR history. We further compare its selection decisions with those of GPT-6-Astra and GPT-6.1-Sol. Both agree with GLM-5.3-Flash on over 95% of decisions.

VerifierGLM-5.3-FlashGPT-6-AstraGPT-6.1-Sol
Agreement with GLM-5.3-Flash96.81%95.21%95.26%

Q8. Isolating gains from review trajectories. We fine-tune the base model without prior post-training to better isolate the gains from our review trajectories. After SFT, it reaches 56.1 F1 on Lego-Review-Bench, compared with 24.4 for the instruction-tuned model.

W2. Keep scaling

We have scaled the dataset to 168k review tasks and 154k teacher trajectories, of which 60k pass verification. Scaling training from 16k to 60k trajectories raises F1 from 52.9 to 56.1, narrowing the gap to the teacher from 12.0 to 8.8 points. The teacher remains ahead with a substantially larger model. The continued gains at 60k show the potential of Lego-Review to scale general-purpose code review across repositories. We provide a detailed analysis of the expanded dataset (see Figures 3 and 4) and the full scaling results (see Figure 6) in the revised manuscript.

ReviewerPrecisionRecallF1Clean Rate
Ours, 16k59.647.652.990.9
Ours, 60k62.550.956.190.9
Teacher63.965.864.993.9
Reviewer_ePUBOpenReview ↗

Response

I thank authors for their response and additional experiment results. I will retain my score.

Thank you for your prompt feedback. Please let us know if you have any further questions. We would be happy to address them individually.

Reviewer_xPYmOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Lego-Review uses PR discussions and subsequent revisions as partial, finding-level supervision for code review. It reconstructs reviewed snapshots, curates supported/opposed/uncertain reference findings, and filters teacher trajectories using finding-level verification. From 356,606 merged PRs it constructs 111,937 tasks and retains 23,955 trajectories. SFT on 16k trajectories improves Lego-Review-Bench F1 from 5.7 to 52.9 and external Martian F1 from 3.2 to 40.2 (Table 4). The paper also audits reference quality, verifier behavior, unmatched findings, and clean tasks, and studies filtering and clean-data mixture choices.

Critical strengths
  1. A useful, audited source of finding-level supervision. The paper turns PR discussions and subsequent revisions into snapshot-specific reference findings, with explicit support labels and deterministic checks on the cited evidence. This addresses a concrete weakness of decision-only filtering: a correct reject decision need not imply that the accompanying findings are correct. The reference, report, unmatched-finding, and clean-task audits provide substantive evidence for the resulting signal rather than relying solely on automatic labels (Section 4; Appendices B–C).

  2. Clear training gains, including an external benchmark. Training on 16k validated trajectories improves Lego-Review-Bench F1 from 5.7 to 52.9 and Martian F1 from 3.2 to 40.2 (Table 4). The finding-level versus decision-level comparison also shows a 3.2-point F1 advantage at both 4k and 8k trajectories, while clean examples improve the clean-task acceptance rate (Figure 3; Table 5). These results support the practical value of the data-construction approach.

Critical weaknesses
  1. The smaller empirical differences need stronger uncertainty estimates. Lego-Review-Bench contains 75 PR snapshots and only 11 clean tasks, and the clean-data proportion is selected using this benchmark (Section 5.4). The reported repetitions are inference runs, whereas Appendix E specifies one training/sampling seed. This is adequate evidence for the large improvement over the base model, but it leaves the stability of the smaller filtering and mixture differences less clear. PR-level paired intervals and additional training seeds would make these comparisons more convincing.

  2. The attribution of the fine-grained filtering gain remains incomplete. The finding-level and decision-level comparisons match trajectory counts and use the same clean subset, which is a useful control. However, the paper does not report whether the resulting defect subsets are comparable in training-token budget, trajectory length, or task/reference difficulty. The verifier also uses a qualitative 0–10 score with a fixed threshold, while the final benchmark uses a separate pairwise matching procedure. Sensitivity and calibration measurements would help establish how much of the observed advantage comes specifically from finding-level verification.

Questions for authors
  1. How stable are the filtering and clean-mixture differences across training seeds and PR-level paired resampling, and was the clean-data proportion selected on a development set separate from the reported evaluation set? The 11 clean tasks also make a task-level breakdown useful for interpreting Clean Rate.

  2. For the finding-level versus decision-level comparison, can you report total training tokens and the distributions of trajectory length and task/reference difficulty, and check whether the advantage persists under a matched token budget? This would strengthen the attribution of the 4k/8k improvement to the filtering criterion.

  3. How sensitive is selection to repeated verifier calls and to the score threshold, and how well does the final pairwise benchmark matcher agree with blinded human matching on a stratified sample? The existing audits substantiate reference quality and production-verifier coverage, but do not directly calibrate the separate final benchmark-matching procedure.

1 comment

Thank you for recognizing our useful, well-audited finding-level supervision and clear training gains across benchmarks. Below, we address your questions about the stability and attribution of the reported gains. We have also revised the manuscript accordingly and highlighted the changes in blue.

W1 / Q1. Training stability and clean-data mixture

Finding-level gains remain stable. We aggregate results across three seeds (see Appendix E.3). Under PR-level paired bootstrap, finding-level verification improves F1 at both scales, with all positive 95% confidence intervals and p < 0.004.

Training scaleDecision-level F1Finding-level F1Δ F1PR-level paired 95% CIp-value
4k44.6547.86+3.22[1.18, 5.19]3.7 × 10⁻³
8k46.1749.96+3.79[1.62, 5.83]2.8 × 10⁻³

Clean-mixture gains remain stable. Our clean-mixture experiment studies the effect of adding clean trajectories rather than seeking an optimal proportion. The 20% clean mixture scales effectively, with F1 increasing from 43.1 at 1k to 52.9 at 16k. The table reports Clean Rate and F1 with PR-level paired 95% confidence intervals across three seeds. Compared with 0%, the 20% mixture improves both Clean Rate and F1, with all positive confidence intervals. The task-level breakdown further shows that passes are non-decreasing from 0% to 40% clean data on 9 of the 11 tasks.

0% clean10% clean20% clean40% clean
Overall results
Clean Rate76.77%88.89%90.91%93.94%
Δ Clean Rate vs. 0%—+12.12+14.14+17.17
95% CI—[6.06, 18.18][9.09, 20.20][7.07, 29.29]
F140.0141.0042.8340.85
Δ F1 vs. 0%—+0.99+2.82+0.84
95% CI—[−1.17, 3.29][0.09, 5.27][−1.18, 3.02]
Clean task passes
go-git/go-git#14460/92/93/96/9
redis/redis#157336/98/98/98/9
helm/helm#325907/99/98/99/9
helm/helm#323037/99/99/99/9
vuejs/core#129747/99/99/99/9
encode/httpx#35718/98/98/99/9
prometheus/prometheus#196828/98/99/99/9
scikit-learn/scikit-learn#349598/98/99/99/9
pytest-dev/pytest#149788/99/99/97/9
ansible/ansible#875308/99/99/99/9
ansible/ansible#874989/99/99/99/9

W2. Attribution of the filtering gains

Q2. Finding-level gains under comparable training budgets. At both 4k and 8k scales, both models are trained on the same number of trajectories for three epochs, with finding-level using slightly fewer training tokens. The two groups also have comparable trajectory lengths and task difficulty, as measured by the number of supported findings per task. These results indicate that the advantage of finding-level over decision-level verification holds under comparable training budgets.

Training DataTraining TokensTraj. Length Median / P90Supported Findings Median / P90
Finding 4k1.0B74.9k / 128.1k2 / 5
Decision 4k1.1B84.4k / 137.6k2 / 5
Finding 8k1.9B74.6k / 127.3k2 / 5
Decision 8k2.2B86.8k / 141.7k2 / 5

Q3. Verifier decisions are stable across repeated calls. We sampled 1,000 selected training trajectories and re-scored them three times with GLM-5.3-Flash. The table reports the mean per-trajectory score standard deviation and pairwise agreement of selection decisions at three thresholds, using complete repeats. At our threshold of 8, agreement reaches 97.40%.

VerifierScore SDAgreement at ≥7Agreement at ≥8Agreement at ≥9
GLM-5.3-Flash0.23498.45%97.40%95.99%

The benchmark matcher agrees with the audit. We audited the benchmark matcher’s finding-to-reference decisions on 200 findings. The matcher achieves 98.0% agreement with the audit on whether each finding matches a reference.

EvaluationFindingsAgreementsAgreement rate
Benchmark matching20019698.0%

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Area Chair

We propose a pioneering data pipeline, Lego-Review, to scale code review agent training through finding-level verification derived from PR histories. We also propose Lego-Review-Bench to evaluate code review agents on defect detection and clean-change recognition. Reviewers recognized the importance and practical relevance of the problem, the scalable verification signal, the comprehensive data construction and evaluation, and the clear training gains across benchmarks.

In response to the main concerns, we strengthened the paper with the following results:

  • Finding-level verification delivers stable gains. Across three seeds, finding-level verification outperforms decision-level verification at both 4k and 8k, with PR-level paired confidence intervals all above zero. The two strategies use comparable training budgets, trajectory lengths, and task difficulty.
  • Lego-Review discovers defects beyond PR history. On Lego-Review-Bench’s beyond-history findings, our SFT reviewer reaches 44.7% recall, surpassing the instruction-tuned model’s 16.1% and matching DeepSeek-V4-Flash.
  • Curation and verification remain consistent across models. Compared with GLM-5.3-Flash, alternative curation models achieve over 98% finding coverage F1, and alternative verifiers agree on over 95% of selection decisions.
  • Review performance continues to scale. Scaling from 16k to 60k verified trajectories improves F1 from 52.9 to 56.1, with continued gains at the largest scale.

These results effectively address the concerns raised by all three reviewers, each of whom gave a positive rating. We have incorporated the corresponding results and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.