acceptodds
Under review as a conference paper at ICLR 2027

CRJudgeBench: Can AI Detect Plausible but Invalid Code Reviews?

12 views

Abstract

Large language models can generate plausible code-review comments, but such comments may contain technically incorrect claims that mislead developers. We study technical trustworthiness judgment: determining whether a review comment’s core technical claims are correct and applicable to the reviewed code in its repository context. Existing code-review benchmarks primarily evaluate review generation, issue discovery, or general comment quality, but do not directly assess whether an agent can determine the technical trustworthy of an individual review comment. To fill this gap, we introduce CRJudgeBench, a benchmark of 1199 instances constructed from real pull requests and expert-verified perturbations, covering both trustworthy and plausible but untrustworthy comments. We further present Sentinel, a repository-grounded agentic judge that actively gathers code evidence to verify review comments before making judgments. Starting from Qwen3-Coder-30B-A3B-Instruct, Sentinel is trained on the CRJudgeBench training split through iterative action-level learning from a privileged teacher. On the 359-instance CRJudgeBench test set, Sentinel achieves 76.60% accuracy, outperforming GLM-5.3 by 6.13 percentage points and its base model by 19.78 points. These results show that even state-of-the-art general-purpose LLMs struggle to identify untrustworthy comments, while iterative action-level learning substantially improves the accuracy of repository-grounded trustworthiness judgments. Our dataset is available at https://anonymous.4open.science/r/CRJudgeBench-3991/.

Primary area:
datasets and benchmarks
Keywords:
Code Review, Large Language Models, Repository-Grounded Agents, Action-Level Distillation, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kswTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Unlike prior work on code review comments, which has mainly focused on review generation, issue discovery, and general comment quality, this paper focuses on technical trustworthiness judgment, motivated by the observation that a review comment may appear plausible while still being technically incorrect when considered in the context of the actual code repository. To address this problem, this paper first constructs CRJudgeBench, a benchmark consisting of 1,199 review-comment instances, including 764 trustworthy comments and 435 untrustworthy comments. The untrustworthy examples consist of 65 human-written incorrect comments and 371 synthetic negatives generated from trustworthy comments through negation, symbol substitution, or location reassignment.

This paper also proposes Sentinel, an agentic judge that directly explores the repository to assess the technical trustworthiness of review claims. From the student-generated trajectory, important states such as code search, relevant change inspection, and final reasoning are selected for next-action supervision from a privileged teacher. Although the teacher has access to the ground-truth label, it does not directly reveal the label to the student. Instead, based on the student’s reasoning and tool observations so far, it suggests what additional patch evidence should be inspected next. The teacher-proposed action is then executed in the environment and reviewed by a separate verifier for usefulness, evidence validity, code-version consistency, and potential label leakage. Only approved actions are used for action-level SFT.

Sentinel achieves 76.6% accuracy, outperforming the evaluated baseline models. The improvement is particularly substantial in untrustworthy-comment recall. At the same time, the baseline models already achieve very high recall on trustworthy comments, suggesting that existing models tend to accept plausible but incorrect reviews too readily.

Critical strengths

1. Clear motivation and careful benchmark construction

This paper clearly formulates technical trustworthiness judgment as a distinct evaluation problem that has received limited attention in prior automated code review research. Rather than assessing whether a review comment is merely fluent or relevant, the task asks whether its technical claim is actually correct given the code patch and repository context, making the problem practically important. The construction of CRJudgeBench is also careful. Repositories and PRs are filtered in multiple stages, followed by rule-based and LLM-assisted screening and expert verification. Synthetic negatives are not included solely based on automatic generation; instead, three Python experts review the resulting comments before they are added to the benchmark.

2. Identifying frontier-model failures and providing an effective training-based solution

The experiments clearly show that strong general-purpose LLMs achieve high recall on trustworthy comments while remaining highly vulnerable to plausible but incorrect reviews. In particular, the baselines show very low recall on untrustworthy comments, and several frontier models classify substantially more comments as trustworthy than the actual trustworthy proportion. This paper also goes beyond merely identifying this failure mode by proposing Sentinel, which learns repository-grounded evidence-seeking behavior. Sentinel improves accuracy from 56.82% to 76.60% over the Qwen3-Coder backbone and, more importantly, substantially increases recall on untrustworthy comments from 2.31% to 37.69%.

Critical weaknesses

1. Limited evidence for generalization to real-world invalid reviews

Among the 435 untrustworthy examples in CRJudgeBench, only 65 are actual human-written incorrect comments, while the remaining 370 are synthetic negatives generated by applying negation, symbol substitution, or location reassignment to positive comments. This paper does not explain why these three perturbation types were selected or whether they were designed to reflect failure patterns observed in the 65 real incorrect comments. Therefore, it remains unclear how well these perturbations represent the error distribution of real-world invalid code reviews.

This paper also does not report how human-written negatives and each perturbation type are distributed across the train, validation, and test splits, nor does it provide performance broken down by negative source. Therefore, it is difficult to determine whether Sentinel’s improvement reflects better general technical trustworthiness detection or is primarily driven by specific synthetic perturbation patterns. It is also difficult to assess the extent to which the improvement may depend on adaptation to those synthetic patterns. To validate this more clearly, the authors could report separate performance on human-written negatives or conduct a hold-out evaluation in which a perturbation type unseen during training is reserved for testing.

2. Missing comparison with direct label supervision and Sentinel ablations

Sentinel shows a large improvement over the untrained Qwen3-Coder backbone, but this paper does not compare it against a direct label supervision baseline trained only on the final trustworthy/untrustworthy labels using the same backbone and CRJudgeBench training split. Such a comparison is necessary to determine whether Sentinel’s gains specifically come from action-level evidence-seeking supervision or whether a substantial portion of the improvement could also be achieved through simpler task-specific supervised adaptation. With the current comparison between the untrained backbone and the full Sentinel model, it is difficult to assess what additional benefit the proposed action-level training provides over simpler label-level supervision.

In addition, this paper lacks ablations of the major components of Sentinel’s training pipeline. Sentinel jointly uses privileged teacher supervision at selected trajectory prefixes, teacher-action verification, action-level distillation, and iterative learning in which the updated student is rolled out again. Comparisons between single-round training and full iterative training, or between training with and without reviewer verification, would help clarify how much each component contributes to the final performance improvement.

Questions for authors

1. Perturbation design and negative-source analysis

What was the rationale for selecting negation, symbol substitution, and location reassignment as the three perturbation types? In particular, were these perturbations designed based on an analysis of the failure patterns observed in the 65 human-written incorrect comments identified during manual annotation, or were they selected using separate heuristics? I think the paper should clarify this point.

2. Direct-label baseline and Sentinel ablations

How does Sentinel compare with a direct label supervision baseline trained only on the final trustworthy/untrustworthy labels using the same Qwen3-Coder backbone and CRJudgeBench training split? In addition, it would be helpful to compare single-round action-level training, training without reviewer verification, and the full Sentinel model to clarify how much each component contributes to Sentinel’s final improvement.

2 comments
AuthorsOpenReview ↗

Response to Reviewer kswT

W1. Limited evidence for generalization to real-world invalid reviews

We thank the reviewer for this valuable suggestion! We agree that performance breakdowns by negative sources are important. Our 370 synthetic negatives consist of 130 Negation (35.14%), 88 Symbol Substitution (23.78%), and 152 Location Reassignment (41.08%) instances. On the test set, Sentinel achieves 25.00% (10/40), 47.83% (11/23), and 45.10% (23/51) recall on these three categories, respectively, with an overall synthetic-negative recall of 38.60% (44/114). For the 16 naturally occurring incorrect comments, Sentinel correctly identifies 5, achieving 31.25% recall. We agree that holding out an unseen perturbation type would provide further evidence of generalization and will include this experiment as a direction for future work.

Q1. Perturbation design and negative-source analysis

Thank you for this important concern. The three perturbation types were selected based on our manual analysis of the 65 naturally occurring incorrect review comments. We identified symbol-related errors (24 instances), location-related errors (21 instances), and negation-related errors (15 instances) as the three most frequent error patterns. Note that these categories are not mutually exclusive, as some comments contain multiple types of errors.

W2. Missing comparison with direct label supervision and Sentinel ablations

Q2. Direct-label baseline and Sentinel ablations

We thank the reviewer for this valuable suggestion. To address this concern, we conducted additional two experiments with direct-label SFT and without reviewer verification. Using the same Qwen3-Coder backbone and training split, direct-label SFT achieves only 39.55% accuracy, substantially lower than both the original backbone (56.82%) and Sentinel (76.60%). Furthermore, removing reviewer verification reduces Sentinel's accuracy from 76.60% to 70.75% (254/359). We will incorporate these additional results into the revised paper.

Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. While the new results clarify several points, the limited evaluation on naturally occurring incorrect reviews and the lack of a single-round training ablation leave my main concerns only partially resolved. Therefore, I maintain my original rating.

Reviewer_2pzfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CRJudgeBench, a benchmark for judging whether individual code‑review comments are technically trustworthy given full repository context, alongside Sentinel, an agentic judge trained via LEAP action‑level distillation.

Critical strengths

The paper identifies an under‑studied problem: existing code‑review benchmarks mostly focus on generating reviews or detecting code defects, rather than verifying whether a given review comment itself is technically correct. The real‑world motivation from prior industry observations (high rejection rate for AI‑generated reviews) is compelling.

Critical weaknesses

Limited practical utility of the core task The task is formulated as binary classification: trustworthy / untrustworthy. However, real‑world code review involves many subjective trade‑offs, engineering preferences and soft suggestions, which are not absolute true/false cases. Though authors filtered subjective comments, there is no quantitative statistics reporting the proportion of objective verifiable comments in the final dataset.

Severe distribution shift risk Out of 435 untrustworthy instances, only 65 are naturally occurring real‑world invalid reviews. The remaining 370 negatives are artificially synthesized by perturbing ground‑truth positive comments via three simple operations: negation, symbol replacement and location relocation.

Limited scope All data are Python‑only. Generalization to other programming languages remains unproven.

Questions for authors

Beyond the aforementioned weaknesses, I also question the overall research direction. Given that GPT‑5.6‑Sol, a state‑of‑the‑art closed‑source model, is already employed as the privileged teacher for trajectory correction, it is unclear why the authors prioritize investigating action‑level distillation onto an open‑source student agent, instead of focusing on improving the intrinsic capabilities of top‑tier closed‑source models (e.g., via prompt engineering, few‑shot learning, or tool‑call optimization) to directly solve this judging task.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 2pzf

W1. Limited practical utility of the core task

We thank the reviewer for this comment. Our work specifically focuses on evaluating the technical trustworthiness of code-review comments. We define a review comment as trustworthy when its judgments are grounded in the code patches and relevant repository context, and contain no errors that could mislead a developer’s understanding or subsequent code refinement actions. Subjective aspects, such as engineering preferences and stylistic suggestions, fall outside our current scope. We will clarify this distinction and discuss extending our framework to subjective review comments as future work.

W2. Severe distribution shift risk

Thank you for this important concern. The three perturbation types were selected based on our manual analysis of the 65 naturally occurring incorrect review comments. We identified symbol-related errors (24 instances), location-related errors (21 instances), and negation-related errors (15 instances) as the three most frequent error patterns. Note that these categories are not mutually exclusive, as some comments contain multiple types of errors.

W3. Limited scope

We thank the reviewer for pointing out this limitation. We agree that evaluating generalization across programming languages is important. We will explicitly acknowledge this limitation in the revised paper and include cross-language evaluation as a direction for future work.

Q1. Beyond the aforementioned weaknesses, I also question the overall research direction. Given that GPT‑5.6‑Sol, a state‑of‑the‑art closed‑source model, is already employed as the privileged teacher for trajectory correction, it is unclear why the authors prioritize investigating action‑level distillation onto an open‑source student agent, instead of focusing on improving the intrinsic capabilities of top‑tier closed‑source models (e.g., via prompt engineering, few‑shot learning, or tool‑call optimization) to directly solve this judging task.

This is a really valuable question! Our motivation for training a model is its lower deployment cost, accessibility, and data privacy advantages compared to GPT-5.6. Moreover, we can’t train the closed-source model GPT-5.6. Importantly, our experiments show that GPT-5.6 with prompting and few-shot learning still underperforms our specialized model.

Reviewer_JZ5LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CRJudgeBench, a benchmark for evaluating whether a coding agent can judge the technical correctness of code-review comments. The benchmark includes 829 natural comments collected from 9 repositories and 370 synthetic incorrect comments. The authors train Sentinel (Qwen3-Coder-30B-A3B-Instruct) on the benchmark's training split using LEAP-style action-level supervision and report improved performance on its held-out test split.

Critical strengths

This paper explores whether coding agents can verify the technical correctness of code-review comments. The dataset links comments to real PR patches and provides fixed, PR-disjoint splits, giving a concrete resource for comparing judges and inspecting their errors.

Critical weaknesses
  1. The benchmark lacks demonstrated realism and a sound scoring protocol. The dataset mixes 829 natural comments (764 correct and 65 incorrect) and 370 synthetic comments (all labeled incorrect), so every positive is natural while 85.1% of negatives are synthetic and limited to three perturbation types. In the test set, 114 / 130 negatives are synthetic, leaving only 16 natural errors. The paper does not justify whether this distribution fully reflects real-world situations. Moreover, the scoring credits judgments on repaired comments rather than requiring a correct verdict on the original comment.

  2. The contribution is limited and generalization remains unproven. Sentinel applies existing LEAP methods without SFT or trajectory distillation baselines, leaving the value of its specific training design unclear. Furthermore, the evaluation (test set) uses the same data construction process and predominantly draws examples from repositories represented in training, without dedicated out-of-distribution or independent real-world evaluation. Therefore, it is unclear whether the training improves the model's ability to identify incorrect information more generally, or simply adapts it to this benchmark's specific data distribution.

Questions for authors
  1. Why did you construct synthetic negatives only using these three perturbation types, and how were their proportions chosen? Do they reflect the errors in real code reviews?
  2. Could you evaluate Sentinel on independent but related benchmarks, and compare it with the baselines and without further training, to show whether training on CRJudgeBench produces general gains beyond this dataset?
  3. Please report the results based on the final verdict about the original comment, and specify how many examples per model receive credit solely through the comment-repair exception?
2 comments
AuthorsOpenReview ↗

Response to Reviewer JZ5L

W1. The benchmark lacks demonstrated realism and a sound scoring protocol.

Q1. Why did you construct synthetic negatives only using these three perturbation types, and how were their proportions chosen? Do they reflect the errors in real code reviews?

Thank you for this important concern. This is a really valuable question! The three perturbation types were selected based on our manual analysis of the 65 naturally occurring incorrect review comments. We identified symbol-related errors (24 instances), location-related errors (21 instances), and negation-related errors (15 instances) as the three most frequent error patterns. Note that these categories are not mutually exclusive, as some comments contain multiple types of errors.

W2. The contribution is limited and generalization remains unproven.

We thank the reviewer for this valuable comment. We will add this limitation in the revised paper. However, we would like to clarify that CRJudgeBench employs repository-level stratified random sampling, and the repositories in the test set differ from those in the training set, requiring the agent to investigate unseen codebases and repository contexts rather than memorize training examples. Furthermore, our additional direct-label SFT experiment achieves only 39.55% accuracy, compared to 76.60% for Sentinel, demonstrating the benefit of action-level supervision over simple label-based adaptation.

Q2. Could you evaluate Sentinel on independent but related benchmarks, and compare it with the baselines and without further training, to show whether training on CRJudgeBench produces general gains beyond this dataset?

We thank the reviewer for this suggestion. However, to the best of our knowledge, no existing benchmark directly evaluates the technical trustworthiness of code-review comments. Related benchmarks focus on different tasks, such as review generation or bug detection. We acknowledge this limitation and will discuss external generalization as future work in the revised paper.

Q3. Please report the results based on the final verdict about the original comment, and specify how many examples per model receive credit solely through the comment-repair exception?

Thank you for the suggestion. We re-evaluated all models strictly based on their final verdict on the original review comment, without applying the comment-repair exception. The updated results are as follows:

ModelAccuracy (%)Trustworthy F1 (%)Untrustworthy F1 (%)
Sentinel76.6084.3353.85
Claude Opus 565.4678.4712.68
GPT-5.565.1778.2512.68
GLM-5.364.7178.207.35
Kimi-K364.7178.207.35
Deepseek-V4.1-Flash64.5377.998.63
Qwen3-Coder-30B-A3B-Instruct56.8272.173.73

Sentinel maintains an 11.14 percentage-point accuracy advantage over the strongest baseline (Claude Opus 5) and a 19.78 percentage-point improvement over its original Qwen3-Coder backbone, demonstrating that its gains remain substantial under the strict final-verdict evaluation protocol.

Thank you for your rebuttal, but it does not establish generalization beyond this benchmark or practical benefits in real code-review workflows (or even more general coding tasks like code generation and software development). It remains unclear whether training improves ability more generally or simply performance on this dataset. I will maintain my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.