- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Unlike prior work on code review comments, which has mainly focused on review generation, issue discovery, and general comment quality, this paper focuses on technical trustworthiness judgment, motivated by the observation that a review comment may appear plausible while still being technically incorrect when considered in the context of the actual code repository. To address this problem, this paper first constructs CRJudgeBench, a benchmark consisting of 1,199 review-comment instances, including 764 trustworthy comments and 435 untrustworthy comments. The untrustworthy examples consist of 65 human-written incorrect comments and 371 synthetic negatives generated from trustworthy comments through negation, symbol substitution, or location reassignment.
This paper also proposes Sentinel, an agentic judge that directly explores the repository to assess the technical trustworthiness of review claims. From the student-generated trajectory, important states such as code search, relevant change inspection, and final reasoning are selected for next-action supervision from a privileged teacher. Although the teacher has access to the ground-truth label, it does not directly reveal the label to the student. Instead, based on the student’s reasoning and tool observations so far, it suggests what additional patch evidence should be inspected next. The teacher-proposed action is then executed in the environment and reviewed by a separate verifier for usefulness, evidence validity, code-version consistency, and potential label leakage. Only approved actions are used for action-level SFT.
Sentinel achieves 76.6% accuracy, outperforming the evaluated baseline models. The improvement is particularly substantial in untrustworthy-comment recall. At the same time, the baseline models already achieve very high recall on trustworthy comments, suggesting that existing models tend to accept plausible but incorrect reviews too readily.
Critical strengths
1. Clear motivation and careful benchmark construction
This paper clearly formulates technical trustworthiness judgment as a distinct evaluation problem that has received limited attention in prior automated code review research. Rather than assessing whether a review comment is merely fluent or relevant, the task asks whether its technical claim is actually correct given the code patch and repository context, making the problem practically important. The construction of CRJudgeBench is also careful. Repositories and PRs are filtered in multiple stages, followed by rule-based and LLM-assisted screening and expert verification. Synthetic negatives are not included solely based on automatic generation; instead, three Python experts review the resulting comments before they are added to the benchmark.
2. Identifying frontier-model failures and providing an effective training-based solution
The experiments clearly show that strong general-purpose LLMs achieve high recall on trustworthy comments while remaining highly vulnerable to plausible but incorrect reviews. In particular, the baselines show very low recall on untrustworthy comments, and several frontier models classify substantially more comments as trustworthy than the actual trustworthy proportion. This paper also goes beyond merely identifying this failure mode by proposing Sentinel, which learns repository-grounded evidence-seeking behavior. Sentinel improves accuracy from 56.82% to 76.60% over the Qwen3-Coder backbone and, more importantly, substantially increases recall on untrustworthy comments from 2.31% to 37.69%.
Critical weaknesses
1. Limited evidence for generalization to real-world invalid reviews
Among the 435 untrustworthy examples in CRJudgeBench, only 65 are actual human-written incorrect comments, while the remaining 370 are synthetic negatives generated by applying negation, symbol substitution, or location reassignment to positive comments. This paper does not explain why these three perturbation types were selected or whether they were designed to reflect failure patterns observed in the 65 real incorrect comments. Therefore, it remains unclear how well these perturbations represent the error distribution of real-world invalid code reviews.
This paper also does not report how human-written negatives and each perturbation type are distributed across the train, validation, and test splits, nor does it provide performance broken down by negative source. Therefore, it is difficult to determine whether Sentinel’s improvement reflects better general technical trustworthiness detection or is primarily driven by specific synthetic perturbation patterns. It is also difficult to assess the extent to which the improvement may depend on adaptation to those synthetic patterns. To validate this more clearly, the authors could report separate performance on human-written negatives or conduct a hold-out evaluation in which a perturbation type unseen during training is reserved for testing.
2. Missing comparison with direct label supervision and Sentinel ablations
Sentinel shows a large improvement over the untrained Qwen3-Coder backbone, but this paper does not compare it against a direct label supervision baseline trained only on the final trustworthy/untrustworthy labels using the same backbone and CRJudgeBench training split. Such a comparison is necessary to determine whether Sentinel’s gains specifically come from action-level evidence-seeking supervision or whether a substantial portion of the improvement could also be achieved through simpler task-specific supervised adaptation. With the current comparison between the untrained backbone and the full Sentinel model, it is difficult to assess what additional benefit the proposed action-level training provides over simpler label-level supervision.
In addition, this paper lacks ablations of the major components of Sentinel’s training pipeline. Sentinel jointly uses privileged teacher supervision at selected trajectory prefixes, teacher-action verification, action-level distillation, and iterative learning in which the updated student is rolled out again. Comparisons between single-round training and full iterative training, or between training with and without reviewer verification, would help clarify how much each component contributes to the final performance improvement.
Questions for authors
1. Perturbation design and negative-source analysis
What was the rationale for selecting negation, symbol substitution, and location reassignment as the three perturbation types? In particular, were these perturbations designed based on an analysis of the failure patterns observed in the 65 human-written incorrect comments identified during manual annotation, or were they selected using separate heuristics? I think the paper should clarify this point.
2. Direct-label baseline and Sentinel ablations
How does Sentinel compare with a direct label supervision baseline trained only on the final trustworthy/untrustworthy labels using the same Qwen3-Coder backbone and CRJudgeBench training split? In addition, it would be helpful to compare single-round action-level training, training without reviewer verification, and the full Sentinel model to clarify how much each component contributes to Sentinel’s final improvement.
2 comments
Thank you for the detailed response and additional experiments. I appreciate the authors' efforts to address my concerns. While the new results clarify several points, the limited evaluation on naturally occurring incorrect reviews and the lack of a single-round training ablation leave my main concerns only partially resolved. Therefore, I maintain my original rating.
Response to Reviewer kswT
W1. Limited evidence for generalization to real-world invalid reviews
We thank the reviewer for this valuable suggestion! We agree that performance breakdowns by negative sources are important. Our 370 synthetic negatives consist of 130 Negation (35.14%), 88 Symbol Substitution (23.78%), and 152 Location Reassignment (41.08%) instances. On the test set, Sentinel achieves 25.00% (10/40), 47.83% (11/23), and 45.10% (23/51) recall on these three categories, respectively, with an overall synthetic-negative recall of 38.60% (44/114). For the 16 naturally occurring incorrect comments, Sentinel correctly identifies 5, achieving 31.25% recall. We agree that holding out an unseen perturbation type would provide further evidence of generalization and will include this experiment as a direction for future work.
Q1. Perturbation design and negative-source analysis
Thank you for this important concern. The three perturbation types were selected based on our manual analysis of the 65 naturally occurring incorrect review comments. We identified symbol-related errors (24 instances), location-related errors (21 instances), and negation-related errors (15 instances) as the three most frequent error patterns. Note that these categories are not mutually exclusive, as some comments contain multiple types of errors.
W2. Missing comparison with direct label supervision and Sentinel ablations
Q2. Direct-label baseline and Sentinel ablations
We thank the reviewer for this valuable suggestion. To address this concern, we conducted additional two experiments with direct-label SFT and without reviewer verification. Using the same Qwen3-Coder backbone and training split, direct-label SFT achieves only 39.55% accuracy, substantially lower than both the original backbone (56.82%) and Sentinel (76.60%). Furthermore, removing reviewer verification reduces Sentinel's accuracy from 76.60% to 70.75% (254/359). We will incorporate these additional results into the revised paper.