- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces a Risk Hypothesis Benchmark to evaluate LLM-based code review beyond final findings. It separates the review process into three stages: risk formulation, prioritization, and verification. The benchmark contains 300 reference risks from 223 review cases across 157 repositories. Experiments with multiple LLMs show that explicit risk hypotheses can improve risk coverage, but do not necessarily lead to better end-to-end code review performance.
Critical strengths
-
The paper addresses an important problem in LLM-based code review. Evaluating intermediate risk formulation and prioritization, rather than only final findings, provides a potentially useful perspective for understanding model failures.
-
The paper investigates multiple stages of code review through controlled comparisons and ablation studies. The reconstruction of review-time evidence also helps make the evaluation more realistic.
Critical weaknesses
-
Limited novelty and insufficient benchmark positioning. The main contribution is a structured risk representation and its associated benchmark, but the underlying ideas of risk generation, prioritization, and verification are not fundamentally new. The paper lacks a systematic comparison with existing code review benchmarks, making its unique contributions unclear. Moreover, the benchmark is insufficiently characterized, with limited analysis of data distributions, difficulty levels, and annotation reliability. Concrete case studies are also missing, making it difficult to understand the benchmark instances and outputs. The related work discussion is limited, with only nine references, further weakening the paper's positioning.
-
Limited experimental insights and unconvincing practical effectiveness. Although the paper includes several experiments, most comparisons are based on internally designed settings, with limited analysis of why different methods succeed or fail. More importantly, the proposed risk-guided workflow performs worse than direct review under matched budgets (Table 4). The positive results from Oracle Selection and Gold Guidance do not establish practical end-to-end effectiveness. The paper lacks deeper error analysis and concrete examples to explain these observations, limiting the insights gained from the benchmark.
Questions for authors
-
What are the key differences between the proposed benchmark and existing code review benchmarks, such as Sphinx, AACR-Bench, and CR-Bench? What important capabilities can this benchmark evaluate that existing benchmarks cannot?
-
Could the authors provide representative benchmark examples and more detailed dataset statistics, including risk categories, difficulty distribution, and annotation quality? How reliable and representative are the 300 reference risks?
-
Why does the risk-guided workflow perform substantially worse than direct review under matched budgets (Table 4)? Is the degradation mainly caused by inaccurate risk generation, poor prioritization, or additional planning overhead? Can the authors provide a detailed failure analysis to support their explanation?