acceptodds
Under review as a conference paper at ICLR 2027

Better Risk Hypotheses, Better Code Reviews? Diagnosing Risk Formulation and Prioritization in LLMs

Abstract

AI-assisted coding can accelerate code production, increasing the need for efficient and reliable code review. Yet evaluating LLM reviewers through final findings alone obscures whether failures arise from overlooking relevant risks, prioritizing unproductive investigations, or failing to verify a concern. We introduce a diagnostic framework centered on Risk Hypotheses: structured, testable descriptions of potential failures and the evidence needed to investigate them. From real review histories, we curate 300 reference risks grounded in reconstructed review-time evidence, distinguishing concerns worth investigating from those verifiable in a fixed environment. The framework evaluates risk formulation, budgeted prioritization, and risk-guided review. Controlled comparisons isolate selection effects using fixed candidate pools and verifiers, while Gold-derived guidance and field ablations probe how hypothesis content affects downstream findings. Initial experiments show that broader risk coverage does not necessarily improve top-ranked coverage, while Gold-derived guidance can improve finding recall for some reviewers. We also identify grounded risks that require evidence unavailable in the review environment. Together, these observations show why recognizing a relevant risk, prioritizing its investigation, and substantiating a finding require separate assessment. Our framework connects these intermediate capabilities to review outcomes, enabling systematic diagnosis of when better risk hypotheses lead to better code reviews.

Primary area:
datasets and benchmarks
Keywords:
code review, large language models, repository-level reasoning, risk hypotheses, software engineering, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_HZEqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a Risk Hypothesis Benchmark to evaluate LLM-based code review beyond final findings. It separates the review process into three stages: risk formulation, prioritization, and verification. The benchmark contains 300 reference risks from 223 review cases across 157 repositories. Experiments with multiple LLMs show that explicit risk hypotheses can improve risk coverage, but do not necessarily lead to better end-to-end code review performance.

Critical strengths
  1. The paper addresses an important problem in LLM-based code review. Evaluating intermediate risk formulation and prioritization, rather than only final findings, provides a potentially useful perspective for understanding model failures.

  2. The paper investigates multiple stages of code review through controlled comparisons and ablation studies. The reconstruction of review-time evidence also helps make the evaluation more realistic.

Critical weaknesses
  1. Limited novelty and insufficient benchmark positioning. The main contribution is a structured risk representation and its associated benchmark, but the underlying ideas of risk generation, prioritization, and verification are not fundamentally new. The paper lacks a systematic comparison with existing code review benchmarks, making its unique contributions unclear. Moreover, the benchmark is insufficiently characterized, with limited analysis of data distributions, difficulty levels, and annotation reliability. Concrete case studies are also missing, making it difficult to understand the benchmark instances and outputs. The related work discussion is limited, with only nine references, further weakening the paper's positioning.

  2. Limited experimental insights and unconvincing practical effectiveness. Although the paper includes several experiments, most comparisons are based on internally designed settings, with limited analysis of why different methods succeed or fail. More importantly, the proposed risk-guided workflow performs worse than direct review under matched budgets (Table 4). The positive results from Oracle Selection and Gold Guidance do not establish practical end-to-end effectiveness. The paper lacks deeper error analysis and concrete examples to explain these observations, limiting the insights gained from the benchmark.

Questions for authors
  1. What are the key differences between the proposed benchmark and existing code review benchmarks, such as Sphinx, AACR-Bench, and CR-Bench? What important capabilities can this benchmark evaluate that existing benchmarks cannot?

  2. Could the authors provide representative benchmark examples and more detailed dataset statistics, including risk categories, difficulty distribution, and annotation quality? How reliable and representative are the 300 reference risks?

  3. Why does the risk-guided workflow perform substantially worse than direct review under matched budgets (Table 4)? Is the degradation mainly caused by inaccurate risk generation, poor prioritization, or additional planning overhead? Can the authors provide a detailed failure analysis to support their explanation?

Reviewer_HYWxOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The widespread adoption of AI-assisted programming has heightened the need for reliable code review. However, simply evaluating the final output of a Large Language Model (LLM) review makes it impossible to pinpoint the root cause of a review failure—whether it stemmed from overlooking relevant risks, prioritizing unproductive lines of inquiry, or failing to substantiate a specific concern. This paper introduces a diagnostic framework centered on "risk hypotheses"—structured, testable descriptions of potential faults and the evidence required to investigate them. Drawing on real-world review histories, the authors constructed a benchmark comprising 300 structured reference risks, grounded in repository evidence available at the time of review. This benchmark evaluates three key capabilities: risk formulation, prioritization, and the impact of risk-driven guidance on the review process. Experiments demonstrate that hypothesis generation yields broader coverage of observed risks, though it does not necessarily improve coverage among top-priority items; improvements in prioritization vary by model; and specific risk attributes provide information crucial for identifying valid risks. By linking intermediate capabilities—such as identifying relevant risks, prioritizing investigations, and substantiating findings—to final review outcomes, this framework connects risk formulation and selection to downstream findings, enabling a systematic diagnosis of when "better risk hypotheses" lead to "better code reviews." This benchmark provides concrete evaluation targets for understanding and improving how LLM-based reviewers transform potential risks into substantiated findings.

Critical strengths

None

Critical weaknesses
  • This paper bears strong hallmarks of having been generated by an LLM agent, exceeding the scope of "language editing and translation" declared in the authors' AI Use Statement. Specific indicators include the frequent use of fabricated concepts, a glaring lack of explanation regarding the research background and introduced terminology, excessively defensive writing, discussion which is full of useless drivel, and an unusually short reference list. The writing is virtually unreadable. Without AI assistance, it is nearly impossible to grasp the paper's core points or content. The writing is disastrously poor.
  • The paper fails to review the state of the art or discuss the current usage of AI-driven programming in code review. Consequently, there is no justification for the risk assumptions proposed in the work, and neither the paper's value nor its technical soundness is demonstrated. In short, I see no substantive contribution to the fields of code review or software engineering.
Questions for authors
  • What is the current state of AI coding in the code-review field? What related lines of work exist in this area, what problems do they address, and how do they address them?
  • How should we understand the relationship between gold risk hypothesis-matching recall and “better code reviews”? The paper argues that text similarity is a poor proxy for review quality, yet the headline numbers are largely coverage of curated reference risks. Please clarify what evidence supports that higher strict finding recall / gold match corresponds to better real-world review outcomes.
  • What is the practical significance of the reported results for real-world code review? As the paper stands, Oracle Selection and Gold Guidance are not deployable, Self underperforms Direct under a matched budget, and the practical selectors show small or inconsistent gains. Beyond a diagnostic benchmark, what concrete implication can a practitioner or tool-builder take from these experiments to improve a real LLM code reviewer?
Reviewer_17cMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a diagnostic framework for LLM-based code review that separates risk formulation, risk selection, and finding verification. The benchmark contains 300 structured reference risks across 223 review cases from 157 repositories, reconstructed from historical reviews and review-time code. Four experiments examine explicit hypothesis generation, selection from fixed candidate pools, risk-guided review under matched total resource limits, and ablations of reference-derived guidance.

Explicit hypothesis generation improves reference coverage relative to normalized review comments in the reported paired comparisons, but does not consistently improve early ranking. Oracle selection improves downstream finding recall, whereas self-generated guidance lowers recall relative to direct review in the tested budget setting. Gold Guidance and field ablations provide exploratory evidence about information useful for verification.

Critical strengths

1. Useful intermediate evaluation targets. The framework makes it possible to distinguish a relevant concern that was never proposed from one that was proposed but not selected, or investigated without being substantiated. Accounting for the evidence available in the review environment further improves the diagnostic value. This is a meaningful contribution beyond measuring the final comments alone.

2. Informative experiments on risk selection and guidance content. RQ2 holds candidate content, verifier settings, and verification budgets fixed while varying the selected hypotheses. Oracle selection improves PR-macro Strict Finding Recall by 12.26 percentage points with GPT-5.6-sol and 18.33 points with DeepSeek Pro, identifying recoverable losses within a fixed candidate pool. RQ4 complements this analysis by removing individual fields from reference-derived guidance while keeping the remaining fields unchanged. This provides a useful way to investigate which information helps verification, with the observed drops after removing the failure mechanism or verification plan suggesting directions for improving risk guidance. Both experiments contribute to the diagnostic value of the framework, although their findings warrant validation at a larger scale.

Critical weaknesses

1. Limited quantitative validation of reference quality and automated evaluation. Sections 3.4-3.6 describe model-assisted admission and manual review, but the paper does not report the number of human reviewers per item, whether reviews were independent, how disagreements were resolved, or how often human review corrected the model-admitted candidates. Finding validity and strict reference matching are also judged automatically, without a reported quantitative calibration against expert assessments. These omissions leave the reliability of the measurement process insufficiently established. Since the benchmark and diagnostic evaluation are the central contributions, evidence about label quality and judgment errors is important to the overall assessment.

2. Small downstream evaluations and an insufficiently explained choice of experimental cohorts and settings. RQ2 uses 53 and 30 paired PRs for the Oracle comparisons, but only 20-25 for the practical selectors. RQ3 uses 60 PRs with 68 eligible risks. RQ4 uses 30 development or previously exposed PRs with 35 eligible risks, with only 25 PRs common to all eight conditions. These are useful initial experiments, but their scale limits confidence in the generalizability of the selection, workflow, and field-ablation results. In particular, the practical selector gains are only 4-5 percentage points, and Appendix D reports substantial uncertainty in the ablation effects.

Different research questions reasonably require different controls, and the paper explains some design choices, such as the fixed candidate pool in RQ2. However, it does not adequately explain why these particular cohorts and sizes were chosen, how they overlap, or why the model and evaluation configurations differ across experiments. The distinction between research-driven choices, verification eligibility, and practical constraints remains unclear. A consolidated account of these choices and substantially larger evaluations would make the diagnostic findings more convincing and their scope easier to assess.

Questions for authors

1. Human review and evaluator calibration. Could you provide the human review protocol, including the number of reviewers per item, whether reviews were independent, how disagreements were resolved, and the frequency of corrections? Have automated finding-validity and reference-matching judgments been calibrated against expert assessments? Quantitative agreement and an analysis of judgment errors would help establish the reliability of the benchmark and comparisons.

2. RQ1 comparison and candidate counts. Please include raw Direct Review (DR) alongside Review-Norm and DH in Table 2, preferably on a common subset across all three conditions, with the numbers of cases and Gold risks reported. This matters for GLM: Appendix C reports 45.2% for DR, 35.7% for Review-Norm, and 40.5% for DH on the common subset. Please also report actual candidate-count distributions by condition, since the text associates longer lists with additional coverage. These additions would help explain how much of the gain comes from proposing more candidates and how much depends on normalization.

3. Rationale for the different experimental settings. Could you explain how the cohorts for RQ2-RQ4 were selected, why their sizes differ, and how the PRs and repositories overlap across experiments? Please provide a compact overview of the cohorts, eligible risks, model roles, and evaluation protocols, and explain which differences are required by the research questions and which arise from practical constraints. Please also distinguish initial cohort selection from exclusions due to unavailable outputs or judgments. This would clarify which conclusions can be connected across the experiments.

4. Scale of downstream evaluation. Please substantially expand RQ2-RQ4 to a larger set of verification-eligible PRs to assess whether the observed selection gains, end-to-end workflow differences, and field-ablation effects are robust across review cases. For RQ4, evaluation on previously unexposed PRs would be particularly valuable. How large is the available pool of eligible cases, and what constraints prevented broader evaluation in the current study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.