acceptodds
Under review as a conference paper at ICLR 2027

Breaking the Illusion of Review Reliability under Static Evaluation: SCOPE Fuzzing for LLM-based Scientific Reviewers

Abstract

The rapid growth of submissions and reviewing workload has accelerated the use of large language models(LLMs) in peer review. Prior studies suggest that LLM-based reviewers can penalize content perturbations, such as overclaiming, indicating a certain degree of reliability. Yet these conclusions are largely based on a narrow set of perturbation strategies instantiated with static templates, providing limited evidence of actual reliability. In this paper, we construct a three-level evaluation framework covering perturbations to surface presentation, argumentative logic, and value judgment. Experiments on representative LLM-based reviewers reveal two limitations of static evaluation: stratified vulnerability, where perturbation effects depend on whether the paper's original review score is high or low, and perturbation undercoverage, where a single template misses vulnerabilities exposed by diverse realizations. To address these limitations, we propose SCOPE-Fuzzer, a strategy-aware fuzzer that combines feedback-driven strategy selection with adaptive mutation of paper content. By iteratively probing reviewers with dynamic perturbations, SCOPE-Fuzzer consistently uncovers vulnerabilities overlooked by static evaluation and other baselines.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM-based Scientific Reviewer, Reliability against Perturbation, Agent Fuzzing

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VU9FOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the reliability of LLM-based scientific reviewers under subtle, semantics-preserving changes to paper abstracts. The authors introduce TREAP, a framework covering six perturbation strategies across form, reasoning, and judgment levels, and show that static single-template evaluation can miss important vulnerabilities. They further propose SCOPE, an adaptive fuzzing method that searches for paper-specific perturbations which increase review scores, and demonstrate that it uncovers reviewer vulnerabilities more effectively than static and non-adaptive baselines.

Critical strengths
  1. Broader perturbation coverage. TREAP evaluates six perturbation strategies across form, reasoning, and judgment levels rather than relying on a single perturbation type.

  2. Adaptive evaluation. SCOPE turns reviewer robustness testing into an iterative fuzzing problem and consistently finds stronger perturbations than static or non-adaptive baselines.

Critical weaknesses

Limited evaluation scope. The multi-probe and fuzzing experiments use only 93 ICLR 2024 papers, which limits how broadly the findings can be generalized.

Questions for authors

Can you extend the evaluation to the full ICLR 2024 and 2025 paper sets to better assess whether the observed vulnerabilities generalize beyond the current subset?

Reviewer_PJKzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether LLM-based peer reviewers can be manipulated by subtle edits to abstracts. It introduces TREAP, a taxonomy of perturbation strategies. It proposes SCOPE, a budgeted black-box fuzzer. An LLM picks an initial strategy, then a heuristic chooses later actions, and an LLM mutator writes paper-specific rewriting instructions. It conducted experiments with multiple reviewers on 300 ICLR 2024 papers for static analysis and a subset for multi-probe evaluation and fuzzing, with a small human study on semantic preservation and plausibility.

Critical strengths
  • The paper excludes prompt injection and focuses on edits that a real author could make without obvious violations, which is a practical threat.
  • The paper finds out that aggregate, single-template evaluation understates risk and suggests PUR/RUR as a reasonable metrics.
  • TREAP provides a structured set of perturbation strategies
Critical weaknesses
  • Missing a content-neutral rewriting baseline to verify whether SV reflects a distinctive vulnerability to perturbations. In Table 2, the authors observe that low-score papers tend to gain scores after perturbation, and vice versa. This pattern can arise from regression to the mean as disabling stochastic decoding does not eliminate the variability.
  • Similarly, without a content-neutral rewriting baseline, the improvements of SCOPE and Perturbation Undercoverage observation might be attributed to a best-of-k selection effect. Eq. (3) selects the maximum score gain over multiple perturbations, which increases the probability of finding a positive score shift even for random rewrites.
  • Semantic similarity is not equal to scientific presentation quality. Some TREAP strategies, like logic adjustment and verbosity increase, may improve clarity. In such cases, a higher score may reflect a reasonable response to improved presentation, not necessarily indicating vulnerability to adversarial manipulation.
Questions for authors

Can you evaluate SV using a content-neutral rewriting baseline?

Can you compare your method against content-neutral rewrites under same semantic similarity constraints and query budgets to quantify the gains beyond the best-of-k selection effect?

Some related works that miss discussion.

  • Non-adversarial perturbation robustness:
    • SCORE: Systematic Consistency and Robustness Evaluation for Large Language Models ;
    • All Prompts Are Created Equal? Evaluating Robustness of LLM Judges Against Non-Adversarial Prompt Variations
  • Vulnerabilities in AI review: Breaking the Reviewer:
    • Assessing the Vulnerability of Large Language Models in Automated Peer Review Under Textual Adversarial Attacks;
    • Does AI Reviewer See the Full Picture? Attacking and Defending Multimodal Peer Review
Reviewer_DwuuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper evaluates LLM reviewers using six abstract-level perturbation strategies and proposes SCOPE-Fuzzer for feedback-guided search. Experiments show that aggregate score decreases can hide exploitable score increases, and multiple probes expose more vulnerabilities than a single template.

Critical strengths
  • The distinction between average score changes and adversarial robustness is useful and supported across seven reviewer systems.
  • SCOPE achieves promising gains, with significant improvements over the tested baselines on SEA and TreeReview.
Critical weaknesses
  • Higher scores do not necessarily indicate reviewer failure: some edits improve presentation. Human validation covers only six abstract pairs, insufficient to validate successful compound perturbations.
  • Stratified vulnerability may partly reflect baseline-score coupling and bounded ratings. MARG under logic adjustment also contradicts the claimed universal pattern.
  • Best-of-ten probing naturally outperforms one probe. RUR near 0.9 alone does not establish realization dependence.
  • Random selection is competitive, and gains over PAA and SPD are nonsignificant on OpenReviewer. Baseline differences also confound the benefit of feedback.
Questions for authors
  • Are successful compound perturbations factually equivalent, and are their score gains unjustified?
  • Does stratified vulnerability persist under matched neutral paraphrasing?
  • How does undercoverage compare with independent sampling at the same budget?
  • Does feedback beat random selection with all other components matched?
  • How sensitive are the expand/switch decisions to using the first perturbed score rather than the original-paper score as the search reference?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.