acceptodds
Under review as a conference paper at ICLR 2027

ARD-Review: Generating Accurate, Reasonable, and Diverse AI Peer Review

17 views

Abstract

Current AI review systems tend to generate generic, unsuited, nitpicky, and invalid (GUNI) feedback, and this failure mode risks homogenizing not only reviews but, through defensive writing, the papers themselves. We propose ARD-Review, a split-role pipeline that combines a high-recall harsh critic, an adversarial merger that aggressively filters GUNI content, and a retrieval-based calibration step that anchors scores to human-reviewed reference papers. We also introduce a new evaluation suite and benchmark to quantify the GUNI tendency in AI-generated reviews. On a leakage-controlled dataset of ICLR 2026 pre-rebuttal reviews, our method outperforms prior systems in scoring accuracy with the least sycophancy bias, is the only method significantly better than the human one-vs-one baseline, and wins 80% pairwise validity comparisons while keeping cross-paper overlap close to the direct-review level. SycoPaperEval: 46 poor papers (average score ) reviewed by three models known by the community to be scyphantic or contrarian toward papers due to their training. We further show that the same calibration mechanism can recalibrate individual human scores and can substantially reduce the spherical nature of backbones.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI Review, Agent, Reviewing Agent

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8TnSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ARD-Review to reduce generic, unsuited, nitpicky, and invalid criticism in AI-generated peer reviews. The pipeline first uses a harsh critic to generate candidate comments, then applies a merger to verify and filter unreasonable feedback, and finally retrieves reference papers with human reviews and scores to calibrate the target paper’s score. The authors also introduce ARD-Bench, consisting of 393 ICLR 2026 papers, to evaluate scoring accuracy, review validity, and cross-paper review overlap. The results show improved agreement with human scores and a high win rate in review-validity comparisons conducted by an LLM judge. Experiments with different backbone models and ablations provide additional evidence for the pipeline’s effectiveness.

Critical strengths
  1. The paper addresses a practically relevant problem in AI reviewing.
    It focuses on generic, inappropriate, trivial, and incorrect criticism, and separates broad identification of potential issues from verification of whether those criticisms are justified. This design addresses an important reviewing need and could reduce unsupported or insufficiently specific feedback.

  2. Score calibration has clear empirical support.
    With the same DeepSeek backbone, the full method improves Pearson correlation from 0.5512 to 0.6137 and reduces MAE from 1.7358 to 1.3335 compared with No-Calibration. These results support the usefulness of calibrating scores against historical human reviews. Results across different backbones provide further evidence for the pipeline’s applicability.

Critical weaknesses
  1. Insufficient validation on established benchmarks. The authors develop both ARD-Review and its evaluation framework, with the method and evaluation criteria closely aligned around reducing GUNI content. This makes it difficult to determine whether the reported gains represent broadly applicable improvements in review quality. Evaluation on existing, publicly available, widely used benchmarks is needed to provide independent validation and make the results more comparable with prior work.

  2. Limited methodological novelty. The method mainly combines candidate generation by a harsh critic, filtering by a merger, and retrieval-based score calibration. These components form a sensible pipeline, but the contribution primarily lies in workflow organization and prompt design. The paper should more clearly establish what methodological advance this combination introduces and why that advance is consequential beyond the reported empirical gains.

  3. Incomplete baseline comparisons. Comparisons with additional strong baselines are missing, leaving the empirical assessment incomplete. While the reported results demonstrate improvements over the included systems, they do not sufficiently establish the method’s competitiveness against a broader set of strong existing approaches. More comprehensive comparisons under consistent experimental settings would strengthen the evidence for the proposed pipeline’s advantage.

Questions for authors

See Critical Weaknesses.

1 comment

Dear Reviewer,

As the discussion period is ending soon, we wanted to kindly check whether our response has addressed your concerns. We would be grateful for any further feedback, and we are happy to clarify anything else in the remaining time.

Thank you for your time!

Best regards, The Authors

Reviewer_8gfZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents ARD-Review, an LLM reviewing pipeline in three stages. A "harsh critic" lists candidate strengths and weaknesses, a filter removes or downgrades generic, out-of-scope, nitpicky or invalid points, and a calibration step sets the score by comparing the paper with retrieved human-reviewed ICLR papers. The authors also build ARD-Bench from ICLR 2026 submissions with their frozen pre-rebuttal scores. They evaluate score agreement with humans, review validity (pairwise comparisons by an LLM judge) and how much the weaknesses overlap across papers.

Critical strengths

The retrieval-based calibration is the most useful part of the paper. Without it, scores are much more inflated and correlation with the human scores also drops, so it does more than shift the mean. The idea is also easy to reuse with any backbone. Using the frozen pre-rebuttal scores is a good choice, since post-rebuttal scores are less independent.

Critical weaknesses
  1. I don't think the comparison with human reviewers supports the claim that the system matches or beats them. As far as I can tell from Figs. 3-4, the AI score is compared with the average of all reviewers, while the human baselines predict a single other reviewer or the average of the rest, which are noisier targets. That alone could explain much of the AI's advantage. The matched hold-out used in Table 5 would be the right way to make this comparison.

  2. I am also not sure what the validity and diversity results show. The filter is designed to keep fewer, safer weaknesses and drops comments on missing related work, so an LLM judge asked which review is more valid will tend to prefer its output, and part of the gain may be built in. Without weakness counts, some measure of what the reviews miss, or a small human evaluation, I can't tell whether the reviews are better or just shorter.

Questions for authors
  1. Could you report Table 1 against the same targets as the human baselines, as in Table 5? This is what could most change my rating.

  2. Which PDF versions did the systems review? Camera-ready versions of accepted papers name the authors, which gives away the decision. And can the calibrator retrieve an earlier version of the same paper, with its reviews, from the calibration corpus (App. G)?

  3. For the Claude runs, was web search enabled, and how does a direct-review baseline with the same model compare?

Reviewer_4EcNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ARD-Reviewer, a methodology for reviewing paper based on: (i) using a model to reviewing harshly the paper by generating different Weaknesses, Strengths, and Comments; (ii) filtering the previous output by removing too generic assessments; (iii) score calibration by finding similar reviews among a corpora of previously-acquired reviews. The ARD-Reviewer is then tested against the real scores of papers, showing that its score aligns better with real reviews.

Critical strengths
  • Interesting idea of breaking down a review in statements (strengths and weaknesses), and using them to find groundtruth of similar comments to provide a score aligned with human-based review activity
  • Analysis of overlapping statements, showing that some techniques produce vague and repeated statements
  • Good coverage of backbones used as critic and filter
Critical weaknesses
  • The merger (which is another LLM) is critical for the overall work, but it is not analyzed how its performance (or under-performance) impact the whole system. The paper would need a more systematic analysis on the merger, since its output is crucial. The manuscript states that GLM is a good substitute for TADDLE, but is this enough? How much of the performances of ARD-Reviewer are based on the backbone and the merger?

  • The score calibration is not discussed: is it a RAG? How documents are picked for finding similar scores to the review to write? How much of this retrieval is responsible for the performances of the proposed model? These questions are currently lacking an answer, and they are crucial to understand the internals of the methodology. Moreover, I imagine that the quality of the scoring mechanism heavily depends on the data used inside the retrieval phase.

  • ARD-Bench is introduced but not really discussed aside from Figure 2, and it is seems a typo rather than a contribution.

  • The paper is not discussing its own limitations, if not for some indications here and there (referring to the appendix).

Questions for authors
  1. How results changes w.r.t. the retrieval component?
  2. How results changes w.r.t. a different corpus of data used as groundtruth?
  3. How results changes w.r.t. using a model different from GLM?
1 comment

Dear Reviewer,

As the discussion period is ending soon, we wanted to kindly check whether our response has addressed your concerns. We would be grateful for any further feedback, and we are happy to clarify anything else in the remaining time.

Thank you for your time!

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.