acceptodds
Under review as a conference paper at ICLR 2027

DynReviewer: A Domain-Aware AI Reviewer for Each Paper from Past Human Reviews

Abstract

AI reviewers are entering peer review, where papers from different domains go to different experts, who judge them by the standards of their domain. Yet most AI reviewers apply one fixed configuration to every paper. They are also evaluated mainly on ICLR, where a single average hides performance differences across domains. We propose DynReviewer, which reads the paper under review first, then dynamically configures a dedicated AI reviewer for it. DynReviewer rewrites general checks distilled from past human reviews, along with reviews of related papers, into a paper-specific rubric, then searches the paper for evidence on each item. The whole process requires no model training and no human-written rubrics. We also build ReviewAtlas-40K+, which collects 45,551 papers and their official reviews from six venues. Reviews up to 2024 serve as historical experience, and papers from 2025 form the benchmark ReviewAtlas-Bench, covering four venues and four ICLR research domains. Across three LLM backbones, DynReviewer outperforms all compared methods in every venue and domain at covering human reviewers' concerns with criticisms grounded in the paper. It leads the strongest same-backbone baseline by 5.6 to 19.9 percentage points. Its reviews also have the highest content quality in most settings, and its ICLR acceptance decisions agree best with actual outcomes. Analyses show that general checks distilled from human reviews clearly outperform LLM-written ones, and that rewriting experience into a paper-specific rubric works much better than using it directly. Past human reviews thus let an AI reviewer adapt to each paper, as human peer review does.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
AI peer review, automated paper reviewing, large language models, paper-specific rubrics, retrieval-augmented generation, LLM agents, benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1GkaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DynReviewer, a training-free reviewing framework that adapts historical human-review experience to each target paper. It combines general checks distilled from past reviews with concerns retrieved from related papers, converts both into paper-specific rubrics, and verifies potential criticisms against the manuscript. The authors also introduce ReviewAtlas-40K+, containing 45,551 papers with associated reviews, and an 800-paper benchmark spanning four venues. Experiments across three backbones report improvements in concern coverage, judge-rated review quality in most settings, and agreement with ICLR acceptance decisions.

Critical strengths
  1. The framework provides a practical way to turn historical reviews into applicable checks rather than directly copying criticisms from other papers. Explicit applicability conditions and evidence requirements make the generated findings easier to inspect.

  2. The evaluation covers four venues and separately reports four ICLR domains across three backbones. Component ablations, source-exclusion experiments, and finding-budget analyses provide useful evidence about the behavior of the complete system.

Critical weaknesses
  1. The methodological novelty appears limited. Adaptive review questions and historical-memory-based reviewing have already been explored by methods such as TreeReview and Agent Reviewers. The training-free, dual-path design is useful, but the contribution currently appears primarily to be an integration and refinement of existing ideas. The paper does not sufficiently establish the distinctive methodological advance over closely related adaptive or rubric-based reviewing systems. A clearer comparison with these approaches would help substantiate the added value of the proposed design.

  2. The analyses provide limited insight into reviewing behavior and the value of historical experience. Given the substantial paper–review corpus, I would expect a deeper investigation of how AI and human reviewers differ in the issues they identify, which concern types AI reviewers systematically miss or overproduce, and how human reviewing priorities change over time within comparable venues and domains. It would also be valuable to identify which concern categories and severity levels are most strongly associated with human ratings and acceptance outcomes, while distinguishing association from causation. Such analyses would clarify whether DynReviewer improves the identification of consequential issues and explain why historical experience helps.

  3. The claimed improvement in review quality lacks independent human evaluation. Concern matching, evidential support, and content quality are predominantly assessed by a single LLM judge, while Appendix A acknowledges that the independent expert audit remains incomplete. Agreement with official acceptance decisions is encouraging, yet does not establish that individual criticisms are correct, relevant to the paper’s claims, or useful to authors. A blinded evaluation by domain experts on a representative sample is needed, comparing factual correctness, with inter-annotator agreement reported.

Questions for authors

See Critical Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 1Gka (1/5): Novelty and Differences from TreeReview and Agent Reviewers (W1)

Thank you for recognizing that DynReviewer turns past reviews into checks with applicability conditions and evidence requirements, and for valuing our evaluations and ablations across venues, domains and backbones. To address your three points, we added direct comparisons with TreeReview and Agent Reviewers, an analysis of review behavior, and an independent blind expert evaluation.

W1: On methodological novelty and the differences from TreeReview and Agent Reviewers

(1) Our core contribution is to configure a dedicated AI reviewer for a paper after reading it. Human peer review assigns each paper to experts in its area, because different papers call for different criteria. Theory papers require checking assumptions and proofs, empirical papers require checking baselines and ablations, and systems papers require checking scalability and overhead. Most existing AI reviewers fix their configuration before reading the paper. Whether they are trained reviewer models or pre-written prompts, review forms or roles, they apply the same criteria to every paper. To our knowledge, no prior work has made it a core problem to automatically build a review rubric for a paper from past human reviews after reading it. Table 9 in the paper compares the method designs of existing AI reviewers. Among them, DynReviewer is the only method that is training-free, adapts to each paper, and derives its rubric from human reviews.

(2) Neither TreeReview nor Agent Reviewers has a step that builds a review rubric for each paper from past human reviews. Section 2 (Related Work) of the paper already discusses both methods, and Table 9 also compares the method design of Agent Reviewers.

  • TreeReview [1] has an LLM pose review questions dynamically based on the manuscript. The questions come from the model's own knowledge and do not use past human reviews. It therefore has no access to what reviewers in a given domain actually care about.
  • Agent Reviewers [2] retrieves past reviews as memory, which predefined reviewer roles use only as reference. The roles and the pipeline are the same for all papers. Past reviews are supplied in their original text, and no paper-specific rubric is built for the manuscript. Its evaluation also does not report results separately by research area within a conference (Table 9 in the paper). So its evaluation cannot test whether it reviews well in every domain.

As you noted among the strengths, DynReviewer turns past reviews into usable checks rather than copying criticisms from other papers. This conversion is exactly the step both methods lack. Two ablations in Table 3 of the paper test the effect of each of these two differences (200 ICLR papers, DeepSeek v4 Flash):

  • Checks come from human reviews, not from the model itself: In TreeReview, the model poses the review questions itself. Replacing DynReviewer's 18 checks distilled from human reviews with 18 LLM-written checks (w/ Generic Rubrics) lowers Q from 42.66% to 36.87%, Overall from 6.98 to 4.80, and decision accuracy from 74.5% to 55.0%. Checks written by the model cannot replace human review experience.
  • Building a paper-specific rubric, not using past reviews as reference: Agent Reviewers gives past reviews directly to the model as reference. When DynReviewer drops the rewriting step and gives the same past experience directly to the model (w/o Compile), Q drops from 42.66% to 17.71%, Overall drops to 5.64, and accuracy drops to 55.0%. Rewriting past experience into questions and evidence requirements for this manuscript is the key step that makes DynReviewer work.

References

[1] TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review. https://aclanthology.org/2025.emnlp-main.790/

[2] Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review. https://proceedings.mlr.press/v267/lu25p.html

AuthorsOpenReview ↗

Response to Reviewer 1Gka (2/5): Direct Comparison (W1, cont.) and AI vs. Human Concern Types (W2)

W1 (continued)

(3) In direct comparison, DynReviewer clearly outperforms both methods (Appendix D.11). We ran all three systems with GLM-5.3-Flash on 100 ICLR 2025 papers, 25 from each of the four domains. For TreeReview, we use the authors' released code. Agent Reviewers has no public code, and its core memory module is not released either. We reproduced it following the pipeline and prompts described in its paper.

Review frameworkUnion-W (%)SFP (%)Q (%)Overall
TreeReview19.2390.2530.447.13
Agent Reviewers18.1466.9027.025.13
DynReviewer31.0998.7545.797.95

Union-W is concern coverage, SFP is finding support (the share of criticisms supported by the manuscript), and Q is the per-paper harmonic mean of the two. Overall is review content quality (−2 to 8).

DynReviewer's Q is 15.35 percentage points (pp) higher than TreeReview's and 18.77 pp higher than Agent Reviewers'. It also uses 46.0% fewer tokens than TreeReview. The gap comes mainly from concern coverage. DynReviewer recovers over 60% more human concerns than either method, and it also has the highest finding support.

W2: On analyzing review behavior and the value of past experience

Thank you for these questions. They bear on whether AI reviewers can truly find important issues. Some of them are hard to answer rigorously. When an AI criticism does not appear in the human reviews, the AI may be wrong, or the human reviewers may have missed the issue. No current method can reliably tell the two apart across a large number of manuscripts. To establish that a type of criticism leads to lower scores, one would need interventional comparisons on the same paper. Public review records can only show associations. Within these limits, we give the following analyses in the order of your questions. DeepSeek v4 Flash categorized the criticisms without knowing whether each came from a human or an AI. The tables list six main categories.

(1) How AI and human reviewers differ in the issues they identify, and which concern types AI reviewers systematically miss or overproduce. We counted the category distribution of human criticisms and of criticisms from three AI systems on 100 ICLR 2025 papers (all systems use DeepSeek v4 Flash).

Issue categoryHumanDynReviewerReviewGrounder [3]AI Scientist-v2 [4]
Experimental design16.5128.3923.8611.88
Method/theory29.6125.4632.0926.24
Results/statistics6.3216.6711.8831.78
Baseline comparison8.1910.4710.652.88
Novelty8.034.191.441.37
Writing/presentation22.904.525.9715.05

Units are %. Shares are computed per paper and then averaged. Only the six main categories are listed.

All three AI systems systematically raise fewer writing/presentation and novelty issues. DynReviewer and ReviewGrounder raise more experimental design issues, and AI Scientist-v2 raises more results/statistics issues. On novelty, DynReviewer's share is about 3 times that of the other two AI systems, but still only half the human share. Under the ScholarPeer protocols, its novelty and significance judgments also beat those of the two online review systems (see our reply to 1Bqq). Novelty judgment remains an area where AI reviewers need to improve.

References

[3] ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents. https://arxiv.org/abs/2604.14261

[4] The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. https://arxiv.org/abs/2504.08066

AuthorsOpenReview ↗

Response to Reviewer 1Gka (3/5): Trends over Time, Links to Ratings, and Value of Past Experience (W2, cont.)

W2 (continued)

(2) How human reviewing priorities change over time within comparable venues and domains. We applied the same method to the public ICLR 2025 reviews and the ICLR 2027 first-round reviews. For both years, we sampled 25 papers from each of the same four domains (in this analysis, DynReviewer uses GLM-5.3-Flash).

Review sourceShare of experimental design + results/statistics: ICLR 2025ICLR 2027
Human reviews22.83%38.78%
DynReviewer47.72%48.89%

Between the two years, DynReviewer's category distribution stays roughly the same. Human reviews give markedly more attention to experimental design and results/statistics in 2027, moving closer to the focus of AI reviews. Possible causes of this shift in human focus include the submission mix, reviewer guidelines, or reviewers' greater use of AI tools. We cannot yet tell these apart.

(3) Which concern categories and severity levels in AI reviews are most strongly associated with human ratings and acceptance outcomes. We grouped papers by DynReviewer's criticisms and matched papers within the same domain by abstract similarity. Matching uses neither human scores nor decisions. We then compared the mean human scores and acceptance rates of the two groups. The two categories below show the clearest associations. For both, the 95% confidence interval of the difference excludes 0:

Matched groups A / BMean human score A / BDifference A−B [95% CI]Acceptance rate A / B
DynReviewer flagged a severe issue / did not4.78 / 5.44−0.66 [−1.25, −0.04]21.4% / 50.0%
More / fewer experimental-design criticisms4.92 / 5.71−0.80 [−1.34, −0.27]37.5% / 43.8%

Papers in which DynReviewer flagged a severe issue receive lower human scores and are accepted much less often. These are associations after matching and do not establish causation. Matching controls only for domain and topical similarity. Other aspects of paper quality may affect both the AI criticisms and the human scores.

(4) Whether DynReviewer is better at finding important issues. In Table 6 of the paper, the judge selects only the K most important criticisms from each review before computing Q. At K = 3, DynReviewer reaches 28.50%, ReviewGrounder 23.90% and AI Scientist-v2 20.19%. Even when only the few most important criticisms count, DynReviewer recovers more human concerns. Combined with (3), the severe issues it flags are consistent with human scores and decisions.

(5) Why past experience helps. In Table 3 of the paper, replacing the 18 checks distilled from human reviews with LLM-written checks (w/ Generic Rubrics) drops Q from 42.66% to 36.87%. Past experience also changes which issues the AI focuses on. Compared with the version that removes all past experience (w/o Experience), the full system's criticisms have 9.38 pp and 6.41 pp higher shares of experimental design and baseline comparison, respectively (95% CI [6.05, 12.54] and [4.34, 8.31]). Appendix F.1 gives examples from the four domains. Each criticism can be traced back to the general check or related-paper review it came from.

Finally, your question about how human review focus changes over time points us to a direction that deserves the community's attention. As AI tools are used more in writing and reviewing, the focus of human reviews may gradually drift toward that of AI. The cross-year results in point (2) already show signs of this. If the trend continues, two questions deserve further study: how much independent expert judgment future human review experience can still offer AI reviewers, and how methods that learn from past reviews should select that experience.

Reviewer_1BqqOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Authors propose DynReviewer, which uses human reviews from relevant papers and a general experience library to generate AI reviews. They also build ReviewAtlas-40K+, containing 46K papers and their official reviews from six venues, and propose ReviewAtlas-Bench that covers four venues and four iclr research domains for evaluation. Across three llm backbones, the proposed method outperforms baselines at covering human-written reviews and estimating iclr acceptance decisions.

Critical strengths
  • The idea is interesting -- building rubric sets by (1) referring to a general experience library and (2) human reviews of relevant papers might be an interesting direction for AI reviewing, and it might resolve the problem that the current AI reviewers have similar tastes which are sometimes misaligned with human's.
  • The analysis and ablation study give valuable lessons for readers.
Critical weaknesses
  • Unjustified evaluation criteria.
    • line 210: Justify the choice of evaluation methods. Why is evidence-grounded recovery important? Why overall scores consider only correctness, depth, actionability, etc.? Why not other criteria? How are those decided?
    • I guess authors measure coverage of human-written concerns as one of evaluation methods. However, ML community is suffering from low-quality reviews, so I'm quite pessimistic about using unfiltered human reviews from AI conference venues as a reference.
  • Weak Baselines. There are lots of baseline agentic reviewers authors should include: ScholarPeer (Goyal et al., 2026), Stanford Agent Reviewer, ReviewerToo (Sahu et al., 2025), CSPaper Review (Cao et al., 2025), etc. I am quite sure that current baselines are too weak.
Questions for authors
  • line 176: Are you sure that these papers are all first drafts? I tried to do similar thing before, but recent venues such as iclr 2024 and 2025 do not offer first drafts. If not, since there is inconsistency in target papers and reviews, the value of the dataset might be much lowered.
  • I'm not convinced that Table 1 clearly motivates an evaluation with cross-venue and domain-level reporting, etc. (line 154-156) -- also, DeepReviewer and CycleReviewer are preliminary reviewer models, so I guess using frontier models or models that perform better than these two would be necessary.
8 comments
AuthorsOpenReview ↗

Response to Reviewer 1Bqq (1/5): Basis for the Evaluation Criteria (W1)

Thank you for recognizing our idea of building review rubrics from a general experience library combined with human reviews of related papers, and the value of our analyses and ablations. To address your concerns, we added comparisons with six stronger review systems, a comparison under ScholarPeer's evaluation protocols, an independent blind expert evaluation, and validation on ICLR 2027 first-submission versions.

W1: On the basis for the evaluation criteria

(1) Why we evaluate grounded concern recovery (Q). To be useful to authors, an AI review must do two things at once. It must raise issues that domain experts would also care about, and its criticisms must be grounded in the manuscript. We measure these with two metrics. Concern coverage (Union-W) measures how many of the issues raised in human reviews the AI recovers. It follows the issue-detection evaluation of Agent Reviewers [1]. Finding support (SFP) measures how many of the AI's criticisms are supported by the main text or appendix. It draws on the evidence-support evaluation of ReviewGrounder [2]. Each metric alone has a loophole. Listing many generic criticisms raises coverage but lowers support. Raising only a few safe criticisms raises support but leaves coverage very low. Q is the per-paper harmonic mean of the two, so a low value on either one pulls Q down.

(2) Why Overall uses these dimensions. DeepReview [3] and ScholarPeer [4] both evaluate reviews on technical accuracy, depth of analysis and constructiveness. We turn these three aspects into four items that authors can check against the manuscript. Each item is scored 0–2: whether a criticism is grounded (evidence correctness), whether it explains the impact of the issue (diagnostic depth), whether authors can revise based on it (actionability), and whether it targets this paper (paper specificity). A review that contains factual errors loses another 0–2 points. We left novelty and significance out of Overall because these two dimensions are more subjective. Kim et al. [5] asked 45 scientists to evaluate reviews of Nature-family papers. Expert agreement was 85.8% on "is the criticism correct" and 88.0% on "is the evidence sufficient", but only 59.9% on "is the criticism important". We evaluate novelty and significance judgments separately in (4) below, using ScholarPeer's protocols.

(3) On the uneven quality of human reviews. We agree that review quality at ML conferences varies. We therefore extract only the substantive issues explicitly raised in the original reviews. We remove summaries, praise and vague comments, merge duplicates, and keep the original sentences and their qualifiers (Appendix B.1, C.4). Human reviews are used only to compute coverage. Whether a criticism holds (SFP) is judged only against the manuscript, without reference to human reviews. In addition, we validate with ScholarPeer's H-Max protocol in (4) below. On each dimension, it takes only the strongest comment among all human reviews as the reference, so low-quality human comments do not affect the scores.

References

[1] Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review. https://proceedings.mlr.press/v267/lu25p.html

[2] ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents. https://arxiv.org/abs/2604.14261

[3] DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process. https://aclanthology.org/2025.acl-long.1420/

[4] ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review. https://arxiv.org/abs/2601.22638

[5] On the limits and opportunities of AI reviewers: Reviewing the reviews of Nature-family papers with 45 expert scientists. https://arxiv.org/abs/2605.20668

AuthorsOpenReview ↗

Response to Reviewer 1Bqq (2/5): Results under ScholarPeer's SxS and H-Max Protocols (W1, cont.)

W1 (continued)

(4) DynReviewer still leads under the evaluation protocols that the Google team proposed in ScholarPeer. To confirm that our conclusions do not depend on our metrics or on the human concerns we extract, we adopt both evaluation modes of ScholarPeer [4]:

  • SxS: No human reviews are given as input. Two anonymized reviews of the same paper are compared side by side. The judge decides which one is better on technical correctness, constructiveness, depth of analysis, novelty and significance judgment, and overall quality.
  • H-Max: On each dimension, only the strongest relevant comment among all human reviews of the paper is chosen as a 5-point reference. Each candidate review is scored independently from 1 to 10. Low-quality human comments never become the reference.

We compare with the two online review systems you mentioned, Stanford Agentic Reviewer [6] and CSPaper Review [7], which serve real users. All three systems reviewed the same 100 ICLR 2025 papers (25 from each of the four domains), and Gemini 3.8 Flash scored the reviews under these protocols.

SxS: DynReviewer's pairwise win rate on the same papers (ties count as half)

Dimensionvs. Stanfordvs. CSPaper
Technical correctness85.5%61.5%
Constructiveness79.0%64.0%
Depth of analysis79.5%63.5%
Novelty and significance judgment74.5%79.0%
Overall quality78.0% (95% CI [70%, 86%])62.0% (95% CI [52%, 72%])

H-Max: Scores relative to the strongest human comment (1–10; 5 = strongest human comment)

SystemTechnical correctnessConstructivenessDepth of analysisNovelty judgmentOverall quality
DynReviewer7.887.817.966.907.81
Stanford6.427.007.275.596.54
CSPaper7.157.407.784.786.77

Under both protocols and on every dimension, DynReviewer outperforms both online systems. This includes novelty and significance judgment, which Overall does not cover. In addition, two experts who could not see the official reviews blindly compared 100 review pairs. Their preference rates for DynReviewer were 86% and 65% (Appendix D.12). None of these evaluations relies on concern coverage, and they give the same system ranking as Q.

References

[4] ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review. https://arxiv.org/abs/2601.22638

[6] Stanford Agentic Reviewer (official technical overview). https://paperreview.ai/tech-overview

[7] CSPaper Review: Fast, Rubric-Faithful Conference Feedback. https://aclanthology.org/2025.inlg-demos.2/

AuthorsOpenReview ↗

Response to Reviewer 1Bqq (3/5): Comparison with Stronger Review Systems (W2)

W2: On stronger review baselines

(1) The paper already includes the main open-source baselines from academia that can be reproduced under the same backbone. These are the released dedicated reviewer models DeepReviewer 7B/14B [3] and CycleReviewer 8B/70B [8]; the AI Scientist-v2 review module [9] and ReviewGrounder [2] (a tool-augmented review agent released in 2026), both run on the same three backbones; and zero-shot and few-shot prompting methods.

(2) The four systems you mentioned are all closed-source systems or commercial services. Unlike the open-source baselines, they cannot be run uniformly on our benchmark with the same backbone. ScholarPeer has no public code, and its hosted service is not accessible. Stanford Agentic Reviewer offers only a web service. CSPaper Review is a commercial service that charges per paper. ReviewerToo has not released its system; only the reviews it generated for ICLR 2025 papers are public. Thank you for these suggestions. The revised Section 2 (Related Work) now describes the design of these four systems. During the rebuttal period, we completed the comparison through whatever access each system offers (Appendix D.11).

SystemAccessVersion used here
ScholarPeer [4]Closed-source; hosted service not accessibleCommunity open-source implementation OpenScholarPeer [10]
Stanford Agentic Reviewer [6]Closed-source; web service onlyOfficial web service (free during our experiments)
CSPaper Review [7]Closed-source; paid commercial serviceOfficial paid service, €1 per paper
ReviewerToo [11]Closed-source; only its reviews for ICLR 2025 papers are releasedFinal meta-reviews from gpt-oss-120b (high), released by the authors

Setup: We took 25 papers from each of the four ICLR 2025 domains, 100 in total. DynReviewer and OpenScholarPeer use GLM-5.3-Flash. For CSPaper, we chose GLM-5.3-Flash from the model options its service offers. Stanford uses the model built into its service. For ReviewerToo, we use the final meta-reviews released by the authors. One of them is empty, so we compare on the remaining 99 papers. On the same papers, we also added TreeReview [12] and Agent Reviewers [1], which Reviewer 1Gka mentioned. Gemini 3.8 Flash evaluates all reviews under the same protocol as in the paper.

References

[1] Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review. https://proceedings.mlr.press/v267/lu25p.html

[2] ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents. https://arxiv.org/abs/2604.14261

[3] DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process. https://aclanthology.org/2025.acl-long.1420/

[4] ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review. https://arxiv.org/abs/2601.22638

[6] Stanford Agentic Reviewer (official technical overview). https://paperreview.ai/tech-overview

[7] CSPaper Review: Fast, Rubric-Faithful Conference Feedback. https://aclanthology.org/2025.inlg-demos.2/

[8] CycleResearcher: Improving Automated Research via Automated Review. https://arxiv.org/abs/2411.00816

[9] The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. https://arxiv.org/abs/2504.08066

[10] OpenScholarPeer (community open-source implementation of ScholarPeer). https://github.com/amirkiarafiei/open-scholar-peer

[11] ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review. https://arxiv.org/abs/2510.08867; released reviews: https://huggingface.co/datasets/demfier/reviewertoo-iclr2025-reviews

[12] TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review. https://aclanthology.org/2025.emnlp-main.790/

Reviewer_WGpWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Practical, training-free design. All adaptation happens in context: past reviews become checks and paper-specific rubrics, and model weights never change. The method therefore transfers across backbones without retraining, and its experience pool can grow as new review cycles become public. I find this more practical than fine-tuned reviewers such as DeepReviewer and CycleReviewer, which DynReviewer also outperforms. Consistent gains across backbones. The ranking holds on all three backbones and in every venue and ICLR domain. The improvement over the strongest same-backbone baseline ranges from 5.6 to 19.9 Q points. This consistency suggests the gain comes from the pipeline rather than from one particular model.' Findings are tied to evidence. Each finding must cite a quotation and location in the paper, and support is measured separately (SFP, unsupported-finding rate) rather than through embedding similarity.

Critical strengths
  1. Practical, training-free design. All adaptation happens in context: past reviews become checks and paper-specific rubrics, and model weights never change. The method therefore transfers across backbones without retraining.

  2. Consistent gains across backbones. The ranking holds on all three backbones and in every venue and ICLR domain.

  3. Thorough ablations and an honest limitations section. The ablations cover generic versus review-derived checks, removing the rewriting step, source and domain exclusion, temporal cutoff, library size, and finding budgets.

Critical weaknesses

All 18 checks are domain-general, and excluding same-domain sources costs only 0.11–0.69 Q points in two of the four domains (Table 5b). The evidence better supports adapting the reviewer to each paper than adapting it to a domain.

how do we validate the judge by the gemini-3-8-flash? it might be worth reporting the human judgement or any type of calibrations. including mind the blind spots show that they could be done cheaply.

Uncertainty and cost are not reported. There are no confidence intervals or repeated runs. Decision accuracy is 74.5% on the main ICLR cohort but 62.0% on the D.10 control cohort under the same configuration.

Questions for authors
  1. The paper does not say which revision of each 2025 paper is used. Accepted papers’ default OpenReview PDF is usually the camera-ready version. which version did we use?
  2. how about resubmissions?
  3. how do we validate the judge by the gemini-3-8-flash? it might be worth reporting the human judgement or any type of calibrations. including midn the blind spots show that they could be done cheaply
5 comments
AuthorsOpenReview ↗

Response to Reviewer WGpW (1/4): The 18 General Checks and Domain Adaptation (W1)

Thank you for recognizing DynReviewer's training-free design that transfers across backbones, its consistent gains on every backbone, and the completeness of our ablations. We respond point by point below.

W1: On the 18 general checks and domain adaptation

(1) We agree with you: the core of DynReviewer is configuring a reviewer for each paper. We want the AI reviewer to act as a domain expert who can review the paper at hand well. As the introduction notes, theory papers require checking assumptions and proofs, empirical papers require checking baselines and ablations, and systems papers require checking scalability and overhead. The paper itself determines which domain knowledge is needed. So the system configures the reviewer after reading the manuscript and needs no predefined domain label. In our method, configuring a reviewer for each paper and giving it review knowledge of that paper's domain are the same process.

(2) The 18 checks cover only general reviewing ability. Domain-specific checks come from a separate path. As shown in Section 4 and the middle of Figure 3 (framework overview), DynReviewer draws past experience from two paths:

  • Experience-library path (B, "Experience-Library Retrieval" in Figure 3): We distill 18 general checks (Appendix E.1) from human reviews of 500 past papers, for example whether comparisons are fair and whether the experiments support the claims. Papers in every domain need these checks. They are cross-domain by design.
  • Related-paper path (A, "Related-Paper Retrieval" in Figure 3): For the current manuscript, the system retrieves up to 3 of the most relevant past papers. From their human reviews, it extracts the issues that reviewers in that domain care about. It then rewrites them into questions and evidence requirements to check for this manuscript. Domain-specific checks come mainly from this path.

Table 3 in the paper shows the role of the related-paper path. Removing it and keeping only the 18 general checks (w/o Related) drops Q from 42.66% to 24.12%. General reviewing ability alone is not enough. The reviewer also needs domain-specific checks drawn from human reviews of related papers.

(3) Table 5b removes same-domain experience from both paths at once, so its results cannot be attributed to the 18 general checks. For each target paper, this experiment removes same-domain historical sources from both paths together. The related-paper path can retrieve only papers from other domains. In the experience-library path, checks supported by same-domain sources are also deleted, leaving only 9–15 checks per domain (Appendix D.9, Table 31). The experiment therefore measures how well other-domain experience can compensate when same-domain experience is entirely absent.

DomainGeneral checks keptΔ Concern coverageΔ Finding supportΔ Q
General ML9/18−0.40+0.02−0.69
FM/NLP11/18−0.05+0.76−0.11
Graphs13/18−2.88−1.05−3.48
Trustworthy AI15/18−3.38+1.31−3.70

Units are percentage points (pp). Data are from Table 5b in the paper and Appendix Table 31.

These results show three things:

  • The drop is unrelated to the general checks. General ML loses 9 general checks, yet its Q drops by only 0.69. Trustworthy AI loses only 3, yet its Q drops by 3.70. If the drop came from the general checks, the pattern would be reversed.
  • The drop comes from the related-paper path losing same-domain papers. In Graphs and Trustworthy AI, almost all of the drop comes from concern coverage, while finding support stays roughly unchanged. Reviews of same-domain papers supply exactly the issues that reviewers in that domain care about.
  • The drops differ because the domains differ. In General ML and FM/NLP, review concerns are mostly methodological, such as baselines, ablations and statistics. Reviews in other domains often raise them too, so cross-domain experience can substitute. Concerns in Graphs and Trustworthy AI rely more on in-domain knowledge, and other-domain experience cannot replace it. Which papers need same-domain experience cannot be judged in advance from domain labels. This is exactly why DynReviewer retrieves related papers only after reading the paper.
AuthorsOpenReview ↗

Response to Reviewer WGpW (2/4): Validating the Gemini 3.8 Flash Judge with Expert Ratings (W2, Q3)

W2, Q3: On validating the Gemini 3.8 Flash judge

(1) Using a single fixed model to evaluate reviews is standard practice in prior work. DeepReview [1] uses Gemini-2.0-Flash-Thinking to compare reviews of the same paper. ScholarPeer [2] uses Claude Sonnet 4.5 as its main judge. We use Gemini 3.8 Flash for all systems, with temperature 0 and the same prompts.

(2) We added a human evaluation (Appendix D.12). Following the idea of Mind the Blind Spots [3], which you mentioned, and the practice of ScholarPeer [2] and ReviewArena [4], we had experts and Gemini 3.8 Flash score the same reviews and then compared their scores.

  • Sample: We took 25 papers from each of the four ICLR 2025 domains, 100 papers in total. Each paper has two reviews: one from DynReviewer and one from a comparison system (ReviewGrounder [5] for 36 papers; Stanford Agentic Reviewer [12] and CSPaper Review [13] for 32 each). This gives 200 reviews. System names were hidden, and the order was randomized.
  • Experts: Two researchers who are not authors of this paper. They work on machine learning, AI for Research and agents. They scored independently of each other.
  • Scoring: The experts checked each review's key criticisms against the main text and appendix of the paper. They recorded the locations that support or refute each criticism. They then scored on the same scale as Gemini 3.8 Flash: evidence correctness, diagnostic depth, actionability and paper specificity at 0–2 each, plus a Pitfall penalty of 0–2 points for factual errors. Overall quality (Overall) is the sum of the five items and ranges from −2 to 8.

(3) The results show that Gemini 3.8 Flash ranks review quality largely in line with the experts. On the 100 comparison-system reviews, its Spearman correlations with the two experts' Overall scores are 0.636 and 0.763. The mean absolute errors are 0.49 and 0.39 points on the −2 to 8 scale. Reviews that the experts rate as better usually also get higher scores from it. We use Gemini 3.8 Flash only to compare the relative quality of reviews from different systems, and this result supports that use. Per-dimension results are below (revised Table 39):

DimensionGemini–Expert 1 ρMAEGemini–Expert 2 ρMAE
Evidence correctness (0–2)0.5910.260.6830.18
Diagnostic depth (0–2)0.7040.010.5710.02
Actionability (0–2)0.7580.070.7580.07
Paper specificity (0–2)0.7040.010.5710.02
Pitfall penalty (−2 to 0)0.6810.180.7660.12
Overall (−2 to 8)0.6360.490.7630.39

The diagnostic depth and paper specificity rows have identical values because almost all reviews got full marks on these two items (out of 100 reviews: 99 for Gemini, 98 for Expert 1, 97 for Expert 2).

In addition, both experts gave DynReviewer higher evidence-correctness scores. When choosing between the two reviews of each paper, their preference rates for DynReviewer were 86% and 65% (ties count as half; Appendix D.12, Table 38).

References

[1] DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process. https://aclanthology.org/2025.acl-long.1420/

[2] ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review. https://arxiv.org/abs/2601.22638

[3] Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews. https://aclanthology.org/2025.emnlp-main.1805/

[4] ReviewArena: A Large-Scale Cross-Conference Dataset and Benchmark for LLM Peer Review. https://openreview.net/forum?id=yugEO52gkR

[5] ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents. https://arxiv.org/abs/2604.14261

[12] Stanford Agentic Reviewer (official technical overview). https://paperreview.ai/tech-overview

[13] CSPaper Review: Fast, Rubric-Faithful Conference Feedback. https://aclanthology.org/2025.inlg-demos.2/

AuthorsOpenReview ↗

Response to Reviewer WGpW (3/4): Cost, Repeated Runs and Decision Accuracy (W3)

W3: On cost, repeated runs and the gap in decision accuracy

(1) Cost per review (Appendix C.5, Table 15). We recorded each system's average cost to produce one review. With GLM-5.3-Flash, DynReviewer uses 310,992 tokens per paper on average.

SystemCost per review
DynReviewer310,992 tokens
ReviewGrounder [5]257,124 tokens
OpenScholarPeer [6]475,249 tokens
TreeReview [7]576,362 tokens
Agent Reviewers [14]58,133 tokens
Stanford Agentic Reviewer [12]Free (5.17 minutes on average)
CSPaper Review [13]€1 per paper

All token counts are measured with GLM-5.3-Flash. Agent Reviewers also needs a one-time topic index over 28,052 past papers.

(2) Repeated runs and confidence intervals (Appendix D.13, Table 40). Because the GLM-5.3-Flash API was under heavy load during the rebuttal period, these experiments use DeepSeek v4 Flash (the backbone of the ablations in the paper). On the same 50 ICLR papers, we independently generated and judged reviews three times. Mean Q was 41.56%, 41.66% and 42.27%, a spread of at most 0.71 pp. We first averaged each paper's three results. We then resampled papers 20,000 times to obtain a 95% confidence interval. We also report the confidence interval for the main experiment in the paper (200 papers), with resampling stratified by domain and decision outcome.

RunPapersMean Q (%)95% CI (%)
Run 15041.56—
Run 25041.66—
Run 35042.27—
Mean of three runs5041.83[39.24, 44.37]
Main experiment in the paper20042.66[40.68, 44.60]

DynReviewer's results are stable across repeated runs. In Table 2 of the paper, the strongest ICLR baseline under the same backbone is ReviewGrounder, with a Q of 24.58%. That is 18.08 pp below DynReviewer, far larger than the run-to-run variation.

(3) The 74.5% and 62.0% figures come from two experiments with different settings. 74.5% is the decision accuracy of the main experiment on all 200 ICLR papers (Figure 4, Table 24). 62.0% comes from the retrieval-year sensitivity experiment in Appendix D.10 (Table 34). D.10 regenerates reviews separately under three retrieval cutoff years: 2022, 2023 and 2024. To control cost, it uses only 50 of the 200 papers and reruns the full pipeline on them. On the same 50 papers, the main experiment gets 34 right (68.0%). The independent rerun in D.10 with the 2024 cutoff gets 31 right (62.0%), only 3 papers fewer. Most of the gap between the two figures comes from the different paper samples: these 50 papers are harder to judge than the 200 as a whole.

References

[5] ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents. https://arxiv.org/abs/2604.14261

[6] OpenScholarPeer (community open-source implementation of ScholarPeer). https://github.com/amirkiarafiei/open-scholar-peer

[7] TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review. https://aclanthology.org/2025.emnlp-main.790/

[12] Stanford Agentic Reviewer (official technical overview). https://paperreview.ai/tech-overview

[13] CSPaper Review: Fast, Rubric-Faithful Conference Feedback. https://aclanthology.org/2025.inlg-demos.2/

[14] Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review. https://proceedings.mlr.press/v267/lu25p.html

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Contributions, Clarifications and Additional Validation

We thank the AC and the three reviewers for their careful reading.

Contributions recognized by the reviewers

  • Method. The design is training-free: all adaptation happens in context, so it transfers across backbones without retraining (WGpW). It turns past reviews into checks with applicability conditions and evidence requirements, rather than copying criticisms from other papers (1Gka). Building rubrics from a general experience library and human reviews of related papers is an interesting direction that may ease the mismatch between AI reviewers and human concerns (1Bqq).
  • Evidence. The evaluation covers three backbones, four venues and four ICLR domains (1Gka), the gains are consistent across all of them (WGpW), and the ablations and analyses offer useful lessons (WGpW, 1Bqq, 1Gka).

Clarifications

  1. Domain-specific knowledge comes from the related-paper path; the 18 general checks cover general reviewing ability (WGpW). Removing the related-paper path drops Q from 42.66% to 24.12% (Table 3). Table 5b removes same-domain sources from both paths at once, so its drop cannot be attributed to the 18 checks.
  2. What separates DynReviewer from TreeReview and Agent Reviewers is building a paper-specific rubric from past human reviews (1Gka). Two ablations in Table 3 test exactly these differences. Replacing the checks with LLM-written ones (w/ Generic Rubrics) and skipping the rewriting step (w/o Compile) both clearly hurt.
  3. The evaluation metrics follow prior work (1Bqq). Concern coverage follows Agent Reviewers, finding support follows ReviewGrounder, and the Overall dimensions follow DeepReview and ScholarPeer.
  4. Manuscript versions do not favor any system (WGpW, 1Bqq). We use the earliest PDF downloadable at collection time, as existing public datasets do. All systems read the same manuscripts and are compared against the same human concerns.
  5. The 74.5% and 62.0% decision accuracies come from different experiments (WGpW). The latter is an independent rerun on a 50-paper subset (Appendix D.10), and the gap comes mainly from the different paper samples.

Additional validation

These results support the paper's conclusions. The method and the main results are unchanged.

  • Blind expert evaluation (Appendix D.12). Gemini 3.8 Flash ranks review quality largely in line with two independent experts. Both experts rated DynReviewer's criticisms as more accurate and preferred its reviews.
  • More review systems (Appendix D.11). Four of the systems the reviewers mentioned (ScholarPeer, Stanford Agentic Reviewer, CSPaper Review and ReviewerToo) are closed-source or paid services, so they cannot be run under the same backbone as our open-source baselines. We evaluated them through the access each one offers, together with TreeReview and Agent Reviewers, on the same 100 ICLR 2025 papers. DynReviewer is the highest on all four metrics. It also leads Stanford Agentic Reviewer and CSPaper Review under ScholarPeer's evaluation protocols.
  • First-submission versions. On ICLR 2027 submissions and their first-round official reviews, DynReviewer still leads ReviewGrounder.
  • Repeated runs (Appendix D.13). DynReviewer's results are stable across three independent runs, and its lead over the baselines is far larger than the run-to-run variation.

Scope of the revision

Changes are marked in blue. Most are in the appendix: A (limitations), B.1 (manuscript versions and the resubmission filter), C.5 (cost), D.10 (cohort description), D.11 (more review systems), D.12 (expert evaluation) and D.13 (repeated runs). Section 2 (Related Work) now describes four more review systems, and Table 1 adds two stronger frameworks. The ICLR 2027 evaluation, the ScholarPeer-protocol comparison and the review behavior analysis are reported in the individual replies, and we will add them to the appendix of the final version.

AuthorsOpenReview ↗

Summary of the Discussion

We thank the AC and the reviewers. As the discussion closes, we summarize what is new in this paper and where each concern stands.

What is new

  • The rubric is formed after reading the paper, from past human reviews. Human peer review sends each paper to experts in its domain, because standards differ across domains. AI reviewers are already being piloted at real conferences, yet most of them fix what to check before reading the paper, through a prompt, a review form or a trained model. DynReviewer reads the paper first, then turns past human reviews into a rubric for that paper, drawing on general checks distilled from past reviews and on what reviewers criticized in the most related papers. It needs no training, so it works with any backbone, and its experience grows with every public review cycle.
  • Why this matters. A fixed configuration misses what experts in each domain care about. Within one venue, the same AI reviewer's recall of the weaknesses found by human reviewers varies by 7 to 12 percentage points across domains (Section 3), a gap that venue-level averages hide. ReviewAtlas-Bench, built from ReviewAtlas-40K+ (45,551 papers and their reviews from six venues), reports results by venue and by domain. DynReviewer leads in every venue and domain on all three backbones, by 5.6 to 19.9 percentage points over the strongest same-backbone baseline.
  • A contamination-free test. To our knowledge, our ICLR 2027 evaluation is the first test of AI reviewers on an ongoing review cycle, with each submitted version matched to its first-round reviews, which appeared after the backbone's training cutoff.

The official reviews make the same point. Reviewer 1Bqq writes that the approach "might resolve the problem that the current AI reviewers have similar tastes which are sometimes misaligned with human's". Reviewer WGpW finds it "more practical than fine-tuned reviewers such as DeepReviewer and CycleReviewer", and Reviewer 1Gka calls it "a practical way to turn historical reviews into applicable checks".

The concerns and our evidence

  • Difference from TreeReview and Agent Reviewers (1Gka). Ablations isolate both differences (Table 3: w/ Generic Rubrics, w/o Compile), and DynReviewer outperforms both methods on the same papers (Appendix D.11).
  • Value of past experience (1Gka). Checks distilled from human reviews outperform LLM-written ones, and past experience shifts the reviewer's focus toward experimental design and baseline comparison.
  • Source of domain adaptation (WGpW). It comes from the related-paper path. Removing this path drops Q from 42.66% to 24.12% (Table 3).
  • Metrics and judge validity (WGpW, 1Bqq). The metrics follow prior work. Two blind experts rank review quality largely in line with the judge (Appendix D.12). DynReviewer also leads Stanford Agentic Reviewer and CSPaper Review under ScholarPeer's SxS and H-Max protocols, which do not use our extracted concerns.
  • Baselines (1Bqq). The four systems named are closed-source or paid services. We evaluated each through the access it offers, and DynReviewer is the highest on all four metrics (Appendix D.11).
  • Human evaluation and stability (1Gka, WGpW). Blind expert evaluation is in Appendix D.12, and three independent runs are in Appendix D.13.
  • Manuscript versions (1Bqq, WGpW). First-submission versions of recent ICLR, ICML and NeurIPS papers are not public [1]. On the 100 matched ICLR 2027 submissions, DynReviewer reaches a Q of 57.93% against 46.97% for ReviewGrounder, and a concern coverage of 43.33% against 33.04%.

Scope of the revision

Reviewers WGpW and 1Gka rated soundness as having "minor or fixable concerns". We answered each concern with additional evidence, mostly in the appendix (A, B.1, C.5, D.10 to D.13). In the main text, Related Work describes four more systems and Table 1 adds two stronger frameworks. The method and the main results are unchanged.

Where the discussion stands

Reviewers WGpW and 1Gka have not yet responded to these results, and Reviewer 1Bqq's follow-up did not address the matched ICLR 2027 test. It states that "the proposed method seems outperforming". The one concern it keeps, version matching, is a limit of the public record that every published benchmark in this area shares, and the ICLR 2027 test addresses it directly.

The final version will add the ICLR 2027 evaluation and the other new analyses, and will state the version limitation and its cause in Section 3 and Appendix B.1.

Every concern raised in the reviews now has a direct answer, with the evidence on record in this thread. We respectfully ask the AC to weigh this evidence, together with the paper's contributions, in the final decision.

[1] PaperScorer: How Predictable Is Paper Acceptance? arXiv:2610.31791, NeurIPS 2026, Table B.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.