- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes DynReviewer, a training-free reviewing framework that adapts historical human-review experience to each target paper. It combines general checks distilled from past reviews with concerns retrieved from related papers, converts both into paper-specific rubrics, and verifies potential criticisms against the manuscript. The authors also introduce ReviewAtlas-40K+, containing 45,551 papers with associated reviews, and an 800-paper benchmark spanning four venues. Experiments across three backbones report improvements in concern coverage, judge-rated review quality in most settings, and agreement with ICLR acceptance decisions.
Critical strengths
-
The framework provides a practical way to turn historical reviews into applicable checks rather than directly copying criticisms from other papers. Explicit applicability conditions and evidence requirements make the generated findings easier to inspect.
-
The evaluation covers four venues and separately reports four ICLR domains across three backbones. Component ablations, source-exclusion experiments, and finding-budget analyses provide useful evidence about the behavior of the complete system.
Critical weaknesses
-
The methodological novelty appears limited. Adaptive review questions and historical-memory-based reviewing have already been explored by methods such as TreeReview and Agent Reviewers. The training-free, dual-path design is useful, but the contribution currently appears primarily to be an integration and refinement of existing ideas. The paper does not sufficiently establish the distinctive methodological advance over closely related adaptive or rubric-based reviewing systems. A clearer comparison with these approaches would help substantiate the added value of the proposed design.
-
The analyses provide limited insight into reviewing behavior and the value of historical experience. Given the substantial paper–review corpus, I would expect a deeper investigation of how AI and human reviewers differ in the issues they identify, which concern types AI reviewers systematically miss or overproduce, and how human reviewing priorities change over time within comparable venues and domains. It would also be valuable to identify which concern categories and severity levels are most strongly associated with human ratings and acceptance outcomes, while distinguishing association from causation. Such analyses would clarify whether DynReviewer improves the identification of consequential issues and explain why historical experience helps.
-
The claimed improvement in review quality lacks independent human evaluation. Concern matching, evidential support, and content quality are predominantly assessed by a single LLM judge, while Appendix A acknowledges that the independent expert audit remains incomplete. Agreement with official acceptance decisions is encouraging, yet does not establish that individual criticisms are correct, relevant to the paper’s claims, or useful to authors. A blinded evaluation by domain experts on a representative sample is needed, comparing factual correctness, with inter-annotator agreement reported.
Questions for authors
See Critical Weaknesses.
6 comments
Response to Reviewer 1Gka (2/5): Direct Comparison (W1, cont.) and AI vs. Human Concern Types (W2)
W1 (continued)
(3) In direct comparison, DynReviewer clearly outperforms both methods (Appendix D.11). We ran all three systems with GLM-5.3-Flash on 100 ICLR 2025 papers, 25 from each of the four domains. For TreeReview, we use the authors' released code. Agent Reviewers has no public code, and its core memory module is not released either. We reproduced it following the pipeline and prompts described in its paper.
| Review framework | Union-W (%) | SFP (%) | Q (%) | Overall |
|---|---|---|---|---|
| TreeReview | 19.23 | 90.25 | 30.44 | 7.13 |
| Agent Reviewers | 18.14 | 66.90 | 27.02 | 5.13 |
| DynReviewer | 31.09 | 98.75 | 45.79 | 7.95 |
Union-W is concern coverage, SFP is finding support (the share of criticisms supported by the manuscript), and Q is the per-paper harmonic mean of the two. Overall is review content quality (−2 to 8).
DynReviewer's Q is 15.35 percentage points (pp) higher than TreeReview's and 18.77 pp higher than Agent Reviewers'. It also uses 46.0% fewer tokens than TreeReview. The gap comes mainly from concern coverage. DynReviewer recovers over 60% more human concerns than either method, and it also has the highest finding support.
W2: On analyzing review behavior and the value of past experience
Thank you for these questions. They bear on whether AI reviewers can truly find important issues. Some of them are hard to answer rigorously. When an AI criticism does not appear in the human reviews, the AI may be wrong, or the human reviewers may have missed the issue. No current method can reliably tell the two apart across a large number of manuscripts. To establish that a type of criticism leads to lower scores, one would need interventional comparisons on the same paper. Public review records can only show associations. Within these limits, we give the following analyses in the order of your questions. DeepSeek v4 Flash categorized the criticisms without knowing whether each came from a human or an AI. The tables list six main categories.
(1) How AI and human reviewers differ in the issues they identify, and which concern types AI reviewers systematically miss or overproduce. We counted the category distribution of human criticisms and of criticisms from three AI systems on 100 ICLR 2025 papers (all systems use DeepSeek v4 Flash).
| Issue category | Human | DynReviewer | ReviewGrounder [3] | AI Scientist-v2 [4] |
|---|---|---|---|---|
| Experimental design | 16.51 | 28.39 | 23.86 | 11.88 |
| Method/theory | 29.61 | 25.46 | 32.09 | 26.24 |
| Results/statistics | 6.32 | 16.67 | 11.88 | 31.78 |
| Baseline comparison | 8.19 | 10.47 | 10.65 | 2.88 |
| Novelty | 8.03 | 4.19 | 1.44 | 1.37 |
| Writing/presentation | 22.90 | 4.52 | 5.97 | 15.05 |
Units are %. Shares are computed per paper and then averaged. Only the six main categories are listed.
All three AI systems systematically raise fewer writing/presentation and novelty issues. DynReviewer and ReviewGrounder raise more experimental design issues, and AI Scientist-v2 raises more results/statistics issues. On novelty, DynReviewer's share is about 3 times that of the other two AI systems, but still only half the human share. Under the ScholarPeer protocols, its novelty and significance judgments also beat those of the two online review systems (see our reply to 1Bqq). Novelty judgment remains an area where AI reviewers need to improve.
References
[3] ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents. https://arxiv.org/abs/2604.14261
[4] The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search. https://arxiv.org/abs/2504.08066
Response to Reviewer 1Gka (3/5): Trends over Time, Links to Ratings, and Value of Past Experience (W2, cont.)
W2 (continued)
(2) How human reviewing priorities change over time within comparable venues and domains. We applied the same method to the public ICLR 2025 reviews and the ICLR 2027 first-round reviews. For both years, we sampled 25 papers from each of the same four domains (in this analysis, DynReviewer uses GLM-5.3-Flash).
| Review source | Share of experimental design + results/statistics: ICLR 2025 | ICLR 2027 |
|---|---|---|
| Human reviews | 22.83% | 38.78% |
| DynReviewer | 47.72% | 48.89% |
Between the two years, DynReviewer's category distribution stays roughly the same. Human reviews give markedly more attention to experimental design and results/statistics in 2027, moving closer to the focus of AI reviews. Possible causes of this shift in human focus include the submission mix, reviewer guidelines, or reviewers' greater use of AI tools. We cannot yet tell these apart.
(3) Which concern categories and severity levels in AI reviews are most strongly associated with human ratings and acceptance outcomes. We grouped papers by DynReviewer's criticisms and matched papers within the same domain by abstract similarity. Matching uses neither human scores nor decisions. We then compared the mean human scores and acceptance rates of the two groups. The two categories below show the clearest associations. For both, the 95% confidence interval of the difference excludes 0:
| Matched groups A / B | Mean human score A / B | Difference A−B [95% CI] | Acceptance rate A / B |
|---|---|---|---|
| DynReviewer flagged a severe issue / did not | 4.78 / 5.44 | −0.66 [−1.25, −0.04] | 21.4% / 50.0% |
| More / fewer experimental-design criticisms | 4.92 / 5.71 | −0.80 [−1.34, −0.27] | 37.5% / 43.8% |
Papers in which DynReviewer flagged a severe issue receive lower human scores and are accepted much less often. These are associations after matching and do not establish causation. Matching controls only for domain and topical similarity. Other aspects of paper quality may affect both the AI criticisms and the human scores.
(4) Whether DynReviewer is better at finding important issues. In Table 6 of the paper, the judge selects only the K most important criticisms from each review before computing Q. At K = 3, DynReviewer reaches 28.50%, ReviewGrounder 23.90% and AI Scientist-v2 20.19%. Even when only the few most important criticisms count, DynReviewer recovers more human concerns. Combined with (3), the severe issues it flags are consistent with human scores and decisions.
(5) Why past experience helps. In Table 3 of the paper, replacing the 18 checks distilled from human reviews with LLM-written checks (w/ Generic Rubrics) drops Q from 42.66% to 36.87%. Past experience also changes which issues the AI focuses on. Compared with the version that removes all past experience (w/o Experience), the full system's criticisms have 9.38 pp and 6.41 pp higher shares of experimental design and baseline comparison, respectively (95% CI [6.05, 12.54] and [4.34, 8.31]). Appendix F.1 gives examples from the four domains. Each criticism can be traced back to the general check or related-paper review it came from.
Finally, your question about how human review focus changes over time points us to a direction that deserves the community's attention. As AI tools are used more in writing and reviewing, the focus of human reviews may gradually drift toward that of AI. The cross-year results in point (2) already show signs of this. If the trend continues, two questions deserve further study: how much independent expert judgment future human review experience can still offer AI reviewers, and how methods that learn from past reviews should select that experience.
Response to Reviewer 1Gka (1/5): Novelty and Differences from TreeReview and Agent Reviewers (W1)
Thank you for recognizing that DynReviewer turns past reviews into checks with applicability conditions and evidence requirements, and for valuing our evaluations and ablations across venues, domains and backbones. To address your three points, we added direct comparisons with TreeReview and Agent Reviewers, an analysis of review behavior, and an independent blind expert evaluation.
W1: On methodological novelty and the differences from TreeReview and Agent Reviewers
(1) Our core contribution is to configure a dedicated AI reviewer for a paper after reading it. Human peer review assigns each paper to experts in its area, because different papers call for different criteria. Theory papers require checking assumptions and proofs, empirical papers require checking baselines and ablations, and systems papers require checking scalability and overhead. Most existing AI reviewers fix their configuration before reading the paper. Whether they are trained reviewer models or pre-written prompts, review forms or roles, they apply the same criteria to every paper. To our knowledge, no prior work has made it a core problem to automatically build a review rubric for a paper from past human reviews after reading it. Table 9 in the paper compares the method designs of existing AI reviewers. Among them, DynReviewer is the only method that is training-free, adapts to each paper, and derives its rubric from human reviews.
(2) Neither TreeReview nor Agent Reviewers has a step that builds a review rubric for each paper from past human reviews. Section 2 (Related Work) of the paper already discusses both methods, and Table 9 also compares the method design of Agent Reviewers.
As you noted among the strengths, DynReviewer turns past reviews into usable checks rather than copying criticisms from other papers. This conversion is exactly the step both methods lack. Two ablations in Table 3 of the paper test the effect of each of these two differences (200 ICLR papers, DeepSeek v4 Flash):
References
[1] TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review. https://aclanthology.org/2025.emnlp-main.790/
[2] Agent Reviewers: Domain-specific Multimodal Agents with Shared Memory for Paper Review. https://proceedings.mlr.press/v267/lu25p.html