acceptodds
Under review as a conference paper at ICLR 2027

PRISM: A Multi-Dimensional Benchmark for Evaluating LLM Peer Reviewers

11 views

Abstract

The rapid growth in submissions to machine learning venues has strained the scientific peer-review system and intensified interest in LLM-based automated peer reviewers. However, how good these systems are actually, especially compared to expert humans at catching scientific gaps, remains poorly understood. In this work, we introduce **PRISM** (**P**eer **R**eview **I**ntelligence via **S**tructured **M**ulti-dimensional assessment), a benchmarking framework that evaluates review quality across four dimensions: **Depth of Analysis**, **Novelty Assessment**, **Flaw Identification & Major Issues Prioritization**, and **Multi-dimensional Constructiveness**. Unlike most existing evaluations based on surface-level metrics like ROUGE and BLEU, or unconstrained LLM-as-a-judge prompting that conflates fluency with rigor, PRISM grounds each dimension in argument mining, retrieval-augmented verification, and consensus-based scoring. We apply PRISM to benchmark five leading automated reviewer systems and expert human reviewers on a stratified corpus of reviews from ICLR, ICML, and NeurIPS. The results reveal that LLMs can match or beat human experts on individual dimensions: comparable depth of analysis, stronger novelty verification, and higher flaw recall. However, no single system consistently matches the balanced performance of the human baseline across all dimensions at once. Each exhibits a distinct specialization profile with characteristic blind spots—failure modes that aggregate metrics miss entirely. The implication is that *LLM reviewers are best understood as targeted supplements to human review, effective within specific dimensions, but unreliable as standalone replacements.*

Primary area:
datasets and benchmarks
Keywords:
Peer review evaluation, large language models, argument mining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_zTb9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

PRISM evaluates peer-review quality across four dimensions: analytical depth, novelty assessment, flaw identification and prioritization, and constructiveness. It combines structured argument extraction, literature retrieval, LLM-based verification, and explicit scoring rules. The framework evaluates five automated reviewer systems against human reviews on 600 papers from three machine-learning conferences.

Critical strengths

The evaluation spans multiple systems and venues. Detailed prompts, worked examples, human annotations, and an alternative-judge experiment make the framework relatively transparent.

Critical weaknesses
  • Limited novelty. The framework largely combines heuristic components, and its contribution beyond existing reviewer evaluation methods remains unclear.
  • Unclear practical insights. The paper does not establish how its findings can guide concrete improvements to reviewer systems.
  • Insufficient validation. Comparisons with existing benchmarks and evaluators are limited. The human validation does not sufficiently establish that the proposed metrics capture review quality.
Questions for authors
  • What is PRISM’s main contribution beyond existing reviewer evaluation approaches?
  • Which key insights suggest specific, testable improvements to reviewer systems?
  • How does PRISM compare with existing evaluators against independent expert judgments, and what evidence demonstrates its reliability?
7 comments

We appreciate your time and effort for reviewing our submission. Your key insight and valuable question help us boost our paper's quality. Regarding the questions, we provide the following clarification:

Q1. Limited novelty. The framework largely combines heuristic components, and its contribution beyond existing reviewer evaluation methods remains unclear. What is PRISM’s main contribution beyond existing reviewer evaluation approaches?

R: PRISM’s main methodological contribution is a unified, evidence-grounded framework that evaluates human and LLM reviews on equal terms, without assuming human reviews are the ground truth. Rather than measuring how closely an LLM review resembles a human review, PRISM evaluates what scientific reviews are expected to achieve: identifying valid and critical flaws, making literature-grounded novelty judgments, prioritizing major issues, providing actionable feedback, and supporting arguments with evidence. These capabilities are assessed separately through structured review units and dedicated evaluation pipelines. The components are grounded in established constructs from the peer-review literature, but their integration enables capability-level comparisons between human and automated reviewers under the same protocol.

Why this matters for the domain. Peer review gates what enters the scientific record, yet human reviews are themselves incomplete and sometimes wrong [8], and reviewing outcomes are noticeably inconsistent [10]. As LLM reviewers enter this process, the central question is no longer whether a review resembles a human review, but whether its criticisms are correct and important. An evaluator that rewards similarity to human reviews inherits their omissions and errors. An evaluator that gives one holistic score cannot show which statement is wrong. Overlap metrics barely change when a review is paired with references from a different paper (a BERTScore drop of 1.8–2.6 points [1, App. C.1, Table 7]). Holistic LLM judges inherit position, verbosity and self-preference biases [3–5], frontier LLMs reach only about 20 F1 when asked to locate deficient review segments [6], and holistic LLM judges align poorly with expert ratings of review quality [9].

We compare existing approaches with PRISM on the same requirements:

Requirement of a good reviewWhy it mattersPRISM’s designExisting approaches
Criticisms are correctWrong criticisms mislead authors and decision-makers, and LLM reviewers hallucinate flawsEach flaw is verified against the manuscript; invalid flaws are excluded and reportedReviewEval [7] also checks factual correctness against the paper; overlap metrics and holistic judges do not
Major flaws are found, including those others missGatekeeping depends on catching critical flaws, and any single review covers only part of them [8]Recall against a pooled verified-flaw reference from all human and LLM reviews, split into Critical and MinorCoCoReviewBench [8] filters human references for correctness but credits only issues humans raised; overlap metrics reward similarity
Novelty is judged against prior workNovelty is a central acceptance criterion and is inherently literature-relative [11,12]Each novelty claim is checked against independently retrieved literature, with a time cutoffRubric-based evaluators score novelty statements without checking them against the literature [7]; ScholarPeer’s search-enabled judge sees literature but gives pairwise preferences
The most important issues come firstArea chairs and authors triage by severitynCPS, a severity-weighted ranking score (Järvelin & Kekäläinen, 2002)REMOR’s training reward includes importance-based prioritization, but without verified severity
Feedback is actionableReviews serve authors as well as decisions [2, 14]Comment-level scores for actionability, specificity, justification, solution and tone (D1–D5)RevUtil [2] rates comment utility for authors; PRISM measures this alongside the other capabilities
Arguments are groundedUnsupported claims cannot be checked or acted on [15]Premise share and grounding of argument units (DoA)Depth is usually scored with holistic rubrics [7]

What this design reveals. These capabilities are measured separately and are only weakly correlated (max |r| = 0.193; App. D), so PRISM exposes findings that a single score would hide. Some LLM reviewers exceed individual human reviewers in Critical-flaw recall (§4.2.3), yet only human review scores differ between accepted and rejected papers (§4.2.5). This is exactly the distinction the domain needs in order to decide where LLMs can assist reviewers and where they cannot. The reliability of each component is validated against experts and across judges and prompts.

In the revision, we will restructure §2 around these requirements and include the table above.

[1] Chang et al. (2025). TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), pp. 15651–15682. https://aclanthology.org/2025.emnlp-main.790/

[2] Sadallah et al. (2025). The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), pp. 28991–29021. https://aclanthology.org/2025.emnlp-main.1476/

[3] Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html

[4] Saito et al. (2023). Verbosity Bias in Preference Labeling by Large Language Models. Workshop on Instruction Tuning and Instruction Following at NeurIPS 2023. https://mlanthology.org/neuripsw/2023/saito2023neuripsw-verbosity/

[5] Panickssery et al. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Main Conference Track. https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html

[6] Du et al. (2024). LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pp. 5081–5099. https://aclanthology.org/2024.emnlp-main.292/

[7] Garg et al. (2025). ReviewEval: An Evaluation Framework for AI-Generated Reviews. Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 20542–20564. https://aclanthology.org/2025.findings-emnlp.1120/

[8] Deng et al. (2026). CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), Proceedings of Machine Learning Research (PMLR), Vol. 306, pp. 23997–24057. https://proceedings.mlr.press/v306/deng26j.html

[9] Ebrahimi et al. (2025). RottenReviews: Benchmarking Review Quality with Human and LLM-Based Judgments. Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM 2025), pp. 5642–5649. https://doi.org/10.1145/3746252.3761506

[10] Cortes & Lawrence (2021). Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. arXiv:2109.09774. https://arxiv.org/abs/2109.09774

[11] Morgan (1985). Originality, Novelty and Priority: Three Words to Reckon with in Scientific Publishing. Canadian Medical Association Journal, 132(1), pp. 8–9. https://pmc.ncbi.nlm.nih.gov/articles/PMC1346489/

[12] Mishra & Torvik (2016). Quantifying Conceptual Novelty in the Biomedical Literature. D-Lib Magazine, 22(9/10). https://doi.org/10.1045/september2016-mishra

[13] Järvelin & Kekäläinen (2002). Cumulated Gain-based Evaluation of IR Techniques. ACM Transactions on Information Systems, 20(4), pp. 422–446. https://doi.org/10.1145/582415.582418

[14] Kennard et al. (2022). DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2022), pp. 1234–1249. https://aclanthology.org/2022.naacl-main.89/

[15] Hua et al. (2019). Argument Mining for Understanding Peer Reviews. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019), pp. 2131–2137. https://aclanthology.org/N19-1219/

Q2. Unclear practical insights. The paper does not establish how its findings can guide concrete improvements to reviewer systems. Which key insights suggest specific, testable improvements to reviewer systems?

R: A key insight of PRISM is that submitted human reviews and automated reviews have complementary capability profiles. This suggests concrete, testable directions for human–AI reviewing workflows rather than assuming that the same system should handle every review task:

(1) LLMs as flaw scanners, humans as verifiers. In the original pooled-reference analysis, some automated systems have higher critical-flaw recall than individual submitted human reviews (§4.2.3), but they hallucinate ~3.6–16.2% of their flaws, with those identified hallucinations categorized as minor (Fig. 6). Improvement: LLMs produce a candidate flaw list and the human reviewer validates and prioritizes it. Test: flaw recall and hallucination rate of the combined review vs. the human review alone.

(2) LLMs supply fixes, humans supply depth. Human reviews are strong on grounded, methodology-focused argumentation (§4.2.1) but rarely propose solutions (D4 = 0.427). A purpose-built LLM pipeline does so far more often (D4 = 0.781) and has the highest constructiveness score (0.636 vs. 0.556 for humans) (§4.2.4). Improvement: use LLMs to turn human-identified weaknesses into actionable suggestions. Test: D1/D4 with DoA preserved.

(3) Novelty and merit judgments should stay human-led. LLM systems tend to endorse author novelty claims (up to 83% “Novel” vs. 62% for humans; Fig. 4). More importantly, only human review scores differ significantly between accepted and rejected papers; the automated-review scores do not show the same statistically significant accepted-versus-rejected separation (§4.2.5). Improvement: LLMs supply retrieved prior work and counter-evidence, while a human reviewer retains responsibility for the novelty verdict and overall recommendation. Test: stance distribution and the accept-vs-reject comparison for the combined review.

For fully automated systems, the same findings identify the gaps to close: claim verification, counter-evidence search for novelty, and sensitivity to paper quality. Because PRISM measures each dimension separately, these proposals yield specific, measurable hypotheses; they are suggested follow-up experiments, not interventions already validated in this study.


Reviewer_TicPOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

PRISM evaluates peer reviews across four dimensions, covering depth of analysis, novelty assessment, flaw identification and prioritization, and constructiveness. The authors develop dedicated evaluation pipelines and compare five automated reviewing systems with human reviewers across 600 papers from three conferences. The experiments reveal different strengths and limitations across systems, with no single automated reviewer consistently matching the human baseline across all dimensions. The authors also conduct human validation with 20 researchers on 30 papers.

Critical strengths
  1. The paper addresses an important problem and presents a comprehensive experimental study of automated peer reviewing. Examining several dimensions of review quality across multiple systems and conferences provides a useful picture of their different strengths and limitations.

  2. The paper is easy to read, and each experimental section communicates a clear message. The authors document the experiments thoroughly, and the extensive appendix provides detailed descriptions of the pipelines, prompts, implementation choices, and human evaluation protocol. This documentation makes the framework and experimental setup easier to understand and assess. Every experiment has a clear takeaway.

Critical weaknesses

My main concern is that the paper introduces a comprehensive set of metrics and then draws substantial conclusions from them before establishing their reliability. For an evaluation framework, validating the metrics should be a central part of the contribution. Here, most of the paper focuses on comparisons between reviewing systems, while the human validation occupies a relatively small section near the end. The study involves 20 experienced researchers, but it covers only 30 papers. This feels limited given the number of dimensions, intermediate judgments, and conclusions that depend on the evaluator.

More importantly, the annotators assess components that the pipeline has already extracted, and the novelty evaluation also supplies the retrieved papers. This setup helps assess scoring consistency, but it leaves important parts of the evaluation untested. For example, the novelty pipeline involves extraction, retrieval, and verification. The paper needs to establish whether extraction preserves the reviewer’s actual claims, whether retrieval finds the necessary evidence, and whether verification interprets that evidence correctly. Without validation at each stage, an incorrect final score could reflect several different failures. I need stronger evidence that these pipelines work before trusting the broader conclusions about the relative abilities of human and automated reviewers.

Another main concern I have is that the additional value of PRISM over existing peer-review evaluation methods remains unclear. The paper discusses some relevant frameworks, including ReviewEval and RottenReviews, but does not provide a direct empirical comparison showing what PRISM captures that these approaches miss. A comprehensive collection of metrics does not by itself establish a better evaluation framework. The important question is whether these metrics reveal meaningful differences in review quality that existing methods overlook, and whether experts agree with those distinctions. Comparing the frameworks on the same reviews would help establish this. Examples where the methods disagree would be particularly informative, especially if independent human judgments clarify which assessment better reflects the quality of the review. Basically the main question is what is PRISM finding that previous eval suite do not? The current experiments mainly demonstrate how reviewing systems perform according to PRISM. They provide less evidence about why PRISM itself offers a more reliable or informative assessment than the available alternatives.

The human validation reports agreement scores without enough analysis of the underlying disagreements. For example, the 63.5% agreement on flaw aspect classification raises questions about which categories cause confusion and why. The paper would benefit from a systematic examination of these cases, including whether the disagreements concern ambiguous labels, misunderstood criticisms, or substantive differences in judgment. Their consequences also matter. Some errors may have little effect on aggregate results, while others could change the comparison between systems.

The flaw-verification stage deserves particular attention. In practice, one reviewer may identify a valid major flaw because of specialized expertise, while other reviewers miss it. Discussions between reviewers and authors often establish the significance of these concerns. PRISM combines verified flaws across reviewers, but the LLM verifier still needs to recognize subtle criticisms that only one expert raises. The current validation does not clearly establish how reliably it handles these cases.

The benchmark uses reviews from 2025 and 2026, when reviewers already commonly use LLM assistance. Some reviews in the human baseline may therefore reflect a combination of human judgment and AI-generated or AI-edited text. This complicates the interpretation of comparisons between human and automated reviewing. Using older reviews might reduce this concern, but it could increase the risk of training-data contamination. I understand this tradeoff and do not think there is an obvious solution. However, the paper should more carefully explain what its human baseline represents and how possible AI assistance affects its conclusions. The distinction matters when interpreting claims about whether automated systems match or exceed human reviewing abilities.

Lastly several relevant papers are absent from the paper.

  • REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning
  • ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review
  • ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents
  • CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers
  • PeerPrism: Peer Evaluation Expertise vs Review-writing AI
  • PeeriScope: A Multi-Faceted Framework for Evaluating Peer Review Quality

A minor presentation concern is that several figures are too small to read comfortably.

Questions for authors
  1. What evidence supports the reliability of each pipeline stage? For example, Section 3.2 involves extraction, retrieval, and verification, but the human study supplies annotators with extracted claims and retrieved papers. How accurately do the earlier stages identify the relevant claims and evidence, and how much do their errors affect the final scores?

  2. How does PRISM compare with existing evaluation suites on the same reviews? Examples where PRISM captures an important distinction that other methods miss, supported by independent expert judgments, would help clarify its contribution. Some important baseline eval suite include: REMOR, ScholarPeer, ReviwewGrounder,RottenReviews, Peeriscope etc. I mean if you would have done the same thing with these baselines, would the results be different?

  3. What explains the disagreements between human annotators and the LLM evaluator? For example, Table 3 reports 63.5% agreement on flaw aspect classification. Which categories or types of cases account for these disagreements, and do they affect the comparisons between reviewing systems?

  4. How reliably does the flaw verifier retain subtle but valid major concerns that only one expert reviewer identifies? The framework combines verified flaws across reviewers if there is one reviewer raising a big flaw which is not in consensus what will happen in this case?

  5. The choice of recent reviews from 2025 2026 helps reduce contamination risk, but it also complicates the interpretation of the human baseline as it perhaps already is human-ai collaboration of reviews not fully human! Any thoughts on this?

8 comments

We thank the reviewer for the thoughtful comments, which will help us improve the paper.

Q1. My main concern is that the paper introduces a comprehensive set of metrics and then draws substantial conclusions from them before establishing their reliability. For an evaluation framework, validating the metrics should be a central part of the contribution. […] The study involves 20 experienced researchers, but it covers only 30 papers. This feels limited given the number of dimensions, intermediate judgments, and conclusions that depend on the evaluator. The annotators assess components that the pipeline has already extracted, and the novelty evaluation also supplies the retrieved papers. […] The paper needs to establish whether extraction preserves the reviewer’s actual claims, whether retrieval finds the necessary evidence, and whether verification interprets that evidence correctly. What evidence supports the reliability of each pipeline stage? […] How accurately do the earlier stages identify the relevant claims and evidence, and how much do their errors affect the final scores?

PRISM is designed to make review evaluation traceable and grounded in established literature on peer review (depth [1], literature-relative novelty [2,3], prioritization [4], and peer-review discourse structure [5]), replacing lexical-overlap metrics and holistic rubric-prompted LLM judging (Section 2). Since PRISM’s goal is to reliably compare reviewer systems on these grounded and literature-defined dimensions, not to claim a perfectly objective absolute score, we validate it at the level where its conclusions are drawn: the alignment of its final judgments with human experts. Accordingly, we focus on robustness (App. E.7) and informativeness (cross-metric independence (App. D), along side human–judge agreement on ranking (Section 4.3). This agreement also covers the intermediate judgments each metric is built on (Table 3): premise grounding for DoA, claim–evidence support for Novelty, flaw validity, i.e., the verification step that filters hallucinated flaws from the consensus ground truth, flaw severity for Recall/nCPS, and per-comment D1–D5 for MCS. The human evaluation also follows the scale of prior work (TreeReview: 20 papers/5 experts; CycleResearcher: 20/3; ReviewEval: 30/5), which already requires extensive annotation effort; we agree with the reviewer a larger scale human evaluation (with more papers, review systems, LLM judges, etc...) is desirable, but it is a substantial effort that requires a separate, independent study.

Nevertheless, to further strengthen the reliability of PRISM, we conducted additional stage-wise validation experiments to assess the accuracy of extraction and retrieval, as well as the impact of errors in these stages on the final evaluation scores. Verification and scoring were previously evaluated against expert judgments in §4.3.

[1] Hua et al. (2019). Argument Mining for Understanding Peer Reviews. NAACL-HLT 2019. https://aclanthology.org/N19-1219/

[2] Morgan (1985). Originality, Novelty and Priority: Three Words to Reckon with in Scientific Publishing. Canadian Medical Association Journal, 132(1), 8–9. https://pmc.ncbi.nlm.nih.gov/articles/PMC1346489/

[3] Mishra & Torvik (2016). Quantifying Conceptual Novelty in the Biomedical Literature. D-Lib Magazine, 22(9/10). https://doi.org/10.1045/september2016-mishra

[4] Järvelin & Kekäläinen (2002). Cumulated Gain-based Evaluation of IR Techniques. ACM Transactions on Information Systems, 20(4), 422–446. https://doi.org/10.1145/582415.582418

[5] Kennard et al. (2022). DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions. NAACL 2022. https://aclanthology.org/2022.naacl-main.89/

We summarize the additional findings below.

Extraction. We report the precision of extracted units, and the share of reviews in which the researchers found at least one important unit that the pipeline missed:

StagePrecision [95% CI]Reviews with ≥1 omission
Flaw arguments90% [85, 95]25%
Constructiveness ARCs90% [86, 93]37%
DoA ADUs85% [75, 93]37%
Novelty claims42% [32, 55]35%

The weak stage is novelty-claim extraction. Its errors are mostly statements that are not novelty judgments, rather than claims that were missed.

Retrieval. This is the weakest stage. The candidate pool recovers only 34% of the works cited by reviewers (matched by title or abstract), and the researchers judged the retrieved evidence sufficient for only 17% of claims. The pool also often contains the reviewed paper itself (56–70% of reviews). Removing it lowers Novelty Score by 0.064–0.074 for every system and leaves the ranking unchanged, except for one near-tied pair (gap < 0.004).

Effect on final scores. We recomputed every score after (P1) removing the units the researchers marked invalid, and (P2) also applying their corrections that need no new judge calls:

  • Flaw and Constructiveness are unaffected: per-review shifts are ≤ 0.017, τ = 1.00, and all LLM–human gap signs are preserved.
  • DoA: the shift is 0.029 under P1 (τ = 0.73). Under P2 it is 0.111, but this overstates the effect, because missing claims are added without their premises.
  • Novelty Score: levels shift by 0.06 and the ranking is not stable (τ = 0.33). Removing non-novelty statements raises the human NS and roughly halves the LLM–human gaps.

We therefore consider the Flaw and Constructiveness conclusions robust to extraction errors. The Novelty Score comparisons, by contrast, should be read as grounding within the retrieved literature, which is how §4.2.2 frames them. In the revision we will (i) exclude the reviewed paper from its own pool, (ii) add a filter for non-novelty statements, (iii) retrieve reviewer-cited works directly by title, and (iv) report these audits in the paper.

Q2. Another main concern I have is that the additional value of PRISM over existing peer-review evaluation methods remains unclear. […] Basically the main question is what is PRISM finding that previous eval suite do not? How does PRISM compare with existing evaluation suites on the same reviews? Examples where PRISM captures an important distinction that other methods miss, supported by independent expert judgments, would help clarify its contribution. […] I mean if you would have done the same thing with these baselines, would the results be different?

R: PRISM answers a different question from these works. Some generate reviews, in which case PRISM can benchmark them as systems. Others evaluate reviews using human reference reviews, rubric-based assessments, or authorship detection. While rubric-based methods may also assess multiple dimensions, PRISM differs in its unit-level, evidence-grounded evaluation. Rather than directly assigning review-level scores, PRISM extracts individual argumentative units, flaw arguments, novelty claims, and atomic comments, evaluates them through dedicated pipelines and aggregates the results into capability-level scores. In particular, PRISM verifies flaws against the manuscript and novelty claims against prior literature, making these judgments traceable. It also evaluates human and LLM reviews under the same protocol, without treating human reviews as the ground-truth reference. We will add all of these works to §2:

WorkTypeHow it evaluates reviewsHow PRISM differs
REMOR [6]Reviewer system (reasoning LLM, SFT + GRPO)Multi-aspect training reward: criticism, examples, importance-based prioritisation, relevance via text overlap with the manuscriptThe reward measures what a review contains, not whether it is correct. PRISM verifies each flaw against the manuscript and each novelty claim against prior literature, and could benchmark REMOR as a system.
ScholarPeer [7]Reviewer system (search-enabled multi-agent)Pairwise LLM judge on 5 dimensions; H-Max scores an AI review 1–10 against the set of human reviewsPairwise and review-level, so scores are not traceable to statements, and human reviews serve as the reference rather than being scored.
ReviewGrounder / ReviewBench [8]Reviewer system + benchmarkPaper-specific rubrics derived from guidelines, the paper and the human reviews; an LLM evaluator scores 8 dimensionsRubrics anchored on human reviews and scored per review. PRISM verifies individual units and pools flaws from all reviewers.
CoCoReviewBench [9]Evaluation benchmark (3,900 papers)Category-level comparison with human references, filtered using reviewer–author–meta-review discussions; categories without human reviews are skippedThe closest to PRISM: both target correctness rather than overlap. It cannot credit issues no human raised and does not score human reviews; PRISM does both. Its use of discussions is complementary.
PeerPrism [10]AI-authorship detection benchmarkWhether detectors identify the origin of a review’s ideas vs. its textA different question: who wrote a review, not how good it is.
PeeriScope [11]Review-quality platformText features, rubric-guided LLM assessments and supervised predictionA review-level quality profile, without verification against the manuscript or the literature.
RottenReviews [12]Review-quality benchmarkExpert annotations, feature-based models and LLM judgesFinds that LLM judges align poorly with experts on holistic quality, which motivates PRISM’s unit-level verification (§2).
Reviewer_9UsuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper presents PRISM, a framework to benchmark the capabilities of LLMs of correctly assessing the quality of research papers according to four main axes (Depth of Analysis, Novelty Assessment, Flow Identification and Major Issues, Multi-dimensional Constructiveness), including also humans in the study. Thus, this framework proposes scores for each of these axes based on an LLM-as-a-judge paradigm, where a Gemini model judges whether the raised claims are correct (i.e., whether they are actually novel or already proposed somewhere else; whether statements are supported or not; etc.) This framework is hence evaluated on papers extracted from ICML 2025, NeurIPS2025, and ICLR 2026 (600 papers in total), and results show that single technique can excel in single aspects rather than outperforming other techniques, with only human-produced reviews being generally optimal.

Critical strengths
  • Good step towards understanding the capabilities of automated tools to rigorously assess the quality of reviews, since the community is now flooded with AI-generated content that (most of the time) lacks depth or convincing arguments;
  • Interesting breakdown of reviews in four axis, allowing a more coherent analysis of AI-based peer reviews;
  • Good choice of technologies to benchmark, since they are the most prominent ones.
Critical weaknesses
  • I understand that the authors already point out that the main limitation lays in the LLM-as-a-judge paradigm, but all results are standing on top of a single model, and it is likely that the analysis will vary consistently by using another tool. The impact of this choice can be measured on all metrics of the benchmark, since the LLM-as-a-judge is one variable itself that can be benchmarked as well. The appendix shows that results vary (the order does not seem to be the same), and this is a clear sign that the paper should investigate more this limitation. What would happen if two LLMs used as judges flip indicate two models as best on a specific axes? Which one is the correct assessment, or are they both valid? The validity of the whole methodology stands on top of these questions, but they are not answered.

  • there is a huge reliance on prompts that is not covered in the main paper. The LLM-as-a-judge produces outputs that are vital to the benchmark, but were the results backed by some manual analysis? It is likely that I have missed this point entirely (thus, read the next point directly, which highlight several writing problems), but it does not seem that the manuscript offer some sort of ablation study on the results of the judge in terms of used prompts. What if one of these prompts align the LLM to avoid including specific claims, thus changing the scores? The manuscript should clarify this issue.

  • The main body of the paper feels farfetched, and all the relevant content has been relocated to the Appendix. The paper needs time to be read, since details are not explained well (metrics are very difficult to understand), some portions of the paper are not even covered (prioritization is never deeply explained in the main text; Figure 1 is shown but at that point of the text is not useful; Figure 2 takes a lot of space but a reader can not grasp useful information from it since it is too packed and the text does not cover all the terminology; Figure 5 has 5 possible values which are not explained how they have been computed; it is not clear how hallucinations are detected). In general, the paper has many interesting bits, but interesting things are not really explained (forcing the reader to fill the blanks, and dangerously and erroneously judging the contribution), while its structure disturbs the focus, preventing a deep understanding of the content.

  • a minor weakness of this paper lays in the avoidance of open-source or commercial tools used as naive reviewers, which could be the majority of cases. It would be interesting to see how these models score in the benchmark when used with single prompts with zero interaction.

Questions for authors
  1. Do the authors have a glimpse of possible changes caused by a different choice of the LLM used as a judge (aside from the preliminary results that point out to differences already)?
  2. How hallucinations have been removed? An AI-reviewer has access to a pool of techniques confined to those 600, or they can query the whole internet?
  3. What if a paper in 2025 was ok, but discredited in 2026? Have this been taken into account, since tools can access the whole literature?
  4. How much results change w.r.t. the prompts used for the various axes?
9 comments

We appreciate your time and effort for reviewing our submission. Your key insight and valuable question help us boost our paper's quality. Regarding the questions, we provide the following clarification.

Q1. I understand that the authors already point out that the main limitation lays in the LLM-as-a-judge paradigm, but all results are standing on top of a single model, and it is likely that the analysis will vary consistently by using another tool. […] The appendix shows that results vary (the order does not seem to be the same), and this is a clear sign that the paper should investigate more this limitation. What would happen if two LLMs used as judges flip indicate two models as best on a specific axes? Which one is the correct assessment, or are they both valid?

Do the authors have a glimpse of possible changes caused by a different choice of the LLM used as a judge (aside from the preliminary results that point out to differences already)?

R: Thank you for the constructive question on evaluators. In Appendix, we already provided an analysis with an additiona judge: Xiaomi MiMo V2.5 Pro. Nevertheless, here, we again rerun the PRISM evaluation pipeline, including extraction, verification, and scoring across all four aspects, using two additional judges from different model families: Claude Haiku 5.5 and GPT-5.6-Sol.

The experiment covers 150 papers from ICLR, ICML, and NeurIPS, including human reviews and reviews generated by all five automated reviewer systems. Both judges evaluate the same papers and reviews using the same PRISM methodology. The following table summarizes the results:

MetricJudgeHumanSEAReviewer2TreeReviewDeepReviewCycleReviewer
ConstructivenessGemini.571.504.570.477.629.530
Haiku.517.432.620.512.628.418
GPT.613.531.670.620.716.567
Depth of AnalysisGemini.494.380.381.357.481.485
Haiku.449.265.389.278.426.330
GPT.441.309.449.311.407.376
Critical RecallGemini.308.237.613.304.332.228
Haiku.374.185.425.144.260.159
GPT.316.197.448.127.281.177
Minor RecallGemini.252.232.442.312.228.176
Haiku.284.145.407.239.215.117
GPT.266.133.375.311.252.116
NoveltyGemini.795.841.798.829.771.787
Haiku.649.708.646.618.629.607
GPT.630.673.613.615.611.587
PrioritizationGemini.973.968.977.974.974.977
Haiku.965.948.972.967.949.894
GPT.902.824.896.840.885.783

To assess how judge choice affects PRISM's conclusions, we also compare the rankings of five automated reviewer systems under each judge using Kendall's W, as shown in the table.

DimensionGemini Flash LiteClaude Haiku 5.5GPT-5.6-SolKendall's W
Minor RecallR2 > Tree > SEA > Deep > CycleR2 > Tree > Deep > SEA > CycleR2 > Tree > Deep > SEA > Cycle0.96
Critical RecallR2 > Deep > Tree > SEA > CycleR2 > Deep > SEA > Cycle > TreeR2 > Deep > SEA > Cycle > Tree0.87
ConstructivenessDeep > R2 > Cycle > SEA > TreeDeep > R2 > Tree > SEA > CycleDeep > R2 > Tree > Cycle > SEA0.82
NoveltySEA > Tree > R2 > Cycle > DeepSEA > R2 > Deep > Tree > CycleSEA > Tree > R2 > Deep > Cycle0.80
Depth of AnalysisCycle > Deep > R2 > SEA > TreeDeep > R2 > Cycle > Tree > SEAR2 > Deep > Cycle > Tree > SEA0.78
PrioritizationR2 ≈ Cycle > Tree ≈ Deep > SEAR2 > Tree > Deep > SEA > CycleR2 > Deep > Tree > SEA > Cycle0.61

Findings consistent across judges. The main capability-level findings remain stable across all three judges. Reviewer2 consistently leads in Critical and Minor Flaw Recall (W = 0.87 and 0.96), DeepReview in Constructiveness (W = 0.82), and SEA in Novelty (W = 0.80). Their exact rankings differ in some cases, but these leading systems remain unchanged. These results support PRISM's main conclusion that automated reviewers have distinct strengths across reviewing capabilities, rather than one system consistently outperforming others across all dimensions.

Where do judges disagree, and why? The differences are more apparent in Depth of Analysis and Prioritization.

Depth of Analysis (W = 0.78): Gemini ranks CycleReviewer first, whereas Haiku and GPT rank DeepReview and Reviewer2 first, respectively. The intermediate analysis indicates differences in how judges identify premises and assess their grounding. Gemini's scores are more influenced by premise share, whereas the additional judges distinguish systems more strongly by grounding quality. These differences help explain the changes in rankings. Nevertheless, DeepReview remains among the top two automated systems across all three judges.

Prioritization (W = 0.61): This dimension has the lowest ranking agreement. However, nCPS scores are close to the ceiling (0.97–0.99) for reviews containing valid flaws. Reviews without valid flaws receive zero, and their frequency varies across judges because of differences in flaw-validity judgments. Thus, the ranking variation appears to be influenced by these validity decisions and score aggregation, rather than necessarily reflecting substantial differences in how reviewers prioritize identified flaws.

Overall, the additional judges confirm our main capability-level findings while revealing greater evaluator sensitivity in Depth and Prioritization. We again thank the reviewer for a constructive question and we will include these results in our revision.

Q2. There is a huge reliance on prompts that is not covered in the main paper. The LLM-as-a-judge produces outputs that are vital to the benchmark, but were the results backed by some manual analysis? […] it does not seem that the manuscript offer some sort of ablation study on the results of the judge in terms of used prompts. What if one of these prompts align the LLM to avoid including specific claims, thus changing the scores? How much results change w.r.t. the prompts used for the various axes?

R: PRISM’s conclusions are largely insensitive to prompt wording. Across all four axes, rewording the prompts changes the system rankings mainly among systems that are already statistically tied, and it does not cause the pipeline to drop specific claims.

The prompts are part of the method and are released in full (App. C.3, Fig. 11–20). Their outputs are checked manually against expert re-scoring (§4.3) and in the stage-wise audit by researchers, and their robustness across LLM backends is reported in App. E.7. To isolate the prompt itself, we re-ran all four pipelines on 30 papers × 6 reviewer types under two variants: (P1) paraphrased prompts with identical content, and (P2) definition-only prompts with all examples and anchor phrases removed. Re-running the original prompts on also 30 papers gives the run-to-run noise floor. Because this study uses a newer judge than the paper, we compare agreement and rankings, not absolute levels.

We assess prompt sensitivity at two levels: per-review Spearman correlation (ρ) measures the consistency of individual review-score rankings, while Kendall's τ measures the stability of reviewer-system rankings under different prompts, relative to the original prompts (P0). Higher values indicate greater agreement.

MetricPer-review ρ vs. original (P1 / P2)Noise floorSystem ranking, Kendall τ (P1 / P2)
Critical recall0.700 / 0.7630.6800.867 / 0.867; top kept
Minor recall0.668 / 0.7070.6910.733 / 1.000; top kept
NS0.380 / 0.2250.2350.200 / 0.733; top kept
MCS0.670 / 0.6430.7690.867 / 0.600; top pair (DeepReview, Reviewer2) swaps, not separated under the original prompts
nCPS0.452 / 0.5810.5670.200 / 0.733; near ceiling
DoA0.389 / 0.2530.4870.867 / 0.067 ; no swapped pair separated under the original prompts

Extraction consistency. The units extracted under P1/P2 show substantial overlap with the original extractions, although some differences remain. Flaw micro-flaws match 0.751 / 0.680 (floor 0.683) and constructiveness comments 0.629 / 0.618 (floor 0.626). Novelty claims match 0.770 / 0.733, and their stance changes in only 8% / 7% of aligned cases. These results suggest that prompt variation does not lead to a broad collapse in extraction, although they do not rule out omissions of particular claims.

Where sensitivity remains, it is localised. For Flaw, agreement is at the noise floor and the top system (Reviewer2) is kept. For Novelty, the residual variation comes from retrieval, not scoring: with claims and evidence fixed, changing only the scoring prompt gives ρ 0.410 / 0.357. For Constructiveness, changing only the rubric stays at the re-scoring baseline (ρ 0.716 / 0.736 vs. 0.755). DoA is the most prompt-dependent axis, by design: P2 removes the anchor examples that calibrate the grounding scale, which raises DoA for every system (+0.054 to +0.170). Paraphrasing alone keeps the ranking (τ 0.867). DoA differences between systems are small relative to run-to-run variation, and no swapped pair is separated under the original prompts. Across all metrics on 30 papers, only one rank swap involves a pair that the original prompts separate: Human/Reviewer2 on nCPS under P1. The paper draws no ranking conclusion from nCPS, since all systems score ≈ 0.97 (§4.2.3).

Comments on the paper

2 comments

Dear Reviewers,

We would like to kindly follow up on our responses to your comments. As the discussion period is approaching its end, we would greatly appreciate it if you could take a moment to review our responses and the additional experimental results.

We believe our responses have addressed all your concerns and demonstrate the significance and completeness of our work. Nevertheless, we’d also be happy to promptly respond to any additional concerns before the discussion deadline.

Thank you again for your time and valuable feedback.

Best regards, Authors

Dear Area Chairs and Reviewers,

We sincerely thank all reviewers for their constructive feedback and for engaging with our responses. During the discussion period, we (1) conducted experiments to additionally confirm the robustness and value of PRISM, clarified the PRISM's contributions, and minorly restructured some sections per reviewer's suggestions, and accordingly revised the manuscript. Briefly, we:

  • conducted additional analyses and stage-wise audits to examine extraction accuracy, retrieval quality, verification, and their effects on final scores. These show that Flaw Identification and Constructiveness comparisons remain robust to expert corrections, while also identifying limitations in novelty-claim extraction and literature retrieval.
  • re-ran the evaluation pipeline with two new LLM judges (in addition to 2 judges in the paper) from different model families and conducted prompt-sensitivity experiments. The leading systems remain consistent across judges for Flaw Recall, Constructiveness, and Novelty, while Depth and Prioritization show greater sensitivity. These establish which conclusions are robust and where additional caution is needed.
  • clarified that PRISM’s contribution lies in its structured, evidence-grounded evaluation methodology rather than prompt engineering. as This allows us to assess human and LLM reviews on equal terms, which is fundamentally different from existing review-evaluation benchmarks, which treat human reviews as ground-truth references.
  • further investigated human–judge disagreements, the treatment of flaws identified by only one reviewer, and the effects of intermediate-stage errors. We also added general-purpose LLM reviewer baselines to better contextualize the performance of specialized reviewer systems.
  • finally, streamlined the framework presentation, and placed validation before the system comparisons for better readability, as suggested by one of the reviewer.

Discussion outcomes:

  • We are encouraged by the positive reception from all three reviewers, especially on PRISM's value to the literature, after the discussion.
  • The reviewers acknowledged that our clarifications and additional analyses helped address their concerns and improve their understanding of PRISM’s contributions.
  • Two reviewers characterized our work as incremental prompt engineering. We clarified that our contribution is a structured, well-grounded framework for evaluating human and AI reviews on equal terms, with prompting serving as an implementation tool.
  • All additional experiments and minor revisions are now updated inside the revised paper.

Overall, we believe the revised manuscript presents a clearer view of our work and complete analysis of PRISM, an important tool to allow us to understand the differences between human and LLM reviewers.

We sincerely appreciate the reviewers’ time, constructive engagement, and evaluation of our work.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.