- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PRISM evaluates peer-review quality across four dimensions: analytical depth, novelty assessment, flaw identification and prioritization, and constructiveness. It combines structured argument extraction, literature retrieval, LLM-based verification, and explicit scoring rules. The framework evaluates five automated reviewer systems against human reviews on 600 papers from three machine-learning conferences.
Critical strengths
The evaluation spans multiple systems and venues. Detailed prompts, worked examples, human annotations, and an alternative-judge experiment make the framework relatively transparent.
Critical weaknesses
- Limited novelty. The framework largely combines heuristic components, and its contribution beyond existing reviewer evaluation methods remains unclear.
- Unclear practical insights. The paper does not establish how its findings can guide concrete improvements to reviewer systems.
- Insufficient validation. Comparisons with existing benchmarks and evaluators are limited. The human validation does not sufficiently establish that the proposed metrics capture review quality.
Questions for authors
- What is PRISM’s main contribution beyond existing reviewer evaluation approaches?
- Which key insights suggest specific, testable improvements to reviewer systems?
- How does PRISM compare with existing evaluators against independent expert judgments, and what evidence demonstrates its reliability?
7 comments
What this design reveals. These capabilities are measured separately and are only weakly correlated (max |r| = 0.193; App. D), so PRISM exposes findings that a single score would hide. Some LLM reviewers exceed individual human reviewers in Critical-flaw recall (§4.2.3), yet only human review scores differ between accepted and rejected papers (§4.2.5). This is exactly the distinction the domain needs in order to decide where LLMs can assist reviewers and where they cannot. The reliability of each component is validated against experts and across judges and prompts.
In the revision, we will restructure §2 around these requirements and include the table above.
[1] Chang et al. (2025). TreeReview: A Dynamic Tree of Questions Framework for Deep and Efficient LLM-based Scientific Peer Review. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), pp. 15651–15682. https://aclanthology.org/2025.emnlp-main.790/
[2] Sadallah et al. (2025). The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authors. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), pp. 28991–29021. https://aclanthology.org/2025.emnlp-main.1476/
[3] Zheng et al. (2023). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems 36 (NeurIPS 2023), Datasets and Benchmarks Track. https://proceedings.neurips.cc/paper_files/paper/2023/hash/91f18a1287b398d378ef22505bf41832-Abstract-Datasets_and_Benchmarks.html
[4] Saito et al. (2023). Verbosity Bias in Preference Labeling by Large Language Models. Workshop on Instruction Tuning and Instruction Following at NeurIPS 2023. https://mlanthology.org/neuripsw/2023/saito2023neuripsw-verbosity/
[5] Panickssery et al. (2024). LLM Evaluators Recognize and Favor Their Own Generations. Advances in Neural Information Processing Systems 37 (NeurIPS 2024), Main Conference Track. https://proceedings.neurips.cc/paper_files/paper/2024/hash/7f1f0218e45f5414c79c0679633e47bc-Abstract-Conference.html
[6] Du et al. (2024). LLMs Assist NLP Researchers: Critique Paper (Meta-)Reviewing. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024), pp. 5081–5099. https://aclanthology.org/2024.emnlp-main.292/
[7] Garg et al. (2025). ReviewEval: An Evaluation Framework for AI-Generated Reviews. Findings of the Association for Computational Linguistics: EMNLP 2025, pp. 20542–20564. https://aclanthology.org/2025.findings-emnlp.1120/
[8] Deng et al. (2026). CoCoReviewBench: A Completeness- and Correctness-Oriented Benchmark for AI Reviewers. Proceedings of the 43rd International Conference on Machine Learning (ICML 2026), Proceedings of Machine Learning Research (PMLR), Vol. 306, pp. 23997–24057. https://proceedings.mlr.press/v306/deng26j.html
[9] Ebrahimi et al. (2025). RottenReviews: Benchmarking Review Quality with Human and LLM-Based Judgments. Proceedings of the 34th ACM International Conference on Information and Knowledge Management (CIKM 2025), pp. 5642–5649. https://doi.org/10.1145/3746252.3761506
[10] Cortes & Lawrence (2021). Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment. arXiv:2109.09774. https://arxiv.org/abs/2109.09774
[11] Morgan (1985). Originality, Novelty and Priority: Three Words to Reckon with in Scientific Publishing. Canadian Medical Association Journal, 132(1), pp. 8–9. https://pmc.ncbi.nlm.nih.gov/articles/PMC1346489/
[12] Mishra & Torvik (2016). Quantifying Conceptual Novelty in the Biomedical Literature. D-Lib Magazine, 22(9/10). https://doi.org/10.1045/september2016-mishra
[13] Järvelin & Kekäläinen (2002). Cumulated Gain-based Evaluation of IR Techniques. ACM Transactions on Information Systems, 20(4), pp. 422–446. https://doi.org/10.1145/582415.582418
[14] Kennard et al. (2022). DISAPERE: A Dataset for Discourse Structure in Peer Review Discussions. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2022), pp. 1234–1249. https://aclanthology.org/2022.naacl-main.89/
[15] Hua et al. (2019). Argument Mining for Understanding Peer Reviews. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT 2019), pp. 2131–2137. https://aclanthology.org/N19-1219/
Q2. Unclear practical insights. The paper does not establish how its findings can guide concrete improvements to reviewer systems. Which key insights suggest specific, testable improvements to reviewer systems?
R: A key insight of PRISM is that submitted human reviews and automated reviews have complementary capability profiles. This suggests concrete, testable directions for human–AI reviewing workflows rather than assuming that the same system should handle every review task:
(1) LLMs as flaw scanners, humans as verifiers. In the original pooled-reference analysis, some automated systems have higher critical-flaw recall than individual submitted human reviews (§4.2.3), but they hallucinate ~3.6–16.2% of their flaws, with those identified hallucinations categorized as minor (Fig. 6). Improvement: LLMs produce a candidate flaw list and the human reviewer validates and prioritizes it. Test: flaw recall and hallucination rate of the combined review vs. the human review alone.
(2) LLMs supply fixes, humans supply depth. Human reviews are strong on grounded, methodology-focused argumentation (§4.2.1) but rarely propose solutions (D4 = 0.427). A purpose-built LLM pipeline does so far more often (D4 = 0.781) and has the highest constructiveness score (0.636 vs. 0.556 for humans) (§4.2.4). Improvement: use LLMs to turn human-identified weaknesses into actionable suggestions. Test: D1/D4 with DoA preserved.
(3) Novelty and merit judgments should stay human-led. LLM systems tend to endorse author novelty claims (up to 83% “Novel” vs. 62% for humans; Fig. 4). More importantly, only human review scores differ significantly between accepted and rejected papers; the automated-review scores do not show the same statistically significant accepted-versus-rejected separation (§4.2.5). Improvement: LLMs supply retrieved prior work and counter-evidence, while a human reviewer retains responsibility for the novelty verdict and overall recommendation. Test: stance distribution and the accept-vs-reject comparison for the combined review.
For fully automated systems, the same findings identify the gaps to close: claim verification, counter-evidence search for novelty, and sensitivity to paper quality. Because PRISM measures each dimension separately, these proposals yield specific, measurable hypotheses; they are suggested follow-up experiments, not interventions already validated in this study.
We appreciate your time and effort for reviewing our submission. Your key insight and valuable question help us boost our paper's quality. Regarding the questions, we provide the following clarification:
R: PRISM’s main methodological contribution is a unified, evidence-grounded framework that evaluates human and LLM reviews on equal terms, without assuming human reviews are the ground truth. Rather than measuring how closely an LLM review resembles a human review, PRISM evaluates what scientific reviews are expected to achieve: identifying valid and critical flaws, making literature-grounded novelty judgments, prioritizing major issues, providing actionable feedback, and supporting arguments with evidence. These capabilities are assessed separately through structured review units and dedicated evaluation pipelines. The components are grounded in established constructs from the peer-review literature, but their integration enables capability-level comparisons between human and automated reviewers under the same protocol.
Why this matters for the domain. Peer review gates what enters the scientific record, yet human reviews are themselves incomplete and sometimes wrong [8], and reviewing outcomes are noticeably inconsistent [10]. As LLM reviewers enter this process, the central question is no longer whether a review resembles a human review, but whether its criticisms are correct and important. An evaluator that rewards similarity to human reviews inherits their omissions and errors. An evaluator that gives one holistic score cannot show which statement is wrong. Overlap metrics barely change when a review is paired with references from a different paper (a BERTScore drop of 1.8–2.6 points [1, App. C.1, Table 7]). Holistic LLM judges inherit position, verbosity and self-preference biases [3–5], frontier LLMs reach only about 20 F1 when asked to locate deficient review segments [6], and holistic LLM judges align poorly with expert ratings of review quality [9].
We compare existing approaches with PRISM on the same requirements: