- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SQUARE, an LLM-based framework for automated scientific paper reviewing. Its core idea is to explicitly model manuscript comprehension via a structured question-answering process prior to final review generation, instead of merely relying on implicit reasoning or iterative review refinement. SQUARE comprises three stages: (1) Sectional Inquiry, which incrementally generates and answers questions when reading individual manuscript sections and keeps structured inquiry notes; (2) Cross-section Inquiry, which inspects the full manuscript to spot cross-sectional inconsistencies, missing evidence and logical gaps; and (3) Comparative Novelty Assessment, which retrieves relevant literature and creates questions to assess the novelty of the manuscript. The aggregated inquiry notes are then leveraged to synthesize a structured review.
Critical strengths
-
The paper addresses an important challenge in automated scientific reviewing: how to encourage LLMs to develop a coherent understanding of a manuscript before producing critical assessments. The idea of explicitly maintaining intermediate representations of manuscript comprehension is intuitively appealing and potentially useful for improving the transparency and reliability of automated review generation.
-
The paper evaluates the proposed framework using multiple complementary approaches, including numerical rating prediction, LLM-based preference judgments, alignment with human-identified weaknesses, and human evaluation. The authors also conduct component ablations, comparisons of intermediate representation strategies, backbone robustness experiments, and cross-domain evaluations. These experiments provide useful evidence regarding the effectiveness of the proposed framework.
Critical weaknesses
-
The central contribution of the paper is to explicitly model manuscript comprehension through structured question-answering. However, the paper does not sufficiently explain why QA is particularly suitable for representing manuscript comprehension, or which specific properties of QA contribute to improved reviewing performance.
- While QA may help organize information, guide attention toward important details, and progressively resolve uncertainties, these advantages are not unique to QA-based representations. Alternative approaches, such as structured summaries, hierarchical evidence notes, knowledge graphs, or hierarchical graphs, could also explicitly represent manuscript content and relationships between claims, assumptions, methods, and experimental evidence.
- Although Section 4.4 compares SQUARE with Interleaved Synthesis and Interleaved Direct Revision, these experiments primarily demonstrate the benefit of maintaining separate inquiry notes rather than directly updating a review draft. They do not sufficiently establish whether the QA format itself is responsible for the observed improvements.
-
The functional distinction between derived questions in Sectional Inquiry (Section 3.1) and Cross-section Inquiry (Section 3.2) is not sufficiently clear. In Sectional Inquiry, derived questions are generated based on the current section and previously accumulated inquiry notes. Consequently, these questions may already capture relationships between information appearing in different sections. The paper does not clearly demonstrate what types of questions are uniquely generated during Cross-section Inquiry or why these questions cannot be sufficiently captured through the existing derived-question mechanism.
-
The proposed framework relies heavily on the quality of generated questions and their corresponding answers. However, the paper provides limited analysis of how the quality of these intermediate QA pairs is ensured or evaluated.
- First, the intended role of QA is somewhat ambiguous. The paper describes QA as a mechanism for building manuscript comprehension. This raises the question of whether the intermediate QA process primarily serves manuscript understanding or already performs critical review generation.
- Second, the paper does not sufficiently investigate whether the generated questions are relevant, non-redundant, comprehensive, or effective at uncovering important technical issues. Similarly, it remains unclear how the framework ensures that generated answers accurately reflect manuscript evidence rather than introducing unsupported interpretations.
- Thrid, the paper sets the maximum number of questions per stage to five. However, the impact of this design choice on reviewing performance is not sufficiently investigated. Since the proposed method relies on progressively accumulating QA pairs, the number of generated questions is an important factor that may directly affect the comprehensiveness of manuscript analysis, computational cost, and quality of the final review. It remains unclear whether the performance improvements are primarily attributable to the proposed inquiry structure or partly to the increased amount of intermediate analysis enabled by additional questions.
-
The experiments use 200 randomly sampled papers from DeepReview-Test. Although Appendix A provides descriptive statistics about the selected subset, the paper does not sufficiently clarify the original sources of the manuscripts and their associated human reviews. In particular, it would be helpful to specify the conferences or venues from which these papers were collected, their publication or submission years, how the human reviews were obtained, and which manuscript and review versions were used. More importantly, the paper does not adequately discuss the potential risk of data contamination in evaluating LLM-based reviewing systems. If the test manuscripts or their corresponding human reviews were publicly available before the training of the evaluated LLMs, some models may have encountered these materials during pretraining or post-training. This possibility is particularly relevant because the evaluation measures agreement with human review ratings and weaknesses.
-
The paper employs multiple evaluation methods, which is commendable. However, in the LLM-as-a-judge evaluation, SQUARE is compared only against baseline-generated reviews rather than human-written reviews. While such comparisons demonstrate relative improvements over existing automated methods, they do not establish how closely SQuARe approaches human-level review quality. Although the authors provide a human-grounded evaluation, it primarily measures semantic alignment between identified weaknesses rather than directly comparing overall review quality.
Questions for authors
See weaknesses above.
Dear reviewers,
Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.
Many thanks for your time and effort.