acceptodds
Under review as a conference paper at ICLR 2027

Making Comprehension Explicit: Step-wise Inquiry for Automated Paper Reviewing

Abstract

Peer review requires more than simply producing a critical review; reviewers must question the assumptions of the manuscript, trace its logic, and build a coherent understanding of its claims. However, while existing automated review methods broadly emulate the workflow of human peer review, most do not explicitly model how reviewers progressively build such an understanding before forming their critique. Although this process often manifests in human review as reading annotations, reflection, and fragmented notes, its informal and piecemeal nature does not justify omitting it from automated reviewing. To operationalize this idea, we propose SQuARe, an LLM-based framework that constructs an evidence-grounded understanding of the manuscript through three inquiry phases: sectional inquiry, cross-section inquiry, and comparative novelty assessment. Across rule-based, LLM-as-a-judge, human-grounded review, and human evaluations, SQuARe demonstrates stronger overall performance than capable general-purpose LLMs, specialized review models, and advanced review-agent frameworks. Further analysis suggests that its benefits stem not simply from additional iterative reasoning, but from explicitly organizing and consolidating manuscript evidence into a dedicated representation of understanding. These findings suggest that explicit manuscript understanding, rather than iterative reasoning alone, can serve as a useful intermediate representation for automated paper reviewing.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
automated peer review, NLP for scientific research, scientific paper reviewing, review generation, comprehension-before-generation, staged question answering

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zZHZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SQUARE, an LLM-based framework for automated scientific paper reviewing. Its core idea is to explicitly model manuscript comprehension via a structured question-answering process prior to final review generation, instead of merely relying on implicit reasoning or iterative review refinement. SQUARE comprises three stages: (1) Sectional Inquiry, which incrementally generates and answers questions when reading individual manuscript sections and keeps structured inquiry notes; (2) Cross-section Inquiry, which inspects the full manuscript to spot cross-sectional inconsistencies, missing evidence and logical gaps; and (3) Comparative Novelty Assessment, which retrieves relevant literature and creates questions to assess the novelty of the manuscript. The aggregated inquiry notes are then leveraged to synthesize a structured review.

Critical strengths
  1. The paper addresses an important challenge in automated scientific reviewing: how to encourage LLMs to develop a coherent understanding of a manuscript before producing critical assessments. The idea of explicitly maintaining intermediate representations of manuscript comprehension is intuitively appealing and potentially useful for improving the transparency and reliability of automated review generation.

  2. The paper evaluates the proposed framework using multiple complementary approaches, including numerical rating prediction, LLM-based preference judgments, alignment with human-identified weaknesses, and human evaluation. The authors also conduct component ablations, comparisons of intermediate representation strategies, backbone robustness experiments, and cross-domain evaluations. These experiments provide useful evidence regarding the effectiveness of the proposed framework.

Critical weaknesses
  1. The central contribution of the paper is to explicitly model manuscript comprehension through structured question-answering. However, the paper does not sufficiently explain why QA is particularly suitable for representing manuscript comprehension, or which specific properties of QA contribute to improved reviewing performance.

    • While QA may help organize information, guide attention toward important details, and progressively resolve uncertainties, these advantages are not unique to QA-based representations. Alternative approaches, such as structured summaries, hierarchical evidence notes, knowledge graphs, or hierarchical graphs, could also explicitly represent manuscript content and relationships between claims, assumptions, methods, and experimental evidence.
    • Although Section 4.4 compares SQUARE with Interleaved Synthesis and Interleaved Direct Revision, these experiments primarily demonstrate the benefit of maintaining separate inquiry notes rather than directly updating a review draft. They do not sufficiently establish whether the QA format itself is responsible for the observed improvements.
  2. The functional distinction between derived questions in Sectional Inquiry (Section 3.1) and Cross-section Inquiry (Section 3.2) is not sufficiently clear. In Sectional Inquiry, derived questions are generated based on the current section and previously accumulated inquiry notes. Consequently, these questions may already capture relationships between information appearing in different sections. The paper does not clearly demonstrate what types of questions are uniquely generated during Cross-section Inquiry or why these questions cannot be sufficiently captured through the existing derived-question mechanism.

  3. The proposed framework relies heavily on the quality of generated questions and their corresponding answers. However, the paper provides limited analysis of how the quality of these intermediate QA pairs is ensured or evaluated.

    • First, the intended role of QA is somewhat ambiguous. The paper describes QA as a mechanism for building manuscript comprehension. This raises the question of whether the intermediate QA process primarily serves manuscript understanding or already performs critical review generation.
    • Second, the paper does not sufficiently investigate whether the generated questions are relevant, non-redundant, comprehensive, or effective at uncovering important technical issues. Similarly, it remains unclear how the framework ensures that generated answers accurately reflect manuscript evidence rather than introducing unsupported interpretations.
    • Thrid, the paper sets the maximum number of questions per stage to five. However, the impact of this design choice on reviewing performance is not sufficiently investigated. Since the proposed method relies on progressively accumulating QA pairs, the number of generated questions is an important factor that may directly affect the comprehensiveness of manuscript analysis, computational cost, and quality of the final review. It remains unclear whether the performance improvements are primarily attributable to the proposed inquiry structure or partly to the increased amount of intermediate analysis enabled by additional questions.
  4. The experiments use 200 randomly sampled papers from DeepReview-Test. Although Appendix A provides descriptive statistics about the selected subset, the paper does not sufficiently clarify the original sources of the manuscripts and their associated human reviews. In particular, it would be helpful to specify the conferences or venues from which these papers were collected, their publication or submission years, how the human reviews were obtained, and which manuscript and review versions were used. More importantly, the paper does not adequately discuss the potential risk of data contamination in evaluating LLM-based reviewing systems. If the test manuscripts or their corresponding human reviews were publicly available before the training of the evaluated LLMs, some models may have encountered these materials during pretraining or post-training. This possibility is particularly relevant because the evaluation measures agreement with human review ratings and weaknesses.

  5. The paper employs multiple evaluation methods, which is commendable. However, in the LLM-as-a-judge evaluation, SQUARE is compared only against baseline-generated reviews rather than human-written reviews. While such comparisons demonstrate relative improvements over existing automated methods, they do not establish how closely SQuARe approaches human-level review quality. Although the authors provide a human-grounded evaluation, it primarily measures semantic alignment between identified weaknesses rather than directly comparing overall review quality.

Questions for authors

See weaknesses above.

Reviewer_G8mTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that existing automated reviewing methods leave manuscript comprehension implicit and proposes SQuARe, an LLM-based framework that makes this comprehension explicit through question answering before a review is written. The system first reads the manuscript section by section, with a question generation agent and an answer generation agent building structured inquiry notes along the way. It then revisits the full text to probe cross-section issues and compares the manuscript with retrieved related papers to assess novelty. On 200 papers sampled from the DeepReview dataset, SQuARe achieves rating errors comparable to fine-tuned review models and well below those of its backbone used directly.

Critical strengths

This paper targets a step that prior automated reviewing methods largely skip, namely how a reviewer builds an understanding of the manuscript before writing the critique. Recording this process as accumulated question and answer notes is a simple design that needs no additional training. The same pipeline gives similar gains in rating error when the backbone is switched to DeepSeek-V3.1. The comparison with Interleaved Synthesis in Section 4.4 is the most informative experiment in the paper. And the evaluation goes beyond rating agreement. Reviews are judged by models from three different families and matched against weaknesses raised by human reviewers. The direct baselines are also rerun with the same retrieved papers, which rules out access to related literature as the main source of the gains. The framework remains efficient as well, with a token cost close to that of AI-Scientist and about half that of TreeReview.

Critical weaknesses

W1: The evidence that SQuARe writes more substantive reviews is weak. In Table 3, its SN-F1 of 44.11 is close to that of Qwen3-32B (43.27) and the backbone used directly (43.00), and no confidence intervals are reported. Its precision is lower than that of six baselines, and only DeepReviewer writes longer reviews (Table 9), so the higher recall may partly come from listing more weaknesses. The human study covers 30 papers with one annotator each. The wins of 14 to 8 against Gemini 2.5 Flash and 16 to 11 against Claude Sonnet 4 would not be significant under a sign test, and they are far from the 89.5 and 100 percent win rates given by the Gemini-2.5-Pro judge.

W2: The contribution of the inquiry notes is not fully isolated. SQuARe's rating Spearman correlation of 0.41 is below that of DeepSeek-V3.1 used directly (0.45) and DeepReviewer (0.43). The lower Rating MAE may therefore mostly reflect predictions shifted toward the dataset mean. The synthesis prompts in Figures 17 and 18 also differ from the direct review prompt in Figure 19, and Figure 18 asks the model to align the rating with the severity of the weaknesses it found. In Section 4.4, the interleaved prompts are not provided. Interleaved Direct Revision has a Rating MAE of 2.86, which is worse than the single-pass backbone. The representational claim therefore effectively rests on one variant.

There are also several errors:

In Table 1, the Contribution MAE and MSE of Gemini 2.5 Flash are identical to its Rating values (2.23 and 6.31), which is implausible on a four-point scale.

The Soundness MSE of SQuARe is 0.28 in Table 1 but 0.29 in Tables 10 and 13.

Figure 16 states that the full content of related papers is provided, while Section 3.3 says only abstracts are used.

In Section 3.1, M denotes both the manuscript and the number of initial questions. Typos: "Absract" in Figure 1, and "40,GB" and "24,GB" in Section 4.1.

Questions for authors

See the weakness.

Reviewer_bBHgOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents SQuARe, which is a LLM based paper review framework ( i do not like the term automated peer review). The authors show the performance of their approach with different LLM frameworks and some related datasets.

Critical strengths
  • The paper is well written and well explained
  • The performance reported is better tha other related frameworks
Critical weaknesses
  • The major weakness is that the paper ignores one of the most important study of published this year regarding peer review - Stop Automating Peer Review Without Rigorous Evaluation Baumann et al - ICML 2026 which showed how much overfitting automated review systems have and what are their biggest failure modes, the paper should be judged according to the current state of the research not on older faulty baseliens
  • Novelty: Overall the work does not propose anything new, its an incontext LLM framework in a pipeline, which can be achieved by prompt optimization between couple of LLMs, is the main contribution here the agent prompt?
  • The paper also misses entire discussion around automating peer reviewing (Position: AI Should Verify, Not Judge, Scientific Work Singh et al, AI4Science workshop ICML)
  • the experimental setup in 4.5 is also not very convincing to me, if you are calling the LLMs the assumption is that they have already seen some of the data
Questions for authors
  • Can you extend your study to accomodate papers discussed in weaknesses?
  • What are the implications of the paper, is there a broader impact, is the work eady to be a go to adaptation for Large conference such as ICLR?

Comments on the paper

1 comment

Dear reviewers,

Thank you for your reviews. Please read the authors’ response and follow up in the discussion, briefly indicating whether your main concerns have been addressed and highlighting any remaining questions. Please also update your review and rating where appropriate.

Many thanks for your time and effort.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.