- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies evidence-grounded meta-review generation using both full papers and their associated review discussions. It introduces MetaBench and proposes FACT, a pipeline combining structured issue extraction, planning, and claim-level verification with targeted revision. The paper argues that access to full context alone is insufficient and that explicit grounding and disagreement tracking improve meta-review quality.
Critical strengths
- The paper addresses meaningful failure modes in automated meta-review generation, particularly unsupported factual claims and misrepresentation of reviewer disagreement.
- The experiments include expert assessment, rather than relying exclusively on text-similarity metrics.
Critical weaknesses
- The presentation makes a relatively straightforward idea unnecessarily difficult to follow. However, the paper introduces numerous intermediate constructs and specialized terms before sufficiently explaining their roles and relationships. For example, Section 4 refers to “claim support, false consensus, and false resolution rates” before providing their operational definitions in Section 5.1.
- The dataset’s incremental contribution is insufficiently established. The paper does not sufficiently establish what substantive capability this dataset enables beyond existing review and meta-review resources augmented with paper text.
- The Method's novelty is limited, and the verifier’s reliability remains insufficiently justified. FACT combines familiar ingredients: structured prompting, intermediate planning, evidence retrieval, claim verification, and iterative revision. The contribution therefore appears primarily to be a task-specific integration of established techniques, rather than a substantially new method.
- The paper includes several baselines, but methodological diversity remains limited: many comparisons involve prompting, planning, or refinement variants. Adapted comparisons with established verification-and-revision approaches and conflict-aware meta-review methods would better clarify the distinctive benefit of FACT.
Questions for authors
What does MetaBench enable beyond existing meta-review datasets augmented with publicly available paper text? Can you quantify the value added by its curation and annotations?
Response to Reviewer AQYH
Thank you for the questions about MetaBench’s contribution, verifier reliability, and the choice of baselines.
1. Definitions and presentation
Figure 1 and §4 now introduce the two sources of evidence with an example. The paper can establish what an experiment found. The review thread can establish whether a reviewer accepted the answer. Section 5.1 then defines paper-claim support, false consensus (reporting agreement despite conflicting stances), and false resolution (reporting an open issue as settled).
2. What does MetaBench add to PeerSum plus a current PDF?
The largest observed difference was in attributing a view to a specific reviewer. For those 66 claims, agreement with the reference judgments rose from 70.5% to 85.7%. We tested the same 768 claims on 100 papers drawn from the 11,851 records matched to PeerSum, without requiring membership in the 4,782-paper generation benchmark. Gemini Pro made the reference judgments from evidence available when the meta-review was written (§3.4, Appendix G.2).
The overall gain of 2.2 percentage points (95% paper-bootstrap CI [0.6, 3.8]) weights the two PDF strata by their 403 and 365 claims. The separate unweighted tally was 58 claims correct only with MetaBench and 29 only with PeerSum. Version and attribution information vary together. The comparison measures the two representations as a whole. It does not separately estimate gains from version alignment, attribution, the curation score, or expert annotation.
DeepReview also pairs full papers with review materials. MetaBench preserves the document version, paragraph identifiers, and speaker roles available at the meta-review date. The issue cards are generated from the thread. In the matched comparison, passage location rates were similar. The clearer difference was in supplying valid evidence identifiers (Table 13b).
3. FACT’s contribution and verifier reliability
FACT combines established techniques to check two kinds of claims. It checks paper facts against identifiable passages, and claims about consensus or resolution against reviewer stances in the discussion. Expert passage annotations cover 200 papers. Separate audits assess how accurately the automatic components perform each task (Table 4b).
The lower F1 scores for false consensus and false resolution are why we report those rates as diagnostics. Support and major-issue coverage are the primary outcomes.
4. Baselines and related methods
We compare against checklist-guided drafting, compute-matched self-refinement, and a shuffled-evidence control. Applying the same full gate to the checklist baseline and FACT gives a direct comparison of the drafting workflows. In the 200-paper expert audit, Plan+Gate had higher support and coverage (Table 3a).
Section 2 also relates FACT to RARR-style post-editing and the Cognitive Alignment Framework. Direct comparisons with those methods under matched paper and thread access remain open.
We’d be glad to discuss any remaining questions about the PeerSum comparison or the verifier audits. If the new evidence addresses your concerns, we hope you’ll consider raising your overall rating.