acceptodds
Under review as a conference paper at ICLR 2027

Full Context Is Not Enough: Evidence-Grounded Meta-Review Generation

Abstract

As some major research venues begin to evaluate tightly bounded AI assistance while retaining strict limits on delegated reviewing, reliable evaluation of meta-review support is becoming increasingly consequential. Meta-reviewing must reconcile paper evidence, reviews, rebuttals, and unresolved disagreement into a defensible decision rationale; fluent automation can instead introduce unsupported paper claims or fictitious reviewer consensus. Existing benchmarks often expose only partial paper context and rely on reference-overlap metrics, leaving these failures difficult to measure. We introduce MetaBench, a 4,782-instance benchmark pairing full papers with review threads and human meta-reviews, and FACT, a pipeline that separates issue-conditioned planning from a return-to-source evidence gate. On a same-drafter 2×2 comparison over 1,520 cases, adding the evidence gate to Direct raises supported paper claims from 73.8% to 84.7%, whereas planning alone reaches 76.0%; Plan+Gate reaches 88.9% support and 80.6% major-issue coverage. On a 200-paper expert-gold slice, the same ordering holds, while claim extraction attains .79 precision, .82 recall, and .80 F1. A 120-case blinded study is descriptively favorable against the strongest gated baseline, although we make no confirmatory inferential claim. Across these evaluations, verification is associated with the largest support change, while planning adds most clearly to issue coverage and reductions in conflict errors when paired with the gate. Within the Direct stack, longer paper context produces smaller observed gains than explicit verification.

Primary area:
datasets and benchmarks
Keywords:
meta-review generation, agentic prompting, evidence anchoring, full-paper benchmark, review summarization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AQYHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies evidence-grounded meta-review generation using both full papers and their associated review discussions. It introduces MetaBench and proposes FACT, a pipeline combining structured issue extraction, planning, and claim-level verification with targeted revision. The paper argues that access to full context alone is insufficient and that explicit grounding and disagreement tracking improve meta-review quality.

Critical strengths
  1. The paper addresses meaningful failure modes in automated meta-review generation, particularly unsupported factual claims and misrepresentation of reviewer disagreement.
  2. The experiments include expert assessment, rather than relying exclusively on text-similarity metrics.
Critical weaknesses
  1. The presentation makes a relatively straightforward idea unnecessarily difficult to follow. However, the paper introduces numerous intermediate constructs and specialized terms before sufficiently explaining their roles and relationships. For example, Section 4 refers to “claim support, false consensus, and false resolution rates” before providing their operational definitions in Section 5.1.
  2. The dataset’s incremental contribution is insufficiently established. The paper does not sufficiently establish what substantive capability this dataset enables beyond existing review and meta-review resources augmented with paper text.
  3. The Method's novelty is limited, and the verifier’s reliability remains insufficiently justified. FACT combines familiar ingredients: structured prompting, intermediate planning, evidence retrieval, claim verification, and iterative revision. The contribution therefore appears primarily to be a task-specific integration of established techniques, rather than a substantially new method.
  4. The paper includes several baselines, but methodological diversity remains limited: many comparisons involve prompting, planning, or refinement variants. Adapted comparisons with established verification-and-revision approaches and conflict-aware meta-review methods would better clarify the distinctive benefit of FACT.
Questions for authors

What does MetaBench enable beyond existing meta-review datasets augmented with publicly available paper text? Can you quantify the value added by its curation and annotations?

1 comment
AuthorsOpenReview ↗

Response to Reviewer AQYH

Thank you for the questions about MetaBench’s contribution, verifier reliability, and the choice of baselines.

1. Definitions and presentation

Figure 1 and §4 now introduce the two sources of evidence with an example. The paper can establish what an experiment found. The review thread can establish whether a reviewer accepted the answer. Section 5.1 then defines paper-claim support, false consensus (reporting agreement despite conflicting stances), and false resolution (reporting an open issue as settled).

2. What does MetaBench add to PeerSum plus a current PDF?

The largest observed difference was in attributing a view to a specific reviewer. For those 66 claims, agreement with the reference judgments rose from 70.5% to 85.7%. We tested the same 768 claims on 100 papers drawn from the 11,851 records matched to PeerSum, without requiring membership in the 4,782-paper generation benchmark. Gemini Pro made the reference judgments from evidence available when the meta-review was written (§3.4, Appendix G.2).

Claim setClaimsPeerSum + current PDFMetaBench at meta-review dateDifference (points)
Overall, weighted by PDF stratum76880.8%83.0%+2.2
Identical PDF versions (48 papers)40382.5%82.5%0.0
Different PDF versions (52 papers)36578.9%83.6%+4.7
Paper content32690.8%91.2%+0.4
Participants’ views31473.8%79.1%+5.3
Discussion outcome12872.0%73.5%+1.5
Claims naming a reviewer (participant-view subset)6670.5%85.7%+15.2

The overall gain of 2.2 percentage points (95% paper-bootstrap CI [0.6, 3.8]) weights the two PDF strata by their 403 and 365 claims. The separate unweighted tally was 58 claims correct only with MetaBench and 29 only with PeerSum. Version and attribution information vary together. The comparison measures the two representations as a whole. It does not separately estimate gains from version alignment, attribution, the curation score, or expert annotation.

DeepReview also pairs full papers with review materials. MetaBench preserves the document version, paragraph identifiers, and speaker roles available at the meta-review date. The issue cards are generated from the thread. In the matched comparison, passage location rates were similar. The clearer difference was in supplying valid evidence identifiers (Table 13b).

Evidence checkPeerSum + current PDFMetaBench
All cited passages located622/640 (97.2%)648/668 (97.0%)
Evidence IDs given and all valid412/518 (79.5%)634/662 (95.8%)
False reference label judged supported (count)42

3. FACT’s contribution and verifier reliability

FACT combines established techniques to check two kinds of claims. It checks paper facts against identifiable passages, and claims about consensus or resolution against reviewer stances in the discussion. Expert passage annotations cover 200 papers. Separate audits assess how accurately the automatic components perform each task (Table 4b).

Audit targetUnitsPrecisionRecallF1
Claim extraction2,400.790.820.805
Unsupported claims220.550.805.653
False consensus110.476.714.571
False resolution110.467.778.583

The lower F1 scores for false consensus and false resolution are why we report those rates as diagnostics. Support and major-issue coverage are the primary outcomes.

4. Baselines and related methods

We compare against checklist-guided drafting, compute-matched self-refinement, and a shuffled-evidence control. Applying the same full gate to the checklist baseline and FACT gives a direct comparison of the drafting workflows. In the 200-paper expert audit, Plan+Gate had higher support and coverage (Table 3a).

200-paper expert auditPaper-claim supportMajor-issue coverage
Checklist-guided drafting + full gate83.8%72.3%
Plan+Gate86.4%78.1%

Section 2 also relates FACT to RARR-style post-editing and the Cognitive Alignment Framework. Direct comparisons with those methods under matched paper and thread access remain open.

We’d be glad to discuss any remaining questions about the PeerSum comparison or the verifier audits. If the new evidence addresses your concerns, we hope you’ll consider raising your overall rating.

Reviewer_dGJTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the challenge of factual faithfulness and disagreement coverage in Meta-Review Auto-Generation (MRAG). The authors argue that existing benchmarks often rely only on abstracts and superficial reference-based overlap metrics (e.g., ROUGE), leaving models prone to hallucinating unsupported paper-specific claims or ignoring unresolved reviewer–author disputes. To tackle these issues, the paper introduces:

  1. MetaBench: A benchmark of 4,782 instances pairing full-text papers (with captions) with complete review threads and human meta-reviews.
  2. FACT: A fact-check-centered prompting pipeline that separates issue-conditioned planning from a "return-to-source" verification gate. The authors demonstrate that the verification gate primarily improves claim support (up to 88.9%), while the planning module significantly enhances major-issue coverage (80.6%) and mitigates conflict misattributions.
Critical strengths
  1. The factorial evaluation on 1,520 cases clearly isolates the roles of key components: return-to-source evidence verification drives the primary gain in factual claim support (+10.9 pp), while issue-conditioned planning drives deliberative coverage (reaching 80.6%) and reduces conflict errors. This provides convincing empirical insight into MRAG architectures beyond simple monolithic prompting.
  2. MetaBench decisively moves MRAG beyond abstract-only setups by pairing 4,782 full papers with complete review threads. The findings are strongly substantiated through a 200-paper expert-gold audit (.80 claim-extraction F1) and blinded human comparisons, establishing a reliable, evidence-grounded evaluation framework for peer-review assistance.
Critical weaknesses
  1. Limited Algorithmic Novelty as an Engineering-Heavy Workflow:The proposed FACT framework is fundamentally a multi-stage prompt orchestration and verification pipeline rather than a novel architectural advance. While the empirical ablation is thorough, the methodological contribution consists largely of structured engineering around off-the-shelf LLMs. This is consequential to my evaluation because it limits the theoretical and foundational ML insights offered to the ICLR community.

  2. Incomplete Latency, Cost, and Scalability Profiling for Deployment: FACT heavily inflates computational requirements compared to direct generation, averaging 4.2 LLM calls and 37.2k input tokens per submission. Despite this multi-pass design, the main paper defers concrete end-to-end latency and serving cost benchmarks entirely to the appendix, while relying on a shared judge family that introduces potential self-alignment bias. This weakens my overall recommendation because feasibility, turnaround time, and token economics are decisive bottlenecks for adoption in high-volume, real-world conference review systems.

Questions for authors
  1. Inference Latency, Serving Costs, and Throughput (Clarification / Assessment-Critical): Figure 4(b) indicates that FACT (Plan+Gate) requires an average of 4.2 LLM calls and 37.2k input tokens per submission, yet wall-clock latency and serving dollar costs are deferred to the appendix. Could the authors report concrete end-to-end latency benchmarks and estimated monetary costs per paper across the evaluated backbones (e.g., GPT-5-mini, Claude-Sonnet-4.5)? Knowing the real-world operational turnaround time and cost per paper is critical to assessing whether FACT is practically deployable in high-volume conference workflows.

  2. Mitigating Evaluator Self-Alignment Bias (Clarification / Assessment-Critical): Section 8 notes that the pipeline and the automated evaluator share the same model family for gating and claim-support scoring, which may artificially inflate support numbers through self-alignment. Was cross-family verification evaluated (e.g., a Claude-based gate/evaluator scoring a GPT-generated draft, or vice versa)? If so, does the +10.9 pp support gain attributable to the gate hold across disjoint evaluator families?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dGJT

Thank you for the thoughtful review. We’ve moved the timing and cost measurements into the main paper and asked a model from a different family to check the same drafts against their sources.

1. Latency, cost, and throughput

For GPT-5-mini, adding verification to Direct increased mean latency from 42 to 96 seconds and cost from USD 0.11 to USD 0.24 per paper. Adding planning to that workflow took another 12 seconds and USD 0.03. Section 5.3 now reports these measurements alongside the main results.

SystemCallsInput tokensMeasured latencyCost (USD)
Direct1.028.4k42 s0.11
Direct+Gate3.635.8k96 s0.24
Plan+Gate4.237.2k108 s0.27

These measurements cover the primary GPT-5-mini runs. The study did not measure matched latency and cost for Claude or the other backbones, or throughput under concurrent load.

2. Does the support gain hold across evaluator families?

Yes, with a smaller estimated gain. On the same 100 papers, the gate improved support by 10.2 points under the GPT-4.1-aligned evaluator and by 5.8 under Gemini 3.8 Flash. Direct+Gate verified each paper’s Direct draft. Both evaluators scored the same frozen texts with identifiers removed, and Gemini extracted and checked its own claims (§5.2, Appendix G.1).

Evaluator on the same 100 papersDirect supportDirect+Gate supportPaired gain in points (95% CI)
E1, GPT-4.1-aligned75.1%85.3%+10.2 [6.8, 13.6]
E2, Gemini 3.8 Flash83.2%89.0%+5.8 [2.8, 8.8]

Both intervals use a paired bootstrap over papers. The 4.4-point difference measures sensitivity to evaluator choice. Extraction and judgment both changed, so the comparison cannot isolate the effect of a shared model family. The separate 200-paper expert audit did not include Direct+Gate.

3. What does the workflow comparison establish?

FACT builds on existing models. The original 1,520-paper experiment tests planning and verification with the same writer and inputs. Adding verification to direct drafting raised paper-claim support by 10.9 points. With verification already in place, adding an issue plan raised major-issue coverage by 8.8 points (Table 1).

Original 1,520-paper comparisonPaper-claim supportMajor-issue coverage
Direct73.8%68.4%
Plan76.0%70.9%
Direct+Gate84.7%71.8%
Plan+Gate88.9%80.6%

We hope the timing measurements and independent scores make it easier to judge when the extra verification pass is worthwhile. We’d be happy to clarify either comparison.

Reviewer_BKuMOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies meta-review auto-generation (MRAG): automatically generating meta-reviews given the paper, review-rebuttal thread, and reviewer scores. The authors collect MetaBench, a benchmark sourced from OpenReview for the MRAG task, and propose FACT, a prompting workflow for MRAG in a planning-and-verify manner. The FACT workflow emphasizes evidence-grounding, steering the LLM to ground each paper-related claim in the meta-review to the paper fact/content, and each strength/weakness/decision rationale to the review -rebuttal thread issues. Experiments with human and automatic evaluation show that FACT attains better evidence grounding with higher human preference.

Critical strengths
  1. This paper addresses a timely research question regarding how to improve the groundedness and quality of AI-generated meta-reviews.
  2. The compiled MetaBench is a potentially useful testbed for meta-review generation.
Critical weaknesses
  1. The presentation is confusing and hard to follow. Many terms appear without proper context or explanation. For example, in lines 21- 22, there's no explanation of what Direct/Gate/Plan refers to, making it hard to assess what and how the displayed results support the contribution of the FACT pipeline. And on lines 241-242, there is a lack of context to understand what "without re-verification against a second gate pass" is trying to convey. Moreover, in Sec 5.1, it seems to me that it delves deep into the discussion of a subset of metrics too early, without first properly outlining what the set of metrics is and what each of them is trying to measure; are they human or automatic metrics, etc. And these necessary elaborations are also missing in each table/figures caption. Overall, it takes a hard time to infer the necessary information from the writing to properly understand the setup and evaluation, as well as the true contributions.

  2. The design choices of the FACT pipeline are partially unsupported/unexplained. In particular, on line 212: "Decision/Rationale from unresolved issues and score signals" - it seems to me that the decision and rationale should be grounded in the full content of the meta-review above them, to form an overall flow of the judgement and defensible rationale, not merely addressing the unresolved issues. Also, lines 230- 231: "Paper-span retrieval uses hybrid lexical/dense retrieval and re-ranking" - LLM tested in your setup should be good enough for lexical/semantic retrieval of evidence; what are the reasons/supporting evidence of using additional lexical/dense retrieval and reranking?

  3. Questionable baseline: The paper claims the superiority of FACT over baselines, especially direct prompting. However, judging from what the direct prompt shows in the appendix, I am skeptical about the comparison, as there is no instruction/steering in the direct prompt on any aspect of "you should ground each of your claim/statement to paper facts/review threads". As such, it confounds the comparison of whether the performance gain should be attributed to the structural/multi-phase design of the FACT workflow, or just the instruction to LLMs to do rigorous grounding. LLMs with proper instruction of grounding might be able to do the FACT-style evidence retrieval and verification with their reasoning capabilities.

Questions for authors

Please see the weaknesses above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer BKuM

Thank you for the feedback on presentation and experimental design. We revised the figures and captions and tested the stronger single-call baseline you suggested.

1. Drafting methods, metrics, and scoring stages

The main Plan+Gate result is scored after verification and before optional style editing. The “second pass” checks the edited draft. The abstract now introduces the three operations before reporting scores. Direct drafts from the paper and review thread, Plan first outlines the major issues, and Gate checks either draft against its sources. Figure 1 marks the scoring stages.

Section 5.1 defines support as the fraction of extracted paper claims backed by passages and coverage as the fraction of a fixed list of major review issues the draft addresses. It then defines false consensus and false resolution. Table 1 labels the automatic measures. In Fig. 3, each line compares verification on and off at a fixed planning setting. The rows below also show how polishing and rechecking change the scores (Tables 1 and 4a).

Draft scored on the same 1,520 papersSupportCoverage
Direct73.8%68.4%
Plan76.0%70.9%
Direct+Gate84.7%71.8%
Plan+Gate immediately after verification (main result)88.9%80.6%
Plan+Gate after optional style editing87.4%80.1%
Edited Plan+Gate draft checked again (second pass)88.5%80.4%
Shuffled-evidence gate (control)70.8%66.5%

2. Decision rationale and passage retrieval

Decision rationale. The rationale draws on the same strengths, weaknesses, and disagreements as the preceding sections. FACT drafts all four sections from a single plan built from the paper and all review issues. Outstanding concerns receive particular attention, with ratings as context. If the draft says an objection was resolved, the gate checks that statement against the reviewer’s recorded position (§4.2–4.3).

Table 3 distinguishes expert scores from blinded preferences. Experts scored support and coverage against annotated evidence on 200 papers. Separately, three blinded readers per paper assessed rationale quality on 120 high-disagreement papers. Of the 360 judgments comparing Plan+Gate with Direct+Gate, 211 favored the planned rationale (Table 9).

Expert audit on 200 papersSupportCoverage
Checklist-guided drafting + full gate83.8%72.3%
Plan+Gate86.4%78.1%
Rationale judged on 120 high-disagreement papersPrefer Plan+GateTiePrefer comparator
vs. Direct+Gate21110148
vs. checklist-guided drafting + full gate20510055

Passage retrieval. For a paper claim that fails verification, lexical and semantic search find candidates. Reranking returns up to five passages with paragraph IDs so the writer and reader can check the same evidence. On the 1,520 papers above, support was 84.7% for Direct+Gate and 70.8% with shuffled evidence. This tests the value of matching a claim to the right passage. A head-to-head comparison with an LLM selecting passages directly remains open.

3. Does verification still help with a stronger prompt?

Yes. The explicit one-call instruction improved support by 2.6 percentage points over Direct. Checking and revising the finished draft added a further 3.2 points over that stronger prompt (95% paired paper-bootstrap CI [0.5, 5.9]). The coverage difference between the latter two methods was 0.3 points.

On the same 100 papers, Direct asked the writer to avoid unsupported claims. Direct+Check explicitly asked it to check its claims in that call. Direct+Gate started from that paper’s Direct draft and revised claims that failed verification. Gemini 3.8 Flash independently extracted and checked claims from the frozen outputs with identifiers removed (§5.2, Table 12).

MethodMean calls/paperSupportCoverage
Direct1.083.2%70.2%
Direct+Check1.085.8%71.5%
Direct+Gate3.689.0%71.8%

Please let us know if anything needs clarification. If the new comparison and revisions address your concerns, we hope you’ll consider raising your overall rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and revision overview

We thank the three reviewers for their careful reading and thoughtful questions. We apologize for our late response amid overlapping submission deadlines.

Your comments helped us put a practical question at the center of the revision. When a meta-review says that an experiment resolved a reviewer’s concern, what evidence supports that sentence? The paper can show what the experiment found. The discussion can show whether the reviewer accepted the answer. Figure 1 now traces these two checks, and Figure 2 shows how we selected the papers and review threads for MetaBench.

The new comparison with PeerSum plus the current PDF shows where the choice of evidence mattered most. The largest gain in agreement with Gemini Pro’s judgments was on claims that attributed a view to a specific reviewer (§3.4).

The results for FACT also clarify why it helps to plan before drafting and verify afterward. In the original experiment, planning helped the writer cover more of the issues reviewers raised (Fig. 3). In the new cross-family evaluation, checking a finished draft against its sources raised the share of supported paper claims by 3.2 percentage points over a single prompt that already asked the writer to check its claims (§5.2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.