acceptodds
Withdrawn from ICLR 2027

Beyond Aesthetics: LogicBench for Paper-to-Poster Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent multimodal models have made it increasingly feasible to generate scientific posters directly from research papers. Existing evaluation largely emphasizes visual quality or isolated fact recovery, leaving unclear whether a poster preserves the scientific logic that connects evidence to conclusions. We introduce LogicBench, a benchmark for measuring logical path coverage in generated posters with source-grounded multi-hop questions derived from validated scientific logic graphs. Questions are verified against the source paper and calibrated with reference posters before held-out poster scoring, so the benchmark measures what the poster makes recoverable. On 134 NIPS25 papers with 1,354 questions, human posters achieve 98.3% accuracy, while five poster-generation methods score 24.4–61.8%. Although these methods achieve high scores on PaperQuiz and P2PEval, which emphasize fact-level correctness, their lower LogicBench scores show that factual recovery does not ensure preserved reasoning. Our analysis shows that accuracy reaches only 66.4% even when all facts needed for a question are present on the poster, suggesting that the logical connections between these facts are not preserved. Moreover, among the evaluated methods, aesthetic scores increase over time while LogicBench accuracy decreases. These findings establish logical path coverage as a distinct dimension of poster quality and motivate methods that preserve scientific logic alongside visual quality.

Primary area:
datasets and benchmarks
Keywords:
paper-to-poster generation, academic poster evaluation, scientific QA benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_M71YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

this paper focuses on the evaluation of the paper-to-poster from a logical path coverage perspective.

the experiments on 134 NeurIPS papers, and the findings suggest that logical connections should be evaluated separately from factual recovery.

Critical strengths

the logic bench treat each paper as a logic graph, and frame the poster evaluation as the assessment of logical path coverage. despite this might be novel but it still incremental and it cannot support a ICLR submission.

Critical weaknesses
  1. the contribution of this paper is incremental. This paper do not introduce new poster generation method and also the idea of logicquiz also no significant new compare with the past evaluation such as paperquiz. the assumption should be inspected carefully, as human posters usually have subjectives, such as visual appeal for attention, rather than objective information presentation. Thus, the logical path might be just part of a rubric rather than a comprehensive framework.

  2. bad evaluation: select 134 paper-poster pairs randomly, this ignore the diversity in terms of paper topics, and only source form nips 25. Which might introduce huge bias. The paper do not disclose the human study details, such as how many people, distributions. There is no ablation about the choices the paper readers and poster readers.

Questions for authors

please refer to the question is weaknesses.

the writing and presentation of this paper look significantly rely on AI. such as the fig2 and fig3 size makes no sense, it almost occupy the half of the paper... also come across the full paper, i do not find any examples, and how different examples perform scores.

5 comments
AuthorsOpenReview ↗

Response to Revewier M71Y

1 Novelty and the Purpose of Poster Evaluation

We respectfully disagree that the paper’s contribution is incremental. The novelty of our work lies in identifying and formalizing a critical blind spot across the entire paper-to-poster field: current generative models optimize heavily for visual appearance while systematically breaking the scientific reasoning of the paper.

This conceptual novelty was explicitly recognized by all three other reviewers:

  • Reviewer 96em: "The paper addresses a relevant and under-measured problem. Whether a poster preserves the argument that links evidence to conclusions is a meaningful quality dimension that current benchmarks do not measure directly."
  • Reviewer Cc3o: "The aesthetics–logic disconnect (Fig. 2, r = −0.56) is the kind of result that should make people in the space stop and re-think. PaperForest and ResearchStudio look great and score 29.4/24.4—this is a real, actionable warning."
  • Reviewer sLar: "The paper identifies a meaningful limitation of existing poster evaluations that primarily focus on aesthetics and factual correctness, and introduces logical fidelity as an additional evaluation dimension."

Below, we address the reviewer’s specific concerns with concrete data from the paper:

  • Scientific posters must be "Content and Logic First: The reviewer argues that posters are subjective tools aimed at "visual appeal for attention, rather than objective information presentation." We agree that visual appeal is necessary to draw readers in, but a scientific poster is ultimately an academic communication medium. A poster that catches the eye but drops the core reasoning steps behind a paper’s claims fails at basic scientific communication.

  • Our contribution is a new benchmark, not a new generation method: Our paper is in Benchmarks track, introducing a benchmark that exposes where existing generation methods fail is an essential contribution. And LogicBench addresses this by providing guidance for future models toward preserving scientific logic integrity alongside visual design.

  • LogicBench is fundamentally different from PaperQuiz: PaperQuiz and P2PEval only test surface-level facts (e.g., retrieving an individual number, dataset name). On PaperQuiz, the poster generation problem looks almost solved: generation methods achieve 90.8%–95.6% on Understanding and 75.4% on Detail. However, on LogicBench, these methods drop to 24.4%–61.8% because the generated posters fail to present the logic (e.g., tracing how theoretical assumptions justify a final regret bound) in the original paper.

2 Insufficient Evaluation

2.1 Data Bias

Topic bias: Our 134 randomly sampled papers represent five distinct areas:

  • Deep Architectures and Representations: 34 papers
  • Optimization and Learning Theory: 31 papers
  • Language and Multimodal Systems: 28 papers
  • Computer Vision: 22 papers
  • Reinforcement Learning: 19 papers

Conference bias: To evaluate conference bias, we conducted evaluations on 30 additional papers with 300 questions across three venues. The results show similar trends across all conferences.

Conference VenuePapers (N)Human Ref. (%)Paper2Poster (%)P2P (%)PosterGen (%)PosterForest (%)ResearchStudio (%)
NIPS 2025 (Main)13498.3%61.8%54.7%50.3%29.4%24.4%
ICML 20251097.0%63.0%52.0%48.0%31.0%23.0%
ACL 20251099.0%59.0%58.0%46.0%27.0%25.0%
CVPR 20251096.0%64.0%50.0%54.0%33.0%21.0%

Sample scale bias: Our dataset size of 134 papers and 1,354 questions exceeds prior poster evaluation sets, including PaperQuiz with 100 papers and P2PEval with 121 papers.

We have added a discussion on this in Appendix B of the revision.

2.2 Clarification on "Human Reference"

Our benchmark includes no humen study. As stated in Section 4.1, "Human Reference" designates the official conference posters created by the original paper authors. Evaluated under the identical scoring pipeline, these author-designed posters establish an objective ceiling, confirming that human posters preserve full reasoning chains. Furthermore, to thoroughly validate human perception and benchmark quality, we have incorporated extensive human evaluations into the revised paper: domain experts confirmed a 93.8% pass rate across audited logic graphs and questions (Fleiss’ =0.85; see 96em-3.1), a direct human perception study closely matched the VLM reader with an average gap of only +0.3% (53.4% vs. 53.1%; see 96em-3.2), and expert reviews verified the soundness of our question filtering criteria (see 96em-1.2).

AuthorsOpenReview ↗

Response to Revewier M71Y

3 AI Overuse

This paper is not AI-generated. AI was only used as an assistive tool for manuscript drafting and editing. Below are our responses to the specific concerns:

  • Large figure size: We deliberately highlight Figures 2 and 3 as they present the core empirical discoveries of the paper. In fact, the current frontier LLMs prefer generating diagrams with small size. Furthermore, we created our figures in PowerPoint and Python, and we can provide the source files as proof.

  • No examples: We have included detailed examples in Appendix C. Each one displays the multi-hop reasoning path alongside its verified question and targeted distractors.

AuthorsOpenReview ↗

Follow up for Reviewer M71Y

Dear Reviewer M71Y,

We appreciate your candid feedback and have carefully considered your concerns about novelty, evaluation diversity, and presentation.

We respectfully offer the following clarifications and revisions:

  1. Novelty (Response 1): LogicBench identifies a blind spot across the field—generators optimize for aesthetics while breaking scientific reasoning. PaperQuiz scores of 90.8%– 95.6% contrast sharply with LogicBench scores of 24.4%–61.8%, demonstrating that existing benchmarks miss this dimension entirely. This is a benchmark contribution, not a generation method, and we believe it provides actionable guidance for future work.

  2. Data diversity (Response 2.1): Papers span five subfields (Deep Architectures: 34, Optimization: 31, Language/Multimodal: 28, CV: 22, RL: 19). Cross-venue validation (ICML, ACL, CVPR) shows consistent trends.

  3. Human study (Response 2.2): We have added three human studies—expert audit (93.8% pass rate), reader perception (0.3% human-VLM gap), and filter validation.

  4. Presentation (Response 3): Figures 2 and 3 are intentionally large to highlight core findings. Detailed examples are in Appendix C.

We understand your concerns about incremental contribution. However, we believe exposing a systematic failure mode across the field is a meaningful contribution to the ICLR community.

If you find the revisions address your concerns, we would appreciate your reconsideration.

Respectfully,

The Authors

Reviewer_sLarOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LogicBench, a benchmark for evaluating whether generated scientific posters preserve the logical connections between evidence and conclusions.

It constructs Scientific Logic Graphs from source papers and generates multi-hop multiple-choice questions to evaluate logical fidelity.

Critical strengths

The paper identifies a meaningful limitation of existing poster evaluations that primarily focus on aesthetics and factual correctness, and introduces logical fidelity as an additional evaluation dimension.

It proposes a structured benchmark construction pipeline using Scientific Logic Graphs and Multi-Hop QA to assess whether generated posters preserve scientific reasoning.

Critical weaknesses
  • The question selection process may bias the reported results. Questions are selected based on high human accuracy (>= 0.90) and a large human-generated gap (>= 0.40), potentially amplifying the reported performance differences. Conversely, selecting questions that favor a particular method could lead to different conclusions. Although Appendix B shows ranking stability for thresholds of 0.2-0.4, PosterGen overtakes P2P when the gap threshold is removed (Figure 6). Further justification of this selection criterion is needed.

  • The LLM/VLM backbones used by each poster generation method are not specified, making it difficult to assess the fairness of the comparison and distinguish framework differences from backbone effects.

  • The evaluation relies on MCQ accuracy from a single VLM (GLM-5.3). It is unclear whether incorrect answers reflect missing logical connections in the posters or limitations in the evaluator's perception and reasoning abilities.

  • The accuracy and validity of the constructed Scientific Logic Graphs are not sufficiently validated. An end-to-end qualitative example tracing source evidence through graph construction, reasoning path extraction, and MCQ generation would help establish the reliability of the proposed pipeline.

  • A question-only baseline would help identify potential shortcuts. In several Appendix C examples, the correct options are substantially longer than the distractors, potentially allowing the questions to be answered without examining the posters.

  • A human study would strengthen the benchmark's validity by examining whether LogicBench scores align with actual human understanding of the posters' scientific reasoning.

  • The sign convention for delta Acc in Table 1 appears to follow Human - Method. Please clarify this definition to avoid confusion.

  • Some details of the question selection process require clarification:

    1. Table 3 reports 89.4% human calibration accuracy, despite the selection criterion requiring human accuracy >= 0.90. How can these results be reconciled?
    2. The paper alternately states "10 questions" and "at least 10 questions" per paper. Please clarify how 1,354 questions were obtained from 134 papers.
Questions for authors

I would appreciate it if the authors could address the concerns raised in the weaknesses above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer sLar

1 Gap Validation

Please refer to our response to 96em-1 for the complete filter ablation table and expert audit data:

1.1 The Gap Filter Does Not Alter the Benchmark Findings

Without poster filters, human posters outperform all generators by 18.7%, and by 23.1 points with the human filter alone (Kendall’s =0.866). PosterGen edges ahead of P2P at threshold 0.0 because clean layouts help locate surface keywords on simple questions. P2P copies dense text, preserving intermediate steps on multi-hop questions. This gap actually reduces the evaluation noise from uninformative candidate items.

1.2 The Necessity of Question Filtering

In our expert review, 85.0% of questions removed by the human filter (Human0.90) probed minor appendix steps or parameter trivia unsuitable for posters. Furthermore, 82.0% of low-gap questions (Gap<0.40) were solved through superficial keyword overlap. These two gates ensure the benchmark tests essential poster-level scientific logic.

2 VLM Backbone Specification

To ensure a fair comparison, we use the same VLM backbone GLM-5.3 for all poster generation methods. We have clarified this setting in the Benchmark and Poster Collection paragraph of Section 4.1 in the revision.

3 Evaluation Limitations

3.1. Human reference posters confirm evaluator sufficiency

Under the identical multiple-choice evaluation protocol, human-authored posters achieve 98.3% accuracy, while generated posters score only 24.4%–61.8%. Since GLM-5.3 accurately answers questions derived from human-authored posters, the evaluator has sufficient visual perception and reasoning capacity when the required scientific logic is present on the canvas. Thus, the low generated-poster scores reflect genuine content omission rather than evaluator incapacity.

3.2 Human audit confirms that benchmark questions are valid and errors do not silently propagate

As detailed in our response to 96em-3.1, experts audited 600 graph nodes and 420 inference edges across 20 sampled papers, yielding high inter-rater agreement (Fleiss’ κ = 0.86, Krippendorff’s α = 0.87) and a 94.8% edge-validity pass rate, confirming that the source graphs are objectively grounded. Downstream MCQ audits further show 93.7% option unambiguity, 92.4% distractor isolation, 91.9% hop necessity, and an overall 93.8% pass rate with high consistency (Fleiss’ κ = 0.85). This confirms that final questions remain valid and avoid cascaded LLM errors.

4 End to End Example

We have added an end-to-end qualitative example in Appendix C that traces one representative item from parsed source evidence, through typed graph construction and a three-hop reasoning path, to MCQ generation and validation.

5 MCQ Option Length Balance

We apologize for the misleading presentation in Appendix C. By design, both o⋆o^{\star} and or−o_r^{-} concatenate all kk claims. In those examples, the distractors display only the modified claim c~r\widetilde{c}_r, while the correct option displays the full sequence C(c1,…,ck)C(c_1,\dots,c_k), making o⋆o^{\star} appear longer.

Empirically, we find the options to be highly balanced, with a median max/min length ratio of 1.07 across all 1,354 questions. To further test for length-based shortcuts, we conduct a question-only baseline (without poster input), which achieves 23.5% accuracy, close to the 24.01% random-choice expectation. Based on these analyses, option length neither affects selection nor offers a shortcut.

We have updated the examples in Appendix C to display all options in their complete form. In addition, we have included another 500 question examples in logicbench_500_questions_anonymized.json in the supplementary material for verification.

6 Human Study

To address concerns regarding benchmark validity and human alignment, we conducted three human studies across 20 papers with domain experts in the revision:

  • Filter Validation, which confirmed that 85.0% of human-discarded candidate questions probed minor trivia and 82.0% of low-gap items relied on surface shortcuts (see our response to 96em-1.2; added to Section 4.3).
  • Pipeline and Graph Audit, where experts verified 200 questions, 600 nodes, and 420 edges, achieving a 93.8% pass rate and high inter-rater reliability (Fleiss’ =0.85; see 96em-3.1 and Cc3o-1; added to Appendix D).
  • Human vs. VLM Reader Perception, where domain researchers answered 200 questions directly from posters, closely matching the GLM-5.3 reader with an average gap of only +0.3% (53.4% vs. 53.1%; see 96em-3.2; added to Appendix D).

7 Sign Convention for ΔAcc\Delta\text{Acc}

The sign convention in Table 1 is defined as ΔAcc=AccHuman−AccMethod\Delta\text{Acc} = \text{Acc}_{\text{Human}} - \text{Acc}_{\text{Method}}. We have clarified this detail in the first paragraph of Section 4.2 in the revision.

AuthorsOpenReview ↗

Response to Reviewer sLar

8 Detail Clarification in MCQ Process

  • Reconciling Human Calibration Accuracy: The 89.4% reported in Table 3 comes from an independent re-evaluation of the calibration split. VLM randomness during re-scoring causes minor fluctuations around the 0.90 threshold. We have clarified this to avoid confusion in the Calibration-Test Generalization Gap paragraph of Section 4.3 in the revision.

  • Derivation of 1,354 Questions: Our target is 10 questions per paper (134 × 10 = 1,340). When more than 10 questions pass selection for a given paper, we retain all of them, adding 14 extra questions for a total of 1,354. We have unified the phrasing in the Question Set and Reference Calibration paragraph of Section 4.1 in the revision.

AuthorsOpenReview ↗

Follow up for Reviewer sLar

Dear Reviewer sLar,

Thank you for your detailed review. Your concerns about selection bias, backbone specification, evaluator limitations, and question validity have been addressed:

  1. Gap filter justification (Response 1): Expert audit shows 85% of human-filter-discarded questions probed minor trivia; 82% of low-gap questions used surface shortcuts. Filtering removes noise, not signal.

  2. VLM backbone (Response 2): All methods use the same GLM-5.3 backbone—clarified in Section 4.1 of the revision.

  3. Evaluator sufficiency (Response 3.1): Human posters score 98.3% under identical protocol, confirming GLM-5.3 perception is not the bottleneck.

  4. Question validity (Response 3.2): Expert audit confirms 93.8% pass rate with high inter-rater reliability (Fleiss' κ = 0.85).

  5. Length shortcuts (Response 5): Question-only baseline achieves 23.5% (near random 24.01%); median max/min option length ratio is 1.07.

  6. Human study (Response 6): Three human studies now included—filter validation, pipeline audit, and reader perception.

  7. Sign convention and details (Responses 7-8): Clarified ΔAcc = Acc_Human − Acc_Method; the 89.4% vs. 90% discrepancy is due to VLM re-scoring randomness; 1,354 = 134×10 + 14 extra questions.

We hope these clarifications address your concerns. If satisfactory, we would be grateful for an updated review.

Respectfully,

The Authors

Reviewer_Cc3oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper points out that existing paper-to-poster benchmarks (PaperQuiz, P2PEval) reward visual quality and surface fact recall, but never check whether the poster preserves the multi-hop logical chain from evidence → results → conclusion. They build LogicBench: a provenance-aware evidence graph from NIPS 2025 papers, sampled connected inference paths, MCQ generation with hop-ablation and reference-calibration, and a held-out evaluation. Human posters score 98.3%; the best generated poster (Paper2Poster) only 61.8%, and newer methods look worse on LogicBench while looking better on aesthetic metrics.

Critical strengths
  • The aesthetics–logic disconnect (Fig. 2, r = −0.56) is the kind of result that should make people in the space stop and re-think. PaperForest and ResearchStudio look great and score 29.4/24.4—this is a real, actionable warning.
  • The construction pipeline (logic graph → directed path sampling → counterfactual MCQ → hop-ablation → reference-calibration → held-out) is more careful than most LLM-judged benchmarks, and the calibration→test generalization gap analysis (Sec 4.3) shows they're aware of overfitting-to-questions.
  • Distractor diagnosis by hop (Fig. 3) is smart: errors concentrate at hop 1, meaning posters preserve conclusions but drop the supporting premise—that's a concrete failure mode a method designer can target.
Critical weaknesses
  • The graph construction and MCQ generation are both LLM-in-the-loop, and the paper reports limited validation of the source graph itself. A mis-parsed evidence edge in step 1 silently becomes a "hard question" in step 4. The 98.3% human ceiling is reassuring, but the inter-rater agreement on graph edges is not shown.
  • 134 papers / 1,354 questions is small. Split into calibration and held-out, the held-out set is maybe ~700 questions. Per-method CIs will be wide, and the ranking between P2P (54.7) and Paper2Poster (61.8) is not obviously stable.
  • Scoring uses GLM-5.3 as the poster reader on both source and poster. A model that's weak at reading posters (a known issue) will depress all methods together; the gap between human and machine readers isn't quantified.
  • The paper is strong on diagnosis but weak on what to do about it. No baseline method is modified to target the hop-1 failure; the paper ends at "existing benchmarks miss this." That's a contribution, but feels half a step short.
Questions for authors

Q1. You sample 134 NIPS 2025 papers. What is the venue/field distribution—are these mostly CS/ML papers with a clean method→experiment→conclusion chain? Would a biology or economics paper, where the logical chain is more narrative, break the pipeline? Q2. GLM-5.3 reads both source paper and poster. Can you quantify how much of the human-vs-generated gap is poster-reading difficulty rather than poster quality? Running a human reader on a subset would de-confound this.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Cc3o

1 Inter-rater Agreement and Error Propagation in MCQ generation Process

We agree that LLM-in-the-loop pipelines risk silent error propagation. With the detailed human audit data provided in our response to 96em-3.1, we address the review’s specific concerns below with concrete evidence:

1.1 Inter-rater Agreement on Graph Construction

Experts audited 600 graph nodes and 420 inference edges. The Inter-rater agreement on graph edges was high, with Fleiss' = 0.86 and Krippendorff's = 0.87. This directly answers the concern about missing reliability metrics, confirming that the source graphs are objectively grounded with a 94.8% edge validity pass rate.

1.2 Preventing Error Propagation in MCQ generation

We further audit the downstream generated questions to ensure early errors do not silently propagate. Experts verified option unambiguity at 93.7%, distractor isolation at 92.4%, and hop necessity at 91.9%. The overall pipeline achieved a 93.8% pass rate with high consistency, Fleiss' = 0.85. This demonstrates that the final questions are mostly valid and are not artifacts of cascaded LLM errors.

2 Dataset Scale and Statistical Stability

We respectfully clarify that the calibration/held-out split applies to posters, not questions. The held-out evaluation paragraph of Section 3.3 defines our held-out split by dividing each method’s 5–10 independent poster runs into a calibration set C (⌊N/2⌋\lfloor N/2 \rfloor) and a sealed held-out set H (⌈N/2⌉\lceil N/2 \rceil), preserving the entire question set. All 1,354 questions are evaluated on H, providing full statistical power across 134 papers, which exceeds Paper2Poster (100 pairs) and P2PEval (121 pairs). Across these 1,354 evaluated questions, the 7.1% margin between Paper2Poster (61.8%) and P2P (54.7%) shows non-overlapping 95% bootstrap confidence intervals ([59.2%, 64.4%] vs. [52.0%, 57.4%]), yielding paired significance at p<0.001. Table 5 confirms this stability: even when subsampling five questions per paper (approximately 670 questions), the methods achieve a 99.2% Top-1 agreement (mean Kendall’s =0.96), demonstrating that the measured ranking remains empirically stable.

3 Quantifying the Human-Machine Reader Gap

To quantify the human-machine reader gap and address the concern of a systematic bottleneck, we conducted a human perception study, as detailed in our response to 96em-3.2.

The human and VLM readers closely align, with an overall average accuracy gap of only +0.3% (53.4% vs. 53.1%). This confirms that the VLM's perception is not a systematic bottleneck.

4 Actionable Baseline Improvement Targeting Hop-1 Failures

To demonstrate that our diagnosis guides concrete solutions, we added a lightweight premise-preservation constraint into the planning stages of Paper2Poster and P2P across 20 papers. This adjustment requires generators to include initial supporting assumptions alongside experimental findings. Evaluating this change yields substantial accuracy gains: Paper2Poster improves from 61.8% to 72.5% (+10.7%), while P2P improves from 54.7% to 63.5% (+8.8%), reducing the Hop-1 error share from 56.1% down to 32.5%. This confirms that LogicBench provides direct diagnostic feedback guidance that allows generative methods to repair broken reasoning chains.

5 Paper Topic Distribution and Generations beyond CS/ML Papers

  • Diverse Field Distribution Across Sourced Papers: We sincerely thank you for raising this insightful question. For the complete subfield distribution across our 134 sampled papers, we kindly refer you to our response to M71Y-2. Our sample spans five distinct research areas, ranging from Reinforcement Learning to Deep Architectures.

  • Applicability to Narrative Scientific Disciplines: Narrative scientific disciplines align directly with our pipeline. Equations 1-4 construct semantic dependency graphs (vpre→evconv_{\text{pre}} \xrightarrow{e} v_{\text{con}}) from text coordinates, operating independently of fixed section templates. Narrative arguments in biology and economics follow explicit causal and conditional dependencies: for example: biological arguments connect genetic interventions to molecular pathways, while economics arguments connect policy shocks to welfare outcomes. In a pilot test across six external papers (three from bioRxiv and three from NBER), our pipeline extracted valid multi-hop reasoning chains, achieving a 91.7% expert pass rate across 60 generated questions.

AuthorsOpenReview ↗

Follow up for Reviewer Cc3o

Dear Reviewer Cc3o,

We thank you for recognizing the aesthetics–logic disconnect as a meaningful contribution and for your constructive feedback on validation and actionable improvements.

We have addressed your concerns:

  1. Graph validity (Response 1): Experts confirmed 94.8% edge validity with high inter-rater agreement (Fleiss' κ = 0.86), addressing the silent error propagation concern.

  2. Statistical stability (Response 2): All 1,354 questions are evaluated on the held-out set (split applies to posters, not questions). Bootstrap CIs are non-overlapping for Paper2Poster vs. P2P (p < 0.001).

  3. Human-machine reader gap (Response 3): Human readers match GLM-5.3 within 0.3%, de-confounding poster-reading difficulty from poster quality.

  4. Actionable baseline (Response 4): Adding a premise-preservation constraint improves Paper2Poster from 61.8% to 72.5% and reduces Hop-1 errors from 56.1% to 32.5%, showing LogicBench provides actionable diagnostic feedback.

  5. Field distribution (Response 5): Papers span five subfields; pilot tests on bioRxiv and NBER papers achieved 91.7% expert pass rate.

If you find them satisfactory, we would appreciate your consideration of an updated review.

Respectfully,

The Authors

Reviewer_96emOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces LogicBench, a benchmark that tests whether a generated academic poster preserves the multi-hop reasoning of its source paper, rather than only its visual quality or isolated facts. A multi-agent pipeline builds a validated logic graph from the paper, samples connected 2–4-hop inference paths, and converts them into multiple-choice questions in which each distractor changes one hop. Questions are filtered by hop ablation and by calibration against human and generated posters, then scored on held-out posters by a VLM reader. On 134 NeurIPS 2025 papers (1,354 questions), human posters reach 98.3% and five generators reach 24.4–61.8%, although the same methods perform well on published PaperQuiz and P2PEval scores. The paper also reports that newer methods receive higher aesthetic scores but lower LogicBench accuracy.

Critical strengths
  1. The paper addresses a relevant and under-measured problem. Whether a poster preserves the argument that links evidence to conclusions is a meaningful quality dimension that current benchmarks do not measure directly. The pipeline is carefully designed for this purpose, with provenance-tagged evidence, connected inference paths, hop ablation against shortcuts, and a held-out split.
  2. The question design makes errors diagnosable. Because each distractor changes exactly one hop, the benchmark can show where a poster breaks a reasoning chain and which inference types a method loses (Fig. 3, Fig. 5), which gives method developers actionable feedback.
Critical weaknesses
  1. The main human–generated gap is largely shaped by the selection rule. Questions are kept only if human posters score at least 0.90 and the human–generated gap is at least 0.40, computed on calibration runs of the same five methods that are later evaluated. Fig. 6 suggests that without this filter the generated methods score roughly 60–90%. Because human posters are also part of the filter, the 98.3% human score is not an independent control, and the benchmark partly measures agreement with the authors' own content choices.
  2. The comparative claims are not measured under the same conditions. Table 1 compares LogicBench, computed on the authors' 134 NeurIPS 2025 papers, with PaperQuiz and P2PEval scores taken from the original publications (Sec. 4.2), so the benchmarks are not evaluated on the same posters. The aesthetic scores are described in a single sentence, without the judge model or prompts. The aesthetics–logic correlation (r = −0.56) rests on six points, the Fig. 2 caption calls it weak, and the conclusion states that visual appeal "has little relation to logical fidelity", which sits uneasily with the trend claimed in Sec. 4.3 and the framing of the title.
  3. The validity of the measurement rests entirely on LLM judgments. Question construction, validation and scoring are all performed by LLM agents, and posters are read by a single VLM. Without human verification, some errors may reflect reader failures on dense posters rather than missing logic; ResearchStudio and PosterForest score 24.4% and 29.4% on questions with four or five options.
Questions for authors
  1. Could the authors report results without the gap filter, and with only the human-poster filter? This would show how much of the human–generated gap comes from question selection.
  2. Could the authors describe the aesthetic scoring setup (judge model, prompts, and whether scores were computed on these posters)? Ideally, PaperQuiz and P2PEval would also be run on the same posters.
  3. Would the authors consider adding a small human audit of sampled questions (correctness and necessity of each hop) and a question-only baseline? This would make the measurement considerably more convincing.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up for Reviewer 96em

Dear Reviewer 96em,

Thank you for your thorough and constructive review. Your concerns about question filtering, comparative conditions, and LLM-judge validity were instrumental in shaping our revision.

Since your review, we have made the following additions:

  1. Filter ablation (Response 1.1): Without poster-based filters, human posters still outperform generators by 18.7% (Kendall's τ = 0.824), confirming the core finding is not an artifact of question selection.

  2. Expert validation of filtering (Response 1.2): Five domain experts confirmed that 85% of human-filter-discarded questions probed minor trivia, and 82% of low-gap questions relied on surface shortcuts—validating both filters.

  3. Human audit (Response 3.1): Experts verified 93.8% pass rate across 200 questions, 600 nodes, and 420 edges (Fleiss' κ = 0.85), directly addressing your LLM-judgment concern.

  4. Human vs. VLM reader study (Response 3.2): Domain researchers matched GLM-5.3 within 0.3%, showing evaluator reliability.

We hope these additions address your soundness concerns. If you find the revisions satisfactory, we would be grateful if you would consider updating your review.

Respectfully,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.