acceptodds
Under review as a conference paper at ICLR 2027

SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

Abstract

Scientific figure quality is bound to the manuscript: a crop can be visually clear and still contradict its caption or the paragraph that cites it. Most image quality assessment and chart-understanding methods target a detached visual surface or a question-answering score, rather than alignment among the figure, its caption, and the citing text. A single end-to-end prompt that sees the image and the text together still mixes what is visible with what the caption and the citing paragraph claim, so fluent wording can raise a visual score and missing text can be treated as low quality. To address this gap, we introduce SciFigQual-Bench, a manuscript-linked benchmark for published CS-conference figures that binds each figure to its caption and to index-resolved citing paragraphs, and scores visual clarity, layout, caption consistency, context consistency, and misleading risk, leaving a dimension unevaluated when its evidence is absent. SFQ-Agent reads the image and the text in separate calls and records modality-specific evidence. A cross-modal judge scores caption and citing-text alignment from those reports, and a deterministic runner keeps the visual scores, sets misleading risk by a fixed rule, and applies the written caps, so each dimension follows the rubric instead of a single end-to-end prompt. Experiments comparing this staged judge with single-pass and OCR-sidecar protocols find the closest point-estimate fit to mean human ratings under staged judging, while the gap between protocols remains small. Caption consistency remains the main gap, and agreement with the rater mean answers a different question from agreement among human raters.

Primary area:
datasets and benchmarks
Keywords:
scientific figure quality, manuscript-linked context, vision-language judge, caption consistency, multimodal benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Zsw4OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SciFigQual-Bench for assessing published scientific figures together with their captions and manuscript-linked citing paragraphs. It contains 7,609 figures from 1,144 CS conference papers, including 6,308 human-rated figures. The rubric covers visual clarity, structure and layout, caption consistency, context consistency, and misleading risk, with unavailable textual evidence gating off the corresponding consistency dimensions. The authors also propose SFQ-Agent, which separates visual and textual evidence collection, performs cross-modal judging, and applies deterministic score constraints. Experiments compare 29 protocol–backend configurations on eval1200, comprising 1,200 figures from 254 papers with at least three human ratings per figure. The best Agent configuration achieves an overall MAE of 0.418 against mean human scores, compared with 0.443 for the best Direct configuration and 0.565 for a constant-mean predictor. The paper also examines inter-rater disagreement, particularly on caption consistency.

Critical strengths
  1. The paper evaluates figures alongside captions and citing paragraphs, addressing consistency issues beyond visual quality. The dataset contains 6,308 human-rated figures, with multiple ratings and written rationales on eval1200. These annotations could provide a useful resource for studying figure–text verification beyond the particular judge proposed in this paper.

  2. The evaluation distinguishes missing text from poor consistency through evidence gating. The authors also report a constant-mean baseline and inter-rater agreement, and explicitly distinguish matching mean human scores from matching individual raters.

Critical weaknesses
  1. The full Agent pipeline improves over same-backbone baselines, but the experiments do not separate the benefits of staging, additional calls, and the Runner’s rules. A separate Runner ablation would help explain what drives the improvement. It is also unclear how conflicts identified during fusion affect the final MR score, since the stated calculation uses the visual risk band and text-stage severity.

  2. Human agreement is modest, particularly for CC (α = 0.49), which leaves uncertainty about the reliability of a central benchmark dimension. Although the rubric distinguishes incomplete descriptions from actual contradictions, the paper does not report agreement and model errors separately for these cases. This makes it difficult to tell whether the reported CC difficulty mainly reflects differing expectations about caption completeness or difficulty identifying factual errors.

Questions for authors
  1. Could you report the results of SFQ-Agent without the Runner? Table 2 already compares Direct, Sidecar and Agent on the same backbone, but the Runner directly changes CC/CTX and recomputes MR. Removing it would help clarify how much of the improvement comes from these rules rather than the staged judging.

  2. Does the disagreement on CC mainly come from captions that leave out details, or from captions that say something incorrect? Appendix A.1 treats these cases differently. Reporting human agreement and model errors separately for the two groups would help explain what makes CC difficult.

  3. It is unclear whether human annotators saw the same two 500-character citing snippets as the models or the longer paragraphs. The paper also does not specify the total number of unique annotators or the number of ratings per figure outside eval1200. These details would help clarify the annotation setup.

  4. It is unclear how a conflict found during fusion affects the final MR score. Appendix C.2 assigns severity to the text-only stage, and Algorithm A1 combines it with the visual risk band. The paper does not explain how a contradiction that becomes apparent only when comparing the image and text enters this calculation. A worked example showing the intermediate outputs would help clarify this.

  5. Could you check Figures A1 and A2 against the original records? The images for Ex-B and Ex-C in A1 appear swapped, and the images in A2 appear misaligned with their captions. In A2, most model Overall values do not equal the stated mean of the five printed dimensions. For example, Ex-H/D2 averages to 6.0, not 6.7, which reduces the claimed +1.5 overestimate to about +0.8. Are these errors limited to figure preparation, and is Table 2 unaffected?

Reviewer_UjVhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SciFigQual-Bench, a benchmark dataset links figures from top-tier CS publications with their corresponding captions and the specific text segments in the main body that reference them. It also proposes SFQ-Agent, a method decouples visual perception from textual verification and employs deterministic runner for scoring. The core contributions include the introduction of multimodal context binding and a 5-dimensional evaluation framework.

Critical strengths
  • Precise Task Definition: By shifting the evaluation of scientific figures from "isolated visual judgment" to "verification of consistency between figures and their textual context," the approach directly addresses a genuine pain point in the peer-review process.
  • Auditable Architecture: SFQ-Agent employs a separation between visual and textual modalities by using a deterministic "Runner" that locks in visual scores (VC/SL), it effectively prevents the "textual hegemony" of LLMs during the evaluation process.
Critical weaknesses
  1. Positive-sample bias: The dataset consists exclusively of papers published at top-tier conferences (positive samples), causing the human-annotated "gold standard" scores to cluster heavily around 8.13. The authors should incorporate rejected papers, low-quality submissions, or synthetically generated negative samples to rigorously validate the model's true capabilities.

  2. Semantic ambiguity regarding the red "X" marks in Figure 1: The "X" marks in the upper section denote "flaws in the actual charts," those in the middle section are used to indicate that traditional methods "fail to cover" specific capabilities, making them quite misleading.

  3. Superficial description and motivation for SFQ-Agent: As the primary innovation, the Agent is described in less than a page of the main text, with many critical mechanisms relegated to the appendix. Furthermore, the rationale for "decoupling the two Agents" is explained only superficially, lacking a deep-dive into the underlying motivations.

  4. Lack of fairness and key baselines in experimental comparisons:

  • Failure to account for the impact of reasoning: The paper does not clarify whether the "Direct" or "Sidecar" prompts trigger the model’s reasoning (Chain-of-Thought/CoT) capabilities. If enabling CoT within a single-call framework yields superior results, the perceived gains from the proposed multi-stage architecture would be significantly diminished.
  • Absence of standard Agent baselines: The study fails to compare its approach against established, state-of-the-art multi-turn Agent frameworks.
  1. Low inter-annotator agreement for the "CC" (Chart-Caption Consistency) metric: Beyond merely noting the inherent difficulty of this dimension, the authors fail to address whether the low agreement stems from an ambiguous scoring rubric, which could lead to inconsistent annotation.

  2. Overemphasis on routine engineering tasks in dataset construction: In the current era of AI Agents, tasks such as using Codex for PDF scraping, parsing, and watermark removal are standard engineering procedures. The authors should not devote significant space to these mundane technical details; instead, they should focus on the specific, targeted methodological design choices they have made.

  3. Premature use of the term "L1-gated" in the introduction, without definition until the methodology section. Additionally, the term "L1" is highly prone to being misinterpreted as "L1 regularization" within the context of machine learning.

  4. Figure 4: One data flow arrow is significantly obscured.

Questions for authors

no question

Reviewer_rBsuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The submission introduces SciFigQual-Bench, a dataset and benchmark with 7,609 scientific figures from 1,144 papers, including 6,308 with human ratings. It assesses figures together with their captions and available citing text, covering clarity, layout, consistency with the text, and the risk of misleading readers.

The submission also proposes SFQ-Agent, which examines images and text separately, compares the findings, and applies fixed rules to produce scores. Experiments compare this approach with single-pass judging based on agreement with average human ratings.

Critical strengths
  • [S1] The dataset is a useful contribution. The released dataset contains 7,609 figures from 1,144 papers, with human ratings for 6,308 figures. Each figure in eval1200 has at least three ratings, and the released scores and explanations make disagreements easier to inspect (Appendix D).
  • [S2] The paper addresses an interesting problem. A figure can be visually clear but still conflict with its caption or the text that cites it. Evaluating these together, while separating visual quality from text consistency, is a sensible approach to assessing scientific figures (Sec. 3.1).
Critical weaknesses
  • [W1] The scoring rules are not applied consistently. Appendix A.1 says a caption should score no higher than 6 if it leaves out a key metric (line 804-805). But in Case 4, Figure 3, the caption does not mention ROC-AUC, and the three raters give it 10, 9, and 6, an average of 8.33 (Table A18). It is unclear how this fits with the claim that humans and models follow the same scoring rules.

  • [W2] The claim of more reliable explanations needs direct evidence. Section 3.4 (line 331) says that separating image and text prevents explanations that sound reasonable but lack support. Table 2 measures agreement with human scores, but matching a score does not mean the explanation is correct. A small human comparison of Direct and Agent explanations would help support this claim.

  • [W3] If I understand correctly, the final MR score does not directly use conflicts found during fusion. Fusion can identify contradictions between the figure and text, but Appendix C.2 and Algorithm A1 compute MR from the earlier visual and text risk outputs, overriding fusion’s MR score.

In general, the main contribution of the submission is the dataset and benchmark, which address an interesting problem. However, the scientific insight remains unclear. I am currently borderline but lean toward acceptance, and I am open to revising my rating based on the rebuttal and discussion with other reviewers.

Questions for authors

Please refer to the weakness

Comments on the paper

1 comment

Thanks for the authors detailed response and the revisions! The additional annotation details and Table A11 are very helpful. After reading the revised manuscript, I have two remaining questions:

  1. Appendix D.5 now explains the Ex-H calculation, but I am still unsure how the text-only stage identified a contradiction. The caption’s reference to a red bar would seem to require image-side evidence to establish that the bar is absent. Could you show the relevant text-stage input and output? Also, the walkthrough gives a final MR of 2, whereas Figure A2 reports 4.2 for Ex-H/F3. Could you clarify how these values relate?

  2. Thank you for adding the human-agreement breakdown in Table A11. Could you also report model CC MAE separately for the omission-only and mismatch groups, at least for the main Direct and Agent configurations? This would help distinguish the human disagreement analysis from the question of which cases contribute most to the model’s CC error.

I remain at weak acceptance and appreciate your efforts to clarify these points!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.