- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SciFigQual-Bench for assessing published scientific figures together with their captions and manuscript-linked citing paragraphs. It contains 7,609 figures from 1,144 CS conference papers, including 6,308 human-rated figures. The rubric covers visual clarity, structure and layout, caption consistency, context consistency, and misleading risk, with unavailable textual evidence gating off the corresponding consistency dimensions. The authors also propose SFQ-Agent, which separates visual and textual evidence collection, performs cross-modal judging, and applies deterministic score constraints. Experiments compare 29 protocol–backend configurations on eval1200, comprising 1,200 figures from 254 papers with at least three human ratings per figure. The best Agent configuration achieves an overall MAE of 0.418 against mean human scores, compared with 0.443 for the best Direct configuration and 0.565 for a constant-mean predictor. The paper also examines inter-rater disagreement, particularly on caption consistency.
Critical strengths
-
The paper evaluates figures alongside captions and citing paragraphs, addressing consistency issues beyond visual quality. The dataset contains 6,308 human-rated figures, with multiple ratings and written rationales on eval1200. These annotations could provide a useful resource for studying figure–text verification beyond the particular judge proposed in this paper.
-
The evaluation distinguishes missing text from poor consistency through evidence gating. The authors also report a constant-mean baseline and inter-rater agreement, and explicitly distinguish matching mean human scores from matching individual raters.
Critical weaknesses
-
The full Agent pipeline improves over same-backbone baselines, but the experiments do not separate the benefits of staging, additional calls, and the Runner’s rules. A separate Runner ablation would help explain what drives the improvement. It is also unclear how conflicts identified during fusion affect the final MR score, since the stated calculation uses the visual risk band and text-stage severity.
-
Human agreement is modest, particularly for CC (α = 0.49), which leaves uncertainty about the reliability of a central benchmark dimension. Although the rubric distinguishes incomplete descriptions from actual contradictions, the paper does not report agreement and model errors separately for these cases. This makes it difficult to tell whether the reported CC difficulty mainly reflects differing expectations about caption completeness or difficulty identifying factual errors.
Questions for authors
-
Could you report the results of SFQ-Agent without the Runner? Table 2 already compares Direct, Sidecar and Agent on the same backbone, but the Runner directly changes CC/CTX and recomputes MR. Removing it would help clarify how much of the improvement comes from these rules rather than the staged judging.
-
Does the disagreement on CC mainly come from captions that leave out details, or from captions that say something incorrect? Appendix A.1 treats these cases differently. Reporting human agreement and model errors separately for the two groups would help explain what makes CC difficult.
-
It is unclear whether human annotators saw the same two 500-character citing snippets as the models or the longer paragraphs. The paper also does not specify the total number of unique annotators or the number of ratings per figure outside eval1200. These details would help clarify the annotation setup.
-
It is unclear how a conflict found during fusion affects the final MR score. Appendix C.2 assigns severity to the text-only stage, and Algorithm A1 combines it with the visual risk band. The paper does not explain how a contradiction that becomes apparent only when comparing the image and text enters this calculation. A worked example showing the intermediate outputs would help clarify this.
-
Could you check Figures A1 and A2 against the original records? The images for Ex-B and Ex-C in A1 appear swapped, and the images in A2 appear misaligned with their captions. In A2, most model Overall values do not equal the stated mean of the five printed dimensions. For example, Ex-H/D2 averages to 6.0, not 6.7, which reduces the claimed +1.5 overestimate to about +0.8. Are these errors limited to figure preparation, and is Table 2 unaffected?
Thanks for the authors detailed response and the revisions! The additional annotation details and Table A11 are very helpful. After reading the revised manuscript, I have two remaining questions:
Appendix D.5 now explains the Ex-H calculation, but I am still unsure how the text-only stage identified a contradiction. The caption’s reference to a red bar would seem to require image-side evidence to establish that the bar is absent. Could you show the relevant text-stage input and output? Also, the walkthrough gives a final MR of 2, whereas Figure A2 reports 4.2 for Ex-H/F3. Could you clarify how these values relate?
Thank you for adding the human-agreement breakdown in Table A11. Could you also report model CC MAE separately for the omission-only and mismatch groups, at least for the main Direct and Agent configurations? This would help distinguish the human disagreement analysis from the question of which cases contribute most to the model’s CC error.
I remain at weak acceptance and appreciate your efforts to clarify these points!