- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates whether high binary accuracy in visual graph reasoning reflects genuine structural understanding. It proposes a four-stage diagnostic framework covering edge extraction, gold-cycle retention, list faithfulness, and cycle verification. Experiments on four VLMs reveal a substantial gap between correctly predicting cycle existence and identifying valid cycles, particularly under challenging graph layouts. The authors further distinguish two failure modes: losing valid cycles during edge extraction and selecting fabricated cycles despite retaining correct ones. These findings highlight the limitations of binary accuracy for evaluating visual graph reasoning.
Critical strengths
- Find an important limitation of existing visual graph reasoning benchmarks, where correct binary answers do not necessarily indicate accurate structural understanding.
- Proposes a four-stage diagnostic framework that separates edge extraction, gold-cycle retention, list faithfulness, and final cycle verification, enabling more fine-grained error analysis.
- Identifies two distinct failure modes, loss and competition, and demonstrates that different VLMs exhibit different error patterns.
Critical weaknesses
- The evaluation is limited to cycle detection on labeled, undirected graphs. It is unclear whether the proposed diagnostic framework and observed failure patterns generalize to other graph reasoning tasks, such as reachability and shortest-path reasoning, or to real-world diagrams.
- The contribution is largely limited to diagnosing existing failure modes rather than improving visual graph reasoning. While the paper identifies loss and competition as two major sources of errors, it does not develop or empirically validate corresponding solutions, limiting the practical impact of the proposed framework.
- Some numerical results are reported inconsistently across sections. For example, the single-edge insertion experiment reports different percentage ranges in the introduction, Section 7.4, and Appendix R, raising concerns about the consistency of the experimental reporting.
Questions for authors
- The reported ranges for the single-edge insertion experiment differ between the introduction (61–100% vs. 2–14%), Section 7.4 (58–100% vs. 4–15%), and Appendix R. Could the authors clarify which values are correct and whether these results refer to different subsets?
- The paper identifies loss and competition as two major failure modes but does not propose concrete methods to address them. Could the authors demonstrate how the proposed diagnostic framework can guide practical improvements in visual graph reasoning, rather than relying primarily on experiments with ground-truth information?