acceptodds
Withdrawn from ICLR 2027

What Does Binary Accuracy Prove? A Diagnostic Decomposition of Visual Graph Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models are increasingly used to read structured diagrams and reason over the relations they encode. Benchmarks typically score only the final answer. Ask a model "Is there a cycle?" and it almost always says yes. Ask "Which nodes form it?" and the cycle it names is often not in the graph. A correct answer thus cannot show whether the model recovered the diagram's structure. We use cycle detection as a diagnostic, since a named cycle can be checked against ground truth. We score four stages: which edges are extracted, whether a real cycle survives among them, whether the named cycle comes from the model's own list, and whether it is real. We test four models on cyclic graphs and trees, each drawn in a standard layout and a stress-test equal-angle ring layout. On ring drawings, models say "cycle" 84-100% of the time, yet the cycle they name is real only 4.8–44.0% of the time. Degree probes and human judgments suggest the drawings are not simply unreadable. Wrong answers follow two patterns: the real cycle never enters the extracted edge list (most common for three models), or it does and a fabricated cycle is named instead (one model). A clean text edge list restores two models fully but not the others, so extraction explains the gap only in part. A "yes" to "Is there a cycle?" does not show that the cycle a model names is real, so such errors can pass unnoticed and cause silent failures in graph-based workflows. The code and dataset are available at https://anonymous.4open.science/r/What-Does-Binary-Accuracy-Prove-31DB.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models (VLMs), Visual Graph Reasoning, Graph Understanding, Cycle Detection, Representation–Decision Dissociation, Visual Reasoning, Graph Extraction, Layout Sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VTWDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether high binary accuracy in visual graph reasoning reflects genuine structural understanding. It proposes a four-stage diagnostic framework covering edge extraction, gold-cycle retention, list faithfulness, and cycle verification. Experiments on four VLMs reveal a substantial gap between correctly predicting cycle existence and identifying valid cycles, particularly under challenging graph layouts. The authors further distinguish two failure modes: losing valid cycles during edge extraction and selecting fabricated cycles despite retaining correct ones. These findings highlight the limitations of binary accuracy for evaluating visual graph reasoning.

Critical strengths
  1. Find an important limitation of existing visual graph reasoning benchmarks, where correct binary answers do not necessarily indicate accurate structural understanding.
  2. Proposes a four-stage diagnostic framework that separates edge extraction, gold-cycle retention, list faithfulness, and final cycle verification, enabling more fine-grained error analysis.
  3. Identifies two distinct failure modes, loss and competition, and demonstrates that different VLMs exhibit different error patterns.
Critical weaknesses
  1. The evaluation is limited to cycle detection on labeled, undirected graphs. It is unclear whether the proposed diagnostic framework and observed failure patterns generalize to other graph reasoning tasks, such as reachability and shortest-path reasoning, or to real-world diagrams.
  2. The contribution is largely limited to diagnosing existing failure modes rather than improving visual graph reasoning. While the paper identifies loss and competition as two major sources of errors, it does not develop or empirically validate corresponding solutions, limiting the practical impact of the proposed framework.
  3. Some numerical results are reported inconsistently across sections. For example, the single-edge insertion experiment reports different percentage ranges in the introduction, Section 7.4, and Appendix R, raising concerns about the consistency of the experimental reporting.
Questions for authors
  1. The reported ranges for the single-edge insertion experiment differ between the introduction (61–100% vs. 2–14%), Section 7.4 (58–100% vs. 4–15%), and Appendix R. Could the authors clarify which values are correct and whether these results refer to different subsets?
  2. The paper identifies loss and competition as two major failure modes but does not propose concrete methods to address them. Could the authors demonstrate how the proposed diagnostic framework can guide practical improvements in visual graph reasoning, rather than relying primarily on experiments with ground-truth information?
Reviewer_9WA7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what binary answers reveal about vision-language models' reasoning over graph drawings, using cycle detection as the main diagnostic task. It evaluates four models on cyclic graphs and trees rendered in a standard Graphviz layout and an equal-angle Ring layout. The proposed evaluation checks extracted edges, retention of a true cycle, consistency of the named cycle with the model's own edge list, and correctness against the ground-truth graph. It distinguishes errors where no true cycle remains in the listed edges from errors where a true cycle remains but the answer is incorrect. The experiments include text-list interventions, geometric analyses, readability probes, and a shortest-path extension. The main finding is that a correct binary answer can accompany an invalid named cycle, with different failure patterns across the evaluated models and conditions.

Critical strengths
  1. I appreciate the effort to check the intermediate edge list and the final answer together. This makes the evaluation more informative than scoring the binary answer or structure extraction separately. The distinction between losing a true cycle and returning an incorrect answer despite retaining one gives a concrete way to inspect failures that aggregate accuracy would miss.

  2. The comparisons using the model's extracted list and the gold edge list provide useful evidence about where performance breaks down. In particular, some models recover fully on clean text while others retain substantial errors, showing that visual extraction alone cannot account for every model's difficulty. The paired statistical analyses and additional probes help establish that the observed performance gaps deserve attention.

Critical weaknesses
  1. The main empirical scope is narrow relative to the broader literature on visual graph reasoning. The study primarily examines visual structure recovery and limitations of answer-only evaluation through cycle detection, whereas existing work such as VisionGraph [1], GITA [2], and Visual Graph Arena [3] considers multiple task categories. I appreciate the detailed analysis and the shortest-path extension in Appendix O.2, but the latter uses short paths on the same Ring drawings, and most of the paper's conclusions remain centered on cycle detection. It is therefore unclear how much the diagnostic reveals about visual graph reasoning beyond this particular task. A clearer justification for choosing cycle detection, and an explanation of which findings are expected to extend to other tasks, would help establish the broader significance of the work. At present, I find the contribution more convincing as a focused study of perception and evaluation than as a broader account of visual graph reasoning.

  2. The introduction could provide a clearer narrative connecting the motivation, prior work, and central contribution. After a brief motivation, it moves quickly into a long list of findings containing detailed results and experimental distinctions. This makes it harder to distinguish the main conceptual contribution from the individual observations that support it. Section 2 and Appendix X do discuss relevant work, including faithfulness, but bringing the most important connections into the introduction would help explain how this study advances existing understanding of perception errors and evaluation limitations. My concern here is primarily about organization and emphasis. A more focused introduction would make it substantially easier for a broader readership to understand the research gap and take away the central insights.

Questions for authors
  1. What makes cycle detection a representative diagnostic for the aspects of visual graph reasoning studied here? In light of the shortest-path results in Appendix O.2, which findings extend beyond cycle detection, and which depend on the task or Ring rendering? This would help clarify the intended scope and significance of the contribution.

  2. Given the paper's emphasis on rethinking how visual graph reasoning is evaluated, could the authors provide a more systematic, high-level discussion of prior work on vision-based graph understanding, learning, and reasoning? Section 2 and Appendix X already discuss relevant studies, but a synthesis of their task coverage, evaluation practices, and known limitations would help clarify which existing conclusions this study challenges or refines. Making this perspective more prominent in the motivation and discussion could strengthen the paper's persuasiveness and broader impact, and help readers see how the insights from cycle detection inform the wider field.

Reviewer_kWFcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that asking LLMs to answer binary questions can give a skewed perspective on how much they truly understand the question. The authors use the following experiment to make this concrete: given an image of a directed graph, an LLM is asked both whether there exists a cycle in the graph, and also to name a cycle. Models frequently can identify the presence or absence of a cycle without being able to correctly identify a cycle.

Critical strengths

None

Critical weaknesses

There is not nearly enough intellectual content here to merit publication in ICLR. The core insight of the mismatch between a yes/no answer and a chain of thought supporting the yes/no is interesting, but doesn’t feel fully developed. The particular case of graph cycles feels like it should be one smaller experiment within a larger more well-developed narrative around this idea. What makes this particular problem of cycle detection so critical to explore?

The paper is very poorly written and structured. It reads as if it were zero-shotted from an LLM with minimal editing or oversight. It is extremely repetitive, and offers very little insight, analysis, or broader interpretation of the results.

The paper only evaluates small fast models: Sonnet 5, GPT 5.6-Luna, Gemini 3.8-Flash, and Qwen 3.8-Flash. Why only these models? Do the same trends hold for larger more capable models? Does more reasoning effort allow models to perform better at this task?

The paper devotes only a single short paragraph to related work, which is woefully insufficient.

Questions for authors

Why did you choose to study this particular problem in such detail? Is there some great practical importance to this particular cycle detection problem? Or is it meant to be emblematic of a larger trend?

Why didn’t you evaluate any larger SOTA LLMs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.