acceptodds
Withdrawn from ICLR 2027

When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Can structured text replace vision for diagram reasoning? A wrong answer after textualization can arise because the representation omits information the question needs, or because the solver fails to use information that is present. We introduce a diagnostic protocol to distinguish these explanations. Using the same solver model and generation settings, we compare three input conditions: the original image, question-blind structure extracted by a vision-language model, or gold structure derived from the diagram source. Validity-triggered recovery tests truncation and schema failure, question-relevant fidelity measures preservation of answer-critical structure, and matched edge interventions test the effect of error location. On a reserved holdout of 240 public FlowGen diagrams, evaluated under a frozen protocol, gold structure reaches 87% accuracy while direct vision and learned text both remain below 30%. The aggregate comparison includes source-specific relation targets and different graph encodings, so it does not isolate extraction error alone. Retrying only invalid extractions makes nearly every public representation schema-valid yet leaves accuracy essentially unchanged; in a post-confirmation diagnostic on generated diagrams, the same policy brings learned-text accuracy close to direct vision. The public learned-text deficit relative to gold more than doubles with structural difficulty. Question-relevant topology predicts correctness better than whole-graph topology. In an exposed intervention study, a single answer-relevant edge edit reduces the primary solver's original-answer accuracy to near zero, while matched irrelevant edits largely preserve it. Supplied structure requires fewer solving tokens than vision, but learned acquisition removes this advantage at single use. These comparisons motivate evaluating acquired text by the answer-relevant evidence it preserves and by the solver's ability to use that representation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Structured Representations, Diagram Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wZCdOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines text-for-vision substitution in diagram reasoning through a fixed-solver input ladder, validity recovery, question-relevant fidelity, and graph interventions. On FlowGen, the learned–gold accuracy gap survives validity recovery and widens with structural difficulty.

Critical strengths
  • Reserved charts, prespecified tests, and chart-level uncertainty estimates support the reported descriptive trends.

  • Validity recovery and recorded examples clearly distinguish syntactic validity from preservation of answer-critical structure.

Critical weaknesses
  • The main comparison includes source-specific targets that are not printed in the images. On 124 connectedTo questions, gold answers 123 correctly while every image-based condition answers none. Although disclosed, this mismatch leaves the share of the headline gap attributable to loss of visually available evidence unresolved.

  • The fixed-solver input ladder does not isolate acquisition failure from difficulty using the acquired representation. Learned inputs use entity IDs and relation objects, whereas gold uses labeled source triplets. Holding the solver fixed does not control this encoding difference, so high gold accuracy does not establish that the solver can use equally correct structure in the learned encoding.

  • Relevant edits are chosen to change the answer, irrelevant edits to preserve it, and both are scored against the original answer. A solver that correctly follows each edited graph would therefore produce the same qualitative contrast, so the result mainly confirms sensitivity to deliberately answer-changing inputs.

  • The exact-match comparison leaves the explanation of its predictive gain unresolved. Question-relevant support changes both which elements are evaluated and how many must match, so the observed improvement does not distinguish the value of answer relevance from the effect of reducing the scope of exact matching.

Questions for authors

See Weakness

Reviewer_G6qaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether structured textual representations can replace visual input in diagram reasoning. Authors aim to distinguish errors caused by information loss during textualization from failures by the solver to use available information, comparing direct image input, learned textual representations extracted by a vision-language model, and gold structures derived from diagram sources on 240 FlowGen diagrams. However, the lack of a sufficiently clear and coherent presentation makes it difficult to determine the precise research questions, the proposed contribution, and how the methodology and experiments address those questions. Consequently, I am unable to provide a reliable summary of the paper's specific scientific contributions beyond its apparent overall objective.

Critical strengths

I am unable to identify and assess the paper's critical strengths with sufficient confidence, given the difficulties in following its presentation. This should not necessarily be interpreted as an absence of strengths, but rather as a limitation on my ability to evaluate them reliably from the submitted manuscript.

Critical weaknesses

My main concern is the manuscript's lack of clarity and coherence, which substantially impedes a meaningful assessment of its scientific contribution. Key concepts are not always introduced before they are used, and the discussion shifts between research questions, related work, and methodological details without sufficiently clear connections. As a result, it is difficult to determine precisely what is being proposed, how the experiments address the central research questions, and which conclusions are supported by the experimental evidence. The severity of these presentation issues raises concerns about whether the manuscript meets the minimum standards of clarity required for a meaningful scientific evaluation. In its current form, I cannot reliably assess the paper's technical merits or scientific contribution.

Questions for authors

My concerns are not limited to specific technical questions that could be addressed individually. Rather, the difficulty lies in understanding the overall narrative of the paper, the relationships between its research questions and methodological choices, and how the experimental results support the stated contributions. Addressing these concerns would require a substantially clearer and more coherent presentation of the work as a whole.

Reviewer_ZkaYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper explores when a structured textual representation can replace an image for diagram reasoning. It compares direct visual QA, QA over question-blind VLM-extracted structure, and QA over source-derived gold structure using a fixed downstream solver. The main evaluation uses 240 held-out FlowGen diagrams (720 questions), with additional controlled and secondary experiments on validity recovery, graph complexity, question-relevant fidelity, edge perturbations, and token cost. The authors argue that retaining answer-relevant relations, rather than merely producing valid JSON or a globally similar graph, is critical for text substitution.

I appreciate the careful attempt to distinguish representation validity from downstream utility. However, the primary learned–gold comparison does not isolate extraction quality, the observed scaling trend remains sensitive to unresolved budget and baseline questions, and the perturbations do not yet explain natural acquisition failures.

Critical strengths
  1. Controlled diagnostic setup. Keeping the solver and its generation settings fixed, while withholding the downstream question from the extractor, is a sensible way to study the acquisition pipeline without changing the solving model.

  2. Useful observations beyond end-task accuracy. On the FlowGen holdout, validity recovery raises schema-valid extractions from 196/240 to 238/240, yet QA moves only from 22.8% to 23.5% (see Table 1). The matched fidelity analysis also shows a meaningful and promising predictive advantage for question-relevant topology exact match (see Table 2).

Critical weaknesses
  1. The mechanism analysis is less conclusive than presented. In Section 4.4, “relevant” edits are constructed to change the original answer, but predictions are still scored against that original answer. A solver that correctly follows the edited graph can therefore be marked wrong by design. The very large relevant-versus-irrelevant differences in Table 3 demonstrate sensitivity to constructed answer changes, but say little about how often these errors occur naturally or how much of the learned–gold gap they explain. Relatedly, question-relevant fidelity is computed using offline support masks from the gold graph and the question. Its predictive improvement is valid as a diagnostic, but the paper does not show how to use it when gold structure is unavailable.

  2. The incremental contribution and scope are not sufficiently clear. FlowGen already studied triplet reconstruction and gold/self-extracted representations; the current paper adds a text-only comparison, relevant-support metrics, and perturbations, but the main takeaway—that errors on answer-critical relations matter more than unrelated errors—is fairly intuitive. The confirmatory results also cover one FlowGen renderer and one model family. The contrasting outcomes on controlled flowcharts (recovered learned 99.5%, direct 99.0%) and QZhou (learned at 2,048 tokens 71.0%, direct 69.8%, source structure 67.7%) highlight how dependent the answer is on task, format, and budget. I would have liked a sharper statement of what generalizable conditions the experiments actually establish.

  3. Readability and figure quality need work. The main text moves between reserved and exposed FlowGen cohorts, controlled program/circuit tasks, and QZhou, often with different question types and output caps. The qualifications needed to interpret the main comparisons are scattered across the paper and appendices, making the narrative difficult to follow. Figure 3 has near-overlapping curves, while Figures 6 and 7 pack many estimates, intervals, and labels into small panels. The figures are technically informative, but the presentation makes the central comparisons harder to identify than necessary.

Questions for authors
  1. After restricting evaluation to relations whose labels/semantics are verifiable from the image, and converting learned and gold structures into the same serialization, could you clarify how much of the Table 1 acquisition gap remains? Please also report whether the Figure 3 trends persist.

  2. What happens on the FlowGen holdout if every chart is re-extracted at 2,048 and 8,192 tokens, rather than retrying only invalid JSON? How do missing-node and missing-edge rates vary with graph size?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.