acceptodds
Under review as a conference paper at ICLR 2027

SpatialGraph-Eval: Intervention-Based Evaluation of Spatial-Network Reasoning in Multimodal Models

Abstract

Multimodal language models increasingly perform well on spatial and visual reasoning benchmarks, but aggregate accuracy does not show how model performance changes when the same problem is presented differently or when its underlying structure is modified. We introduce SpatialGraph-Eval, a diagnostic benchmark for spatial-network reasoning that separates graph structure from its presentation and enables targeted interventions on both. Graphs are procedurally generated with exact shortest-path solutions and controlled variation in path length and decision complexity, while graph size and mean branching are matched within path-length conditions. We evaluate three contemporary multimodal model systems on identical graph structures presented symbolically and visually, under topology-preserving rotations and reflections, and following minimal topology-changing counterfactuals. Representation sensitivity differed substantially across models, with optimal-route accuracy decreasing by 3.9 percentage points for Terra (68.1% to 64.2%), 42.8 for Claude (85.6% to 42.8%), and 17.2 for Gemini (100.0% to 82.8%) when comparing visual with symbolic presentation. Despite unchanged topology, optimal-route accuracy varied substantially across rotations and reflections, with significant heterogeneity across models (Model × Transform, p = 1.83 × 10⁻¹¹). Among instances solved optimally before intervention, models found the new shortest route in only 2.0–7.1% of cases after a one-edge counterfactual rewiring changed the correct route. Failure analysis showed that degradation included graph-invalid transitions as well as valid but suboptimal routes. As an external validity check, we evaluated 50 subgraphs sampled from the San Francisco OpenStreetMap road network; across models, optimal-route accuracy decreased from 90–100% under symbolic presentation to 2–16% under visual presentation. These results reveal substantial model-dependent behavioral sensitivity to representation, topology-preserving presentation changes, and minimal structural interventions, motivating intervention-based evaluation of spatial-network reasoning beyond aggregate task accuracy.

Primary area:
datasets and benchmarks
Keywords:
multimodal models, spatial reasoning, graph reasoning, visual reasoning, spatial networks, representation sensitivity, counterfactual evaluation, benchmark evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QfLhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether multimodal models can still find shortest paths when a graph’s format, orientation, or connections change. It compares three models using paired experiments and algorithmic scoring.

Critical strengths
  1. Comparing the same graphs across input conditions makes the performance differences easier to assess.
  2. Algorithmic ground truth and acceptance of multiple valid shortest paths make the scoring objective.
Critical weaknesses
  1. Rotations and reflections also change the text labels, so some failures may simply come from misreading them.
  2. The rewiring experiment needs clearer setup details and controls to explain the large accuracy drop.
Questions for authors

How do the rewired graphs perform with symbolic input, and how stable are repeated answers to unchanged graphs? How much does accuracy drop when the graph is transformed but the labels stay upright?

Reviewer_iDiyOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SpatialGraph-Eval, a diagnostic benchmark for evaluating spatial-network reasoning in multimodal models through shortest-path tasks. Its main contribution is a paired intervention framework that tests sensitivity to symbolic versus visual representations, robustness to topology-preserving rotations and reflections, and adaptation to single-edge rewiring that changes the optimal route while preserving selected graph properties. The benchmark uses procedurally generated graphs with exact algorithmic ground truth and controls graph size when varying decision complexity within each path-length condition. Validation covers three multimodal model systems on 360 main-benchmark graphs, 72 graphs under six visual transformations, 144 original/counterfactual pairs, and 50 OpenStreetMap subgraphs. Results show model-dependent representation gaps, substantial sensitivity to some rotations and reflections, and low counterfactual success among originally solved instances (2.0–7.1%). Failure analysis distinguishes invalid routes, valid but suboptimal routes, and parsing failures. The findings motivate evaluating invariance and structural adaptation alongside aggregate accuracy.

Critical strengths
  1. Controlled comparisons on matched graph instances. The benchmark evaluates the same underlying graphs across representations and visual transformations, and constructs counterfactuals that preserve selected graph properties while changing the optimal route. Matching graph size when varying decision complexity further reduces confounding. This design is a substantive strength because it makes performance differences more interpretable than comparisons across unrelated task sets.
  2. Analysis beyond aggregate accuracy. Exact graph-based scoring distinguishes valid but suboptimal routes from connectivity violations, while paired analyses reveal failures that average scores conceal. For example, Gemini’s small net accuracy change under 90° rotation masks seven previously correct instances becoming incorrect and five changing in the opposite direction. This provides a concrete demonstration of why aggregate accuracy alone is insufficient to characterize reliability.
Critical weaknesses
  1. Limited diagnostic insight and unclear practical significance. The experiments establish behavioral brittleness but do not adequately distinguish failures in label recognition, connectivity extraction, and shortest-path computation. Consequently, the substantial performance drops provide limited insight into which capability is deficient or how to improve it. A baseline that asks the model to extract an adjacency list and then applies BFS would help separate visual understanding from algorithmic search. Since the routing objective is exactly solvable by standard algorithms, the paper also needs a stronger argument for why direct route generation is an important capability to evaluate, or evidence that its scores predict performance on practical tasks. These omissions limit the contribution beyond documenting benchmark-specific failure rates.
  2. The interventions leave important alternative explanations unresolved. Appendix A.1 states that rotations and reflections transform the entire image, including node labels. The resulting degradation therefore mixes graph reasoning with recognition of rotated or mirrored text; a control that transforms the graph while keeping labels upright is needed to assess their respective contributions. Similarly, preserving graph size, shortest-path length, and selected complexity measures after rewiring does not establish comparable perceptual or search difficulty. The near-zero counterfactual accuracy is striking, but without more detailed intervention examples and diagnostic controls, its interpretation remains uncertain. These issues weaken the framework’s ability to isolate the reasoning capabilities it seeks to assess.
Questions for authors
  1. What underlying capability does the benchmark assess, and why is it important for general-purpose multimodal models? Successful performance combines label recognition, connectivity extraction, and shortest-path computation. Which capability is the central target, and what supports interpreting the combined score as spatial-network reasoning? In particular, what motivates evaluating direct execution of graph search, given that search can be delegated to established algorithms? Clarifying the intended capability and scope of the conclusions would materially affect my assessment of the benchmark’s significance.
  2. How much of the transformation sensitivity reflects label recognition rather than graph reasoning? Appendix A.1 states that rotations and reflections transform node labels along with the graph. What evidence from the existing experiments or error analyses distinguishes difficulty reading transformed labels from difficulty understanding connectivity or planning routes? This distinction is essential to interpreting the reported failures of invariance.
  3. What explains the near-zero counterfactual performance? How were the 144 pairs selected, and why is original-condition accuracy substantially lower than main-benchmark visual accuracy? Beyond matching the stated graph properties, what evidence distinguishes failure to respond to changed connectivity from increased perceptual or search difficulty? Were the original and counterfactual conditions evaluated independently, or did the model receive the original problem and response before answering the counterfactual? These details would materially affect the interpretation of adaptation failure.
Reviewer_N2DqOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents SpatialGraph-Eval, a benchmark for shortest-route finding on spatial graphs. The same graph is shown as text, as an image, transformed, or rewired. Three closed models do worse on images than on text. After rewiring, they rarely find the new route.

Critical strengths

(1) The paired design is clean. The same stored graph is reused across text, image, transform, and rewiring conditions. Every route is checked exactly against the graph.

(2) The rewiring test is useful. In 27.5% to 33.3% of solved pairs, models repeat the old route that uses the removed edge.

Critical weaknesses

(1) The interventions confound graph reading with graph reasoning. Flips and rotations are applied to raw pixels, so node labels are mirrored too, and two models fall to 1.4% under vertical flip. No probe asks models to transcribe the edge list from the image.

(2) The main finding is known, and the new evidence is thin. Prior work such as IsoBench already shows that models solve the same problem better as text than as an image. In the rewiring set, accuracy on the original graphs is only 9.7% to 35.4%, versus 42.8% to 82.8% on the main image set.

Questions for authors

(1) Do the flip and rotation effects remain if graphs are re-rendered from transformed coordinates with upright labels? How accurately can each model transcribe the edge list from the image?

(2) Each cell has 15 aligned and 15 misleading geometry instances. Can you report results for these two splits, and add repeated runs and at least one open model?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.