acceptodds
Under review as a conference paper at ICLR 2027

Solvable, Yet Still Wrong: An Axis-Wise Perspective on Spatial Reasoning in Vision-Language Models

Abstract

Despite significant progress in Vision-Language Models (VLMs), spatial reasoning, which involves inferring the spatial relationships between objects, remains challenging. Existing benchmarks typically report a single aggregated accuracy per task, making it difficult to determine whether a spatial relation is solvable from the available geometric evidence or how models fail. In practice, different spatial axes (e.g., horizontal, vertical, and depth) require different geometric evidence. We therefore introduce an axis-wise analysis that examines, for each relation, what information makes it solvable and how models fail when relevant information is supplied. Our analysis reveals a gap between what VLMs can spatially infer and what they ultimately answer: models can possess the relevant geometric information yet select a relation from the wrong axis. Through controlled analysis on real-world datasets, we reveal three key findings: i) Horizontal, vertical, and depth relations differ substantially in their geometric solvability and in how accurately VLMs can solve them from the available evidence, showing that spatial reasoning is not uniform across axes. ii) We observe that, when VLMs make mistakes, they often select a relation from an incorrect axis, even when at least one geometrically valid spatial relation is among the options. Interestingly, restricting the multi-axis options to the correct axis substantially improves accuracy, highlighting axis selection as an important error source. iii) This axis-wise failure can be alleviated by supplying axis-specific geometric evidence. Providing geometry as the relation implied by grounded objects significantly improves accuracy, whereas directly providing object coordinates sometimes even hurts performance, indicating that how geometric evidence is represented matters. Based on these findings, we introduce AxisCue, a training-free, axis-aware inference method that allows axis-conditioned spatial reasoning. AxisCue identifies the queried axis, grounds the reference objects, and converts them into axis-specific geometric evidence, eliciting relevant geometric information from the model to improve accuracy. Overall, our analysis demonstrates that an axis-wise perspective exposes failure modes hidden by aggregated spatial accuracy and provides a principled basis for deriving simple, targeted improvements to spatial reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Spatial Reasoning, Axis-Wise Analysis, Axis-Aware Inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HsDWOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this work, the authors introduce and perform an axis-wise analysis of capital-reasoning, that examines, for each relation, what information makes it solvable and how models fail when relevant information is given. Their analysis reveals different axis-related problems that lead to poor performance in VLMs. Based on these findings, they also introduce AxisCue, a training-free, axis-aware inference method that allows axis-conditioned spatial reasoning; which identifies the queried axis, grounds the reference objects, and converts them into axis-specific geometric evidence, eliciting relevant geometric information from the model to improve accuracy.

Critical strengths

Introduced an interesting and interpretable axis-wise decomposition (in X,Y,Z axes) to understand and diagnose spatial reasoning failures; which is very intuitive and aligns with how we humans reason spatially.

Critical weaknesses

Two of my most critical weaknesses are:

  1. The authors claim that AxisCue improves visual spatial reasoning by providing geometric evidence on different axes. But, it seems to give away the answer; and the authors did not separate axis-aware reasoning from textual answer-following or grounding quality. For example, as shown in Figure 3(b), the clue from AxisCue gives away the final answer directly (Vertical (0=top to 100=bottom): the person spans 65-99 and the bowl spans 26-63, so the person is below the bowl). So, after this, there is no need for the model to perform any spatial "reasoning" on its own; it can answer just by reading the text. It shows that the model is treating the cue as soft text prior, rather than actively verifying and analyzing it with the image.
  2. Another thing is, I think the paper's more motivation and diagnostic foundation presented in Section 3, has some major flaws. Firstly, in Table 3, in the oracle-axis setting, the improvements are huge; but the random guess baseline also raised to 50% from 20-25% before. Also, the solvability baseline on Z-axis relies on a heuristics that lower objects are nearer; but it is not always true and can differ heavily, depending on camera angles, tilt, and scene layouts; for example, objects on higher shelves. So, overall, the grounding motivation seems unproven to me. Also, the results undermine AxisCue and related motivation too.
Questions for authors
  1. Regarding critical weakness 1, can you please provide some more analysis? Like, a controlled analysis with three things: a cue that gives axis-projected position, but not the relation word; an oracle-box version vs the predicted-box version; and a length-matched, axis-mismatched cue. These will help me to understand if any actual reasoning is happening.
  2. Regarding critical weakness 1, can you please strengthen the claims and results? Table 1 shows averaged results of 4 models, why?
  3. The datasets used are very old, weak, and has easier questions; there are some recent benchmarks released in 2026, like SpatiaLab, OmniSpatial and more.This is why frontier models in Table 5 performed around 80%; where in latest benchmarks they score around 40-60% only. So, can you please include some newer/latest benchmarks too?
  4. Another related case is, in Z-axis analysis, it is done on a very small sample size (228 items from WhatsUp Z axis), CV-Bench is also saturated. The dataset selection is poor and requires full experiments on newer, harder and complex datasets. Can you please include some newer/latest datasets too?
  5. Another related question is, what happens to multi-hop questions? Recent works also show that models are much weaker in multi-hop questions; but it is not clear if AxisCue can operate on complex multi-hop questions reliably. In that sense, it seems the contribution is not timely and may not be that useful in current systems as even GPT-5-mini, Claude-Sonnet-4.5, Qwen3.5-122B-A10B got 80% in EmbSpatial; and latest GPT-6.1 and Claude-5.5, Qwen3.8 will perform much much better without any support. It weakens the overall purpose of the paper and nullifies its contributions. Please check, add new experiments, explain and justify the contributions properly.
  6. The paper says they reuse KV cache to reduce inference overhead, but no supporting computational overhead/latency analysis was done to show and justify the extra overhead.
2 comments
AuthorsOpenReview ↗

Response to Reviewer HsDW (1/2)

W1 & Q1 & Q2: Separating axis-aware reasoning from answer-following and grounding.

We would like to clarify that the paper already separates these factors through controlled experiments.

(i) Axis-projected position without the relation word (Q1, first request). The "+ Axis-position" row in Table 4 (prompt in Line 292) supplies axis-projected position based on ground-truth boxes without stating the relation. It does not consistently improve accuracy.

(ii) Oracle boxes versus predicted boxes (Q1, second request). Sec. 3.3 constructs every cue from ground-truth boxes, whereas Sec. 4 uses the model's predicted boxes (Tables 5-9). With predicted boxes, AxisCue improves over the baseline on every model and benchmark (Table 5), yet the boxes alone do not produce the gain, since answers derived directly from them fall below the baseline in all four cases (Table 9). The improvement therefore comes from the axis-specific cue rather than from grounding alone.

(iii) Answer-following with a length-matched, axis-mismatched cue (Q1, third request). We interpret this as a cue whose stated relation does not match the true one on the queried axis. The "+ AxisCue, relation flipped" setting in Table 9 implements this by swapping the objects' projected positions, so the cue states an incorrect relation while preserving its format and length. For Qwen2.5-VL-7B and Qwen3-VL-8B, accuracy remains near baseline on WhatsUp (77.1 and 76.2 vs. 77.0 and 77.3) and drops by 5.8 and 10.5 on EmbSpatial. This suggests that VLMs do not blindly follow an incorrect cue but instead evaluate it against visual evidence. If the reviewer intended a different setting, we are glad to provide additional results.

For clarity, Table 16 in Appendix C.6 provides a more direct comparison.

W2: Section 3's chance accuracy and Z-axis solvability.

We would like to clarify that both concerns are already addressed in the paper.

Chance accuracy. Our central comparison between oracle-axis and cross-axis distractors is chance-matched, and the performance drop demonstrates interference from cross-axis options. Table 3 reports baseline chance accuracy for each axis (21.5, 23.0, and 24.1 on WhatsUp and 25.0 on EmbSpatial) for reference. Both oracle-axis and cross-axis settings contain two options, yielding the same chance accuracy of 50.0. They differ only in the distractor's axis: the queried axis (e.g., left vs. right) or another axis (e.g., left vs. above). On the WhatsUp X-axis, accuracy drops from 97.9, 98.3, 97.5, and 98.9 to 68.7, 75.1, 71.0, and 54.7 across the four models, highlighting the impact of cross-axis distractors.

The Z-axis rule. The limitation of the perspective prior does not affect our key finding that geometric representation substantially affects VLM accuracy. This finding is independently supported by the X- and Y-axis analyses, which require no perspective assumption (Table 4). In Sec. 3.1, we use the perspective prior only to estimate how much 2D geometry determines depth relations when depth annotations are unavailable. Its accuracy varies across WhatsUp, EmbSpatial, and the CV-Bench depth subset (99.6, 67.3, and 83.8, respectively; Table 1), reflecting its dependence on scene characteristics. We acknowledge that the prior is not universally valid and treat it as a reference for depth-related solvability, without claiming that the resulting cues fully resolve depth relations.

Q2: The average results of 4 models in Table 1.

We report averaged accuracy because the conclusion holds for every model. Table 4 provides per-model results, where every model falls below the 2D object-box reference on both benchmarks.

AuthorsOpenReview ↗

Response to Reviewer HsDW (2/2)

Q3 & Q4: Evaluation on more recent dataset and Z-axis samples.

The benchmark choice is the key to reveal a new finding that VLMs still exhibit substantial limitations even on relatively simple questions. If we used more complicated and entangled benchmarks, the factors and finding may be more unclearly isolated due to less controlled experiments, which is critically undesirable for analysis. Also, we follow prior work to evaluate on widely used spatial reasoning benchmarks, which are well-established and suitable for diagnosing VLM capabilities.

Nonetheless, in response to the reviewer's comment, we have additionally evaluated on SpatialMQA [C2] (ACL 2025) and included the results in Appendix C.5. SpatialMQA is a recent and more challenging benchmark covering both out-of-image perspective questions (posed from the camera's viewpoint) and in-image perspective questions (posed from the viewpoint of an object or another observer). We also report accuracy on its Z-axis (depth) subset of 312 questions, which is drawn from both perspective splits. Table R2 shows that AxisCue consistently improves over baselines, including Qwen3.5-122B-A10B and Qwen3.8-27B. This further validates the effectiveness of our method on a more challenging dataset and larger models.

[C2] Can Multimodal Large Language Models Understand Spatial Relations? ACL 2025.

Table R2: Results on SpatialMQA. Out-of-image and In-image are the official perspective splits in [C2], and Overall covers both. The Z-axis (depth) subset consists of the front/behind questions, which are drawn from both perspectives and reported separately.

ModelSettingOut-of-imageIn-imageOverallZ-axis (depth)
Qwen2.5-VL-7BBaseline51.725.136.230.8
+ AxisCue59.4 (+7.7)26.2 (+1.1)40.1 (+3.9)35.6 (+4.8)
Qwen3-VL-8BBaseline55.227.539.135.3
+ AxisCue65.0 (+9.8)29.3 (+1.8)44.2 (+5.1)42.9 (+7.6)
LLaVA-OV-2-8BBaseline54.822.636.114.4
+ AxisCue62.7 (+7.9)26.6 (+4.0)41.7 (+5.6)33.0 (+18.6)
Qwen3.5-122B-A10BBaseline66.330.145.350.0
+ AxisCue74.3 (+8.0)31.7 (+1.6)49.5 (+4.2)52.9 (+2.9)
Qwen3.8-27BBaseline65.433.046.652.2
+ AxisCue73.8 (+8.4)35.0 (+2.0)51.3 (+4.7)57.1 (+4.9)

Q5: Contribution timeliness and multi-hop questions.

Contribution. We would like to highlight that Reviewers u6Tc and 7r8Z recognized the value of our analysis. Reviewer u6Tc described the cross-axis error analysis as an interesting and unexpected finding, while Reviewer 7r8Z noted that our controlled analysis reveals useful behavioral insights beyond aggregate accuracy. Our contributions lie in the axis-wise diagnosis and the training-free method it motivates. We further support these contributions with new experiments on SpatialMQA and two larger models, as detailed below.

Generalization to recent frontier models. We additionally evaluate AxisCue on SpatialMQA using Qwen3.5-122B-A10B and Qwen3.8-27B, with results also reported in Appendix C.5. Table R2 shows that (i) scaling up does not eliminate spatial reasoning failures, as the two models achieve only 45.3 and 46.6 overall accuracy, respectively, and (ii) AxisCue improves their accuracy by 4.2 and 4.7, respectively, with consistent gains across subsets. These results indicate that supplying axis-specific evidence remains effective at this scale.

Multi-hop questions. AxisCue targets relation questions where the queried axis is identifiable from the question and options. Multi-hop reasoning requires composing relations across steps, which is beyond the scope of this work.

Q6: Latency.

Latency is already reported in Table 12 (Appendix B). On a single A100, KV-cache reuse reduces AxisCue's per-item latency from 1,878 to 1,532 ms, compared with 618 ms for the baseline.

Reviewer_7r8ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AxisCue improves VLM spatial-relation answering by predicting object boxes, converting them into axis-specific text, and answering again without training. The paper analyzes axis-dependent errors and compares geometric evidence representations across models and benchmarks.

Critical strengths
  1. Error distributions, matched two-option comparisons, and relation-flip interventions provide useful behavioral insights beyond aggregate accuracy (Tables 2–3, 9).
  2. All four models improve overall accuracy on both main benchmarks. Fixed-box comparisons support the value of evidence representation, alongside a rule-based baseline and grounding/cost analysis (Tables 5, 8–9; Appendix B–C).
Critical weaknesses
  1. Some interpretations are stronger than the evidence: heuristic accuracy is not an information ceiling, and cross-axis answers do not uniquely identify an internal axis-selection failure (§3; Appendix A). The conclusions should distinguish observed behavior from mechanism.
  2. Table 10's counting scope is unclear. Qwen2.5's EmbSpatial errors total 365 there, while Table 5's 71.0% accuracy over 3,640 items implies approximately 1,056 errors. Different subsets or counting conventions may explain this discrepancy.
Questions for authors

What sample set and counting convention does Table 10 use, and how does it relate to Tables 2 and 5?

Reviewer_u6TcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper conducts an axis-wise analysis of spatial reasoning, disentangling different axes that are aggregated in common spatial benchmarks like WhatsUp. It finds that models often choose the wrong axis entirely, and that their performance is much better when the options are restricted to a single axis. It then provides different hints to the model, and puts forth a training-free method to improve VLMs’ spatial reasoning by having it perform two passes: one to ground the objects, then another to identify the spatial relation between them.

Critical strengths
  1. Disentangling axes in spatial reasoning is an interesting question
  2. Interesting and unexpected finding, that cross-axis errors are so abundant
  3. The paper is well-written and quite clear
Critical weaknesses
  1. The biggest weakness I see of this paper is that the “hint” effectively provides the answer to the model. It effectively turns the visual spatial reasoning task into a text-solvable task that just requires math. Consider the “axis-position” hint, which is also used in their AxisCue method. It provides the horizontal coordinate of each of the two objects, then the vertical coordinates. The model does not at all need to attend to the image to answer the question, only to perform some basic math on the coordinates. Even worse, the “axis-relation” hint literally provides the answer (L317 “the cabinet is to the right of the bag”); then what is the VLM doing? Also, why aren’t VLMs achieving 100% with this hint? I strongly believe that even if the image is not provided, the VLM could correctly answer the question just given the hint alone (an experiment not provided in the paper). In which case, the proposed method is not improving VLMs’ spatial reasoning, but just using their grounding capabilities and turning the problem into a text-solvable problem, which is much less interesting. The authors mention (L146) that existing methods guide VLMs where to look rather than “provide geometric evidence”, but I would argue that the former methods can more fairly say they improve VLMs’ spatial reasoning capabilities. Further, even after the authors' method, there are still substantial cross-axis errors on WhatsUp (Table 10).
  2. The paper claims that ~10% of X and Y axis questions in WhatsUp are not solvable as the bounding box coordinates can be both right and front of the other object. However, COCO-spatial and GQA-spatial have only two options, not six. In fact, no subset of WhatsUp has 6 options, unlike what is stated in Table 2. This raises the question of whether the authors changed the evaluation from WhatsUp — if the models in the work are all evaluated with the same method, the findings would still hold, but it should be specified clearly. Further, calling parts of the dataset “unsolvable” after changing the options is inaccurate. Given that the premise of the paper is to disentangle axes, it feels counter-intuitive to take a benchmark that already has disentangled axes (COCO-spatial, GQA-spatial in WhatsUp), entangle them by adding options, and then disentangle them again.
Questions for authors
  1. My biggest question I hope to see addressed in the rebuttal relates to Weakness 1. Are the authors merely turning the spatial reasoning question into a text-only task? Does the VLM even need to see the image for the second pass? If the VLM can answer the second pass without access to the image (i.e., if it can select the answer when just the AxisCue is provided with no image), then I think the contribution of the paper isn’t really “improving VLMs’ spatial reasoning”, and the paper should be significantly reframed into something more like CoT: breaking the question down into two fixed/hard-coded steps to improve performance.
  2. In Table 3, Random chance accuracy is ~25%. Given that the options were reduced to single-axis, i.e. 2 options, shouldn’t this be ~50%?
  3. Why are there still so many cross-axis errors on WhatsUp after implementing AxisCue (Table 10)?
  4. The most interesting research question raised by this work is: if VLMs can accurately ground objects in the image, and can perform math on the (disentangled) coordinates, why are they unable to answer the spatial relation question to start with? I don't expect an answer to this, but I believe it would have been a more interesting research question than merely sticking existing grounding and math capabilities together to answer a spatial reasoning question.

Comments on the paper

3 comments

Dear Reviewers,

It is important that we thoroughly discuss our perspectives before reaching a final recommendation. Now that the authors' rebuttal is available, I kindly ask each of you to respond in the discussion and clarify whether your original concerns have been addressed or still remain.

Thank you in advance for your timely engagement before the discussion period ends.

Best, Your AC

Thank you for the response.

I still disagree with some of the points.

Firstly, regarding the multi-hop claim, I agree that you did not make that claim. However, this is not TMLR, where I would judge the work solely on whether the claims are supported by the evidence. I also consider broader aspects such as novelty and impact. To me, this work feels 1.5 to 2 years late. Had it been submitted to ICLR 2025, the contribution might have been much stronger. However, as models have improved and tasks have become harder, I do not feel this work adds much value at this point. This is why I kept insisting on this aspect.

For example, in the original work, you used EmbSpatial and WhatsUp. WhatsUp was accepted in EMNLP 2023; meaning the work was developed in mid -2023; EmbSpatial was accepted in ACL 2024, so it was developed in late 2023. Now, it is end of 2026 (around 3 year gap); and a lot better, stronger, and more capable models and benchmarks there now.

Secondly, regarding the contributions, it is still unclear to me why this is a strong work. In my previous response, I explicitly asked to answer that, despite all the weaknesses and limited practical impact or use, what makes this method or work's contribution strong enough for ICLR? I still cannot see a strong enough answer, and I am sorry.

Thirdly, nobody judged you on whether your training-free method resolves most of the questions that the base model gets wrong. That was a metaphor I used to illustrate how the very logic you are using to argue for scaling (that scaling still leaves a good portion of errors unresolved) could work against your own argument. I was pointing out the logical fallacy in your previous response and I hoped you'd understand that.

We thank the reviewer for the clarification and address each point below.

On timeliness. The age of a benchmark does not imply that the problem it exposes has been solved. We do evaluate both stronger models and newer benchmarks, and the same failures persist. The four recent models evaluated in the paper fall 8.8–21.4 points short of the performance supported by the available geometry (Table 1). Moreover, the two frontier models added during the discussion achieve only 45.3 and 46.6 on SpatialMQA (ACL 2025), respectively, and our method improves both. The axis-wise analysis is our contribution, not something provided by the datasets. We use benchmarks with fully annotated object geometry because isolating what the available evidence supports requires controlled conditions. If the reviewer is aware of a more recent dataset that supports the same analysis, we would be glad to evaluate on it.

On contribution. We would appreciate clarification on which aspect of the contribution is considered insufficient. Regarding novelty, to our knowledge, no prior work has reported that VLMs possess the geometric information required to determine a spatial relation yet predict a relation along a different axis, or that identical geometric evidence can yield substantially different accuracy depending on how it is expressed. Both findings are supported by controlled experiments. Regarding impact, our method requires no training or changes to model parameters, improves every model and benchmark we evaluate, and outperforms AdapVis (ICML 2025), the closest training-free method, by 15.0 points on WhatsUp (Table 7). It also raises Qwen3-VL-8B to 44.2 on SpatialMQA, approaching the 45.3 achieved by Qwen3.5-122B-A10B at a small fraction of the computational cost.

On the scaling argument. We thank the reviewer for clarifying the intent. We did not argue that scaling is ineffective. Instead, we argue that spatial reasoning remains unresolved even at larger model scales, as supported by the SpatialMQA results. This makes our analysis relevant regardless of continued model scaling.

Overall, this work identifies a previously unreported failure mode, isolates it under chance-matched and geometry-matched conditions, and converts that finding into a method that improves every model and benchmark we evaluate at no training cost, including frontier models on a recent benchmark (SpatialMQA, ACL 2025). We believe this constitutes a substantive contribution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.