- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
In this work, the authors introduce and perform an axis-wise analysis of capital-reasoning, that examines, for each relation, what information makes it solvable and how models fail when relevant information is given. Their analysis reveals different axis-related problems that lead to poor performance in VLMs. Based on these findings, they also introduce AxisCue, a training-free, axis-aware inference method that allows axis-conditioned spatial reasoning; which identifies the queried axis, grounds the reference objects, and converts them into axis-specific geometric evidence, eliciting relevant geometric information from the model to improve accuracy.
Critical strengths
Introduced an interesting and interpretable axis-wise decomposition (in X,Y,Z axes) to understand and diagnose spatial reasoning failures; which is very intuitive and aligns with how we humans reason spatially.
Critical weaknesses
Two of my most critical weaknesses are:
- The authors claim that AxisCue improves visual spatial reasoning by providing geometric evidence on different axes. But, it seems to give away the answer; and the authors did not separate axis-aware reasoning from textual answer-following or grounding quality. For example, as shown in Figure 3(b), the clue from AxisCue gives away the final answer directly (Vertical (0=top to 100=bottom): the person spans 65-99 and the bowl spans 26-63, so the person is below the bowl). So, after this, there is no need for the model to perform any spatial "reasoning" on its own; it can answer just by reading the text. It shows that the model is treating the cue as soft text prior, rather than actively verifying and analyzing it with the image.
- Another thing is, I think the paper's more motivation and diagnostic foundation presented in Section 3, has some major flaws. Firstly, in Table 3, in the oracle-axis setting, the improvements are huge; but the random guess baseline also raised to 50% from 20-25% before. Also, the solvability baseline on Z-axis relies on a heuristics that lower objects are nearer; but it is not always true and can differ heavily, depending on camera angles, tilt, and scene layouts; for example, objects on higher shelves. So, overall, the grounding motivation seems unproven to me. Also, the results undermine AxisCue and related motivation too.
Questions for authors
- Regarding critical weakness 1, can you please provide some more analysis? Like, a controlled analysis with three things: a cue that gives axis-projected position, but not the relation word; an oracle-box version vs the predicted-box version; and a length-matched, axis-mismatched cue. These will help me to understand if any actual reasoning is happening.
- Regarding critical weakness 1, can you please strengthen the claims and results? Table 1 shows averaged results of 4 models, why?
- The datasets used are very old, weak, and has easier questions; there are some recent benchmarks released in 2026, like SpatiaLab, OmniSpatial and more.This is why frontier models in Table 5 performed around 80%; where in latest benchmarks they score around 40-60% only. So, can you please include some newer/latest benchmarks too?
- Another related case is, in Z-axis analysis, it is done on a very small sample size (228 items from WhatsUp Z axis), CV-Bench is also saturated. The dataset selection is poor and requires full experiments on newer, harder and complex datasets. Can you please include some newer/latest datasets too?
- Another related question is, what happens to multi-hop questions? Recent works also show that models are much weaker in multi-hop questions; but it is not clear if AxisCue can operate on complex multi-hop questions reliably. In that sense, it seems the contribution is not timely and may not be that useful in current systems as even GPT-5-mini, Claude-Sonnet-4.5, Qwen3.5-122B-A10B got 80% in EmbSpatial; and latest GPT-6.1 and Claude-5.5, Qwen3.8 will perform much much better without any support. It weakens the overall purpose of the paper and nullifies its contributions. Please check, add new experiments, explain and justify the contributions properly.
- The paper says they reuse KV cache to reduce inference overhead, but no supporting computational overhead/latency analysis was done to show and justify the extra overhead.
2 comments
Response to Reviewer HsDW (2/2)
Q3 & Q4: Evaluation on more recent dataset and Z-axis samples.
The benchmark choice is the key to reveal a new finding that VLMs still exhibit substantial limitations even on relatively simple questions. If we used more complicated and entangled benchmarks, the factors and finding may be more unclearly isolated due to less controlled experiments, which is critically undesirable for analysis. Also, we follow prior work to evaluate on widely used spatial reasoning benchmarks, which are well-established and suitable for diagnosing VLM capabilities.
Nonetheless, in response to the reviewer's comment, we have additionally evaluated on SpatialMQA [C2] (ACL 2025) and included the results in Appendix C.5. SpatialMQA is a recent and more challenging benchmark covering both out-of-image perspective questions (posed from the camera's viewpoint) and in-image perspective questions (posed from the viewpoint of an object or another observer). We also report accuracy on its Z-axis (depth) subset of 312 questions, which is drawn from both perspective splits. Table R2 shows that AxisCue consistently improves over baselines, including Qwen3.5-122B-A10B and Qwen3.8-27B. This further validates the effectiveness of our method on a more challenging dataset and larger models.
[C2] Can Multimodal Large Language Models Understand Spatial Relations? ACL 2025.
Table R2: Results on SpatialMQA. Out-of-image and In-image are the official perspective splits in [C2], and Overall covers both. The Z-axis (depth) subset consists of the front/behind questions, which are drawn from both perspectives and reported separately.
| Model | Setting | Out-of-image | In-image | Overall | Z-axis (depth) |
|---|---|---|---|---|---|
| Qwen2.5-VL-7B | Baseline | 51.7 | 25.1 | 36.2 | 30.8 |
| + AxisCue | 59.4 (+7.7) | 26.2 (+1.1) | 40.1 (+3.9) | 35.6 (+4.8) | |
| Qwen3-VL-8B | Baseline | 55.2 | 27.5 | 39.1 | 35.3 |
| + AxisCue | 65.0 (+9.8) | 29.3 (+1.8) | 44.2 (+5.1) | 42.9 (+7.6) | |
| LLaVA-OV-2-8B | Baseline | 54.8 | 22.6 | 36.1 | 14.4 |
| + AxisCue | 62.7 (+7.9) | 26.6 (+4.0) | 41.7 (+5.6) | 33.0 (+18.6) | |
| Qwen3.5-122B-A10B | Baseline | 66.3 | 30.1 | 45.3 | 50.0 |
| + AxisCue | 74.3 (+8.0) | 31.7 (+1.6) | 49.5 (+4.2) | 52.9 (+2.9) | |
| Qwen3.8-27B | Baseline | 65.4 | 33.0 | 46.6 | 52.2 |
| + AxisCue | 73.8 (+8.4) | 35.0 (+2.0) | 51.3 (+4.7) | 57.1 (+4.9) |
Q5: Contribution timeliness and multi-hop questions.
Contribution. We would like to highlight that Reviewers u6Tc and 7r8Z recognized the value of our analysis. Reviewer u6Tc described the cross-axis error analysis as an interesting and unexpected finding, while Reviewer 7r8Z noted that our controlled analysis reveals useful behavioral insights beyond aggregate accuracy. Our contributions lie in the axis-wise diagnosis and the training-free method it motivates. We further support these contributions with new experiments on SpatialMQA and two larger models, as detailed below.
Generalization to recent frontier models. We additionally evaluate AxisCue on SpatialMQA using Qwen3.5-122B-A10B and Qwen3.8-27B, with results also reported in Appendix C.5. Table R2 shows that (i) scaling up does not eliminate spatial reasoning failures, as the two models achieve only 45.3 and 46.6 overall accuracy, respectively, and (ii) AxisCue improves their accuracy by 4.2 and 4.7, respectively, with consistent gains across subsets. These results indicate that supplying axis-specific evidence remains effective at this scale.
Multi-hop questions. AxisCue targets relation questions where the queried axis is identifiable from the question and options. Multi-hop reasoning requires composing relations across steps, which is beyond the scope of this work.
Q6: Latency.
Latency is already reported in Table 12 (Appendix B). On a single A100, KV-cache reuse reduces AxisCue's per-item latency from 1,878 to 1,532 ms, compared with 618 ms for the baseline.
Response to Reviewer HsDW (1/2)
W1 & Q1 & Q2: Separating axis-aware reasoning from answer-following and grounding.
We would like to clarify that the paper already separates these factors through controlled experiments.
(i) Axis-projected position without the relation word (Q1, first request). The "+ Axis-position" row in Table 4 (prompt in Line 292) supplies axis-projected position based on ground-truth boxes without stating the relation. It does not consistently improve accuracy.
(ii) Oracle boxes versus predicted boxes (Q1, second request). Sec. 3.3 constructs every cue from ground-truth boxes, whereas Sec. 4 uses the model's predicted boxes (Tables 5-9). With predicted boxes, AxisCue improves over the baseline on every model and benchmark (Table 5), yet the boxes alone do not produce the gain, since answers derived directly from them fall below the baseline in all four cases (Table 9). The improvement therefore comes from the axis-specific cue rather than from grounding alone.
(iii) Answer-following with a length-matched, axis-mismatched cue (Q1, third request). We interpret this as a cue whose stated relation does not match the true one on the queried axis. The "+ AxisCue, relation flipped" setting in Table 9 implements this by swapping the objects' projected positions, so the cue states an incorrect relation while preserving its format and length. For Qwen2.5-VL-7B and Qwen3-VL-8B, accuracy remains near baseline on WhatsUp (77.1 and 76.2 vs. 77.0 and 77.3) and drops by 5.8 and 10.5 on EmbSpatial. This suggests that VLMs do not blindly follow an incorrect cue but instead evaluate it against visual evidence. If the reviewer intended a different setting, we are glad to provide additional results.
For clarity, Table 16 in Appendix C.6 provides a more direct comparison.
W2: Section 3's chance accuracy and Z-axis solvability.
We would like to clarify that both concerns are already addressed in the paper.
Chance accuracy. Our central comparison between oracle-axis and cross-axis distractors is chance-matched, and the performance drop demonstrates interference from cross-axis options. Table 3 reports baseline chance accuracy for each axis (21.5, 23.0, and 24.1 on WhatsUp and 25.0 on EmbSpatial) for reference. Both oracle-axis and cross-axis settings contain two options, yielding the same chance accuracy of 50.0. They differ only in the distractor's axis: the queried axis (e.g., left vs. right) or another axis (e.g., left vs. above). On the WhatsUp X-axis, accuracy drops from 97.9, 98.3, 97.5, and 98.9 to 68.7, 75.1, 71.0, and 54.7 across the four models, highlighting the impact of cross-axis distractors.
The Z-axis rule. The limitation of the perspective prior does not affect our key finding that geometric representation substantially affects VLM accuracy. This finding is independently supported by the X- and Y-axis analyses, which require no perspective assumption (Table 4). In Sec. 3.1, we use the perspective prior only to estimate how much 2D geometry determines depth relations when depth annotations are unavailable. Its accuracy varies across WhatsUp, EmbSpatial, and the CV-Bench depth subset (99.6, 67.3, and 83.8, respectively; Table 1), reflecting its dependence on scene characteristics. We acknowledge that the prior is not universally valid and treat it as a reference for depth-related solvability, without claiming that the resulting cues fully resolve depth relations.
Q2: The average results of 4 models in Table 1.
We report averaged accuracy because the conclusion holds for every model. Table 4 provides per-model results, where every model falls below the 2D object-box reference on both benchmarks.