acceptodds
Withdrawn from ICLR 2027

Beyond Disentanglement: Adaptive Partial Sharing for Spatial Reasoning Control in Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

How should the feature subspaces used to control spatial reasoning in vision– language models be organized across relations: fully shared, relation-selective, or partially shared? We investigate this question through controlled spatial coun- terfactuals that connect changes in visual geometry to internal causal sensitiv- ity and learned low-rank control. Our TriAxis-CF dataset comprises 300 fac- tual/counterfactual image pairs, each reversing one spatial relation while pre- serving the other two, with geometry-derived labels and role-reversed questions. These paired contrasts provide controlled donors for activation interventions. Across three models evaluated under a shared protocol and an independent Qwen3 pilot, horizontal reasoning exhibits a repeated distribution of causal sensitivity across early visual regions, intermediate relation-text positions, and later final- input positions. Increasing visual state differences coexist with diminishing out- put effects, establishing a distinction between representation change and its causal influence on spatial judgments. A comparison of fully shared, relation-selective, and shared–private control in Qwen3 identifies adaptive partial sharing as an effec- tive structure for balancing corrective intervention and answer preservation. Our SP-LoReFT combines a shared branch with relation-specific branches that adapt into partially overlapping subspaces. On held-out internal scenes, it attains the highest accuracy point estimate among the evaluated configurations, improving from 88.33% to 90.56% over shared rank-8 control. Break decreases from 6.18% to 3.89%, while Repair increases from 65.05% to 66.99%. These findings support adaptive partial sharing for spatial reasoning control, with its principal observed benefit arising from improved preservation of already-correct judgments.

Primary area:
interpretability and explainable AI
Keywords:
VLLM, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_cvtGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduced a TriAxis-CF dataset which includes 300 factual/counterfactual image pairs to discover causal represented spatial features. They compared fully shared, relation-selective, and shared–private control in Qwen 3 to find that adaptive partial sharing is the best for error correction and correct answer preservation. The proposed SP-LoReFT demonstrated improved accuracy and decreased breaks.

Critical strengths

The paper is generally well motivated. How spatial reasoning is represented and controlled in VLMs, especially with specific relations, provides useful insights for improving spatial reasoning capabilities. The paper presents a systematic framework including controlled spatial counterfactuals, activation-based causal analysis, and low-rank representation intervention.

Critical weaknesses
  1. Dataset size of 300 is small. The small dataset raises concerns about the representativeness of the mechanistic findings. The diversity of spatial configurations and scenes does not capture real-world spatial reasoning. Given the small evaluation set, the modest improvement results on SP-LoReFT (88.33% to 90.56%) further limit the effectiveness of the conclusion.
  2. The evaluation uses three spatial relationships (far/near, left/right, tall/short) under controlled visual settings, without sufficiently establishing whether the learned interventions transfer to more complex spatial reasoning tasks or real-world images.
Questions for authors
  1. The evaluated Qwen3-VL-8B-Instruct, InternVL3.5-8B, Idefics3-8B-Llama3, and DeepSeekVL-7B-Chat differ in their visual encoders, multimodal fusion mechanisms, and internal architectures. Since the paper investigates the distribution of causal sensitivity across model layers and token positions, could the authors clarify how these architectural differences affect the observed patterns? How are the intervention sites defined and aligned across models with different visual-token representations and fusion mechanisms? To what extent can the findings be generalized to alternative VLM architectures, such as cross-attention-based or late-fusion models?
  2. The paper is only 6 and a half pages, using the full 9 pages to include more comprehensive analysis and results will strengthen the paper.
  3. Minor: the AI USE STATEMENT is supposed to be put before the references.
Reviewer_ftfaOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper suggests a dataset and analysis framework for understanding how the feature subspaces of VLMs should be formulated for spatial reasoning. For this, the authors generate TriAxis-CF dataset which consists of counterfactual image pairs of reversed spatial relations. Using this data, the authors evaluate three VLMs through activation intervention technique. The analysis concludes that out of fully shared, relation-selective and partially shared subspaces, the adaptive partial sharing case is found to be the most effective structure for spatial reasoning.

Critical strengths
  • The suggested TriAxis-CF dataset could be practically useful for evaluating subtle aspects of VLMs on spatial reasoning, and could further be utilized as training data.
Critical weaknesses
  • The importance of the main research question is unclear. The authors ask whether spatial reasoning controllers should use shared, relation-selective, or partially shared feature subspaces. But this doesn't clearly state why this question is practically important, and why we need to understand this.

  • Lack of coherence and unclear structure. The motivation and connection between the analyses and method seem inefficient. While the authors ask a clear question about the shared and relation-specific control, it was difficult to grasp how each section's experiments and analyses coherently answer the proposed research questions. Moreover, the motivation of each analysis is also not sufficiently described in the paper. The paper could be better structured to provide a direct connection between the data, causal analysis, and the final conclusions and takeaways.

  • Overall presentation quality is not convincing. For instance, Figure 3 included extremely small plots, label, and legends, making it difficult to interpret the main results. And the y-axis range is different for each figure. The main figure should be more clear and organized to support the central claims of the paper.

  • Missing related work. The paper misses out some closely related works that should have been thoroughly discussed.

    • Rel3D [1] already introduced minimally contrastive scene pairs in which a spatial relation holds in one scene and fails in the other. TriAxis-CF seems to share its core ideas with this work. The authors should explain more precisely what new diagnostic capability this new dataset enables beyond Rel3D and why a separate dataset is necessary.

    • Why Far Looks Up [2] studies spatial-axis organization within VLM inner representations, identifies vertical-distance entanglement, and relates representation structure to spatial robustness. Could the authors please elaborate on how the proposed method differs from the anlaysis in this work?

  • Controller only used Qwen3-VL-8B-Instruct. This limits the generalizability of the takeaway that adaptive partial sharing is preferable.


[1] Rel3D: A Minimally Contrastive Benchmark for Grounding Spatial Relations in 3D, Goyal et al., NeurIPS 2020

[2] Why Far Looks Up: Probing Spatial Representation in Vision-Language Models, Min et al., ECCV 2026

Questions for authors
  • What are the practical insights that the paper's analysis could bring to the broader area of vision-language understanding? Why is it important to analyze the feature subspaces for spatial reasoning?

  • Why is the main manuscript only 7 pages while the appendix includes much more information? Why didn't the authors include more appendix information in the main text?

Reviewer_HqmXOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors construct a dataset of images with counterfactuals along exactly one of three axes (horizontal, depth, height). They aim to perform activation patching on counterfactual images to find what mechanisms control spatial reasoning along different axes, and how much they overlap. They perform Low-rank Linear Subspace ReFT (LoReFT) to improve spatial reasoning.

Critical strengths

The goal of the paper is interesting, i.e., trying to disentangle how much information is shared between spatial reasoning along different axes. However, due to the substantial weaknesses discussed below, the actual takeaways from the paper's experiments and results are unfortunately far less clear.

Critical weaknesses
  1. Their entire premise is that the only thing that changes is one axis, while the other two remain same. However, there are two major flaws in the premise: 1.1. In Figure 1, look at the height examples. While only the object height is supposed to change from Factual to Counterfactual, keeping horizontal and depth the same, the depth appears to change too. Perhaps the authors meant that the ranking of the objects doesn't change along other axes, but even that appears ambiguous in the height examples (e.g. in the first and second height examples, the depth of the two objects appears to have switched as well). Given that it is a render, perhaps the ranking of the depth did not indeed change, but I believe the minimum delta between depths (which is never discussed) to ensure unambiguity is insufficient. No manual verification was specified, even though the set is small (300 pairs). The authors mention “human acceptance” and “ambiguity flags”, but do not mention a human study, and do not filter out the ambiguous examples. They in fact mention that "ambiguity flags and height-subset lateral irregularities remain" [in the constructed dataset] (L159), which throws significant doubt on the conclusions of the paper based on this dataset. 1.2. There are differences between image pairs beyond just the spatial relation, e.g. “occlusions and other rendering consequences” (L054-55). The authors lose the sole advantage of rendering, i.e. to isolate the changes between images to the spatial relation alone.

  2. The writing quality is insufficient for a venue like ICLR, and is very hard to follow. The introduction contains no definitions of the terms used, and no citations; the main paper is only about 6 pages but the appendix is over 12 (why not reorganize?); and many important points are left unclear (e.g., L159 "ambiguity flags and height-subset lateral irregularities remain" as mentioned in W1.1).

  3. The results seem relatively minor, and their statistical significance is not discussed ("ten of twelve additional correct answers" L101). The uncertainty bands in the plots appears quite large.

Questions for authors

In addition to the questions raised by the weaknesses section above (particularly regarding the dataset construction):

  1. Qwen has the same house, but other models don’t. Why?
  2. Why is only one model in Chinese? What impact does that have?
  3. Why does the constructed dataset contain only 300 pairs, if it’s rendered anyway?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.