acceptodds
Under review as a conference paper at ICLR 2027

S-Space: Exploring Spatial Workspace in Multimodal Language Models

10 views

Abstract

Multimodal language models (MLLMs) exhibit promising spatial reasoning capabilities, yet the internal mechanisms supporting them remain poorly understood. We uncover S-Space, a linear representation subspace in which object-token representations encode continuous coordinates along horizontal, vertical, and distance axes. Although we identify these axes by differentiating spatial-answer scores with respect to object-token activations, S-Space is intrinsic rather than induced by specific data or prompts, remaining readable under non-spatial questions. We show that S-Space functions as an internal spatial workspace of MLLMs. Perception encodes object locations into it, forming spatial representations that can be directly read out to support accurate spatial judgments. Internal reasoning transforms it for changing viewpoints. Verbal responses selectively access it, evidenced by causal interventions that alter spatial judgments while largely preserving non-spatial responses. Case studies further show that S-Space assigns locations to unseen entities and the observer, and connects physical directions with abstract linguistic meanings, suggesting both the capabilities and limitations of large-scale language training. Finally, we use S-Space as a diagnostic tool for spatial reasoning failures. Direction-word readouts reveal systematic coupling between cardinal and image-relative reference frames. More importantly, a perception-computation decomposition exposes a key bottleneck: MLLMs can form capable spatial representations, yet fall short in the spatial transformations required to reason reliably across viewpoint changes.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, spatial reasoning, multimodal large language models, VLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_u2PBOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes S-Space, a three-dimensional linear subspace in the intermediate hidden representations of multimodal large language models (MLLMs). By computing gradients of spatial answer-logit contrasts with respect to object-token representations, the authors identify horizontal, vertical, and depth directions, enabling continuous spatial coordinates to be read out without additional training.

The authors validate S-Space across multiple MLLMs and spatial reasoning benchmarks, examining its coordinate accuracy, robustness across prompts, and causal influence on model predictions. They further explore its representations of unseen entities and abstract spatial concepts. Through explicit coordinate rotations, chain-of-thought (CoT) tracking, and reference-frame analysis, the paper distinguishes spatial perception from subsequent spatial computation. The results suggest that MLLMs can encode relatively accurate spatial information yet still struggle with viewpoint transformations and reference-frame reasoning.

Critical strengths

1. A unified and interpretable framework for analyzing spatial representations.

The paper uses a simple gradient-based method to identify spatial directions and applies the same S-Space representation to coordinate readout, causal intervention, cross-prompt analysis, and reasoning-process tracking. The causal intervention experiments are particularly convincing, as they compare targeted coordinate edits with random perturbations and examine their effects on both spatial and non-spatial predictions. The experiments are systematic and mutually supportive, providing evidence beyond simple correlations between internal representations and ground-truth positions.

2. Valuable insights into the mechanisms underlying spatial reasoning failures.

A key contribution is the distinction between spatial perception and spatial computation. On SpinBench, reading out S-Space coordinates and applying explicit viewpoint transformations substantially outperforms both direct prediction and CoT reasoning. Together with the analyses of reference-frame confusion and representation changes during reasoning, these results provide useful insights into the limitations of MLLM spatial reasoning. This mechanistic understanding is more informative than simply improving benchmark accuracy.

Critical weaknesses

1. Insufficient empirical comparison with closely related prior work.

Prior studies, particularly Kang et al. [1] and Gurnee et al. [2], have investigated linear spatial representations in object tokens and Jacobian-based internal workspaces, respectively. Although the paper cites and discusses these studies, the distinctions are primarily conceptual, without direct comparisons under matched experimental settings.

Consequently, it remains unclear whether the proposed gradient-based identification method provides meaningful advantages over existing approaches in readout accuracy, robustness, or causal effectiveness. This makes it difficult to assess the methodological novelty.

2. Some claims about internal spatial computation require stronger evidence.

The substantial gains from S-Space + Rotation rely on predefined viewpoint transformations. Moreover, SpinBench mainly involves discrete right-angle rotations, implemented through coordinate permutations and sign changes. These results do not necessarily establish that S-Space supports more general three-dimensional spatial computations.

Additionally, the CoT tracking experiments use controlled reasoning sequences containing correct spatial relations, which may not fully reflect the model's internal computations during autonomous reasoning. While these experiments demonstrate that spatial information can be extracted and used effectively, how the model itself performs spatial transformations remains less clear.

Questions for authors

1. Comparison with existing spatial representation methods.

Could the authors provide a direct comparison between S-Space and the spatial direction extraction method of Kang et al. [1], using the same models and datasets?

  • Does the gradient-based method offer measurable advantages in coordinate readout, cross-prompt generalization, or SpinBench viewpoint transformations?
  • How much does the query-reference gradient difference contribute compared with simply averaging gradients?

2. Generalization to more complex spatial transformations.

SpinBench mainly evaluates discrete right-angle viewpoint changes, where rotations can be implemented through coordinate permutations and sign changes.

  • Is there evidence that S-Space supports more general, continuous viewpoint transformations?
  • This would help establish whether S-Space captures a geometrically consistent three-dimensional representation rather than three independently readable spatial directions.

3. Spatial representation changes during autonomous reasoning.

Section 5.2 analyzes S-Space using controlled CoT sequences containing correct spatial relations.

  • Do similar coordinate transformations emerge during the model's self-generated reasoning, particularly when comparing correct and incorrect reasoning trajectories?
  • Such evidence would help clarify whether S-Space participates in the model's actual spatial computation process.

References

[1] Kang, R., Chen, H., Gkioxari, G., and Perona, P. Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models. International Conference on Learning Representations (ICLR), 2026. https://arxiv.org/abs/2601.12626

[2] Gurnee, W., Sofroniew, N., Pearce, A., Piotrowski, M., Kauvar, I., Chen, R., Soligo, A., Bogdan, P., Ong, E., Wang, R., Thompson, B., Abrahams, D., Kantamneni, S., Ameisen, E., Batson, J., and Lindsey, J. Verbalizable Representations Form a Global Workspace in Language Models. Transformer Circuits, 2026. https://transformer-circuits.pub/2026/workspace/index.html

4 comments

We sincerely thank Reviewer u2PB for the constructive review and insightful questions. We organize our response into Part 1/2 (Q1a below) and Part 2/2 (Q1b, Q2, and Q3), alongside the global response.

Q1a (W1 / Q1): Comparisons with Spatial ID and Jacobian Lens

We benchmark S-Space against Kang et al.'s Spatial ID [1] and Gurnee et al.'s Jacobian Lens [2] on identical checkpoints, splits, and same-layer object-token hidden states (hq,hr)(h_q, h_r) (global response Q1; Sec. 3.1-3.2, Appendix C.2-C.3). Because Spatial ID [1] relies on a synthetic 4x4 grid dataset and extracts only 2D axes without depth, we conduct a two-part readout comparison against it—first (Tab. 9) on 2D tasks and second (Tab. 10) adapting it to natural COCO images with depth labels—followed by causal swaps (Tab. 16, Fig. 13) and pretrained Jacobian Lenses [2] (Tab. 11):

  1. 2D readout, cross-prompt transfer, and SpinBench vs. Spatial ID (Appendix C.2, Tab. 9): Under Default, S-Space uses unlabeled COCO images while Spatial ID uses labeled synthetic 4x4 grids; under Same COCO, Spatial ID is given ground-truth COCO boxes. Across relative, single-object, and non-spatial color 2D QA plus horizontal-axis SpinBench without/with premises, unsupervised S-Space beats grid-supervised Spatial ID on relative and color 2D QA on both models plus single-object 2D on Qwen3.5-4B, surpasses COCO-supervised Spatial ID across all tasks on Molmo2-ER, and stays closely competitive elsewhere:
Model & settingMethod (data)CVEmbSTAll 2DSingleColorSpin w/oSpin w/
Molmo2-ER, DefaultS-Space (COCO)91.9%95.0%99.8%95.8%94.2%95.8%94.3%100%
Molmo2-ER, DefaultSpatial ID (4x4 grid)88.3%94.8%99.8%94.7%95.0%95.2%92.9%100%
Molmo2-ER, Same COCOS-Space (COCO)91.9%95.0%99.8%95.8%94.2%95.8%94.3%100%
Molmo2-ER, Same COCOSpatial ID (COCO)86.6%92.0%99.0%92.9%93.7%94.0%91.4%100%
Qwen3.5-4B, DefaultS-Space (COCO)88.9%95.3%100%95.1%94.2%94.6%92.9%98.5%
Qwen3.5-4B, DefaultSpatial ID (4x4 grid)86.0%94.8%99.4%93.9%93.3%93.6%94.3%98.5%
Qwen3.5-4B, Same COCOS-Space (COCO)88.9%95.3%100%95.1%94.2%94.6%92.9%98.5%
Qwen3.5-4B, Same COCOSpatial ID (COCO)91.9%94.6%99.6%95.6%94.9%95.2%94.3%100%
  1. 3D depth extension and disentanglement (Appendix C.2, Tab. 10): Spatial ID's 4x4 grids cannot extract 3D depth. Extending it to natural COCO images on Molmo2-ER with Depth Anything V2 [3] quartiles suffers from perspective, where farther ground-plane objects appear higher [4]: static depth binning severely entangles vyv_y and vzv_z, whereas differentiating close/far logits in S-Space keeps them nearly orthogonal and achieves much higher accuracy on height-decorrelated SpatialTunnel [4]:
Method (Molmo2-ER)Construction datacos⁡(vy,vz)\cos(v_y, v_z)SpatialTunnel distance
S-SpaceUnsupervised COCO0.1490.5%
Spatial ID [1]Supervised 4x4 grid--
Spatial ID [1]Supervised COCO + Depth0.8277.5%
  1. Causal specificity (Appendix C.3, Tab. 16, Fig. 13): Under matched single-layer swaps (Eq. 12) on Molmo2-ER while monitoring What color is the {query}?, S-Space achieves higher counterfactual flips and higher color retention across α∈{1,…,5}\alpha \in \{1, \ldots, 5\}:
Axis method (Molmo2-ER)Spatial flip (α=1/2/3/4/5\alpha = 1 / 2 / 3 / 4 / 5)Color kept (α=1/2/3/4/5\alpha = 1 / 2 / 3 / 4 / 5)
S-Space (COCO)1.3 / 14.3 / 40.2 / 48.8 / 54.1%98.9 / 98.1 / 95.8 / 92.8 / 91.1%
Spatial ID [1] (4x4 grid)1.3 / 7.8 / 31.0 / 41.9 / 41.1%98.1 / 96.1 / 90.9 / 83.6 / 80.0%
Random orthonormal basis0.0 / 0.0 / 0.1 / 0.1 / 0.3%100 / 99.9 / 99.6 / 99.4 / 99.3%
  1. Pretrained Jacobian Lenses (Appendix C.2, Tab. 11): Decoding (hq,hr)(h_q, h_r) with official pretrained Jacobian Lenses [2] (ℓ^(ht)=WU RMSNorm(Jℓht)\hat{\ell}(h_t) = W_U\,\mathrm{RMSNorm}(J_\ell h_t), Eq. 4 sign rule), S-Space substantially outperforms Jacobian Lens across all 3D benchmarks:
ModelS-Space (CV / Emb / ST / Overall)Jacobian Lens [2] (CV / Emb / ST / Overall)
Qwen3.5-4B89.4 / 86.4 / 90.1 / 88.6%67.1 / 67.5 / 67.8 / 67.5%
Qwen3.6-27B91.8 / 89.0 / 92.9 / 91.3%72.2 / 69.9 / 89.8 / 77.2%

[1] Kang et al., ICLR 2026. [2] Gurnee et al., Transformer Circuits, 2026. [3] Yang et al., NeurIPS 2024. [4] Min et al., ECCV 2026.

(Continued from Part 1/2)

Q1b (Q1): Gradient difference vs. gradient averaging

In relative spatial queries, query qq and reference rr carry opposite spatial sensitivities while sharing the same image and prompt framing: gq=βpug+dp,qcontextg_q = \beta_p u_g + d^{\mathrm{context}}_{p,q} and gr=−βpug+dp,rcontextg_r = -\beta_p u_g + d^{\mathrm{context}}_{p,r} (Sec. 2.2 and Appendix B.1, Eq. 11). Consequently, role-averaging (gq+gr)/2(g_q + g_r)/2 cancels the opposing spatial terms and isolates prompt drift, whereas single-token gqg_q retains uncancelled prompt-context bias on every example. Taking the role-antisymmetric difference (gq−gr)/2(g_q - g_r)/2 reinforces both spatial terms while cancelling shared prompt context, substantially improving readout, cross-prompt transfer, and SpinBench rotation (Appendix B.1, Tab. 2, Fig. 11):

Method (Molmo2-ER)Gradient estimatorCV-BenchEmbSpatialSpatialTunnelSingle-object / Non-spatialSpinBench
S-Space(gq−gr)/2(g_q - g_r)/291.9%86.9%96.5%87.3% / 88.6%94.5%
Query tokengqg_q86.0%84.7%92.5%83.4% / 83.4%85.6%
Role average(gq+gr)/2(g_q + g_r)/230.2%26.7%15.3%33.2% / 30.4%33.6%

Q2 (W2 / Q2): Continuous 3D viewpoint rotations

SpinBench evaluates discrete 90∘90^\circ and 180∘180^\circ viewpoint changes whose rotation matrices reduce to axis permutations and sign flips, which could still succeed even if vxv_x and vzv_z had incompatible scales. To test whether S-Space forms a geometrically consistent space that supports continuous viewpoint transformations, we evaluate 24×15∘24 \times 15^\circ clockwise observer yaw rotations across 200 real RGB-D test scenes from SUN-RGBD [5] using Qwen3.6-27B with frozen COCO axes and no scale calibration (Sec. 5.1 and Appendix C.5, Eq. 13, Tab. 19, Fig. 18).

Crucially, we read (x,z)(x, z) only once per scene from the front RGB image via standard left-right and closer-farther prompts, and then transform coordinates analytically for every target yaw angle θ∈{0∘,15∘,…,345∘}\theta \in \{0^\circ, 15^\circ, \ldots, 345^\circ\} via x′=cos⁡θ x+sin⁡θ zx' = \cos\theta\,x + \sin\theta\,z and z′=−sin⁡θ x+cos⁡θ zz' = -\sin\theta\,x + \cos\theta\,z, directly mixing xx and zz at intermediate angles. On the same 8,464 target-view queries, S-Space + Rotation substantially outperforms both Direct greedy decoding and Reasoning chain-of-thought generation while maintaining high accuracy on non-orthogonal views on par with orthogonal views:

Viewpoint subsetQuestionsMethodLeft / rightNearer / fartherOverall
Orthogonal, θ∈{0∘,90∘,180∘,270∘}\theta \in \{0^\circ, 90^\circ, 180^\circ, 270^\circ\}1,600Direct54.4%55.5%54.9%
Reasoning66.9%74.9%70.9%
S-Space + Rotation86.0%86.5%86.3%
Non-orthogonal, θ∉90∘Z\theta \notin 90^\circ\mathbb{Z}6,864Direct52.6%51.9%52.3%
Reasoning70.0%78.2%74.1%
S-Space + Rotation86.0%88.8%87.4%
All 24 angles (0∘–345∘0^\circ\text{--}345^\circ, 15∘15^\circ step)8,464Direct53.0%52.6%52.8%
Reasoning69.4%77.6%73.5%
S-Space + Rotation86.0%88.4%87.2%

[5] Song et al., SUN RGB-D: A RGB-D Scene Understanding Benchmark Suite, CVPR 2015.

Q3 (W2 / Q3): Trajectories during autonomous reasoning

To examine whether similar coordinate updates emerge during the model's own spontaneous reasoning without controlled CoT templates, we evaluate Qwen3.6-27B on SpinBench given only the image and multiple-choice question with no CoT prefill (global response Q2.3; Sec. 5.2 and Appendix C.5, Tab. 21, Fig. 21). Across 8,505 complete self-generated <think> rollouts, we pair adjacent mentions of options A and B over normalized reasoning progress p∈[0,1]p \in [0, 1] and fit the linear slope of the target-aligned coordinate difference.

Although free-form CoT is noisy rather than an exact geometric rotation at every step, rollouts reaching the correct final answer steadily shift object coordinates toward the target perspective (mean slope +2.19+2.19 across 7,115 rollouts), whereas incorrect rollouts remain flat or drift away (−0.57-0.57 across 1,390 rollouts). Controlling for question difficulty across the 72 matched questions that contain both correct and incorrect rollouts yields the same contrast (+1.27+1.27 vs. −0.53-0.53, with higher mean slope for correct CoTs on 69 of 72 questions), confirming that S-Space actively participates in autonomous spatial reasoning.

Thank you for the detailed response and the additional experiments. The new results directly address my main concerns regarding methodological novelty, generalization to continuous viewpoint transformations, and spatial representations during autonomous reasoning. These additions substantially strengthen the paper's contributions and conclusions. Overall, I find the rebuttal convincing, and my concerns have been largely resolved. I appreciate the authors' efforts in addressing my questions.

Thank you for your thoughtful feedback and positive reassessment. We are pleased that our response and additional experiments have largely addressed your concerns. We would appreciate it if you could consider updating your rating to reflect your revised assessment.

Reviewer_V2mDOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how VLMs represent spatial information and why they still fail at spatial reasoning. It identifies S-Space, a three-dimensional linear subspace of intermediate object-token representations that encodes horizontal position, vertical position, and viewer-relative depth. Readout and intervention experiments link this subspace to spatial judgments, and case studies show it extends to non-visible objects and responds to linguistic context. Applying viewpoint transformations to the readout coordinates substantially improves perspective-taking accuracy, exposing a gap between the spatial information models encode and what they use when answering. The main contribution is an interpretable framework for probing internal spatial representations and diagnosing spatial reasoning failures.

Critical strengths
  1. S-Space provides a useful way to study how VLMs represent and use spatial information.
  2. The readout and intervention experiments provide evidence that S-Space is closely related to VLMs' spatial judgments.
  3. The Section 4 case studies are interesting, particularly the representation of unseen entities and spatial relations inferred from context.
Critical weaknesses
  1. The hypothesis in Section 4 that S-Space emerges from predicting spatial language lacks direct experimental support. The case studies show how spatial representations respond to context, but do not establish their origin during training. Supporting this claim would require controlled training ablations with and without spatial-language supervision. The authors should reframe Section 4 and its takeaway around the observed behavior.
  2. The paper does not sufficiently examine the relationship between S-Space and the vision encoder. Spatial structure may already be present in visual features and then be transferred to object-token representations through attention to the relevant image regions. Probing object-level features pooled from the vision encoder would help test this possibility. Comparisons across vision encoders, and between VLMs with and without a separate vision encoder, would also help clarify which aspects of S-Space are inherited from visual representations and which arise during multimodal processing.
Questions for authors

Please refer to weaknesses.

1 comment

We sincerely thank Reviewer V2mD for the constructive review and insightful questions. Common points are addressed in the global response and individual questions below.

W1: Reframing Section 4 around observed behavior

We fully agree with distinguishing inference-time contextual behavior—which our case studies directly demonstrate—from training-time origin, which would require controlled pretraining ablations. Following your suggestion to reframe Section 4 and its takeaway around the observed behavior, we revised Sec. 4 ('Beyond Visible Objects: Case Studies of S-Space') and Sec. 6 in two ways:

  • Restructuring around observed contextual behavior: Sec. 4 and its takeaway box now present the inference-time visual and linguistic case studies first, showing how a single frozen 3D subspace assigns coherent coordinates to off-screen entities, the observer, and contextually updated descriptions before discussing potential origins and physical-abstract concept blending.
  • Framing pretraining origin as a literature-grounded hypothesis: Across Sec. 4 and Sec. 6, we explicitly frame pretraining origins as a plausible hypothesis rather than an established mechanism, connecting it to prior work on emergent world models [1] with linear spatial states [2, 3] and language-aligned visual representations [4].

[1] Li et al., ICLR 2023. [2] Nanda et al., BlackboxNLP 2023. [3] Gurnee & Tegmark, ICLR 2024. [4] Radford et al., ICML 2021.

W2: Relationship with the vision encoder

Prior work has shown that spatial structure in VLMs originates in visual features and is progressively routed to text tokens across layers [5-8]. Our core contribution does not dispute this perceptual origin, but rather characterizes how this spatial information is organized and used inside the LLM as an internal spatial workspace. To clarify how S-Space relates to vision-encoder features, we probe object-level visual features alongside LLM object tokens across two distinct vision towers, SigLIP2 in Molmo2-ER and QwenViT in Qwen3.5-4B, on 600 held-out COCO pairs (Sec. 4 and Appendix C.4, Tab. 17, Fig. 16). Specifically, we compare box-pooled visual tokens (averaging post-projector image patch tokens inside each object's ground-truth COCO box) against text-only object tokens (read from the object-name token in the relative prompt with no boxes supplied):

ModelToken representationReadoutBoxesHoriz.Vert.Dist.Overall
Molmo2-ERBox-pooled visualSup. probeYes99.0%93.5%90.5%94.3%
Molmo2-ERBox-pooled visualS-Space L22Yes90.0%83.5%44.5%72.7%
Molmo2-ERObject text (L0)S-Space L0No51.0%63.0%69.5%61.2%
Molmo2-ERObject text (L22)S-Space L22No97.0%97.0%88.5%94.2%
Qwen3.5-4BBox-pooled visualSup. probeYes100.0%100.0%91.5%97.2%
Qwen3.5-4BBox-pooled visualS-Space L15Yes91.5%90.5%71.0%84.3%
Qwen3.5-4BObject text (L0)S-Space L0No56.0%62.5%53.0%57.2%
Qwen3.5-4BObject text (L15)S-Space L15No99.5%96.5%85.0%93.7%
  1. Comparison with vision features: Two key observations emerge from Tab. 17. (i) Without any grounding boxes or coordinate supervision, mid-layer S-Space readouts from text-only object tokens closely match supervised probes trained on ground-truth box-pooled visual features (94.2% vs. 94.3% on Molmo2-ER; 93.7% vs. 97.2% on Qwen3.5-4B). (ii) At the same time, S-Space is a higher-level aggregated subspace specific to object tokens rather than a property of the visual feature space itself: projecting box-pooled visual tokens directly onto S-Space degrades noticeably on 2D axes and sharply on 3D distance (44.5% on Molmo2-ER; 71.0% on Qwen3.5-4B), while object tokens rise from near chance at L0 to form this unified 3D subspace in intermediate LLM layers.
  2. S-Space beyond vision features: Whereas vision-encoder features remain strictly static once an image is given, S-Space functions as a dynamic workspace that integrates linguistic context and internal reasoning. Keeping the input image and its vision-encoder outputs fixed, object-token S-Space coordinates shift systematically under implied motion (Fig. 6c-d), spatial premises (Fig. 17), our single-word paired viewpoint control (Tab. 20, Fig. 20), and autonomous CoT reasoning (Tab. 21, Fig. 21), and even localize unseen entities and abstract concepts with no image patches (Figs. 6-7).

[5] Tong et al., NeurIPS 2024. [6] Kang et al., ICLR 2026. [7] Cui et al., arXiv 2026. [8] Chen et al., ICML 2025.

Reviewer_DMdWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how spatial information is represented and used in multimodal language models. The authors introduce three fixed linear readout directions from object-token activations, evaluate their generalization across datasets and prompting conditions, and investigate their causal relevance through activation interventions. They further use these readouts to analyze perspective-taking and reference-frame errors.

I find the transferable spatial information and the distinction between accessible spatial information and final answer generation very interesting. My main concern is that some claims—such as autonomous viewpoint transformation and the functional role of the proposed workspace—are a bit too strong compared to what the current experiments actually establish.

Critical strengths

Transferable spatial information. Fixed linear directions found on COCO can recover useful spatial information across different benchmarks, and they still work under single-object and non-spatial prompts.

A useful separation between spatial information and answer generation. On SpinBench, applying fixed viewpoint transformations to the latent representations achieves much better results than the original model responses. Although this does not prove that the model uses an internal transformation mechanism, it shows that the extracted spatial information can help make better predictions than the model's default outputs.

Critical weaknesses
  1. The controlled-CoT evidence does not clearly show autonomous spatial transformation. Figure 9 uses provided reasoning that contains the ground-truth transformed relation. Therefore, the changes in the late-stage representations might just reflect the encoding of that explicit relation instead of the model internally computing the viewpoint transformation. This limits support for the main claim that S-Space acts as a workspace used during reasoning.

  2. The causal role of the decoded coordinates is still not fully established. Table 7 shows 98.71% answer retention after swapping exact coordinates at ”\alpha = 1“, and large effects only happen under amplified interventions. This does not rule out causal relevance, because information can be redundant or reconstructed downstream, but it leaves the connection between the decoded coordinates and natural answer computation unclear.

Questions for authors
  1. Clarification of Figure 9: Because the provided CoT includes the ground-truth transformed relation, how do we know if the model really computes the viewpoint transformation internally, or if it just encodes the text relation? Also, are the middle-stage changes (before the target relation is given) statistically significant? This is key for me to evaluate the transformation claim.

  2. Clarification of intervention results: How should we understand the 98.71% answer retention when swapping exact coordinates at “\alpha = 1”? Is there any direct evidence showing if later layers repair the original coordinates, or if the answers follow the new counterfactual relation?

3 comments

We sincerely thank Reviewer DMdW for the constructive review and insightful questions. Common points are addressed in the global response and individual questions below.

Q1 (W1 / Q1): Viewpoint transformation vs. explicit relation encoding

In Fig. 9, the controlled CoT template included ground-truth relation words. To test whether these S-Space shifts reflect internal viewpoint processing rather than supplied directional words, we run three tests on Qwen3.6-27B with frozen COCO axes across the 146 SpinBench questions without premises (global response Q2; Sec. 5.2, Appendix C.5):

  1. Removing ground-truth reasoning relations (Fig. 19): Replacing both Step 1 and Step 3 spatial relations in Fig. 9a with <masked> while keeping the 3-step template unchanged, direct readout still rises steadily across stages while rotated readout declines:
CoT prefillReadoutEarlyMiddleLate
Original (Fig. 9b)Direct / Rotated44.5% / 96.6%54.8% / 89.0%86.3% / 67.1%
Direction-maskedDirect / Rotated38.4% / 95.2%50.7% / 84.9%71.2% / 71.9%
  1. Significance of Middle-stage changes and paired viewpoint control (Tab. 20, Fig. 20): Yes, the Middle-stage changes before the target relation is given are statistically significant in both the original template (p=0.033p = 0.033 direct, p=0.041p = 0.041 rotated) and the masked template (p=0.0019p = 0.0019 and p=4.9×10−4p = 4.9 \times 10^{-4}).

    Because Step 2 mainly introduces the target viewpoint context, we isolate its effect without object-order or Step 3 cues by fixing {OBJECT_A} and {OBJECT_B} to option order, masking Step 1, omitting Step 3, and varying only whether Step 2 adopts {TARGET_VIEW} or retains the front view:

    1. Track the viewpoint: The question asks about the {TARGET_VIEW} view. For this intermediate step, I use the {TARGET_VIEW} / front view. In that view, I consider the {OBJECT_A} and the {OBJECT_B}.
Intermediate prefixEarlyMiddleΔ\Delta (Middle −- Early)Mean paired shift
Adopt target view-1.91-0.46+1.46+0.31
Retain front view-1.91-0.77+1.14-

Entries report the target-aligned coordinate difference m=y u⊤(cA−cB)m = y\,u^\top(c_A - c_B); Mean paired shift averages per-question Δtarget−Δfront\Delta_{\text{target}} - \Delta_{\text{front}} across all 146 questions to cancel shared phrasing drift, confirming a significant shift toward the target perspective (p<10−4p < 10^{-4}) on 87/146 questions and across all six viewpoint-relation categories before any answer relation is given.

  1. Autonomous reasoning trajectories (Tab. 21, Fig. 21): Across Qwen3.6-27B's self-generated <think> rollouts with no CoT prefill, correct CoTs steadily shift coordinates toward the target perspective (slope +2.19+2.19), whereas incorrect CoTs remain flat or drift away (−0.57-0.57), showing that S-Space updates spontaneously despite the noise of free-form CoT.

Q2 (W2 / Q2): Downstream repair and counterfactual steering

Thank you for this constructive question on why exact swaps at α=1\alpha = 1 retain 98.71% of the original answers. Based on your question, we supplemented the downstream-layer analysis and discussion in Sec. 3.2 and Appendix C.3 (Tab. 15, Fig. 12). The 98.71% answer retention at α=1\alpha = 1 arises because a mild single-layer edit is gradually repaired across subsequent layers, not because final answers are unaffected by counterfactual S-Space coordinates. Tracking downstream layers L18–L21 after L17 swaps on Molmo2-ER confirms both points.

L17 swap interventionL18L19L20L21 restoredFlipped to opposite relation
Exact swap (α=1\alpha = 1)31.0%77.4%93.4%96.1%1.3%
Amplified swap (α=5\alpha = 5)2.1%2.7%3.8%6.1%54.1%
  • At α=1\alpha = 1, gradual downstream repair restores the original coordinates: Editing object-name tokens at L17 leaves image patch tokens untouched, so subsequent multimodal layers re-attend to unedited visual patches and progressively restore 96.1% of the original coordinate orderings across L18–L21, consistent with Transformer self-repair [1, 2] and why α>1\alpha > 1 is standard for single-layer steering [3].
  • At α=5\alpha = 5, once the swap survives repair, answers follow the counterfactual relation: Amplifying to α=5\alpha = 5 keeps the swapped coordinates reversed through L21 (only 6.1% restored), and 54.1% of answers flip directly to the opposite spatial direction (e.g., left to right or above to below), following the newly swapped relation.

[1] McDougall et al., BlackboxNLP 2024. [2] Rushing & Nanda, ICML 2024. [3] Gurnee et al., Transformer Circuits, 2026.

Thanks for the detailed explanations to my questions. It addressed my major concerns and I have updated my final decision, thanks.

We sincerely appreciate your dedicated evaluation of our paper and the subsequent score increase. Your constructive feedback has significantly improved our work.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response (Part 1/2)

We sincerely thank all reviewers for their constructive feedback. We organize our global response into Part 1/2 (Q1 and Q2 below) and Part 2/2 (Q3, an index mapping each reviewer's question to our new experiments and revised PDF locations), with detailed replies posted under each review. All revisions are highlighted in blue in the updated PDF (main text within 10 pages).

Q1: Comparisons with linear representation baselines (Reviewer u2PB W1/Q1)

To evaluate how our unsupervised gradient-based identification compares with prior linear representation methods in readout accuracy, prompt robustness, 3D geometry, and causal selectivity, we benchmark S-Space against Spatial ID [1] and the Jacobian Lens [2] on identical checkpoints, evaluation splits, and same-layer object-token hidden states (Sec. 3.1-3.2; Appendix C.2-C.3). Because Spatial ID [1] relies on synthetic 4x4 grids and extracts only 2D axes without depth, we compare against it in two parts—first on 2D tasks and second on a natural-image 3D depth extension—followed by causal swaps and pretrained Jacobian Lenses [2]:

  • 2D readout, cross-prompt transfer, and SpinBench vs. Spatial ID (Appendix C.2, Tab. 9): Across Default (unlabeled COCO vs. labeled 4x4 grids) and Same COCO (ground-truth boxes for Spatial ID) on Molmo2-ER and Qwen3.5-4B, unsupervised S-Space beats grid-supervised Spatial ID on relative 2D QA (95.8% vs. 94.7%; 95.1% vs. 93.9%) and non-spatial color prompts on both models plus single-object prompts on Qwen3.5-4B, beats COCO-supervised Spatial ID across all 2D prompts and SpinBench on Molmo2-ER, and stays closely competitive elsewhere without synthetic grids or box labels.
  • 3D depth extension and disentanglement (Appendix C.2, Tab. 10): Spatial ID's 4x4 grids cannot extract 3D depth. Extending it to natural COCO images with Depth Anything V2 [4] quartiles entangles vertical and distance axes under perspective [3] (cos⁡(vy,vz)=0.82\cos(v_y, v_z) = 0.82 vs. 0.140.14 for S-Space), degrading SpatialTunnel distance readout (77.5% vs. 90.5% ).
  • Causal specificity (Appendix C.3, Tab. 16, Fig. 13): Under matched single-layer swaps on Molmo2-ER, S-Space yields higher spatial flips (54.1% vs. 41.1% at α=5\alpha = 5) and higher color retention (91.1% vs. 80.0%) than Spatial ID; random bases stay ≤0.3%\le 0.3\% flip.
  • Pretrained Jacobian Lenses (Appendix C.2, Tab. 11): On identical object-token states, S-Space substantially outperforms official pretrained Jacobian Lenses [2] across all 3D benchmarks on Qwen3.5-4B (88.6% vs. 67.5% overall) and Qwen3.6-27B (91.3% vs. 77.2%).

[1] Kang et al., ICLR 2026. [2] Gurnee et al., Transformer Circuits, 2026. [3] Min et al., ECCV 2026. [4] Yang et al., NeurIPS 2024.

Q2: Viewpoint dynamics during CoT reasoning (Reviewer DMdW W1/Q1; Reviewer u2PB W2/Q3)

To verify that S-Space updates during CoT reflect internal viewpoint processing rather than supplied relation words—and also arise spontaneously during free-form reasoning—we run three experiments on Qwen3.6-27B with frozen COCO axes across all 146 SpinBench questions without premises (Sec. 5.2; Appendix C.5):

  1. Direction-masked CoT (Appendix C.5, Fig. 19): Replacing both Step 1 and Step 3 spatial relations in Fig. 9a with <masked> still yields rising direct readout (38.4% →\to 50.7% →\to 71.2%) and declining rotated readout (95.2% →\to 84.9% →\to 71.9%).
  2. Middle-stage significance and paired control (Appendix C.5, Tab. 20, Fig. 20): The Early-to-Middle shift is statistically significant in both Fig. 9b and the masked template. Because Step 2 mainly introduces the target viewpoint context, we fix {OBJECT_A} and {OBJECT_B} to option order, mask Step 1, omit Step 3, and vary only I use the {TARGET_VIEW} view vs. I use the front view in Step 2; adopting {TARGET_VIEW} significantly shifts coordinates toward the target perspective (+0.31+0.31, p<10−4p < 10^{-4}) across all six viewpoint-relation categories.
  3. Autonomous CoT trajectories (Appendix C.5, Tab. 21, Fig. 21): With no CoT prefill, pairing adjacent A/B mentions in self-generated <think> traces over progress p∈[0,1]p \in [0, 1] shows that although free-form CoT is noisy rather than an exact rotation, correct rollouts steadily shift coordinates toward the target view whereas incorrect rollouts stay flat or drift away:
Autonomous CoT outcomeRolloutsMean slopeMatched-question slope
Correct final answer7,115+2.19+1.27
Incorrect final answer1,390-0.57-0.53
AuthorsOpenReview ↗

Global Response (Part 2/2)

(Continued from Part 1/2)

Q3: Index of reviewer questions, new experiments, and revised PDF locations

To facilitate review, the table below maps each reviewer's concern, our corresponding experiment or manuscript revision, and its blue-highlighted location in the updated PDF:

ReviewerQuestion / concernOur experiment / revisionRevised PDF location (in blue)
DMdW (W1 / Q1)Do CoT shifts reflect viewpoint transformation or template directional words, and is the Middle shift significant?Direction-masked CoT, Early-to-Middle significance & paired Step 2 viewpoint control, and autonomous CoT trajectoriesSec. 5.2 (p. 9); Appendix C.5 (pp. 31–33; Tabs. 20–21, Figs. 19–21)
DMdW (W2 / Q2)Why do exact swaps (α=1\alpha = 1) retain answers while α=5\alpha = 5 flips them?Tracked downstream layers L18–L21 showing gradual visual self-repair at α=1\alpha = 1 vs. counterfactual relation flips at α=5\alpha = 5Sec. 3.2 (p. 5); Appendix C.3 (pp. 24–25; Tab. 15, Fig. 12)
V2mD (W1)Distinguish observed inference-time behavior from pretraining-origin claimsRestructured Sec. 4 around inference-time workspace behavior; framed pretraining origins as a hypothesisSec. 4 (pp. 6–7); Sec. 6 (p. 10); Appendix C.6 (p. 33)
V2mD (W2)Is S-Space merely routed from vision-encoder features via attention?Probed box-pooled visual tokens vs. LLM object tokens across SigLIP2 and QwenViT; contrasted with dynamic context/CoT updatesSec. 4 (p. 7); Appendix C.4 (pp. 26–28; Tab. 17, Fig. 16)
u2PB (W1 / Q1a)How does S-Space compare on equal footing with Spatial ID and Jacobian Lens?Matched 2D readout/transfer, natural-image 3D depth, and causal swaps vs. Spatial ID; 3D QA vs. pretrained Jacobian LensesSec. 3.1–3.2 (pp. 4–5); Appendix C.2–C.3 (pp. 20–22, 25; Tabs. 8–11, 16; Fig. 13)
u2PB (Q1b)Why use gradient difference (gq−gr)/2(g_q - g_r)/2 over single-token or averaged gradients?Derived prompt-context cancellation and ablated (gq−gr)/2(g_q - g_r)/2 against gqg_q and (gq+gr)/2(g_q + g_r)/2Sec. 2.2 (p. 4); Appendix B.1 (pp. 16–17; Eq. 11, Tab. 2, Fig. 11)
u2PB (W2 / Q2)Does S-Space support continuous non-90∘90^\circ 3D viewpoint rotations?Evaluated 24×15∘24 \times 15^\circ continuous yaw rotations on real 3D SUN-RGBD scenes with zero scale calibrationSec. 5.1 (pp. 8–9); Appendix C.5 (pp. 30–31; Tab. 19, Fig. 18)
u2PB (W2 / Q3)Do S-Space shifts also emerge during the model's own autonomous CoT?Trajectory slope analysis across self-generated <think> rollouts comparing correct vs. incorrect answersSec. 5.2 (p. 9); Appendix C.5 (pp. 32–33; Tab. 21, Fig. 21)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.