acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Spatial Reasoning in MLLMs: Learning to Form and Use Depth

Abstract

Spatial reasoning is essential for multimodal models to understand scenes and support embodied interaction, yet representing spatial information does not ensure that it guides their answers. We investigate how models can learn to form useful depth representations and turn them into evidence for spatial judgments. Our finding that depth information becomes readable before it has its strongest influence on answers motivates a training strategy that couples early depth formation with its subsequent use. Using existing question-answering annotations, we supervise object depth in early layers and train later layers to answer consistently with controlled changes in represented depth relations, requiring no additional geometry data, encoder, or teacher. Across VSI-Bench, CV-Bench, BLINK spatial tasks, MMSI-Bench, and ViewSpatial-Bench, our method raises mean scores by 3.8, 1.7, and 2.5 percentage points over supervised fine-tuning on identical data for Qwen2.5-VL-3B, Qwen3-VL-4B, and Qwen3.5-4B, respectively. These findings support early depth formation and downstream use as complementary training targets for improving spatial reasoning while preserving the original inference architecture.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Spatial Reasoning, Multimodal Large Language Models, Mechanistic Interpretability, Depth Representations, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1EE3OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the relationship between depth information in multimodal models’ internal representations and its influence on spatial answers. Layerwise readouts and interventions suggest that peak depth readability precedes peak answer influence. Motivated by this observation, the authors combine early-layer depth supervision with edited-answer supervision that trains downstream layers to respond to modified depth relations. The method improves spatial QA across three Qwen backbones while retaining the original inference architecture.

Critical strengths
  1. An interesting mechanistic hypothesis with supporting evidence. The paper connects layerwise depth readability with downstream answer influence. Readouts, directional interventions, and channel ablations provide complementary evidence for depth-related internal information and its behavioral relevance. This is a useful direction for understanding spatial reasoning failures.
  2. A practical training approach with controlled empirical improvements. The proposed objectives improve all five benchmark scores over same-data SFT across three backbones without adding inference components. Replay, random-perturbation, and gradient-separation controls help establish that the improvements are associated with the proposed training design.
Critical weaknesses
  1. The generality of the insight remains insufficiently established. All evaluated backbones belong to the Qwen family and are limited to 3B–4B. The detailed mechanistic evidence is narrower than the cross-backbone QA evaluation. It remains unclear whether the same structure and layerwise ordering hold for other model families, larger models, or different amounts and mixtures of spatial SFT data. On VSI-Bench, the gains over VG-LLM are 1.5, 0.4, and 0.5 points across the three backbones. These smaller margins on the newer backbones motivate further investigation, although they do not establish a scaling trend.
  2. The evidence does not fully distinguish the proposed mechanism from alternative explanations. Reduced accuracy under the chosen linear readout need not imply reduced depth information. Later layers could retain spatial information in a different, potentially more semantic encoding that the readout does not recover. Fixed-polarity accuracy below 50% also indicates inverse correspondence rather than an absence of information. The interventions establish answer sensitivity to the selected directions, but further evidence is needed to establish their geometric specificity. Moreover, early and late configurations may differ in which examples pass the reliability and feasibility filters, potentially confounding the placement comparison.
Questions for authors
  1. Can the core readout and intervention findings be replicated on a larger model and a model family outside Qwen? Please also discuss their dependence on spatial SFT data amount and composition, and clarify the VG-LLM training setup used in Table 2.
  2. Can independently trained geometric probes, including probes with greater capacity, recover depth information from later layers? Such comparisons, together with validation-calibrated polarity and intervention selectivity controls, would help distinguish changes in encoding from changes in information availability and use.
  3. The reliability and feasibility filters depend on the depth readout at the selected layer. Do the early- and late-layer configurations therefore receive different amounts or types of edited-answer training? Please report the number of examples actually edited, the proportions of relation-preserving and relation-reversing edits, and the perturbation magnitudes for each configuration. Can you repeat the comparison on examples eligible in both configurations, with matched edit types and comparable perturbation magnitudes? This would help determine whether the early-layer advantage comes from placement itself or from differences in the edited training examples.

If my phrasing of the issue is not sufficiently precise, I encourage the author to engage in further discussion with me.

5 comments

Thank you for the time and care you devoted to reviewing our work and for your valuable suggestions on the mechanistic analysis. Below, we address your questions about generalization, depth readouts, and the early–late comparison.

W1. Generality across models and data, and VG-LLM training (Q1)

Response.{\color{green}\mathbf{Response.}} The readout–response ordering also appears at 9B scale and outside Qwen. The additional experiments use Qwen3.5-9B and InternVL3-9B [1]. The latter uses an InternLM3-8B language backbone, providing a comparison across model families.

ModelDataMethodHigh-readout layersStrong-response window centers
Qwen3.5-9BOriginalSFTL12–L18L16–L23
Qwen3.5-9BOriginalOursL7–L18L10–L22
Qwen3.5-9BExpandedOursL7–L17L10–L22
InternVL3-9BOriginalSFTL18–L26L24–L34
InternVL3-9BOriginalOursL11–L26L16–L32

Directions are estimated separately per model and evaluated on scene-disjoint confirmation examples. For original-data SFT/Ours comparisons, shared thresholds within each backbone are 50%+0.8(ASFT,peak−50%)50\%+0.8(A_{\mathrm{SFT,peak}}-50\%) for readout and 0.8USFT,max0.8U_{\mathrm{SFT,max}} for response. Each interval begins with the first run of at least three consecutive positions above threshold. It ends before the next subthreshold position. Interventions use three-layer windows at ρ=10%\rho=10\%. Layers are zero-indexed.

High-readout onset precedes strong-response onset in every reported configuration. Training advances both intervals on each backbone. The expanded Qwen3.5-9B model retains this ordering.

The expanded mixture contains 361K examples, including additional VLM-3R and VSI-590K data. Sampling of the added sources follows SpatialStack [2]. Within each comparison, SFT and Ours share natural QA, training steps, tuning mode, and evaluation settings. Scores below are five-benchmark means (%).

ModelDataSFT meanOurs meanGain (pp)
Qwen3.5-9BOriginal59.262.5+3.3
Qwen3.5-9BExpanded60.364.5+4.2
InternVL3-9BOriginal58.461.4+3.0

All five benchmarks improve in each comparison. The gains persist at 9B across Qwen and InternLM backbones and across the original and expanded data mixtures. The layerwise diagnostics also show earlier depth readout and answer influence after training.

VG-LLM in Table 3 (Table 2 in the original submission) is retrained on each corresponding backbone, separately from the Section 3 analysis pair. All groups start before spatial fine-tuning and receive SPAR-234K, LLaVA-Hound-64K, and natural QA for the 30K auxiliary pairs. VGGT, the visual encoder, and the visual merger are frozen. The geometry mapper is trained.

Language tuning matches SFT/Ours. Qwen2.5-VL-3B uses full tuning at 10−510^{-5}. The two 4B backbones use LoRA with rank 64, α=128\alpha=128, dropout 0.05, and learning rate 5×10−55\times10^{-5}. Training uses batch 64 and one epoch. Checkpoints are selected using QA from independent calibration scenes. Appendix B.2 specifies feature alignment. VG-LLM retains VGGT at inference. Ours uses the original architecture.

[1] InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models.

[2] SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning.

W2. Late-layer information, polarity, and geometric specificity (Q2)

Response.{\color{green}\mathbf{Response.}} You are right that depth information can remain available in late layers. Independent linear probes confirm this directly:

DiagnosticQwen-SFTVG-LLM
Independent depth probe: peak accuracy87.17% (L22)88.50% (L23)
Independent depth probe: final-layer accuracy84.83% (L35)86.00% (L35)
Answer-gradient readout: peak accuracy (Figure 1)70.00% (L17)74.00% (L8)
Answer-margin response: peak center (Figure 3)L21L17

Both models support high late-layer probe accuracy, with similar peak locations. The answer-gradient readout peaks much earlier in VG-LLM. These results distinguish linearly decodable depth from its alignment with the model's answer computation.

A probe learns an additional decoder. Our readout measures geometry along the model's own answer-sensitive direction. Let FℓF_\ell be the final near/far answer-logit difference and r=hA−hBr=h_A-h_B. With the pair mean and other token states fixed,

Fℓ(r+Δr)−Fℓ(r)≈∇rFℓ(r)⊤Δr.F_\ell(r+\Delta r)-F_\ell(r)\approx\nabla_rF_\ell(r)^\top\Delta r.

Section 3.1 fixes the averaged direction before evaluating ground-truth relations. Section 3.2 then edits states and reruns the downstream network. This separates recoverable information from its influence on answers, consistent with the knowledge–prediction distinction studied by Park et al. [1]. Independent probes show similar depth recoverability in the two models, while our answer-gradient readout and controlled interventions reveal differences in when and how strongly depth-related states influence their answers. The interventions establish these causal effects within the models' own answer computation, which probe accuracy alone does not test.

Your polarity interpretation also matches Appendix A.1: below-50% fixed-polarity accuracy indicates inverse ordering. After polarity calibration, the gap from independent probes and the between-model differences in layerwise answer-related readout remain. Together, the probes and interventions identify differences in answer-related accessibility and downstream influence.

To address your geometric-specificity concern, the following controls test visual correspondence and behavioral effects separately.

Visual grounding. In 200 real view pairs, the same objects have opposite annotated depth orderings. Object names, wording, mention order, and options stay fixed. Success requires correct readouts for both views, testing whether the direction follows the visual reversal under identical text.

Answer changes. Depth edits preserve the image, question, pair mean, and edited-layer H/V readings. We evaluate 320 non-tied judgments per model that initially oppose the target. Flip rates are reported at each model's highest-flip layer: Qwen-SFT L22 and VG-LLM L17.

Open-ended descriptions. On the same 200 pairs, the target view's natural depth projection supplies the source-view edit target. The source image stays fixed. Prompts ask for the spatial relation without providing options or the target relation. Directions and layers are fixed before evaluating descriptions. Human evaluators extract each description's near/far relation, blind to the condition and target answer. Agreement is measured against target-view annotations. Each condition uses all 200 examples, with ambiguous, missing, or contradictory answers counted as failures. Values below are percentages.

TestConditionQwen-SFTVG-LLM
Cross-view readout: both views correctGradient direction61.066.5
Cross-view readout: both views correctRandom direction (mean)14.517.0
Conditional answer flipDepth edit55.658.8
Conditional answer flipEqual-norm random edit0.91.25
Description: agreement with target relationSource view, natural state17.015.5
Description: agreement with target relationTarget view, natural state74.578.0
Description: agreement with target relationSource view, depth edit58.564.0
Description: agreement with target relationSource view, equal-norm random edit15.016.0

The readout follows real depth reversals. Targeted edits change both choices and open-ended descriptions far more than random edits. The natural target view provides visual evidence for the edited relation, and the source-view intervention measures its causal influence on the answer.

[1] Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions.

W2. Do filtering and edit distributions confound early versus late placement? (Q3)

Response.{\color{green}\mathbf{Response.}} The early configuration retains a 1.6-point advantage after matching edited examples, edit types, target answers, and the use-loss update range. Each counted instance is a completed paired edit contributing to answer supervision. ρ\rho is the larger per-object relative state change, capped at 10%.

ConfigurationEdited instancesPreserving: count (%)Reversing: count (%)ρ\rho (%), mean ±\pm SDMean score
R2: early26,53914,103 (53.1)12,436 (46.9)4.27±2.624.27\pm2.6254.5
R3: late26,77313,894 (51.9)12,879 (48.1)4.13±2.694.13\pm2.6952.3
Early, common L23 suffix26,42713,836 (52.4)12,591 (47.6)4.06±2.484.06\pm2.4853.8
Late, common L23 suffix26,69113,467 (50.5)13,224 (49.5)3.88±2.563.88\pm2.5652.1
Matched early, common L23 suffix23,75212,317 (51.9)11,435 (48.1)3.93±2.383.93\pm2.3853.6
Matched late, common L23 suffix23,75212,317 (51.9)11,435 (48.1)3.89±2.443.89\pm2.4452.0

The first four scores are from Table 5 (Table 3 in the original submission). Edit statistics and matched-training scores are additional results. Early/late sets are L8/L10/L12 and L18/L20/L22. Both matched conditions restrict use-loss updates to L23–L35. They share initialization, QA batches and order, steps, loss weights, and checkpoint selection. Corresponding rotating layers edit the same jointly eligible example with the same edit type and target answer. Both skip when a common edit is infeasible.

Mean perturbations differ by only 0.04 percentage points. The matched advantage, 53.6 versus 52.0, closely retains the original common-suffix gap of 53.8 versus 52.1. The early advantage therefore persists with common examples, matched edit types, and nearly identical perturbation magnitudes. These controlled comparisons support early placement of the joint depth-supervision and editing configuration.

Reviewer_cRwVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when depth information becomes readable in multimodal language models and when it affects spatial answers. It proposes supervising depth in early layers and training later layers to respond to controlled depth edits. The method improves average spatial benchmark scores over same-data supervised fine-tuning across three backbones.

Critical strengths

1, The paper connects internal representations to downstream behavior. It combines depth readout, activation edits, and answer supervision rather than relying only on aggregate benchmark scores.

2, The training objectives are practically motivated. They use existing QA annotations and do not add inference-time components.

Critical weaknesses

1, The evidence does not fully establish that depth formation and use are the main bottlenecks in spatial reasoning. The analysis focuses on depth and selected same-view relations, while spatial tasks also require orientation, viewpoint, and interactions among multiple relations.

2, The depth-edit intervention may not capture realistic changes in visual scenes. Editing an internal relation while holding the image fixed tests model responsiveness to the representation, but it does not show that the model grounds such edits in valid visual evidence.

Questions for authors

1, How well does the proposed formation/use framework extend to orientation, viewpoint, and multi-relation spatial reasoning?

  1. How do you verify that edited depth states correspond to plausible visual changes, instead of steering answer logits?
4 comments

We appreciate your time and thoughtful comments on the scope of spatial reasoning and the visual grounding of our interventions. We address these points below with broader-task results and controlled geometric evaluations.

W1. Why depth, and does the framework transfer to broader spatial reasoning? (Q1)

Response.{\color{green}\mathbf{Response.}} You are right that spatial reasoning also requires orientation, viewpoint, and relation composition. Depth formation and downstream use provide complementary targets for training spatial reasoning. Depth offers a measurable relation beyond image-plane position. In Figure 1, VG-LLM's best depth-readout window precedes Qwen-SFT's by eight layers, versus one layer for H/V. Figure 3 places peak answer response after peak readout in both models.

This motivates supervising both depth representations and their use in answering. Table 5 (Table 3 in the original submission) separates their contributions: depth supervision scores 51.9, QA replay 52.0, and depth plus edited-answer supervision 54.5. Figure 6 connects these gains to stronger early readout and earlier answer influence.

Transfer extends beyond near/far questions. With identical natural QA and training steps, Qwen3.5-9B improves all eight VSI-Bench subtasks:

VSI-Bench subtaskSFTOursGain (pp)
Object count64.8366.07+1.24
Absolute distance49.6655.10+5.44
Object size75.7581.69+5.94
Room size72.9577.99+5.04
Relative distance39.5846.34+6.76
Relative direction60.7264.93+4.21
Route planning51.5555.15+3.60
Appearance order80.4283.01+2.59
Mean61.9366.28+4.35

Without further training, the same checkpoints also improve all seven tasks in Spatial457 [1]:

Spatial457 subtaskSFTOurs
Single-object recognition72.172.8
Multi-object understanding66.467.7
2D spatial relations61.863.3
Occlusion relations58.161.5
3D orientation46.248.7
6D spatial relations43.846.2
Collision prediction45.247.7
Seven-task mean56.258.3

The 6D and collision tasks combine object position and orientation, providing direct evidence on multi-relation judgments. ViewSpatial-Bench, which covers camera- and person-centered localization, also improves:

Qwen3.5-9B evaluationSFTOurs
ViewSpatial-Bench, overall53.756.6

Depth-based auxiliary supervision produces these gains across orientation, scale, and compositional tasks. ViewSpatial's overall score also rises by 2.9 points across questions covering camera- and person-centered perspectives. Together, these evaluations show that training depth formation and use benefits several components of spatial reasoning.

[1] Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models.

W2. Do the edits reflect visual geometry or merely steer answer scores? (Q2)

Response.{\color{green}\mathbf{Response.}} The edited relation is anchored to a real target view, and its effect is visible in open-ended descriptions. The controls below address your question by connecting visual changes, represented depth, and generated answers.

Visual grounding. In 200 real view pairs, the same objects have opposite annotated depth orderings. Object names, wording, mention order, and options stay fixed. Success requires correct readouts for both views, testing whether the direction follows the visual reversal under identical text.

Answer changes. Depth edits preserve the image, question, pair mean, and edited-layer H/V readings. We evaluate 320 non-tied judgments per model that initially oppose the target. Flip rates are reported at each model's highest-flip layer: Qwen-SFT L22 and VG-LLM L17.

Open-ended descriptions. On the same 200 pairs, the target view's natural depth projection supplies the source-view edit target. The source image stays fixed. Prompts ask for the spatial relation without providing options or the target relation. Directions and layers are fixed before evaluating descriptions. Human evaluators extract each description's near/far relation, blind to the condition and target answer. Agreement is measured against target-view annotations. Each condition uses all 200 examples, with ambiguous, missing, or contradictory answers counted as failures. Values below are percentages.

TestConditionQwen-SFTVG-LLM
Cross-view readout: both views correctGradient direction61.066.5
Cross-view readout: both views correctRandom direction (mean)14.517.0
Conditional answer flipDepth edit55.658.8
Conditional answer flipEqual-norm random edit0.91.25
Description: agreement with target relationSource view, natural state17.015.5
Description: agreement with target relationTarget view, natural state74.578.0
Description: agreement with target relationSource view, depth edit58.564.0
Description: agreement with target relationSource view, equal-norm random edit15.016.0

The original directions track visual depth reversals under identical text. With the source image fixed, targeted edits exceed random edits in open-ended target agreement by 43.5 points for Qwen-SFT and 48.0 for VG-LLM. Their effects therefore extend to the spatial content of generated descriptions.

The target view supplies both a natural depth reading and an annotated near/far relation. Editing the source state toward that reading tests how downstream computation responds to a relation with a specific visual reference. Natural QA supervision continues to preserve the original image–answer association during training.

Dear Reviewer cRwV,

As the discussion period is coming to an end, we would appreciate your feedback on whether our responses have adequately addressed your concerns. We have responded to each of your points in detail and would be happy to clarify any remaining questions before the deadline.

Thank you again for your time and valuable feedback.

Best wishes,
The Authors

Dear Reviewer cRwV,

As the discussion period draws to a close, we would appreciate your feedback on whether our updated responses address your concerns. We have added:

  • Broader spatial reasoning: additional results on VSI-Bench, Spatial457, and ViewSpatial-Bench cover orientation, scale, viewpoint, and relation composition. The same Qwen3.5-9B checkpoints improve all seven Spatial457 tasks without further training.
  • Visual grounding of edits: real cross-view tests and human evaluation of open-ended descriptions show that the edits reflect depth relations observed in real images. Targeted edits produce the target relation in both answer choices and open-ended descriptions far more often than random edits.

Please let us know if any concerns remain. We would be happy to provide further clarification.

Thank you again for your time and valuable feedback.

Best wishes,
The Authors

Reviewer_uDFdOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how multimodal models can learn to form useful depth representations and actually use them for spatial judgments. The authors find that depth information becomes readable in early layers before it strongly influences the final answers. To fix this disconnect, they propose a training strategy that couples early depth formation with downstream use. They supervise object depth in early layers and train later layers to answer consistently with controlled changes in these depth relations. This method requires no additional geometry data, encoders, or teacher models. Experiments show it improves spatial reasoning scores across multiple benchmarks and model sizes compared to standard fine-tuning.

Critical strengths

The method uses existing question-answering data without needing extra geometry labels or complex teacher models, and the mechanistic analysis shows that depth readability and answer influence happen at different network stages.

Critical weaknesses

A major concern is that the method focuses entirely on depth reasoning while relying heavily on the SPAR dataset. The authors even note in the conclusion that future work must examine how depth interacts with orientation and viewpoint. This narrow focus risks the model simply memorizing the statistical patterns of specific scenes rather than learning true, generalizable three-dimensional spatial reasoning.

Another critical issue is that the reported performance improvements over standard supervised fine-tuning are quite small, ranging from only 1.7 to 3.8 percentage points. While these gains are consistent across benchmarks, such minor bumps might not justify the added training complexity for real-world embodied applications. It remains unclear if these small score increases actually translate to meaningful improvements in complex scene understanding.

The experiments are exclusively conducted on the Qwen family of models, which share very similar underlying architectural choices. This raises valid concerns about whether the early depth formation principle is a universal property of multimodal models or just a specific artifact of the Qwen design. Without testing on fundamentally different models, the broad applicability of these findings remains highly questionable.

The paper defines readable depth using a gradient-based readout on residual states, which is an indirect and abstract mathematical proxy. It is not entirely clear if this readout accurately reflects the true geometric understanding of the model or just a superficial statistical correlation. If this proxy is flawed, the entire premise of early depth formation might be built on shaky ground.

Questions for authors

First, a critical concern is that the paper focuses entirely on depth reasoning while ignoring other crucial spatial relations like orientation, scale, or complex multi-object arrangements. Relying solely on depth might not generalize to broader spatial reasoning tasks that require a holistic understanding of three-dimensional space. To improve this, could you provide evidence or ablation studies showing how this early formation strategy performs when extended to orientation or multi-relational spatial queries?

About W2, what specific failure cases does your method resolve that standard fine-tuning completely fails on, to prove the practical value of these marginal gains?

About W3, without testing on fundamentally different models, the generalizability of these findings remains highly questionable. Have you tested this training strategy on non-Qwen architectures, such as LLaVA or InternVL, to confirm the method is broadly applicable?

Fourth, the paired edit mechanism used to train downstream responses involves complex mathematical constraints to preserve pair means and orthogonal components. This adds significant implementation overhead and creates potential points of failure during the training process. If the edit is even slightly miscalibrated, it could introduce harmful noise rather than useful learning signals. How sensitive is the model performance to the relative state-change cap and the feasibility thresholds used during the paired editing process?

Fifth, the depth pairs and training signals are heavily derived from the SPAR dataset, which uses specific scene groups and strict parsing rules. The model might simply be memorizing the statistical patterns of these specific scenes rather than learning true geometric reasoning. This could easily explain why the gains on some benchmarks, like MMSI-Bench for the smallest model, are minimal or even slightly worse than the baseline. Can you show performance on a completely out-of-distribution spatial reasoning dataset that does not share any scene or formatting overlap with the SPAR dataset?

About W4, what alternative methods or human evaluations could validate that the gradient-based readout truly corresponds to meaningful geometric understanding?

Seventh, the paper claims the method preserves the original inference architecture, which is certainly a strong point. However, it does not discuss whether the early layer supervision affects the activation sparsity or computational cost during inference. Even without adding new modules, changes in layer activations could subtly impact inference speed or memory usage. Did you measure any changes in inference latency, memory footprint, or activation sparsity as a result of this training method?

Eighth, the analysis shows that removing certain negative spatial channels actually raises accuracy, implying some channels actively harm performance. The proposed training method does not explicitly address how to suppress these negative spatial channels during learning. Ignoring this issue might leave obvious room for further improvement by actively penalizing harmful representations. Could incorporating a regularization term that explicitly suppresses the identified negative spatial channels lead to even better overall performance?

6 comments

Thank you for taking the time to review our work and for your detailed suggestions on generalization, efficiency, and geometric interpretation. Below, we respond to each question with the relevant experiments and clarifications.

W1. Does depth training help orientation, scale, and multi-object reasoning? (Q1)

Response.{\color{green}\mathbf{Response.}} The gains extend beyond depth comparison. Table 18 (Table 9 in the original submission) shows mean gains across the three backbones of 6.8 points in relative distance, 5.0 in room size, 4.8 in absolute distance, and 1.7 in relative direction. To address your broader-task concern, we also test Qwen3.5-9B with identical natural QA and training steps:

VSI-Bench subtaskSFTOursGain (pp)
Object count64.8366.07+1.24
Absolute distance49.6655.10+5.44
Object size75.7581.69+5.94
Room size72.9577.99+5.04
Relative distance39.5846.34+6.76
Relative direction60.7264.93+4.21
Route planning51.5555.15+3.60
Appearance order80.4283.01+2.59
Mean61.9366.28+4.35

All eight subtasks improve, including relative direction (+4.21), room size (+5.04), and route planning (+3.60). Direct evaluation of the same checkpoints on Spatial457 [1], without further training, also improves orientation and relation composition:

Spatial457 subtaskSFTOurs
Single-object recognition72.172.8
Multi-object understanding66.467.7
2D spatial relations61.863.3
Occlusion relations58.161.5
3D orientation46.248.7
6D spatial relations43.846.2
Collision prediction45.247.7
Seven-task mean56.258.3

The 6D and collision tasks combine object position and orientation.

The same Qwen3.5-9B checkpoints also improve the overall ViewSpatial-Bench score from 53.7 to 56.6, a gain of 2.9 points. This benchmark covers camera- and person-centered localization, providing further evidence of transfer to spatial judgments involving different reference perspectives.

Both auxiliary objectives use depth and near/far labels, and their gains transfer to judgments involving orientation, viewpoint, and multiple spatial relations.

[1] Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models.


W1. Are the gains SPAR memorization, and what happens on MMSI-Bench? (Q5)

Response.{\color{green}\mathbf{Response.}} To address your concern about scenes and question formats, the existing Qwen3.5-9B checkpoints are evaluated directly on Spatial457 [1], with no further training. Its SuperCLEVR-based rendered scenes and question-generation pipeline differ from SPAR's training construction.

Spatial457 subtaskSFTOurs
Single-object recognition72.172.8
Multi-object understanding66.467.7
2D spatial relations61.863.3
Occlusion relations58.161.5
3D orientation46.248.7
6D spatial relations43.846.2
Collision prediction45.247.7
Seven-task mean56.258.3

All seven tasks improve, including 3D orientation, 6D relations, and collision prediction. This supports transfer beyond SPAR's scenes and question construction. Appendix B.1 also separates training, calibration, and evaluation by physical scene. Auxiliary prompts omit numerical depth labels and balance colors, entity order, near/far wording, and option positions. SFT receives the same natural QA from all auxiliary examples.

Ours improves MMSI-Bench by 1.8 points over same-data SFT. Table 3 (Table 2 in the original submission) reports:

Qwen2.5-VL-3B trainingMMSI-Bench
Same-data SFT26.9
VG-LLM29.0
Ours28.7

The 28.7 score is within 0.3 points of VG-LLM's 29.0 while using the original inference architecture. VG-LLM additionally uses a pretrained geometry encoder at inference.

The auxiliary supervision requires object locations and camera-axis depth. These geometric labels are also available from sources such as Omni3D [2], which provides object-center camera coordinates. Here, SPAR parsing supplies the training labels, and Spatial457 provides an independent test of transfer.

[1] Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Models.

[2] Omni3D: A Large Benchmark and Model for 3D Object Detection in the Wild.

W2. Are the gains meaningful relative to training complexity, and which errors are corrected? (Q2)

Response.{\color{green}\mathbf{Response.}} The reported 1.7–3.8 points are five-benchmark mean gains. To address your cost concern, we compare training under the same time budget. The 9B experiments also test a larger model and a broader data mixture. These results support the scalability of our training method with respect to both model size and training data.

Qwen3.5-9B dataMethodVSICVBLINK-SpatialMMSIViewSpatialMean
OriginalSFT61.985.262.832.353.759.2
OriginalOurs66.388.765.335.756.662.5
ExpandedSFT66.985.462.532.754.160.3
ExpandedOurs72.789.467.436.156.964.5

The expanded set contains 361K examples after adding VLM-3R and VSI-590K data. Sampling of the added sources follows SpatialStack [1]. Each SFT/Ours pair shares natural QA, steps, and evaluation settings. All five benchmarks improve, with mean gains of 3.3 and 4.2 points.

On the original set, Qwen3.5-9B uses the manuscript's 4B LoRA configuration and four A100 GPUs:

Training conditionMean scoreGPU-hoursPeak memory (GiB)
SFT59.2~9325.95
Ours, SFT time budget62.1~9326.50
Ours, SFT step budget62.5~10226.50

The matched-time gain is 2.9 points. Full training costs approximately 10% more GPU-hours and 0.55 GiB more peak memory.

To avoid GPU out-of-memory errors, we shift the labels for next-token prediction and compute the output-head projection and cross-entropy loss only at positions whose shifted labels are not -100. We process these positions in token chunks and checkpoint each chunk, recomputing the projection and loss during backpropagation. This avoids allocating the full [batch, sequence, vocab] logits tensor and reduces the intermediate tensors retained for backpropagation.

Representative corrections under identical inputs are:

ExampleRequired judgmentSFT answerOurs answerGround truth
CV-Bench omni3d_hypersim_44Lamp or counter closer to the camera?Counter (B)Lamp (A)Lamp (A)
VSI-Bench 1334Chair, stool, stove, or sofa closest to the TV?Sofa (D)Chair (A)Chair (A)
VSI-Bench 980Standing by the stove and facing the stool, where is the TV?Back-left (D)Front-left (C)Front-left (C)

These illustrate corrected camera-depth ordering, object-to-object distance, and a reference-frame judgment. The mechanism motivating training is that available geometry must also guide downstream computation. Table 5 (Table 3 in the original submission) supports this distinction: the full method scores 54.5 versus 51.9 with depth supervision alone and 52.0 with matched natural-QA replay. The controlled intervention supplies a relation-dependent training signal beyond repeated QA exposure.

[1] SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning.


W3. Does the finding hold outside Qwen? (Q3)

Response.{\color{green}\mathbf{Response.}} InternVL3-9B reproduces the ordering and benefits from the training strategy. Its InternLM3-8B language backbone [1] addresses your request for evidence from another model family. Directions are estimated separately and evaluated on scene-disjoint confirmation examples.

InternVL3-9BHigh-readout layersStrong-response window centersEarly readout (%)Early response
SFTL18–L26L24–L34670.06
OursL11–L26L16–L32710.11

Shared thresholds are 50%+0.8(ASFT,peak−50%)50\%+0.8(A_{\mathrm{SFT,peak}}-50\%) and 0.8USFT,max0.8U_{\mathrm{SFT,max}}. Intervals are the first runs of at least three consecutive above-threshold positions, ending before the next subthreshold position. Responses use three-layer windows at ρ=10%\rho=10\%. Early means use L11/L14/L16 and windows L10–L12/L13–L15/L15–L17, respectively.

Training advances the high-readout onset by seven layers and the strong-response onset by eight. The high-readout interval still starts before the strong-response interval. With matched QA, steps, tuning mode, and evaluation, all five benchmarks improve:

InternVL3-9BVSICVBLINK-SpatialMMSIViewSpatialMean
SFT57.685.172.829.746.958.4
Ours62.287.874.432.550.261.4

On InternVL3, training improves early depth readout, advances the onset of strong answer responses, and raises the five-benchmark mean by 3.0 points. The mechanistic observation and the training benefit therefore both extend to this model family.

[1] InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models.

W4. Does the gradient readout capture meaningful geometry? (Q6)

Response.{\color{green}\mathbf{Response.}} The depth directions track ground-truth geometry on held-out scenes. Section 3.1 and Appendix A.1 estimate directions on one split, then fix them for evaluation against camera-frame object depths. On 1,000 scene-disjoint confirmation examples, peak accuracy is 70.0% for Qwen-SFT and 74.0% for VG-LLM. Confirmation labels are used solely for evaluation.

To address your request for further validation, the following tests hold textual cues fixed and include human evaluation of open-ended descriptions.

Visual grounding. In 200 real view pairs, the same objects have opposite annotated depth orderings. Object names, wording, mention order, and options stay fixed. Success requires correct readouts for both views, testing whether the direction follows the visual reversal under identical text.

Answer changes. Depth edits preserve the image, question, pair mean, and edited-layer H/V readings. We evaluate 320 non-tied judgments per model that initially oppose the target. Flip rates are reported at each model's highest-flip layer: Qwen-SFT L22 and VG-LLM L17.

Open-ended descriptions. On the same 200 pairs, the target view's natural depth projection supplies the source-view edit target. The source image stays fixed. Prompts ask for the spatial relation without providing options or the target relation. Directions and layers are fixed before evaluating descriptions. Human evaluators extract each description's near/far relation, blind to the condition and target answer. Agreement is measured against target-view annotations. Each condition uses all 200 examples, with ambiguous, missing, or contradictory answers counted as failures. Values below are percentages.

TestConditionQwen-SFTVG-LLM
Cross-view readout: both views correctGradient direction61.066.5
Cross-view readout: both views correctRandom direction (mean)14.517.0
Conditional answer flipDepth edit55.658.8
Conditional answer flipEqual-norm random edit0.91.25
Description: agreement with target relationSource view, natural state17.015.5
Description: agreement with target relationTarget view, natural state74.578.0
Description: agreement with target relationSource view, depth edit58.564.0
Description: agreement with target relationSource view, equal-norm random edit15.016.0

The readout follows visual depth reversals under fixed text. Directional edits change both answer choices and the spatial content of descriptions assessed by blinded human evaluators. These tests connect ground-truth depth order with readout accuracy, answer choices, and open-ended descriptions, providing direct evidence of geometric meaning and behavioral influence.


Q4. Is paired editing complex or sensitive to feasibility thresholds?

Response.{\color{green}\mathbf{Response.}} The paired edit has a closed-form solution that satisfies the constraints you mention in a single update. In Equation (7), let aea_e be the depth-head direction and d^=ae⊤(hA−hB)\hat d=a_e^\top(h_A-h_B):

\qquad h_A'=h_A+\delta,\qquad h_B'=h_B-\delta.$$ The opposite shifts preserve the pair mean. Their alignment with $a_e$ preserves orthogonal components and sets the depth difference to $d'$. Appendix B.3 computes the feasible target interval before sampling. Auxiliary examples retain natural QA and depth supervision. The edit branch processes examples that pass the reliability and feasibility checks. Depth warmup and downstream-only use-loss gradients further separate coordinate learning from edited-answer training (Appendix B.4). Matched Qwen2.5-VL-3B sensitivity runs give: | Setting | State-change cap | Minimum depth margin | Object/pair error caps | Mean score | |---|---:|---:|---:|---:| | Default | 10% | 0.10 | 0.50 | 54.5 | | Relaxed state-change cap | 15% | 0.10 | 0.50 | 54.4 | | Relaxed reliability filters | 10% | 0.05 | 0.75 | 54.2 | Thresholds use standardized depth units, and the correct-sign requirement is retained. Scores vary by at most 0.3 points under these relaxations. Both settings outperform same-data SFT (50.7) and depth-only training (51.9), showing that the gain persists with a larger edit cap or looser reliability filters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.