- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the relationship between depth information in multimodal models’ internal representations and its influence on spatial answers. Layerwise readouts and interventions suggest that peak depth readability precedes peak answer influence. Motivated by this observation, the authors combine early-layer depth supervision with edited-answer supervision that trains downstream layers to respond to modified depth relations. The method improves spatial QA across three Qwen backbones while retaining the original inference architecture.
Critical strengths
- An interesting mechanistic hypothesis with supporting evidence. The paper connects layerwise depth readability with downstream answer influence. Readouts, directional interventions, and channel ablations provide complementary evidence for depth-related internal information and its behavioral relevance. This is a useful direction for understanding spatial reasoning failures.
- A practical training approach with controlled empirical improvements. The proposed objectives improve all five benchmark scores over same-data SFT across three backbones without adding inference components. Replay, random-perturbation, and gradient-separation controls help establish that the improvements are associated with the proposed training design.
Critical weaknesses
- The generality of the insight remains insufficiently established. All evaluated backbones belong to the Qwen family and are limited to 3B–4B. The detailed mechanistic evidence is narrower than the cross-backbone QA evaluation. It remains unclear whether the same structure and layerwise ordering hold for other model families, larger models, or different amounts and mixtures of spatial SFT data. On VSI-Bench, the gains over VG-LLM are 1.5, 0.4, and 0.5 points across the three backbones. These smaller margins on the newer backbones motivate further investigation, although they do not establish a scaling trend.
- The evidence does not fully distinguish the proposed mechanism from alternative explanations. Reduced accuracy under the chosen linear readout need not imply reduced depth information. Later layers could retain spatial information in a different, potentially more semantic encoding that the readout does not recover. Fixed-polarity accuracy below 50% also indicates inverse correspondence rather than an absence of information. The interventions establish answer sensitivity to the selected directions, but further evidence is needed to establish their geometric specificity. Moreover, early and late configurations may differ in which examples pass the reliability and feasibility filters, potentially confounding the placement comparison.
Questions for authors
- Can the core readout and intervention findings be replicated on a larger model and a model family outside Qwen? Please also discuss their dependence on spatial SFT data amount and composition, and clarify the VG-LLM training setup used in Table 2.
- Can independently trained geometric probes, including probes with greater capacity, recover depth information from later layers? Such comparisons, together with validation-calibrated polarity and intervention selectivity controls, would help distinguish changes in encoding from changes in information availability and use.
- The reliability and feasibility filters depend on the depth readout at the selected layer. Do the early- and late-layer configurations therefore receive different amounts or types of edited-answer training? Please report the number of examples actually edited, the proportions of relation-preserving and relation-reversing edits, and the perturbation magnitudes for each configuration. Can you repeat the comparison on examples eligible in both configurations, with matched edit types and comparable perturbation magnitudes? This would help determine whether the early-layer advantage comes from placement itself or from differences in the edited training examples.
If my phrasing of the issue is not sufficiently precise, I encourage the author to engage in further discussion with me.
5 comments
W2. Late-layer information, polarity, and geometric specificity (Q2)
You are right that depth information can remain available in late layers. Independent linear probes confirm this directly:
| Diagnostic | Qwen-SFT | VG-LLM |
|---|---|---|
| Independent depth probe: peak accuracy | 87.17% (L22) | 88.50% (L23) |
| Independent depth probe: final-layer accuracy | 84.83% (L35) | 86.00% (L35) |
| Answer-gradient readout: peak accuracy (Figure 1) | 70.00% (L17) | 74.00% (L8) |
| Answer-margin response: peak center (Figure 3) | L21 | L17 |
Both models support high late-layer probe accuracy, with similar peak locations. The answer-gradient readout peaks much earlier in VG-LLM. These results distinguish linearly decodable depth from its alignment with the model's answer computation.
A probe learns an additional decoder. Our readout measures geometry along the model's own answer-sensitive direction. Let be the final near/far answer-logit difference and . With the pair mean and other token states fixed,
Section 3.1 fixes the averaged direction before evaluating ground-truth relations. Section 3.2 then edits states and reruns the downstream network. This separates recoverable information from its influence on answers, consistent with the knowledge–prediction distinction studied by Park et al. [1]. Independent probes show similar depth recoverability in the two models, while our answer-gradient readout and controlled interventions reveal differences in when and how strongly depth-related states influence their answers. The interventions establish these causal effects within the models' own answer computation, which probe accuracy alone does not test.
Your polarity interpretation also matches Appendix A.1: below-50% fixed-polarity accuracy indicates inverse ordering. After polarity calibration, the gap from independent probes and the between-model differences in layerwise answer-related readout remain. Together, the probes and interventions identify differences in answer-related accessibility and downstream influence.
To address your geometric-specificity concern, the following controls test visual correspondence and behavioral effects separately.
Visual grounding. In 200 real view pairs, the same objects have opposite annotated depth orderings. Object names, wording, mention order, and options stay fixed. Success requires correct readouts for both views, testing whether the direction follows the visual reversal under identical text.
Answer changes. Depth edits preserve the image, question, pair mean, and edited-layer H/V readings. We evaluate 320 non-tied judgments per model that initially oppose the target. Flip rates are reported at each model's highest-flip layer: Qwen-SFT L22 and VG-LLM L17.
Open-ended descriptions. On the same 200 pairs, the target view's natural depth projection supplies the source-view edit target. The source image stays fixed. Prompts ask for the spatial relation without providing options or the target relation. Directions and layers are fixed before evaluating descriptions. Human evaluators extract each description's near/far relation, blind to the condition and target answer. Agreement is measured against target-view annotations. Each condition uses all 200 examples, with ambiguous, missing, or contradictory answers counted as failures. Values below are percentages.
| Test | Condition | Qwen-SFT | VG-LLM |
|---|---|---|---|
| Cross-view readout: both views correct | Gradient direction | 61.0 | 66.5 |
| Cross-view readout: both views correct | Random direction (mean) | 14.5 | 17.0 |
| Conditional answer flip | Depth edit | 55.6 | 58.8 |
| Conditional answer flip | Equal-norm random edit | 0.9 | 1.25 |
| Description: agreement with target relation | Source view, natural state | 17.0 | 15.5 |
| Description: agreement with target relation | Target view, natural state | 74.5 | 78.0 |
| Description: agreement with target relation | Source view, depth edit | 58.5 | 64.0 |
| Description: agreement with target relation | Source view, equal-norm random edit | 15.0 | 16.0 |
The readout follows real depth reversals. Targeted edits change both choices and open-ended descriptions far more than random edits. The natural target view provides visual evidence for the edited relation, and the source-view intervention measures its causal influence on the answer.
[1] Bridging the Knowledge-Prediction Gap in LLMs on Multiple-Choice Questions.
W2. Do filtering and edit distributions confound early versus late placement? (Q3)
The early configuration retains a 1.6-point advantage after matching edited examples, edit types, target answers, and the use-loss update range. Each counted instance is a completed paired edit contributing to answer supervision. is the larger per-object relative state change, capped at 10%.
| Configuration | Edited instances | Preserving: count (%) | Reversing: count (%) | (%), mean SD | Mean score |
|---|---|---|---|---|---|
| R2: early | 26,539 | 14,103 (53.1) | 12,436 (46.9) | 54.5 | |
| R3: late | 26,773 | 13,894 (51.9) | 12,879 (48.1) | 52.3 | |
| Early, common L23 suffix | 26,427 | 13,836 (52.4) | 12,591 (47.6) | 53.8 | |
| Late, common L23 suffix | 26,691 | 13,467 (50.5) | 13,224 (49.5) | 52.1 | |
| Matched early, common L23 suffix | 23,752 | 12,317 (51.9) | 11,435 (48.1) | 53.6 | |
| Matched late, common L23 suffix | 23,752 | 12,317 (51.9) | 11,435 (48.1) | 52.0 |
The first four scores are from Table 5 (Table 3 in the original submission). Edit statistics and matched-training scores are additional results. Early/late sets are L8/L10/L12 and L18/L20/L22. Both matched conditions restrict use-loss updates to L23–L35. They share initialization, QA batches and order, steps, loss weights, and checkpoint selection. Corresponding rotating layers edit the same jointly eligible example with the same edit type and target answer. Both skip when a common edit is infeasible.
Mean perturbations differ by only 0.04 percentage points. The matched advantage, 53.6 versus 52.0, closely retains the original common-suffix gap of 53.8 versus 52.1. The early advantage therefore persists with common examples, matched edit types, and nearly identical perturbation magnitudes. These controlled comparisons support early placement of the joint depth-supervision and editing configuration.
Thank you for the time and care you devoted to reviewing our work and for your valuable suggestions on the mechanistic analysis. Below, we address your questions about generalization, depth readouts, and the early–late comparison.
Response. The readout–response ordering also appears at 9B scale and outside Qwen. The additional experiments use Qwen3.5-9B and InternVL3-9B [1]. The latter uses an InternLM3-8B language backbone, providing a comparison across model families.
Directions are estimated separately per model and evaluated on scene-disjoint confirmation examples. For original-data SFT/Ours comparisons, shared thresholds within each backbone are 50%+0.8(ASFT,peak−50%) for readout and 0.8USFT,max for response. Each interval begins with the first run of at least three consecutive positions above threshold. It ends before the next subthreshold position. Interventions use three-layer windows at ρ=10%. Layers are zero-indexed.
High-readout onset precedes strong-response onset in every reported configuration. Training advances both intervals on each backbone. The expanded Qwen3.5-9B model retains this ordering.
The expanded mixture contains 361K examples, including additional VLM-3R and VSI-590K data. Sampling of the added sources follows SpatialStack [2]. Within each comparison, SFT and Ours share natural QA, training steps, tuning mode, and evaluation settings. Scores below are five-benchmark means (%).
All five benchmarks improve in each comparison. The gains persist at 9B across Qwen and InternLM backbones and across the original and expanded data mixtures. The layerwise diagnostics also show earlier depth readout and answer influence after training.
VG-LLM in Table 3 (Table 2 in the original submission) is retrained on each corresponding backbone, separately from the Section 3 analysis pair. All groups start before spatial fine-tuning and receive SPAR-234K, LLaVA-Hound-64K, and natural QA for the 30K auxiliary pairs. VGGT, the visual encoder, and the visual merger are frozen. The geometry mapper is trained.
Language tuning matches SFT/Ours. Qwen2.5-VL-3B uses full tuning at 10−5. The two 4B backbones use LoRA with rank 64, α=128, dropout 0.05, and learning rate 5×10−5. Training uses batch 64 and one epoch. Checkpoints are selected using QA from independent calibration scenes. Appendix B.2 specifies feature alignment. VG-LLM retains VGGT at inference. Ours uses the original architecture.
[1] InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models.
[2] SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning.