acceptodds
Withdrawn from ICLR 2027

Gated Depth Attention: Preserving Vision-Language Understanding for Generalist Robot Policies

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models (VLMs) underpin modern vision-language-action (VLA) policies, yet remain fragile on spatially grounded queries and manipulation scenarios where 3D geometry matters. Prior 3D enhancements typically either align visual representations to geometry-centric teachers or explicitly inject depth/geometry tokens, both of which can perturb the pretrained RGB–language interface. We propose Gated Depth Attention (GDA), a lightweight module that distills a latent geometry branch from a frozen 3D foundation teacher and fuses it as a gated residual. GDA retrieves geometry through cross-attention while a learned per-channel gate controls how strongly the retrieved correction modifies each visual token; the external 3D teacher is removed at inference. Across VLM benchmarks spanning relational grounding, VQA/OCR, and robustness, GDA consistently outperforms alignment and concatenation baselines and matches or improves strong RGB-only models, yielding a better trade-off between geometry sensitivity and generic vision-language utility. When transferred to VLA policies, GDA also improves manipulation success in LIBERO, RoboTwin 2.0, and targeted real-world spatial/contact tasks. Teacher-swap experiments further preserve the benefit across two distinct 3D foundation teachers. These results suggest that stronger 3D supervision does not automatically preserve vision-language compatibility, and that geometry is better integrated as a selective refinement than as an always-on replacement.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, spatial reasoning, 3D geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TAAzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to introduce geometric knowledge into VLMs and VLA policies while preserving their existing vision-language capabilities. GDA trains an auxiliary latent geometry branch using a frozen 3D teacher, retrieves geometry through cross-attention, and injects the resulting correction into RGB tokens through a per-channel gate. The external teacher is removed at inference. The paper evaluates both vision-language compatibility and manipulation performance, including LIBERO, RoboTwin 2.0, and four targeted real-world tasks.

Critical strengths
  1. The compatibility-versus-geometry question is useful for generalist robot policies. Evaluating both VLM capabilities and downstream control makes the contribution more informative than reporting manipulation success alone. Relevant controls include capacity/no-distillation baselines, shuffled teacher targets, teacher swaps, and gate interventions. Spatial Forcing is already compared across VLM and robotic evaluations, with an additional distillation-weight sweep. The supplementary code and rollout examples also improve inspectability.

  2. The reported manipulation gains are substantial in some matched settings. Table 2 reports 0.5B LIBERO average success rates of 86.3% for the reproduced base policy, 83.6% for Spatial Forcing, and 95.8% for GDA. Table 3 reports 49.4%, 58.8%, and 68.1%, respectively, across the targeted real-world conditions. These results constitute meaningful positive evidence within the evaluated settings.

Critical weaknesses
  1. The specific mechanism contribution is not fully isolated by the ablations. Geometry distillation with teacher removal at inference is already established by Spatial Forcing, which the paper appropriately compares against. GDA's specific increment is the latent-branch/gated-fusion design, but a matched simpler channel gate or RGB-only-conditioned gate would help identify its independent benefit. Table 14's three-seed overall VLM improvement is not statistically detectable at that sample size, and the matched 7B LIBERO advantage over Spatial Forcing is only 0.1 percentage points. These limitations do not negate the larger gains in other settings.

  2. The evidence for an advantage on the stronger pi0 policy is limited specifically on LIBERO. In Table 2, average success increases from 93.1% to 94.3% (+1.2 percentage points), mainly from Goal (92.0% to 96.2%); Object is unchanged and Long slightly decreases (85.6% to 85.4%). The high baseline leaves limited headroom, but Table 2 also lacks a matched pi0 + Spatial Forcing comparison, so GDA's incremental benefit over geometry distillation is not established in this setting. Spatial Forcing comparisons are provided for the pi0-based real-world experiments (Table 3) and RoboTwin (Table 18); this concern is specific to pi0-LIBERO.

Questions for authors
  1. Can the existing ablations distinguish the proposed gate conditioning from a matched learned channel gate or RGB-only-conditioned gate? Please clarify what the evidence supports about the specific fusion mechanism, beyond the reported task improvements.

  2. For pi0-LIBERO, are matched pi0 + Spatial Forcing results available under the same fine-tuning recipe? If not, please qualify the claim of an advantage over geometry distillation for this setting, and discuss the modest, Goal-concentrated gain and possible ceiling effects. This request is limited to the existing pi0-LIBERO setting.

  3. The supplementary fusion paths appear to use different residuals. Which implementation corresponds to the reported experiments, and is the difference simply an outdated supplementary file? A version clarification would suffice initially; no full retraining is requested.

Reviewer_pqgDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Gated Depth Attention (GDA). A linear branch on DINO features is distilled toward a frozen 3D teacher (DA3, with VGGT as an alternative) and projected into the LLM embedding space. RGB tokens cross-attend to the resulting geometry tokens, and a per-channel sigmoid gate controls how much of that residual is added to the RGB tokens. The teacher is removed at inference, so the LLM interface is unchanged.

The method is evaluated on 10 VLM benchmarks (Qwen2.5 0.5B–7B), LIBERO (OpenVLA-OFT and π0), 7 RoboTwin 2.0 tasks, and 4 real-robot tasks. The paper reports that geometry baselines degrade VLM utility while GDA preserves it, and that GDA improves VLA success.

Critical strengths
  1. Practical design. The teacher is used only during training, the module adds ~8.5M parameters with negligible inference cost, and the token count and LLM input format are unchanged.
  2. Strong gains in harder settings. On RoboTwin-hard, success rises from 23.0 to 42.7, versus 31.3 for Spatial Forcing. In the paired real-world evaluation, exact McNemar tests give 36:6 against the base policy and 20:5 against SF.
  3. Thorough and candid controls. These include λ=0\lambda=0, a shuffled teacher, a direct-teacher control, gate interventions, and matched-seed repeats. The authors openly report that the overall VLM gain is not statistically significant.
Critical weaknesses
  1. The code and the paper describe different fusion operations. In the released VLM pretraining code (prismatic/models/vlms/prismatic.py, ~l.1214), the output is rgb_tokens + g * depth_tokens, i.e. V^+g⊙D^\hat V + g \odot \hat D, and the attention output only feeds the gate. The VLA model (modeling_prismatic.py, ~l.594) computes V^+g⊙Δ\hat V + g \odot \Delta, matching Eq. 2. If this is what was run, the VLM results do not evaluate Eq. 2, and pretraining+FT loads weights trained for one computation into another.
  2. The Table 2 statistics do not fit the stated protocol. With 500 rollouts per run, a three-run mean should round to an odd tenth about two-thirds of the time, yet all 32 controlled per-suite means are even tenths. The across-seed s.d. values (0.1–0.3pp, including on the 69% Long suite) are also implausibly small.
  3. The baselines may be misconfigured or weak. The SF implementation aligns hidden_states[24], which is the final layer of Qwen2.5-0.5B; consistent with this, SF hurts at 0.5B but not at 7B. Dualhead-SF loses ~24 points on VQA-v2 despite an identity-initialized branch. The reproduced base policies are weak: OFT-0.5B scores 87.8 on LIBERO-Object, and π0\pi_0 scores 92.0 on Goal against a published 95.8. GDA-π0\pi_0 (94.3) is roughly equal to the published π0\pi_0 (94.2).
  4. The VLM claims and the explanation of the mechanism are overstated. The authors' own seed analysis finds no detectable gain over RGB-only. The gains come mostly from noisy TallyQA and VizWiz, and S-Avg improves by only +0.3 on DINO+SigLIP. VSR sits at chance (48.5) yet underpins the gate interventions. Each ablation is a single seed. UDA's constant α\alpha can be absorbed into WVW_V. Distillation also trains the encoder through a linear head, which blurs the claimed difference from alignment methods.
Questions for authors
  1. Which fusion operation produced each result, and how does VLA fine-tuning handle the mismatch? Please report Tables 1 and 4 using Eq. 2 as written.
  2. Please give per-run success counts for the controlled rows of Table 2 and explain how the means and s.d. values were computed.
  3. Which LLM layer and projector does SF use at 0.5B? Do the results change if an intermediate layer is aligned?
Reviewer_L1wnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GDA distills a geometry branch from a frozen 3D teacher (used only in training) and adds it to the RGB tokens through cross-attention and a learned gate. This keeps general VLM ability: the average goes 43.2→44.1, while other 3D methods drop below RGB-only. It also improves robot success: LIBERO 86.3→95.8 at 0.5B, and 49.4→68.1 on a real robot.

Critical strengths
  • a good observation that directly adding a new modality (3d) into the VLA model might destroy the internal representation/knowledge prior of the model.
  • gating is generally a plausible and reasonable attempts to avoid model distribution deviation
Critical weaknesses
  • VQA co-training at policy training stage is a common practice and the paper did not discuss it. previous work including: π0.5(2025, CoRL), Knowledge Insulation(2025, NeurIPS), ChatVLA(2025, EMNLP), GR-3(2025, arXiv), Qwen-VLA(2026, arXiv).
  • evaluation is far from supportive: -- by the authors' own seed analysis, GDA only matches the RGB-only baseline. -- The geometry claim isn't tested: the "spatial" benchmarks are at chance or near their floor, and none of them measures 3D. -- The robot gains are inflated: they show up only against a weak reproduced baseline, disappear on strong ones, and the reported variance is too small to be credible.
Questions for authors
  • a baseline of VQA cotraining at policy trainig stage

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.