acceptodds
Withdrawn from ICLR 2027

SpatialAnything: Bidirectional Attention and Multi-Teacher Distillation for Embodied Spatial Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large vision-language models (LVLMs) recently demonstrate promising capabilities in embodied spatial reasoning, spanning spatial understanding, object localization, navigation, and long-horizon planning. These tasks require models to process both language and visual grounding primitives such as bounding boxes, keypoints, and trajectories. However, existing methods typically generate coordinates one at a time through autoregressive decoding, so coordinates within the same visual grounding primitive depend on different contexts and generation time grows with primitive length. Meanwhile, different spatial tasks build on shared localization and spatial relation understanding, yet differ in output formats and optimization objectives. We therefore present SpatialAnything, a natively multimodal model for embodied spatial reasoning built around Primitive Bidirectional Attention (PBA). PBA enables bidirectional attention among coordinate representations within a primitive through bidirectional masks in full-attention layers and query rewriting with write compensation in gated linear-attention layers. SpatialAnything further adopts adaptive parallel decoding, which treats each primitive as a whole and predicts all of its coordinates at once, removing the wait inherent in coordinate-by-coordinate generation. The predictions are then written into the context to support subsequent generation. Building on this, SpatialAnything uses multi-task joint training through a two-stage strategy based on Spatial Multi-Teacher On-Policy Distillation (Spatial MOPD), which first establishes a foundation for localization prediction with a unified primitive teacher and then lets task-specific teachers guide the student on its own generations, giving a single model complementary spatial capabilities. Across a range of embodied spatial reasoning benchmarks, SpatialAnything achieves the highest scores on multiple metrics, exceeding GPT 6 Astra by 5.2 points on average over ten spatial understanding and reasoning benchmarks, and delivers a 2.9× inference speedup.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Spatial Reasonsing, Embodied Intelliengence, Locate grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fAYmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SpatialAnything for spatial reasoning, grounding, egocentric interaction, and navigation. First, it organizes coordinate prediction around complete visual grounding primitives rather than individual coordinate tokens. Primitive Bidirectional Attention (PBA) permits interaction among masked coordinate positions within each primitive, using attention-mask changes in full-attention layers and query rewriting with write compensation in gated linear-attention layers. Adaptive parallel decoding predicts the coordinates together and causally replays the selected values into the context before continuing generation. Second, a two-stage Spatial Multi-Teacher On-Policy Distillation procedure integrates task-specific teachers on student-generated responses.

Critical strengths

I think this paper showing the below strengths:

  1. A technically motivated extension of structured prediction to hybrid attention. PBA addresses a concrete limitation of mask-only primitive decoding: gated linear-attention layers do not expose an attention matrix whose visibility can simply be modified. Table 4(a), where all variants share masked inputs and parallel decoding, reports the strongest localization performance when both attention types are modified. This makes the hybrid-attention treatment a substantive contribution beyond simply grouping coordinate tokens.

  2. Broad evaluation with evidence that grounding warm-up helps multi-task distillation. The paper evaluates coordinate grounding, spatial question answering, egocentric interaction, and navigation at three model scales. Table 4(b) is particularly informative: ordinary MOPD improves CV-Bench and MindCube over SFT but reduces RefCOCOg from 81.3 to 76.6, whereas Spatial MOPD reaches 88.3, 96.2, and 91.6 on the three respective benchmarks. This suggests that maintaining shared grounding competence is consequential when integrating task-specific expertise.

Critical weaknesses

This paper show the below weaknesses:

  1. Many benchmarks in Section 4 and training datasets in Appendix B are listed without direct citations to their original sources. This makes their provenance and evaluation protocols difficult to verify.

  2. LocateAnything already predicts boxes and points in parallel. The paper should distinguish this existing idea from its new contributions, particularly the extension to gated linear attention. Plus, despite emphasizing box/point localization and decoding speed, the paper does not compare with this closely related model on overlapping tasks. The benefit over existing parallel grounding methods therefore remains unclear.

  3. Appendix B describes the overall SFT corpus, but the datasets, splits, and sampling ratios for teacher training and the two distillation stages are not clearly reported. Several training and evaluation subsections are also incomplete.

  4. SA-4B scores 42.8 on ScanRefer in Table 1, whereas Full PBA without posttraining scores 52.1 in Table 4(a). MindCube is 90.6 in Table 2 versus 91.6 in Table 4(b). The paper does not explain whether these results use different checkpoints or protocols.

Questions for authors

Please see the weakness.

Reviewer_CtjmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

SpatialAnything combines primitive-level bidirectional attention in a hybrid full/linear-attention backbone with parallel coordinate prediction and two-stage multi-teacher distillation. The framework supports grounding, spatial question answering, interaction, and navigation at three model scales. The authors evaluate it on 25 benchmarks and report competitive performance alongside consistent inference speedup.

Critical strengths
  1. PBA addresses a specific challenge in hybrid backbones. It rewrites queries with future-write compensation to read primitive-end states while preserving the original recurrence. This is a technically meaningful design, supported by detailed derivations and author-provided numerical validation.
  2. The primitive interface is coherent across spatial outputs. It accommodates points, 2D/3D boxes, and trajectories. The decoding procedure explicitly restores context and causally replays predicted coordinates before continuing generation, addressing the risk of retaining placeholder-derived states in the cache.
  3. The evaluation includes useful controlled evidence. Full PBA improves ScanRefer from 40.8 to 52.1 and RefSpatial from 37.7 to 49.8 over causal parallel decoding, providing encouraging initial evidence for the attention modifications.
Critical weaknesses
  1. PBA enables interaction among masked coordinate representations, but the final coordinates are selected separately by per-position argmax. This does not itself demonstrate that the resulting primitive is geometrically coherent. The reported localization scores do not directly test the structural motivation. Invalid-box rates, trajectory continuity, and performance on ambiguous targets should be compared against matched autoregressive and parallel baselines.
  2. The comparison with standard MOPD matches teachers, task proportions, and rollout budgets, but Spatial MOPD adds a primitive warm-up stage. Its gains could therefore reflect additional training rather than the proposed transfer of shared localization knowledge.
  3. The main-text ablations contain an unresolved baseline inconsistency. Table 4(c)’s full-data setting, explicitly described as excluding reinforcement learning and distillation, reports 88.3/96.2/91.6 on RefCOCOg/CV-Bench/MindCube. These values exactly match the Spatial MOPD row in Table 4(b), while its SFT-only baseline reports 81.3/93.8/89.5. Unless different checkpoints or training configurations are involved, these comparisons are difficult to reconcile. The distinction must be clarified before the contribution of post-training can be assessed reliably.
Questions for authors

What accounts for 320→110 ms when both variants decode in parallel? Please provide an autoregressive baseline, component timings, and primitive-length scaling.

Reviewer_BbmbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper treats each grounding primitive (box, point, 3D box, trajectory) as a unit. Primitive Bidirectional Attention (PBA) lets coordinate tokens inside one primitive see each other. Full-attention layers do this with a mask; Gated DeltaNet layers use an exact query rewrite plus a correction term, so every coordinate reads the end-of-primitive state. Coordinates are then decoded in parallel from mask tokens and replayed causally. Training is SFT, then a two-stage multi-teacher on-policy distillation (Spatial MOPD) from GRPO-trained task teachers. Models at 4B/9B/27B on Qwen3.5 are evaluated on about 25 benchmarks.

Critical strengths
  • The linear-attention query rewrite is the most interesting part. It is exact, adds no parameters, needs no reverse scan, and the appendix validates it numerically. It addresses a real gap for hybrid backbones, where mask editing does nothing in linear layers.
  • Broad evaluation across grounding, spatial QA, egocentric reasoning and navigation, with consistent scaling from 4B to 27B.
  • The training corpus and system details are documented well.
Critical weaknesses
  • The main speed claim lacks the right baseline. The 2.9x speedup (320 → 110 ms) is measured against "causal parallel" decoding, not against standard token-by-token autoregressive decoding, which is the status quo the paper criticizes. It is also unclear why changing visibility inside a primitive changes latency threefold when both variants decode in parallel.
  • The comparisons are not apples to apples. The SFT corpus includes training data from VSI, MindCube, SenseNova-SI and related sources, so the gains on MindCube (94.8) and VSI-Bench (73.9) are partly in-domain, while the closed models are zero-shot. Baseline numbers are copied from other papers. The large gaps on RoboAfford (55.8 vs 80.4) and VABench-Point (14.6–20.5 vs 65.2) are not discussed.
Questions for authors
  1. Please explain the duplicated numbers in Table 4(b)/(c) and the ScanRefer mismatch between Table 1 and Table 4(a).
  2. How do accuracy and latency compare with plain autoregressive coordinate decoding?
  3. Coordinates within a primitive are taken by independent per-position argmax. Does this produce inconsistent outputs, such as x2 < x1 or jagged trajectories, for long primitives?
  4. What is the total compute of training four GRPO teachers plus two distillation stages?

I would raise this if the table inconsistencies are typos and the AR baseline is added.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.