- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes SpatialLAM, a latent action model (LAM) with explicit spatial structure. SpatialLAM uses spatial latent action queries that attend with a locality bias based on their grid positions, implemented by adding a distance bias to the pre-softmax logits. Combined with policy pretraining and finetuning on a BAKU backbone, SpatialLAM’s performance on LIBERO, LIBERO-Plus, and real world experiments demonstrates its effectiveness for pretraining policies without action labels and for generalization. In addition, its world model prediction performance on BridgeV2 and SSv2 shows that it can serve as a rich and effective control signal for world model prediction. The ablations and diagnostics are extensive, ranging from comparisons with various design choices and external baselines, cross position generalization tests with shuffled grids and permutations, and action retrieval, to world model visualizations and failure cases.
Critical strengths
- Straightforward motivation, idea and methodology. Motivated by the observation that prior latent actions are mostly an unbiased list of tokens, adding an explicit spatial bias is a straightforward way to improve latent actions, since actions and visual dynamics inherently have spatial properties. SpatialLAM’s methodology, which aligns the spatial latent action queries with the locality bias, is simple and clear, and it effectively bridges the gap between spatial representations and the structure of latent actions. The design is well evaluated in simulation and the real world, and it is ablated and analyzed extensively to justify each design choice and to show effective extensions to policies and world models.
Critical weaknesses
-
Fair comparison with other designs. SpatialLAM is a complementary design that could be applied to existing latent action based models such as UniVLA. Yet the analysis is almost entirely limited to unstructured LAM vs. SpatialLAM within the authors’ own BAKU policy pipeline. Prior methods appear only as numbers reported in their papers, with different policy architectures and pretraining. Although OTF-LAM and PoLAR are compared in the ablation using the authors’ policy design, there is no experiment applying SpatialLAM to an existing LAM design to show that it actually complements them. This matters because such evidence would support SpatialLAM not only as an improvement over the unstructured baseline or existing baselines, but also as a broader claim about latent action design.
-
Experimental Rigor. LIBERO-Plus is used as evidence of generalization performance, but the perturbation-wise performance relative to other baselines is inconsistent. The policy is much stronger under camera-view shifts and sensor noise, while being weaker under background texture changes. The unstructured baseline shares this property, so it most likely stems from the common policy pipeline rather than from SpatialLAM. These factors are under-analyzed, and while the authors state that they used only color jitter, it is not stated whether the baselines used comparable data augmentation. SpatialLAM might show much higher performance with proper improvements to the training pipeline. Also, the performance gain under camera view shifts can be explained by its spatial generalization capability, but the low performance under background texture changes raises the question of whether SpatialLAM can extract task-relevant information in OOD situations.
Questions for authors
-
Questions for Weaknesses. Could you explain (a) whether applying SpatialLAM to existing LAMs such as UniVLA, or even to other structured LAM baselines like OTF-LAM (e.g., Spatial OTF-LAM, if applicable), leads to meaningful improvements over those baselines, and (b) why the policy’s perturbation-wise performance differs from the baselines, being weakest under background texture changes? Is there evidence that explains this pattern (e.g., the rich semantics of DINOv3 ironically hindering generalization under different semantics, or the limited capacity of the policy’s ResNet-based vision branch)?
-
Does SpatialLAM itself act as a bottleneck?. In Table 12, SpatialLAM’s performance at 8x8 resolution matches that at 4x4, while the unstructured LAM degrades. Could SpatialLAM itself act as a semantic bottleneck, not by limiting its capacity but by restricting information flow through the spatial bias? Or would performance eventually drop at denser resolutions even for SpatialLAM, requiring an explicit bottleneck design?
5 comments
Q2. Spatial structure and latent bottleneck
Thank you for this insightful question. To examine whether spatial structure can replace an explicit bottleneck, we extend the capacity ablation to a 16×16 latent grid.
Table R15. Latent capacity ablation extended to a 16×16 grid (Table 12). Success rate (%) on LIBERO-Long and the LIBERO-Plus subset.
| Grid | Total dim. | Unstructured: Long | Unstructured: Plus | SpatialLAM: Long | SpatialLAM: Plus |
|---|---|---|---|---|---|
| 2×2 | 16 | 63.4 | 42.4 | 69.4 | 59.2 |
| 4×4 | 64 | 76.8 | 63.2 | 83.0 | 69.6 |
| 8×8 | 256 | 71.6 | 57.1 | 82.6 | 70.1 |
| 16×16 | 1,024 | 66.8 | 53.5 | 72.6 | 61.7 |
At 8×8, SpatialLAM maintains performance comparable to the 4×4 grid, whereas the unstructured LAM drops by 5.2/6.1 pp. At 16×16, SpatialLAM degrades by 10.0/8.4 pp relative to 8×8, while still outperforming the capacity-matched unstructured LAM (72.6/61.7 vs. 66.8/53.5).
Since ρ is measured in grid spacings (Equation 7, default ρ=1), the locality width in the image narrows from four patches at 4×4 to a single patch at 16×16. To separate this effect from the larger capacity, we further train the 16×16 SpatialLAM with ρ=4, which matches the four-patch locality width of the default 4×4 grid. This variant recovers 2.8/2.5 pp (75.4/64.2) but still trails the 4×4 grid by 7.6/5.4 pp. The degradation persists even when the spatial locality width is matched.
These results suggest that spatial structure mitigates degradation at larger latent capacities but does not eliminate sensitivity to bottleneck size. Spatial organization and a compact latent bottleneck therefore serve complementary roles. The spatial bias also restricts information flow at a fixed capacity. At the same KL weight, SpatialLAM has a 15.7% lower posterior rate than the unstructured LAM (279.7 vs. 331.7 nats) while achieving a lower next-frame feature MSE (Appendix C.6), consistent with your hypothesis of an implicit bottleneck. Notably, when the KL weight is tuned to match the posterior rate, SpatialLAM still outperforms the unstructured LAM by 5.6 to 6.4 pp on LIBERO-Long, so its advantage does not stem from the lower rate alone.
Dear reviewer, thank you again for your insightful comments on our paper, and we genuinely hope that our response could address your concerns. As the discussion is about to end, we are sincerely looking forward to your feedback. Please feel free to contact us if you have any further inquiries.
W1 & Q1(a). Integration with existing LAMs
Thank you for the thoughtful suggestion. To examine whether the benefits extend beyond BAKU, we integrate SpatialLAM into UniVLA and change only the latent action interface. The LAM keeps the native design of UniVLA, including its visual feature backbone, the two-stage training of language-conditioned task-irrelevant and task-centric latents, and VQ quantization. We only arrange the latent action queries as a spatial grid with locality-biased attention. As in UniVLA, the quantized codebook indices are added to the vocabulary of the Prismatic-7B policy as special tokens and predicted with cross-entropy, and the action decoder reads the latent action embeddings through the 3×3 convolutional projector. Following UniVLA's Bridge V2 policy-pretraining recipe, we pretrain for 20k steps (LR 2e-5) using latent-token cross-entropy, followed by LIBERO fine-tuning with LoRA (rank 32, α=16) for 30k steps (40k for LIBERO-Long), using latent-token cross-entropy and action L1 losses.
Table R13. UniVLA as the base pipeline. Success rate (%) on LIBERO-Long, LIBERO (average), and LIBERO-Plus (total).
SpatialLAM improves over the reported UniVLA results by 2.1 pp on LIBERO-Long, 1.1 pp on LIBERO average, and 5.7 pp on LIBERO-Plus, with the largest gains under camera-view (1.8 to 12.9) and object-layout (31.9 to 42.2) shifts. These results suggest that the benefits of spatial structure extend to a stronger VLA pipeline and are not specific to BAKU. We have added these results to Appendix C.5.
W2 & Q1(b). Perturbation-wise robustness on LIBERO-Plus
Thank you for the valuable question. The background-shift weakness is already present in BAKU without latent actions, which suggests that it is largely associated with the shared policy pipeline rather than introduced by SpatialLAM. We provide the no-latent control and a comparison with UniVLA:
Table R14. Selected LIBERO-Plus perturbation factors. Success rate (%) under the BAKU and UniVLA pipelines.
All BAKU variants share the same ResNet-18 policy encoder, LIBERO fine-tuning data, and color-jitter augmentation. DINOv3 provides features for LAM training but is not used as the policy's visual encoder at inference. The prior methods in Table 2 use different architectures and training recipes, so their perturbation profiles are not augmentation-controlled comparisons.
UniVLA performs much better under background shifts, suggesting that absolute per-factor performance depends substantially on the policy pipeline. This contrast is consistent with your second hypothesis. BAKU encodes observations using a ResNet-18 trained from scratch, whereas UniVLA employs large pretrained DINOv2 and SigLIP visual encoders. The same weakness in the no-latent baseline indicates that this pattern does not require DINOv3-derived latent targets. Nevertheless, SpatialLAM improves background-shift success within both pipelines: by 7.4 pp over the unstructured LAM and 16.8 pp over the no-latent baseline in BAKU, and by 2.6 pp in UniVLA, indicating that it retains useful task information under appearance shifts. Larger gains under camera-view (+10.2 pp in BAKU and +11.1 pp in UniVLA) and robot-initialization (+9.1 pp in BAKU) shifts are consistent with the benefits of spatial grounding.