acceptodds
Under review as a conference paper at ICLR 2027

SpatialLAM: Latent Actions Need Spatial Structure

Abstract

Latent action models (LAMs) learn compact representations of visual transitions, enabling policies and world models to learn from videos with little or no action annotation. However, most existing LAMs represent latent actions as spatially unstructured tokens, leaving their correspondence to the locations of visual changes implicit. This provides limited structural support for reusing local transition content across positions, especially in tasks that require predicting or controlling similar local interactions at different image locations. To address this limitation, we propose SpatialLAM, which organizes latent actions on a 2D grid aligned with the observation plane and uses locality-biased attention to ground each token in a consistent visual neighborhood. We evaluate SpatialLAM in both policy learning and world modeling under matched data, optimization, and latent capacity. SpatialLAM improves policy success from 76.8% to 83.0% on LIBERO-Long, from 64.6% to 70.7% on out-of-distribution LIBERO-Plus, and from 45.0% to 66.7% across three real-world manipulation tasks. It also reduces world-model FVD by 16.9% on Something-Something V2 and 10.9% on Bridge V2. These results suggest that the utility of latent actions depends not only on compactly encoding visual transitions, but also on explicitly organizing their spatial structure.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Latent Action, Robot Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CMCkOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SpatialLAM, a latent action model (LAM) with explicit spatial structure. SpatialLAM uses spatial latent action queries that attend with a locality bias based on their grid positions, implemented by adding a distance bias to the pre-softmax logits. Combined with policy pretraining and finetuning on a BAKU backbone, SpatialLAM’s performance on LIBERO, LIBERO-Plus, and real world experiments demonstrates its effectiveness for pretraining policies without action labels and for generalization. In addition, its world model prediction performance on BridgeV2 and SSv2 shows that it can serve as a rich and effective control signal for world model prediction. The ablations and diagnostics are extensive, ranging from comparisons with various design choices and external baselines, cross position generalization tests with shuffled grids and permutations, and action retrieval, to world model visualizations and failure cases.

Critical strengths
  • Straightforward motivation, idea and methodology. Motivated by the observation that prior latent actions are mostly an unbiased list of tokens, adding an explicit spatial bias is a straightforward way to improve latent actions, since actions and visual dynamics inherently have spatial properties. SpatialLAM’s methodology, which aligns the spatial latent action queries with the locality bias, is simple and clear, and it effectively bridges the gap between spatial representations and the structure of latent actions. The design is well evaluated in simulation and the real world, and it is ablated and analyzed extensively to justify each design choice and to show effective extensions to policies and world models.
Critical weaknesses
  • Fair comparison with other designs. SpatialLAM is a complementary design that could be applied to existing latent action based models such as UniVLA. Yet the analysis is almost entirely limited to unstructured LAM vs. SpatialLAM within the authors’ own BAKU policy pipeline. Prior methods appear only as numbers reported in their papers, with different policy architectures and pretraining. Although OTF-LAM and PoLAR are compared in the ablation using the authors’ policy design, there is no experiment applying SpatialLAM to an existing LAM design to show that it actually complements them. This matters because such evidence would support SpatialLAM not only as an improvement over the unstructured baseline or existing baselines, but also as a broader claim about latent action design.

  • Experimental Rigor. LIBERO-Plus is used as evidence of generalization performance, but the perturbation-wise performance relative to other baselines is inconsistent. The policy is much stronger under camera-view shifts and sensor noise, while being weaker under background texture changes. The unstructured baseline shares this property, so it most likely stems from the common policy pipeline rather than from SpatialLAM. These factors are under-analyzed, and while the authors state that they used only color jitter, it is not stated whether the baselines used comparable data augmentation. SpatialLAM might show much higher performance with proper improvements to the training pipeline. Also, the performance gain under camera view shifts can be explained by its spatial generalization capability, but the low performance under background texture changes raises the question of whether SpatialLAM can extract task-relevant information in OOD situations.

Questions for authors
  • Questions for Weaknesses. Could you explain (a) whether applying SpatialLAM to existing LAMs such as UniVLA, or even to other structured LAM baselines like OTF-LAM (e.g., Spatial OTF-LAM, if applicable), leads to meaningful improvements over those baselines, and (b) why the policy’s perturbation-wise performance differs from the baselines, being weakest under background texture changes? Is there evidence that explains this pattern (e.g., the rich semantics of DINOv3 ironically hindering generalization under different semantics, or the limited capacity of the policy’s ResNet-based vision branch)?

  • Does SpatialLAM itself act as a bottleneck?. In Table 12, SpatialLAM’s performance at 8x8 resolution matches that at 4x4, while the unstructured LAM degrades. Could SpatialLAM itself act as a semantic bottleneck, not by limiting its capacity but by restricting information flow through the spatial bias? Or would performance eventually drop at denser resolutions even for SpatialLAM, requiring an explicit bottleneck design?

5 comments

W1 & Q1(a). Integration with existing LAMs

Thank you for the thoughtful suggestion. To examine whether the benefits extend beyond BAKU, we integrate SpatialLAM into UniVLA and change only the latent action interface. The LAM keeps the native design of UniVLA, including its visual feature backbone, the two-stage training of language-conditioned task-irrelevant and task-centric latents, and VQ quantization. We only arrange the latent action queries as a spatial grid with locality-biased attention. As in UniVLA, the quantized codebook indices are added to the vocabulary of the Prismatic-7B policy as special tokens and predicted with cross-entropy, and the action decoder reads the latent action embeddings through the 3×3 convolutional projector. Following UniVLA's Bridge V2 policy-pretraining recipe, we pretrain for 20k steps (LR 2e-5) using latent-token cross-entropy, followed by LIBERO fine-tuning with LoRA (rank 32, α=16) for 30k steps (40k for LIBERO-Long), using latent-token cross-entropy and action L1 losses.

Table R13. UniVLA as the base pipeline. Success rate (%) on LIBERO-Long, LIBERO (average), and LIBERO-Plus (total).

MethodLongLIBERO Avg.LIBERO-Plus Total
UniVLA87.592.542.9
UniVLA + SpatialLAM89.693.648.6

SpatialLAM improves over the reported UniVLA results by 2.1 pp on LIBERO-Long, 1.1 pp on LIBERO average, and 5.7 pp on LIBERO-Plus, with the largest gains under camera-view (1.8 to 12.9) and object-layout (31.9 to 42.2) shifts. These results suggest that the benefits of spatial structure extend to a stronger VLA pipeline and are not specific to BAKU. We have added these results to Appendix C.5.

W2 & Q1(b). Perturbation-wise robustness on LIBERO-Plus

Thank you for the valuable question. The background-shift weakness is already present in BAKU without latent actions, which suggests that it is largely associated with the shared policy pipeline rather than introduced by SpatialLAM. We provide the no-latent control and a comparison with UniVLA:

Table R14. Selected LIBERO-Plus perturbation factors. Success rate (%) under the BAKU and UniVLA pipelines.

PipelineLatentBackgroundCameraRobot Init.Total
BAKUNo latent30.169.411.459.0
BAKUUnstructured39.580.916.164.6
BAKUSpatialLAM46.991.125.270.7
UniVLANative LAM81.01.846.242.9
UniVLASpatialLAM83.612.948.448.6

All BAKU variants share the same ResNet-18 policy encoder, LIBERO fine-tuning data, and color-jitter augmentation. DINOv3 provides features for LAM training but is not used as the policy's visual encoder at inference. The prior methods in Table 2 use different architectures and training recipes, so their perturbation profiles are not augmentation-controlled comparisons.

UniVLA performs much better under background shifts, suggesting that absolute per-factor performance depends substantially on the policy pipeline. This contrast is consistent with your second hypothesis. BAKU encodes observations using a ResNet-18 trained from scratch, whereas UniVLA employs large pretrained DINOv2 and SigLIP visual encoders. The same weakness in the no-latent baseline indicates that this pattern does not require DINOv3-derived latent targets. Nevertheless, SpatialLAM improves background-shift success within both pipelines: by 7.4 pp over the unstructured LAM and 16.8 pp over the no-latent baseline in BAKU, and by 2.6 pp in UniVLA, indicating that it retains useful task information under appearance shifts. Larger gains under camera-view (+10.2 pp in BAKU and +11.1 pp in UniVLA) and robot-initialization (+9.1 pp in BAKU) shifts are consistent with the benefits of spatial grounding.

Q2. Spatial structure and latent bottleneck

Thank you for this insightful question. To examine whether spatial structure can replace an explicit bottleneck, we extend the capacity ablation to a 16×16 latent grid.

Table R15. Latent capacity ablation extended to a 16×16 grid (Table 12). Success rate (%) on LIBERO-Long and the LIBERO-Plus subset.

GridTotal dim.Unstructured: LongUnstructured: PlusSpatialLAM: LongSpatialLAM: Plus
2×21663.442.469.459.2
4×46476.863.283.069.6
8×825671.657.182.670.1
16×161,02466.853.572.661.7

At 8×8, SpatialLAM maintains performance comparable to the 4×4 grid, whereas the unstructured LAM drops by 5.2/6.1 pp. At 16×16, SpatialLAM degrades by 10.0/8.4 pp relative to 8×8, while still outperforming the capacity-matched unstructured LAM (72.6/61.7 vs. 66.8/53.5).

Since ρ is measured in grid spacings (Equation 7, default ρ=1), the locality width in the image narrows from four patches at 4×4 to a single patch at 16×16. To separate this effect from the larger capacity, we further train the 16×16 SpatialLAM with ρ=4, which matches the four-patch locality width of the default 4×4 grid. This variant recovers 2.8/2.5 pp (75.4/64.2) but still trails the 4×4 grid by 7.6/5.4 pp. The degradation persists even when the spatial locality width is matched.

These results suggest that spatial structure mitigates degradation at larger latent capacities but does not eliminate sensitivity to bottleneck size. Spatial organization and a compact latent bottleneck therefore serve complementary roles. The spatial bias also restricts information flow at a fixed capacity. At the same KL weight, SpatialLAM has a 15.7% lower posterior rate than the unstructured LAM (279.7 vs. 331.7 nats) while achieving a lower next-frame feature MSE (Appendix C.6), consistent with your hypothesis of an implicit bottleneck. Notably, when the KL weight is tuned to match the posterior rate, SpatialLAM still outperforms the unstructured LAM by 5.6 to 6.4 pp on LIBERO-Long, so its advantage does not stem from the lower rate alone.

Dear reviewer, thank you again for your insightful comments on our paper, and we genuinely hope that our response could address your concerns. As the discussion is about to end, we are sincerely looking forward to your feedback. Please feel free to contact us if you have any further inquiries.

Reviewer_DpoJOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles a relevant design space question in the field of LAMs -- spatial organization of latent actions. The proposed approach is SpatialLAM, which combines structured latent representation of actions, a locality bias term in the attention and a convolutional projection layer to favor local features.

Critical strengths

The paper is well written, has a clear motivation and clearly describes the approach. The proposed solution is reasonably simple and easy to understand and implement, which is a big plus. The approach is also basically free of additional cost in terms of model size, training time and inference time, which is also valuable. I believe the direction has some practical value even though the experiments are in controlled (e.g., fixed camera only where anchored latents might be favorable) and relatively closed settings.

Critical weaknesses

The most critical weakness with this paper, as submitted, from my perspective is regarding the claims drawn from the experiments: while I am not questioning the validity of the experiments in general, I do not find them to clearly support the conclusions drawn. Particularly, it's not clear if the approach solely leads to faster convergence or is a general fix. Some evidence is for example in Figure 4 where the gap shrinks as spatial coverage grows, and Table 12 where more latent capacity leads to worse results (unstructured latent; relatedly, the induced structure could also be considered a bottleneck, so the experiments should explicitly account for that). This could be clarified by extending the training duration (e.g., to 80k and 160k iterations) of the unstructured LAM and SpatialLAM models and providing validation metrics throughout training. Further, there are many potential further confounders such as the policy's head capacity (a more structured latent might either carry less information or be generally easier to model, so insufficient head size by itself might explain the gains seen), which are not explored/accounted for sufficiently.

Questions for authors

Beyond the main critical weakness mentioned above, I'd also appreciate clarifications on the following points:

  • Is there a baseline missing in Tab. 1 etc of just BAKU with all the finetuning, but skipping the extra LAM step?
  • Encoder locality or decoder locality is not separated and would benefit from a more detailed analysis and an additional ablation (e.g., encoder locality only, decoder locality only, etc.). Additionally, I'm not convinced that the shuffled-grid control isolates persistence (if decoder keeps the anchors while the encoder is permuted).
  • The related work section is unusually short and misses discussion of important related works such as (Learning Latent Action World Models in the Wild, Garrido et al., ICML 2026) that report actions becoming localized without explicit structure enforcment.
4 comments

W1. Training duration, bottleneck, and policy-head capacity

Thank you for these constructive suggestions. To directly address the convergence and capacity concerns, we extend LAM training to 80k and 160k steps (up to 4× the default) and enlarge the downstream policy heads. SpatialLAM retains a 6.3 pp advantage on LIBERO-Long after extended training and a 5.6 pp advantage with larger heads. The larger setting increases the latent head from 2 to 4 layers and the action decoder from 3 to 6 blocks, widening both from 256 to 512.

Table R7. Extended LAM training and larger policy heads. Held-out next-frame feature MSE of each LAM and success rate (%) on LIBERO-Long.

LAM stepsDownstreamMSE ↓ (Unstructured)MSE ↓ (SpatialLAM)Success ↑ (Unstructured)Success ↑ (SpatialLAM)
40kDefault0.21100.207476.883.0
80kDefault0.21070.207277.183.1
160kDefault0.21060.207177.183.4
160kLarge head0.21060.207177.983.5

All LAMs use the same cosine learning-rate schedule (1e-4 → 1e-5) stretched to the full training length, and all policies use the same training budget (10 epochs of Bridge V2 pretraining and 80k steps of LIBERO-Long fine-tuning). Both LAMs are nearly converged by the default 40k steps, with held-out next-frame feature MSE differing by less than 0.25% between the 40k- and 160k-step LAMs. We also evaluate both policies at intermediate fine-tuning checkpoints:

Table R8. Intermediate policy fine-tuning checkpoints. Success rate (%) on LIBERO-Long.

Fine-tuning stepsUnstructured LAMSpatialLAMGap
40k75.681.6+6.0
60k76.882.6+5.8
80k76.883.0+6.2

Both policies nearly plateau, and the gap remains stable at 5.8 to 6.2 pp. We have added these validation metrics to Appendix C.7.

We would also like to clarify the observations you raised. In Figure 4, the gap naturally narrows as training covers more positions, since less cross-position generalization is required. It does not close with more data, remaining +3.0 pp at full coverage and +2.6 pp under the natural budget, where the number of demonstrations grows 16-fold from 1 to 16 training positions. In Table 12, SpatialLAM outperforms the unstructured LAM at every matched capacity, including the 8×8 grid. To account for the spatial structure itself acting as a bottleneck, we also match the posterior rate by tuning the KL weight. At three matched rates, SpatialLAM still outperforms the unstructured LAM by 5.6 to 6.4 pp on LIBERO-Long (Appendix C.6).

Together, these results indicate that SpatialLAM shows a general improvement rather than merely faster convergence, as its advantage holds after extended training, with larger policy heads, and at matched posterior rates.

Q1. No-latent BAKU baseline

Thank you for the suggestion. This baseline is the no-latent control in Table 4: the same BAKU policy and fine-tuning, without a LAM or latent objective. It follows the official BAKU training recipe and matches SpatialLAM in all other settings, except that it skips the latent action pretraining on Bridge V2 and is fine-tuned on LIBERO with the behavior-cloning loss alone. We have added it to Tables 1 and 2 under the same protocol:

Table R9. No-latent BAKU baseline, added to Table 1. Success rate (%) on LIBERO.

LIBEROSpatialObjectGoalLongAvg.
No latent (BAKU)93.499.494.872.490.0
Unstructured LAM96.699.897.476.892.7
SpatialLAM98.2100.097.483.094.7

Table R10. No-latent BAKU baseline, added to Table 2. Success rate (%) on LIBERO-Plus.

LIBERO-PlusBackgroundCameraLang.LightLayoutRobot Init.SensorTotal
No latent (BAKU)30.169.462.776.168.511.489.259.0
Unstructured LAM39.580.970.080.269.416.191.164.6
SpatialLAM46.991.170.488.174.225.294.970.7

Q2. Encoder/decoder locality and the shuffled-grid control

Thank you for the valuable suggestions. We apply the locality bias separately in the encoder and the decoder, keeping the grid, capacity, recipe, and 3×3 projector fixed:

Table R11. Ablation of encoder and decoder locality. Success rate (%) on LIBERO-Long and the LIBERO-Plus subset.

Encoder localityDecoder localityLongPlus
OffOff77.463.4
OnOff81.067.3
OffOn79.665.5
OnOn (SpatialLAM)83.069.6

The variant without locality bias in either module retains the spatial grid and the convolutional projector. Both encoder-only and decoder-only locality improve over the variant without locality, and using both gives the best results.

Regarding the shuffled-grid control, the locality bias is applied only at the permuted encoder anchors, and the decoder reads the latent in query order without locality bias, so the decoder does not retain fixed anchors. We have clarified this in Section 3.4 and Appendices B.2 and B.3. To isolate persistence from the decoder, we freeze the encoder-only-locality LAM from Table R11 and reorder only its latent tokens used as policy targets. The spatially aligned ordering keeps the native grid order, the fixed permutation applies one constant permutation to this order, and the transition-dependent permutation varies the permutation across transitions. The latent contents and the LAM decoder therefore remain unchanged, and all policies share the same architecture, 3×3 projector, and training budget:

Table R12. Reordering the policy targets of a frozen encoder-only-locality LAM. Success rate (%) on LIBERO-Long and the LIBERO-Plus subset.

Target orderingLongPlus
Transition-dependent permutation75.863.9
Fixed permutation78.865.8
Spatially aligned80.667.5

With the LAM fixed, a persistent ordering improves success by 3.0/1.9 pp over a transition-dependent one, and spatial alignment adds another 1.8/1.7 pp. The spatially aligned ordering closely matches the encoder-only result (81.0/67.3, Table R11), and the transition-dependent ordering falls to the level of the shuffled grid in Table 4 (76.0 on LIBERO-Long). These results support the benefits of persistent correspondence and spatial adjacency independently of the decoder. We have added both analyses to Appendix C.8.

Q3. More related work

Thank you for the suggestion. We have expanded the Related Work section with more discussion of latent actions for downstream policy pretraining and world modeling (e.g., CoLA-World, DreamDojo, and LAC-WM), spatially grounded manipulation methods (e.g., PerAct, RVT, and DINO-WM), and Garrido et al. (ICML 2026) as you suggested. The finding that latent actions can exhibit spatial locality without explicit structural constraints is consistent with our findings, as the unstructured LAM also shows local concentration (Table 11). Our work goes one step further by examining whether individual latent tokens maintain consistent spatial roles across transitions.

I thank the authors for the extensive reply and appreciate the effort for the extra evaluations. They have reasonably addressed my concerns, so I'm raising my recommendation to weak accept. I'd appreciate it if the authors could integrate the relevant conclusions (even if just by reference to the related section in the appendix) drawn from the new evaluations (incl. other relevant ones from the other replies) into the main paper more tightly for the final version.

Thank you very much for your positive feedback and for reconsidering your recommendation. We appreciate your valuable suggestion and will incorporate the key findings from the additional evaluations into the main paper in the final version. Thank you again for your time and constructive feedback.

Reviewer_vef8OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SpatialLAM keeps the standard inverse-forward latent action model and changes three interfaces. The bottleneck queries are arranged on a 4×4 grid aligned to the DINOv3 patch grid, a Gaussian penalty on the distance between a query anchor and a visual patch is added to the pre-softmax attention logit (Eq. 6-7), and downstream models read the grid with a 3×3 convolution rather than a per-token linear map. The comparison is against the same model with 1D positional encodings and no bias, matched on data, optimizer and latent capacity, with identical LAM parameter counts (Table 9). Authors evaluate on LIBERO and LIBERO-Plus over three full-pipeline seeds, on three real-world Flexiv tasks over 60 trials, and on action-conditioned world modelling for Bridge V2 and SSv2. The mechanistic claim rests on a component ladder (Table 4), two correspondence controls that vary only the query-to-anchor assignment, a cross-position study on translated LIBERO-Long (Figure 4), and intervention measurements of token-to-region correspondence (Table 11).

Critical strengths

The paper asks which property of the latent interface is doing the work, rather than only reporting that a new bottleneck scores higher, and it builds the controls that are needed to answer that question. In this literature, the standard move is to change the bottleneck and report benchmark deltas, and the properties that drive the change are typically left unseparated. Here, the authors instead vary one property of the representation at a time, while capacity, training recipe, readout and architecture are held fixed. I note that this design is what makes the ablations readable as evidence rather than as a sequence of configurations.

The shuffled-grid variant is the clean case. It keeps the locality bias and removes only stability of the token-to-region assignment across transitions, and LIBERO-Long falls from 81.8 to 76.0. Unstructured bottlenecks are already spatially local, 0.408 vs 0.181 in the Table 11, so locality was not the deficit.

What i found interesting is that what is missing is a binding that is the same from one transition to the next. I stress that this reorientation of the problem is worth knowing even if SpatialLAM is not the right fix for it. For that reason the recommendation here is a weak reject and not a reject.

Critical weaknesses

The title claims that latent actions need spatial structure, while the experiments support only that they need persistent structure. However, persistence is generic, and any stable token-to-region assignment delivers it, including one with no relation to image plane. Image-plane geometry is the specific and interesting claim, and the one that ties this paper to Transporter-style spatial equivariance.

To separate the two, the authors build exactly the right control - the fixed permutation that keeps a persistent anchor while scrambling its spatial arrangement. Its numbers are not reported. It is not a row in Table 4, only an unlabelled curve in Figure 4 that cannot be told apart from its neighbours, so the preference between the two arms is taken on the authors' word. This decides the paper, since by the authors' own ladder the 2D grid alone does nothing, the locality bias is a Gaussian attention prior of a known family, and the convolution is standard. In other words, what is left is an empirical claim about what organisation brings here, and I believe that half of it is unreported.

Note, that the demonstration also sits in a pipeline weaker than published latent action models in the authors' own Table 1, 83.0 on LIBERO-Long against UniVLA 87.5 and MVP-LAM 90.8. We therefore do not learn whether the effect survives at the scale where LAMs are used, and I recommend weak reject.

Questions for authors
  1. I suggest to report the fixed-permutation arm as a table row, on the same metrics and read through the same projector as SpatialLAM. This one row decides whether the claim is about spatial structure or only about persistent structure, and it is the answer that would move my score.

  2. What posterior rate does each arm actually achieve? Here, capacity is matched in dimension, while the locality bias changes what the posterior can express, so "the spatial bottleneck simply carries more about the next frame" is currently an untested explanation for every downstream gain.

  3. Does the effect survive on a strong latent action model? The three components add almost no parameters, so applying them to UniVLA or MVP-LAM should be cheap, and it would tell us, if this is a general property or only a pipeline-dependent repair.

4 comments

W1 & Q1. Fixed-permutation control

Thank you for the thoughtful comments. The fixed permutation applies a single random query-to-anchor permutation to all transitions, in both the encoder and the decoder. Downstream models read the latent through the same 3×3 projector as SpatialLAM, and all other settings are unchanged. This control preserves persistent token-to-region correspondence, but only SpatialLAM additionally aligns latent grid adjacency with image-plane adjacency. We now report its results in Table 4 under the same training and evaluation protocol:

Table R1. Fixed-permutation control, added to Table 4. Success rate (%) on LIBERO-Long and the LIBERO-Plus subset, and FVD on Bridge and SSv2.

MethodLong ↑Plus ↑Bridge FVD ↓SSv2 FVD ↓
Unstructured + conv. projector77.062.6829.2668.4
Fixed permutation79.466.5812.7591.4
SpatialLAM83.069.6767.7551.3

SpatialLAM outperforms the fixed permutation on all four metrics, improving success by 3.6/3.1 pp and reducing FVD by 45.0/40.1 on Bridge/SSv2. Since both variants preserve persistent correspondence, these results show an additional benefit from preserving image-plane adjacency, beyond persistence alone. Figure 4 also shows that the advantage is largest under sparse spatial coverage. We have added the fixed-permutation row to Table 4 and explicitly labeled the corresponding curves in Figure 4.

To further verify that this result does not depend on a specific permutation, we evaluate three independently drawn permutations, each with its own LAM and three policy seeds:

Table R2. Three independently drawn fixed permutations. Success rate (%) on LIBERO-Long.

MethodLong ↑
Permutation 179.8 ± 1.4
Permutation 278.6 ± 1.2
Permutation 380.0 ± 1.6
Mean over permutations79.5 ± 0.8

Permutation rows report the mean ± std over three policy seeds, and the mean row reports the std across permutations. The std across the three permutations is only 0.8 pp, and all three remain below SpatialLAM (83.0, Table R1). The benefit of spatial adjacency is therefore not specific to one permutation.

W2 & Q3. Integration with a stronger LAM

Thank you for the suggestion. Our BAKU pipeline uses a ResNet-18 policy trained without large-scale visual pretraining, whereas UniVLA builds on a 7B VLM policy, so absolute success rates are not directly comparable across pipelines. The controlled comparison is against the unstructured LAM in the same pipeline. To evaluate whether the benefits extend beyond our BAKU-based pipeline, we integrate SpatialLAM into UniVLA and change only the latent action interface. The LAM keeps the native design of UniVLA, including its visual feature backbone, the two-stage training of language-conditioned task-irrelevant and task-centric latents, and VQ quantization. We only arrange the latent action queries as a spatial grid with locality-biased attention. As in UniVLA, the quantized codebook indices are added to the vocabulary of the Prismatic-7B policy as special tokens and predicted with cross-entropy, and the action decoder reads the latent action embeddings through the 3×3 convolutional projector.

Following UniVLA's Bridge V2 policy-pretraining recipe, we pretrain the policy to predict latent action tokens using cross-entropy for 20k steps (LR 2e-5), followed by LIBERO fine-tuning with LoRA (rank 32, α=16) for 30k steps (40k for LIBERO-Long), using latent-token cross-entropy and action L1 losses.

Table R3. UniVLA as the base pipeline. Success rate (%) on the four LIBERO suites.

LIBEROSpatialObjectGoalLongAvg.
UniVLA95.295.491.987.592.5
UniVLA + SpatialLAM96.695.892.489.693.6

Table R4. UniVLA as the base pipeline. Success rate (%) on LIBERO-Plus.

LIBERO-PlusBackgroundCameraLang.LightLayoutRobot Init.SensorTotal
UniVLA81.01.869.669.031.946.221.242.9
UniVLA + SpatialLAM83.612.970.273.842.248.428.648.6

SpatialLAM improves over the reported UniVLA results across all four LIBERO suites (+2.1 pp on Long and +1.1 pp on average) and all seven LIBERO-Plus perturbation factors (+5.7 pp overall). These results suggest that the benefits of spatially structured latent actions extend to a stronger VLA pipeline and are not specific to BAKU. We have added these results to Appendix C.5.

Q2. Posterior rate analysis

Thank you for raising this concern. SpatialLAM achieves a lower posterior KL than the unstructured LAM and retains its policy advantage at closely matched posterior rates. We report the posterior KL to the prior (nats per transition, summed over 64 latent dimensions) and held-out next-frame feature MSE. All variants use the same convolutional projector. Under the same KL weight β, SpatialLAM achieves higher success with a 15.7% lower posterior KL than the unstructured LAM:

Table R5. Posterior rate and reconstruction at the default β. KL (nats), held-out next-frame feature MSE, and success rate (%) on LIBERO-Long.

LAMKLFeature MSE ↓Long ↑
Unstructured + conv. projector331.70.211077.0
Fixed permutation281.60.207679.4
SpatialLAM279.70.207483.0

All three LAMs use the default β = 2e-4, and KL and MSE are averaged over held-out validation transitions of the Open X-Embodiment training mixture. The fixed permutation achieves nearly identical KL and next-frame feature MSE to SpatialLAM, yet trails it by 3.6 pp in policy success.

To control for posterior rate directly, we further tune β to closely match the achieved KL across three target rates, using the same convolutional projector for both models:

Table R6. Rate-matched comparison. Achieved KL (nats) and success rate (%) on LIBERO-Long.

Target KLUnstructured LAM: KLUnstructured LAM: Success (%)SpatialLAM: KLSpatialLAM: Success (%)
290291.074.9289.281.3
305304.175.6306.081.5
320319.076.3321.081.9

The KL weight β is tuned separately for each model and target rate, and KL is measured on the same held-out transitions. SpatialLAM consistently outperforms the unstructured LAM by 5.6 to 6.4 pp at matched rates, so its advantage is not explained by a higher posterior rate. Moreover, lowering the rate of the unstructured LAM reduces its success (76.3 to 74.9), so a lower rate does not by itself improve the policy. We have added both analyses to Appendix C.6.

Dear reviewer, thank you again for your insightful comments on our paper, and we genuinely hope that our response could address your concerns. As the discussion is about to end, we are sincerely looking forward to your feedback. Please feel free to contact us if you have any further inquiries.

Dear Reviewer vef8,

Thank you again for your constructive feedback. We have included the fixed-permutation control in Table 4, as you suggested, along with additional experiments and analyses to address your concerns. As the discussion period is approaching its end, we would greatly appreciate any further thoughts you may have on our rebuttal.

Thank you for your time and consideration!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the area chair for managing the review process, and all the reviewers for their detailed and constructive comments. We appreciate the reviewers’ positive assessments of the paper’s clear motivation, simple and effective design, carefully controlled analyses, and extensive experiments. Based on the reviewers’ valuable comments, we have provided additional experiments and clarifications to address their concerns, including:

  • [Reviewer vef8]: we have added the fixed-permutation control to Table 4, which the reviewer identified as key to their assessment, and SpatialLAM outperforms it on all four metrics and across three independently drawn permutations. We have also provided a rate-matched posterior analysis and integrated SpatialLAM into UniVLA.
  • [Reviewer DpoJ]: we have extended LAM training and enlarged the policy heads, added the no-latent BAKU baseline, ablated encoder and decoder locality separately, clarified the shuffled-grid control, and expanded the related work.
  • [Reviewer CMCk]: we have integrated SpatialLAM into UniVLA, analyzed perturbation-wise robustness on LIBERO-Plus, and extended the latent capacity ablation to a 16×16 grid.

We are glad that Reviewers DpoJ and CMCk have confirmed that our responses addressed their concerns. We will incorporate the key findings from the additional analyses more tightly into the main paper in the final version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.