- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LMP, which formulates iterative, adaptive "reasoning" for robot control as variational inference with a variable-length autoregressive latent variable: an encoder (posterior) maps observation–action pairs to latent token sequences terminated by EOS, and a decoder reconstructs actions; the ELBO is optimized via latent-space RL (a PPO-style clipped surrogate with a stepwise KL decomposition), and a variance-decay schedule over latent length induces compression and adaptive allocation of compute. The framework instantiates both a policy (LMP-π) and a variable-length action tokenizer (LMP-tok). On real-robot DROID and simulated LIBERO/D3IL/RoboMimic, LMP-π outperforms Diffusion Policy, OpenVLA, VAE-policy and VQ-BeT baselines; LMP-tok outperforms VQ-VAE, FAST, and OAT tokenizers under the same downstream autoregressive policy. Analyses show variable-length latents and compression both contribute, that more latent steps express more behavioral modes, and that latent length negatively correlates with local action variance (e.g., gripper movements use fewer steps).
Critical strengths
- Clean and principled formulation: casting adaptive computation as variational inference over variable-length discrete latents is elegant, and the variance-decay compression schedule yields genuinely adaptive latent lengths with a clear mechanism (each additional step must "pay for itself" via sharper reconstruction).
- Strong and unusually fair empirical evaluation: real-robot plus three simulated benchmarks, two instantiations from one framework, and tokenizer comparisons that control for the downstream policy; ablations isolate the contributions of variable length, compression strength, and vocabulary size.
- Insightful mechanistic analysis: test-time truncation experiments, mode-visualization, and the latent-length vs. action-variance correlation clarify what the latent sequence actually encodes, going beyond headline numbers.
Critical weaknesses
- The "reasoning" framing overstates the mechanism: the paper itself shows that LMP does not perform iterative refinement — longer latents simply enumerate more behavioral modes. The memory-palace/reasoning analogy may mislead readers about what is being learned; the contribution is more accurately a variable-length latent VAE policy.
- The RL-based optimization is acknowledged to be fragile and hyperparameter-sensitive, but the paper gives no training-cost or stability comparison against the much simpler baselines (diffusion policy, VQ-BeT); scalability beyond 1B models and multi-task datasets remains unclear.
- Statistical and evaluation rigor is limited: DROID results are 20 trials without variance/CIs, LIBERO comparisons are marginal (0.933 vs 0.909 overall), the "bottom-10 tasks" framing is a post-hoc lens, and no significance testing is reported.
- The tokenizer evaluation uses a single downstream policy class and reports no latency/compression-rate analysis (average token length, decoding overhead), limiting the practical case for LMP-tok over FAST/OAT.
Questions for authors
- Since latent steps encode modes rather than iterative refinement, in what sense is this "reasoning"? Can you show a task where the latent sequence exhibits temporal/planning structure (e.g., plan-then-act) rather than mode enumeration, or otherwise clarify the analogy?
- What is the training cost (GPU-hours, buffer size, PPO hyperparameter sensitivity) of latent-space RL relative to the baselines, and are the DROID baselines tuned with a comparable budget? Can you report per-task variance/CIs?
- Would a simpler fixed-length VAE with a larger codebook recover most of the gap, i.e., how much of the benefit comes from variable length per se versus the latent formulation plus compression?
- For LMP-tok: what are the average sequence lengths and decoding latencies compared to FAST/OAT, and does the tokenizer remain effective on high-frequency or dexterous control (e.g., 30Hz+ tasks)?
3 comments
(W3, Q2) DROID no variance / CI. LIBERO gains are marginal.
We report the DROID results with confidence intervals as well as the per-trial outcomes.
DROID Results with Confidence Interval
The following table shows success rate over 20 trials, with 95% Wilson confidence intervals. Bold marks methods whose success rate lies within the 95% confidence interval of the best method. We further provide p-values from paired McNemar tests of LMP-π against each baseline. Against OpenVLA, LMP-π is significantly better (paired McNemar p < 0.05) on every measure except clean-table ≥1/3. Against DP, the per-task difference is significant on marker-mug (p = 0.031) and clean-table ≥2/3 (p = 0.008), borderline on peg-hole (p = 0.057) and clean-table 3/3 (p = 0.065), and not significant on block-bowl (p = 0.27). Pooling the paired trials across the four tasks, LMP-π succeeds where DP fails on 34 trials versus 7 the other way, which is statistically significant (p = 2.5 × 10^-5).
| block-bowl (zero-shot) | marker-mug (zero-shot) | peg-hole (finetuned) | clean-table ≥1/3 (finetuned) | clean-table ≥2/3 (finetuned) | clean-table 3/3 (finetuned) | |
|---|---|---|---|---|---|---|
| OpenVLA | 0.30 [0.15, 0.52] | 0.00 [0.00, 0.16] | 0.30 [0.15, 0.52] | 0.80 [0.58, 0.92] | 0.30 [0.15, 0.52] | 0.05 [0.01, 0.24] |
| DP | 0.40 [0.22, 0.61] | 0.25 [0.11, 0.47] | 0.30 [0.15, 0.52] | 0.80 [0.58, 0.92] | 0.55 [0.34, 0.74] | 0.25 [0.11, 0.47] |
| LMP-π | 0.65 [0.43, 0.82] | 0.55 [0.34, 0.74] | 0.70 [0.48, 0.85] | 0.95 [0.76, 0.99] | 0.95 [0.76, 0.99] | 0.60 [0.39, 0.78] |
| p vs. DP | 0.27 | 0.031 | 0.057 | 0.25 | 0.008 | 0.065 |
| p vs. OpenVLA | 0.039 | 0.001 | 0.021 | 0.38 | 0.0002 | 0.001 |
Per-trial DROID results
1 = success, 0 = failure. For clean-table, the entry is the number of objects cleared out of 3.
| Task | Method | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 | 12 | 13 | 14 | 15 | 16 | 17 | 18 | 19 | 20 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| block-bowl | OpenVLA | 1 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| DP | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | |
| LMP-π | 1 | 1 | 0 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 1 | 0 | 1 | 1 | |
| marker-mug | OpenVLA | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| DP | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 1 | 1 | 0 | |
| LMP-π | 0 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 0 | 1 | 1 | 1 | |
| peg-hole | OpenVLA | 0 | 0 | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | 0 | 0 |
| DP | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 | 1 | 0 | 0 | 0 | |
| LMP-π | 1 | 0 | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 1 | 0 | 1 | |
| clean-table | OpenVLA | 2/3 | 1/3 | 1/3 | 2/3 | 1/3 | 3/3 | 1/3 | 1/3 | 1/3 | 0/3 | 0/3 | 2/3 | 2/3 | 1/3 | 0/3 | 1/3 | 0/3 | 2/3 | 1/3 | 1/3 |
| DP | 3/3 | 3/3 | 1/3 | 1/3 | 0/3 | 1/3 | 1/3 | 1/3 | 0/3 | 2/3 | 0/3 | 2/3 | 2/3 | 3/3 | 2/3 | 3/3 | 3/3 | 0/3 | 2/3 | 2/3 | |
| LMP-π | 2/3 | 3/3 | 3/3 | 3/3 | 0/3 | 3/3 | 3/3 | 3/3 | 2/3 | 2/3 | 2/3 | 3/3 | 3/3 | 3/3 | 3/3 | 3/3 | 2/3 | 2/3 | 3/3 | 2/3 |
We have added the CI and per-trial results as Tables 6 and 7 in the revised paper.
We emphasize that while the aggregate LIBERO results may seem marginal, the individual task results in Figure 12 show that DP success rates taper off significantly on the low-success tasks. The bottom-10 result is intended to convey this in a tabular format.
(W2, Q2) No training-cost or stability comparison against baselines.
Training on DROID for 150k gradient steps took ~28h for LMP and ~21h for DP on 32xH100s. Training on single-task Robomimic Square for 100k gradient steps took ~6.5h for LMP and ~2.5h for DP on 4xA100s. On the tokenizer side, training on LIBERO-90 for 100k gradient steps took 2.5h for LMP-tok on 1xA100 and 2h for OAT on 1xA100. VQ-VAE took 0.5 h, and FAST took 1.5 min on CPU. LMP is generally slower than baselines because of the explicit latent-space rollout and buffer collection overhead.
Thank you for your response, I will keep my score.
We thank the reviewer for the positive review and for finding our formulation "clean and principled," our experiments "strong and unusually fair," and our analysis "insightful." We address the questions below.
We clarify that LMP does not follow the conventional interpretation of reasoning in a reinforcement learning setting, where more reasoning leads to higher return. We consider the behavior cloning setting, and here more reasoning corresponds to a better fit of the expert distribution. However, whether or not this directly translates to higher success depends on the property of the task. For example, in the DROID real-world tasks where recovery is important, more reasoning steps tend to improve performance (Figure 9 (b)). We additionally find that on a canonical multimodal benchmark, Push-T, LMP shows a strong correlation between the number of test-time reasoning steps and the coverage score. The mean coverage score and standard deviation over 3 seeds are shown in the following table, and we have added these results to Figure 9 (b) of the revised paper.
We train 1-step LMP with larger codebooks of size 16, 256, and 4096, matching the capacity of a vocabulary of 4 over 2, 4, and 6 steps, respectively. The following table shows mean success +- std over 3 seeds on robomimic square. We find that while vocab size 16 and 256 follow their autoregressive counterparts closely, vocab size of 4096 shows regression (compared to the closest neighbor from Figure 9.c, i.e. 4x8=65536). This suggests that the autoregressive latent space does more than provide capacity: it facilitates learning structured codes to populate the latents (e.g., because of shared prefixes).
We report the average token length, decoding latency, and total inference latency in the following table (evaluated on a single H200 with batch size 1). We find that, on average, LMP tokens are shorter than OAT tokens and decode faster. FAST produces the longest codes but decodes the fastest because it does not go through a learned decoder. Combining token generation and decoding, LMP and OAT have comparable overall latency. FAST's longer token sequences outweigh its faster decoding, resulting in the highest overall latency.