- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper replaces the one-step latent transition in LeWM with direct multi-step prediction conditioned on action prefixes. A causal Transformer encodes the current state and a candidate action sequence into prefix representations, and a predictor maps these representations and the same observed anchor state to future latents under dense multi-horizon supervision. Planning still uses CEM to search for action sequences; the main change is how candidate actions are evaluated by the dynamics model. Table 1 reports an increase in average success from 85.8% to 90.5%. Table 2 reports a reduction in cumulative dynamics time from 19.7 s to 4.1 s and cumulative CEM solve time from 33.7 s to 16.1 s. The efficiency gains are practically useful, but the specific methodological contribution and the evidence for improved long-horizon prediction require further clarification.
Critical strengths
The strongest result is the simultaneous improvement in efficiency and task success. Under the reported matched evaluation settings, the dynamics module is approximately 4.8 times faster, cumulative full-CEM solve time decreases by 52.2%, and average success in the main table increases by 4.7 percentage points. This demonstrates a practical planning benefit from reducing sequential model calls, beyond architectural parallelism alone.
Critical weaknesses
- The contribution of the key design choices is not sufficiently isolated. The method changes the transition interface, action encoding, and supervision density together. The ablations in Table 4 are useful, but the long-action baseline with terminal-only supervision does not replace a direct multi-step baseline with matched supervision. Moreover, adding dense supervision to LeWM reduces average success from 85.8% to 79.0%, which warrants an explanation of whether the training objective and optimization settings are suitable for that variant. The current evidence does not adequately distinguish the benefits of fewer autoregressive calls, multi-horizon supervision, and the specific causal prefix encoder. A simpler action-sequence encoder with direct multi-step prediction, matched for state conditioning, data, model size, and training budget, would help establish the methodological contribution.
- Cross-model latent MSE does not yet establish more accurate dynamics prediction. Section 4.2 and Figure 3 compare errors in each model's own learned latent space, where representation scale and geometry can affect the numerical values. Both models use SIGReg, which helps constrain representations, but the comparability of their error scales still needs verification. The physical-property probes in Table 3 mainly measure information recoverable from encoded real observations; they do not directly establish the accuracy of predicted future states. Please evaluate predicted latents in common physical units, such as future position or angle errors, or support the comparison through a shared encoder, appropriate normalization, and representation statistics.
- The scope of the long-horizon error advantage needs stronger evidence. Within a prediction window, all horizons share an observed anchor, avoiding recursive feedback of intermediate predicted states. Nevertheless, direct prediction can become harder as action prefixes grow longer. When a predicted terminal state anchors the next window, recursive error propagation reappears across windows. In the 50-environment-step experiment, corresponding to 10 latent transitions, Fast-LeWM chains two five-step windows. It therefore reduces the number of recursive transitions without eliminating cross-window propagation. The paper already qualifies the within-window mechanism, so this is not itself a methodological error. The missing evidence is a systematic analysis of different window lengths, longer chains of windows, and errors at window boundaries, which would clarify how far the reported advantage extends.
Questions for authors
- How does a simpler direct multi-step predictor perform when state conditioning, multi-horizon supervision, parameter count, and optimization budget are matched to Fast-LeWM? Can you also explain the performance drop when dense supervision is added to LeWM in Table 4?
- Can you report physical-space errors for predicted future latents, or provide cross-model representation-scale and variance statistics to support the comparison in Figure 3?
- How do prediction error and task success change with window length and the number of chained windows? Please distinguish within-window errors from errors after re-anchoring on a predicted state, and clarify whether each replanning cycle re-anchors on a real observation.
- Do the seven seeds in Appendix A vary model training, task initialization, planning samples, or a combination of these? Please provide paired results or confidence intervals. Also clarify whether Table 2 uses terminal-only or all-horizon prediction, and how many planning calls contribute to its cumulative timings. Per-call latency would make these results easier to interpret.
6 comments
Response to Reviewer 4CsG (2/4)
Q2: Physical-space prediction errors and latent representation scales.
We thank the reviewer for raising this important concern. We agree that latent prediction errors measured in independently learned representation spaces may not be directly comparable. To address this issue, we conducted additional evaluations in physical coordinates and examined the empirical scales of both latent representations.
Physical-space prediction errors. We trained physical-state probes for both models to map predicted future latents to common physical coordinates: agent XY (Two-Room), block XY (PushT), cube XYZ (Cube), and finger XY (Reacher). The table reports physical-space RMSE, with each entry showing Fast-LeWM/LeWM (lower errors in bold).
Each prediction step corresponds to five raw environment steps. Fast-LeWM directly predicts horizons within a single prediction window, while longer horizons require chaining.
| Env. steps | Two-Room (px) | PushT (px) | Cube (mm) | Reacher (mm) |
|---|---|---|---|---|
| Direct prediction () | ||||
| 5 | 1.3/5.8 | 2.0/1.9 | 1.9/1.9 | 1.6/3.4 |
| 10 | 1.0/5.8 | 1.9/2.5 | 1.6/1.9 | 2.1/4.0 |
| 15 | 1.0/5.3 | 1.7/2.8 | 1.4/2.2 | 2.0/4.5 |
| 20 | 1.2/5.0 | 1.8/3.0 | 1.3/2.5 | 3.2/5.3 |
| 25 | 1.3/4.3 | 1.7/3.3 | 1.8/4.5 | 2.8/6.3 |
| Chained prediction () | ||||
| 30 | 1.6/4.6 | 2.3/5.1 | 2.4/5.6 | 3.5/6.9 |
| 35 | 2.0/4.6 | 3.5/7.5 | 2.9/7.2 | 6.9/7.3 |
| 40 | 1.8/4.2 | 3.9/8.9 | 3.5/8.4 | 7.4/7.9 |
| 45 | 1.7/4.7 | 5.4/11.2 | 4.3/8.5 | 7.4/8.4 |
| 50 | 1.6/4.7 | 6.2/13.3 | 5.7/8.8 | 7.4/8.6 |
Fast-LeWM achieves lower physical-space prediction errors, including both direct and chained prediction horizons. These results provide evidence that the advantage observed in Figure 3 extends to task-relevant physical quantities.
Latent representation scales. Both models employ SIGReg regularization to encourage a standard normal distribution of the learned latents. To assess whether differences in latent MSE could arise from representation-scale discrepancies, we computed the variance of the encoded latent representations.
| Task | LeWM | Fast-LeWM |
|---|---|---|
| PushT | 1.0088 | 1.0101 |
| Two-Room | 0.9844 | 1.0182 |
| Reacher | 0.9894 | 0.9741 |
| Cube | 1.0146 | 1.0605 |
The variances are close to one for both models across all four environments, suggesting that the differences in Figure 3 are unlikely to be explained merely by differences in latent scale.
Variance-normalized latent prediction errors. We further normalize each model's prediction MSE by its own latent variance. The following table reports normalized errors at the first prediction step, averaged over the direct prediction window (), and averaged over subsequent chained predictions ().
| Task | Model | Avg. | Avg. | |
|---|---|---|---|---|
| PushT | LeWM | 0.0075 | 0.0316 | 0.1298 |
| Fast-LeWM | 0.0038 | 0.0049 | 0.0179 | |
| Two-Room | LeWM | 0.0615 | 0.0926 | 0.2103 |
| Fast-LeWM | 0.0054 | 0.0179 | 0.0538 | |
| Reacher | LeWM | 0.0833 | 0.1199 | 0.1385 |
| Fast-LeWM | 0.0373 | 0.0414 | 0.0633 | |
| Cube | LeWM | 0.0044 | 0.0192 | 0.0717 |
| Fast-LeWM | 0.0009 | 0.0012 | 0.0028 |
Fast-LeWM retains lower normalized latent prediction errors across all four environments, both within the direct prediction window and during subsequent chained rollouts. This confirms that the advantage in Figure 3 persists after accounting for overall representation scale.
We have incorporated these results into the revised manuscript.
Response to Reviewer 4CsG (3/4)
Q3. Window length and re-anchoring.
We thank the reviewer for the important question.
We first clarify that multi-window chaining is used only for a more comprehensive evaluation of long-horizon open-loop prediction accuracy, rather than in actual planning. In our receding-horizon planning protocol, each CEM cycle evaluates candidate action sequences within a single prediction window (). After executing the planned actions, the next cycle starts from a newly observed environment state, rather than the previously predicted terminal latent.
Nevertheless, we agree that when multiple prediction windows are chained in open-loop evaluation, error propagation can reappear because subsequent windows use predicted terminal states as their anchors. To examine this effect, we provide additional analyses of open-loop prediction errors and planning performance under different window decompositions.
Open-loop prediction across chained windows. We evaluate two consecutive prediction windows, as reported in the physical-space error table in Q2. Predictions at environment steps 5–25 share the initial observed anchor. The predicted terminal state at step 25 subsequently serves as the anchor for predictions at steps 30–50. No intermediate real observations are used in this evaluation.
Fast-LeWM achieves lower physical-space RMSE on all four tasks at every evaluated horizon from steps 10 to 50, including after the first predicted-state re-anchoring. These results indicate that Fast-LeWM's prediction advantage can persist across chained windows.
Closed-loop planning performance under different window decompositions. To systematically examine the effects of window length and recursive predicted-state feedback, we additionally evaluate all 16 ordered partitions of the fixed five-action-block planning horizon across seven random seeds.
Each partition determines how candidate actions are divided into sequential prediction windows. For example, [2,3] first predicts two steps from the observed anchor and then predicts three additional steps using the intermediate predicted latent as the new anchor. The candidate goal cost is computed from the resulting terminal prediction, without any self-consistency penalty.
All variants share the same model weights, total planning horizon, action space, and CEM optimization budget. Results are planning success rates (%), reported as mean sample standard deviation across seven seeds.
| Partition | PushT | TwoRoom | Reacher | Cube | Avg. |
|---|---|---|---|---|---|
| [5] | 91.7±4.1 | 95.4±3.2 | 84.0±5.9 | 71.7±8.4 | 85.7 |
| [1,4] | 90.0±5.4 | 93.7±5.5 | 77.1±6.3 | 70.6±6.6 | 82.9 |
| [2,3] | 91.4±3.6 | 93.1±5.3 | 78.6±5.7 | 70.3±7.4 | 83.4 |
| [3,2] | 90.9±5.8 | 93.1±4.3 | 77.7±5.0 | 70.9±9.2 | 83.2 |
| [4,1] | 90.6±4.6 | 93.1±5.3 | 79.1±5.3 | 69.4±7.1 | 83.1 |
| [1,1,3] | 91.4±4.4 | 93.1±3.8 | 75.4±5.3 | 67.7±7.6 | 81.9 |
| [1,2,2] | 91.7±4.7 | 92.9±5.1 | 72.3±7.0 | 65.7±8.4 | 80.7 |
| [1,3,1] | 91.7±4.1 | 92.6±4.9 | 73.7±9.1 | 66.0±5.3 | 81.0 |
| [2,1,2] | 90.6±5.0 | 92.3±3.9 | 76.3±5.0 | 69.7±9.5 | 82.2 |
| [2,2,1] | 91.4±4.7 | 92.3±4.4 | 77.1±8.6 | 67.4±6.3 | 82.1 |
| [3,1,1] | 90.9±3.4 | 91.7±5.8 | 73.7±7.3 | 67.4±6.3 | 80.9 |
| [1,1,1,2] | 91.1±5.3 | 92.6±5.0 | 69.4±8.0 | 68.0±6.1 | 80.3 |
| [1,1,2,1] | 90.3±7.2 | 92.0±4.9 | 73.4±4.3 | 65.4±7.2 | 80.3 |
| [1,2,1,1] | 91.1±4.5 | 92.0±5.5 | 75.4±9.8 | 67.1±8.0 | 81.4 |
| [2,1,1,1] | 88.9±5.3 | 91.7±4.8 | 73.7±5.2 | 65.7±8.6 | 80.0 |
| [1,1,1,1,1] | 89.7±3.9 | 92.3±5.7 | 71.1±6.5 | 65.7±6.4 | 79.7 |
The single-window configuration [5] achieves the highest average success (85.7) among all 16 partitions. Overall, configurations involving fewer chained windows tend to perform better, supporting the benefit of reducing recursive feedback from predicted intermediate states.
We have included this analysis in the revised manuscript.
Response to Reviewer 4CsG (1/4)
Q1. Comparison with direct multi-step baseline and LeWM with dense supervision.
We thank the reviewer for these questions.
Direct Multi-Step LeWM. As suggested, we constructed a minimally modified Direct Multi-Step LeWM baseline with state-conditioned action encoding and direct prediction of future latents. We matched multi-horizon supervision, training data, model size, hyperparameters, and training budget. Results are planning success rates (%), reported as mean ± sample standard deviation across seven seeds.
Fast-LeWM improved average planning success by 5.6 percentage points. These results support the benefit of our action-prefix formulation beyond a straightforward direct multi-step extension under matched training and model-capacity settings.
LeWM with Dense Loss. This variant retains LeWM's original architecture and training hyperparameters. The only modification is to perform additional autoregressive forward passes during training and apply a prediction loss at each rollout step. All horizon-specific losses backpropagate through the full unrolled computation graph.
For a more comprehensive evaluation, we additionally evaluated both LeWM and LeWM w/ Dense Loss across seven random seeds:
These results show that dense supervision does not consistently improve performance of LeWM. Unlike Fast-LeWM, which directly supervises each horizon from the same observed latent state, autoregressive LeWM propagates later-horizon losses through intermediate predicted states. We hypothesize this difference in optimization structure may contribute to the performance of dense supervision.
We have included these results in the revised paper.