acceptodds
Under review as a conference paper at ICLR 2027

Fast LeWorldModel

Abstract

Joint-Embedding Predictive Architectures (JEPAs), including recent LeWorldModel (LeWM), have become a promising foundation for reconstruction-free visual world models. For visual planning, however, LeWM evaluates candidate action sequences by repeatedly applying a local one-step latent transition model. This autoregressive rollout makes planning computationally expensive and causes accumulated latent prediction errors as the horizon grows. We propose We propose Fast LeWorld-Model (Fast-LeWM), a fast latent world model that replaces repeated local rollout with action-prefix prediction. Given the current latent state and a candidate action sequence, Fast-LeWM encodes its prefixes and predicts the future latents reached after executing those prefixes in parallel. By making action prefixes the basic prediction unit, Fast-LeWM directly models action effects accumulated over different horizons. This prefix-level supervision forces Training these multi-horizon predictions jointly encourages the model to learn how states continuously evolve under different action prefixes, rather than only fitting one-step state transitions. During planning, the predictor uses the prefix from the encoded action sequence to evaluate the corresponding future latent directly, without explicitly rolling through each intermediate imagined state. Across multiple tasks, Fast-LeWM improves average success over LeWM while substantially reducing planning time. It also achieves lower open-loop latent prediction loss, with errors growing considerably more slowly as the rollout horizon increases.

Primary area:
applications to robotics, autonomy, planning
Keywords:
latent world model, action prefix prediction

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4CsGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper replaces the one-step latent transition in LeWM with direct multi-step prediction conditioned on action prefixes. A causal Transformer encodes the current state and a candidate action sequence into prefix representations, and a predictor maps these representations and the same observed anchor state to future latents under dense multi-horizon supervision. Planning still uses CEM to search for action sequences; the main change is how candidate actions are evaluated by the dynamics model. Table 1 reports an increase in average success from 85.8% to 90.5%. Table 2 reports a reduction in cumulative dynamics time from 19.7 s to 4.1 s and cumulative CEM solve time from 33.7 s to 16.1 s. The efficiency gains are practically useful, but the specific methodological contribution and the evidence for improved long-horizon prediction require further clarification.

Critical strengths

The strongest result is the simultaneous improvement in efficiency and task success. Under the reported matched evaluation settings, the dynamics module is approximately 4.8 times faster, cumulative full-CEM solve time decreases by 52.2%, and average success in the main table increases by 4.7 percentage points. This demonstrates a practical planning benefit from reducing sequential model calls, beyond architectural parallelism alone.

Critical weaknesses
  1. The contribution of the key design choices is not sufficiently isolated. The method changes the transition interface, action encoding, and supervision density together. The ablations in Table 4 are useful, but the long-action baseline with terminal-only supervision does not replace a direct multi-step baseline with matched supervision. Moreover, adding dense supervision to LeWM reduces average success from 85.8% to 79.0%, which warrants an explanation of whether the training objective and optimization settings are suitable for that variant. The current evidence does not adequately distinguish the benefits of fewer autoregressive calls, multi-horizon supervision, and the specific causal prefix encoder. A simpler action-sequence encoder with direct multi-step prediction, matched for state conditioning, data, model size, and training budget, would help establish the methodological contribution.
  2. Cross-model latent MSE does not yet establish more accurate dynamics prediction. Section 4.2 and Figure 3 compare errors in each model's own learned latent space, where representation scale and geometry can affect the numerical values. Both models use SIGReg, which helps constrain representations, but the comparability of their error scales still needs verification. The physical-property probes in Table 3 mainly measure information recoverable from encoded real observations; they do not directly establish the accuracy of predicted future states. Please evaluate predicted latents in common physical units, such as future position or angle errors, or support the comparison through a shared encoder, appropriate normalization, and representation statistics.
  3. The scope of the long-horizon error advantage needs stronger evidence. Within a prediction window, all horizons share an observed anchor, avoiding recursive feedback of intermediate predicted states. Nevertheless, direct prediction can become harder as action prefixes grow longer. When a predicted terminal state anchors the next window, recursive error propagation reappears across windows. In the 50-environment-step experiment, corresponding to 10 latent transitions, Fast-LeWM chains two five-step windows. It therefore reduces the number of recursive transitions without eliminating cross-window propagation. The paper already qualifies the within-window mechanism, so this is not itself a methodological error. The missing evidence is a systematic analysis of different window lengths, longer chains of windows, and errors at window boundaries, which would clarify how far the reported advantage extends.
Questions for authors
  1. How does a simpler direct multi-step predictor perform when state conditioning, multi-horizon supervision, parameter count, and optimization budget are matched to Fast-LeWM? Can you also explain the performance drop when dense supervision is added to LeWM in Table 4?
  2. Can you report physical-space errors for predicted future latents, or provide cross-model representation-scale and variance statistics to support the comparison in Figure 3?
  3. How do prediction error and task success change with window length and the number of chained windows? Please distinguish within-window errors from errors after re-anchoring on a predicted state, and clarify whether each replanning cycle re-anchors on a real observation.
  4. Do the seven seeds in Appendix A vary model training, task initialization, planning samples, or a combination of these? Please provide paired results or confidence intervals. Also clarify whether Table 2 uses terminal-only or all-horizon prediction, and how many planning calls contribute to its cumulative timings. Per-call latency would make these results easier to interpret.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 4CsG (1/4)

Q1. Comparison with direct multi-step baseline and LeWM with dense supervision.

We thank the reviewer for these questions.

Direct Multi-Step LeWM. As suggested, we constructed a minimally modified Direct Multi-Step LeWM baseline with state-conditioned action encoding and direct prediction of future latents. We matched multi-horizon supervision, training data, model size, hyperparameters, and training budget. Results are planning success rates (%), reported as mean ±\pm sample standard deviation across seven seeds.

TaskDirect Multi-Step LeWMFast-LeWM
PushT87.1 ±\pm 4.391.7 ±\pm 4.1
Two-Room83.1 ±\pm 2.695.4 ±\pm 3.2
Reacher86.8 ±\pm 2.684.0 ±\pm 5.9
Cube63.4 ±\pm 6.671.7 ±\pm 8.4
Average80.185.7

Fast-LeWM improved average planning success by 5.6 percentage points. These results support the benefit of our action-prefix formulation beyond a straightforward direct multi-step extension under matched training and model-capacity settings.

LeWM with Dense Loss. This variant retains LeWM's original architecture and training hyperparameters. The only modification is to perform additional autoregressive forward passes during training and apply a prediction loss at each rollout step. All horizon-specific losses backpropagate through the full unrolled computation graph.

For a more comprehensive evaluation, we additionally evaluated both LeWM and LeWM w/ Dense Loss across seven random seeds:

TaskLeWMLeWM w/ Dense Loss
PushT87.1±7.287.5±4.0
Two-Room87.0±4.991.8±4.0
Reacher83.4±5.781.8±4.5
Cube67.4±8.662.8±5.6
Average81.281.0

These results show that dense supervision does not consistently improve performance of LeWM. Unlike Fast-LeWM, which directly supervises each horizon from the same observed latent state, autoregressive LeWM propagates later-horizon losses through intermediate predicted states. We hypothesize this difference in optimization structure may contribute to the performance of dense supervision.

We have included these results in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 4CsG (2/4)

Q2: Physical-space prediction errors and latent representation scales.

We thank the reviewer for raising this important concern. We agree that latent prediction errors measured in independently learned representation spaces may not be directly comparable. To address this issue, we conducted additional evaluations in physical coordinates and examined the empirical scales of both latent representations.

Physical-space prediction errors. We trained physical-state probes for both models to map predicted future latents to common physical coordinates: agent XY (Two-Room), block XY (PushT), cube XYZ (Cube), and finger XY (Reacher). The table reports physical-space RMSE, with each entry showing Fast-LeWM/LeWM (lower errors in bold).

Each prediction step corresponds to five raw environment steps. Fast-LeWM directly predicts horizons k=1,…,5k=1,\ldots,5 within a single prediction window, while longer horizons require chaining.

Env. stepsTwo-Room (px)PushT (px)Cube (mm)Reacher (mm)
Direct prediction (k≤Hk\leq H)
51.3/5.82.0/1.91.9/1.91.6/3.4
101.0/5.81.9/2.51.6/1.92.1/4.0
151.0/5.31.7/2.81.4/2.22.0/4.5
201.2/5.01.8/3.01.3/2.53.2/5.3
251.3/4.31.7/3.31.8/4.52.8/6.3
Chained prediction (k>Hk>H)
301.6/4.62.3/5.12.4/5.63.5/6.9
352.0/4.63.5/7.52.9/7.26.9/7.3
401.8/4.23.9/8.93.5/8.47.4/7.9
451.7/4.75.4/11.24.3/8.57.4/8.4
501.6/4.76.2/13.35.7/8.87.4/8.6

Fast-LeWM achieves lower physical-space prediction errors, including both direct and chained prediction horizons. These results provide evidence that the advantage observed in Figure 3 extends to task-relevant physical quantities.

Latent representation scales. Both models employ SIGReg regularization to encourage a standard normal distribution of the learned latents. To assess whether differences in latent MSE could arise from representation-scale discrepancies, we computed the variance of the encoded latent representations.

TaskLeWMFast-LeWM
PushT1.00881.0101
Two-Room0.98441.0182
Reacher0.98940.9741
Cube1.01461.0605

The variances are close to one for both models across all four environments, suggesting that the differences in Figure 3 are unlikely to be explained merely by differences in latent scale.

Variance-normalized latent prediction errors. We further normalize each model's prediction MSE by its own latent variance. The following table reports normalized errors at the first prediction step, averaged over the direct prediction window (k=1,…,5k=1,\ldots,5), and averaged over subsequent chained predictions (k=6,…,10k=6,\ldots,10).

TaskModelk=1k=1Avg. k=1 ⁣: ⁣5k=1\!:\!5Avg. k=6 ⁣: ⁣10k=6\!:\!10
PushTLeWM0.00750.03160.1298
Fast-LeWM0.00380.00490.0179
Two-RoomLeWM0.06150.09260.2103
Fast-LeWM0.00540.01790.0538
ReacherLeWM0.08330.11990.1385
Fast-LeWM0.03730.04140.0633
CubeLeWM0.00440.01920.0717
Fast-LeWM0.00090.00120.0028

Fast-LeWM retains lower normalized latent prediction errors across all four environments, both within the direct prediction window and during subsequent chained rollouts. This confirms that the advantage in Figure 3 persists after accounting for overall representation scale.

We have incorporated these results into the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 4CsG (3/4)

Q3. Window length and re-anchoring.

We thank the reviewer for the important question.

We first clarify that multi-window chaining is used only for a more comprehensive evaluation of long-horizon open-loop prediction accuracy, rather than in actual planning. In our receding-horizon planning protocol, each CEM cycle evaluates candidate action sequences within a single prediction window (H=5H=5). After executing the planned actions, the next cycle starts from a newly observed environment state, rather than the previously predicted terminal latent.

Nevertheless, we agree that when multiple prediction windows are chained in open-loop evaluation, error propagation can reappear because subsequent windows use predicted terminal states as their anchors. To examine this effect, we provide additional analyses of open-loop prediction errors and planning performance under different window decompositions.

Open-loop prediction across chained windows. We evaluate two consecutive prediction windows, as reported in the physical-space error table in Q2. Predictions at environment steps 5–25 share the initial observed anchor. The predicted terminal state at step 25 subsequently serves as the anchor for predictions at steps 30–50. No intermediate real observations are used in this evaluation.

Fast-LeWM achieves lower physical-space RMSE on all four tasks at every evaluated horizon from steps 10 to 50, including after the first predicted-state re-anchoring. These results indicate that Fast-LeWM's prediction advantage can persist across chained windows.

Closed-loop planning performance under different window decompositions. To systematically examine the effects of window length and recursive predicted-state feedback, we additionally evaluate all 16 ordered partitions of the fixed five-action-block planning horizon across seven random seeds.

Each partition determines how candidate actions are divided into sequential prediction windows. For example, [2,3] first predicts two steps from the observed anchor and then predicts three additional steps using the intermediate predicted latent as the new anchor. The candidate goal cost is computed from the resulting terminal prediction, without any self-consistency penalty.

All variants share the same model weights, total planning horizon, action space, and CEM optimization budget. Results are planning success rates (%), reported as mean ±\pm sample standard deviation across seven seeds.

PartitionPushTTwoRoomReacherCubeAvg.
[5]91.7±4.195.4±3.284.0±5.971.7±8.485.7
[1,4]90.0±5.493.7±5.577.1±6.370.6±6.682.9
[2,3]91.4±3.693.1±5.378.6±5.770.3±7.483.4
[3,2]90.9±5.893.1±4.377.7±5.070.9±9.283.2
[4,1]90.6±4.693.1±5.379.1±5.369.4±7.183.1
[1,1,3]91.4±4.493.1±3.875.4±5.367.7±7.681.9
[1,2,2]91.7±4.792.9±5.172.3±7.065.7±8.480.7
[1,3,1]91.7±4.192.6±4.973.7±9.166.0±5.381.0
[2,1,2]90.6±5.092.3±3.976.3±5.069.7±9.582.2
[2,2,1]91.4±4.792.3±4.477.1±8.667.4±6.382.1
[3,1,1]90.9±3.491.7±5.873.7±7.367.4±6.380.9
[1,1,1,2]91.1±5.392.6±5.069.4±8.068.0±6.180.3
[1,1,2,1]90.3±7.292.0±4.973.4±4.365.4±7.280.3
[1,2,1,1]91.1±4.592.0±5.575.4±9.867.1±8.081.4
[2,1,1,1]88.9±5.391.7±4.873.7±5.265.7±8.680.0
[1,1,1,1,1]89.7±3.992.3±5.771.1±6.565.7±6.479.7

The single-window configuration [5] achieves the highest average success (85.7) among all 16 partitions. Overall, configurations involving fewer chained windows tend to perform better, supporting the benefit of reducing recursive feedback from predicted intermediate states.

We have included this analysis in the revised manuscript.

Reviewer_YRkbOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Fast-LeWM, a latent world model built using LeWorldModel (LeWM) as a backbone for faster latent planning via multi-horizon prediction. Standard latent world model such as LeWM rolls a single step predictor forward autoregressively to score CEM candidates during planning. Instead, Fast-LeWM first aggregates each action prefix at:t+k−1a_{t:t+k-1} of a given trajectory of k steps into a single conditioning token, and then predicts the latent zt+kz_{t+k} reached after that prefix directly from the observed anchor ztz_t. For this purpose, the proposed method train a causal transformer to aggregate an initial state + a sequence of action [st;at,…,at+H−1][s_t; a_t,\dots,a_{t+H-1}] (with sts_t an MLP projection of ztz_t) into a single token pt,kp_{t,k} per prefix length kk. Training uses a dense loss averaged over all prefixes plus SIGReg, trained jointly similarly to the original LeWM framework. Fast-LeWM also introduces a self-consistency penalty during test-time by decomposing the single long-horizon prediction into two steps which is possible due to the multi-horizon prediction. The proposed method is compared on the same suite of benchmarks (Push T, Two Room, Cube, Reacher) as LeWM with both a performance and efficiency comparison against the original model. On the four tasks with H=5H=5, Fast-LeWM reports higher CEM success overall (85.8 to 90.5% with a single seed and 81.2 to 85.7% over seven seeds), as well as 4.8x faster dynamics evaluation and 2.1x faster CEM solve. The author ablate the state token, the dense loss and the interface.

Critical strengths
  1. Multi-horizon latent prediction in an end-to-end JEPA world model that enables faster CEM solve with matched or slightly improved planning performance. This tackles one direct limitation of planning with current JEPA world model which is the cost of doing search by single-step rollouts. The proposed solution is simple and effective. A sequence of actions is aggregated, conditioned on the current state, into a single token per prefix, and the model is trained jointly with the encoder to predict the latent at multiple horizons. All horizons are then queries in one parallel pass, greatly reducing the dynamics evaluation from 19.7s to 4.1s and the CEM solve time from 33.7s to 16.1s at comparable parameter count.
Critical weaknesses
  1. The main claim of avoiding recursive error propagation is not fully supported by the experiment shown in the paper. First, the action prefix encoder is a causal Transformer over (MLP(zt),at:t+k−1)(MLP(z_t), a_{t:t+k-1}), so actions are still composed but inside the prefix encoder's latent directly. Also the state token ztz_t enters twice in the predictor forward and other than a few within-noise performance boost in single-seed, adding the state token to the action sequence is not a design choice that is well supported, as now the conditioning is not just on an aggregated action sequence anymore but on a transition outcomes. Also Fig 3. compares MSE across two different latent spaces. The gap already exists at the first horizon, before any compounding is possible. Fast-LeWM's error also seems to jump once the chaining starts, so mitigating error propagation is not clearly supported. What is supported is less forward with comparable success, not directly the fact that aggregating long action sequences mitigates error accumulation.

  2. The contribution is framed around speed (title included) and only evaluated against single-step baseline. The scientific contribution behind the proposed method is the multi-horizon prediction, not the speed, and leading with "fast" understates what actually needs to be validated by this method. The speedup is only a direct byproduct of the multi-horizon interface. Doing a single forward instead of several is always faster at equal parameter count, so the critical consideration should be comparing the method of doing that single forward (here through action prefixes) to other approaches that address the same problem. Predicting longer strides has been a natural approach to build hierarchical world models which the author mention but never compared against, or methods such as jumpy world models with adaptable horizon predictions.

Questions for authors
  1. Under error normalized by latent variance (or relative to a copy-ztz_t baseline), or in a shared latent space, does Fast-LeWM still beat LeWM at k=1k=1? Does the difference in error growth survive? Please also report per-horizon error for k≤Hk \le H separately from the chained region (k>Hk > H)

  2. ztz_t enters both through the state token and through GϕG_\phi. Please report multi-seed results for (a) removing ztz_t from GϕG_\phi while keeping the state token, and (b) the converse. If the prefix encoder alone carries the dynamics, how does the method differ from a causal sequence model whose hidden states are read out, as opposed to “action-prefix” conditioning?

  3. What are the FLOPs per candidate and the per-call latency for each predictor? How much of the speedup remains if the same MLP predictor is run autoregressively?

  4. Why a fixed 2 + 3 split and β=1\beta=1? Were other decompositions (16 ordered splits for H=5H=5, or even multi-way splits) or other values of β\beta tried? Why is the consistency term used only at planning time rather than also as a training loss, given that the decomposed path anchors on a predicted latent, which the model never saw as an anchor during training?

  5. How was LeWM w/ Dense Loss tuned, and does its drop below LeWM (especially Cube, 74 → 58) hold across seeds?

  6. The prefix token acts as a learned high-level action predicting at a longer, but fixed stride. Is the method intended as a replacement for hierarchical or jumpy approaches (e.g., with adaptive strides, or planning over abstract actions), or as a component of them, for instance as the high-level predictor? Does a single predictor shared across horizons perform as well as separate per-horizon heads?

  7. Following the previous question, it the higher stride predictor is indeed better, then should it be composed with a lower level, single step predictor to do hierarchical planning? It seems like the natural composition of longer stride prediction is to be used as intermediate subgoals for lower-level rollouts, and that there is a balance to find between the horizon prediction of methods such as the one proposed here, and the compositionality with single-step executable rollouts. Else we could just increase the prediction horizon to reach the goal in a single step. What does the author think about this concern?

7 comments
AuthorsOpenReview ↗

Response to Reviewer YRkb (1/3)

We appreciate the reviewer's valuable feedback.

Q1: Normalized latent prediction error and error in different regions.

We thank the reviewer for raising this important question. Both LeWM and Fast-LeWM employ the same SIGReg regularization, which encourages their latent representations to follow a standard Gaussian distribution. We further measured the empirical latent variances across four datasets:

DatasetLeWMFast-LeWM
PushT1.00881.0101
Two-Room0.98441.0182
Reacher0.98940.9741
Cube1.01461.0605

As shown above, the latent variances are consistently close to one for both methods, indicating comparable representation scales.

Following the reviewer's suggestion, we normalized the prediction MSE by each model's empirical latent variance:

{V^{(m)}},$$ where $V^{(m)}$ denotes the empirical variance of model $m$'s latent representations. $\hat z_{t}^{(m)}$ and $z_{t}^{(m)}$ are predicted and GT latents, respectively. **The normalized error curves remain nearly unchanged from the original Fig. 3, and our conclusions regarding prediction accuracy remain consistent.** Furthermore, as suggested by the reviewer, we separately report the normalized error at the first horizon $e_1$, the average error for $k < 5$, and the average error for the chained region $k > 5$: | Dataset | Method | $e_1$ ↓ | Avg. $e_{1:5}$ ↓ | Avg. $e_{6:10}$ ↓ | |:---:|:---|---:|---:|---:| | PushT | LeWM | 0.0075 | 0.0316 | 0.1298 | | | Fast-LeWM | **0.0038** | **0.0049** | **0.0179** | | Two-Room | LeWM | 0.0615 | 0.0926 | 0.2103 | | | Fast-LeWM | **0.0054** | **0.0179** | **0.0538** | | Reacher | LeWM | 0.0833 | 0.1199 | 0.1385 | | | Fast-LeWM | **0.0373** | **0.0414** | **0.0633** | | Cube | LeWM | 0.0044 | 0.0192 | 0.0717 | | | Fast-LeWM | **0.0009** | **0.0012** | **0.0028** | **Fast-LeWM consistently achieves lower normalized prediction error across all datasets, both within the direct prediction window and after chaining.** We agree that the improvement at $k=1$ cannot be attributed to reduced recursive error propagation. It instead reflects improvement of the learned representations thanks to our proposed dynamic interface and training scheme. We have updated Fig. 3 and added the corresponding quantitative analysis in the revised paper. ### **Q2: State conditioning.** We thank the reviewer for this question. The two uses of $z_t$ serve different purposes: the state token provides contextual information for action-prefix encoding, while the direct $z_t$ input to $G_\phi$ serves as the anchor for future latent prediction. To examine their individual contributions, we conducted the requested ablations across seven random seeds: | Variant | PushT | Two-Room | Reacher | Cube | Avg. | |:---|:---:|:---:|:---:|:---:|:---:| | (a) w/o $z_t$ in $G_\phi$ | 88.0±2.8 | 92.3±4.5 | 80.0±4.0 | 66.0±7.6 | 81.6 | | (b) w/o state token | 88.8±4.5 | 93.8±3.1 | 82.3±4.3 | 71.3±4.3 | 84.1 | | Fast-LeWM | **91.7±4.1** | **95.4±3.2** | **84.0±5.9** | **71.7±8.4** | **85.7** | Removing the direct $z_t$ input to the predictor reduces average planning success from 85.7 to 81.6, whereas removing the state token from the action-prefix encoder results in a smaller decrease to 84.1. These results suggest that explicit state anchoring in the predictor plays an important role, while state-dependent prefix encoding provides additional benefits. Regarding the architecture, we acknowledge that our action-prefix encoder is implemented as a causal Transformer. However, we would like to clarify that our contribution lies in the action-prefix-conditioned multi-horizon prediction formulation, rather than in the causal sequence encoding architecture itself. We have added these results in the revised paper.
AuthorsOpenReview ↗

Response to Reviewer YRkb (2/3)

Q3: Efficiency analysis.

We thank the reviewer for this question. The predictor FLOPs per candidate action sequence (H=5H=5) are 259.129 MFLOPs for LeWM and 17.990 MFLOPs for Fast-LeWM. We further measure per-call latency using the same batch of 300 candidate sequences:

MetricLeWMFast-LeWM
Action encoder latency / call (ms)0.191.07
Predictor latency / call (ms)2.441.67
Total latency / call (ms)2.632.74

Although Fast-LeWM's predictor is faster per call, its action-prefix encoder introduces additional overhead. Consequently, the total latency of a single dynamics evaluation is slightly higher for Fast-LeWM (2.74 vs. 2.63ms). This indicates that our overall speedup does not result from a faster network.

Autoregressive comparison using the same MLP. We additionally execute the same Fast-LeWM dynamics network autoregressively for five steps (Fast-LeWM-AR).

MetricLeWMFast-LeWMFast-LeWM-AR
Dynamics latency (s)19.74.120.8
Full CEM time (s)33.716.134.9

The total dynamics time is accumulated over 50 goal images and 30 CEM iterations, with 300 candidates per iteration. With the same network, autoregressive execution increases dynamics latency from 4.1 to 20.8s and full CEM planning time from 16.1 to 34.9s, yielding an even larger speedup than the comparison with LeWM.

Q4: Self-consistency decomposition.

We thank the reviewer for these suggestions. The original 2+32+3 decomposition and β=1\beta=1 were selected empirically using the validation set, without exhaustive hyperparameter tuning. We do not claim that this configuration is optimal.

Sensitivity to decomposition. Following the reviewer's suggestion, we additionally evaluated all 16 ordered compositions of H=5H=5 across seven random seeds. The complete results are reported below, where [5] denotes the baseline without self-consistency.

SplitPushTTwoRoomReacherCubeAvg.
[5]91.7±4.195.4±3.284.0±5.971.7±8.485.7
[1,4]92.0±4.694.6±3.684.3±7.776.9±5.887.0
[2,3]92.3±5.194.3±4.884.4±3.675.4±6.686.6
[3,2]92.6±4.695.1±3.886.0±2.376.9±6.387.7
[4,1]93.4±4.995.1±3.685.7±4.276.0±7.587.6
[1,1,3]92.0±4.894.9±5.083.1±5.477.7±5.986.9
[1,2,2]92.3±5.794.9±5.386.9±3.878.0±6.788.0
[1,3,1]93.7±5.194.9±3.682.6±6.678.6±7.387.5
[2,1,2]92.9±5.195.1±6.083.4±3.878.0±5.287.4
[2,2,1]93.1±4.994.0±5.282.6±3.678.6±7.687.1
[3,1,1]91.7±5.294.6±4.686.9±2.376.9±5.687.5
[1,1,1,2]93.4±4.994.0±4.278.9±4.579.4±6.986.4
[1,1,2,1]93.7±5.194.9±4.182.9±3.878.9±6.687.6
[1,2,1,1]92.3±4.594.3±5.081.4±9.678.9±6.786.7
[2,1,1,1]93.1±4.595.1±3.885.7±3.778.0±5.988.0
[1,1,1,1,1]91.7±4.495.0±5.783.1±5.878.3±6.087.0

All 15 nontrivial decompositions improve the average planning success over [5] (without self-consistency). Although some configurations outperform our default setting [2,3], the results indicate that the benefit does not depend on a particular decomposition.

Why planning-time rather than training-time consistency? Our motivation is to use agreement between direct and decomposed predictions as an implicit indicator of prediction reliability. Since the consistency between different prediction paths is not explicitly enforced in training, their agreement can therefore provide an additional signal for ranking candidate action sequences during planning.

Explicitly optimizing this agreement during training could weaken its usefulness as an independent diagnostic signal, since predictions may become consistent without necessarily becoming more accurate. Moreover, training-time consistency regularization would introduce additional computational overhead. We therefore use self-consistency as an optional planning-time criterion, while retaining the original objective for training.

We have added the complete decomposition analysis and clarified our design choices in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer YRkb (3/3)

Q5: LeWM w/ Dense Loss.

We thank the reviewer for raising this question. We retained LeWM's original architecture and training hyperparameters. The only modification was to unroll additional autoregressive prediction steps during training and apply a prediction loss at each horizon. All horizon-specific losses are backpropagated through the full unrolled autoregressive computation graph. All other settings, including the training data, optimizer, learning-rate schedule, and regularization, remained unchanged. The planning and evaluation protocols were also identical.

As suggested, we additionally evaluated LeWM w/ Dense Loss across seven random seeds:

TaskLeWMLeWM w/ Dense Loss
PushT87.1 ±\pm 7.287.5 ±\pm 4.0
Two-Room87.0 ±\pm 4.991.8 ±\pm 4.0
Reacher83.4 ±\pm 5.781.8 ±\pm 4.5
Cube67.4 ±\pm 8.662.8 ±\pm 5.6
Average81.281.0

Dense supervision improves performance on Two-Room, yields comparable results on PushT, and reduces performance on Reacher and Cube. These results clarify our conclusion: simply adding dense multi-horizon supervision to LeWM's autoregressive prediction interface does not yield a consistent improvement. We have included the seven-seed results in the revised paper.

Q6. Relation to hierarchical/jumpy world models and shared prediction heads.

We thank the reviewer for this question. We would like to clarify that Fast-LeWM operates at a different level from hierarchical planning methods. Our action-prefix tokens are not learned high-level actions or subgoals. Instead, they encode given primitive-action prefixes, allowing the predictor to directly estimate the corresponding future latent states. Fast-LeWM retains LeWM's original CEM planning framework and does not introduce a hierarchy of actions or controllers. Our method is therefore not intended to replace hierarchical or jumpy planning, although its prediction interface could potentially be incorporated into such frameworks, as discussed in Section 5 (Limitations and Future Work).

Shared vs. separate per-horizon predictors. Following the reviewer's suggestion, we additionally evaluated separate per-horizon prediction heads across seven random seeds:

TaskSeparate headsFast-LeWM (Shared)
PushT91.7 ±\pm 4.791.7 ±\pm 4.1
Two-Room92.0 ±\pm 4.295.4 ±\pm 3.2
Reacher85.7 ±\pm 4.884.0 ±\pm 5.9
Cube72.0 ±\pm 5.471.7 ±\pm 8.4
Average85.485.7

The two variants achieve comparable average planning performance (85.4 vs. 85.7). Separate per-horizon heads increase the parameter count and GPU memory usage without improving average planning performance in our experiments.

Q7. Hierarchical composition.

We appreciate the reviewer's perspective, but would like to clarify an important distinction between our approach and subgoal-based hierarchical planning. In the latter, a high-level module typically generates intermediate target states, which a low-level controller then attempts to reach. In contrast, Fast-LeWM predicts future states conditioned on explicitly specified sequences of primitive actions. These actions are already optimized by CEM, so our predicted future states do not need to serve as subgoals for an additional low-level planner.

Importantly, Fast-LeWM operates within a receding-horizon planning framework. At each planning cycle, CEM optimizes a finite-horizon action sequence from the current real observation. The planned actions are then executed, and the next planning cycle starts from a newly observed state rather than the previously predicted terminal latent. Therefore, our method does not require a single long-horizon prediction to reach the final goal, nor does it accumulate prediction errors across successive replanning cycles through predicted-state re-anchoring.

This also explains why simply increasing the prediction horizon indefinitely is not necessary. Longer horizons increase both the dimensionality of CEM action-sequence optimization and the difficulty of learning accurate long-horizon dynamics. Receding-horizon planning instead allows the model to operate over manageable prediction windows while periodically incorporating real observations.

We agree that hierarchical planning could further improve scalability to complex long-horizon tasks, but it is a complementary extension rather than a prerequisite for our approach. We have already discussed this possibility in Section 5 (Limitations and Future Work).

Reviewer_XxxcOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a common computational bottleneck in JEPA-based world models: recursively applying one-step latent predictions makes the planning process slow and computationally expensive. The authors propose Fast-LeWM, which introduces an action-prefix encoder to process action sequences and a parallel predictor to predict multiple future latent states in a single forward pass, rather than relying on recursive rollout as in the original LeWM. By replacing multiple sequential forward passes with parallel prediction, the method reduces dynamics-evaluation time from 19.7s to 4.1s.

Critical strengths
  1. Significant efficiency improvement. The proposed method achieves a substantial speedup by replacing multiple recursive forward passes with a single parallel prediction process, reducing dynamics-evaluation time from 19.7s to 4.1s. This is a meaningful improvement for planning efficiency.

  2. Clear presentation and intuitive design. The paper is generally well written and easy to understand. Replacing recursive latent prediction with one-pass parallel prediction is a reasonable and intuitive approach to addressing the computational bottleneck in world-model planning.

Critical weaknesses
  1. Using an action sequence to predict multiple future latent states in a single forward pass seems to be a relatively straightforward idea. Similar approaches may already have been existed without claim in code of related works, such as ThinkJEPA or other JEPA-based architectures.

  2. The paper should more clearly explain the novelty of the proposed action-prefix encoding mechanism and how it differs from existing approaches.

  3. In addition, the method is only evaluated on LeWM. I do not see an obvious architectural reason why it could not be applied to other models, such as DINO-WM or V-JEPA 2-AC. Its generalizability beyond LeWM remains unclear.

  4. Limited efficiency analysis and comparison. Since this is primarily an efficiency-focused paper, more computational metrics would be helpful, such as FLOPs, a more detailed speedup breakdown, and the additional computational overhead introduced by the action-prefix encoder. Although the method achieves faster inference, it would also be useful to understand whether the same approach could be applied to VLM-based world models, such as ThinkJEPA, and how its efficiency compares with related architectures.

  5. The title is somewhat generic and does not clearly convey the key technical contribution. I suggest making it more descriptive by highlighting the action-prefix encoding mechanism or parallel multi-horizon prediction that makes LeWorldModel faster.

Questions for authors
  1. Action-prefix length. How is the length of the action prefix determined when only a target image is provided? Is the planning horizon fixed, or can it be dynamically adjusted based on the target?

  2. Generalizability. Is the proposed action-prefix encoding approach specific to LeWM, or could it also be applied to other JEPA-based models, such as DINO-WM, V-JEPA 2-AC, or VLM-based architectures like ThinkJEPA? Are there any architectural limitations that would prevent such extensions?

  3. Computational overhead. How much additional computational overhead does the action-prefix encoder introduce compared with the original LeWM? Could the authors provide a more detailed breakdown of the parameter count and computational cost?

  4. Novelty. What is the key technical novelty of the proposed approach compared with existing methods that predict multiple future latent states from an action sequence in a single forward pass?

7 comments
AuthorsOpenReview ↗

Response to Reviewer Xxxc (1/3)

We thank the reviewer for the constructive comments and address the raised issues below.

W1/W2/Q4: Novelty and distinction from prior work.

We thank the reviewer for raising this important question. We agree that predicting multiple future latent states in parallel is conceptually simple. However, our key contribution is not parallel prediction itself, but an action-prefix-conditioned dynamics formulation that enables direct, horizon-specific latent prediction without autoregressive rollout, together with an effective training design that improves both planning efficiency and accuracy.

1. Distinction from prior work. To the best of our knowledge, existing JEPA-based visual world models for action-conditioned planning, including LeWM and DINO-WM, rely on autoregressive latent rollout to evaluate candidate action sequences. Fast-LeWM is the first, to our knowledge, to replace this recursive planning interface with direct action-prefix-conditioned latent prediction in this setting. While some existing architectures can predict multiple future frames in a single forward pass, they generally do not formulate explicit action-prefix-conditioned transitions for planning. In some cases, multi-frame prediction simply changes the temporal granularity of the prediction unit, while the inference process still relies on recursive transitions. Importantly, Fast-LeWM is not restricted to fixed-window, multi-output forecasting. Instead, it supports variable-length action prefixes and flexible horizon-specific prediction. Given the current latent state and an action prefix of length kk, its predictor directly estimates the corresponding future latent:

p_{t,k}=E_\psi(a_{t:t+k-1}).$$ This formulation allows the model to **directly query future states at different horizons using action prefixes of different lengths**, without generating intermediate states. While multiple prefixes can be evaluated in parallel when needed, **the fundamental change is the dynamics interface itself, rather than simply increasing the number of predicted frames.** Across all LeWM planning benchmarks, Fast-LeWM directly predicts the terminal latent of each action sequence without intermediate rollout, achieving a 4.8\$\times\$ speedup in dynamics evaluation while improving planning success. **Comparison with ThinkJEPA.** Except for the above, there are also other substantial differences between ThinkJEPA and Fast-LeWM. (i) The main contribution of ThinkJEPA is incorporating pretrained VLM reasoning representations into a V-JEPA predictor, primarily to improve semantic grounding and latent forecasting. In contrast, Fast-LeWM targets the computational inefficiency and error accumulation of autoregressive dynamics in action-conditioned planning, and introduces a direct action-prefix transition interface. (ii) ThinkJEPA relies on pretrained V-JEPA representations and cached VLM features, whereas Fast-LeWM jointly learns visual representations and action-conditioned dynamics from scratch with explicit anti-collapse regularization. (iii) ThinkJEPA employs supervised downstream trajectory heads for its trajectory-prediction tasks, while Fast-LeWM directly supports reward-free goal-conditioned planning without task-specific trajectory supervision. **Thus, the two methods address different research problems and differ substantially in their dynamics formulation, learning objectives, and downstream applications.**
AuthorsOpenReview ↗

Response to Reviewer Xxxc (2/3)

W1/W2/Q4 (continued).

2. Straightforward multi-state prediction is insufficient. To directly address the reviewer's concern, we additionally implemented a naive multi-step LeWM baseline that takes an action sequence as input and predicts multiple future latents in a single forward pass with dense loss. Under the same seven-seed evaluation protocol, this baseline achieves only 80.1 average planning success, compared with 81.2 for LeWM and 85.7 for Fast-LeWM. This demonstrates that simply extending a JEPA predictor to accept action sequences and predict multiple future states does not reproduce our improvements.

3. Beyond straightforward parallel prediction. While the basic idea of parallel prediction may appear straightforward, developing an effective latent world model that achieves both faster prediction and higher planning accuracy is by no means straightforward. Achieving these two objectives simultaneously requires addressing several interconnected challenges through a carefully integrated design.

Specifically, Fast-LeWM combines four complementary components: (i) a shared-anchor, action-prefix dynamics interface that eliminates sequential dependence on intermediate predicted states; (ii) causal action-prefix encoding that ensures each horizon-specific representation depends only on its corresponding action prefix; (iii) dense prefix-level supervision that directly learns accumulated action effects across different horizons; and (iv) state conditioning that resolves ambiguity in how the same action sequence affects different initial configurations.

Crucially, these components are interdependent. Without direct prefix prediction, dense multi-horizon supervision operates through autoregressive rollout, where errors and gradients propagate through successive predicted states. Conversely, parallel prediction alone does not ensure accurate multi-horizon dynamics without dense supervision. Causal encoding ensures proper alignment between action prefixes and prediction horizons, while state conditioning provides the context necessary to interpret their effects.

Our Table 4 provides further empirical evidence: simply extending LeWM's action horizon reduces success from 85.8 to 71.0; adding dense supervision to autoregressive LeWM yields 79.0; removing dense supervision from Fast-LeWM reduces success to 84.5; and removing state conditioning reduces success to 87.0, compared with 90.5 for the complete Fast-LeWM.

In summary, the central novelty of Fast-LeWM lies in reformulating action-conditioned latent dynamics around causal action prefixes and developing the corresponding architecture and learning objective to make this formulation effective for planning. The resulting method achieves a 4.8× speedup in dynamics evaluation while improving planning success.

We have revised Section 2 to clarify this distinction.

Q1: Action-prefix length.

We strictly follow LeWM's planning protocol, which uses a fixed planning horizon of (H=5). Although our action-prefix encoder supports variable-length prefixes, we retain LeWM's fixed-horizon setting throughout all experiments to ensure a fair comparison.

AuthorsOpenReview ↗

Response to Reviewer Xxxc (3/3)

W3/Q2: Generalizability beyond LeWM.

We thank the reviewer for raising this important question. Our action-prefix formulation is not specific to LeWM and can be extended to other action-conditioned latent world models. In fact, our method is designed as a general dynamics interface that does not depend on LeWM's particular visual encoder, latent regularization, or predictor architecture.

To demonstrate the generalizability of our approach beyond LeWM, we incorporated our action-prefix encoder and parallel predictor into DINO-WM, keeping the datasets and training settings unchanged. The results are summarized below:

DatasetDINO-WMDINO-WM + Ours
Two-Room100100
Reacher7984
PushT7482
Cube8672
Average84.884.5

Importantly, our approach achieves an approximately 11.9× speedup in CEM dynamics evaluation (32.76s to 2.76s), while maintaining comparable average planning success (84.5 vs. 84.8). These results provide direct empirical evidence that our action-prefix formulation can be transferred to a different latent world model, delivering substantial computational benefits without relying on LeWM-specific architectural components. Note that the planning gains on DINO-WM are less pronounced than those on LeWM, possibly because DINO-WM's frozen encoder cannot benefit from representation improvements induced by our dense supervision. Further investigation in DINO-WM would be an interesting direction for future research.

In summary, Fast-LeWM's action-prefix formulation is transferable beyond LeWM and can substantially accelerate planning across different visual representation frameworks. While improvements may vary across backbones, the results support our central claim that action-prefix-conditioned dynamics provide a broadly applicable and computationally efficient alternative to autoregressive latent rollout.

These results and discussions are included in Appendix F.

W4/Q3: Computational overhead and efficiency analysis.

We thank the reviewer for raising this important question. Although the action-prefix encoder introduces additional computation, this overhead is substantially outweighed by the savings from parallel latent prediction. We provide a detailed breakdown of parameter counts and FLOPs, measured per action sequence of five transition steps.

ModuleLeWM Params (M)Our Params (M)LeWM MFLOPsOur MFLOPs
Action encoder0.1561.8034.65718.322
Predictor10.7919.023259.12917.990

While the action encoder becomes more expensive, predictor computation decreases by 93.1% (259.129 to 17.990 MFLOPs). The combined parameter count remains comparable (10.9M vs. 10.8M).

This computational reduction is also reflected in the measured runtime. As shown in Figure 1 of our manuscript, under the same CEM planning budget, action encoding time increases only slightly from 1.4s to 1.6s, whereas latent prediction time decreases from 18.3s to 2.5s. Consequently, Fast-LeWM achieves a 4.8× speedup in dynamics evaluation (19.7s to 4.1s).

We have included the detailed computational analysis in Appendix E.

W5: Paper title.

We thank the reviewer for this helpful suggestion. We have revised the title to “Fast LeWorldModel with Action-Prefix Multi-Horizon Prediction” to better highlight the key technical contribution of our approach.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Additional Evidence

Dear Area Chair and Reviewers,

We sincerely thank all reviewers for their constructive feedback and careful consideration of our work. As the discussion phase comes to an end, we would like to summarize the central contribution of our work, its recognized strengths, and the substantial additions and revisions made during rebuttal.

1. Central Contribution

The central contribution of Fast-LeWM is reformulating action-conditioned latent dynamics around variable-length action prefixes, enabling direct prediction of future latent states at different horizons from a shared observed anchor, without recursively propagating intermediate predicted states within each prediction window.

Rather than merely predicting multiple future states in parallel, Fast-LeWM combines causal action-prefix encoding, state-conditioned prediction, and dense multi-horizon supervision to learn the accumulated effects of action sequences. This formulation provides a more efficient dynamics interface for model-based planning while improving prediction accuracy and planning performance.

2. Recognized Strengths

We are encouraged that the reviewers have recognized several strengths of our work, including:

  • “A solid paper addressing an important bottleneck in JEPA-based world models” (Reviewer Xxxc).
  • Significant improvements in planning efficiency (Reviewers Xxxc, YRkb, 4CsG).
  • Competitive or improved planning success alongside substantial computational savings (Reviewers YRkb, 4CsG).
  • A clear and intuitive approach to improving latent world-model planning (Reviewer Xxxc).

3. Key Additional Evidence in the Rebuttal

To directly address the reviewers' concerns, we conducted extensive additional experiments and analyses:

(1) Novelty and controlled comparisons.

We implemented a Direct Multi-Step LeWM baseline matched for state conditioning, multi-horizon supervision, model size, training data, hyperparameters, and training budget. This baseline achieves 80.1% average planning success, compared with 85.7% for Fast-LeWM.

We also evaluated LeWM with dense multi-horizon supervision, confirming that dense supervision alone does not consistently improve planning performance.

These results help distinguish the contribution of our action-prefix formulation from straightforward parallel prediction and additional supervision.

(2) Generalizability and efficiency.

We extended our approach to DINO-WM, achieving an 11.9× speedup in dynamics evaluation while maintaining comparable average success (84.5% vs. 84.8%).

We further provided detailed parameter counts, FLOPs, and per-call latency analyses. In particular, executing the same Fast-LeWM network autoregressively increases cumulative dynamics time from 4.1 s to 20.8 s and full CEM planning time from 16.1 s to 34.9 s.

These results clarify that the efficiency improvement arises primarily from eliminating sequential prediction calls, rather than relying on a faster individual network.

(3) Accuracy and error propagation.

To address concerns about comparing independently learned latent representations, we supplemented the original evaluation with physical-space prediction errors in common coordinates, empirical latent-variance statistics, and variance-normalized prediction errors.

We also systematically evaluated all 16 ordered partitions of the five-step planning horizon under identical model weights and planning budgets. The single-window configuration achieves the highest average success among all 16 partitions, and configurations involving fewer chained windows tend to perform better.

These results provide additional evidence for the benefit of reducing recursive feedback through predicted intermediate states.

(4) Architectural ablations and statistical reliability.

We performed additional seven-seed experiments examining state conditioning, dense supervision, shared versus separate prediction heads, and different self-consistency decompositions.

We also reported paired performance improvements between Fast-LeWM and LeWM across seven seeds. The overall average improvement is 4.46 percentage points, with a 95% paired confidence interval of [2.60, 6.33] percentage points.

4. Closing Remarks

We sincerely appreciate the reviewers' detailed comments, which have helped us substantially strengthen the paper. We have carefully addressed the raised concerns through additional experiments, methodological clarifications, and corresponding manuscript revisions.

We believe that the combined evidence highlights the scientific contribution and practical significance of action-prefix-conditioned dynamics as an effective alternative to autoregressive latent rollout for visual planning.

We hope these additional results and revisions provide a more complete basis for evaluating our work, and we would be grateful if they could be taken into account in the final assessment.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.