acceptodds
Withdrawn from ICLR 2027

Beyond Myopic World Models: Long-Horizon End-to-End Training for Direct Future Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

World models are expected to support imagination over extended temporal horizons, yet most are still trained through local few-step prediction objectives and deployed by recursively rolling out their own predictions. This creates a fundamental mismatch: few-step losses optimize local transition fidelity, while long-horizon prediction depends on how errors and gradients propagate through the entire trajectory. As a result, transitions with different downstream influence on the endpoint are treated uniformly during training, and small local errors are amplified through recursive inference. We argue that long-horizon accuracy is better achieved by optimizing directly, through an end-to-end endpoint prediction objective. To instantiate this paradigm, we introduce the Direct Prediction World Model (DPWM), a non-recursive architecture that compresses an action sequence of arbitrary length into a single embedding and predicts the endpoint observation in a single forward pass. This design avoids recurrent rollout in both prediction and gradient propagation, making long-horizon end-to-end training practical. Empirically, DPWM substantially improves long-horizon endpoint prediction over recursive world-model baselines on continuous-control and pixel-based benchmarks, with larger gains as the prediction horizon increases. We further show that a recurrent any-step baseline improves substantially at long horizons when retrained with the same long-horizon endpoint objective, supporting our central claim that the training horizon, rather than the backbone alone, is a major driver of long-horizon prediction accuracy. Our results suggest that world models can benefit from being trained and evaluated at the temporal scales where they are ultimately used, shifting the focus from local transition modeling toward long-horizon predictive accuracy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
world models, long-horizon prediction, dynamics modeling

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_f8QDOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies long-horizon predictions in action-conditioned world models. It proposes DPWM, which encodes a variable-length action sequence using a Transformer to predict the endpoint latents directly. The complete pipeline is trained with endpoint reconstruction losses over sampled prediction horizons. Experiments evaluate DPWM through prediction error on four state-based continuous-control tasks from DeepMind Control Suite and a pixel-based Atari Game.

Critical strengths
  1. The idea of future prediction with a variable-length action sequence is appealing. To that end, the paper proposes a simple action sequence encoder and a dynamic model to predict the endpoint observation.
  2. The paper is easy to understand.
Critical weaknesses
  1. The contribution beyond prior work is insufficiently clarified. The paper emphasizes that "the training horizon, rather than the backbone alone, is a major driver of long-horizon prediction accuracy" (Line 29-31). However, it is common knowledge that training a model with supervision at longer horizons improves its prediction accuracy at correspondingly long test horizons. More importantly, the core idea of predicting a future outcome conditioned on an initial state and a variable-length action sequence has already been explored by ADM and ADM-v2. This concern is further amplified by the fact that DPWM predicts only the endpoint observation. Since the motivation for long-horizon world models is closely tied to planning and decision making, it would be important to explain how the proposed model is intended to support downstream policy evaluation or policy learning.
  2. The evaluation is too limited to establish a fair and practically meaningful comparison with prior world-model methods. The evaluation is almost entirely based on endpoint MSE. While these metrics directly measure the stated endpoint-prediction objective, they provide only a narrow view of world-model quality and make the reported results difficult to compare with the broader evaluation protocols used in closely related work. In particular, the main results in Table 2 are not fully controlled: DPWM is trained with Kmax⁡=100K_{\max}=100, whereas ADM uses its short-horizon setting with Kmax⁡=3K_{\max}=3, and the other recurrent baselines are trained with local one-step objectives. More importantly, all the compared baselines in the paper (i.e. DreamerV3, ADM-v2, and MoSim) evaluate long-horizon modeling in terms of downstream policy learning or policy evaluation rather than prediction error alone. ADM-v2, for example, reports off-policy evaluation on DOPE and policy optimization results on D4RL and NeoRL. In contrast, the current paper explicitly removes the actor–critic component when evaluating Dreamer and does not report any policy-level metric. Thus, it remains unclear whether the reduction in endpoint MSE translates into any meaningful improvement in planning, policy evaluation, or control performance.
Questions for authors
  1. Lines 413–415 state that the native 210×160210\times160 frame is cropped to 160×160160\times160 to remove the "static scoreboard region". However, the Pong scoreboard changes as the game progresses. Could the authors clarify the motivation for removing this region and whether this preprocessing is necessary?
  2. Table 2 reports endpoint prediction error in raw state space, whereas Table 3 reports MSE in normalized state space. Why are different evaluation spaces used? Please clarify whether the conclusions remain unchanged under a consistent metric.
  3. Which ADM or ADM-v2 implementation is used in each experiment? How are predictions beyond the model’s training horizon obtained?
  4. Figure 3 reports normalized training time, inference latency, and peak GPU memory. Could the authors also report the absolute peak memory in GB, training time, and inference latency at each evaluated horizon, together with the hardware configuration and model size?
  5. Lines 365–367 state that Table 3 is intended to isolate the effect of long-horizon training from the architectural choice. While ADM${}_{K=3}$ versus ADM$_{K=100}$ isolates the training-horizon effect within the same backbone, ADM$_{K=100}$ versus DPWM$_{K=100}$ may still differ in model capacity, optimization, and compute budget. Which factors are strictly controlled? Why is this comparison reported only on Humanoid?
  6. The paper motivates long-horizon world models in the context of planning and decision making, but evaluates DPWM almost exclusively using endpoint MSE. Why are established downstream protocols, such as policy evaluation or policy learning, not included?
Reviewer_f4ErOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies whether world models should be trained directly for the long-horizon endpoint predictions at which they are evaluated. Its Direct Prediction World Model (DPWM) encodes an entire variable-length action sequence using a local-attention Transformer, mean-pools the resulting tokens, and conditions a non-recursive endpoint predictor on that embedding and the initial observation. Experiments evaluate state-space prediction on four continuous-control tasks and pixel prediction on Pong, with additional training-horizon and computational-scaling comparisons (pp. 3–9, Sections 2–3).

The most informative result retrains the recurrent ADM baseline with longer endpoint supervision: much of its prediction gap closes, supporting the paper's central training-horizon hypothesis. The main reservations concern horizon information in the documented encoder, an overextended gradient argument, and incomplete experimental reporting. These reservations qualify the interpretation of promising endpoint results rather than erase the useful controlled observation (pp. 7–8, Table 3).

Critical strengths
  1. A relevant control separates training horizon from backbone. Table 3 retrains ADM with Kmax⁡=100K_{\max}=100, rather than comparing DPWM only with ADM trained at Kmax⁡=3K_{\max}=3. On matched-policy Humanoid evaluation at K=100K=100, ADM error falls from 0.7756 to 0.2999, a 61.33% reduction; DPWM then reaches 0.2658, a further 11.37% reduction. This is strong descriptive evidence that supervision horizon explains a substantial part of the improvement (pp. 7–8, Section 3.2/Table 3).

    The direction also holds under policy-to-random evaluation: ADM improves from 0.8011 to 0.6318, with DPWM at 0.6058. These results support a substantive conclusion about the recurrent baseline, even though broader generality requires care (p. 8, Table 3).

  2. The primary state results show substantial comparative endpoint gains. DPWM is best in 27 of Table 2's 32 task/distribution/horizon cells, including 15 of 16 cells at horizons 100 and 200. This covers four tasks and both random-action and expert-intermix data. The pattern is meaningful under the explicitly different training paradigms; it should not be reduced to the favorable individual examples (pp. 6–7, Table 2 and experimental setup).

    The exceptions also help calibrate the claim: MoSim is better on random-action Cheetah at horizon 200, and other models win some short-horizon Hopper/Walker cases (p. 6, Table 2).

  3. The prediction interface has a clear computational motivation. Encoding the action sequence and directly predicting its endpoint avoids recursively generating every intermediate observation. The paper distinguishes fixed sequential depth from total computation and acknowledges the memory tradeoff. Figure 3 supports milder horizon-dependent time growth, rather than asserting constant runtime or uniformly lower resource use (pp. 5–6, Figures 2–3/Table 1; p. 8).

  4. The study includes useful bounded probes beyond the main table. Figure 4 varies the maximum training horizon and includes a Cheetah policy-to-random transfer test. Pong compares DPWM100 with ADM100, and the appendix shows the full 70-step independent-prefix visualization. These tests broaden the empirical picture across horizons and observation modalities without by themselves proving arbitrary-policy or arbitrary-horizon generalization (pp. 7–9, Figure 4/Table 4; pp. 14–15, Appendix B/Figure 6).

Critical weaknesses
  1. The documented encoder cannot identify horizon for constant-action sequences. Section 2.3 and Appendix C specify action projections or embeddings, standard RoPE attention, tokenwise layers, and mean pooling. They do not specify an explicit horizon input, boundary token, or positional feature added to values (pp. 5, 16–17).

    The concern has explicit assumptions: deterministic inference with dropout disabled; standard RoPE rotating queries/keys but not values; no additional position, length, or boundary feature; and averaging over valid action tokens. For a sequence containing KK copies of action aa, all input tokens are identical. Each attention head averages identical value vectors, so ∑jpijv=v\sum_j p_{ij}v=v regardless of the attention weights, local-window boundary, or sequence length. Residual connections and tokenwise normalization/feedforward layers preserve this identity. Induction through the layers therefore gives the same pooled embedding for every positive KK.

    With the initial observation fixed, the documented model consequently predicts the same endpoint for one repeated action and for many repetitions. Even the simple dynamics st+K=st+Kas_{t+K}=s_t+Ka contradict this restriction. This is a counterexample to the described variable-horizon design, not a verified failure of an implementation that has not been specified fully (pp. 3, 5, 16–17).

    Explicit length conditioning, special tokens, positional information entering values, or a pooling rule that exposes padding/length could invalidate the argument. The exact mechanism matters: finite attention-window boundaries alone do not distinguish identical values. Nor does this argument claim that RoPE makes all action permutations indistinguishable. Clarification should establish which model actually produced the results; a proposed architectural fix requires validation.

  2. The gradient-dominance explanation exceeds the analysis. Equation 5 expands recursive endpoint parameter sensitivity, but a Lipschitz upper bound L>1L > 1 does not establish that early transitions dominate. Consider fθ(s,a)=diag⁡(2,1/2)s+θ(0,1)⊤f_\theta(s,a)=\operatorname{diag}(2,1/2)s+\theta(0,1)^\top. Its state Lipschitz constant is 2, yet the parameter perturbation lies in the contracting direction: its contribution from step kk is proportional to 2−(K−1−k)(0,1)⊤2^{-(K-1-k)}(0,1)^\top. Earlier contributions shrink rather than dominate (p. 4, Equation 5 and following discussion).

    Directional Jacobians, per-step parameter derivatives, and the endpoint residual all affect the endpoint-loss gradient. Uniformly averaging local losses also does not create uniform vector gradients. The manuscript does not define or bound the claim that misalignment grows with horizon. Endpoint supervision targets endpoint error directly, but the direct network does not thereby acquire the recursive Jacobian factorization of Equation 5 (p. 4, Equations 3–5).

    Appendix A's recurrence and geometric-sum bound are correct under their assumptions. However, a growing worst-case upper bound is not a lower bound on actual rollout error. The direct approximation error δK\delta_K is defined rather than controlled, and may itself grow with horizon. The authors acknowledge this qualification; the stronger gradient language should be brought into alignment with it (pp. 3, 14, Section 2.1 and Appendix A, Equations 6–10).

  3. The objective-versus-backbone conclusion is only partly isolated. Table 2 changes both architecture and supervision horizon. Table 3 provides the essential within-ADM control, but only on Humanoid, and changes the action-repeat and normalization conventions relative to Table 2. Its values cannot directly partition the main table's performance gaps (pp. 6–8, Tables 2–3).

    At horizons 16, 100, and 200, DPWM's additional reductions relative to ADM100 are 7.10%, 11.37%, and 7.15% under matched-policy evaluation, and 8.28%, 4.12%, and 4.79% under random-action evaluation. These are useful differences, but uncertainty is needed to judge their robustness. The short-trained ADM remains best at horizon 1 in both Table 3 distributions, so the benefit also has a short-horizon tradeoff (p. 8, Table 3).

    Section 3.1 cites both ADM versions without clearly identifying the implemented variant; Appendix C supplies DPWM settings but not baseline capacity, optimization budget, or tuning/model-selection details. This does not demonstrate unfairness. It limits verification of the broader causal interpretation and the residual architectural advantage (p. 6, Section 3.1; pp. 16–17, Appendix C).

  4. The metrics and data distribution qualify the dynamics interpretation. Endpoint MSE is a legitimate objective. Nevertheless, Table 2 combines raw position/velocity coordinates with different units, while the normalization used in Table 3/Figure 4 is undefined. Pong's predominantly static background can dilute errors on its small moving objects. A foreground-sensitive diagnostic or simple endpoint reference would clarify what the aggregate gains represent (pp. 6–9, Tables 2–4/Figure 4).

    Pong additionally uses a single frame without frame stacking and a deterministic ball-chasing policy. Ball velocity is not directly observed, while the supplied future actions can carry policy-specific information about ball motion. Action conditioning is appropriate, but same-policy prediction may combine dynamics inference with this policy information. The full appendix visualization helps qualitatively without resolving that distinction quantitatively (pp. 8–9, Section 3.3; pp. 14–16, Appendix B/Figure 6 and preprocessing).

    The horizon gains also need bounded wording. In Pong, the relative MSE reduction versus ADM100 is 13.45% at horizon 100, 19.17% at 200, and 11.22% at 400. In Table 2, DPWM's policy-Walker error increases from 3.7131 to 18.9293 between horizons 100 and 200. Comparative superiority can coexist with substantial extrapolation degradation; it does not establish uniformly stable long-horizon fidelity (pp. 6, 9, Tables 2 and 4).

  5. Replication and uncertainty details remain incomplete after the appendix. The submission omits trajectory/sample counts and lengths, exact intermix probability/segment lengths, split and initialization procedures, independent training-run counts, evaluated-example counts, and which uniform/log-uniform horizon sampler generated each experiment. These choices determine the effective prediction problem (pp. 4, 7–9; pp. 16–17, Appendix C).

    Figure 4's shaded bands are undefined, and Tables 2–4 report point values without uncertainty. The epoch schedule does not establish actual updates or matched baseline resources. Figure 3 normalizes each model to its own horizon-16 reference without absolute times, hardware, or measurement variability. It therefore supports relative scaling, not absolute efficiency. These are reporting and evidential gaps, rather than demonstrated leakage or invalid measurements (pp. 5–9, Figures 3–4/Tables 2–4; pp. 16–17, Table 5).

Questions for authors
  1. What exposes horizon to the action encoder? Please specify all length/position/boundary inputs, whether RoPE affects values, and the exact masking/pooling denominator. For fixed initial observations and repeated actions, do predictions vary appropriately with KK? If an omitted mechanism was used for the reported results, explaining it and supplying this diagnostic would materially affect the soundness assessment (Weakness 1; pp. 5, 16–17).

  2. What assumptions support the gradient claims? How do early-transition dominance and increasing misalignment follow despite the contracting-direction counterexample? Can the text distinguish endpoint sensitivity, endpoint-loss gradients, and the valid worst-case rollout bound, or narrow the claims accordingly (Weakness 2; p. 4, Equation 5; p. 14, Appendix A)?

  3. How were the objective-aligned comparisons matched? Which ADM variant, capacities, optimizer updates, data, tuning, and model-selection rules were used? Please provide uncertainty for Table 3 and explain the action-repeat/normalization differences. A focused matched comparison on another already studied task would strengthen the broader interpretation without requiring a new benchmark suite (Weaknesses 3 and 5; pp. 6–8, Tables 2–3).

  4. How much accuracy depends on action-conditioned dynamics? On the existing held-out data, how does performance compare with a simple endpoint reference or an action-ablated predictor? For Pong, can ball-location or foreground error distinguish dynamic accuracy from low background error, while accounting for single-frame ambiguity (Weakness 4; pp. 8–9, Section 3.3/Table 4)?

  5. What are the evaluation and uncertainty units? Please give data sizes/splits, intermix settings, horizon sampler by experiment, state normalization, run/evaluation counts, and Figure 4's band definition. For Figure 3, please give absolute timing/memory references and hardware. Which smaller reported gaps remain stable across independent runs (Weakness 5; pp. 4–9; pp. 16–17)?

Reviewer_YLeeOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether direct regression of future states within world models behaves better compared to the autoregressive setup. In the context of long-horizon future modeling, error accumulation could be significant, therefore this paper seeks to measure whether autoregressive inference accumulates more error than direct prediction.

Critical strengths

The research question (autoregressive vs direct prediction) is interesting to investigate.

Critical weaknesses

I am fairly confident the core contribution is already investigated in Lambert et al. [1]. Also, there are practical aspects of direct prediction presented in this submission.

  1. K_max is a fixed parameter that limits prediction beyond this step (moving beyond K_max would make the model autoregressive).
  2. The model is conditioned on a single frame, meaning it cannot model important motion features.
  3. It is unclear from the paper whether the method achieves better MSE due to learning a mean of possible trajectories (autoregreesive models that lead to sharp rollouts of a single trajectory might deviate from the mean, but still output more plausable trajectories). The math in section 2 is unclear: i) gg and gθg_{\theta} are used interchangeably, ii) ∣∣gθ−f(K)∣∣|| g_{\theta} - f^{(K)}|| is wrong since gθg_{\theta} should be a function of an input, iii) Eq (3) is a function of θ\theta that is not present on the right hand side, and, compared to Eq (4) does not state what is expectation computed over.

[1] Lambert, Nathan, et al. "Learning accurate long-term dynamics for model-based reinforcement learning." 2021 60th IEEE Conference on decision and control (CDC). IEEE, 2021.

Questions for authors

Can you clarify the difference compared to [1], and use the method as a baseline if there are methodological differences? Can you address my concerns 1-3?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.