acceptodds
Under review as a conference paper at ICLR 2027

TS-Dreamer: Aligning Representations in World Models at Different Temporal Scales

Abstract

World models let agents learn from imagined trajectories, so their latent states must preserve information for prediction and control. Aligning states with image embeddings provides supervision without pixel reconstruction. Single-step alignment directly supervises correspondence with the current image, but not explicitly over multiple observations, where motion and other temporal properties can become apparent. Different observation intervals can reveal longer-term patterns. We ask whether alignment at different temporal scales can provide complementary supervision. TS-Dreamer maps latent states into the image embedding space and jointly aligns the two representations at individual steps and over temporal windows of different lengths. At each scale, normalized sums over matching windows are aligned using Barlow Twins, without adding trainable parameters or changing inference. TS-Dreamer leads the evaluated methods in mean and median returns on DeepMind Control Suite and remains competitive on Atari 100K. In FlappyBird, adding alignment windows up to 32 decisions yields the two-scale score at fixed capacity and training budget. Representation analyses show improved linear readout of physical quantities and their changes. These results highlight temporal scale as a design choice for world model supervision.

Primary area:
reinforcement learning
Keywords:
World Models, Representation Learning, Temporal Representation Learning, Model-Based Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xot7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes TS-Dreamer, which extends the single-step representation alignment in R2-Dreamer to multiple temporal scales. Specifically, the method aggregates image embeddings and recurrent state projections over temporal windows of different lengths using normalized sums, and applies the Barlow Twins objective to align the aggregated representations at each scale. The alignment losses across different temporal scales jointly supervise the same world model without introducing additional trainable parameters or modifying policy inference.

Critical strengths

1.Broad evaluation across control benchmarks. The paper evaluates TS-Dreamer on both DeepMind Control Suite (DMC) and Atari 100K, covering continuous and discrete action control tasks.

2.Additional analyses of learned representations. Beyond standard return-based evaluations, the authors investigate different temporal scale configurations, compare alternative alignment objectives, and conduct physical-state readout and ordered-motion recognition experiments. These analyses provide useful insights into how temporal-scale supervision affects the representations learned by the world model.

Critical weaknesses
  1. Limited methodological novelty and insufficient evidence for the distinctive role of temporal-scale alignment.

    The proposed method primarily extends the Barlow Twins alignment objective in R2-Dreamer by aggregating representations within temporal windows. This appears to be a relatively incremental modification.

    More importantly, the normalized summation used to aggregate image embeddings is permutation-invariant within each window. Reordering observations within a window does not change the aggregated image representation. Although recurrent states depend on the ordered observation history, the additional alignment target itself cannot directly distinguish different temporal orderings.

    The current experiments do not sufficiently establish whether the improvements arise from complementary information across temporal scales or simply from the regularization effect introduced by temporal aggregation. Stronger evidence is needed to demonstrate the distinctive contribution of the proposed temporal-scale alignment mechanism.

  2. Limited performance improvements.

    The performance improvements over R2-Dreamer, the most directly relevant baseline, are relatively modest on the main benchmarks. On DMC, the mean return increases only from 630.91 to 636.68. On Atari 100K, the median human-normalized score improves slightly from 17.27 to 18.04.

    Furthermore, the main benchmark experiments use only three training seeds per task. Given the relatively small performance differences, the current evidence is insufficient to establish whether TS-Dreamer consistently and reliably outperforms R2-Dreamer across tasks and random seeds.

Questions for authors

Since image embeddings are aggregated through normalized summation within temporal windows, the resulting alignment targets are invariant to the temporal ordering of observations. What direct evidence demonstrates that the additional alignment objective improves the world model's ability to capture temporal-order information? How can the authors distinguish between learning richer temporal information and merely making temporal information already encoded in the recurrent states easier to extract through downstream classifiers?

7 comments
AuthorsOpenReview ↗

Response to Reviewer xot7

Thank you for recognizing our evaluation and representation analyses, and for your constructive feedback, which we have incorporated into the revised manuscript.

Limited methodological novelty ... a relatively incremental modification.

While the aggregation operation itself is simple, this conciseness is deliberate: our main contribution is a new temporal-scale perspective on world model supervision. Introducing more complex aggregation modules would confound their extra capacity with the effect of temporal scale itself.

... normalized summation ... is permutation-invariant within each window ...

Although the summed image target is order-invariant, the full model retains single-step alignment at k=1k=1 while the causal RSSM processes observations in order. For A→BA\to B, the k=1k=1 term pairs p1↔eAp_1\leftrightarrow e_A and p2↔eBp_2\leftrightarrow e_B, whereas for B→AB\to A the targets reverse. Building on this stepwise alignment, windows at k>1k>1 introduce cross-time covariance terms that encourage states to capture multi-step changes and motion trends.

... complementary information across temporal scales or simply ... regularization effect ...

We agree that auxiliary objectives can act as regularizers [1, Ch. 7]. Window aggregation is used only in the alignment loss, while the policy acts on the full state sts_t. Beyond generic regularization, our objective introduces explicit cross-time covariance terms in Eq. 3, such as Cov⁡(pt,et−1)\operatorname{Cov}(p_t,e_{t-1}) and Cov⁡(pt−1,et)\operatorname{Cov}(p_{t-1},e_t) at k=2k=2.

Empirically, linear readout of Walker physical quantities improves across all window lengths (Table 7). For full states, the largest gain is in the two-step change component D2D_2 (ΔR2=0.0191\Delta R^2=0.0191), about 2.3 times the level-component gain L2L_2 (0.00840.0084; L452 and Table 8), showing improved linear accessibility of temporal changes beyond window averages alone.

... richer temporal information ... or merely making it easier to extract?

This is an insightful question, and our analyses show both easier linear extraction and stronger multi-step trajectory retention. Making temporal features linearly accessible is itself a core criterion of representation quality [2], as online actors and critics must extract state information within finite update budgets (Section 4.5). Complementing this, our new visualizations and probing tests directly examine open-loop imagination, memory recovery, and readout-training efficiency:

  • New imagination and memory visualizations in Figure 8 and Appendix C.7: Across 10 held-out DMC and Atari cases, we evaluate open-loop imagination without future images and historical trajectory recovery from one current state (up to 32 decisions). In both settings, broader-scale TS follows forward dynamics and motion history more closely than R2 and Kmax⁡=2K_{\max}=2.
  • Ordered-motion classification in Table 9: At 400 updates, TS reaches 80.73% vs. 71.88% for R2, and transfers better to held-out speeds at 83.59% vs. 57.23% and altered geometry at 74.61% vs. 60.74%.

Limited performance improvements ... only three training seeds per task.

Beyond the default Kmax⁡=2K_{\max}=2, we evaluated shared scale sets Kmax⁡∈{2,4,8,16,32}K_{\max}\in\{2,4,8,16,32\} across all 20 DMC tasks and 26 Atari games (Section 4.3 and Appendix C.4):

  • Atari 100K: Kmax⁡=4K_{\max}=4 raises mean HNS from 74.14% to 84.67%, a gain of +10.53 percentage points with paired 95% bootstrap CI [3.16, 17.80], and Kmax⁡=16K_{\max}=16 yields +7.64 with 95% CI [0.13, 15.17], exceeding R2 on 17 of 26 games.
  • DMC: Effects are task-dependent, with Kmax⁡=2K_{\max}=2 achieving the highest overall mean.
  • FlappyBird with five seeds: At Kmax⁡=32K_{\max}=32, the mean score rises from 816.7 for R2 to 3621.1, or 4.4× (4.43×), and the median rises from 929.8 to 2677.4.

Our three-training-seed protocol (with 30 evaluation episodes per model) follows established precedent on Atari 100K, such as EfficientZero [3], and comprises 60 DMC and 78 Atari runs per configuration (1,104 training runs overall) with paired-bootstrap 95% CIs and five-seed FlappyBird results. That said, we agree that five seeds per task is even stronger; we will run two additional seeds as far as compute permits, though completing all 46 tasks may extend beyond the discussion window.

References

[1] Goodfellow et al. Deep Learning. MIT Press, 2016, Ch. 7.

[2] Alain and Bengio. Understanding intermediate layers using linear classifier probes. arXiv:1610.01644, 2016.

[3] Ye et al. Mastering Atari Games with Limited Data. NeurIPS, 2021.


We hope these clarifications, additional results, and manuscript revisions resolve your concerns, and look forward to further discussion.

AuthorsOpenReview ↗

Following up on temporal-order retention and new analyses

Dear Reviewer xot7,

Before the discussion period concludes, we wanted to check whether our response and new analyses address your questions on temporal-order capture and representation retention—particularly the open-loop imagination and memory-recovery visualizations from frozen states (Figure 8 and Appendix C.7), alongside the cross-time covariance analysis (Eq. 3) and shared-scale results.

Please let us know if any concerns remain so we can address them. If our updates resolve your concerns, we would be grateful if you could reassess our paper. Thank you very much for your time.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Reviewer_NtNVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TS-Dreamer augments R2-Dreamer with a multi-scale Barlow Twins loss that aligns normalized sums of image embeddings and recurrent-state projections across temporal windows. It introduces no extra parameters or inference overhead and is evaluated on DMC, Atari 100K, FlappyBird, and representation-probing tasks.

Critical strengths
  1. The multi-scale alignment loss is simple, adds no trainable parameters or runtime overhead at inference, and is compared fairly against R2-Dreamer under identical architectures and interaction budgets.

  2. The empirical suite covers 20 DMC tasks and 26 Atari 100K games (3 seeds each) plus 5-seed FlappyBird runs, and the physical-state readout and ordered-motion probing experiments cleanly test how quickly temporal information can be extracted under a fixed classifier-training budget.

Critical weaknesses
  1. On standard benchmarks, the performance gains over R2-Dreamer (using the default scale set {1,2}\{1,2\}) are marginal and inconsistent across aggregate metrics. In Table 4, DMC mean improves slightly from 630.91 to 636.68 (median 732.66 to 766.80), while on Atari 100K the median edges up from 17.27 to 18.04 but the mean drops from 74.14 to 71.41. Without paired confidence intervals on the per-task differences between TS-Dreamer and R2-Dreamer, it is difficult to distinguish these shifts from seed noise.

  2. The main evidence for longer temporal scales relies on post-hoc scale selection and incomplete variance reporting. In Figure 5, the ~3.8× FlappyBird improvement is measured against the 2-scale TS-Dreamer variant and plots 5-seed means without error bars or per-seed spread. Furthermore, Appendix C.4 (Table 5) picks the best-performing scale out of four candidates directly from evaluation returns, leaving it unclear how a practitioner should choose temporal scales without tuning on test environments. Finally, the ordered-motion probing advantage is purely a sample-efficiency effect during readout training, as both methods converge to the exact same 94.60% accuracy after 800 updates (Table 9).

Questions for authors
  1. Can you report paired bootstrap confidence intervals for the TS-Dreamer minus R2-Dreamer differences on DMC and Atari 100K, as well as the per-seed returns (or error bars) for all curves in Figure 5?

  2. In Table 5 (Appendix C.4), were the reported temporal scales chosen based on the test evaluation returns? Please provide the full table across all four candidate scales for every task, and clarify whether a single fixed long-horizon scale works consistently without per-task tuning.

3 comments
AuthorsOpenReview ↗

Response to Reviewer NtNV

Thank you for reading our paper carefully. We have supplemented the statistical points you raised and incorporated them into the revised manuscript.

Paired bootstrap confidence intervals ... per-seed returns (or error bars) for Figure 5?

Paired bootstrap 95% CIs for TS-Dreamer minus R2-Dreamer [1], default Kmax⁡=2K_{\max}=2:

  • DMC: mean +5.77 [−21.72, 32.26]; median +34.14 [−88.44, 102.49].
  • Atari human-normalized score (HNS): mean −2.73 [−18.82, 13.10]; median +0.78 [−4.76, 5.63], in percentage points.

We also evaluated the shared scale sets (Kmax⁡∈{2,4,8,16,32}K_{\max}\in\{2,4,8,16,32\}) across all 20 DMC tasks and 26 Atari games (Section 4.3 and Appendix C.4). On Atari, Kmax⁡=4K_{\max}=4 raises mean HNS from 74.14% to 84.67%, with a paired mean gain of +10.53 percentage points [3.16, 17.80] (95% CI). Kmax⁡=16K_{\max}=16 also yields a positive paired mean gain (+7.64 [0.13, 15.17]) and exceeds R2 on 17 of 26 games. On DMC, Kmax⁡=2K_{\max}=2 achieves the highest overall mean.

Figure 5 (FlappyBird): We have added 95% bootstrap CIs to Figure 5. The five-seed means, medians, and per-seed scores (mean pipes) are:

Kmax⁡K_{\max}Mean [95% CI]MedianPer-seed scores (seeds 41, 42, 43, 61, 62)
1 (R2)816.7 [334.5, 1313.0]929.8222.8, 1780.3, 929.8, 149.5, 1001.2
2958.2 [369.3, 1794.1]854.7874.3, 45.7, 2540.0, 854.7, 476.4
41160.8 [546.6, 1774.9]1140.11805.1, 2062.2, 426.0, 370.4, 1140.1
8699.4 [266.6, 1270.9]399.71821.1, 385.1, 81.5, 399.7, 809.5
162083.6 [820.9, 3703.2]1372.91372.9, 5102.3, 733.7, 2577.3, 632.0
323621.1 [1007.8, 7458.7]2677.411113.9, 2964.0, 362.4, 2677.4, 987.9

The 3.8× is the observed mean ratio of Kmax⁡=32K_{\max}=32 to Kmax⁡=2K_{\max}=2 (4.4× relative to R2); the median rises from 854.7 to 2677.4. Four of five paired seeds improve from Kmax⁡=2K_{\max}=2 to 32; excluding seed 41, which gives the highest Kmax⁡=32K_{\max}=32 score, from both settings still leaves means of 1747.9 vs. 979.2.

Table 5: were scales chosen by evaluation returns?

Yes, the original Table 5 reported the highest mean among Kmax⁡∈{4,8,16,32}K_{\max}\in\{4,8,16,32\} by evaluation return to illustrate multi-scale potential. We have replaced Table 5 in Appendix C.4 with the complete six-task scale comparison below (DMC/Atari single-step R2 controls were independently retrained). Notably, without per-task selection, a single shared Kmax⁡=16K_{\max}=16 outperforms R2 on 5 of the 6 tasks, and Kmax⁡=32K_{\max}=32 outperforms R2 across all three DMC tasks and FlappyBird.

TaskR2 (1)Kmax⁡=2K_{\max}=2481632
Walker Walk778.2886.9867.8945.2867.6903.5
Cheetah Run517.7532.0638.6638.4427.4605.6
Finger Turn Hard290.1309.7296.6342.2312.9343.4
Seaquest389.1313.1386.4293.3390.7347.8
Pong−4.5−9.1−9.3−6.8−1.8−8.2
FlappyBird816.7958.21160.8699.42083.63621.1

... a single fixed long-horizon scale works consistently without per-task tuning.

All benchmark sweeps apply a single shared scale set across tasks without per-task tuning, and the preferred scale reflects the temporal structure of each domain: in high-frequency continuous control (DMC), supervising local two-step changes (Kmax⁡=2K_{\max}=2) suffices to capture immediate posture and velocity dynamics, whereas discrete-action games (Atari and FlappyBird) involve longer-horizon events and benefit from broader shared scale sets (Kmax⁡=4K_{\max}=4 or 1616 on Atari, 3232 on FlappyBird). Showing that temporal scale matters with a fixed backbone is a necessary first step before studying automatic scale selection in future work.

... the exact same 94.60% accuracy after 800 updates.

While both models reach 94.60% given enough updates, readout-training efficiency within finite update budgets is relevant to online MBRL [2], where actors and critics must extract state information from limited online updates (Section 4.5). More importantly, at the 400-update budget, TS not only reaches 80.73% vs. 71.88% on the original condition, but also transfers better to held-out speeds (83.59% vs. 57.23%) and altered geometry (74.61% vs. 60.74%) (Table 9).

In the revised manuscript, Figure 8 and Appendix C.7 further visualize decoded trajectories from frozen states: TS with larger scales imagines multi-step future trajectories without future images and recovers past trajectories from one current state more accurately than R2.

References

[1] Morihira et al. R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation. ICLR, 2026.

[2] Alain and Bengio. Understanding intermediate layers using linear classifier probes. arXiv:1610.01644, 2016.


We hope these additional results and revisions resolve your concerns, and look forward to further discussion.

AuthorsOpenReview ↗

Following up on the requested confidence intervals and Table 5

Dear Reviewer NtNV,

With the discussion phase drawing to a close, we wanted to follow up on the additions in our response addressing your two questions: the paired-bootstrap 95% CIs, the per-seed returns and error bars for Figure 5, the complete six-task Table 5 across all candidate scales, and the aggregate results under shared fixed scales.

We would be glad to clarify any remaining points while time permits. If these updates address your concerns, we would be grateful if you would consider updating your evaluation. Thank you for your time.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Reviewer_NTB2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper proposes TS-Dreamer, an interesting extension of R2-Dreamer to temporal windows. While the extension makes sense, I had some concerns if simpler baselines could do just as well, and the the order-invariant (within the window) encoding is desirable for learning control or dynamics based representations.

Critical strengths
  1. Well written: mostly clear and easy to follow (some of the math explanations got a little complex, but fixable, see "small weaknesses")
  2. Important problem
  3. Nice approach building off prior work without a large additional compute cost
  4. Nice comparisons against prior methods like R2-Dreamer and DreamerV3
Critical weaknesses

Large weaknesses:

  1. Baseline: many of the original DMC-based papers like DQN/DDQN simply expanded the observation space to concat (say) 4 images (depending on the task). While the method here is interesting, how would it compare against such (simpler) baseline. Especially since Line 404 mentions only “All analyzed TS-Dreamer models use K = {1, 2}”.
  2. Unclear the performance benefit over baselines on DMC and Atari. Figures 3 and 4 show significant overlap in the baselines presented, which by itself is not a weakness of a paper, but I would consider this a weakness given the additional complexity introduced by the method, which arguably is only warranted where it can offer a clear benefit.

Small weaknesses:

  • Line 012: “Aligning states” seems a little vague: does this mean the state’s definition, or something else?
  • Line 048: “groups of observations” → “{sequential, successive, or contiguous} observations” ? (presumedly not just any group)
  • Line 066: I found this paragraph difficult to understand. For example, “single-step alignment” still seem vague to me, and I didn’t understand what “measured in agent decisions” means exactly. I also didn’t (at first) understand why there appears to be multiple scales with “At each selected scale”.
  • Line 152: h_t seems to be a function of (stochastic) z_{t-1}, even though it was described as “deterministic memory” on line 148. Which is correct, line 152 or line 148?
  • Line 156: is the output of “Reward prediction” and “Continuation prediction” used anywhere? (they don’t have outputs like the others in (1))
  • Line 218: k^{-1} or k^{-2} (given the double sum over a covariance to compute another covariance) ?
  • Line 366: “we add windows of 4, 8, 16, and 32 decisions” is this k? (if so, prepend a “k=” perhaps?)
  • Figure 5: 95% uncertainty needed like with Figures 3 and 4?
Questions for authors
  1. Line 212: “Reordering image embeddings within a window leaves their sum unchanged” is this counter to the paper’s intent toward dynamic awareness? I assume this gives rise to features stable over time, but is that a useful representation for control of dynamical systems (e.g., would an object going left-to-right encode similar to an object going right-to-left?)
4 comments
AuthorsOpenReview ↗

Response to Reviewer NTB2

Thank you for the careful review and constructive suggestions; your detailed line-by-line comments have greatly helped us improve the paper's readability and clarify our key points.

Baseline: ... DQN/DDQN ... concat (say) 4 images ... how would it compare against such (simpler) baseline ... Line 404 mentions only K = {1, 2}.

To isolate the alignment objective, we deliberately kept Dreamer's input and architecture unchanged. Frame stacking is an informative control; this experiment is running and we will post the results. Note that L404 (now L419) refers only to the Walker and ordered-motion analyses; we have now completed a full evaluation of larger shared scale sets Kmax⁡∈{4,8,16,32}K_{\max}\in\{4,8,16,32\} across all 20 DMC tasks and 26 Atari games in Section 4.3 and Appendix C.4.

Unclear the performance benefit ... given the additional complexity introduced by the method.

The default configuration's gains are modest, while broader temporal scales on Atari yield larger improvements. Our aim in exploring these scales is to bring a temporal-scale perspective to world model supervision: whereas H-JEPA [1, 2] models temporal scales through hierarchical architectures, we vary the alignment window with the backbone fixed, adding no parameters or inference overhead. With this fixed backbone, extending the longest window to Kmax⁡=4K_{\max}=4 across all 26 Atari games raises the mean human-normalized score (HNS) from 74.14% to 84.67%, a gain of +10.53 percentage points, while on DMC the default Kmax⁡=2K_{\max}=2 achieves the highest mean.

Small weaknesses.

Thank you for these helpful pointers; we have revised the text to make it more self-contained:

  • L012: Following PlaNet [3], the state is st=[vec⁡(zt),ht]s_t=[\operatorname{vec}(z_t),h_t]. We align normalized sums of state projections and image embeddings over matching windows in the Abstract and Section 1.

  • L048: We replaced "groups of observations" with "contiguous observations".

  • L066: Barlow Twins aligns state projections with image embeddings, as in R2-Dreamer [4]. Single-step alignment uses same-step pairs. A temporal scale kk is a contiguous window length in agent decisions, not raw frames: at step 4, k=1,2,4k=1,2,4 use windows [4][4], [3,4][3,4], and [1,2,3,4][1,2,3,4], respectively. Our evaluated scale sets include powers of two up to Kmax⁡K_{\max} in Sections 1 and 3.2.

  • L148/L152: The transition is deterministic given (ht−1,zt−1,at−1)(h_{t-1},z_{t-1},a_{t-1}), although hth_t is random through the sampled zt−1z_{t-1}, as clarified below Eq. 1.

  • L156: These heads are trained via Eq. 7 and yield r^t\hat r_t and c^t\hat c_t for the λ\lambda-returns in Eq. 11. We added r^t,c^t\hat r_t,\hat c_t to Eq. 1 and clarified their role.

  • L218: Each summary is scaled by k−1/2k^{-1/2}, so the raw covariance carries 1/k1/k, which cancels from the standardized correlation, as noted below Eq. 3.

  • L366 & Fig. 5: We wrote k=4,8,16,32k=4,8,16,32 explicitly and added 95% bootstrap CIs to Figure 5.

Line 212: "Reordering image embeddings within a window leaves their sum unchanged" ... would an object going left-to-right encode similar to an object going right-to-left?

The summed image target is order-invariant, but the retained k=1k=1 term still anchors each state projection to its same-step image embedding. For A→BA\to B, it aligns p1↔eAp_1\leftrightarrow e_A and p2↔eBp_2\leftrightarrow e_B, whereas for B→AB\to A the image targets are reversed, while the causal RSSM processes observations in order. The policy acts on the full state sts_t directly, while alignment supervises only its projection ptp_t. Building on this stepwise alignment, windows at k>1k>1 introduce cross-time covariance terms in Eq. 3 that capture multi-step changes and motion trends, differing from future-prediction objectives [3, 5, 6] in Section 3.2.

Empirically, evaluating states on the ordered-motion task (identical frames presented in different orders) confirms that temporal order remains accessible: after 400 classifier updates, TS-Dreamer reaches 80.73% vs. 71.88% for R2 [4], and 82.94% vs. 70.44% under reversed direction (Table 9), dropping to near chance when encoded states are temporally shuffled.

References

[1] Dawid and LeCun. Introduction to latent variable energy-based models: a path toward autonomous machine intelligence. Journal of Statistical Mechanics: Theory and Experiment, 2024.

[2] Zhang et al. H-JEPA: End-to-End Learning of Hierarchical World Models for Visual Planning. 2026.

[3] Hafner et al. Learning Latent Dynamics for Planning from Pixels. ICML, 2019.

[4] Morihira et al. R2-Dreamer: Redundancy-Reduced World Models without Decoders or Augmentation. ICLR, 2026.

[5] Bredis et al. Next Embedding Prediction Makes World Models Stronger. ICLR Workshop on World Models, 2026.

[6] Burchi and Timofte. Learning Transformer-based World Models with Contrastive Predictive Coding. ICLR, 2025.


We hope these revisions address your concerns and look forward to further discussion.

AuthorsOpenReview ↗

Follow-up to Reviewer NTB2: Frame-Stacking Baseline Results

Thank you again for suggesting frame stacking as a simpler baseline. We found this comparison particularly useful and have evaluated stacked-input baselines on six representative tasks under the same experimental protocol (3 training seeds and 30 evaluation episodes per model).

1. 4-frame DQN / DDQN. On Assault (100k decisions), 4-frame DQN (408.10) and DDQN (409.50) both perform below the world model baselines: single-frame R2-Dreamer (486.72) and TS-Dreamer with Kmax⁡=2K_{\max}=2 (570.91).

2. 2-frame R2-Dreamer. To provide a more direct comparison with the two-step temporal window of Kmax⁡=2K_{\max}=2, we concatenate two consecutive RGB frames channel-wise, [ot−1,ot][o_{t-1}, o_t], as input to the R2-Dreamer image encoder, repeating the initial frame o0o_0 at the beginning of each episode. All other components remain unchanged, including the single-step Barlow Twins alignment and recurrent architecture.

The results across six tasks show that the two approaches exhibit complementary strengths. 2-frame R2-Dreamer achieves higher scores on Cheetah Run, Finger Spin, Breakout, and Seaquest, while TS-Dreamer (Kmax⁡=2K_{\max}=2) performs better on Walker Run (422.60 vs. 349.96) and Assault (570.91 vs. 514.03).

TaskR2-Dreamer (1-frame)TS-Dreamer (Kmax⁡=2K_{\max}=2, 1-frame)R2-Dreamer (2-frame stack)
Walker Run393.84422.60349.96
Cheetah Run643.06532.00653.45
Finger Spin766.74767.06826.74
Assault486.72570.91514.03
Breakout6.888.038.76
Seaquest401.33313.11370.44

Takeaways.

These comparisons highlight two complementary ways of incorporating temporal information into world model learning.

Frame stacking introduces recent visual history directly at the input level: the image encoder receives two consecutive frames, so each target embedding ete_t already contains information from both observations.

In contrast, TS-Dreamer introduces temporal information at the supervision level. It retains single-frame inputs, the original encoder architecture, and unchanged policy inference, while extending representation alignment across temporal windows. This formulation also naturally supports longer temporal windows, including Kmax⁡=4,16,K_{\max}=4, 16, and 3232, as evaluated in our full benchmark sweeps.

Update (Final 3-seed results). We have now completed all three training seeds for the additional 4-frame DQN/DDQN controls and the 2-frame R2-Dreamer sweep across all 20 DMC tasks and 26 Atari games. Your suggestion proved very insightful, as 2-frame stacking is indeed a strong baseline that confirms the value of short-term temporal context:

  • 4-frame DQN / DDQN: Across all three tested Atari games (Assault, Breakout, and Seaquest), both 4-frame DQN and DDQN remain well below R2-Dreamer and TS-Dreamer (Kmax⁡=2K_{\max}=2).

  • DMC (20 tasks): At Kmax⁡=2K_{\max}=2, 2-frame stacking helps on tasks such as Cheetah Run and Finger Spin and matches the overall mean return of TS-Dreamer, while TS-Dreamer achieves a substantially higher median return (766.80 vs. 681.16) by avoiding drops on locomotion tasks such as Hopper Stand and Quadruped Walk.

  • Atari 100K (26 games): At Kmax⁡=2K_{\max}=2, the two approaches remain complementary across games: 2-frame stacking helps on games such as Alien and Freeway, whereas TS-Dreamer scores higher on games such as Amidar, Assault, Bank Heist, HERO, and Ms. Pac-Man. Furthermore, whereas stacking longer histories requires expanding the encoder input channels, TS-Dreamer extends directly to broader windows (Kmax⁡=4,16,32K_{\max}=4, 16, 32, reaching 84.67% mean HNS at Kmax⁡=4K_{\max}=4) without changing the input, encoder architecture, or policy inference.

AuthorsOpenReview ↗

Update on extended frame-stacking comparisons

Dear Reviewer NTB2,

We have further extended the 2-frame R2-Dreamer comparison you suggested in our comment above.

As the discussion window closes soon, please let us know if any questions remain regarding these baselines, our line-by-line revisions, or within-window ordering; if your concerns are resolved, we would appreciate your updated assessment. Thank you again for your helpful suggestions.

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions, Additional Experiments, and Core Perspective

We sincerely thank the Area Chair and all reviewers for their thorough reviews and constructive feedback. We are encouraged that the reviewers recognized the clarity of our presentation (NTB2, NtNV, xot7), the computational efficiency of our parameter-free alignment objective (NTB2, NtNV), and the breadth of our evaluations and representation analyses (NtNV, xot7).

We have revised the manuscript (with changes highlighted in blue) and provided additional experiments and clarifications in our responses. The main updates are:

  • Full-benchmark scale sweeps and paired-bootstrap 95% CIs across all 20 DMC tasks and 26 Atari 100K games without per-task scale selection, together with the complete six-task scale comparison and 95% bootstrap CIs for the FlappyBird scale sweep (Abstract, Section 4.3, Figure 5, Table 5, and Appendix C.4; Reviewers NtNV, xot7, NTB2).
  • Multi-step imagination and memory visualizations using frozen world models across 10 DMC and Atari cases, covering 24–32-decision open-loop imagination and recovery of the preceding 32-decision path from one current state (Section 4.5, Figure 8, and Appendix C.7; Reviewers xot7, NtNV, NTB2).
  • Frame-stacking baseline comparisons contrasting frame stacking at the input level with temporal alignment at the supervision level (follow-up comment to Reviewer NTB2; Reviewer NTB2).
  • Methodological and exposition clarifications on world model notation, single-step alignment (k=1k=1) with causal recurrence, cross-time covariance terms (Eq. 3), and readout-training efficiency within finite update budgets (Sections 1, 3.1, 3.2, and 4.5; Reviewers NTB2, xot7, NtNV).

Core Perspective and Key Findings

World model supervision is typically framed around what a latent state should match (pixels, prototypes, or image embeddings). TS-Dreamer instead asks over what temporal span image and state representations should be aligned, using a single parameter-free window-aggregation objective without added predictors or hierarchical modules.

  • Temporal information in recurrent states: Readout gains appear consistently in the recurrent states rather than in the per-image target embeddings, with larger improvements for two-step changes than for levels (Tables 7 and 8).
  • Multi-step dynamics and history recovery: Without adding multi-step prediction or memory losses, broader alignment scales improve 24–32-decision open-loop imagination and single-state recovery of past trajectories in our held-out analyses (Figure 8 and Appendix C.7).
  • Temporal scale as an open design dimension: At fixed model capacity and training budget, varying only the alignment windows lifts the 26-game Atari 100K mean with a shared scale set and reaches 4.4× R2-Dreamer in FlappyBird, while effective scales differ across domains.

We view temporal scale as an underexplored supervision dimension for world models and hope this study provides a clear starting point for further work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.