- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes TS-Dreamer, which extends the single-step representation alignment in R2-Dreamer to multiple temporal scales. Specifically, the method aggregates image embeddings and recurrent state projections over temporal windows of different lengths using normalized sums, and applies the Barlow Twins objective to align the aggregated representations at each scale. The alignment losses across different temporal scales jointly supervise the same world model without introducing additional trainable parameters or modifying policy inference.
Critical strengths
1.Broad evaluation across control benchmarks. The paper evaluates TS-Dreamer on both DeepMind Control Suite (DMC) and Atari 100K, covering continuous and discrete action control tasks.
2.Additional analyses of learned representations. Beyond standard return-based evaluations, the authors investigate different temporal scale configurations, compare alternative alignment objectives, and conduct physical-state readout and ordered-motion recognition experiments. These analyses provide useful insights into how temporal-scale supervision affects the representations learned by the world model.
Critical weaknesses
-
Limited methodological novelty and insufficient evidence for the distinctive role of temporal-scale alignment.
The proposed method primarily extends the Barlow Twins alignment objective in R2-Dreamer by aggregating representations within temporal windows. This appears to be a relatively incremental modification.
More importantly, the normalized summation used to aggregate image embeddings is permutation-invariant within each window. Reordering observations within a window does not change the aggregated image representation. Although recurrent states depend on the ordered observation history, the additional alignment target itself cannot directly distinguish different temporal orderings.
The current experiments do not sufficiently establish whether the improvements arise from complementary information across temporal scales or simply from the regularization effect introduced by temporal aggregation. Stronger evidence is needed to demonstrate the distinctive contribution of the proposed temporal-scale alignment mechanism.
-
Limited performance improvements.
The performance improvements over R2-Dreamer, the most directly relevant baseline, are relatively modest on the main benchmarks. On DMC, the mean return increases only from 630.91 to 636.68. On Atari 100K, the median human-normalized score improves slightly from 17.27 to 18.04.
Furthermore, the main benchmark experiments use only three training seeds per task. Given the relatively small performance differences, the current evidence is insufficient to establish whether TS-Dreamer consistently and reliably outperforms R2-Dreamer across tasks and random seeds.
Questions for authors
Since image embeddings are aggregated through normalized summation within temporal windows, the resulting alignment targets are invariant to the temporal ordering of observations. What direct evidence demonstrates that the additional alignment objective improves the world model's ability to capture temporal-order information? How can the authors distinguish between learning richer temporal information and merely making temporal information already encoded in the recurrent states easier to extract through downstream classifiers?
7 comments
Following up on temporal-order retention and new analyses
Dear Reviewer xot7,
Before the discussion period concludes, we wanted to check whether our response and new analyses address your questions on temporal-order capture and representation retention—particularly the open-loop imagination and memory-recovery visualizations from frozen states (Figure 8 and Appendix C.7), alongside the cross-time covariance analysis (Eq. 3) and shared-scale results.
Please let us know if any concerns remain so we can address them. If our updates resolve your concerns, we would be grateful if you could reassess our paper. Thank you very much for your time.
Dear Reviewer,
Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.
Thank you for your time and cooperation.
Response to Reviewer xot7
Thank you for recognizing our evaluation and representation analyses, and for your constructive feedback, which we have incorporated into the revised manuscript.
While the aggregation operation itself is simple, this conciseness is deliberate: our main contribution is a new temporal-scale perspective on world model supervision. Introducing more complex aggregation modules would confound their extra capacity with the effect of temporal scale itself.
Although the summed image target is order-invariant, the full model retains single-step alignment at k=1 while the causal RSSM processes observations in order. For A→B, the k=1 term pairs p1↔eA and p2↔eB, whereas for B→A the targets reverse. Building on this stepwise alignment, windows at k>1 introduce cross-time covariance terms that encourage states to capture multi-step changes and motion trends.
We agree that auxiliary objectives can act as regularizers [1, Ch. 7]. Window aggregation is used only in the alignment loss, while the policy acts on the full state st. Beyond generic regularization, our objective introduces explicit cross-time covariance terms in Eq. 3, such as Cov(pt,et−1) and Cov(pt−1,et) at k=2.
Empirically, linear readout of Walker physical quantities improves across all window lengths (Table 7). For full states, the largest gain is in the two-step change component D2 (ΔR2=0.0191), about 2.3 times the level-component gain L2 (0.0084; L452 and Table 8), showing improved linear accessibility of temporal changes beyond window averages alone.
This is an insightful question, and our analyses show both easier linear extraction and stronger multi-step trajectory retention. Making temporal features linearly accessible is itself a core criterion of representation quality [2], as online actors and critics must extract state information within finite update budgets (Section 4.5). Complementing this, our new visualizations and probing tests directly examine open-loop imagination, memory recovery, and readout-training efficiency:
Beyond the default Kmax=2, we evaluated shared scale sets Kmax∈{2,4,8,16,32} across all 20 DMC tasks and 26 Atari games (Section 4.3 and Appendix C.4):
Our three-training-seed protocol (with 30 evaluation episodes per model) follows established precedent on Atari 100K, such as EfficientZero [3], and comprises 60 DMC and 78 Atari runs per configuration (1,104 training runs overall) with paired-bootstrap 95% CIs and five-seed FlappyBird results. That said, we agree that five seeds per task is even stronger; we will run two additional seeds as far as compute permits, though completing all 46 tasks may extend beyond the discussion window.
References
[1] Goodfellow et al. Deep Learning. MIT Press, 2016, Ch. 7.
[2] Alain and Bengio. Understanding intermediate layers using linear classifier probes. arXiv:1610.01644, 2016.
[3] Ye et al. Mastering Atari Games with Limited Data. NeurIPS, 2021.
We hope these clarifications, additional results, and manuscript revisions resolve your concerns, and look forward to further discussion.