- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PreShort, a unified online model-based reinforcement learning framework that integrates frozen pretrained visual representations with efficient stochastic world modeling and imagination-based actor-critic learning. To make this integration practical, the proposed Differential Shortcut Dynamics combines differential attention for temporal context modeling with few-step shortcut generation of successor states. Experiments on Atari 100K, the DeepMind Control Suite, and Crafter demonstrate competitive performance, supported by component ablations, alternative visual backbones, rollout-accuracy evaluation, and efficiency analysis.
Critical strengths
- The paper is well written and easy to understand.
- Meaningful framework integration.PreShort combines frozen pretrained visual representations, efficient stochastic world modeling, and imagination-based actor-critic learning in a unified online framework, reducing the need to jointly learn perception and dynamics from limited interactions.
Critical weaknesses
- Limited methodological novelty. The framework mainly combines existing components, including frozen pretrained visual encoders, differential attention, shortcut generation, and standard imagination-based actor-critic learning. The contribution is therefore mainly system integration and with moderate methodological novelty.
- The ablations do not clearly isolate the contribution of each component. The “No Shortcut” variant simultaneously removes stochastic initialization, iterative refinement, token attention, and shortcut-consistency training. It is therefore unclear whether the improvement comes from stochastic modeling, the shortcut objective, additional computation, or the architecture itself.
Questions for authors
- Figure 3 and Appendix G.3 — attribution of the shortcut improvement. The “No Shortcut” variant replaces four-step stochastic generation with a single-pass deterministic MLP, simultaneously removing noise initialization, iterative refinement, token attention, and shortcut consistency. Can the authors provide a comparison against ordinary conditional flow matching with the same four refinement steps, temporal encoder, and comparable parameter budget? This would clarify whether the performance gap specifically supports shortcut consistency or primarily reflects the benefit of stochastic iterative generation.
- Figure 4(a) — evidence for reduced rollout error accumulation. Section 4.4.1 discusses prediction accuracy as the rollout horizon increases, but Figure 4(a) plots 16-step rollout PSNR against environment interactions. How is PSNR aggregated across the 16 prediction steps, and how are stochastic samples evaluated—for example, using one sample, an average over samples, or the best sample?
- Does the reported efficiency include the inference and feature-caching costs of the frozen visual encoder?
3 comments
Response to Reviewer wJcj (2/2)
Q3: Does the reported efficiency include visual encoding and feature caching?
Yes. The reported training time of 1.1 hours per Atari 100K game includes the total online training time, including Wan-VAE forward encoding of newly collected observations, feature-cache writes and reads, world model updates, imagined rollout generation, actor–critic updates, and environment interaction. PreShort uses the official Wan2.2_VAE.pth checkpoint released with Wan2.2-TI2V-5B, performs no additional visual fine-tuning, and keeps the pretrained visual encoder frozen throughout online RL.
Thank you again for the detailed and constructive feedback. We are happy to provide any further experiments or details as needed.
Follow-up on our response
Thank you again for your constructive feedback, which helped us clarify the methodological contribution of DSD, particularly how reusing the encoded history across refinement steps improves computational efficiency. We hope this explanation, together with our clarifications regarding the PSNR evaluation and efficiency measurements, has helped address your questions and concerns. If any points remain unclear or new questions arise, please let us know, and we would be very happy to provide further clarification and continue the discussion.
Response to Reviewer wJcj (1/2)
We thank the reviewer for recognizing the empirical effectiveness of PreShort, and for the constructive feedback. We address each concern below and the corresponding revisions have been incorporated into the revised manuscript and highlighted in blue.
W1: Methodological and efficiency contributions of PreShort.
We would like to clarify that PreShort’s contribution extend beyond an online world-modeling architecture designed for frozen pretrained visual representations to the design of Differential Shortcut Dynamics (DSD) for accurate and efficient dynamics prediction. PreShort also contributes a latent-space diffusion design that improves efficiency by reusing compressed context to avoid excessive reliance on historical trajectories. DSD firstly introduces differential attention into online world modeling, which is used to learn a more informative compact representation of visual and action histories. Unlike diffusion-based methods that repeatedly process historical trajectories during denoising [1], DSD reuses the compact history across refinement steps, enabling efficient successor prediction in the continuous pretrained feature space. In terms of computational efficiency, PreShort achieved the highest performance in the shortest training time.
The comparisons in Appendix D.2 further motivate this design. Incorporating frozen Wan-VAE features into categorical world models does not improve control performance in the evaluated settings. Re-encoding continuous features into categorical latent variables (as in most world-model architectures) may weaken the preservation of fine-grained, control-relevant information (as shown in Figure 2). PreShort supports general pretrained visual models through a common flattened-feature interface (Appendix D1 and D3 for other visual backbones), with DSD directly modeling transitions in their continuous spaces. These results support the importance of the downstream DSD dynamics design in translating pretrained visual information into control performance.
[1] Alonso E, Jelley A, Micheli V, et al. Diffusion for world modeling: Visual details matter in atari[J]. Advances in Neural Information Processing Systems, 2024, 37: 58757-58791.
W2/Q1: Isolating the contribution of shortcut consistency.
We have added PreShort without Lsc, which removes the shortcut consistency loss while retaining the other framework (the full learning curves are provided in Appendix D.6 of the revised manuscript).
This variant is equivalent to parameter-matched conditional flow matching with x-space prediction (same prediction space as PreShort). Removing Lsc leaves endpoint regression along the noise–data interpolation path as the dynamics prediction objective. And the equivalence follows because the endpoint regression corresponds to a time-weighted flow-matching objective, while the predictor architecture and parameter count remain unchanged. The results are summarized in the following table. Full PreShort performs better across all three benchmarks, supporting an additional benefit from shortcut consistency beyond these shared components.
Q2: Definition and aggregation of rollout PSNR.
PSNR measures the pixel fidelity of frames decoded from predicted visual features relative to the corresponding ground-truth frames and higher values indicate closer agreement with the ground truth. Figure 4(a) plots this metric against environment interactions during training and the prediction horizon remains fixed at 16 steps throughout. Thus, the figure measures how rollout fidelity improves with training, rather than how prediction error varies with rollout horizon. We have revised the main text and figure caption to clarify the evaluation procedure and the interpretation of Figure 4(a).
All variants perform autoregressive prediction from the same observation histories and recorded action sequences, and their predicted features are decoded using the frozen Wan-VAE decoder. At each checkpoint, we evaluate 1,024 imagined trajectories using the same distribution of contexts and actions across variants. We compute PSNR separately for each predicted frame and its corresponding ground-truth frame, average over the 16 prediction steps and 1,024 trajectories, and then average the results across five random seeds. Every generated trajectory contributes to the metric; no best-sample selection is used. PreShort achieves the highest average rollout PSNR under this protocol (indicating more accurate imagined trajectories than those generated by the other methods).
Respond to Question 3 continued on the next comment.