- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Visual world models commonly rely on image-pretrained encoders to extract features that serve as prediction targets and as the basis for planning costs. However, these features are extracted frame by frame and carry no temporal context — the burden of modeling how frames evolve is left entirely to the downstream predictor. TRIP addresses this by first adapting the image encoder to downstream videos through self-supervised learning with causal temporal attention, so that each frame's representation is informed by the past, and then reusing this temporal encoder for two purposes. For prediction, the encoder's high-dimensional features are distilled into compact low-dimensional targets: a diffusion model predicts future states in this feature space, and a separate generator decodes them into RGB video. For planning, TRIP introduces a training-free scoring scheme: the goal image is appended to each imagined trajectory, and the encoder checks whether the goal's representation stays consistent with its standalone encoding. Experiments validate both uses: video prediction on SSv2 and LIBERO surpasses baselines such as Diffusion Forcing, with slower degradation on longer sequences; closed-loop planning on four Procgen games outperforms all compared methods (65.6% vs. 59.8% for the strongest baseline); and notably, the scoring encoder trained on Procgen transfers to the VP2 manipulation benchmark with no adaptation and still improves planning, suggesting the scoring scheme generalizes across domains.
Critical strengths
A training-free planning scorer, cleanly validated. Appending the goal to a candidate rollout and checking the consistency of its encoding requires no reward training at all. Under ground-truth simulator trajectories, which remove world-model error entirely, TRIP reaches 84.2% while all frame-wise and consistency baselines stay ≤51% — the gain comes from the scoring design itself. The scorer also transfers zero-shot from Procgen to VP2, suggesting a genuinely reusable module.
A complete evidence chain for temporalized representations. Prediction improves consistently on two benchmarks of different nature, degradation slows on longer sequences, and the encoder ablation (image-only / frame-wise / shuffled / ordered) plus compression comparisons corroborate each other .
Critical weaknesses
The "unified framework" is not actually realized, and the planning protocol is opaque. Planning never uses TRIP's own prediction pipeline — rollouts come from separately trained RGB diffusion world models, and TRIP only scores them. Moreover, Table 2 does not state whether the DF baseline shares the same world model with only the cost changed; VP2 baselines are published numbers while TRIP is self-run. Planning is one of the two headline claims, so these gaps directly weaken the conclusions.
Attribution is unclear: how much is "temporal order" vs. "domain adaptation", and why the scorer works, is never established. The shuffled ablation shows most of the gain comes from video-domain fine-tuning rather than ordered temporal context (220.8 vs. 212.6 vs. 255.5 frame-wise); the consistency cost has only an intuitive justification and prediction is never compared with feature-space forecasting methods at matched settings.
Questions for authors
Can you report the correlation between the consistency cost and true remaining distance/steps to the goal? Are there failure cases — e.g., severely distorted rollouts or out-of-distribution goals — where the cost becomes unreliable?
What is the practical planning overhead? Every candidate requires generating an RGB rollout and encoding the whole sequence; how much slower is this than frame-wise scoring?
8 comments
2. Moreover, Table 2 does not state whether the DF baseline shares the same world model with only the cost changed.
Thank you for raising this question. For each Procgen game in Table 2, TRIP and the Diffusion Forcing baseline share the exact same RGB Diffusion Forcing world-model checkpoint, evaluation scenes, and MPC settings, with only the scoring cost changed. The temporal encoder used for the TRIP cost is trained on the same Procgen training data as the Diffusion Forcing, so both components see identical training data. We will state this explicitly in Section 4.1 and the Table 2 caption in the revision.
3. VP2 baselines are published numbers while TRIP is self-run.
We thank the reviewer for emphasizing a fair comparison on VP2. We re-evaluated the official iVideoGPT cost and our method under identical settings (same world-model checkpoint, initial states, goals, seeds, and MPC budget): iVideoGPT (Reproduced): 72.96 (vs. 73.22 published).
1. The "unified framework" is not actually realized.
Thank you for pointing out the need to clarify how our generation and planning pipelines relate. In TRIP, the common foundation is the self-supervised temporal encoder, which learns representations used across both tasks. Rather than combining both components into a single end-to-end architecture, we evaluate two distinct uses of these representations: Section 3.2 uses distilled compact features for video prediction, while Section 3.3 uses the frozen encoder as a plug-and-play consistency scorer on top of existing RGB world models (Diffusion Forcing and iVideoGPT). We will make this distinction clearer in the revised Introduction.