acceptodds
Under review as a conference paper at ICLR 2027

TRIP: Self-Supervised Temporal Representations for Imagination and Planning

Abstract

Visual world models often use frame-wise image representations as prediction targets and to construct planning costs, without explicitly incorporating temporal context into the representations themselves. We introduce TRIP (Temporal Representations for Imagination and Planning), a framework that adapts an image-pretrained visual encoder through causal video self-supervised learning and uses the resulting representations for both future prediction and visual planning. For prediction, a compact bottleneck adapter learns to match the temporal encoder's prototype distributions, providing low-dimensional targets for a diffusion-based feature predictor whose outputs condition RGB video generation. For planning, we reuse the frozen temporal encoder to score imagined rollouts by comparing the goal's prototype distributions with and without rollout context. The resulting goal-representation consistency cost enables model predictive control without additional cost-specific training. TRIP outperforms Diffusion Forcing in video generation on SSv2 and LIBERO and achieves higher average closed-loop planning success than AdaWorld on Procgen. Its scoring encoder also transfers from Procgen to VP without further adaptation, improving planning when paired with an existing world model.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
World Models, Model Predictive Control

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uAwoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

Visual world models commonly rely on image-pretrained encoders to extract features that serve as prediction targets and as the basis for planning costs. However, these features are extracted frame by frame and carry no temporal context — the burden of modeling how frames evolve is left entirely to the downstream predictor. TRIP addresses this by first adapting the image encoder to downstream videos through self-supervised learning with causal temporal attention, so that each frame's representation is informed by the past, and then reusing this temporal encoder for two purposes. For prediction, the encoder's high-dimensional features are distilled into compact low-dimensional targets: a diffusion model predicts future states in this feature space, and a separate generator decodes them into RGB video. For planning, TRIP introduces a training-free scoring scheme: the goal image is appended to each imagined trajectory, and the encoder checks whether the goal's representation stays consistent with its standalone encoding. Experiments validate both uses: video prediction on SSv2 and LIBERO surpasses baselines such as Diffusion Forcing, with slower degradation on longer sequences; closed-loop planning on four Procgen games outperforms all compared methods (65.6% vs. 59.8% for the strongest baseline); and notably, the scoring encoder trained on Procgen transfers to the VP2 manipulation benchmark with no adaptation and still improves planning, suggesting the scoring scheme generalizes across domains.

Critical strengths

A training-free planning scorer, cleanly validated. Appending the goal to a candidate rollout and checking the consistency of its encoding requires no reward training at all. Under ground-truth simulator trajectories, which remove world-model error entirely, TRIP reaches 84.2% while all frame-wise and consistency baselines stay ≤51% — the gain comes from the scoring design itself. The scorer also transfers zero-shot from Procgen to VP2, suggesting a genuinely reusable module.

A complete evidence chain for temporalized representations. Prediction improves consistently on two benchmarks of different nature, degradation slows on longer sequences, and the encoder ablation (image-only / frame-wise / shuffled / ordered) plus compression comparisons corroborate each other .

Critical weaknesses

The "unified framework" is not actually realized, and the planning protocol is opaque. Planning never uses TRIP's own prediction pipeline — rollouts come from separately trained RGB diffusion world models, and TRIP only scores them. Moreover, Table 2 does not state whether the DF baseline shares the same world model with only the cost changed; VP2 baselines are published numbers while TRIP is self-run. Planning is one of the two headline claims, so these gaps directly weaken the conclusions.

Attribution is unclear: how much is "temporal order" vs. "domain adaptation", and why the scorer works, is never established. The shuffled ablation shows most of the gain comes from video-domain fine-tuning rather than ordered temporal context (220.8 vs. 212.6 vs. 255.5 frame-wise); the consistency cost has only an intuitive justification and prediction is never compared with feature-space forecasting methods at matched settings.

Questions for authors

Can you report the correlation between the consistency cost and true remaining distance/steps to the goal? Are there failure cases — e.g., severely distorted rollouts or out-of-distribution goals — where the cost becomes unreliable?

What is the practical planning overhead? Every candidate requires generating an RGB rollout and encoding the whole sequence; how much slower is this than frame-wise scoring?

8 comments

1. The "unified framework" is not actually realized.

Thank you for pointing out the need to clarify how our generation and planning pipelines relate. In TRIP, the common foundation is the self-supervised temporal encoder, which learns representations used across both tasks. Rather than combining both components into a single end-to-end architecture, we evaluate two distinct uses of these representations: Section 3.2 uses distilled compact features for video prediction, while Section 3.3 uses the frozen encoder as a plug-and-play consistency scorer on top of existing RGB world models (Diffusion Forcing and iVideoGPT). We will make this distinction clearer in the revised Introduction.

2. Moreover, Table 2 does not state whether the DF baseline shares the same world model with only the cost changed.

Thank you for raising this question. For each Procgen game in Table 2, TRIP and the Diffusion Forcing baseline share the exact same RGB Diffusion Forcing world-model checkpoint, evaluation scenes, and MPC settings, with only the scoring cost changed. The temporal encoder used for the TRIP cost is trained on the same Procgen training data as the Diffusion Forcing, so both components see identical training data. We will state this explicitly in Section 4.1 and the Table 2 caption in the revision.

3. VP2 baselines are published numbers while TRIP is self-run.

We thank the reviewer for emphasizing a fair comparison on VP2. We re-evaluated the official iVideoGPT cost and our method under identical settings (same world-model checkpoint, initial states, goals, seeds, and MPC budget): iVideoGPT (Reproduced): 72.96 (vs. 73.22 published).

Reviewer_bkD7OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRIP adapts DINOv2 to videos through causal temporal attention and self-supervised learning. It distills compact features for a two-stage video predictor and introduces a planning cost that compares goal-image representations with and without rollout context. Experiments report improved video prediction on SSv2 and LIBERO, higher planning success on Procgen, and transfer of the scoring encoder to VP2.

Critical strengths
  • The generation ablations provide evidence that temporal context helps beyond frame-wise adaptation. Ordered temporal features perform best, and the compression experiments distinguish reconstruction quality from usefulness for prediction. These comparisons make the representation-learning contribution more convincing.
  • The goal-representation consistency cost is an interesting idea. It uses the temporal encoder to assess candidate rollouts without additional reward-model training. The simulator-based comparison is particularly useful because it evaluates scoring quality separately from world-model prediction errors.
Critical weaknesses
  • The planning comparisons do not fully isolate the proposed scoring mechanism. Figure 6 changes temporal context, representation space, and distance measure together, leaving it unclear how much improvement comes from contextualizing the goal versus using prototype distributions and cross-entropy.
  • The connection between goal-representation consistency and goal achievement needs stronger validation. The cost measures how much rollout context changes the representation of the same goal image, but representation stability alone does not establish that a trajectory approaches the goal. The simulator results are encouraging; controlled tests with incorrect or shuffled trajectories would help show whether the score tracks goal progress.
Questions for authors
  • With the same temporal encoder and prototype head, how does frame-wise prototype cross-entropy scoring perform? This would help separate the benefit of rollout context from the choice of representation and metric.
  • Can the original iVideoGPT cost and TRIP cost be evaluated using the same checkpoint, initial states, goals, and MPC budget on VP2?
4 comments

1. The planning comparisons do not fully isolate the proposed scoring mechanism.

We thank the reviewer for catching this factor. To isolate the contribution of the prototype space and cross-entropy from temporal contextualization, we evaluated a controlled ablation using the exact same temporal encoder and prototype head as TRIP, but scoring each frame independently without rollout conditioning. While T-DINO achieves 46.46% and frame-wise prototype CE reaches 48.13%, TRIP jumps to 84.17%. Second, we retained contextual goal-consistency scoring but replaced cross-entropy with L1 or cosine distance, obtaining 82.08% and 81.67% success, respectively.

2. Can the original iVideoGPT cost and TRIP cost be evaluated using the same checkpoint, initial states, goals, and MPC budget on VP2?

For VP2, we follow iVideoGPT’s planning protocol, as described in Section 4.1. Table 3 reports baseline results from the published papers. To provide a controlled comparison, we additionally evaluated the official iVideoGPT cost and the TRIP cost using the same world-model checkpoint, initial states, goals, seeds, and MPC budget. The reproduced iVideoGPT baseline achieved a normalized score of 72.96, close to its published score of 73.22.

3. The connection between goal-representation consistency and goal achievement needs stronger validation.

We thank the reviewer for this suggestion. To test whether our consistency cost is sensitive to temporal order, we kept the goal image and both endpoint frames fixed while reversing the intermediate frames in our Procgen trajectory set. The original sequence received a lower cost in 82%, 85%, 95%, and 88% of trajectory pairs on Jumper, CaveFlyer, Heist, and Maze, respectively. These results show that the cost can distinguish temporal order even when the endpoints and frame content are identical.

To further assess whether the cost is associated with goal progress, we measured its Spearman correlation with remaining steps on four Procgen games and five RoboDesk tasks, obtaining 0.55 on Procgen and 0.73 on RoboDesk. These positive correlations indicate that lower costs tend to accompany fewer remaining steps along trajectories. Together, these analyses provide empirical evidence of sensitivity to temporal order and an association with goal progress.

Thank you for the additional experiments. The frame-wise prototype control helps isolate the benefit of rollout context, and the frame-reversal and goal-progress analyses clarify what the planning cost captures. I also appreciate the comparison with iVideoGPT under the same VP2 setup. These results address my main concerns, and I’ll keep my score of 3.

Reviewer_M25kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper adds causal temporal attention to DINOv2 and adapts it to video using DINO/iBOT objectives. It uses the resulting features for video generation and introduces a planning score: append the goal image to a candidate rollout and measure how much its prototype distributions change compared with encoding the goal alone. The planning results are encouraging.

Critical strengths
  • The planning score is interesting and requires no additional reward training.
  • The simulator experiment removes world-model prediction errors and shows a substantial advantage over the tested scoring alternatives.
  • Improvements with learned dynamics and transfer of the scoring encoder from Procgen to VP2 provide useful evidence beyond that experiment.
Critical weaknesses
  1. Why should goal-representation consistency imply a better plan? The encoder sees the goal image in both cases; it is not predicting the likelihood of the goal from the rollout. A history-independent encoder would also produce consistent representations, but no planning signal. This does not invalidate the method, since the encoder is frozen, but it leaves the central question unanswered: what makes changes in its representation track whether a trajectory leads toward the goal? The self-supervised training objective does not explicitly establish this connection.

  2. The comparisons do not isolate representation quality from scoring. The contextual V-JEPA comparison changes both the representation and the metric: feature L1 distance versus prototype cross-entropy. Meanwhile, the adapted DINO encoder does not improve over the original DINOv2 under the tested frame-wise scoring. The results support the proposed combination, but do not establish whether the gain comes from temporal adaptation, the scoring construction, or their interaction.

  3. The generation and planning contributions are evaluated separately. Planning uses RGB Diffusion Forcing on Procgen and an existing iVideoGPT model on VP2, rather than the proposed feature-generation pipeline. The generation results therefore do not establish a planning benefit from that pipeline. The VP2 comparison against published baseline results also leaves some uncertainty about the size of the improvement under a fully controlled comparison.

Questions for authors

Suggestions / Comments

  • Test whether the score distinguishes dynamically compatible and incompatible trajectories with similar endpoints and appearance. Reversing or shuffling otherwise matched trajectories would be informative.
  • Hold the encoder fixed while comparing contextual feature distances and prototype-based scores. Compare DINO and video encoders under matched adaptation and scoring where possible.
  • Separate the claims supported by the generation experiments from those supported by the planning experiments.

Overall assessment

I am borderline, leaning weak accept. The planning score is a potentially useful contribution, and the empirical results warrant attention. However, the paper demonstrates a successful scoring recipe more clearly than it explains the underlying mechanism or establishes which component is responsible for the gain.

4 comments

1. Why Does Goal-Representation Consistency Provide a Useful Planning Signal?

We thank the reviewer for this question. We agree that a history-independent encoder would provide no signal for ranking rollouts, and that our self-supervised objective does not explicitly optimize goal likelihood. Our hypothesis is that conditioning the goal representation on the rollout history provides a useful planning signal. To assess this hypothesis, we conducted two empirical analyses:

  1. Temporal Order Sensitivity: Keeping the goal image and both endpoint frames fixed while reversing the intermediate frames, our scorer assigned lower cost to the original order in 82% of trajectory pairs in Jumper, 85% in CaveFlyer, 95% in Heist, and 88% in Maze. This demonstrates sensitivity to temporal order despite identical endpoints and frame content.

  2. Association with Goal Progress: We measured the Spearman correlation between our consistency cost and remaining steps to the goal, obtaining 0.55 on Procgen and 0.73 on RoboDesk. These positive correlations indicate that lower costs align with fewer remaining steps along trajectories.

We will include these analyses and clarify the scope of our claims in the revision.

2. The comparisons do not isolate representation quality from scoring.

We would like to clarify that our additional ablations suggest that the performance gain is not primarily explained by the choice of cross-entropy as the scoring metric. We initially selected L1 distance for V-JEPA 2.1 to align with its latent-space pre-training objective. To verify that the metric choice did not disadvantage V-JEPA 2.1, we also tested contextual cosine scoring. For V-JEPA 2.1, contextual cosine scoring achieves 23.96%, below the 39.79% obtained with L1 in a same comparison.

To further decouple the scoring metric from temporal context, we evaluated the adapted T-DINO encoder with the same prototype head and cross-entropy, achieving 48.13%, which is nearly identical to its frame-wise cosine score 46.46%. We also evaluated our goal representation using L1 and cosine distances instead of cross-entropy, achieving comparable success rates of 82.08% and 81.67%. These results suggest that the advantage of contextual goal-consistency scoring is not specific to cross-entropy. We will add this ablation to the revision.

3. The generation and planning contributions are evaluated separately.

We thank the reviewer for this feedback and clarify both points below:

  • Separating Generation & Planning Claims: Indeed, rather than claiming that the feature-generation pipeline directly improves planning, our goal is to show that the learned temporal representations independently support two applications. Specifically, the generation experiments evaluate the feature-generation pipeline, whereas the planning experiments evaluate the frozen encoder’s goal-representation consistency cost when paired with fixed world models. We will state this scope clearly in the revision.

  • Controlled VP2 Comparison: For VP2, we follow iVideoGPT’s planning protocol, as described in Section 4.1. Table 3 reports baseline results from the published papers. To address the uncertainty in this comparison, we additionally evaluated the official iVideoGPT cost and our submitted scoring configuration using the same world-model checkpoint, initial states, goals, seeds, and planning budget. The reproduced iVideoGPT baseline achieved a normalized score of 72.96, slightly below its published score of 73.22.

Reviewer_M25kOpenReview ↗

Rebuttal acknowledgment

Thanks for running the matched-endpoint reversal test and the scoring ablations. The results give me more confidence that the planning score uses temporal context and that the gain holds across several distance functions. The contribution of encoder choice is still less clear. My rating remains 3 (weak accept).

AI assistance was used to draft this acknowledgment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.