acceptodds
Under review as a conference paper at ICLR 2027

Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning

Abstract

A key design choice in visual world models is what information the latent state is encouraged to preserve. Models that learn a latent space with only a latent-prediction objective adapt to environment dynamics, but the predictive objective alone admits collapsed solutions and provides no explicit constraint on the visual information retained by the latent. Models built directly on pretrained visual features inherit a strong visual prior, but tie prediction and planning to a frozen state space. We propose Anchor-WM, which instead uses pretrained visual features as a teacher for learning a compact latent state rather than as the world-model state itself. Anchor-WM learns this compact latent with action-conditioned dynamics, while distilling the dense DINO representations through a training-only distillation head. This encourages the compact latent to retain visual information captured by the pretrained representation while discouraging representational collapse. At inference, the DINO encoder and distillation head are discarded, allowing prediction and planning to operate entirely in the learned latent space. Anchor-WM achieves the best average success rate across the four evaluated environments while enabling substantially faster planning than world models operating directly on DINO patch features. These results show that pretrained visual features can effectively guide world-model learning without defining the state space used for prediction and planning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Latent World Models, JEPA, Pretrained Visual Features, Feature Distillation, Visual Planning, Representation Learning, DINO

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DzB5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Anchor-WM learns a 192-dimensional state using action-conditioned prediction and reconstruction of DINO patch features. The DINO teacher and reconstruction head are removed at inference. It reports the highest four-task mean success rate in the standard setting, 84.7%, and the farther-goal setting, 71.1%.

Critical strengths
  • Planning results The standard-task mean improves over LeWM's 79.9% and DINO-WM's 76.6%. Planning is about as fast as LeWM and much faster than patch-based DINO-WM.

  • Distillation choices Table 5 compares loss functions, patch versus global targets, and final-state distillation versus intermediate alignment across all four tasks. The appendix also tests distillation weight and latent size on Cube.

Critical weaknesses
  • Compact baselines I am missing a comparison with learned compression of DINO patch features, or a compact student distilled before learning dynamics. DINO-WM(CLS) drops patch information rather than learning to compress it, so its poor performance does not answer this question. The patch/global ablation changes the teacher target too. Since distillation and compact prediction are familiar ingredients, these direct comparisons matter for judging what the proposed combination adds.

  • Joint training The experiments do not isolate the benefit of adapting the state jointly with dynamics. Prediction-only is a weak comparison because it can collapse; the LeWM comparison also changes the regularizer and introduces pretrained supervision. The speedup over DINO largely comes from planning in a smaller state, while speed is already similar to LeWM. Under color changes, Anchor-WM averages 61.9%, versus DINO-WM's 74.5%. That is a reasonable speed–robustness tradeoff, but it leaves open whether joint training improves on an ordinary compact distilled encoder.

  • Toy experimental setup Please provide more real world experiments or more complex task other than the toy examples

Questions for authors

Please refer to the Weaknesses

1 comment

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Reviewer_bGVAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a new way of using pretrained visual features to train latent world models. They claim that when using pre-trained features and only training a predictor, features don’t retain information relevant for dynamics and planning, and that when training end-to-end with a regularization (LeWM with SIGReg), no objective constraints which visual information is retained in the latent space. They propose using pre-trained visual features as a teacher for learning a compact latent space: they use the standard next latent prediction objective and add a distillation objective where an MLP head reconstructs pretrained DINO features. They evaluate short and longer horizon planning tasks on the standard benchmarks (TwoRoom, Reacher, PushT, Cube) and compare against baselines. On some tasks, the introduced method, Anchor-WM, has higher success rates than DINO-WM baselines, but it is always significantly faster.

Critical strengths
  • Simple yet original method to have a smaller dimension latent retain information from pretrained DINO features.
  • The distillation term helps prevent collapse, removing the need for using an extra regularizer.
  • Relevant ablations, namely ablate over the nature of the distillation loss (MSE vs cosine), and over what representation type to use as a target (patch vs CLS).
  • Improvement over baselines on average using their method.
Critical weaknesses
  • The probes are trained to show that physical information is captured, but it does not seem to be captured better in the Anchor-WM latent than in the others. What explains the higher success rate of Anchor-WM?
  • The tasks shown in Fig. 2 are overly simplistic. In TwoRoom, the agent stays on the same side. In PushT, only the agent moves with no interaction with the T block.
  • We don’t see the trainable parameters in L_pred (Eq. 3).
Questions for authors
  • The paper claims that regularizing the latent space, as LeWM does with SIGReg, doesn’t tell which visual information the learned state should retain. From my understanding, LeWM trains both the representation and the predictor end-to-end, so the representation is encouraged to retain visual information relevant for the prediction. Could the authors clarify?
  • Are the trajectories optimal? If not, a goal sampled further along the trajectory is not necessarily further from the start.
  • On the longer-horizon tasks, is the planning horizon also longer?
  • What motivates retaining the visual information in DINO features? Why would this information be the one needed for dynamics and planning?
  • Does the distillation objective / the distillation MLP add extra training time?
1 comment

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

Reviewer_1Ty4OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

World models such as LeWM and PLDM plan in the embedding space by learning compact embeddings from scratch via next embedding prediction. Operating in the small embedding space keeps planning cheap, but the models need a regulariser to stop the embedding collapse, and nothing guides which visual details it keeps. DINO-WM plans directly on frozen DINOv2 patch features which is a much richer starting point, but the state is large, so planning is far slower.

Anchor-WM tries to get the best of both worlds. It keeps LeWM's setup: a small encoder trained from scratch turns each frame into one vector, a predictor rolls it forward, and CEM searches for actions that reach a goal image. The main difference is that LeWM's regulariser is replaced by a training-only head that reconstructs DINO patch features from the embedding. The hypothesis is that this both prevents collapse and passes DINO's visual knowledge into the small embedding. DINO and the head are dropped after training, so planning costs the same as in LeWM.

The paper evaluates on four simulated tasks: a two-room maze, a reaching arm, Push-T and cube pick-and-place. Anchor-WM gets the best average success against LeWM, PLDM and DINO-WM, without winning every task, and plans about 100× faster than DINO-WM. It also does well on longer-horizon goals, and under colour changes it lands between LeWM and DINO-WM. The most informative ablation shows that reconstructing DINO's patch features works far better than reconstructing its single global feature.

Critical strengths

The idea is simple and cheap - only requires a single-loss change to the public LeWM code. Using DINO only as a training signal removes DINO-WM's planning cost.

The patch-versus-global ablation in Table 5 is a clear finding about what kind of teacher signal helps a compact world model and holds regardless of how the headline comparison holds up.

Critical weaknesses

I have concerns about the comparison with LeWM.

  • Every baseline in Table 1 comes in below its own published result on this benchmark. Against LeWM's published numbers, Anchor-WM wins only on the two-room maze, which the LeWM paper seems to name as a weak spot of its regulariser.
  • On the cube task, a random policy already succeeds 48% of the time (per the LeWM paper). Several baselines and the no-distillation model sit right at that level. So on that task they are effectively at chance, and the meaningful range is roughly 48–100%, not 0–100%.
  • The ± is undefined, and many per-task margins are small relative to it.

I'm not convinced DINO is the reason it works.

  • The only model without the DINO head is the prediction-only one which has no anti-collapse term and performs at chance. That shows some regulariser is needed, not that DINO's features help.
  • The claim that distillation prevents collapse only rules out a constant embedding. If the embedding isn't normalised, the encoder can shrink it while the head scales back up. The prediction loss then falls without the dynamics getting any better. This could then also explain why results barely change across a 1000× range of the distillation weight (Fig. 7).
  • The text gives the colour-change and probing results more merit than Tables 3 and 6 show. Under colour changes, Anchor-WM is worse than LeWM on the reaching arm. On the PushT probes, it is no better than LeWM.
Questions for authors
  1. Why are the baselines below their published numbers on this benchmark? Does your pipeline reproduce LeWM's published results?
  2. How does DINO compare with a same-size head that reconstructs pixels, random-network features or another encoder's features? Does adding the DINO head on top of LeWM's regulariser help?
  3. Is the embedding normalised, and is the prediction target stop-gradiented? Reporting the embedding's norm and effective rank over training would show whether it shrinks.
  4. Which distillation weight is used in Table 1? Fig. 7 never reaches the reported cube score. How large is the distillation head, and what does training cost with DINO included?
1 comment

Dear Reviewer,

Thank you for your efforts in reviewing this submission. As the discussion period is approaching its end, I would like to kindly remind you to read the authors' rebuttal and engage in the discussion. Your input is important for reaching a well-informed final assessment.

Thank you for your time and cooperation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.