acceptodds
Withdrawn from ICLR 2027

WorldPrior: Internalizing Multimodal Semantics into Latent World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Latent world models learn compact predictive representations of visual dynamics, but matching future features alone does not guarantee that their hidden states expose the object identity, affordance, interaction phase, relational structure, and contextual cues that determine which futures matter for embodied prediction and control. Vision-language models (VLMs) contain complementary multimodal priors, yet using them online changes the deployment interface and introduces a large inference-time cost. We introduce WorldPrior, a training-only adaptation framework that internalizes multimodal semantics into a latent world model while preserving a VLM-free deployment graph. A frozen VLM processes only the observed clip, and detached observation-side representations are aligned with selected internal predictor states through cosine distillation. Because high-level multimodal supervision can perturb fine spatial and dynamical structure, a frozen reference predictor initialized from the same baseline checkpoint acts as a functional anchor, while the backbone-native predictive objective remains active throughout adaptation. All VLM-specific projections, semantic readouts, reference branches, and training-only adaptation objectives are removed at inference. On EgoDex, WorldPrior reduces ADE/FDE from 0.0705/0.0646 to 0.0597/0.0572 and raises threshold accuracy from 0.436 to 0.564. On EgoPAT3D, it obtains the best or tied-best result on all four 3D metrics and three of four 2D metrics across seen and unseen scenes. With LeWM as the control backbone, average success increases from 85.5% to 90.0%; the same principle transfers to DINO-WM, improving the matched four-task average from 83.0% to 85.5%. WorldPrior retains the inference graph and practical latency of the corresponding VLM-free backbone, whereas our online VLM-guidance reproduction is more than 22× slower in forecasting. These results support a broader view of multimodal foundation models as training-time representational priors that can strengthen perception, relational understanding, and task-relevant inference in latent world models without becoming deployment-time dependencies.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
latent world models, vision-language models, multimodal priors, representation alignment, teacher-free inference, embodied prediction and control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_7eX4OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WorldPrior, a training-only adaptation framework for internalizing multimodal semantics in latent world models. Its cosine semantic distillation objective aligns intermediate predictor states with multimodal semantic representations through trainable projections, while reference-predictor retention constrains deviations from the original predictive function. Experiments show that applying WorldPrior to V-JEPA2 improves performance on EgoDex and EgoPAT3D.

Critical strengths
  1. Deployment efficiency. WorldPrior addresses the inference overhead associated with VLM-guided world modeling by incorporating multimodal semantic supervision only during training. The auxiliary branches are removed at deployment.
  2. Evaluation across multiple benchmarks and backbones. The evaluation covers three backbones (V-JEPA2, LeWM, and DINO-WM) and two benchmarks (EgoDex and EgoPAT3D) providing a broader assessment of the framework’s applicability.
Critical weaknesses
  1. Limited novelty. ThinkJEPA has demonstrated the utility of VLM-derived representations for JEPA-style prediction, which is closely related to WorldPrior’s use of multimodal semantic supervision.
Questions for authors

Please see the weaknesses above.

Reviewer_eWNkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WorldPrior, a training-time adaptation framework that incorporates VLM-derived multimodal knowledge into latent world models without requiring the VLM during inference. A frozen VLM provides representations aligned with selected predictor states through cosine distillation, while a frozen reference predictor constrains deviations from the original predictive function. Experiments on EgoDex, EgoPAT3D, and goal-conditioned control tasks using LeWM and DINO-WM report improved prediction and control performance without additional deployment-time VLM overhead.

Critical strengths
  1. Efficient deployment: WorldPrior introduces multimodal supervision during training while preserving the original inference architecture. This is practically valuable for embodied prediction and control, where online VLM guidance can be computationally expensive.
  2. Broad empirical evaluation: The paper evaluates multiple forecasting datasets and control environments across different backbones. On EgoDex, ADE improves from 0.0705 to 0.0597, while average LeWM control success increases from 85.5% to 90.0%. Component ablations also support the effectiveness of the combined framework.
Critical weaknesses
  1. Limited methodological novelty. WorldPrior combines intermediate-feature cosine distillation, output regularization toward a frozen reference predictor, and the backbone’s original training objective. Removing the teacher and auxiliary heads at deployment is an established property of knowledge distillation. Although applying this combination to latent world models is useful, the paper offers limited conceptual novelty beyond integrating these mechanisms. The component ablations demonstrate their empirical usefulness, but do not establish a distinctive learning principle for resolving the proposed conflict between semantics and dynamics.

  2. Underconstrained semantic alignment. Although the raw VLM features are detached, both the teacher-side projection and student-side readouts remain trainable. Cosine agreement can therefore improve through changes in the alignment heads without necessarily preserving the teacher’s semantic structure in predictive states. The native and retention losses constrain world-model outputs but do not directly prevent degenerate auxiliary representations. The paper provides no analysis of representation diversity or information preservation to establish that the alignment objective produces meaningful semantic transfer.

  3. Unisolated multimodal contribution. The motivation emphasizes object identity, affordances, interaction phases, and relations, yet these capabilities are not directly evaluated. Removing distillation establishes the benefit of additional teacher supervision, but does not distinguish multimodal knowledge from generic representation regularization or transfer from a strong visual encoder. A matched visual-only teacher comparison and targeted semantic diagnostics would help establish the necessity of VLM-derived supervision. Without such evidence, the interpretation of the gains as internalized multimodal semantics remains insufficiently supported.

  4. Incomplete dynamics-preservation evidence. Reference retention is intended to preserve fine spatial and dynamical structure, but its removal is evaluated only on EgoDex. The control results provide mixed evidence, including an 80.2% increase in TwoRoom prediction NMSE despite improved success. This does not invalidate the control gain, but leaves unclear what predictive properties the reference anchor preserves. Control-side retention ablations and longer rollout evaluations are needed to assess whether semantic adaptation maintains reliable dynamics beyond the reported task outcomes.

  5. Uncertain control robustness. The control evaluation reports average improvements of 4.5 percentage points for LeWM and 2.5 for DINO-WM across four environments, with task-dependent outcomes. The success-rate tables provide point estimates without establishing variability across independently trained models. The bootstrap analysis of prediction errors does not resolve uncertainty in closed-loop success. Consequently, the reliability of the smaller gains and the consistency of the adaptation benefit across control backbones remain insufficiently established.

Questions for authors

See the weaknesses.

Reviewer_3ADDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a solution to enhance latent world models by distilling the knowledge from VLMs into them. With a frozen VLM, a predictive module is designed to map the latent features of WMs and align with the VLM's latents. A reference model regularization is also used to prevent WMs from functional drift. The method is evaluated on EgoDex with both LeWM and DINO-WM.

Critical strengths
  • The proposed method is well-motivated and reasonable. The idea of adding complementary knowledge from VLMs to WMs is insightful.

  • The technical solution with frozen VLMs, a predictor, and a reference model to prevent functional drift makes sense.

  • The results look good. Without introducing extra compute during inference, the method can improve both LeWM and DINO-WM with healthy margins.

Critical weaknesses
  • The technical novelty is a bit limited. The general idea of improving generative multimodal models with latents from understanding models is not new. The paper's core technical contribution is to introduce widely known solutions, such as REPA, for general diffusion models to WMs.

  • Some key ablation studies are missing. Table 6 shows the component-removal results. But as an academic paper, core designs of the proposed method should also be validated. For example, how to design the predictor? Will the selection of latent features in WMs/VLMs affect the results? I also found a concurrent paper (arXiv:2610.23984) that uses DINO features to improve WMs. How about using DINO/CLIP/ features or just the features from the ViT of VLMs instead of LLM features from VLMs? Visual backbones also capture semantic representations related to spatial structures.

  • There is also no in-depth analysis of why the WMs are improved. Can we observe better latent features from fine-tuned WMs? More studies on the complementary knowledge from VLMs would make the paper stronger.

  • Would the fine-tuning affect the generalization ability of pretrained WMs? It would be better to have some OOD evaluations to test the models' performance on OOD benchmarks. It would be interesting to see how much data is needed to generally improve the base WMs.

Questions for authors

Please refer to my comments in Critical Weaknesses. Responses for W2/W3 may affect my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.