acceptodds
Under review as a conference paper at ICLR 2027

Roz: A Taste Test for World Model Latents

Abstract

State of the art latent dynamics world models achieve impressive performance for robot planning. But do they encode enough information to accurately unfold dynamics? In this paper, we show that popular world models today achieve strong results on tasks that reduce to image similarity but forget key physical details like mass, deformation, friction, and bounciness. We achieve this by defining and justifying commonsense properties for what ought to be encoded in such a world model and introducing two new scalable evaluation methods. Our evaluation, which we call Roz, totals 1734 individual scenarios. We run Roz on a series of models and validate our eval on control tasks. On our own model, AnonWM, we find that performance roughly improves with training steps, while not necessarily being correlated with reconstruction MSE. We argue current model failures are a symptom of evaluation strategies (probes and planners) that are opaque, model-specific, and high-compute. Roz requires only a model that can ingest still frames or video, a way to extract the latents it predicts from, and a distance function between them, and it can be run in one pass. We hope future evaluations will adopt Roz's evaluation techniques. More broadly, we hope Roz can be a case study for a future where benchmarks allow rapid iteration and comparison across latent world models.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
world models, latent dynamics models, representation evaluation, benchmark, evaluation methodology, contrastive evaluation, nearest-neighbor evaluation, probe-free evaluation, triangle test, distance functions, latent similarity, belief states, bisimulation, physical reasoning, intuitive physics, state abstraction, robot learning, robotics, video representation learning, memory in world models, revealed physical properties

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Tq7QOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper appears to have been submitted to ICLR by mistake. It uses the NeurIPS 2026 template and explicitly states that it was submitted to NeurIPS 2026, so it does not comply with the ICLR 2027 submission format.

Critical strengths

This paper appears to have been submitted to ICLR by mistake. It uses the NeurIPS 2026 template and explicitly states that it was submitted to NeurIPS 2026, so it does not comply with the ICLR 2027 submission format.

Critical weaknesses

This paper appears to have been submitted to ICLR by mistake. It uses the NeurIPS 2026 template and explicitly states that it was submitted to NeurIPS 2026, so it does not comply with the ICLR 2027 submission format.

Questions for authors

This paper appears to have been submitted to ICLR by mistake. It uses the NeurIPS 2026 template and explicitly states that it was submitted to NeurIPS 2026, so it does not comply with the ICLR 2027 submission format.

Reviewer_khVDOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses an important problem and proposes a simple, model-agnostic evaluation framework for world-model latents.

Critical strengths

The paper addresses an important and underexplored problem.

Roz provides a simple, training-free evaluation framework.

Critical weaknesses

The paper requires substantial improvement in its problem formulation, coverage of related work, experimental details, and overall presentation.

Questions for authors

See the Weakness

Reviewer_XWHAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a benchmark for evaluating whether world models encode physically relevant information, such as mass, friction, deformation, and motion history. The paper introduces two evaluation methods, contrastive tests and continuous nearest-neighbor tests, which evaluate distances among latent representations. The benchmark is evaluated on several models, including V-JEPA 2, Qwen, Cosmos, and FastWAM.

Critical strengths

The paper presents an interesting idea of evaluating the physical information encoded in world-model latent representations. The proposed evaluation is relatively simple and evaluates several different model architectures and multiple physical properties.

Critical weaknesses

My main concern is that the paper does not feel fully ready in its current form.

  • The writing feels unfinished in some places. For example, Section 7.1 introduces AnonWM, but the paper does not clearly describe its architecture or training setup, and the analysis of the results is limited. The writing is also not very well organized.
  • The examples of different evaluation tasks could be better visualized and presented to help readers understand what is being tested.
  • The experimental setup and scenario counts are unclear to me. For example, the paper reports 1,734 scenarios, but Table 1 seems to include fewer than 200 scenarios. It is unclear how the total number of scenarios is distributed across the different tasks.

Some models might not be evaluated in an appropriate way. For example, FastWAM only takes the final frame as input in this evaluation, which might not make sense for tasks that require information from earlier observations.

Questions for authors

Please see the weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.