acceptodds
Under review as a conference paper at ICLR 2027

ConceptWorldModel: Can World Models Learn Dynamics in Concept Space?

Abstract

abstract World models have demonstrated substantial capability in predicting future states in complex environments; however, their internal representations often remain difficult to interpret. Consequently, it remains unclear whether such models merely encode information required for prediction or develop structured semantic representations of the surrounding environment. This work investigates whether higher-order semantic information can become recoverable from hierarchical representations built from simpler grounded concepts without direct supervision of those higher-order concepts. To address this question, ConceptWorldModel introduces a hierarchical world model for autonomous driving that transforms visual observations into grounded concepts, relational representations, and higher-order scene representations. These representations define a structured concept space that is propagated through action-conditioned temporal dynamics, enabling recursive prediction of future concepts across multiple horizons. The model is evaluated on Bench2Drive using a structured concept ontology derived from driving-scene annotations and metadata. Experimental results demonstrate sustained predictive performance over extended rollouts, achieving an average precision of \(0.783\) for binary concepts and a normalized mean absolute error of \(0.435\) across scalar regression targets at a 20-step prediction horizon. Evaluation of concepts excluded from direct supervision further shows that higher-order semantic information remains recoverable across the learned representation hierarchy. Overall, the results show that future driving states can be represented and predicted through an explicit semantic concept space, providing a structured alternative to exclusively opaque latent world-model representations.

Primary area:
learning on time series and dynamical systems
Keywords:
World Models, Concept-Based Representation Learning, Concept-Space Dynamics, Autonomous Driving, Hierarchical Representations, Action-Conditioned Dynamics, Semantic World Models, Long-Horizon Prediction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KujUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

ConceptWorldModel studies action-conditioned prediction using a hierarchy of grounded factors, relational slots, and higher-order representations. A frozen visual encoder provides features, thirteen annotated concepts supervise the grounding module, and a dynamics model recursively predicts future states. The evaluation uses Bench2Drive and includes concept forecasting, architectural ablations, and probes for three concepts excluded from direct supervision. The paper reports improved forecasting over adapted latent-dynamics baselines and different patterns of concept recoverability across representation levels.

Critical strengths

The paper addresses a useful question: how semantic information is organized and preserved in a predictive model. Separating directly supervised concepts from held-out probes gives the study a clear experimental focus.

Critical weaknesses

I am not convinced by the main baseline comparison. In Table 1, DreamerV3 diverges on every continuous-prediction horizon, while MILE already has an nMAE of 6.715 at the first step, compared with 0.410 for CWM. The idea of comparing dynamics models on a shared state is reasonable, but these results raise questions about the adaptations and optimization. The paper does not provide enough detail about the adapted objectives, output scaling, or training settings to rule out implementation or convergence issues. Until those issues are resolved, the large performance gaps are difficult to interpret. A stable simple recurrent predictor and a persistence baseline would also help establish how demanding this forecasting task actually is.

The evidence for hierarchical semantic dynamics is incomplete. Table 2 measures whether held-out concepts can be recovered from representations of observed scenes. Future rollout evaluation, however, measures only the thirteen directly supervised concepts. The paper therefore does not show whether the held-out semantic information remains recoverable after recursive prediction. The interpretation of the probing results also needs stronger controls: closing speed is derived from lead-distance history, and TTC is then computed from distance and closing speed. Recovering these quantities may reflect retention of lower-level information rather than a distinct benefit of the proposed hierarchy. Probes on alternative representations and uncertainty estimates for the small differences between levels would make this conclusion more convincing.

The comparison with recent work is not sufficiently convincing. Although Section 2 discusses driving world models from 2025–2026, the experiments are limited to adapted versions of MILE, TD-MPC2, and DreamerV3. It remains unclear whether these comparisons adequately represent competitive approaches to the proposed task, particularly given the numerical divergence of DreamerV3 in Table 1. The authors should better explain their baseline choices and provide stronger, task-matched comparisons. The distinction from existing concept-based and relational modeling approaches also needs more concrete discussion. Taken together, these gaps leave me unconvinced that the paper has sufficiently established either its incremental novelty or its empirical advantage.

I lean toward rejection. The research question is worthwhile, and some of the architectural controls are useful. However, the main performance comparison relies on baselines whose training behavior is not adequately explained. At the same time, the semantic analysis does not establish whether held-out concepts remain accessible during future rollouts. These gaps affect the central claims of the paper.

Questions for authors

What changes were made to DreamerV3, MILE, and TD-MPC2 for the controlled comparison? Please explain the divergence in DreamerV3 and MILE’s large one-step error, including the relevant losses, normalization, and convergence checks.

Were the held-out probes evaluated on recursively predicted states, or only on observation-encoded states? If results on predicted states are available, please report them across rollout horizons using the same frozen probes.

Please clarify the probe training and model-selection splits, and report uncertainty for Table 2. Are comparable probe results available for the flat representation or frozen visual features? Also, what duration in seconds does the 20-step horizon represent?

Reviewer_4bBUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a new hierarchical world model for autonomous driving. The authors present an architecture that learns to predict 13 of 17 available concepts. A concept is a hand-picked attribute of the state that is relevant to autonomous driving in this case (e.g. lead vehicle distance). The model is trained to predict the concepts for future time-steps. The results show that it outperforms TD-MPC2, MILE, and DreamerV3 on concept prediction over several different horizons. Furthermore, some held-out concepts can be probed from different location (L1, L2 and L3) in the architecture. Finally, the authors also include an ablation study on the different design decision of the architecture and a small study on counterfactual action sensitivity.

Critical strengths
  • The writing is clear and the paper is easy to follow
  • The setting is interesting an insights into world models is of high relevance to the research community
Critical weaknesses
  1. Just another model architecture: The method section only presents the architecture of the model. It does not state what problem motivates these decisions, what existing models lack or how the architecture differs from prior work. The only difference I see is that it is supervised mostly by the presented concepts, which are nothing different than hand-engineered features that the authors find useful for the task of autonomous driving.
  2. The evaluation does not hold what the abstract/introduction promise: The evaluation consists mainly of two things: a) how well the concepts can be predicted over time and b) how well the held-out concepts can be recovered from different location in the model's architecture. I find a) not giving much new insights: That a purpose-built model outperforms other general world models is not surprising. Furthermore, it is not stated in the paper, how the baselines are adapted to the setting that the evaluation works in. There is not way for the reader to assess whether the evaluation has been fair. For b) the main issue I have is that it doesn't actually show whether an understanding of higher-level concepts is contained in the world model. The held-out concepts are very related to the concepts that the world model is trained on and the relationship is straight-forward so that it is not surprising that these can be predicted by the model. Indeed, table 3 confirms this since it doesn't really matter where in the model one probes these concepts: the performance does not differ by much.
Questions for authors
  1. Did you also try to see whether these concepts can be predicted without concept supervision, e.g. just by visual supervision?
  2. Can you explain how the baselines are implemented? Do they simply replace the TCDM? Are the losses for the CGM applied across all baselines?
  3. Does the world model actually still work for autonomous driving? Can we still derive a working driving policy? Or does it only predict concepts successfully, but may not contain information outside of the concepts that may still be necessary for a driving policy?
Reviewer_VsAjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ConceptWorldModel (CWM) investigates whether world models can learn dynamics in a semantically organized concept space. It encodes driving observations into grounded concepts, relational slots, and higher-order representations, then recursively predicts future concepts through action- and route-conditioned dynamics. Experiments cover multi-step prediction, held-out concept probing, architectural ablations, and action interventions. At a 20-step horizon, CWM achieves 0.783 AP and 0.435 nMAE.

Critical strengths

The ontology distinguishes 13 supervised concepts, three held-out probe concepts, and exogenous route commands, making their roles in training and evaluation explicit.

ConceptDriver is explicitly presented as an example of optimization within the learned model. The paper does not equate reduced predicted cost with improved closed-loop driving performance, keeping its claims appropriately scoped.

Critical weaknesses

In Table 1, MILE already has an nMAE of 6.715 at the one-step horizon, while DreamerV3’s continuous-valued predictions diverge. The reliability of the baseline adaptations and training procedures needs verification.

The paper does not adequately specify which original architectural components, losses, normalization, and stabilization mechanisms are retained in each baseline. Training and validation curves are also missing.

Lead-vehicle closing speed is derived from changes in distance, and time-to-collision is computed from distance and closing speed. These quantities are therefore closely related to the histories of supervised lower-level concepts. Comparisons using frozen visual features, atomic concept histories, and simple compositional models are needed to determine how much the hierarchy improves their recoverability.

The paper does not compare predictions with actual intervention outcomes from the same initial states, nor does it evaluate counterfactual changes in relations with other vehicles or risk-related concepts.

Equation (6) assigns separate state blocks to the three representation levels, Equation (7) updates the joint state, and Equation (8) decodes it jointly. However, the paper does not explain how each block retains its semantic role after prediction.

Prediction steps are not mapped to a clearly specified physical timescale, and speeds are reported only in standardized units. This makes it difficult to interpret the duration of a 20-step rollout or assess whether the braking response is physically plausible.

Questions for authors

Please clarify how MILE, DreamerV3, and the other baselines were adapted, including the retained architectural components, losses, normalization, and stabilization mechanisms, and provide training and validation curves. This would help determine whether the high one-step nMAE and numerical divergence stem from implementation or training issues.

Please explain how the state blocks in Equations (6)–(8) retain their semantic roles during prediction, and specify the physical time represented by each prediction step.

1 comment

The authors have not yet responded; therefore, I will maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.