acceptodds
Under review as a conference paper at ICLR 2027

Structuring Privileged States with LLMs for Robust World Models

Abstract

Visual world models enable control directly from high-dimensional visual observations through latent imagination, yet their learned representations can remain brittle under distribution shifts. While prior work studies generalization under complex visual distractions, we discover that simple changes to environment configurations and physical dynamics are sufficient to cause substantial performance degradation under zero-shot evaluation. Privileged states expose rich structural information about the underlying environment and have been broadly adopted to support robust generalization, but directly predicting raw states may require modeling task-irrelevant or visually inaccessible quantities, while manually designed auxiliary targets scale poorly across diverse tasks. In this work, we propose a framework that harnesses the reasoning capabilities and commonsense priors of Large Language Models (LLMs) to automatically structure privileged states into compact, task-relevant semantic properties. These LLM-derived properties serve as auxiliary targets that explicitly regularize the world model toward representations that remain stable across distribution shifts. To seamlessly integrate this supervision, we introduce an adaptive multi-loss calibration mechanism to dynamically balance the semantic prediction objective against the original world model losses, removing the need for task-specific manual weight tuning. Across Meta-World, DeepMind Control Suite, and CARLA, our approach consistently achieves substantial zero-shot generalization improvements over strong model-free and model-based baselines under both held-out configuration and physical dynamics shifts.

Primary area:
reinforcement learning
Keywords:
World Models, Reinforcement Learning, Representation Learning, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jMmYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper augments a visual DreamerV3 world model with auxiliary targets derived from simulator privileged states. An LLM is prompted once per task to write a Python extractor for compact relational or kinematic descriptors, and the resulting program supplies masked, normalized targets during training, while deployment remains pixel based. A learnable loss scaling scheme balances this objective with reconstruction, reward, and KL losses. The authors evaluate zero shot transfer to unseen Meta-World layouts and shifted DMC and CARLA dynamics. The reported averages improve over the strongest listed baselines on Meta-World, DMC, and CARLA.

Critical strengths

Using LLMs for semantic structuring of privileged states combined with adaptive calibration is both clean and general. By using LLMs to compress privileged states into task-relevant relative geometric quantities (e.g., relative distance) and integrating them seamlessly via learnable loss scaling, the approach avoids two major pain points that per-task manual design of auxiliary targets and per-task manual weight tuning, and is consistently effective across three highly diverse domains, indicating strong generality and practical value.

Critical weaknesses
  1. The core increment is limited, and the methodological novelty is weak. Semantic/auxiliary prediction targets and learnable multi-loss weighting are both established techniques. This work is essentially an innovation at the implementation level, i.e., using LLMs to automatically generate auxiliary targets.

  2. The paper does not yet isolate the claimed value of LLM-based semantic structuring. Appendix D’s prompt already mandates relative geometry and topological indicators, while the displayed extractors are described as representative. Appendix E.1 compares against raw-state prediction only on CARLA, without a matched non-LLM relational-feature control. Consequently, the results establish the value of the full pipeline more clearly than the need for an LLM to specify its targets.

  3. The evidence that the method specifically reduces degradation under distribution shift is uneven across domains. Additionally, the large reported spreads on some Meta-World tasks, and the advantage of the proposed method falls within the noise range.

Questions for authors
  1. Some questions in Weaknesses.

  2. Can you compare the LLM-generated targets against simple hand-specified or rule-generated relative features with matched descriptor dimension, prediction head, normalization, and loss treatment?

  3. How were the task-specific extractor programs generated, checked, and selected?

  4. Has the "visually inferable" constraint actually been verified? Do the LLM-generated semantic properties contain components that are difficult to infer visually?

  5. For Meta-World, can you compare success on the training-layout distribution with the held-out layout distribution and characterize variation across held-out configurations?

  6. What is the additional time/memory overhead introduced by the semantic prediction head and the offline LLM generation during training?

  7. The fact that "simple changes to environment configurations and physical dynamics can cause substantial performance degradation" is already well-known in the fields of embodied AI and humanoid robotics, and it is nothing new.

Reviewer_4Y2AOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes an auxiliary loss for DreamerV3 to improve zero-shot generalization under distribution shift related to spatial layout and physical dynamics. Before training, an LLM is prompted with a task description and simulator states to generate a Python function that computes a vector of geometric features alongside a validity mask. During training Dreamer, an MLP head is added to predict these features from the RSSM latent state via an MSE loss. To balance this new loss against the Dreamer losses, the authors adopt a series of normalization schemes. Experiments are done on Meta-World and DMC and CARLA with perturbed physical dynamics.

Critical strengths
  1. Evaluating visual world models under physical perturbations and OOD spatial configurations without relying on synthetic background distractors is a sensible question to explore.
Critical weaknesses
  1. The framework is over-engineered and not Occam's razor-applied, and fails to justify the use of LLM for extracting privileged information. The LLM is not discovering latent structure autonomously and humans must prompt it with task details and provide a simulator. Given access to such details and the simulator, the python code does not even need to be written by the LLM. We can even simply train an MLP head directly on the privileged state vector extracted from the env via a simple script during training without involving an LLM at all.

  2. The baseline comparisons regarding privileged information are unfair. The method only compares with conventional model-based and model-free RL methods and does not evaluate baselines that use privileged info such as [1], which makes the improvements in Table 2 artificially inflated.

Overall, the paper is an incremental (and unnecessary) change to DreamerV3, and does not meet the criteria for publication in a top-tier ML conference.

[1] Lambrechts, Gaspard, et al. "Reinformed Dreamer: An Asymmetric World Model Efficiently Trained through Latent Guidance."

Questions for authors
  1. Can you provide results on the existing RL baselines that use privileged info? What is the advantage of using a complex pipeline to extract such info compared to the existing baselines?

  2. How can you justify the complex normalisation schemes for the loss functions? Do you have ablations results on them and can you provide details on how you tuned them?

Reviewer_eyf7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the performance degradation of visual world models under perturbations to simulator configurations and physical parameters. It proposes an LLM-guided framework built on DreamerV3, using LLM-generated executable Python functions to transform privileged states into semantically meaningful properties. These properties serve as auxiliary prediction targets, encouraging latent representations to capture task-relevant structure. A learnable multi-loss calibration mechanism further integrates semantic prediction with standard world-model objectives by adaptively adjusting their relative weights. Experiments on Meta-World, DeepMind Control Suite, and CARLA demonstrate good improvements in zero-shot generalization under distributional shifts. Component ablations and additional analyses further investigate the contributions of semantic supervision and loss calibration, alongside their effects on the learned representations.

Critical strengths
  1. Good Experimental Results Across Shifted Distributions. The proposed method consistently outperforms strong model-free and model-based baselines across Meta-World, DMC, and CARLA, demonstrating good improvements in zero-shot generalization under both configuration and physical dynamics shifts.
  2. Effective Ablation Results on Components and Supporting Analysis The ablation studies demonstrate the effectiveness of the proposed semantic targets and adaptive loss calibration. Additional analyses, including linear probing, attention visualization, and gradient analysis, further support the benefits of the proposed framework.
Critical weaknesses
  1. The specific contribution of LLM-based semantic generation is insufficient, and analysis of prompt design is limited. The meta-prompt explicitly instructs the LLM to prioritize relative distances, spatial alignments, and topological indicators, while the generated extractors in Appendix D largely follow these predefined rules using standard geometric operations. This raises the question of whether the improvements stem from LLM reasoning or simply from following human-specified feature-design principles. Furthermore, the paper provides limited analysis of the generation process itself, including how outputs are selected, the first-attempt success rate, and whether retries or manual corrections are required.

  2. Concerning on novelty The core idea of using LLMs to generate task-relevant properties from privileged states and incorporating them as auxiliary prediction targets in world models closely resembles DreamTIP (Liang et al., 2026), and the adaptive multi-loss calibration mechanism largely follows HarmonyDream. The methodological contributions beyond combining existing techniques might be insufficient.

Questions for authors
  1. Regarding LLM (Weakness 1)
(a) Whether the same meta-prompt template is used across all tasks and whether any task-specific prompt engineering, manual revisions, or repeated generation attempts are required? Like, are the semantic extractors generated successfully on the first LLM attempt for every task?
(b) Comparisons on different LLM templates and even manual designs will be beneficial.
  2. Given the substantial overlap with DreamTIP, what is the fundamental methodological novelty beyond adaptive loss calibration and broader benchmark evaluation? (Weakness2)
  3. Clean-to-Shift: Could you report no-shift performance on Meta-World and DMC? Why no-shift experiment is only conducted on CARLA?
  4. Why can the improved attention patterns in Figure 4(b) be attributed solely to semantic descriptors? Could the authors provide attention map visualizations for the +target and +calibration variants to verify whether the improvements arise from semantic supervision rather than adaptive loss calibration?
  5. How does the method perform when only the semantic loss is adaptively calibrated while the original world-model loss weights remain fixed?

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.