- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper augments a visual DreamerV3 world model with auxiliary targets derived from simulator privileged states. An LLM is prompted once per task to write a Python extractor for compact relational or kinematic descriptors, and the resulting program supplies masked, normalized targets during training, while deployment remains pixel based. A learnable loss scaling scheme balances this objective with reconstruction, reward, and KL losses. The authors evaluate zero shot transfer to unseen Meta-World layouts and shifted DMC and CARLA dynamics. The reported averages improve over the strongest listed baselines on Meta-World, DMC, and CARLA.
Critical strengths
Using LLMs for semantic structuring of privileged states combined with adaptive calibration is both clean and general. By using LLMs to compress privileged states into task-relevant relative geometric quantities (e.g., relative distance) and integrating them seamlessly via learnable loss scaling, the approach avoids two major pain points that per-task manual design of auxiliary targets and per-task manual weight tuning, and is consistently effective across three highly diverse domains, indicating strong generality and practical value.
Critical weaknesses
-
The core increment is limited, and the methodological novelty is weak. Semantic/auxiliary prediction targets and learnable multi-loss weighting are both established techniques. This work is essentially an innovation at the implementation level, i.e., using LLMs to automatically generate auxiliary targets.
-
The paper does not yet isolate the claimed value of LLM-based semantic structuring. Appendix D’s prompt already mandates relative geometry and topological indicators, while the displayed extractors are described as representative. Appendix E.1 compares against raw-state prediction only on CARLA, without a matched non-LLM relational-feature control. Consequently, the results establish the value of the full pipeline more clearly than the need for an LLM to specify its targets.
-
The evidence that the method specifically reduces degradation under distribution shift is uneven across domains. Additionally, the large reported spreads on some Meta-World tasks, and the advantage of the proposed method falls within the noise range.
Questions for authors
-
Some questions in Weaknesses.
-
Can you compare the LLM-generated targets against simple hand-specified or rule-generated relative features with matched descriptor dimension, prediction head, normalization, and loss treatment?
-
How were the task-specific extractor programs generated, checked, and selected?
-
Has the "visually inferable" constraint actually been verified? Do the LLM-generated semantic properties contain components that are difficult to infer visually?
-
For Meta-World, can you compare success on the training-layout distribution with the held-out layout distribution and characterize variation across held-out configurations?
-
What is the additional time/memory overhead introduced by the semantic prediction head and the offline LLM generation during training?
-
The fact that "simple changes to environment configurations and physical dynamics can cause substantial performance degradation" is already well-known in the fields of embodied AI and humanoid robotics, and it is nothing new.
Dear authors and reviewers,
The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.
AC