- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes EvoGraphWM, a world model for long-horizon RL in environments where new objects keep appearing as the agent explores. Built upon a standard DreamerV3, it maintains an object graph whose structure changes during training. Image regions are assigned to a growing set of object-type prototypes; a new prototype is added when poorly reconstructed regions form a cluster far from existing types. Reward relevance decides which types get dedicated graph nodes, and an attention-based transition over nodes yields a learned relation matrix. The graph is connected to the RSSM through an auxiliary loss in which the RSSM predicts future node states, and through a node readout appended to the policy input. The validation is conducted on Crafter and Procgen Heist.
Critical strengths
-
The research problem is important. Dreamer-style holistic world models’ representation of frontier states (like objects the agent has only just met) requires many rounds of repeated training before imagination becomes accurate on them. These are often exactly the states that matter most for task progress. Representing the world with a growing graph lets the model add a node for a new object as soon as it is identified, giving frontier objects an explicit representation without waiting for the holistic latent to absorb them.
-
The analysis directly tests the motivating claim, beyond final return. The paper measures open-loop imagination error over 50 steps and separates important materials from all materials in Crafter, where EvoGraphWM has the lowest error at every horizon, the gap is widest on the important materials. This is good because it links the proposed mechanism to the effect the paper claims, rather than relying on aggregate return alone.
Critical weaknesses
-
As a piece of scientific comunication, this paper fail to make itself understood. Sec. 3 provides no preliminaries, no algorithm box, and no description of how the components run within the DreamerV3 training loop. The writing is dense with technical details but offers little explanation of it. The quality of the writing is clearly below the threshold for acceptance.
-
The central claim, that the model discovers new objects from its own reconstruction error, without labelling, as they appear during training, is not convincing. The annotation is not eliminated; it is merely moved to before training. The front end phi, which determines object types, is pertrained with supervision on ground-truth tile identities. The "new objects" are therefore new only to the prototype set, not to the front end. Crafter is chosen as the experimental setting precisely to simulate an agent learning in a real open-ended environment, and a real open-ended environment has no predefined ground-truth identities. What this paper does amounts to reward hacking.
Questions for authors
-
How does this work run within the Dreamer framework? What is the general training pipeline?
-
On what data, and with how many labelled tiles, was the front end phi trained? Which object types were included as labels? How was phi trained for Procgen Heist, where no tile identities are described?
-
Can you elaborate on what’s the relationship between this work and the 2015 “On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models” system, which does not do any explicit rollouts—instead, the controller is a reinforcement learning prompt engineer which learns to actively query the latent space of its world model for abstract reasoning and planning and decision making?