acceptodds
Withdrawn from ICLR 2027

EvoGraphWM: A Structure-Evolving World Model with Adaptive Object Graph for Long-Horizon Tasks in Dynamic Environments

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon tasks are challenging due to their sparse rewards, where useful behaviour takes many steps to pay off. World models address such tasks by learning environment dynamics and optimising behaviour through imagined rollouts, which therefore need to stay accurate over long horizons. In dynamic environments, new objects continually appear as the agent explores, changing the structure required to represent the environment state. Existing world models fix the structure of their state representation as a holistic vector or a predefined set of object slots, leaving new objects uncaptured and their relations unmodelled, and the resulting errors compound over long-horizon imagination. To solve this problem, in this paper we propose a world model that evolves its state structure with the objects encountered during training. However, realising such structural evolution raises three challenges: 1) how to discover and prioritise new objects during training; 2) how to model the relations between newly discovered objects and those already known; 3) how to learn dynamics and behaviour over a structure-evolving state. To address them, we present EvoGraphWM, a structure-evolving world model centred on an adaptive object graph, whose nodes, focus, and relations evolve as the agent learns. EvoGraphWM consists of: (i) a Self-evolving Node Extractor that grows the node set, discovering new objects through the reconstruction error and prioritising the reward-relevant ones as dedicated nodes while the rest stay in a shared background; (ii) a Relational Dynamics Modeller that learns the graph's transition through collaborative node prediction, where object relations emerge; (iii) an Evolved Graph Integrator that anchors the adaptive object graph to a steady holistic state through two complementary routes, giving dynamics and behaviour a stable footing to learn from. Experiments demonstrate that EvoGraphWM outperforms strong baselines on both Crafter and Procgen Heist, and further keeps the imagined rollouts accurate over remarkably longer horizons.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
self-evolving world model, structure evolving, adaptive object graph

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_txpXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes EvoGraphWM, a world model for long-horizon RL in environments where new objects keep appearing as the agent explores. Built upon a standard DreamerV3, it maintains an object graph whose structure changes during training. Image regions are assigned to a growing set of object-type prototypes; a new prototype is added when poorly reconstructed regions form a cluster far from existing types. Reward relevance decides which types get dedicated graph nodes, and an attention-based transition over nodes yields a learned relation matrix. The graph is connected to the RSSM through an auxiliary loss in which the RSSM predicts future node states, and through a node readout appended to the policy input. The validation is conducted on Crafter and Procgen Heist.

Critical strengths
  • The research problem is important. Dreamer-style holistic world models’ representation of frontier states (like objects the agent has only just met) requires many rounds of repeated training before imagination becomes accurate on them. These are often exactly the states that matter most for task progress. Representing the world with a growing graph lets the model add a node for a new object as soon as it is identified, giving frontier objects an explicit representation without waiting for the holistic latent to absorb them.

  • The analysis directly tests the motivating claim, beyond final return. The paper measures open-loop imagination error over 50 steps and separates important materials from all materials in Crafter, where EvoGraphWM has the lowest error at every horizon, the gap is widest on the important materials. This is good because it links the proposed mechanism to the effect the paper claims, rather than relying on aggregate return alone.

Critical weaknesses
  • As a piece of scientific comunication, this paper fail to make itself understood. Sec. 3 provides no preliminaries, no algorithm box, and no description of how the components run within the DreamerV3 training loop. The writing is dense with technical details but offers little explanation of it. The quality of the writing is clearly below the threshold for acceptance.

  • The central claim, that the model discovers new objects from its own reconstruction error, without labelling, as they appear during training, is not convincing. The annotation is not eliminated; it is merely moved to before training. The front end phi, which determines object types, is pertrained with supervision on ground-truth tile identities. The "new objects" are therefore new only to the prototype set, not to the front end. Crafter is chosen as the experimental setting precisely to simulate an agent learning in a real open-ended environment, and a real open-ended environment has no predefined ground-truth identities. What this paper does amounts to reward hacking.

Questions for authors
  • How does this work run within the Dreamer framework? What is the general training pipeline?

  • On what data, and with how many labelled tiles, was the front end phi trained? Which object types were included as labels? How was phi trained for Procgen Heist, where no tile identities are described?

  • Can you elaborate on what’s the relationship between this work and the 2015 “On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models” system, which does not do any explicit rollouts—instead, the controller is a reinforcement learning prompt engineer which learns to actively query the latent space of its world model for abstract reasoning and planning and decision making?

Reviewer_hduZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a world model for long-horizon RL in open-world games where the types of objects the agent deals with are not known in advance. The model is based on a standard DreamerV3-like recurrent state-space model, but adds a novel flexible object graph that grows during training, mainly by adding new reward-relevant objects. The relations between objects are estimated by attention in the transition model. The graph is linked to the recurrent model via an auxiliary prediction loss, and the summary of the graph nodes is fed to the policy. The approach is evaluated on Crafter and Procgen, outperforming traditional monolithic and object-centric baselines.

Main contribution points:

  1. A world model whose state structure is not fixed in advance but changes while the agent learns.
  2. A procedure to build the graph flexibly: new object types are added when the model keeps failing to reconstruct part of the image, and the objects that matter most for the reward are given dedicated graph nodes. The links between objects are learned from the attention of a prediction model rather than being hand-specified.
  3. A novel mechanism used to attach the graph to a standard recurrent world model via a loss function that makes the recurrent model predict the graph, with a small summary of the graph passed to the policy.
  4. Experiments on Crafter and one Procgen game that show better scores than traditional monolithic and object-centric baselines. The results of the ablation studies suggest that the gains come from the discovered objects and their relations.
Critical strengths
  1. The authors design an approach that can solve the known problem of building an object-centric world model that can accommodate a different number of objects. A similar problem arises in object-centric learning when Slot Attention-based models are applied to dynamic environments where the number of objects is not known in advance. The authors propose to use a recurrent backbone of the traditional DreamerV3 architecture, grow the object vocabulary on demand, give the reward-relevant objects their own nodes, and learn relations by prediction. The two-way dependency to the backbone, with the policy input starting at zero so that structural changes do not destabilise a policy trained in imagination, is a clean solution to a real engineering problem.
  2. Good ablation study targeting the three key mechanisms, and a good analysis of the results.
Critical weaknesses
  1. The appendix shows that the frozen region encoder is trained with the environment’s true tile identities, and the region grid lines up exactly with the game’s tiles, so both the segmentation and the separation between object classes are handed to the model from outside, while the authors criticise other object-centric methods for using similar privileged information. This makes the comparison with a pixels-only baseline unfair. Without a run using an encoder trained without labels, I cannot tell whether the model finds new objects or just re-finds the classes it was trained on. This is the decisive issue for my recommendation.
  2. I think that AXIOM should be discussed in the related work, as it grows its object-slot and object-type mixtures in online RL. AdaSlot is not an RL method, but it can infer a variable number of slots.

Heins et al. AXIOM: Learning to Play Games in Minutes with Expanding Object-Centric Models.

Fan et al. Adaptive Slot Attention: Object Discovery with Dynamic Slot Number. CVPR 2024.

Questions for authors
  1. Please clarify the contradiction between the statement in Section 4.4 that “ground truth never enters the model” and Appendix F, which says that ϕ\phi is trained by supervised contrastive learning on tile identities. Were tile labels used to train ϕ\phi? Which labels were used on Procgen Heist, which exposes no tile semantics?
  2. Can you report Crafter results with a ϕ\phi trained without labels? If the prototype census and the stone/iron gains survive, the discovery claim stands.
  3. Please provide more details on how FOCUS was adapted to Crafter. Specifically: the number of slots, the source of the masks, and so on.
  4. What exactly is removed in the “no relations” ablation? Does the “no births” ablation keep the full 12-node graph over the 8 initial prototypes?
Reviewer_kdWdOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses a meaningful problem: how to adapt a world model’s representation to newly encountered objects and accurately model their dynamics during exploration. However, my main concerns lie in the following. (i) The computational overhead of the additional object-processing and graph-learning components is insufficiently characterized. (ii) Although the paper describes prototype discovery, node reassignment, and learned relations, the scope of structural adaptation needs clarification.

Critical strengths
  1. The problem the paper is interesting and important.
Critical weaknesses
  1. The computational overhead of the additional object-processing and graph-learning components is insufficiently characterized. Reporting training time, memory consumption, and inference latency relative to the backbone would help assess the practical trade-off between performance and complexity.
  2. Although the paper describes prototype discovery, node reassignment, and learned relations, the scope of structural adaptation needs clarification.
Questions for authors
  1. Could you clarify what changes structurally in the adaptive object graph during training? Given the fixed node budget, how should we distinguish prototype discovery and node reassignment from growth of the graph itself?
  2. Would the benefits of online prototype discovery persist against a fixed-prototype baseline with the same number of active prototypes, node budget, and visual front end? This comparison would help separate adaptive discovery from increased representational capacity.
  3. What additional training time, memory consumption, and inference latency do the object discovery, relational modelling, and graph integration components introduce relative to DreamerV3?
  4. How would the method handle scenes with multiple instances of the same object type, occlusion, or objects that do not align with the predefined grid? Since nodes represent object types, how is instance-specific information preserved?
  5. Could you reconcile the statement in Section 4.4 that ground truth is used only for evaluation with the supervised contrastive pretraining on tile identities described in Appendix F? Which discovered types were also unseen during front-end pretraining, and how is this additional supervision accounted for in baseline comparisons?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.