acceptodds
Under review as a conference paper at ICLR 2027

World-as-Graph: Relational World Modeling Through Latent Space Graphs

Abstract

World models aim to learn representations of real-world environments and predict their future evolution. Recent object-centric world models have made expressive progress by representing visual scenes as sets of object-level latent states, but object-object relations are often captured only implicitly, which limits explicit relational and temporal structure modeling and object-centric dynamic memory modeling. To address such challenges, we propose World-As-Graph (WAG), a graph-based object-centric world model that introduces relational inductive bias into JEPA-style predictive representation learning. The proposed WAG contains two main modules: (1) Relation-aware structure induction, which constructs time-varying latent graphs from object-centric slots and designs relation-aware object masking policies to guide relational object representation learning in latent space; (2) Object-centric memory transition, which maintains and updates object-level dynamic states by combining relational information from neighboring objects with historical memory, enabling effective autoregressive future prediction. Extensive experiments on both visual reasoning and robotic manipulation tasks could demonstrate the superior performance of our proposed WAG.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Dynamic graphs, World models, JEPA, Representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DPW6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on incorporating relational structure, temporal structure, and memory into world models. To address the limitations of previous world models, it leverages graph structure to represent observations and encode the relational structure among different objects. The graph structure can also capture memory from previous observations. Through experiments on VQA, dynamic interaction, and some simulation manipulation tasks, the proposed approach outperforms previous baselines.

Critical strengths

1: Using graph structure is a good idea to capture object interactions due to its inductive bias.

2: The paper contains extensive experiments on VQA, dynamic interactions, and manipulation tasks.

3: The paper also contains ablations to study which module is most important to the overall performance.

Critical weaknesses

1: Using graph structure to represent the world has been extensively studied in robotics literature. Here is some related work in robot manipulation: https://arxiv.org/pdf/2209.11943, https://arxiv.org/abs/2109.08771, https://arxiv.org/abs/2012.01693. The authors should carefully study the literature, discuss the differences between this work and prior work, and consider adding some literature to the baselines.

2: In 2.1, the authors mentioned: “N is the fixed number of object slots.” Does this mean the proposed approach can only work for a fixed number of objects? If so, the proposed approach only works for very limited scenarios.

3: The paper only contains one robot manipulation scenario (push-T) and only in simulation. I would encourage the authors to deploy this approach on real-world robots in more diverse tasks.

4: The authors should also explain in more detail how the world model is deployed in robot manipulation. For example, what planning algorithm did you use to achieve the results in Table 2?

Questions for authors

1: Is memory actually important in your proposed approach? In Table 3, performance only decreases by 1% when we remove the memory prediction head.

2: In the current paper, relational structure is added implicitly through graph networks. Have you tried/considered using an explicit relation head that can actually output relations between objects (e.g., on, inside, above)?

3: How are the “token x d” parameters chosen in Table 2? Why did you choose “196 x 384” and “6 x 128”?

Reviewer_atyNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Current Image and Video based JEPA models performed random masking of images/videos and learned to reconstruct representations corresponding to the masked regions in the latent space. The authors conjecture that an object centric view point enables an implicit structure of the scene and helps induce a relational perspective of the interacting objects in the world. So they attempt to avoid random masking and use object centric masking in JEPA style predictive representation learning.

Critical strengths
  1. The idea is sound and helps bridge graph structured dynamics modeling and predictive representation learning.
  2. The presented architecture is simple and light-weight.
Critical weaknesses
  1. Related Work & Positioning:
    • Object-centric notion for future prediction is not new, some recent works worth noting include [1,2,3,4]
    • It is crucial to include and contrast these works as they tried to advocate a similar object-centric representation learning.
  2. Experiments:
    • Although the idea is sound, I am a bit concerned about the experiments
    • Previous works before 2024 attempted to show that their methods work on real-world datasets but this paper still relies on toy datasets such as CLEVERER.
    • I encourage authors to add new experiments to check and compare if the idea works in more realistic settings.
  3. Technical Contribution:
    • Apart from advocating a relational object-centric learning in JEPA style architectures, other technical contributions corresponding to World-as-graph are simple, problem-specific and nothing notably novel. (It is absolutely okay to have simple things and should not be penalized for simplicity)
    • I assume the authors shall encounter some hard engineering/architecture level choices that can potentially add to the novelty when they work on more real-world datasets.

[1] https://arxiv.org/abs/2403.04899

[2] https://arxiv.org/abs/2503.06182

[3] https://arxiv.org/abs/2107.01181

[4] https://cdn.aaai.org/ojs/16402/16402-13-19896-1-2-20210518.pdf

Questions for authors

See weaknesses for more details.

Overall, I am concerned about the experiments, that said, I am also open to discussion and potentially increase the score (in presence of good experimental results)

2 comments
Reviewer_atyNOpenReview ↗

Comment by Reviewer atyN

Thanks authors for the rebuttal. My concerns regarding novelty, positioning the paper and citing relevant work have been addressed

However, my primary concern remains the lack of experiments on real-world tasks. Even if they are simple, I encourage the authors to include real-world experiments. For this reason, I am maintaining my original score.

AuthorsOpenReview ↗

Round-2: Response to Reviewer atyN

We would like to further clarify that we have added a real-world benchmark evaluation in the response above, and here we provide more details about the dataset and our evaluation.

First, we would like to point out that this research area is still at a relatively early stage, and there are still relatively few real-world benchmarks specifically suitable for evaluating world models, especially for future-state prediction and visual reasoning. Following the reviewer’s suggestion, we found the recent AAAI-26 work, JRDB-Reasoning: A Difficulty-Graded Benchmark for Visual Reasoning in Robotics.

JRDB-Reasoning is built on the JRDB dataset, collected by a social mobile robot in real-world indoor and outdoor environments, with an egocentric, human-like view and challenging occlusions and human interactions. It contains 64 minutes of video across 54 sequences from both stationary and moving robot viewpoints. JRDB-Reasoning further adds human-object interaction and geometric relationship annotations for difficulty-graded visual reasoning.

As shown in the table above, in our first-round response, we also evaluated WAG on real-world JRDB videos using shared annotated person tracks, frozen DINOv2 features, the same training targets and update budgets, and compared it with C-JEPA and a no-graph predictor. WAG achieves lower prediction error than both comparators under both observation conditions. With all observed history, WAG achieves 0.7270 ± 0.0039, compared with 0.7975 ± 0.0039 for C-JEPA and 0.7352 ± 0.0007 for the no-graph predictor. With partially masked history, WAG achieves 0.7340 ± 0.0016, compared with 0.7572 ± 0.0033 for C-JEPA and 0.7681 ± 0.0066 for the no-graph predictor. This corresponds to an 8.84% and 3.06% reduction in MSE over C-JEPA, respectively, while using 92.6% fewer predictor parameters.

We hope this additional experiment addresses your concern regarding real-world evaluation. Thank you for your engagement and for all the valuable suggestions. We sincerely hope these additional results address your concerns, and we would be happy to discuss any further questions. We also hope you will consider reconsidering your rating in light of these real-world results.

Reviewer_n1djOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Summary: The authors propose a novel method, WaG, for modeling environment dynamics. It builds a graph over slots obtained from a pretrained, frozen slot-attention-based object-centric encoder. Similarity between nodes is measured with the cosine metric, and nodes are connected using a KNN procedure. The resulting structure is processed by a GNN, while each object’s memory is modeled with a GRU. The model is trained by regressing the masked history slots and the future slots. The nodes to be masked are selected either by graph centrality or by the change in their neighbourhood over time. WaG outperforms JEPA-based and DINO-WM-based baselines on CLEVRER and PushT benchmarks.

Main contribution points:

  1. An object-centric JEPA model enhanced with a graph structure to improve the modeling of object interactions and relations. The approach consists of a graph-building procedure based on KNN over object-centric representation vectors, graph-node masking, and a GRU-based memory module.
  2. Experiments on the CLEVRER and PushT benchmarks show that the proposed WaG method outperforms the JEPA- and DINO-WM-based baselines in terms of both accuracy and training time.
Critical strengths
  1. WaG performs better than or comparably to the baselines while requiring less training time and less VRAM.
  2. A useful ablation study shows the contributions of the different components.
Critical weaknesses

Despite the promising experimental results, I see the following major weaknesses:

  1. The authors do not justify the key ideas of their approach. Which objects are considered similar in WaG: those that look the same, those that are close to each other, or those that interact often? Why is the cosine metric used, and how does it help to model relations between objects? What is the interpretation of centrality-based masking?
  2. I ask the authors to add a graph-based baseline using a simple fully-connected GNN with the same GRU memory, whose nodes are the same frozen slots as in WaG. This experiment would show whether edge pruning via KNN gives any gain over a plain GNN predictor.
  3. Standard deviations or confidence intervals are not provided in Tables 1, 3, 4, 7, etc. Please report results for at least 3 seeds.
  4. A large amount of related work is not cited: C-SWM, G-SWM, SlotGNN, STOVE, OCVP. Some of these models could be used as baselines.

C-SWM: Kipf, van der Pol, Welling, Contrastive Learning of Structured World Models, ICLR 2020

G-SWM: Lin et al., Improving Generative Imagination in Object-Centric World Models, ICML 2020

STOVE: Kossen et al., Structured Object-Aware Physics Prediction for Video Modeling and Planning, ICLR 2020

SlotGNN: Rezazadeh et al., SlotGNN: Unsupervised Discovery of Multi-Object Representations and Visual Dynamics, 2023

OCVP: Villar-Corrales, Wahdan, Behnke, Object-Centric Video Prediction via Decoupling of Object Dynamics and Interactions, ICIP 2023

Questions for authors

See weaknesses.

2 comments

I thank the authors for the detailed response and the additional experiments. I appreciate the effort, and thank you for pointing out that C-SWM is discussed in Appendix A.1. However, some of the new results strengthen my main concern.

  1. If I understand the reported results correctly, the fully connected predictor reaches 91.16 versus 91.37 for K=3. If we assume that its seed standard deviation is comparable to the one you report for WAG_B (0.16), this 0.21 pp difference is within noise. The fully connected predictor is also better than WAG on predictive questions (90.16 vs 89.54). I therefore do not see evidence that KNN-based structure induction improves over a plain GNN predictor.

  2. In the multi-seed table, C-JEPA averages 90.43 ± 0.89 instead of the 89.40 in Table 1, so the overall gain is 0.78 pp (91.21 ± 0.16 vs 90.43 ± 0.89), which is below one standard deviation of the baseline. How many seeds were used, and were the hyperparameters selected on the evaluated split? That said, the lower variance of WAG can be seen as an advantage.

  3. SlotGNN-BN on SAVi (92.23) is within 0.5 pp of WAG (92.34/92.72). On the same SAVi slots, C-JEPA scores 83.88, so WAG's advantage over C-JEPA is 8.84 pp. A simple GNN baseline recovers 8.35 of those 8.84 points (92.23 − 83.88). That leaves at most 0.49 pp for everything specific to WAG: the K-NN graph, the centrality masking and the GRU memory, and that 0.49 pp is a single-seed number. This indicates that the large gain over C-JEPA on SAVi comes from the GNN predictor in general. The C-SWM adaptation results (41.19, 36.67) look like a training failure and are hard to interpret. Please also clarify the PushT numbers: the text states 83.11% vs 79.33%, while the table shows 90.70 vs 80.67. OCVP is mentioned as added but does not appear in the tables.

  4. The answer that similarity may mean appearance, proximity or interaction confirms that the graph is not shown to capture object interactions. An analysis of the edges against CLEVRER collision annotations, or an ablation of the similarity metric, would address this.

In summary, the evidence supports WAG as an efficient and stable GNN+GRU predictor on frozen slots, but not the central claim that relation-aware structure induction and graph-guided masking are responsible for the gains. I therefore intend to keep my rating.

AuthorsOpenReview ↗

Round-2: Response to reviewer n1dj

We thank the reviewer for the follow-up and for recognizing WAG’s efficiency and lower observed variability. We agree that accuracy differences require careful interpretation, but respectfully disagree that differences between complete models provide a causal decomposition or an upper bound on the contribution of WAG’s components.

1. Regarding the fully-connected predictor (91.16 for K=7 vs. 91.37 for K=3), please note that the fully-connected predictor is still a GNN architecture, meaning that it assumes all object pairs are worth constructing connections between. As detailed in our ablation study, this result supports our claim that KNN provides a sparse and dynamically induced relational inductive bias, which allows the predictor to operate on learned object-to-object affinities. At the same time, it significantly reduces training time and improves computational efficiency (2.4× faster training and up to 16.4× lower peak GPU memory).

Thus, we believe that evaluating WAG solely by the 0.21 pp accuracy difference misses an important aspect of our contribution: the goal is not simply to obtain a marginally higher accuracy than a fully connected GNN, but to introduce an efficient, sparse, and dynamically induced relational inductive bias for object-centric world modeling.

2. We do not think it is appropriate to directly compare the **[0.78 pp improvement]**with the [0.89 std] of C-JEPA. This is not a statistical test of significance. The uncertainty of the estimated mean depends on the number of seeds, and when the same seeds are used, paired seed-wise differences provide a more appropriate comparison. We will report the number of seeds and the corresponding statistics more explicitly. We agree that the substantially lower variance of WAG is an important advantage, and appreciate the reviewer highlighting this point.

3. Most importantly, we do not think the proposed 8.84 = 8.35 + 0.49 decomposition provides an upper bound on the contribution of WAG-specific components. SlotGNN-BN cannot be directly applied to our setting. We made additional adaptations to its architecture to make it work with our object-centric slots and prediction setting. It is a different predictor, rather than WAG with only its proposed components removed. Therefore, the 8.35 pp gain should not simply be treated as the gain of a generic GNN predictor and subtracted from WAG's gain. The 0.49 pp difference is a difference between two adapted systems, not an upper bound on the contribution of WAG's relational structure, masking, and memory mechanisms.

And the strong performance of SlotGNN-BN is consistent with our motivation for introducing relational inductive bias into object-centric prediction. WAG focuses on how that structure is constructed and used, through dynamic KNN graphs, relation-aware masking, and object-centric memory.

4. Reporting Thanks for pointing out the reporting issue. For clarity, under the additional three-seed PushT evaluation, WAG achieves 83.11 ± 0.77 versus 79.33 ± 1.33 for SlotGNN-BN. The additional experimental baselines are C-SWM and SlotGNN-BN, while OCVP is discussed as related work.

We sincerely thank you for all the comments and suggestions, which we will incorporate into the revised manuscript. We also greatly appreciate your active engagement in this discussion. We sincerely hope that these open discussions can help further advance the exploration of this still-early research direction. We are grateful for the time and effort you have devoted to reviewing our work, and we hope you may have a chance to reconsider the score. We would also be very happy to continue the discussion with you.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Common Response to All Reviewers

We thank all reviewers for their careful reviews and constructive suggestions. We appreciate their recognition of the following strengths:

  1. A graph-enhanced object-centric JEPA framework, combining graph-guided masking with GRU-based object memory (Reviewer n1dj);

  2. A sound connection between relational dynamics and predictive representation learning, with graph structure providing a useful inductive bias for object interactions (Reviewers atyN and Reviewer DPW6);

  3. A simple, lightweight architecture (Reviewer atyN), achieving better or comparable performance with less training time and GPU memory (Reviewer n1dj); and

  4. Complementary reasoning and manipulation evaluations with informative component ablations (Reviewers n1dj and Reviewers DPW6).

Following the reviewers’ suggestions, our responses provide additional experiments, including: (1) Fully connected slot relation exploration; (2) Adapting and reimplementing existing methods, shared-slot C-SWM and SlotGNN, for more baseline comparisons; and (3) Real-world video JRDB future-feature prediction experiments. We have also clarified the distinction from semantic relation forecasting and skill-level robotics methods. We will incorporate these improvements into the revised manuscript.

We would like to reiterate that, to the best of our knowledge, this is the first work that brings graph machine learning, object-centric representation learning, and world modeling into a unified framework for exploring relational inductive bias in world models. We would like to emphasize that this area is still at a very early stage. We believe and hope that, to better develop this direction, it deserves more exploration at this stage from a graph learning perspective. Our work is not only about “simple” representation learning; we also include complex rollouts from representation to graph to VQA/Robot Manipulation.

Therefore, we hope the contribution of this work can be viewed not only through the complexity of individual architectural components, but also through the problem setting and the complete learning pipeline that we establish. We appreciate the reviewers' feedback, which has helped us clarify these points and better position the work as an early exploration of this direction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.