- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on incorporating relational structure, temporal structure, and memory into world models. To address the limitations of previous world models, it leverages graph structure to represent observations and encode the relational structure among different objects. The graph structure can also capture memory from previous observations. Through experiments on VQA, dynamic interaction, and some simulation manipulation tasks, the proposed approach outperforms previous baselines.
Critical strengths
1: Using graph structure is a good idea to capture object interactions due to its inductive bias.
2: The paper contains extensive experiments on VQA, dynamic interactions, and manipulation tasks.
3: The paper also contains ablations to study which module is most important to the overall performance.
Critical weaknesses
1: Using graph structure to represent the world has been extensively studied in robotics literature. Here is some related work in robot manipulation: https://arxiv.org/pdf/2209.11943, https://arxiv.org/abs/2109.08771, https://arxiv.org/abs/2012.01693. The authors should carefully study the literature, discuss the differences between this work and prior work, and consider adding some literature to the baselines.
2: In 2.1, the authors mentioned: “N is the fixed number of object slots.” Does this mean the proposed approach can only work for a fixed number of objects? If so, the proposed approach only works for very limited scenarios.
3: The paper only contains one robot manipulation scenario (push-T) and only in simulation. I would encourage the authors to deploy this approach on real-world robots in more diverse tasks.
4: The authors should also explain in more detail how the world model is deployed in robot manipulation. For example, what planning algorithm did you use to achieve the results in Table 2?
Questions for authors
1: Is memory actually important in your proposed approach? In Table 3, performance only decreases by 1% when we remove the memory prediction head.
2: In the current paper, relational structure is added implicitly through graph networks. Have you tried/considered using an explicit relation head that can actually output relations between objects (e.g., on, inside, above)?
3: How are the “token x d” parameters chosen in Table 2? Why did you choose “196 x 384” and “6 x 128”?
Comment by Reviewer atyN
Thanks authors for the rebuttal. My concerns regarding novelty, positioning the paper and citing relevant work have been addressed
However, my primary concern remains the lack of experiments on real-world tasks. Even if they are simple, I encourage the authors to include real-world experiments. For this reason, I am maintaining my original score.