acceptodds
Under review as a conference paper at ICLR 2027

ReL-WM: Grounding World Model in Progress-Aware Scene Graphs

Abstract

Successful robot manipulation depends on how interactions between entities evolve toward task completion, yet standard learned world models do not explicitly represent this structure. We introduce ReL-WM (Relational Latent World Model), a framework that grounds latent dynamics in progress-aware scene graphs. These scene graphs retain task-relevant entities and describe their physical interactions, spatial configurations, and affordance compatibility, while temporal qualifiers capture intermediate relational changes. Using scene graph supervision, ReL-WM learns a semantic state that evolves alongside visual dynamics during imagination. Relational milestones extracted from successful demonstrations define a task-progress potential, whose predicted changes guide policy learning. Our experiments show improved success over visual and object-centric model-based baselines across tabletop and mobile manipulation, together with stronger transfer to new objects and zero-shot generalization to unseen layouts and lighting conditions. These findings support explicitly modeling interactions and task advancement as a useful foundation for world models in robot manipulation.

Primary area:
reinforcement learning
Keywords:
Model-based Reinforcement Learning, World Models, Robot Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yQupOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReL-WM adds a relational state to DreamerV3. Scene graphs describe object positions, physical interactions, and changes in task progress. The model learns to predict this information during imagined rollouts. Milestones extracted from demonstrations provide an additional reward-shaping signal. Experiments cover tabletop manipulation, mobile manipulation, object transfer, and changes in scene layout and lighting.

Critical strengths
  1. Modeling interaction states such as contact, grasp, and support is a useful direction for manipulation world models. These states can distinguish situations that look similar in images but require different actions.

  2. The experiments show consistent improvements across several settings. The ablations also separate progress prediction from reward shaping: on PlaceSphere, the method reaches 78.2% success without shaping and 86.7% with the default shaping weight. This helps show that the gains are not entirely due to the added reward.

Critical weaknesses
  1. ReL-WM receives simulator-derived graphs during both training and execution. These graphs use object poses, contact forces, and grasp predicates. The paper does not include a baseline given the same information without the proposed relational dynamics. Removing the semantic branch also removes the graph input, so that ablation does not isolate the benefit of predicting relations. The lighting results are similarly difficult to interpret because simulator-derived relations remain available when images become harder to read.

  2. The real-robot experiments do not yet resolve this concern. Appendix A.1 evaluates an imitation policy trained on frozen world-model features, rather than the proposed imagination-based RL agent. It also does not clearly explain how the real-robot graphs and progress labels are obtained during training and execution. These details are needed to understand what information the deployed policy requires and how practical the approach is outside simulation.

Questions for authors
  1. How does ReL-WM compare with DreamerV3 given the same simulator information or graph embeddings, but without a separate relational dynamics model? Can you retain graph input while removing only relational prediction?

  2. How are scene graphs constructed in the real-robot experiments? Which quantities are measured, estimated, or manually annotated, and which are needed at inference?

  3. Which parts of the progress schedule are generated automatically from the five demonstrations?

  4. Do the main ablation results hold beyond PlaceSphere, particularly on peg insertion or mobile manipulation? It would also help to test missing objects or incorrect grasp states, beyond changing relation labels to neighboring categories.

5 comments

Thank you for your detailed, thoughtful, and constructive review. We address your questions and concerns below, and all locations refer to the revised PDF.

Q1 / W1. Additional baseline. We added two baselines with matched graph information.

  • DreamerV3 + flat graph receives the same node attributes, relation labels, and temporal labels as ReL-WM during training and acting, and the same progress supervision during training. It encodes flattened graphs with an MLP and then standard DreamerV3 latent dynamics, without a separate relational dynamics model (Page 41, Lines 2160–2169).
  • ReL-WM w/o semantic branch retains the graph input and our encoder and removes only the separate semantic branch with its relational prior (Pages 8–9, Lines 429–442). Its graph embedding enters the stochastic latent together with the image, so it reconstructs and predicts the relations through the shared latent state.

Tables 3 and 8 report the comparisons, summarized below

Success (%)PlaceSpherePegInsertionSideMS-HAB
DreamerV33.10.845.7
DreamerV3 + flat graph31.29.749.8
ReL-WM w/o semantic branch57.538.458.9
ReL-WM86.790.865.3

Both variants remain below ReL-WM on every task, so access to the graph does not by itself explain the gain. Under both MS-HAB lighting changes, both matched variants receive the same graphs as ReL-WM, ReL-WM still leads, and its gap to ReL-WM w/o semantic branch widens (Table 8, Page 19, Line 973, and Page 20, Lines 1030–1032).

Q2 / W2. Real-robot scene graphs. A VLM takes the place of the simulator, and no quantity is annotated manually (Page 15, Lines 787–797).

  • Before training. The VLM reads the task instruction and five demonstrations and writes a task specification. It selects the entities and relations (Table 6, Page 16, Line 828) and sets the definition and thresholds of every relation label (Table 7, Page 17, Line 865), so no number in Table 7 is set by hand.
  • Labelling. During training and inference, the VLM labels the graph at every step from both cameras and the proprioceptive state of the current and five previous steps, so it never sees future frames.
  • Sources. Table 5 (Page 16, Line 811) lists the source of every quantity. Camera frames and proprioceptive state are measured, entity boxes and relation labels are estimated by the VLM, centroids are back-projected from depth, and progress labels are computed from the graphs.
  • Inference. The same labeller produces the graphs for training and inference, so inference needs the camera frames and proprioceptive state but no progress label.

ReL-WM also predicts the VLM-labelled relations of the held-out robot demonstrations more accurately than both matched variants and a persistence baseline (Table 11, Page 21, Line 1114).

Q3. Automatic progress schedule. Every stage is generated automatically, and Table 17 (Page 30, Line 1567) lists each stage with its input and output.

  • Entities. The VLM selects the task-relevant entities from the instruction and frames of the five demonstrations.
  • Assets and milestones. The five demonstrations provide the interaction types, affordance components, thresholds, and change scales, together with the recurring milestones and their onset order.
  • Schedule. The VLM organizes these milestones into weighted phases with completion predicates.

The labelling prompt and its output format are included in Appendix A.1 (Page 15, Line 805 to Page 18, Line 918), and Appendix D lists the generated schedules (Page 41, Line 2206).

Q4. Ablations beyond PlaceSphere and graph errors.

  • More tasks. We repeat every ablation on PegInsertionSide and in MS-HAB (Pages 19–20, Lines 1022–1032).
  • Graph errors. Missing objects remove an object with all its relations, and flipped grasps invert the grasp label, both at 20 random steps of each training and evaluation episode. The flipped-grasp runs receive no grasp flag from the environment, so the flag cannot correct the flipped label (Page 41, Lines 2193–2201).

Tables 3 and 9 report the ablations, summarized below

Success (%)PlaceSpherePegInsertionSideMS-HAB
DreamerV3 + flat graph (clean graphs)31.29.749.8
ReL-WM w/o progress71.947.252.7
ReL-WM w/o shaping (β = 0)78.263.557.4
ReL-WM + missing objects66.868.656.9
ReL-WM + flipped grasps70.875.458.3
ReL-WM86.790.865.3

On all three tasks, predicting progress without shaping improves over removing progress, and shaping adds a further gain. With corrupted graphs, ReL-WM still outperforms DreamerV3 + flat graph with clean graphs on all three tasks.

We hope that these results address your concerns, and we would be grateful if you would reconsider your score.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer yQup,

We hope this message finds you well. We try to address your concerns. Should there be any concerns or points you wish us to clarify further? Please do not hesitate to reach out. We are more than willing to extend our conversation and eagerly anticipate any further discussions that may arise.

Best regards,

All Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer yQup,

We hope this message finds you well. We understand that you are busy with revision for your paper, but can you spend a little time to check out our rebuttal ? This work is very important for us too. We are more than willing to extend our conversation and eagerly anticipate any further discussions that may arise.

Best regards,

All Authors

Reviewer_4dS9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReL-WM, which introduces a scene-graph-supervised semantic latent state into DreamerV3 and predicts how it evolves in response to actions. The scene graphs describe spatial relations, physical interactions, affordance compatibility, and temporal changes between entities. The authors further use a small number of successful demonstrations to provide supervision for task-progress prediction and reward shaping.

Critical strengths

The paper addresses a practically meaningful problem. In robotic manipulation, relations such as contact, grasping, and support directly influence the outcomes of subsequent actions. Incorporating these relations into world models is a reasonable and promising direction. The paper is well written and generally clear.

Critical weaknesses

1.ReL-WM uses privileged simulator information to construct scene graphs during both training and execution, and its policy also relies on the graph posterior. Therefore, compared with the reported baselines, the method introduces both structured information and a new model architecture. My concern is that the main text does not present a baseline receiving the same scene-graph information and progress supervision, leaving the source of the performance gains unclear. The “without semantic latent dynamics” variant in Table 3 also removes the semantic latent state and the associated modeling components, so it does not isolate the necessity of predicting relational dynamics.

2.Visual world models may confuse states such as contact and secure grasping, leading to physically inconsistent predictions. A key question for this paper is therefore: does incorporating relational semantics enable the world model to predict changes in interaction states more accurately? The main text primarily presents task success rates, visualizations of relations along successful trajectories, and component ablations. Figure 4 helps illustrate the interpretability of the representation, but it does not by itself demonstrate that the model can accurately predict these relations when future observations are unavailable. Higher task success rates alone are also insufficient to establish that the gains come from better modeling of interaction dynamics.

Questions for authors

Please see the specific concerns and questions raised in the Weaknesses section.

4 comments

Thank you for your detailed, thoughtful, and constructive review. We address your questions and concerns below, and all locations refer to the revised PDF.

W1. Additional baseline. We added two baselines with matched graph information.

  • DreamerV3 + flat graph receives the same node attributes, relation labels, and temporal labels as ReL-WM during training and acting, and the same progress supervision during training. It encodes flattened graphs with an MLP and then standard DreamerV3 latent dynamics, without a semantic branch (Page 41, Lines 2160–2169).
  • ReL-WM w/o semantic branch keeps our encoder and removes only the semantic branch (Pages 8–9, Lines 429–442).

Tables 3 and 8 report the comparisons, summarized below

Success (%)ParamsPlaceSpherePegInsertionSideMS-HAB
DreamerV350.1M3.10.845.7
DreamerV3 + flat graph59.6M31.29.749.8
ReL-WM w/o semantic branch54.2M57.538.458.9
ReL-WM57.9M86.790.865.3

The advantage of ReL-WM remains under matched information, supervision, and budget. Because ReL-WM w/o semantic branch keeps the graph input, its gap to ReL-WM is consistent with the benefit of a separate semantic state with its own objectives rather than a latent state shared with the image.

W2. Predicting interaction changes without future observations. We evaluate this directly with prior-only prediction (Pages 20–21, Lines 1061–1133).

  • Protocol. Five held-out real-robot demonstrations per task and 100 held-out episodes each of PlaceSphere and PegInsertionSide (Page 20, Lines 1061–1066). Each model starts from the same observed prefix and predicts 15 steps from the recorded actions without future images or graphs.
  • Interaction changes. We detect four changes in the predicted relations, which are grasp acquisition (a secure grasp begins, the transition that you highlight), grasp loss (the grasp ends), support onset (one object starts to rest on another), and containment onset (the peg enters the hole or a cube enters the cup). A predicted change is correct when the recorded episode has the same change for the same objects within one step. We report event F1, which is high only when a model catches the recorded changes without predicting false ones.
  • Relation macro-F1. After 15 prediction steps, we compare every predicted relation label with the recorded one and average the F1 score over labels and relation types, so rare interaction labels count as much as frequent ones.
  • Persistence baseline. This baseline copies the last observed relations to all 15 future steps. It never predicts a change, so its event F1 is zero.

The first table summarizes the held-out simulated episodes, and the second summarizes the held-out real-robot demonstrations. In both tables, the first four columns are event F1 (%) and the last column is relation macro-F1 (%).

SimulationGrasp acquisitionGrasp lossSupport onsetContainment onsetRelation macro-F1, h=15
Persistence0.00.00.00.048.9
DreamerV3 + flat graph24.819.322.615.438.6
ReL-WM w/o semantic branch43.636.240.731.849.7
ReL-WM83.776.980.374.678.4
Real robotGrasp acquisitionGrasp lossSupport onsetContainment onsetRelation macro-F1, h=15
Persistence0.00.00.00.041.2
DreamerV3 + flat graph17.413.912.510.831.7
ReL-WM w/o semantic branch33.227.625.022.439.8
ReL-WM66.160.457.053.671.7

Without any future image or graph, ReL-WM achieves the highest event F1 for every type of interaction change and the highest relation macro-F1 after 15 prediction steps, both in simulation and on the labelled real-robot demonstrations (Tables 10 and 11, Page 21, Lines 1101 and 1114). DreamerV3 + flat graph and ReL-WM w/o semantic branch receive the same graph as ReL-WM, so this advantage is consistent with how ReL-WM models relations rather than with access to the graph.

We hope that these results address your concerns, and we would be grateful if you would reconsider your score.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 4dS9,

We hope this message finds you well. We try to address your concerns. Should there be any concerns or points you wish us to clarify further? Please do not hesitate to reach out. We are more than willing to extend our conversation and eagerly anticipate any further discussions that may arise.

Best regards,

All Authors

Thank you for the response.After considering it,I’ll keep my original score.

Thank you for your contributive comments. They help us a lot to improve the quality of our paper. If you have any further question, please feel free to reach out. We are very happy to further discuss and improve our work.

Reviewer_1NebOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ReL-WM, which augments DreamerV3's visual latent dynamics with a semantic state grounded in progress-aware scene graphs. The graph encodes persistent entities, physical interactions, spatial relations, affordance compatibility, and temporal changes. At execution the model uses a graph posterior, while during imagination a learned semantic prior predicts the next semantic state. From five successful demonstrations per task, the authors extract relational milestones and build a bounded progress potential whose changes provide an auxiliary reward for policy learning.The system is evaluated on manipulation tasks in ManiSkill and MS-HAB, including multi-object learning, fine-tuned object transfer, and zero-shot scene and illumination generalization. Appendix reports imitation learning with frozen world-model representations in simulation and on a real robot. The full system outperforms the evaluated baselines by large margins. However, the current controls do not fully separate the contribution of extra structured information from that of relational dynamics modeling, and the accuracy of multi-step relational prediction during imagination is not directly validated.

Critical strengths

1.The full system achieves large gains across several manipulation settings, which supports the practical value of the direction. In Table 1, ReL-WM reaches 41.3% average success in multi-object learning vs. 21.7% for DreamerV3. After 3M additional environment steps for object transfer, the averages are 36.4% vs. 14.9%. In Table 2, success on unseen scenes is 56.4% vs. 34.8%. These results cover changes in objects and environments, although they do not by themselves identify which component drives the gain.

2.The paper makes a serious attempt to separate progress supervision, reward shaping, and the scope of the theoretical guarantee. Table 3 includes a variant without the progress head, a variant with progress prediction but β = 0, and variants with progress shaping. Proposition 1 and Appendix E limit policy invariance to a fixed imagined model and note that the raw progress advantage is zero with an exact progress critic. This is a useful methodological clarification, because it avoids equating policy invariance with guaranteed training improvement.

Critical weaknesses

1.The comparison does not adequately separate extra structured input from the proposed relational dynamics modeling. Appendix B.1.1 states that the scene graph uses privileged information such as object poses, contact forces, and simulator grasp predicates, and that these graphs are available during both training and acting. Appendix C.3 matches tasks, rewards, control interfaces, and interaction budgets, but it does not report a control that gives the baselines the same graph information and demonstration-derived supervision. In Table 3, "w/o semantic latent dynamics" drops from 86.7 ± 6.3% for the full model to 24.9 ± 5.5%, but this variant removes both the semantic latent state and graph dynamics modeling. The gap therefore cannot be attributed to learned relational dynamics alone. The illumination experiments have a similar limitation, because simulator-generated structured information does not undergo the same illumination degradation as RGB inputs. A matched-information and matched-supervision comparison with a simple baseline that encodes the same graph attributes is needed before the core methodological claim can be supported.

2.The paper does not directly validate whether relational dynamics are accurate during imagination, which is the central claim. Appendix A.2 and Figure 8 report node and relation-label reconstruction accuracy from the graph posterior on training batches, eventually above 99%. This shows that the semantic state preserves information in the input graph. It does not show that the semantic prior, which receives no future graph observations, predicts relation changes accurately. The perturbation analysis in Appendix A.3 shows that the encoder is sensitive to relation changes, but it also does not test future dynamics. The current results lack multi-step relation prediction on held-out trajectories, errors on key interaction transitions, and error growth with imagination length. The evidence for improved success is therefore stronger than the evidence that the world model predicts physical interactions more accurately. Direct validation of the semantic prior is needed to determine whether this module actually alleviates the interaction-modeling problem emphasized in the paper.

3.The manuscript still contains several unfinished editing artifacts. On page 3 at the end of Section 3, the sentence "The subsection currently risks appearing ad h" is incomplete. The same page has "vertex counts.Formally" with a missing space. Section 4.1 on page 5 begins with "Eq. The semantic posterior" and an incomplete equation reference. On page 14, "Table 4 reports success rates..." appears twice in a row. These issues affect readability and the impression of completion, although they are straightforward to fix.

Questions for authors

1.Please describe exactly which low-dimensional task states, object information, and demonstration-derived supervision each baseline receives. Please also specify what "w/o semantic latent dynamics" keeps and removes in terms of inputs, losses, and prediction heads. Can the authors add a DreamerV3 baseline that receives the same node attributes, relation labels, temporal information, and progress supervision, uses a simple encoder and standard latent dynamics, and has no separate semantic dynamics branch? With matched interaction budgets and reported model sizes, does the advantage of ReL-WM remain?

2.Can the authors evaluate prior-only multi-step prediction on held-out real-environment trajectories, starting from the same initial posterior and using recorded actions, over the H = 15 imagination length used in training? Please report prediction quality for key relational transitions such as grasp acquisition and loss, support establishment, and insertion, as well as error as a function of prediction step. Please also include a persistence baseline that keeps the current relation unchanged. Does the semantic prior learn useful action-conditioned dynamics rather than mainly reproducing relation persistence?

3.Please clean up the incomplete sentences, missing spaces, broken equation references, and repeated sentences noted above.

4 comments

Thank you for your detailed, thoughtful, and constructive review. We address your questions and concerns below, and all locations refer to the revised PDF.

Q1 / W1. Terminology clarification. Low-dimensional state follows the default ManiSkill and MS-HAB settings. In ManiSkill, low-dimensional state contains the joint state and the TCP pose, and PickCube and PlaceSphere add the goal or bin position and the grasp flag. In MS-HAB, low-dimensional state also contains the target-object pose, the goal position, and the grasp flag. DreamerV3, TD-MPC2, and SOLD use no demonstration-derived supervision. We revised Appendix C.2 (Page 38, Lines 2038–2048) to state the low-dimensional state of each benchmark, and Table 19 (Page 39, Line 2083) to list the graph input and supervision of every method.

Q1 / W1. Additional baseline. We added two baselines with matched graph information and renamed one existing variant.

  • DreamerV3 + flat graph receives the same node attributes, relation labels, and temporal labels as ReL-WM during training and acting, and the same progress supervision during training. It encodes flattened graphs with an MLP and then standard DreamerV3 latent dynamics, without a semantic branch (Page 41, Lines 2160–2169).
  • ReL-WM w/o semantic branch keeps our encoder and removes only the semantic branch (Pages 8–9, Lines 429–442).
  • DreamerV3 + progress reward is the renamed "w/o semantic latent dynamics". It keeps the inputs, losses, and heads of DreamerV3 and adds the graph-derived progress reward to the environment reward. It has no graph input, semantic state, progress head, or progress critic.

All models use the same interaction budget. Tables 3 and 8 report the comparisons, summarized below

Success (%)ParamsPlaceSpherePegInsertionSideMS-HAB
DreamerV350.1M3.10.845.7
DreamerV3 + flat graph59.6M31.29.749.8
ReL-WM w/o semantic branch54.2M57.538.458.9
ReL-WM57.9M86.790.865.3

The advantage of ReL-WM remains under matched information, supervision, and budget, and ReL-WM even has fewer parameters than DreamerV3 + flat graph (Table 8, Page 19, Line 973). Access to the graph therefore does not by itself explain the gain. Under both MS-HAB lighting changes, both matched variants receive the same graphs as ReL-WM, and ReL-WM still leads (Table 8, Page 19, Line 973, and Page 20, Lines 1030–1032).

Q2 / W2. Prior-only multi-step prediction. We added this evaluation in Appendix A.5.

  • Protocol. Five held-out real-robot demonstrations per task and 100 held-out episodes each of PlaceSphere and PegInsertionSide (Page 20, Lines 1061–1066). Each model starts from the same observed prefix and predicts 15 steps from the recorded actions without future images or graphs.
  • Relations. Tables 10 and 11 (Page 21, Lines 1101 and 1114) report relation macro-F1 at h = 0, 1, 5, 10, and 15 against a persistence baseline, together with event F1 for grasp acquisition, grasp loss, support onset, and containment onset. Containment onset marks the peg entering the hole in simulation and a cube entering the cup on the robot.
  • Images. Tables 12 and 13 (Page 22, Lines 1135 and 1146) report the PSNR and LPIPS of the same rollouts.

We provide a summary below

Sim. Relation F1, h=0Sim. Relation F1, h=1Sim. Relation F1, h=15Sim. Grasp-acq. event F1Robot relation F1, h=15Sim. LPIPS, h=15
Persistence–90.448.90.041.2–
DreamerV3–––––0.161
DreamerV3 + flat graph84.779.238.624.831.70.149
ReL-WM w/o semantic branch95.190.249.743.639.80.118
ReL-WM97.495.378.483.771.70.073

The macro-F1 gap between ReL-WM and ReL-WM w/o semantic branch grows from 2.3 points at the observed posterior (97.4 vs. 95.1) to 28.7 points after 15 prediction steps (78.4 vs. 49.7). This supports better preservation and prediction of relations during open-loop rollouts, and together with the event F1 it suggests that the semantic prior predicts relational transitions rather than mainly reproducing persistence. At h = 15, ReL-WM also has the lowest LPIPS in simulation and on the real robot, less than half that of DreamerV3 in simulation.

Q3 / W3. Thank you for pointing out the editing artifacts. We fixed all four and proofread the full manuscript.

LocationBeforeAfter
Sec. 3"The subsection currently risks appearing ad h"Removed (Page 4, Line 202)
Sec. 3"vertex counts.Formally"Space added (Page 3, Line 141)
Sec. 4.1"Eq. The semantic posterior""The graph encoder" (Page 4, Line 215)
App. A.1"Table 4 reports success rates..." twiceDuplicate removed (Page 15, Line 776)

We hope that these results address your concerns, and we would be grateful if you would reconsider your score.

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 1Neb,

We hope this message finds you well. We try to address your concerns. Should there be any concerns or points you wish us to clarify further? Please do not hesitate to reach out. We are more than willing to extend our conversation and eagerly anticipate any further discussions that may arise.

Best regards,

All Authors

AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewer 1Neb,

We hope this message finds you well. We understand that you are busy with revision for your paper, but can you spend a little time to check out our rebuttal ? This work is very important for us too. We are more than willing to extend our conversation and eagerly anticipate any further discussions that may arise.

Best regards,

All Authors

Dear reviewer 1Neb,

As the time for rebuttal is gonna end soon, it is our pleasure if we can have your feedback. We strongly believe that we have already solve all your questions. If you can reconsider the score, it is our pleasure.

Again, thank you for your contributive comments. You help us a lot in improving our paper quality.

The Authors.

Comments on the paper

2 comments

We thank all reviewers for their detailed, thoughtful, and constructive reviews. The reviews share two main concerns. The first is whether the gain of ReL-WM comes from the additional graph information or from how the model represents and predicts relations. The second is whether the semantic prior predicts interaction changes when future observations are unavailable. We address both concerns with new experiments and list the revisions below. New and revised content in the revised PDF is in blue, and line numbers are those printed in the margin.

Additional baselines

ChangeWhereRaised by
New baseline DreamerV3 + flat graph, which receives the same graph labels as ReL-WM during training and acting and the same progress supervision during trainingPage 8, Line 429 to Page 9, Line 442; Page 41, Lines 2160–21691Neb Q1, 4dS9 W1, yQup Q1
New variant ReL-WM w/o semantic branch, which keeps our relation-aware encoder and removes only the separate semantic branchPage 8, Line 429 to Page 9, Line 442; Page 41, Lines 2160–21691Neb Q1, 4dS9 W1, yQup Q1
Table 3 extended with both variants, and the former "w/o semantic latent dynamics" renamed DreamerV3 + progress rewardTable 3, Page 9, Line 4611Neb Q1, 4dS9 W1
New Table 8 with both variants on PegInsertionSide and in MS-HAB under unseen scenes and lighting changes, together with parameter countsTable 8, Page 19, Line 973; Page 19, Line 1016 to Page 20, Line 10321Neb Q1, 4dS9 W1, yQup Q1
New Table 19 with the graph input and supervision of every method, and the low-dimensional state of each benchmark added to Appendix C.2Table 19, Page 39, Line 2083; Page 38, Lines 2038–20481Neb Q1

Prior-only prediction

ChangeWhereRaised by
New Appendix A.5, which rolls each model 15 steps from the recorded actions without future images or graphs on held-out simulated episodes and held-out real-robot demonstrationsPage 20, Line 1059 to Page 21, Line 11331Neb Q2, 4dS9 W2
New Tables 10 and 11 with relation macro-F1 from h = 0 to 15 and event F1 for grasp acquisition, grasp loss, support onset, and containment onset, against a persistence baselineTables 10 and 11, Page 21, Lines 1101 and 11141Neb Q2, 4dS9 W2
New Tables 12 and 13 with the image prediction of the same rollouts by PSNR and LPIPSTables 12 and 13, Page 22, Lines 1135 and 11461Neb Q2

Ablations and graph errors

ChangeWhereRaised by
New Table 9, which repeats every ablation on PegInsertionSide and in the MS-HAB training sceneTable 9, Page 20, Line 1035; Page 19, Line 1016 to Page 20, Line 1032yQup Q4
New graph-error variants that remove objects or flip the grasp label at 20 random steps of each training and evaluation episode, added to Tables 3 and 9Page 9, Lines 465–474; Page 41, Lines 2193–2201yQup Q4

Real-robot pipeline and presentation

ChangeWhereRaised by
New description of the real-robot scene graphs, which a VLM labels at every step from the camera frames and robot state of a causal windowPage 15, Lines 787–797yQup Q2
New Tables 5 to 7 with the source of every quantity and the facts, label definitions, and thresholds that the VLM generates, together with the labelling prompt and outputTables 5 to 7, Page 16, Lines 811 and 828; Page 17, Line 865; Page 15, Line 805 to Page 17, Line 913yQup Q2
New Table 17 with every stage of the progress schedule and its input and outputPage 29, Lines 1559–1562; Table 17, Page 30, Line 1567yQup Q3
Editing artifacts fixed and the full manuscript proofreadThroughout1Neb Q3
AuthorsOpenReview ↗

Gentle Reminder

Dear Reviewers,

We hope this message finds you well. We have tried our best to address your concerns. As the discussion will end very soon on November 18, we are grateful if you can consider our rebuttal and give us your following up comments. We are more than willing to extend our conversation and eagerly anticipate any further discussions that may arise.

Best regards,

All Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.