- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to add explicit object and interaction structure to a JEPA-style latent world model. Qwen3-VL and SAM3 provide object names and tracked bounding boxes, while a VLM proposes a graph from motion-salient frames. Object-specific readouts and graph-restricted predictors then regularize the shared representation. A trajectory-based predictive-error test is used to assess graph agreement. The evaluation covers Push-T, OGBench-Cube, and ten LIBERO-Goal tasks, with CEM planning in the first two environments and behavior cloning on a frozen encoder in LIBERO.
Critical strengths
- The pipeline offers a practical alternative to discovering object slots entirely from scratch. Section 4.1 specifies confidence-based interpolation and object-wise episode filtering, and Appendix F.3 provides the VLM and an example graph-discovery prompt. These details make the proposed use of semantic priors reasonably concrete.
- The dense, object-centric, and causal variants share a backbone and training recipe. Figure 4 reports eight-seed planning results on Push-T and Cube: OCCAM obtains 89.5 ± 2.7% and 76.0 ± 3.9%, compared with LeWM at 90.0 ± 3.1% and 75.9 ± 4.4%. Thus, the added structure appears compatible with strong planning performance in these settings, even though it does not improve it.
Critical weaknesses
- The graph validation does not establish the central causal claim. The causal validation is not fully convincing. Appendix F.1 estimates “CMI” using a held-out MSE gap, which measures incremental predictive value rather than conditional mutual information without additional assumptions. Appendix C also acknowledges missing orientation and action–manipulator collinearity, weakening the identifiability conditions. On LIBERO, graph agreement is only F1 = 0.52, with precision 0.37 (Figure 7). The text and caption appear to use different reference graphs when defining recall and precision. Please clarify the metric definitions and validate direct dependencies through controlled simulator interventions.
- The stated architectural guarantees are not demonstrated by the formulation. Equation (8) restricts inputs to selected object projections, but each projection comes from the same global representation. Box supervision therefore does not guarantee that non-parent information is excluded, or that grounding errors remain confined to one object as claimed in Appendix E.2. The implementation is also unclear: Section 4.2 replaces the dense predictor, whereas Algorithm 2 retains it and adds an auxiliary loss; Eq. (3) excludes self-parents despite Appendix C assuming self-loops. Clarifying the rollout path and comparing against sparsity-matched random or simple graphs would strengthen the contribution.
- The strongest result evaluates representations rather than causal planning. The main improvement, 57.0% versus 51.2% on LIBERO (Figure 4), uses a frozen encoder with a BC head rather than world-model planning. It therefore demonstrates representation gains, not directly better causal rollouts. Deterministic evaluation does not eliminate variability from pretraining or BC-head training, so multiple training seeds are needed. Table 3 also shows no improvement on bowl tasks (0.548 versus 0.560), where grounding fails. A corrected-grounding control and multi-object planning evaluation would help establish whether the gains come from causal structure rather than robustness to one annotation failure.
Questions for authors
- Can you specify the exact causal-mode training and rollout path, including self-state inputs, object-state dimensions, the graph-to-embedding loss, and parameter counts for all three variants?
- Which graph is the reference in Figure 7? Please provide the complete edge sets, threshold sensitivity, and evidence from controlled simulator interventions that separates direct from mediated effects.
- Does the LIBERO gain persist across pretraining and BC-head seeds, after correcting the bowl grounding, and relative to simple or random graphs matched for sparsity and supervision?
- Can you demonstrate a benefit of the causal predictor itself on a multi-object planning task, and test predicted object isolation under distractor or object interventions?
4 comments
T4zx_response_2
Response 2 (W2, Q1): architectural guarantee, implementation and sparsity-matched graphs
-
The guarantee, as now stated: each transition head reads only the projected states of its parents (Eq. 8). The projections are linear maps of the shared embedding, s^(n) = W_n z, and grounding a projection to box state does not by itself exclude information from other objects; App. E.2 now says what each head reads and no longer says that a grounding error stays confined to one object.
-
Implementation, one pipeline in Sec 4.2 and Algorithm 2:
- The autoregressive predictor P_θ is trained in every mode as the shared scaffold.
- In causal mode the graph branch g is trained toward P_θ with L_gr = ||P_θ − g||² (λ = 0.05, stop-gradient on P_θ).
- Planning uses g on Cube and P_θ on Push-T; dense and object-centric modes use P_θ.
-
Eq. 3 and self-loops. Each head takes only its parents' states and has no own-state input; Eq. 3 excludes self-edges, and the self-loop in Appendix C is an assumption on the environment dynamics.
-
Alternative graph controls. We additionally evaluate a fully connected graph and a random graph on Push-T, obtaining 89.6 ± 2.1% and 89.8 ± 2.4% planning success, respectively, compared with 89.5 ± 2.7% for the VLM graph over eight evaluation seeds. Thus, planning performance is largely insensitive to the chosen graph topology in this setting. The VLM graph is not presented as a source of improved Push-T success, but as an explicit, interpretable structure whose dependencies can be assessed against observational and interventional evidence.
Related Revised Sections: Sec. 4.2 and Algorithm 2 (training and rollout pipeline; Eq. 9 with L_gr); Appendix C (self-loop assumption); Appendix E.2 (scope of architectural guarantee); Sec. 5.1 (planning path and parameter counts); Appendix E.6 (alternative graph controls, Table 6).
T4zx_response_3
Response 3 (W3, Q3, Q4): representation versus planning, seeds, bowl tasks and the benefit of the graph
-
What the LIBERO-Goal result measures. Language specifies the goal on LIBERO-Goal, so the benchmark is evaluated with a behavior-cloning head on frozen encoders and measures the representation. The revised Sec 5.2 states this. Reported quantities: object-state probing, sign test across objects p = 0.004 (random frame-level split), and BC success with the original head seed (3072): LeWM 0.512, object-centric 0.366, causal 0.570. Linear-probe r by arm (dense, object-centric, causal): Cube 0.835, 0.851, 0.854; LIBERO 0.738, 0.758, 0.768. On Cube the gain sits on the manipulated object (cube-x r 0.749, 0.819, 0.833).
-
Pretraining and BC-head seeds. We extend the LIBERO-Goal evaluation to three paired encoder-pretraining configurations, each evaluated using three independently trained BC-head seeds. The dense/causal success rates are 0.369/0.411, 0.449/0.353, and 0.333/0.497, respectively. Across configurations, the descriptive means are 0.384 for dense and 0.420 for causal, with causal encoders outperforming dense encoders in two of three comparisons. The results therefore retain a positive average effect while showing substantial variability across pretrained encoders. Dense vs. Causal Encoder Comparison (3 head seeds)
| Comparison | Dense Mean | Causal Mean | Δ (Causal − Dense) |
|---|---|---|---|
| Paper encoders | 0.369 | 0.411 | +0.042 |
| Pretraining pair 1 | 0.449 | 0.353 | −0.096 |
| Pretraining pair 2 | 0.333 | 0.497 | +0.164 |
| Overall descriptive mean | 0.384 | 0.420 | +0.037 |
- Bowl tasks. On the bowl tasks the dense arm reaches 0.560, the causal arm 0.548 and the object-centric arm 0.248. On the five tasks that never involve the mis-grounded object, causal reaches 0.592 against 0.464 for LeWM (250 rollouts per arm).
Benefit of Causal structure:
-
An auditable dependency structure: each object's next state is predicted only from its listed parents, checked against CMI (recall 0.73–0.93 on Cube, 0.80 on Push-T, 0.88 on LIBERO) and Push-T interventions.
-
Over dense predictors such as LeWM and DINO-WM, whose dependencies are implicit: an explicit structure at +1.07% parameters with matched planning success.
-
Over C-JEPA, which induces a causal bias by masking without an explicit graph and whose authors note that its influence neighbourhoods are not validated against explicit temporal causal graphs (Nam et al., 2026, Remark 2 and Conclusion): a graph that is explicit and validated.
Related Revised Sections: Abstract and Sec. 1 (LIBERO evaluation scope); Sec. 5.2 (LIBERO results, Table 2, bowl and non-bowl split); Sec. 5.3 (object-state probing); Sec. 2 (causal structure and C-JEPA comparison).
References
- Kipf, T., van der Pol, E., and Welling, M. (2020). Contrastive Learning of Structured World Models. https://arxiv.org/abs/1911.12247.
- Nam, H., Le Lidec, Q., Maes, L., LeCun, Y., and Balestriero, R. (2026). Causal-JEPA: Learning World Models through Object-Level Latent Masking. https://arxiv.org/abs/2602.11389.
- Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. (2021). Towards Causal Representation Learning. https://arxiv.org/abs/2102.11107.
- Wang, Z., Xiao, X., Xu, Z., Zhu, Y., and Stone, P. (2022). Causal Dynamics Learning for Task-Independent State Abstraction. https://arxiv.org/abs/2206.13452.
- Richens, J., and Everitt, T. (2024). Robust Agents Learn Causal World Models. ICLR.
- Liu et al. (2023). Causal Triplet: An Open Challenge for Intervention-centric Causal Representation Learning. CLeaR.
- World Models: A Comprehensive Survey of Architectures, Methodologies, Reasoning Paradigms, and Applications. (2026). arXiv:2606.00133.
- Yuan et al. (2026). Representation World Model: Learning States, Transition and Executable Plans in Representation. arXiv:2609.29171.
Thank you and we are happy to take further questions if any
Dear Reviewer T4zX,
Thank you once again for your insightful review. We have provided a detailed response along with related discussions and revisions in the paper.
With the rebuttal deadline approaching, please feel free to let us know if you have any further questions or concerns. We will be happy to address them in the revision before the deadline.
We sincerely appreciate your time, effort, and valuable comments.
Thanks and regards,
Submission 20795 Authors
T4zx_response_1
Reviewer T4zX
We thank the reviewer. We respond to the main concerns as follows. (1) Metric definitions: the VLM edges are the reference for recall and precision, and we have corrected the Fig. 7 caption in the revised manuscript; we give the full edge sets and threshold-sensitivity tables for all three environments (Q2). (2) Direct versus mediated effects: simulator interventions on Push-T show that the action reaches the block only through the pusher (Q2). (3) Implementation: we state the exact training and rollout path, how self-state is handled, and the parameter counts of all three variants (Q1). (4) Isolation: the guarantee is at the level of what each transition head reads, not of the shared embedding (Q4 and our reply to Reviewer YVrE, Q3).
Response 1 (W1, Q2): graph validation, reference graph, orientation, collinearity and direct dependencies
Reference graph and metrics. The VLM edges are the reference in both Fig. 7 and its text. Recall is the share of VLM edges recovered by CMI; precision is the share of CMI edges that are in the VLM graph. The caption and text now state this, and "false positive" for action→T-block is replaced by "CMI-only edge".
MSE gap versus CMI. Appendix F.1 previously described the leave-one-out MSE difference directly as CMI. We clarify that this difference first measures incremental predictive value. Under a Gaussian predictive-likelihood approximation, we convert the full and leave-one-out prediction errors into a log-likelihood-based dependency score, following the motivation of Wang et al. (2022). The resulting score approximates conditional mutual information under the stated predictive-model assumptions; it is not an assumption-free causal identification statistic.
CMIj→i=21log(1+MSEfullΔj→i)
Orientation. Every tested edge goes from the state at t to the state at t+H, so time fixes its direction. The reverse edge scores lower on Push-T: pusher→T-block 0.022 ± 0.006 against T-block→pusher 0.010 ± 0.005 (5 estimator seeds).
Collinearity and direct dependencies (Push-T simulator interventions, 600 start states, 5 steps). The action's effect on the T-block displacement, by pusher start distance, is 14.48 px (50–80), 11.79 px (80–110), 3.51 px (110–160) and 0.37 px (160–260). With the pusher trajectory held fixed the effect is below 0.002 px. Dropping the action from the conditioning set raises pusher→T-block from 0.023 to 0.198.
LIBERO precision of 0.37. The CMI union has 19 edges against 8 VLM edges, and 7 are shared (recall 0.88, F1 0.52). The one VLM-only edge is action→robot. The 12 CMI-only edges are action→bottle, bowl, plate, rack, stove; bottle→robot; cabinet→bottle, plate; plate→bowl, cabinet; rack→bottle, cabinet.
Cube. robot→cube scores 0.043 ± 0.036 over 5 estimator seeds and is recovered in 1 of 5 seeds at ε = 0.05, 3 of 5 at ε = 0.04 and 4 of 5 at ε = 0.03.
Threshold sensitivity. Recall of the VLM edges is 0.73–0.93 on Cube for ε from 0.02 to 0.10, 0.80 on Push-T at ε = 0.015–0.016, and 0.88 on LIBERO at ε = 0.015–0.03.
Cube (5 estimator seeds)
Push-T (5 estimator seeds)
LIBERO (5 seeds per task, union over tasks)
Related Revised Sections: Sec. 3.3 (conditional-dependence formulation); Sec. 5.3 (CMI validation and Fig. 7); Appendix C (orientation, collinearity and identifiability); Appendix E.4 (Push-T interventions); Appendix F.1 (CMI estimator and threshold sensitivity).