- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes EventCone-WM, which compares candidate-action and matched-control latent rollouts, predicts typed entity relations, and propagates them into finite-horizon event cones. The cones suppress unsupported changes, preserve reachable effects, and guide MPC. Across 36,000 simulated episodes, the method improves mean OOD success from 61.4% to 76.8% over JEPA-WM while reducing cone false positives and collisions. The evaluation is unusually comprehensive, but the method relies heavily on entity decomposition, event supervision, and simulator-derived action-pair labels, limiting its demonstrated applicability beyond controlled simulation.
Critical strengths
- Novel and intuitive event-cone formulation for localizing action effects.
- Large evaluation and extensive ablations.
- Strong and consistent improvements across three environments and three OOD shifts.
- It can be integrated into existing latent world models such as JEPA-WM without redesigning the world-model backbone.
Critical weaknesses
- Strong dependence on entity supervision. Performance drops from 76.8% with oracle entities to 73.9% with tracking and 69.8% with image-only tokens, which still require event or pseudo-label supervision. Entity splitting, merging, and occlusion remain major barriers to real-robot deployment.
- Training requires contact, support, and other event labels that are readily available in simulation but costly to obtain in real settings.
- Event-graph propagation scales quadratically with the number of entities, causing rapidly increasing latency and memory usage.
- All experiments use rigid-body simulation, with no real-robot evaluation.
Questions for authors
- Can the method be trained without simulator event labels or geometry-derived pseudo-labels?
- The method is highly sensitive to entity segmentation and tracking quality. How could entity splitting, merging, and occlusion errors be mitigated in real-robot deployment?
- How is a matched future camera/proprioceptive control trajectory constructed during online deployment?
- Can the authors provide real-robot experiments or evaluation on unseen tasks and embodiments?
Response to Reviewer X96s: Event Supervision, Tracking Robustness, Scaling, and Online Controls
We thank Reviewer X96s for the encouraging review and for highlighting the intuitiveness of the event-cone formulation, our evaluation breadth, and plug-in compatibility with existing JEPA world models. We address each question below.
1. Reducing Reliance on Simulator Event Labels (W2 & Q1)
Event supervision is used strictly at training time (Table 7) and never at inference. Even under 40% corrupted event labels (Table 25), EventCone-WM retains 71.1% OOD success (+9.7 pp over JEPA-WM; +3.4 pp over EventHead), supported by confidence weighting wi,h (Eq. 17) and the reachability curriculum (Appendix B.3).
For real-world training without simulator or geometry pseudo-labels, two pathways apply: (i) proprioceptive/motion self-supervision, detecting robot-object contact from force-torque or gripper-current spikes and object-object interactions from correlated 3D flow/point-cloud discontinuities; and (ii) sim-to-real decoder transfer, pretraining the lightweight event decoder in simulation atop frozen visual encoders (e.g., DINOv2 / V-JEPA 2) and adapting to real trajectories while Eq. (17) down-weights uncertain edges.
2. Mitigating Entity Splitting, Merging, and Occlusion (W1 & Q2)
Although absolute success drops from oracle entities (76.8%) to tracked proposals (73.9%) and image-only tokens (69.8%) (Table 8), EventCone-WM retains large relative gains over JEPA-WM across all interfaces (+15.4, +13.1, and +9.9 pp) and under image-only + occlusion (66.5% vs. 50.6%, Table 21).
Two built-in mechanisms mitigate perception errors:
In real-robot deployment, merging and occlusion can be further mitigated via multi-view RGB-D sensing with memory-bank re-ID trackers and persistent latent entity slots that carry over soft reachability across brief occlusions.
3. Addressing Quadratic Entity Scaling (W3)
Dense reachability propagation scales as O(HN2∣K∣), which we profiled up to N=32 (Table 27). Because physical contact is spatially local, Appendix A.2 notes that sparse top-k edge pruning (e.g., spatial k-NN or interaction-radius gating) reduces complexity to O(HNk∣K∣) without altering the cone definition. Pre-filtering entities outside the robot's H-step kinematic reachability radius further bounds graph size in cluttered scenes.
4. Online Construction of Matched Control (Q3)
Constructing the matched control a0 online requires no unknown future exogenous states. Per Section 7 and Appendix D.2, commands over horizon H decompose into shared/nuisance actuation ashared (known camera/base commands or passive dynamics) and task intervention aint (end-effector displacement/grasping):
a0=(ashared,0int)
For fixed cameras, a0 zeroes end-effector displacement; for wrist/active cameras, the known planned camera/base trajectory ashared over horizon H is copied directly into a0 while setting aint=0. Table 6 confirms this lowers no-contact residual from 0.18 to 0.07.
5. Real-Robot Evaluation and Unseen Embodiments (W4 & Q4)
We candidly note that physical hardware trials are unavailable within the rebuttal window, and we frame rigid-body simulation as our current scope boundary (Section 8, Appendix J). Toward broader embodiments, Table 21 already evaluates an expanded 7-DoF manipulation space, where EventCone-WM maintains a +14.2 pp gain over JEPA-WM (63.1% vs. 48.9%). Because our method plugs into frozen visual backbones and supports tracked/image-only interfaces, sim-to-real transfer with independent safety monitors (Ethics Statement) is our immediate next step.