- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies an interesting problem in world models that the predictive information in in their latent representation is difficult to localize, interpret and use for relational reasoning. The core idea to address this difficulty is to first predict the future, and then combine the current observations with the predicted future to construct a structured state. Specifically, Abductive World Modeling (AWM) is proposed and uses HASP to sequentially construct entity, dynamics, and relational states on top of a frozen V-JEPA 2, and then employs factor supervision so that these states have explicit organizational granularity. Experiments demonstrate clear gains over backbone baseline, and probes/masking tests examine what information the representations contain and how predictions depend on it.
Critical strengths
- This paper is well written and organized, making it easy to follow. The design of AWM that organizes distributed video features into three levels:entity, entity-time, and entity-pair-time, to make the states easier to localize/inspect/exploit is well motivated.
- The experiments support the strength of AWM and show that AWM improves performance over V-JEPA 2 by 6.99, 7.11, and 17.45 percentage points on the three main metrics, across both synthetic scenes and real-world first-person videos. 3.The interpretability analysis is made. Experiments uses probes to check whether the states contain information about objects, motion, and interactions. Masking experiments also help explain what information the model’s representations contain and how the model uses it.
Critical weaknesses
- CLEVRER includes four question types: descriptive, predictive, explanatory, and counterfactual. The experiments in the paper only focus on the predictive task. However, the performance improvement in predictive task may stem from additional supervision signals such as objects, motion, and interactions. Comparing methods with the same supervision would help isolate the benefit of the hierarchical design. In addition, the predictive scenario in experiment design cannot fully support the claim that explicit causal structure are included in the representations of AWM for explaining how the world evolves.
- The current experiments do not clearly isolate the benefits of predicted futures and the hierarchical design. Additional ablation studies are needed to assess their separate contributions.
Questions for authors
There is a clear gap between demonstrating sensitivity to representation masking and proving that the model can correctly reason about causal/counterfactual outcomes. Changing a prediction by hiding information does not mean the model can correctly predict what would happen if the physical situation changed. Additional explanation and discussions are required.
3 comments
2. Ablations assessing the separate contributions of predicted futures and hierarchical design
To examine these two design choices separately, we added a crossed comparison in the revised implementation: hierarchical versus parallel architecture × an interface using observations only versus one using observations and predicted futures. Table R3 reports the results.
The architectural comparison holds the video backbone, semantic supervision, data split, training budget, and task readout constant. Both the hierarchical and parallel state modules contain 4,457,304 active trainable parameters. The parallel architecture removes the transfer of state features along Entity→Dynamic and Dynamic→Relation, while retaining each module’s access to visual features, the shared slot basis, and the label-to-slot matching procedure.
For the input comparison, the two input conditions within each architecture and seed share the same state checkpoint. A task readout with the same architecture and training budget is then trained separately for each condition. Predicted futures are generated by the frozen video predictor; ground-truth future images are not provided as input.
Table R3. Ablations of architecture and the interface incorporating predicted futures on the CLEVRER predictive task using the revised implementation. Question accuracy is reported on the common set of 3,557 predictive questions as mean ± sample standard deviation across three random seeds (%). Each option is evaluated independently, and a question is counted as correct only when all its options are classified correctly. This table uses revised state modules and task readouts and is not directly compared with the Predictive results in Table R1.
| Architecture | Interface using observations only | Interface using observations + predicted futures |
|---|---|---|
| Parallel | 74.19 ± 1.88 | 76.38 ± 0.98 |
| Hierarchical | 83.90 ± 1.05 | 89.49 ± 0.46 |
Table R3 provides evidence on two aspects:
- Architectural contribution under matched supervision. With observations and predicted futures, the hierarchical architecture outperforms the parallel architecture by 13.10 percentage points, with a paired bootstrap 95% interval of [11.41, 14.64]. With observations only, the gain is 9.71 percentage points, with an interval of [7.27, 12.25]. These differences cannot be explained solely by whether additional semantic labels are available.
- Task value of the interface incorporating predicted futures. Adding this interface improves accuracy by 5.59 percentage points in the hierarchical architecture, with a 95% interval of [4.47, 6.70], and by 2.19 percentage points in the parallel architecture, with an interval of [0.82, 3.59]. Each comparison has the same direction across all three seeds.
We will also clarify the limits of this ablation. The observation-only condition reads states from the observed period, whereas the condition incorporating predicted futures permits readout from the predicted period. The comparison therefore changes both the availability of predicted features and the temporal range available to the readout. It evaluates the contribution of the complete interface incorporating predicted futures, rather than isolating the contribution of predicted features at fixed query times.
Furthermore, predicted futures are computed from existing observations. Their value lies in making task-relevant information easier to organize and decode, rather than supplying independent new observations. These supplementary experiments use existing development and validation data; the intervals describe uncertainty under the current protocol and are not adjusted for multiple comparisons. We will accordingly avoid generalizing these findings into a universal architectural advantage.
3. The gap between sensitivity to representation masking and correct causal/counterfactual reasoning
We agree with the reviewer’s distinction. Representation masking changes the information available inside the model, whereas a physical intervention changes objects, motion conditions, or interaction mechanisms in the scene. A prediction change following masking can indicate functional dependence on the corresponding representation, but it does not establish whether the model correctly predicts the outcome of a changed physical situation. Masking may also produce inputs outside the usual representation distribution, further limiting its causal interpretation.
In the revision, we will distinguish the following forms of evidence:
- Representation-masking experiments assess whether the model uses particular information and which representation components its predictions are sensitive to.
- Explanatory and counterfactual question answering assesses whether the model produces correct answers under the corresponding task conditions. The improvements on Explanatory and Counterfactual questions in Table R1 provide direct task-level evidence for these capabilities.
- Formal identification of causal structure additionally requires explicit variables, intervention semantics, identification assumptions, and corresponding validation. It cannot be established by masking sensitivity or question-answering accuracy alone.
Accordingly, we will replace overly strong wording such as “interventions confirm causal dependence” with “representation perturbations reveal functional dependencies in the model,” and describe the structure supported by the current evidence specifically as “explicit object, motion, and interaction structure.”
The innovation we aim to emphasize is a representation interface connecting predictive world models with structured task inference: first obtaining latent predictions of the future, then organizing them into states with explicit semantic roles for downstream decoding and inspection. The four-question-type results support the applicability of this paradigm, while comparisons under matched supervision and architectural ablations help explain the sources of its gains. We will refine the contribution statements around this evidence and clearly identify the additional validation required for stronger causal claims.
We thank the reviewer for the careful assessment. Your comments have helped us clarify the scope of our work and distinguish the contributions of additional supervision, hierarchical design, and predicted futures.
This paper presents foundational work on predictive modeling of physical interactions. We propose a new paradigm of prediction-driven structured state inference: conditioning on a video model’s observed and predicted representations to construct Entity, Dynamic, and Relation states, thereby turning latent predictive information into decodable, inspectable representation interfaces for downstream tasks. Given this research objective, the original submission focused on CLEVRER’s Predictive category, which directly evaluates judgments about future physical events. In response to the reviewer’s comments, we have added results for the other three question types, comparisons under matched supervision, and architectural ablations. We respond point by point below.
1. CLEVRER question-type coverage and whether the performance gains mainly stem from additional supervision
We agree that evaluating other question types helps assess the applicability of the proposed paradigm. We therefore added experiments on Descriptive, Explanatory, and Counterfactual questions. Table R1 reports the results for all four types.
Table R1. Accuracy of the baseline and the full E+D+R model on the four CLEVRER question types. The macro-average assigns equal weight to each question type. This evaluation and the revised-implementation ablations in Table R3 are reported separately; no numerical comparisons are made across these settings.
As shown in Table R1, the full model improves accuracy on all four question types, increasing the macro-average by 5.09 percentage points. The gains on Predictive, Explanatory, and Counterfactual questions are 7.11, 6.51, and 5.28 percentage points, respectively, while Descriptive accuracy improves by 1.47 percentage points.
Regarding additional supervision, we agree that comparing the full model with simplified models alone is insufficient to distinguish its contribution from that of architectural design. We therefore conducted comparisons between hierarchical and parallel architectures under matched supervision. Table R2 presents the architecture × supervision experiment on Physion++.
Table R2. Architecture and supervision controls on the Physion++ internal holdout under the supplementary joint-training protocol. All conditions retain OCP task supervision; semantic supervision adds the Entity, Dynamic, and Relation objectives. OCP AUROC is reported as mean ± sample standard deviation across three random seeds (%). The two architectures have approximately matched parameter counts and use the same data, training budget, and checkpoint-selection rule.
Table R2 shows that adding semantic supervision improves AUROC by 5.75 and 6.38 percentage points for the hierarchical and parallel architectures, respectively. Additional supervision is therefore an important source of the performance gains, and we will explicitly acknowledge this contribution in the revision.
Under this protocol, the hierarchical-minus-parallel difference is −0.49 percentage points, with an empirical 95% interval of [−1.44, 0.42], which does not establish a consistent hierarchical advantage. In contrast, the revised CLEVRER implementation discussed below achieves higher accuracy with the hierarchical architecture under matched semantic supervision and capacity. We therefore scope our conclusion as follows: semantic supervision provides clear value, and hierarchical information transfer further improves performance under the current CLEVRER setting; its advantage depends on the task and evaluation protocol.