acceptodds
Under review as a conference paper at ICLR 2027

Abductive World Modeling via Causal Representation Learning

Abstract

The central challenge of world modeling is to learn representations that capture how the world evolves. However, existing world models predominantly represent future states without explicitly capturing the latent causes underlying their evolution, limiting their ability to reason about why and how the world changes. To address this limitation, we propose *Abductive World Modeling (AWM)*, a framework that learns structured causal representations by abductively inferring latent causes from predicted futures. Specifically, we realize AWM through the Hierarchical Abductive State Pyramid (HASP), which organizes the inferred world state into three complementary components—*Entity*, *Dynamic*, and *Relation*—capturing what exists, how it changes, and how entities interact, respectively. By jointly reasoning over the current observation and its predicted future, HASP abductively infers these latent factors and integrates them into a structured state representation for downstream reasoning. To the best of our knowledge, AWM is the first framework to introduce abductive state inference into latent-space world modeling for learning structured representations of world dynamics. Experiments across physical prediction, causal reasoning, and action understanding demonstrate the effectiveness of our approach. Compared with V-JEPA 2, a state-of-the-art latent-space world model, AWM improves physical prediction AUROC by 10.7%, causal reasoning accuracy by 16.8%, and action Top-1 accuracy by 68.0%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
World Models, Abductive Inference, Latent Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bpRNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an interesting problem in world models that the predictive information in in their latent representation is difficult to localize, interpret and use for relational reasoning. The core idea to address this difficulty is to first predict the future, and then combine the current observations with the predicted future to construct a structured state. Specifically, Abductive World Modeling (AWM) is proposed and uses HASP to sequentially construct entity, dynamics, and relational states on top of a frozen V-JEPA 2, and then employs factor supervision so that these states have explicit organizational granularity. Experiments demonstrate clear gains over backbone baseline, and probes/masking tests examine what information the representations contain and how predictions depend on it.

Critical strengths
  1. This paper is well written and organized, making it easy to follow. The design of AWM that organizes distributed video features into three levels:entity, entity-time, and entity-pair-time, to make the states easier to localize/inspect/exploit is well motivated.
  2. The experiments support the strength of AWM and show that AWM improves performance over V-JEPA 2 by 6.99, 7.11, and 17.45 percentage points on the three main metrics, across both synthetic scenes and real-world first-person videos. 3.The interpretability analysis is made. Experiments uses probes to check whether the states contain information about objects, motion, and interactions. Masking experiments also help explain what information the model’s representations contain and how the model uses it.
Critical weaknesses
  1. CLEVRER includes four question types: descriptive, predictive, explanatory, and counterfactual. The experiments in the paper only focus on the predictive task. However, the performance improvement in predictive task may stem from additional supervision signals such as objects, motion, and interactions. Comparing methods with the same supervision would help isolate the benefit of the hierarchical design. In addition, the predictive scenario in experiment design cannot fully support the claim that explicit causal structure are included in the representations of AWM for explaining how the world evolves.
  2. The current experiments do not clearly isolate the benefits of predicted futures and the hierarchical design. Additional ablation studies are needed to assess their separate contributions.
Questions for authors

There is a clear gap between demonstrating sensitivity to representation masking and proving that the model can correctly reason about causal/counterfactual outcomes. Changing a prediction by hiding information does not mean the model can correctly predict what would happen if the physical situation changed. Additional explanation and discussions are required.

3 comments

We thank the reviewer for the careful assessment. Your comments have helped us clarify the scope of our work and distinguish the contributions of additional supervision, hierarchical design, and predicted futures.

This paper presents foundational work on predictive modeling of physical interactions. We propose a new paradigm of prediction-driven structured state inference: conditioning on a video model’s observed and predicted representations to construct Entity, Dynamic, and Relation states, thereby turning latent predictive information into decodable, inspectable representation interfaces for downstream tasks. Given this research objective, the original submission focused on CLEVRER’s Predictive category, which directly evaluates judgments about future physical events. In response to the reviewer’s comments, we have added results for the other three question types, comparisons under matched supervision, and architectural ablations. We respond point by point below.

1. CLEVRER question-type coverage and whether the performance gains mainly stem from additional supervision

We agree that evaluating other question types helps assess the applicability of the proposed paradigm. We therefore added experiments on Descriptive, Explanatory, and Counterfactual questions. Table R1 reports the results for all four types.

Table R1. Accuracy of the baseline and the full E+D+R model on the four CLEVRER question types. The macro-average assigns equal weight to each question type. This evaluation and the revised-implementation ablations in Table R3 are reported separately; no numerical comparisons are made across these settings.

Question typeBaselineE+D+R
Descriptive79.52%80.99%
Explanatory57.15%63.66%
Predictive42.20%49.31%
Counterfactual26.32%31.60%
Macro-average over four types51.30%56.39%

As shown in Table R1, the full model improves accuracy on all four question types, increasing the macro-average by 5.09 percentage points. The gains on Predictive, Explanatory, and Counterfactual questions are 7.11, 6.51, and 5.28 percentage points, respectively, while Descriptive accuracy improves by 1.47 percentage points.

Regarding additional supervision, we agree that comparing the full model with simplified models alone is insufficient to distinguish its contribution from that of architectural design. We therefore conducted comparisons between hierarchical and parallel architectures under matched supervision. Table R2 presents the architecture × supervision experiment on Physion++.

Table R2. Architecture and supervision controls on the Physion++ internal holdout under the supplementary joint-training protocol. All conditions retain OCP task supervision; semantic supervision adds the Entity, Dynamic, and Relation objectives. OCP AUROC is reported as mean ± sample standard deviation across three random seeds (%). The two architectures have approximately matched parameter counts and use the same data, training budget, and checkpoint-selection rule.

ArchitectureOCP supervision onlyOCP + Entity, Dynamic, and Relation supervision
Hierarchical80.70 ± 0.2086.45 ± 1.74
Parallel80.56 ± 0.4486.94 ± 1.74

Table R2 shows that adding semantic supervision improves AUROC by 5.75 and 6.38 percentage points for the hierarchical and parallel architectures, respectively. Additional supervision is therefore an important source of the performance gains, and we will explicitly acknowledge this contribution in the revision.

Under this protocol, the hierarchical-minus-parallel difference is −0.49 percentage points, with an empirical 95% interval of [−1.44, 0.42], which does not establish a consistent hierarchical advantage. In contrast, the revised CLEVRER implementation discussed below achieves higher accuracy with the hierarchical architecture under matched semantic supervision and capacity. We therefore scope our conclusion as follows: semantic supervision provides clear value, and hierarchical information transfer further improves performance under the current CLEVRER setting; its advantage depends on the task and evaluation protocol.

2. Ablations assessing the separate contributions of predicted futures and hierarchical design

To examine these two design choices separately, we added a crossed comparison in the revised implementation: hierarchical versus parallel architecture × an interface using observations only versus one using observations and predicted futures. Table R3 reports the results.

The architectural comparison holds the video backbone, semantic supervision, data split, training budget, and task readout constant. Both the hierarchical and parallel state modules contain 4,457,304 active trainable parameters. The parallel architecture removes the transfer of state features along Entity→Dynamic and Dynamic→Relation, while retaining each module’s access to visual features, the shared slot basis, and the label-to-slot matching procedure.

For the input comparison, the two input conditions within each architecture and seed share the same state checkpoint. A task readout with the same architecture and training budget is then trained separately for each condition. Predicted futures are generated by the frozen video predictor; ground-truth future images are not provided as input.

Table R3. Ablations of architecture and the interface incorporating predicted futures on the CLEVRER predictive task using the revised implementation. Question accuracy is reported on the common set of 3,557 predictive questions as mean ± sample standard deviation across three random seeds (%). Each option is evaluated independently, and a question is counted as correct only when all its options are classified correctly. This table uses revised state modules and task readouts and is not directly compared with the Predictive results in Table R1.

ArchitectureInterface using observations onlyInterface using observations + predicted futures
Parallel74.19 ± 1.8876.38 ± 0.98
Hierarchical83.90 ± 1.0589.49 ± 0.46

Table R3 provides evidence on two aspects:

  • Architectural contribution under matched supervision. With observations and predicted futures, the hierarchical architecture outperforms the parallel architecture by 13.10 percentage points, with a paired bootstrap 95% interval of [11.41, 14.64]. With observations only, the gain is 9.71 percentage points, with an interval of [7.27, 12.25]. These differences cannot be explained solely by whether additional semantic labels are available.
  • Task value of the interface incorporating predicted futures. Adding this interface improves accuracy by 5.59 percentage points in the hierarchical architecture, with a 95% interval of [4.47, 6.70], and by 2.19 percentage points in the parallel architecture, with an interval of [0.82, 3.59]. Each comparison has the same direction across all three seeds.

We will also clarify the limits of this ablation. The observation-only condition reads states from the observed period, whereas the condition incorporating predicted futures permits readout from the predicted period. The comparison therefore changes both the availability of predicted features and the temporal range available to the readout. It evaluates the contribution of the complete interface incorporating predicted futures, rather than isolating the contribution of predicted features at fixed query times.

Furthermore, predicted futures are computed from existing observations. Their value lies in making task-relevant information easier to organize and decode, rather than supplying independent new observations. These supplementary experiments use existing development and validation data; the intervals describe uncertainty under the current protocol and are not adjusted for multiple comparisons. We will accordingly avoid generalizing these findings into a universal architectural advantage.

3. The gap between sensitivity to representation masking and correct causal/counterfactual reasoning

We agree with the reviewer’s distinction. Representation masking changes the information available inside the model, whereas a physical intervention changes objects, motion conditions, or interaction mechanisms in the scene. A prediction change following masking can indicate functional dependence on the corresponding representation, but it does not establish whether the model correctly predicts the outcome of a changed physical situation. Masking may also produce inputs outside the usual representation distribution, further limiting its causal interpretation.

In the revision, we will distinguish the following forms of evidence:

  • Representation-masking experiments assess whether the model uses particular information and which representation components its predictions are sensitive to.
  • Explanatory and counterfactual question answering assesses whether the model produces correct answers under the corresponding task conditions. The improvements on Explanatory and Counterfactual questions in Table R1 provide direct task-level evidence for these capabilities.
  • Formal identification of causal structure additionally requires explicit variables, intervention semantics, identification assumptions, and corresponding validation. It cannot be established by masking sensitivity or question-answering accuracy alone.

Accordingly, we will replace overly strong wording such as “interventions confirm causal dependence” with “representation perturbations reveal functional dependencies in the model,” and describe the structure supported by the current evidence specifically as “explicit object, motion, and interaction structure.”

The innovation we aim to emphasize is a representation interface connecting predictive world models with structured task inference: first obtaining latent predictions of the future, then organizing them into states with explicit semantic roles for downstream decoding and inspection. The four-question-type results support the applicability of this paradigm, while comparisons under matched supervision and architectural ablations help explain the sources of its gains. We will refine the contribution statements around this evidence and clearly identify the additional validation required for stronger causal claims.

Reviewer_wYb3OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a "predict forward, then abduce backward" paradigm to remedy the limitation of current world models, which focus on representing future states while neglecting the modeling of the latent causes underlying state evolution. Specifically, Abductive World Modeling (AWM) is realized through the Hierarchical Abductive State Pyramid (HASP), which uses three Attributors to convert the observation and the predicted future into an abductive state for downstream tasks, reporting substantial performance improvements over V-JEPA 2.

Critical strengths
  1. The paper achieves consistent performance improvements on all three downstream tasks.
  2. The paper proposes the Hierarchical Abductive State Pyramid, which performs a complementary representation decomposition of the predicted observation, emphasizing the importance of latent inference factors.
Critical weaknesses
  1. The ablation studies are insufficient, and the source of the performance gains cannot be determined. The method applies supervision alignment to HASP, but the paper lacks ablations of the individual losses, so it is unclear whether the gains come from the proposed method or from the additional supervision signals.
  2. The V-JEPA 2 paper does not report EK100 action recognition results, and the source of the numbers in Table 1 lacks necessary explanation.
  3. Why are the three subcaptions (a), (b), and (c) in the Figure 2 caption identical?
  4. The paper neither cites nor compares recent related work, such as Causal-JEPA: Learning World Models through Object-Level Latent Masking (ICML 2026).
Questions for authors

Please see the weaknesses.

3 comments

We thank the reviewer for the careful assessment and for recognizing HASP's approach to representation decomposition. Your questions about ablations, baseline provenance, figure descriptions, and related work help us define the methodological contribution more precisely. We have conducted additional experiments and will clarify the experimental settings and sources of the results in the revised manuscript. We respond point by point below.

1. Ablation studies and whether the performance gains mainly come from additional supervision

The probes and representation-masking experiments in the original submission are not sufficient to distinguish the contribution of module design from that of auxiliary supervision. To address this issue, we have added comparisons of module configurations and ablations of the supervision objectives.

First, on CLEVRER, we retrained the Entity-only and Entity-plus-Dynamic configurations. Table R1 reports these results alongside the full model's result from the main paper as a historical reference.

Table R1. CLEVRER results for the retrained simplified configurations and the full model's historical result. The full-model row is a historical reference, not a newly retrained result under the same comparison.

ModelOption accuracyQuestion accuracy
Entity only (E; retrained)68.65%46.39%
Entity + Dynamic (E+D; retrained)68.48%47.12%
Entity + Dynamic + Relation (E+D+R; historical reference)70.89%49.31%

The historical full-model result is higher than those of both simplified configurations, consistent with the design motivation that representations at multiple granularities may be complementary.

Second, we completed experiments with different combinations of losses on Physion++ using three random seeds. Table R2 summarizes the results.

Table R2. Loss-group ablations for HASP on the Physion++ internal holdout under the supplementary joint-training protocol. OCP supervision is retained in every condition; checkmarks indicate the additional semantic objectives. AUROC is reported as mean ± sample SD across three seeds.

Entity supervisionDynamic supervisionRelation supervisionOCP AUROC (%)
———80.70±0.20
✓——81.20±0.67
—✓—80.81±0.04
——✓82.43±2.64
✓✓—82.43±2.80
✓—✓85.60±1.73
—✓✓84.46±2.23
✓✓✓86.45±1.74

These results support the usefulness of supervision for Entity, Dynamic, and Relation states under this protocol.

We also compared the hierarchical architecture with a parallel architecture of approximately matched parameter count. Adding semantic supervision improved AUROC by 5.75 and 6.38 percentage points, respectively.

2. The source of the V-JEPA 2 results on EK100 in Table 1

Thank you for pointing out that the provenance of these results was not sufficiently clear. The EK100 values in Table 1 of the manuscript are our reproduction results obtained using the V-JEPA 2 code and publicly released pretrained weights under the action-recognition protocol in this paper. We will explicitly label them as author-reproduced results and provide the following settings:

  • Data and input. We use the official EK100 training set to train the recognition readout, splitting it by video_id into approximately 90% for training and 10% for checkpoint selection, followed by evaluation on the official validation set. We uniformly sample 16 frames from each annotated action segment, resize them to 256×256 using bilinear interpolation, and normalize them using the ImageNet mean and standard deviation.

  • Backbone and features. We freeze the pretrained V-JEPA 2 ViT-H encoder and its native predictor. The observed features CC, predicted-future features FF, and their difference are mean-pooled separately and concatenated into a 3,840-dimensional input:

\operatorname{mean}(C),\\; \operatorname{mean}(F),\\; \operatorname{mean}(F-C) \right]\in\mathbb R^{3840}.$$ - **Recognition readout.** We use an MLP with a hidden dimension of 960, LayerNorm, GELU, and dropout of 0.1 to predict verbs and nouns separately. Each action-combination score is obtained by summing the corresponding verb and noun logits. The training objective is the sum of the verb and noun cross-entropy losses and the action-combination cross-entropy loss with a weight of 0.2. - **Training and checkpoint selection.** We use AdamW with a learning rate of $10^{-3}$, weight decay of $10^{-4}$, a configured batch size of 256, and 30 training epochs. Checkpoints are selected by the mean of verb and noun Top-1 accuracies on the internal holdout, and we report the average results over three random seeds (239, 240, and 241). This reproduction evaluates **action recognition** within annotated action segments, whereas the original V-JEPA 2 paper reports **action anticipation** on EK100. The two tasks differ in their input conditions and evaluation metrics. We will explicitly distinguish them in the revision so that readers do not interpret our values as results directly reported in the V-JEPA 2 paper. ## 3. The identical titles of the three subfigures in Figure 2 Thank you for highlighting this potential ambiguity. Figure 2(a)–(c) all illustrate **Egocentric Entity grounding**, but correspond to three different egocentric scenes. The original submission therefore used the same analysis-category title for all three. This naming did not sufficiently distinguish the content of the individual subfigures. In the revision, we will retain the common analysis category while adding a scene-specific description to each subfigure. The overall caption will explicitly state that (a)–(c) show Entity grounding in different egocentric scenes, while (d) shows Entity grounding across consecutive CLEVRER frames, thereby removing the ambiguity.

4. The missing citation and comparison with Causal-JEPA

We agree that Causal-JEPA is an important and closely related work that should be included in both the related-work discussion and experimental comparisons. Causal-JEPA uses object-level latent masking to control the information observable during training, encouraging the world model to use contextual objects and their interaction dependencies for prediction. Our work instead focuses on constructing supervised Entity, Dynamic, and Relation interfaces on top of existing predictive features. We will distinguish the two approaches in terms of where they operate in training, their representation interfaces, and their supervision, and correct the overly broad characterization of prior object-centric and relational modeling work.

To provide an empirical comparison, we used the official implementation and publicly released weights to complete native ALOE experiments with three QA training seeds under the main VideoSAUR/mask4 configuration. The Causal-JEPA QA model was trained on all four question types for 400 epochs, with the final-epoch checkpoint fixed for evaluation.

On the common set of 3,557 CLEVRER predictive questions, we aligned the temporal sampling of the inputs and evaluated each option independently, counting a question as correct only when all its options were classified correctly. The results are shown in Table R3.

Table R3. Supplementary comparison on 3,557 common CLEVRER predictive questions with aligned temporal sampling and independent option scoring. Question accuracy is mean ± SD across three seeds; option accuracy is the mean. The Causal-JEPA row uses the fixed final checkpoint at epoch 400. This comparison is separate from the historical-reference comparison in Table R1.

MethodOption accuracy (%)Question accuracy (%)
Causal-JEPA (VideoSAUR/mask4; native ALOE)92.5886.30±0.77
HASP94.5489.49±0.46

Question accuracy is reported as the mean ± standard deviation across three QA seeds, while option accuracy is reported as the mean. HASP's mean question accuracy is higher by 3.19 percentage points, with a bootstrap 95% interval of [1.94,4.43] percentage points obtained by paired resampling over seeds and videos. The difference is positive for all three seeds. These results suggest that HASP can make information encoded in latent representations more useful and improve downstream task performance under this evaluation protocol.

We sincerely thank the reviewer for the valuable feedback. These suggestions have helped us strengthen the experimental evaluation and clarify the presentation of our work. We hope that the additional experiments and explanations address the reviewer's concerns and make our contribution and practical value clearer. We would welcome further discussion and respectfully ask the reviewer to reconsider their assessment in light of the new evidence.

Reviewer_y8QWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Abductive World Modeling (AWM), which augments a frozen V-JEPA 2 world model with a Hierarchical Abductive State Pyramid (HASP). HASP maps current and predicted-future representations into entity, dynamic, and relation states, supervised using corresponding factor annotations. The resulting states are evaluated through downstream prediction, probing, qualitative examples, and representation-masking experiments.

Critical strengths
  1. The separation of object, temporal, and pairwise information is an intuitive way to organise representations from a predictive video model.
  2. The paper attempts to inspect the learned states through probes, examples, and masking experiments rather than relying only on downstream task scores.
Critical weaknesses
  1. The method is not self-contained. Core paths, modules, temporal indices, slot matching, supervision, and losses are not specified sufficiently to reconstruct HASP. The appendix mainly restates tensor maps and shapes rather than resolving these gaps (I1).
  2. The evidence does not establish the intended causal or abductive hierarchy. It is unclear what formally makes the supervised hierarchical readouts causal or abductive, or how they differ from related object-centric and relational representations. Point estimates, individual examples, and representation masks show suggestive sensitivity but do not establish causal intervention or isolate consecutive stages that inherit information from one another (I2-I3).
Questions for authors

I1. Architecture and training specification

  • P4 — "information paths" and "establishes a persistent object-centered coordinate system" What are these paths, and what mechanism makes the object basis persistent and stable?
  • P4 — 'attend to ZcZ_c', 'attribution layer ll', LN(Zc)LN(Z_c), and "iterative query refinement" What is the exact layer structure, how many attribution blocks and refinement iterations are used, and how are the normalisation and attention operations defined? How does this construction relate to existing object-centric and causal representation models?
  • P5 — "EfutureE_{future}", "preserves" identity, "integrated" temporal dependencies, and TemporalTransformer(HD)TemporalTransformer(H_D) Is the future input a single state or a sequence? How are identity and temporal information preserved, and what architecture and temporal indexing does TemporalTransformer use? A reference should be added for reused modules/methods.
  • P5 — "The main attribution path" and "redefine entity identity" What is the main path, how does it interact with the residual paths, and why should later evidence be prevented from revising entity identity?
  • P5 — "shared" pair terms, TemporalAttention, and "−SjE-S^E_j" without a temporal index Are the pair features summed or otherwise combined, what is TemporalAttention, and how are its temporal and slot indices handled?
  • P6 — "supervise" and "HASP implementation details" What labels are available at each level, how are they constructed and matched to learned slots, and what losses train FEF_E, FDF_D, FRF_R, GDG_D, and GRG_R?

I2. Causal and abductive interpretation

  • P1 — "learns structured causal representations by abductively inferring latent causes from predicted futures" In what formal sense are the representations causal and the inference abductive, beyond applying supervised hierarchical readouts to frozen predictive features?
  • P2 — "object-level relations" How do the proposed relation states differ from the object-level relations already attributed to prior work?
  • P3 — "These directions ... generally do not infer structured causes of world evolution from predicted states" What supports this distinction from the cited causal representation and object-centric literature? Does the proposed method infer from one predicted state or from an evolution over several states?

I3. Evidence for the hierarchy

  • P2 — "targeted interventions confirm selective dependence" Are these the representation-masking experiments? If so, what do they establish beyond functional sensitivity, given that they are not interventions on the data-generating process?
  • P7 — "outperforming all three baselines" How many independent runs support the comparison, and what uncertainty or statistical tests establish that the differences are reliable?
  • P7 — "readability" What does readability mean here, and how do the moderate probe scores support this interpretation?
  • P8 — "provide qualitative evidence" and the entity examples How representative are these examples of the full evaluation? What do TTC and OCP denote?
  • P8 — "Relation integrates these cues into explicit pairwise interaction states" and "readable object, motion, and interaction information" How do the examples or masking results isolate information at each level when the stages are consecutive and connected by residual paths? Component, residual-path, and ordering ablations would clarify the contribution of each stage.
  • P9 — "provides important evidence for" How strong is this conclusion when the temporal-static comparison is reported in one table without variation across runs?
6 comments

We thank the reviewer for the careful assessment. Your comments helped us clarify the central contribution: a paradigm for prediction-driven structured state inference that organizes predictive video representations into a world-state interface with object binding, temporal structure, and interaction semantics for direct downstream use.

I1.1 — P4: Information paths and the stable object basis

Information follows these dependencies:

Observed C → Entity; (C, predicted F, Entity) → Dynamic; (Entity, Dynamic states and object temporal memory) → Relation.

Entity reads only the 256 spatial tokens of the last observed tubelet, producing eight object slots, region ownership, and object attributes. Dynamic uses the same object representations and spatial centers as anchors to aggregate object memory across 16 observed/predicted steps; Relation reuses these object indices to build 28 object pairs. Stability here means reference object indices retain explicit correspondence across object-wise temporal and pairwise interaction modeling, rather than each level generating unrelated queries.

I1.2 — P4: Attribution-layer structure, attention normalization, and iterative query refinement

Entity uses a competitive attribution module of width 128 with three parameter-sharing iterations, eight object queries, and one background query. The input passes through LayerNorm and a linear projection and receives coordinate embeddings on a 16×16 grid. Each iteration computes

\frac{Q\bigl(\mathrm{LN}(q_k)\bigr)^{\top}K\bigl(\mathrm{LN}(x_n)\bigr)}{\sqrt{128}} \right),$$ then normalizes $A$ across the 256 spatial tokens for value aggregation, followed by a GRUCell and residual MLP to update queries. Dynamic uses two iterations of object-conditioned spatial attribution. Each cross-attention block uses pre-LN multi-head attention, a query residual, and a residual FFN with fourfold hidden expansion. The temporal Transformers in Dynamic and Relation each have two layers, four heads, FFN dimension 512, GELU, dropout 0.1, and a final LayerNorm. Attention is bidirectional over the full available window; no autoregressive causal mask is used. ### I1.3 — P5: Future input: single state or sequence? Identity and temporal dependencies The input is a predicted sequence. Observed C and predicted F both have shape `[B,8,256,1280]`. F is generated from C by the frozen predictor; no ground-truth future enters the forward pass. Dynamic initializes object attribution at each step from a projection of Entity object queries and applies a Gaussian attention bias based on object spatial centers, producing memory of shape `[B,8 objects,16 input times,128]`. Object indices stay explicit; temporal positions use five-dimensional relative-time features, plus an observed/predicted source flag. Each object's memory passes through a two-layer temporal Transformer, and temporal queries decode 80 nine-dimensional states (displacement, velocity, spatial extent). ### I1.4 — P5: Interaction of the main attribution path with information-preserving paths; current identities as references Three mechanisms preserve information: object anchoring in Dynamic retains reference-object correspondence; Relation reads both predicted geometry and object temporal visual memory; structured residuals supplement the shared video readout downstream. In revision, these actual computational paths will replace the schematic $G_D/G_R$ notation, so each edge maps to explicit inputs, outputs, and modules without implying undefined standalone networks. The downstream logit is $$z = z_S + \sum_{\ell\in\\{E,D,R\\}} g_{\ell}\delta_{\ell} + \delta_{\mathrm{joint}}.$$ E/D/R are each read by separate task queries; the two role-query outputs are kept separate and then concatenated. Final layers are zero-initialized, so epoch 0 equals the shared S baseline exactly. The interface thus starts from the baseline and progressively supplements prediction with structured states. Defining reference objects from current observations separates "what currently exists" from "how that object subsequently evolves," reducing the risk of treating predictive hallucinations as direct evidence for current objects. This is a motivated modeling choice, not a requirement that later evidence never revise identity. During joint training, Dynamic, Relation, and task losses can still update Entity parameters.

I1.5 — P5: Pair-feature combination; TemporalAttention and pair/time indices

Relation maintains 28 unordered pairs with i<ji<j. Inputs contain two complementary types: predicted geometry per mode and interaction-related visual cues in object temporal memory.

The geometric branch builds symmetric pair features from relative position, relative velocity, spatial extent, distance/gap, and related quantities. The 80 states are grouped in consecutive pairs into 40 event bins, with predicted role information added. After presence-weighted message aggregation through shared objects, the temporal Transformer operates on [B×M×28,40,128], retaining explicit temporal indices for each pair's 40-step sequence.

Visual content is

L(H_{i,t})\odot L(H_{j,t}),\; \bigl|L(H_{i,t})-L(H_{j,t})\bigr| \Bigr],$$ where $L$ projects 128→16 dimensions and brackets denote concatenation, yielding 32-dimensional content. The 40 pair queries cross-attend to visual memory over 16 steps, and a gated update incorporates the result into relation features. These symmetric combinations are invariant to exchanging $i$ and $j$. "Shared" here means information jointly used to construct a pair, not statistical independence or an identified common cause. Each scene mode independently produces 40 first-event classes and a `none` class, mixed using shared mode weights. The event-existence score is $$p_{\mathrm{event}} = 1-q(\mathrm{none}),$$ equal to the total mass of the first-event PMF. ### I1.6 — P6: Labels, slot matching, and training losses Supervision aligns with the granularity of the three states. Table 1 summarizes label sources, targets, and losses. **Table 1. Supervision targets, label sources, and training losses for each module in the current HASP implementation.** | Module | Label sources and targets | Corresponding losses | |---|---|---| | Entity | Current position/extent, presence, and target/zone/other roles from metadata; available instance regions; pooled frozen features within regions as appearance targets | Geometry L1, presence BCE, role CE, balanced region CE/Dice, appearance-reconstruction MSE | | Dynamic | Displacement, velocity, extent, and validity from metadata; observed motion and available temporal regions | Trajectory soft-min energy with mode weights, observed-motion SmoothL1, temporal ownership CE | | Relation | Collisions and frame-level target-zone events from metadata; distance/gap constructed from GT states; first event and `none` | Distance/gap SmoothL1, contact BCE, first-event/none NLL, event BCE | | Task interface | Whether the designated target contacts the designated zone anywhere in the episode | OCP BCE and squared-residual regularization | The matching cost is four times the mean absolute error in six-dimensional geometry, plus $2(1-\mathrm{Dice})$ when masks are available. Within each call, the same correspondence is used for all future steps and pairs; matching is not recomputed with the GT future. Missing regions and invalid states use validity masks. GT is used only for supervision/evaluation; the online interface receives only C/F. The auxiliary objectives are split into three parts. For the Entity module: $$\mathcal L_E = 5\mathcal L_{\mathrm{geometry}} + \mathcal L_{\mathrm{presence}} + 0.5\mathcal L_{\mathrm{role}} + \mathcal L_{\mathrm{mask\text{-}balanced}} + 0.05\mathcal L_{\mathrm{appearance}}$$ For the Dynamic module: $$\mathcal L_D = \mathcal L_{\mathrm{mode\text{-}energy}} + 0.25\mathcal L_{\mathrm{observed}} + 0.25\mathcal L_{\mathrm{temporal\text{-}mask}}$$ For the Relation module: $$\mathcal L_R = \mathcal L_{\mathrm{distance}} + \mathcal L_{\mathrm{gap}} + \mathcal L_{\mathrm{contact}} + 0.5\mathcal L_{\mathrm{first}} + \mathcal L_{\mathrm{event}}$$ The joint objective is $$\mathcal L_{\mathrm{joint}} = \mathcal L_{\mathrm{OCP}} + 0.001\,\mathrm{mean}(r^2) + 0.25\mathcal L_E + 0.5\mathcal L_D + 0.25\mathcal L_R,$$ where $r=z-z_S$ is the task residual. E/D/R are pretrained for 6/8/8 epochs, with upstream modules frozen during subsequent-stage pretraining. S is trained separately for 20 epochs and frozen; the full model is then trained jointly for eight epochs. Stage checkpoints are selected using only the auxiliary validation objective or validation task AUROC. The added supervision and structural changes will be disclosed in the revision. ### References [1] Battaglia, P. W., Pascanu, R., Lai, M., Rezende, D. J., & Kavukcuoglu, K. (2016). Interaction networks for learning about objects, relations and physics. Advances in Neural Information Processing Systems, 29. (URL: https://proceedings.neurips.cc/paper/2016/hash/3147da8ab4a0437c15ef51a5cc7f2dc4-Abstract.html) [2] Kipf, T., Fetaya, E., Wang, K. C., Welling, M., & Zemel, R. (2018). Neural relational inference for interacting systems. In International Conference on Machine Learning (pp. 2688-2697). PMLR. (URL: https://arxiv.org/abs/1802.04687)

I2.1 — P1: In what sense is the method abductive, and how does it differ from standard supervised hierarchical readouts?

We emphasize prediction-conditioned inference of structured explanatory states: a predictive model first generates future representations, after which cues in observations and predictions are organized into explicit states describing which objects are present, how they change, and how they interact, for downstream decoding. "Predict forward, then abduce backward" describes this direction of inference.

Experiments show the interface remains useful without direct OCP gradients to the Attributors. With gradients only from auxiliary losses, task performance reaches 87.79±1.82%, close to the full model's 87.89±2.04%; frozen pretrained Attributors also achieve 86.85±2.00%. These configurations still use OCP labels to train the task readout; the controlled factor is the direct OCP gradient to the Attributors. This supports the usefulness of the structured auxiliary states themselves for downstream decoding.

We will explicitly distinguish this operational definition from causal identification and SCM-style abduction–action–prediction, and will not present the framework as establishing causal identifiability. F is a deterministic prediction from C, not an independent new observation; our focus is on making existing predictive information more accessible to structured inference and downstream tasks.

I2.2 — P2: How do the proposed relation states differ from existing object-level relations?

We agree that object relations are an important foundation established by prior work. Interaction Networks and NRI have shown that relational structure can support interaction reasoning and dynamics modeling [1, 2]; we will expand this connection in related work.

The paradigm addresses a specific interface question: how can distributed predictive cues from a fixed predictive visual model be converted into object-bound, temporally resolved interaction states that are pair-addressable and directly usable by task readouts? In the current instantiation, Relation combines per-mode motion geometry with low-rank pairwise temporal visual information, producing distance/gap, contact, first-event, and event-existence fields.

This design turns "relations aid prediction" into a supervised, inspectable task interface whose utility is measured with shared predictive features. Against a control that zeros pairwise visual content while retaining module depth and parameters, the full model improves by +0.69 pp, paired empirical 95% interval [+0.04,+1.32], giving limited support for local visual cues as a complement to explicit motion geometry.

I2.3 — P3: Accurate distinction from related work; single predicted state vs. evolution over several states

We will replace broad exclusionary comparisons with specific comparisons of inputs, inference targets, supervision, and identification conditions. For example, CITRIS studies causal factor identification with temporal sequences and observed intervention targets, whereas our setting learns structured explanatory states and task interfaces on top of a frozen predictive video model. The questions are related but differ in objectives and evidential requirements.

HASP uses eight predicted and eight observed steps, thus drawing on evolution cues across a sequence. Dynamic attributes these cues to reference objects and decodes states over a longer horizon; Relation organizes them into events for object pairs. We will describe this precisely as conditioning structured state inference on a predicted sequence, avoiding conflation of a single predicted state with temporal evolution.

We believe this paradigm is worth studying because as predictive video models become more capable, converting their representations into inspectable, directly usable world states is an important interface problem connecting prediction to downstream reasoning.

I3 Experimental evidence and significance

I3.1 — P2: What do targeted interventions establish?

We describe masking/replacement within the computational graph as functional perturbations, testing whether the task readout uses specified inputs or state fields.

On the new 1,000-sample validation set, replacing only R's first-event PMF/CDF and event-existence fields with their training-set standardization means reduces OCP AUROC by 4.61, 3.69, and 5.98 pp across three seeds. The consistent direction shows functional sensitivity to explicit event information and supports the semantic interface's participation in task prediction.

Comments on the paper

2 comments

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

Dear Area Chair,

Thank you for coordinating the review process for our submission, #12706.

We have responded to the reviewers’ comments and provided additional evidence to address their concerns. However, reviewers y8QW, wYb3 and bpRN have not yet had an opportunity to follow up on our responses.

As the author–reviewer discussion period is scheduled to end on November 18 AoE, we would be grateful if you could help encourage them to review our responses and let us know whether any concerns remain. We would be happy to provide further clarification if needed.

Thank you for your time and assistance.

Best regards, The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.