acceptodds
Under review as a conference paper at ICLR 2027

Learning Implicit Causal World Models from Multi-Agent Demonstrations

34 views

Abstract

In model-based reinforcement learning, world models exist as internal simulators, but their training often conflates statistical correlations with causal mechanisms. This problem is exacerbated in multi-agent systems where physical transitions are intertwined with strategic agent intents, causing world models to fail under distribution shift. We introduce Implicit Causal World Models to recover environmental dynamics from offline demonstrations without requiring pre-defined causal graphs. By incorporating policy variance, we render world models discoverable via the multi agent system sequential backdoor condition. Evaluations across coordination tasks (Two-Door, Navigation, and Giveway) demonstrate that these models provide interpretable causal representations under both full and partial observability, with model accuracy scaling directly with interventional strength.

Primary area:
reinforcement learning
Keywords:
Causality, Reinforcement Learning, Multi-Agent Systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HWg8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Implicit Causal World Models (ICWM) to learn environment transition dynamics from offline multi-agent demonstrations. The authors inject uniform action noise into expert policies (termed stochastic soft interventions), derive a Multi-Agent Sequential Backdoor Condition (MAS-SBC) and a tabular sample-complexity bound, and apply the time-series causal discovery algorithm PCMCI+ [1] (discarding contemporaneous edges) to construct a soft input-feature mask for a 3-layer MLP or LSTM transition model. Experiments on three low-dimensional multi-agent tasks (Two-Door, Giveway, and Navigation) evaluate graph recovery and one-step state prediction error under policy and layout shifts.

Critical strengths

Studying how correlated agent policies confound transition-model learning in multi-agent systems is a sensible problem setup, and the observation that policy exploration variance is the primary driver of out-of-distribution generalization is empirically sound.

Critical weaknesses

1.The paper's own empirical analysis directly refutes the utility of its proposed causal-masking architecture. While the title and methodology center on discovering causal graphs via PCMCI+ [1] and gating neural inputs, Figure 5 and Appendix V (Figures 26–27) demonstrate that the proposed ICWM provides zero statistically significant improvement over a standard unconstrained MLP (AWM) trained on the exact same data, while the recurrent variant R-ICWM is significantly worse. Controlling for the exploration noise parameter σ\sigma reduces the correlation between recovered graph quality and out-of-distribution accuracy to zero. In addition, unfair baseline comparisons in Section 5 train the competing causal baselines (FOCUS [2] and BC-WM) exclusively at σ=0\sigma = 0 while evaluating ICWM across σ>0\sigma > 0 (Table 6), confounding architectural comparisons with dataset coverage.

2.I am a bit confused by the theoretical novelty in Sections 3 and 4, though please correct me if I am mistaken. First, how does the "stochastic soft intervention" in Eq. 4 differ from standard ϵ\epsilon-greedy exploration? Second, in Theorem 1, since the physical transition is Markovian and all agents' joint actions ata_t and states sts_t are observed in the dataset, why do we need a sequential backdoor adjustment over HtH_t instead of directly conditioning on (st,at)(s_t, a_t)? Third, in Theorem 2, doesn't the derivation simply apply the tabular Simulation Lemma [3] and finite-state L1L_1 concentration bounds (2∣S∣2^{\vert{}\mathcal{S}\vert{}}) to show that uniform exploration guarantees a σn\sigma^n lower bound on joint-action visitation?

Questions for authors
  1. Since controlling for σ\sigma eliminates the correlation with OOD error and unmasked AWM matches ICWM (Fig. 5, App. V), what justifies the overhead of PCMCI+ over a standard world model?
  2. Given Axiom 1 (App. F.1) makes sk+1s_{k+1} independent of intent UU given (sk,ak)(s_k, a_k), why is backdoor adjustment over P(ak−i∣Hk)P(a_k^{-i} \mid H_k) needed when full joint actions aka_k are already observed?
  3. Why were FOCUS and BC-WM restricted to σ=0\sigma = 0 (Table 6, Figs. 23–25) while ICWM/AWM used σ∈[0.01,0.99]\sigma \in [0.01, 0.99], and why are multiple cited appendices (e.g., G, M–O, R–U) missing from the supplement?

[1] Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasets

[2] Offline reinforcement learning with causal structured world models

[3] Near-optimal reinforcement learning in polynomial time

2 comments

No rebuttal provided; the original score remains unchanged.

AuthorsOpenReview ↗

Author rebuttal

We thank Reviewer HWg8 for participating in the science with us. Revisions are in blue.

C1. Our paper separates two things the critique treats as one: what makes the physics learnable (intervention in the training data) and what protects the learned model under shift (the causal mask). Intervention is the main lever: once σ\sigma is controlled, graph-based and graph-free models match under policy shift (Fig. 6b). The mask is a prior whose value appears only when the shift reaches the model's inputs, under partial observation or corrupted and unseen variables (Fig. 6a; see Q1). We added illegal tic-tac-toe levels (App. G.1) that separate the gain from data coverage from the gain from intervention (Fig. 3a, 3b), and compare graph-based and graph-free models on them (Fig. 26a, 26b). The same view explains the baselines: FOCUS and BC-WM differ from ICWM only in σ\sigma and the discovery method, so training them at σ>0\sigma>0 gives back ICWM (see Q3).

C2. The theory has one new piece, Theorem 1, and two parts that support it. (i) Eq. 4 has the form of ϵ\epsilon-greedy (ϵ=σ\epsilon=\sigma with uniform ζ\zeta) but a different role: it is applied to the demonstrators curated datasets, for identification rather than reward. Because ζ\zeta does not depend on the agents' hidden intent and is applied to each agent independently, it breaks the intent-action link and decorrelates peer actions, giving Theorem 1 the positivity it needs. σ\sigma and ζ\zeta are design choices: σ\sigma trades identifiability against coverage of the states the expert visits (Fig. 16, Fig. 28), and the shape of ζ\zeta matters, since one random move at a random time reaches 90% of OOD test moves with 49× fewer episodes than uniform randomness (Fig. 17). ϵ\epsilon-greedy is one member of this family (Sec. 3.3.1). (ii) Theorem 1 is needed because the query concerns one agent's action while a shared intent couples all agents' actions; why observing the joint action is not enough is answered in Q2. (iii) Theorem 2 uses standard tools (Simulation Lemma, Weissman's L1 bound, Markov-chain Chernoff bound for dependent samples), and we claim no new proof technique. It makes Theorem 1 quantitative (Sec. 4.3, App. H): without intervention the physics off the expert's path cannot be learned; independent interventions cost σn\sigma^n samples, exponential in team size, which motivates targeted interventions; and too much intervention can starve the coordination states, which the bound alone does not capture (Fig. 28).

Q1. The mask's value depends on the shift (Fig. 6a, 6b). In the three multi-agent environments, when only the policy changes and the state is fully observed, intervention does the work, and the graph-free AWM matches ICWM (Fig. 6b). The mask helps under partial observation: on the Stretched Two-Door layout, every masked model beats AWM (App. V, Fig. 42). Tic-tac-toe shows the same split: on legal games, intervention drives transfer; on illegal games (moves out of turn, never seen in training), the masked model transfers almost exactly while the unmasked one gets worse (Fig. 3b, Fig. 26). The mask is a firewall for corrupted or unseen inputs.

Q2. Axiom 1 (App. F) makes conditioning on the full joint action valid: it removes the hidden intent from the physics, so P(sk+1∣aki,ak−i,Hk)P(s_{k+1} \mid a^i_k, a_k^{-i}, H_k) is the physics fsf_s that ICWM learns, and this conditioning is the backdoor adjustment. The averaging in Theorem 1 is needed because the question is about one agent's action under shift, P(sk+1∣do(aki),Hk)P(s_{k+1} \mid do(a^i_k), H_k). Intent ties the agents' actions together (Axiom 3), so in the data the peers move with agent ii; changing agent ii's action does not change the others' (Axiom 2), so we average fsf_s over what the peers do on their own, P(ak−i∣Hk)P(a_k^{-i} \mid H_k) (Eq. 7). HkH_k stays because intent persists and past actions reveal it. Our data are offline: with expert-only data (σ=0\sigma=0) each action is fixed by intent, so the model learns intent-driven correlations instead of fsf_s at joint actions the expert never takes; σ>0\sigma>0 fixes this.

Q3. FOCUS and BC-WM stay at σ=0\sigma=0 because their original methods learn from expert demonstrations only. BC-WM uses the same PCMCI+ graph and mask as ICWM (Table 8), so with interventional data it is ICWM; FOCUS with interventional data is ICWM with a PC graph. Keeping them at σ=0\sigma=0 isolates our claim that the data, not the architecture, drives transfer. The matched comparison is in our tic-tac-toe study: Fig. 19 compares the graphs found by PC and PCMCI+ at every dataset size and intervention level, and Fig. 21 trains world models on the same data with a PC mask (the FOCUS setting), a PCMCI+ mask, the true mask, a shuffled true mask of the same sparsity, or no mask. Learned masks stay close to the true one; the shuffled mask predicts OOD states worse. The cited appendices (e.g., M–O, R–U; manuscript line 870) are in the detailed appendix in the supplementary material (.zip).

Reviewer_8utWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an Implicit Causal World Model (ICWM) for learning multi-agent transition dynamics. The approach collects demonstrations using a mixture of expert and random actions, applies PCMCI+ to discover temporal dependencies, removes contemporaneous edges, and incorporates the resulting structure into a neural transition model through soft masks. The paper also presents a multi-agent sequential backdoor condition and a generalization bound, with experiments in several simulated coordination environments.

Critical strengths

The paper addresses a relevant problem: transition models trained on coordinated expert demonstrations can generalize poorly to different policies. The experiments investigate several useful factors, including intervention strength, partial observability, agent count, and history length. In particular, the comparison against a graph-free model trained on matched data is valuable, and the paper acknowledges that this baseline matches ICWM under full observability.

Critical weaknesses
  1. The central causal-identification claim is insufficiently connected to the transition-learning problem. The stated SCM defines the next state as a function of the current state, joint action, and environmental noise, while latent intent influences agent actions. Under the fully observed Markov setting and the stated separation between intent and environmental dynamics, conditioning on the state and complete joint action already appears to remove the policy-dependent ambiguity relevant to transition prediction. Limited state-action coverage remains a substantial problem, but this is different from establishing that the transition mechanism requires the proposed causal adjustment for identification. Theorem 1 instead concerns intervening on one agent’s action while marginalizing over other agents’ actions. The paper does not clearly establish why identifying this particular query is necessary for learning the joint-action-conditioned transition model used in the experiments. Consequently, the relationship between the theorem, the implemented estimator, and the claimed resolution of causal confusion remains unclear.
  2. The generalization analysis contains a substantive logical error. Appendix H.1 claims that, because the upper bound contains 1/Nσn1/\sqrt{N\sigma^n}, the estimation error approaches infinity as σ→0\sigma\to0, establishing structural unidentifiability. This conclusion does not follow: a diverging upper bound becomes uninformative; it does not imply that the actual error diverges. In particular, the total variation distance bounded in Lemma 1 is itself bounded. There are additional gaps in the finite-sample argument. Appendix H applies a Chernoff bound to state-action visitation counts from sequential data without specifying the independence or mixing conditions needed for this step. The stationary visitation quantity dmin⁡d_{\min}, absorbed into the constant, can also depend on the intervention policy. These issues undermine the claimed interpretation of the bound as a characterization of how intervention strength determines generalization and identifiability.
  3. The causal-discovery guarantees are not established under the acknowledged hidden confounding. The paper explicitly acknowledges that PCMCI+ assumes causal sufficiency and that its setting violates this assumption. Its proposed remedy is to discard contemporaneous edges. However, removing those edges from an estimated graph does not by itself establish that the remaining lagged edges are unconfounded or correctly recovered. The argument in Appendix I asserts that autonomous physical mechanisms imply strictly positive conditional mutual information under the relevant conditioning sets. Mechanism autonomy alone is insufficient to establish this claim; the argument needs explicit conditions addressing faithfulness and the validity of the discovery procedure under latent confounding. Moreover, preserving true lagged edges would not establish the absence of spurious lagged edges. The architectural convenience of obtaining a directed mask is not a statistical justification for the discovery procedure.
  4. The experiments do not establish the claimed contribution of causal structure. Figure 5 and the discussion on page 9 report that the graph-free AWM matches ICWM’s out-of-distribution performance under full observability when trained on matched data. This is an informative finding, but it substantially narrows the contribution: much of the improvement appears attributable to the intervention-induced training distribution rather than the recovered causal graph. The mask-corruption experiments do not close this gap. Showing that corrupting a mask harms a model does not demonstrate that the learned causal mask is better than appropriately controlled alternatives. Comparisons against sparsity-matched random masks, predictive-dependency masks, and comparable regularization would help distinguish causal structure from generic architectural constraints. The partial-observability and recurrent-model results suggest potentially useful benefits, but do not yet isolate their causal origin.
  5. The methodological novelty and presentation are insufficient for the scope of the claims. The implemented pipeline combines expert/random policy mixing, an existing causal-discovery algorithm, and masked neural transition models. The paper does not convincingly establish a sufficiently novel algorithmic contribution beyond this combination. Its strongest empirical observation that intervention-induced coverage can dominate the benefit of graph structure, which could support a more focused diagnostic contribution, but the current framing emphasizes causal-identification guarantees that are not adequately justified. The presentation also makes the technical claims difficult to verify. Deeply nested intervention annotations and subscripts are hard to decipher around pages 5–6, wraparound figures create cramped text blocks, and Figure 5 and parts of Figures 6–7 contain undersized labels. Long equations and inconsistent typographic choices further increase the reading burden.
Questions for authors
  1. What specific confounding remains in P(st+1∣st,at)P(s_{t+1}\mid s_t,a_t) under the fully observed Markov SCM, beyond insufficient state-action coverage?
  2. How does the estimator used to train ICWM implement or require the adjustment in Theorem 1?
  3. Can the authors provide a valid identification argument independent of the divergence of an upper bound, and a discovery guarantee that addresses both missing and spurious lagged edges?
  4. Does ICWM outperform noncausal masks with matched sparsity and comparable model capacity?
2 comments

We thank the reviewer 8utW for comments. Changes in the revised manuscript are shown in blue. We address each weakness and question below. Response: 1/3 (continued.)

C1. Identification is the transition-learning problem. Line 84 treats each agent's action as a sequential intervention on the environment [1], and SBC/MAS-SBC define the next state as the outcome of that intervention under said conditions. The new 'Implication' paragraph after Theorem 1 makes the link explicit: Theorem 1 reduces the identifiable query of an agent's action, P(st+1∣do(ati),Ht)P(s_{t+1} \mid do(a^i_t), H_t), to the transition kernel fsf_s that ICWM learns. Identification, kernel learning and MAS-SBC are one chain, and because conditioning on the joint action is the adjustment, intent need not be modeled, so the model is implicit. The claim that conditioning on the state and joint action already resolves the problem is incorrect. That conditioning identifies fsf_s only where MAS-SBC holds, i.e. where the joint action has positive probability. With offline expert data, actions are fixed by intent, so off the expert path it does not. Our results show exactly this failure under full observation at σ∈{0,0.25}\sigma \in \{0, 0.25\} (Fig. 4C, 4E, 5-IV, 21a, 37, 38, 39). Line 37 explains the mechanism with a moving-car example; App. E and F give the theory. Coverage is not the explanation either. Positive coverage is a building block, as in DFA recovery (Eqn. 6), but our new illegal tic-tac-toe experiments separate the coverage from intervension: data with the same boards coverage but different intervention strength drives OOD performance even at very low coverage (Fig. 3A, 3B, 26), as MAS-SBC/SBC predicts. The statement that the transition mechanism does not need the adjustment contradicts prior peer-reviewed work: identifying a mechanism under confounding and selection requires an explicit adjustment [2, 3]. Theorem 1 isolates one agent's action because model-based RL must reason about that agent under shift while the other agents keep their own behaviour. This is the premise of our evaluation.

C2. Generalization analysis. At σ=0\sigma=0 the bound is now stated as vacuous (App. H.1). The finite-sample argument now states the mixing condition for temporally correlated samples (line 2315,2414). We state openly that the analysis assumes uniform interventions and a visitation floor dmin⁡d_{\min}, which depends on σ\sigma; Fig. 28 measures dmin⁡(σ)d_{\min}(\sigma), evaluates the bound with it, and shows the training error of each architecture. High σ\sigma starves the coordination states (13% at σ=0.99\sigma=0.99 in Give-Way), which motivates targeted interventions (Sec. 3.3.1). Fig. 5-I validates the bound empirically.

C3. Discovery guarantees. The reviewer applies the global assumptions of PCMCI+ without our explicit domain specific SCM (App. B.3), where uintentu_{intent} is contemporaneous and uenvu_{env} is absorbed into fsf_s. Soundness and completeness. Under Axiom 1 and Assumptions 4.4(i), (ii), uintentu_{intent} coordinates actions only at τ=0\tau=0 (EcontempE_{contemp}) and has no edge into st+1s_{t+1}. For τ≥1\tau \ge 1, the MCI test (Eq. 26) conditions on the target's parents and d-separates st+1s_{t+1} from uintentu_{intent}, so no spurious lagged edge enters (we add this to App. I), and the true lagged edges are kept (App. I.1). Faithfulness is now stated explicitly. Soft interventions (σ>0\sigma>0) add independent action variation that keeps I(Vt−τi;Vtj∣S)I(V^i_{t-\tau}; V^j_t \mid \mathbf{S}) positive; it fails at σ=0\sigma=0 in deterministic settings and with collinear sensors (LIDAR, App. P). Finite samples. These guarantees are asymptotic, so the mask is continuous, Γij∈(0,1)\Gamma_{ij} \in (0,1), a sigmoid of each edge's pp-value. The mask is grounded in the statistical significance of the test, not in architectural convenience (writing fixed).

References:

[1] Sequential Causal Imitation Learning with Unobserved Confounders

[2] Recovering from Selection Bias in Causal and Statistical Inference

[3] Causal Effect Identification by Adjustment under Confounding and Selection Biases

[4] A Calculus for Stochastic Interventions:Causal Effect Identification and Surrogate Experiments

[5] Pearl, Causality: Models, Reasoning, and Inference

[6] Markov Decision Processes with Unobserved Confounders: A Causal Approach

We thank the reviewer 8utW for coments. Changes in the revised manuscript are shown in blue. We address each weakness and question below. Response: 2/3 (continued.)

C4. The mask's contribution is shown, not assumed. AWM matches ICWM only under full observation and policy shift, which we state ourselves (Sec. 5 Q(iii)); the review generalizes this one cell to the whole paper. Fig. 6a shows the mask across shift types: neutral under policy shift, largest gain when the joint state shifts (also Fig. 5-IV, 5-V, 6b, 26b, 26c, 39, 42). The theory is MAS-SBC: a covariate set Zk⊆VZ_k \subseteq V that d-separates the agent's action (App. E.2). Controls. Mask corruption hurts more at higher σ\sigma, where the mask reflects d-separation tests more closely (Fig. 5-V). We added the requested sparsity-matched permuted masks (Fig. 21a, 21b): recovered masks stay within about 1x of the oracle, and permuted masks are worse in every seed, by 10410^4 to 105×10^5\times at 10610^6 episodes. A wrong mask selects the wrong ZkZ_k. On "partial-observability and recurrent results do not isolate their causal origin": they do. Under partial observation fsf_s is only locally preserved and the mask recovers performance (Fig. 42). A recurrent hidden state carries memory outside the mask, so R-ICWM cannot enforce d-separation over ZkZ_k; its edge over ICWM grows as the agent velocity/speed drops (Fig. 6c).

C5. We disagree that the method is a combination of existing parts. Our contributions are (1) MAS-SBC and Theorem 1; (2) differentiable statistical soft masking for finite-sample discovery noise; (3) flow-based OOD uncertainty and intervention diagnostics; (4) targeted empirical tests of each claim. The "intervention-induced coverage" observation is now tested directly: Fig. 3A, 3B and 26 separate intervention from coverage, including the gain of the graph-informed model (Fig. 26b). The identification framing is built step by step: Identifiability at Layer 2 (line 77), actions as interventions (line 86), sequential backdoor [2] (line 89), d-separation (line 90), identification and its reduction to observational data (lines 247, 255), graph surgery and SBC on HkH_k (lines 262 to 278), MAS-SBC (Sec. 4.1), Theorem 1 and proof (line 307), bound (Sec. 4.3), Q(i) to Q(vii) (Sec. 5). Presentation: on graph surgery our notation follows [4] (table 3). Cramped text is fixed and Fig. 4 to 6 enlarged. Long equations: Eq. 6 (explains: positivity, qrejectq_{reject}), Eq. 13 (explains: reduction to observational data), Eq. 21 (explains: worst-case error).

Q1. uintentu_{intent} and uenvu_{env} are unobserved. uenvu_{env} is absorbed into fsf_s; conditioning on the joint action constitutes the adjustment for uintentu_{intent}, and intervention variance makes that adjustment identifiable. Only when this holds and uenvu_{env} is unchanged at test time does the question reduce to coverage, where a graph-free model does as well as graph based baseline in full observability. The graph helps under partial observation, a shift in fsf_s, or a shift in uenvu_{env} (Fig. 6a, 26b, 26c).

Q2. Theorem 1 guarantees identification of an agent's action under shift, given the other agents' policies (Implication paragraph). Policy evaluation, credit assignment and imagination reduce to an adjustment over observational data: fix or plan the agent's move, sample the others (Fig. 3C, 27).

Q3. By definition [5] (line 247), a query is identified if it is computable from observational data given the causal graph; this applies to sequential decisions [6]. Theorem 1 gives the formula. For our SCM, with σ>0\sigma>0, Axiom 1 and Assumptions 4.4(i), (ii) trajectories, PCMCI recovery is complete and asymptotically sound under faithfulness. The generalization bound is not the identification argument; it is a downstream connection between asymptotic identification to finite-sample generalization coming from the intervension policy, connections are made so the targeted interventions(future work) are cheaper in sample complexicity (Fig. 17). Empirically, the shifted state is predictable (Fig. 3B, 4C, 5-IV, 6a, 26, 37 to 39), future consequences can be imagined (Fig. 3C), and OOD transfer tracks intervention strength (Fig. 6b, 40).

Q4. In tic-tac-toe at equal capacity (Fig. 21a, 21b). Permuted masks consistently degrade ICWM; the gap is small at p=0p=0 and large once p>0p>0, matching Fig. 5-V.

References:

[1] Sequential Causal Imitation Learning with Unobserved Confounders

[2] Recovering from Selection Bias in Causal and Statistical Inference

[3] Causal Effect Identification by Adjustment under Confounding and Selection Biases

[4] A Calculus for Stochastic Interventions:Causal Effect Identification and Surrogate Experiments

[5] Pearl, Causality: Models, Reasoning, and Inference

[6] Markov Decision Processes with Unobserved Confounders: A Causal Approach

Reviewer_x6KMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies world models learned from multi-agent demonstrations in which the agents' behavior can obscure the underlying transition dynamics. During data collection, the authors mix expert actions with random actions at a specified intervention strength. They use PCMCI+ to estimate temporal dependencies and turn those estimates into a soft input mask for a neural transition model.

The proposed contributions include a multi-agent sequential backdoor condition and a bound relating transition and value-estimation error to intervention strength. Experiments on Two-Door, Navigation, and Give-Way vary the amount of random action, observability, and agent count. The evaluation measures graph recovery and next-state prediction under policy and layout shifts, with comparisons against models that use different structural assumptions or no graph.

Critical strengths
  1. The paper examines whether graph recovery explains improvements in prediction. The pooled analysis includes a graph-free model and controls for intervention strength. It finds that the apparent relationship between graph quality and prediction largely disappears after that control, while identifying particular conditions under which masking helps. Reporting this result helps separate the effects of the data collection procedure from those of the architecture.

  2. The experiments vary intervention strength across several environments and observability settings, with additional studies of agent count and sample size. The reported results support the empirical finding that more randomized actions can improve graph recovery and prediction under policy shift. These comparisons provide useful evidence about the importance of the collected data.

Critical weaknesses
  1. The connection between the identification result, the implemented model, and the evaluation needs to be made explicit. Equation 7 identifies an interventional transition by marginalizing over the other agents' actions. The architecture instead applies a soft mask derived from PCMCI+ statistics and predicts the next state from state-action inputs. Section 4.5 says the construction satisfies the sequential backdoor condition, but I cannot see where the adjustment in Equation 7 is performed or why the mask implements it. Appendix L provides architectural details but leaves this connection unclear.

The generalization bound also needs a more precise relationship to the experiments. Appendix H scopes the analysis to tabular ergodic MDPs and bounds value error, while the experiments include continuous environments and evaluate one-step prediction error. The bound contains a dependence on agent count through the term sigma raised to n. The main-text description of inverse-square-root scaling in sigma needs to reflect that dependence. The claim that excessively high interventions are harmful is also not established by the displayed bound, which decreases with sigma when its other terms are fixed.

  1. The scope of the causal-model contribution needs to match the empirical results. The authors explicitly report that intervention strength explains much of the predictive improvement and that a graph-free model can match the masked model on the same data. This is useful evidence, but it makes the remaining contribution of the causal architecture more specific. The paper should distinguish improved data coverage from the benefit of the learned graph and limit broader claims accordingly.

The descriptions of transfer also need reconciliation. Anti-Causal is described as reversed dynamics, an inverted cue-to-dynamics mapping, and swapped door states with unchanged geometry. The shifted environments are said to preserve physics, yet some listed shifts change transition rules. These distinctions affect whether the experiments test invariance to behavior, changes in observation, or changes in the underlying dynamics. The main text would be easier to assess with a concrete end-to-end example and a concise definition of each shift. The full supplementary appendix is available; the concern is the clarity of the main account, not missing supplementary material.

Questions for authors
  1. Where does training or inference perform the marginalization in Equation 7? How does the PCMCI+ mask implement the stated identification condition when the predictor already receives the joint action?

  2. What does the partial random-action mixture establish about identification, beyond improving support for different actions? How do the state distribution and dependence between successive observations affect the assumptions, including the statement that lagged histories are almost IID?

  3. How does the tabular value-error bound support the claims made for continuous environments and one-step prediction? Can you clarify its dependence on agent count and identify which result supports the claimed harm from excessively high intervention strength?

  4. Under which conditions does the learned mask improve on an otherwise comparable graph-free model trained on the same data? Can you reconcile the descriptions of Anti-Causal and state exactly which shifts preserve the transition mechanism and which change it?

4 comments

We thank the reviewer x6KM for their constructive comments and curious spirit. We address them in this response and have updated the manuscript; changes from the original upload are shown in blue text.

C1. After Theorem 1, we added a paragraph, "Implication", that explains how the identification result is built up: curated datasets, the SBC condition holding, the soft mask used for finite-sample d-separation of cause and outcome prediction, and a given policy at inference. For the learned ICWM model, Equation 7 shows that a new agent behavior under intervention can be uniquely computed from observational data by conditioning. Section 4.5 is updated to explain the relationship between the SBC, the conditioning set, soft interventions and the mask.

Generalization error bound. Section 4.3 and Appendix H now explain more clearly: how the generalization error bound relates to agent count; its sample-size dependence and its future scope; the related empirical results and the limiting factors under stronger intervention in these domains; targeted interventions; the possible extension of the proof from tabular MDPs to continuous MDPs.

C2. We highlight results on the importance of the causal mask under distortions of the state, the transition kernel and the observation space, and summarize the empirical results in Fig. 6a and Fig. 6b. In the main paper we added experiments that separate the OOD performance gained from training-dataset coverage from that gained from the intervention strength used to collect the training data (Fig. 3a, Fig. 3b). These use newly proposed illegal tic-tac-toe game levels (Appendix G.1), together with the nuanced comparison of graph-based and graph-free baselines (Fig. 26a, Fig. 26b).

Clear descriptions of transfer. Fig. 6a, Fig. 3a and Fig. 27 show the various transfer pairs compared to the original environment.

Q1. Where is the marginalization in Eq. 7 performed? At inference, not in training. The network (ICWM) learns the joint-state-action kernel P(st+1∣ati,at−i,Ht)P(s_{t+1} \mid a^i_t, a^{-i}_t, H_t), which stays the same when the agents' policies change. The sum over at−ia^{-i}_t is done by sampling the other agents' actions. Since the identifiability result holds on observational data, one can simply condition on (ati,at−i,Ht)(a^i_t, a^{-i}_t, H_t). For finite data, the soft PCMCI mask helps d-separate cause–effect relationships in interventional datasets (σ>0\sigma>0), i.e., it filters Zk⊆SkZ_k\subseteq S_k (Appendix E.2).

Q2. The partial random-action mixture establishes the following about identification (Section 3.3.1): the random part ζ\zeta runs a natural experiment that breaks the link to agent intent uintentu_{intent}; it produces a distributionally separate state–action target policy for the agent; it improves data coverage of the state space.

Interventions change only the behavior policy, not the physics, so every reachable state keeps positive visitation (dmin>0d_{min}>0 over the σ\sigma range used). Lag-augmented samples are approximately conditionally IID given their lagged parents, even with overlapping windows (Appendix W).

Q3. We prove our bounds for tabular MDPs. For continuous domains the results are qualitative only; extending them requires Lipschitz assumptions and covering numbers (future work). Section 4.3 now states the bound's dependence on agent count and its sample complexity, and the added Figure 28 shows the negative impact of strong interventions through dmind_{min} and training error.

Q4. When enviorment physics fsf_s is preserved and only the policy shifts, the mask adds no value under full observation. Under environment shifts, it helps when the shift corrupts a variable's value or stretches the layout along meaning-preserving axes (Anti-Causal, illegal tic-tac-toe, Stretched). The mask firewalls the surprise by keeping the corrupted variable out of predictions it is not a parent of (Fig. 6a, 26b, 27).

Thanks for clarifying Eq. 7. Which reported experiment averages predictions over sampled peer actions, and what sampling distribution and sample count does it use?

How the evaluation trajectories are generated

Two-Door, Navigation and Give-Way. We generate the peer agents' behaviour at three intervention levels, σ∈{0.1,0.5,0.9}\sigma \in \{0.1, 0.5, 0.9\}:

  • σ=0.1\sigma = 0.1 is close to the training policy (in-distribution).
  • σ=0.5\sigma = 0.5 is a shifted policy.
  • σ=0.9\sigma = 0.9 is close to random exploration.

Each evaluation set has 50 episodes. Full details are in Table 14 (Appendix O of the supplementary zip).

Navigation with 2, 3 and 4 agents. In each episode, a randomly chosen half of the agents head north-east, which the training data never show. We use 50 of these out-of-distribution episodes (Figure 30, supplementary appendix).

Monte Carlo study on shifted Tic-Tac-Toe (Figures 3c and 27). Imagined moves follow a simple tactical rule: win if possible, otherwise block, otherwise play at random. The move is always one that is legal under the new game's rules. The world model then predicts the board that results from it.

Layout-shift studies (Appendix U, supplementary material). One agent is fixed next to the goal door. The other starts in a uniformly random remaining cell, facing a random direction. We draw 200 samples per cell in the discrete environments and 100 samples per grid cell in the continuous ones.

Which reported experiment averages predictions over sampled peer actions?

The planning experiment in tic-tac-toe (Fig. 27, App. G.2, summarized in Fig. 3C). Before each move, the agent tries every legal move. For each one, it imagines 256 possible futures inside the world model, and the other player's moves in those futures are sampled: win if possible, else block, else a random legal move. The agent then plays the move whose futures turn out best on average.

All other results test the world model on data from shifted policies, where the other agents' actions come from evaluation trajectories (Fig. 4C to 4E, 5-II, 5-IV, 5-V, 21, 26b, 26c, 37 to 39, and App. U in the supplementary .zip).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.