- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Implicit Causal World Models (ICWM) to learn environment transition dynamics from offline multi-agent demonstrations. The authors inject uniform action noise into expert policies (termed stochastic soft interventions), derive a Multi-Agent Sequential Backdoor Condition (MAS-SBC) and a tabular sample-complexity bound, and apply the time-series causal discovery algorithm PCMCI+ [1] (discarding contemporaneous edges) to construct a soft input-feature mask for a 3-layer MLP or LSTM transition model. Experiments on three low-dimensional multi-agent tasks (Two-Door, Giveway, and Navigation) evaluate graph recovery and one-step state prediction error under policy and layout shifts.
Critical strengths
Studying how correlated agent policies confound transition-model learning in multi-agent systems is a sensible problem setup, and the observation that policy exploration variance is the primary driver of out-of-distribution generalization is empirically sound.
Critical weaknesses
1.The paper's own empirical analysis directly refutes the utility of its proposed causal-masking architecture. While the title and methodology center on discovering causal graphs via PCMCI+ [1] and gating neural inputs, Figure 5 and Appendix V (Figures 26–27) demonstrate that the proposed ICWM provides zero statistically significant improvement over a standard unconstrained MLP (AWM) trained on the exact same data, while the recurrent variant R-ICWM is significantly worse. Controlling for the exploration noise parameter reduces the correlation between recovered graph quality and out-of-distribution accuracy to zero. In addition, unfair baseline comparisons in Section 5 train the competing causal baselines (FOCUS [2] and BC-WM) exclusively at while evaluating ICWM across (Table 6), confounding architectural comparisons with dataset coverage.
2.I am a bit confused by the theoretical novelty in Sections 3 and 4, though please correct me if I am mistaken. First, how does the "stochastic soft intervention" in Eq. 4 differ from standard -greedy exploration? Second, in Theorem 1, since the physical transition is Markovian and all agents' joint actions and states are observed in the dataset, why do we need a sequential backdoor adjustment over instead of directly conditioning on ? Third, in Theorem 2, doesn't the derivation simply apply the tabular Simulation Lemma [3] and finite-state concentration bounds () to show that uniform exploration guarantees a lower bound on joint-action visitation?
Questions for authors
- Since controlling for eliminates the correlation with OOD error and unmasked AWM matches ICWM (Fig. 5, App. V), what justifies the overhead of PCMCI+ over a standard world model?
- Given Axiom 1 (App. F.1) makes independent of intent given , why is backdoor adjustment over needed when full joint actions are already observed?
- Why were FOCUS and BC-WM restricted to (Table 6, Figs. 23–25) while ICWM/AWM used , and why are multiple cited appendices (e.g., G, M–O, R–U) missing from the supplement?
[1] Discovering contemporaneous and lagged causal relations in autocorrelated nonlinear time series datasets
[2] Offline reinforcement learning with causal structured world models
[3] Near-optimal reinforcement learning in polynomial time
2 comments
Author rebuttal
We thank Reviewer HWg8 for participating in the science with us. Revisions are in blue.
C1. Our paper separates two things the critique treats as one: what makes the physics learnable (intervention in the training data) and what protects the learned model under shift (the causal mask). Intervention is the main lever: once is controlled, graph-based and graph-free models match under policy shift (Fig. 6b). The mask is a prior whose value appears only when the shift reaches the model's inputs, under partial observation or corrupted and unseen variables (Fig. 6a; see Q1). We added illegal tic-tac-toe levels (App. G.1) that separate the gain from data coverage from the gain from intervention (Fig. 3a, 3b), and compare graph-based and graph-free models on them (Fig. 26a, 26b). The same view explains the baselines: FOCUS and BC-WM differ from ICWM only in and the discovery method, so training them at gives back ICWM (see Q3).
C2. The theory has one new piece, Theorem 1, and two parts that support it. (i) Eq. 4 has the form of -greedy ( with uniform ) but a different role: it is applied to the demonstrators curated datasets, for identification rather than reward. Because does not depend on the agents' hidden intent and is applied to each agent independently, it breaks the intent-action link and decorrelates peer actions, giving Theorem 1 the positivity it needs. and are design choices: trades identifiability against coverage of the states the expert visits (Fig. 16, Fig. 28), and the shape of matters, since one random move at a random time reaches 90% of OOD test moves with 49× fewer episodes than uniform randomness (Fig. 17). -greedy is one member of this family (Sec. 3.3.1). (ii) Theorem 1 is needed because the query concerns one agent's action while a shared intent couples all agents' actions; why observing the joint action is not enough is answered in Q2. (iii) Theorem 2 uses standard tools (Simulation Lemma, Weissman's L1 bound, Markov-chain Chernoff bound for dependent samples), and we claim no new proof technique. It makes Theorem 1 quantitative (Sec. 4.3, App. H): without intervention the physics off the expert's path cannot be learned; independent interventions cost samples, exponential in team size, which motivates targeted interventions; and too much intervention can starve the coordination states, which the bound alone does not capture (Fig. 28).
Q1. The mask's value depends on the shift (Fig. 6a, 6b). In the three multi-agent environments, when only the policy changes and the state is fully observed, intervention does the work, and the graph-free AWM matches ICWM (Fig. 6b). The mask helps under partial observation: on the Stretched Two-Door layout, every masked model beats AWM (App. V, Fig. 42). Tic-tac-toe shows the same split: on legal games, intervention drives transfer; on illegal games (moves out of turn, never seen in training), the masked model transfers almost exactly while the unmasked one gets worse (Fig. 3b, Fig. 26). The mask is a firewall for corrupted or unseen inputs.
Q2. Axiom 1 (App. F) makes conditioning on the full joint action valid: it removes the hidden intent from the physics, so is the physics that ICWM learns, and this conditioning is the backdoor adjustment. The averaging in Theorem 1 is needed because the question is about one agent's action under shift, . Intent ties the agents' actions together (Axiom 3), so in the data the peers move with agent ; changing agent 's action does not change the others' (Axiom 2), so we average over what the peers do on their own, (Eq. 7). stays because intent persists and past actions reveal it. Our data are offline: with expert-only data () each action is fixed by intent, so the model learns intent-driven correlations instead of at joint actions the expert never takes; fixes this.
Q3. FOCUS and BC-WM stay at because their original methods learn from expert demonstrations only. BC-WM uses the same PCMCI+ graph and mask as ICWM (Table 8), so with interventional data it is ICWM; FOCUS with interventional data is ICWM with a PC graph. Keeping them at isolates our claim that the data, not the architecture, drives transfer. The matched comparison is in our tic-tac-toe study: Fig. 19 compares the graphs found by PC and PCMCI+ at every dataset size and intervention level, and Fig. 21 trains world models on the same data with a PC mask (the FOCUS setting), a PCMCI+ mask, the true mask, a shuffled true mask of the same sparsity, or no mask. Learned masks stay close to the true one; the shuffled mask predicts OOD states worse. The cited appendices (e.g., M–O, R–U; manuscript line 870) are in the detailed appendix in the supplementary material (.zip).
No rebuttal provided; the original score remains unchanged.