- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies model-based RL with physical parameters that stay fixed within an episode and vary across episodes. CausalDream extends DreamerV3 with a supervised CNN–GRU estimator trained on simulator parameter labels. Before each target episode, the estimator processes eight observation–action pairs from a separate probe environment with the same physical parameters. Its estimate conditions the RSSM recurrent update. A Causal Consistency Regularizer (CCR) reconstructs this estimate from latent trajectories. Counterfactual Imagination (CFI) mixes policy rollouts with rollouts that start with a uniformly sampled action, masking that first step from actor–critic losses. Experiments cover eight DMC tasks and LunarLander, with contextual baselines, component ablations, action-effect prediction, and OOD tests using ground-truth parameter inputs.
Critical strengths
-
CausalDream achieves higher learning AUC than ContextDream on all eight DMC tasks, with a 12.91% improvement in aggregate normalized AUC (Figs. 5a and 13a). ContextDream is the authors' control model: it uses the same estimator, conditioning pathway, and backbone, but omits CCR and CFI. This comparison supports the usefulness of the proposed additions beyond simply providing estimated physical parameters. The component ablations further help identify their contributions.
-
CausalDream also has lower mean return-effect prediction error than DreamerV3 and ContextDream on all four evaluated tasks (Table 2), although the differences on Cheetah Run are small. The evaluation changes the first action from a shared simulator state while keeping subsequent actions and physical conditions fixed. CausalDream and ContextDream receive the same ground-truth parameters, so their difference cannot be attributed to estimator accuracy or access to context. These results provide evidence of better action-effect prediction alongside the gains in control performance.
Critical weaknesses
-
CausalDream starts with predefined physical factors and learns to estimate them from simulator labels. Given existing contextual world models (Appendix B), I struggle to see what the causal formulation adds to conditioning on those estimates, retaining them in the latent state, and broadening imagined experience. The gains over ContextDream support these design choices, but do not establish that deconfounding explains the improvement. Appendix F.3 illustrates the gap: CCR can reconstruct the estimate even when predictions ignore it. This leaves me unconvinced by the claimed methodological advance.
-
I am also not convinced that the experiments show how far the complete method generalizes. Most settings vary gravity or torso mass, and Figure 7 reports estimator accuracy on only two single-factor tasks, despite the additional two-factor experiments. The OOD tests then supply true parameters (Tables 1 and 10), bypassing the estimator’s fixed training grid. These results show how the world model and policy extrapolate when parameters are known, but leave open whether the short probe and learned estimator can identify unfamiliar dynamics well enough for control.
Questions for authors
-
The introduction says that the policy uses context-relevant history omitted from the model state, but Equation (5) conditions the actor on that same state, . For a fixed policy and representation, with independent action-sampling randomness, what creates residual action–parameter dependence after conditioning on in the training data? Please clarify how the state used to collect an action relates to the state reconstructed during replay, including any representation changes or latent-state resampling.
-
How does CCR compare with supplying the same estimated parameters directly to the other RSSM components and policy, as in cRSSM (Appendix B), while keeping CFI and the training budget fixed? Please also test CFI on DreamerV3 without the estimator under the same imagination budget. For the main AUC and action-effect comparisons, please report confidence intervals. The seed counts for Tables 1, 2, and 10 should also be explicit.
-
What are the per-factor and joint estimation errors in the two-factor Cheetah Run and Hopper Hop settings? Please report these alongside returns and describe how the fixed probe distinguishes the factors under randomized initial conditions. This would provide evidence for Assumption 2 beyond the single-factor examples in Figure 7.
-
On the OOD settings in Table 8, how do CausalDream and ContextDream perform when using the learned estimator? Please report parameter errors and returns relative to oracle conditioning. Testing intermediate values between training-grid points would also help assess how the complete method handles continuous parameter variation.
5 comments
Clarification of Q1 (2/5)
Our citation to Lu et al. (2018) was intended as an analogy. They study observational RL in which an unobserved confounder enters both the behavior-action model and the reward model. In our setting, the physical factor enters the transition, while Eq. (5) conditions the actor on the model state . The original passage did not make this distinction clear and could be read as saying that our actor uses history outside . We will revise the passage to distinguish the two settings explicitly.
Here, denotes the latent state actually used by the actor to select the recorded action during environment interaction, whereas denotes the latent state reconstructed from the stored sequence for the same time step during replay training.
Under the fixed policy and representation in your question, if is the exact state realization used to select the action, then independent action sampling gives $ p(a_t\mid s_t^{\mathrm{col}},u^c) =\pi_\xi(a_t\mid s_t^{\mathrm{col}}) =p(a_t\mid s_t^{\mathrm{col}}), $ and hence . Therefore, nothing creates residual action–parameter dependence when conditioning on this same collection-time state.
The distinction relevant to training is that the recorded action was generated from , whereas the model is trained using a replay-reconstructed state . The buffer stores the raw transition rather than the collection-time latent state. During replay, the model reconstructs the state from the stored sequence and draws a fresh posterior sample . Thus, even with a fixed representation, need not be the latent realization from which the recorded action was generated. During training, representation updates provide an additional source of mismatch.
If does not preserve action-relevant information contained in , and that information is correlated with , then may remain informative about the recorded action after conditioning on . For data collected by a single fixed behavior policy, with replay reconstruction based on the observation–action prefix through and fresh posterior randomness, averaging over the original collection states gives $ p(a_t\mid s_t^{\mathrm{rep}},u^c) =\int \pi_\xi(a_t\mid s_t^{\mathrm{col}}) ,p(s_t^{\mathrm{col}}\mid s_t^{\mathrm{rep}},u^c) ,\mathrm d s_t^{\mathrm{col}}. $
At the same replay state, different physical factors can imply different distributions over the original collection states. When these distributions yield different policy-weighted action probabilities, the recorded action remains dependent on after conditioning on the replay state. Consequently, $ a_t\perp u^c\mid s_t^{\mathrm{col}} \quad\not\Rightarrow\quad a_t\perp u^c\mid s_t^{\mathrm{rep}}. $
For the fixed behavior policy considered above, conditional independence is preserved when replay reconstruction retains the information needed to recover the collection-time action distribution. We will revise the text to distinguish collection-time and replay-reconstructed states and explain how posterior resampling and representation updates can leave residual action–parameter dependence.
Clarification of Q2 (3/5)
The supplementary figures referenced in this response are available at this anonymous link.
Direct multi-point injection versus CCR. We implemented a cRSSM-style direct-conditioning control, denoted DirectCond+CFI. It uses the same estimator and estimated parameters, the same CFI procedure, and the same training budget as CausalDream. The estimated parameters are supplied directly to the sequence model, observation decoder, reward and continue predictors, actor, and critic, without CCR.
Figure R1 presents the results of CausalDream, DirectCond+CFI, and ContextDream in terms of AUC, learning curves on the four tasks, cross-task aggregate curves, and normalized returns. Direct multi-point injection does not yield a statistically clear improvement over ContextDream, with an aggregate AUC ratio of 0.998 [0.899, 1.138], whereas CausalDream improves over both controls.
With the estimator, CFI procedure, and training budget held fixed, CausalDream outperforms DirectCond+CFI, with an aggregate AUC ratio of 1.184 [1.077, 1.340]. Thus, directly supplying the estimated parameters to more components does not replace the benefit of CCR in CausalDream.
Testing CFI on DreamerV3 without the estimator. We implemented a DreamerV3+CFI variant under the same imagination budget. This control neither infers nor fixes physical parameters, thereby isolating the data-augmentation effect of CFI.
Figure R2 presents the results of CausalDream, DreamerV3+CFI, and DreamerV3 in terms of AUC, learning curves on the four tasks, cross-task aggregate curves, and normalized returns. Used on its own, CFI yields an aggregate AUC ratio of 1.013 [0.944, 1.085] relative to DreamerV3, showing no statistically resolved improvement. CausalDream outperforms both DreamerV3 and DreamerV3+CFI, with aggregate AUC ratios of 1.109 [1.052, 1.179] and 1.096 [1.031, 1.194], respectively. Therefore, the gains of CausalDream cannot be attributed to applying CFI to DreamerV3 alone.
Together with the component ablations, these controls support complementary roles for CCR and CFI within CausalDream’s causally motivated framework: CCR encourages latent trajectories to retain information about the estimated exogenous factors, while CFI broadens imagined experience through action interventions with those factors held fixed. We will incorporate these comparisons and conclusions into the revision alongside Figures R1 and R2.
Confidence intervals and seed counts. We report 95% bootstrap confidence intervals for the main AUC and action-effect comparisons below, with resampling over seeds stratified by task, and will add them to the revision. Tables 1, 2, and 10 use 5 seeds.
Table R3: Aggregate normalized AUC over the 8 DMC tasks, using all available seeds per method and task over 20k–500k environment steps (ContextDream = 1).
| Method | Normalized AUC, 95% CI |
|---|---|
| CausalDream | 1.129 [1.066, 1.218] |
| HarmonyDream | 1.020 [0.948, 1.111] |
| ContextDream | 1.000 (reference) |
| DreamerV3 | 0.980 [0.908, 1.069] |
| OMPO | 0.811 [0.751, 0.883] |
| DALI-S-χ | 0.766 [0.726, 0.816] |
| TD-MPC2 | 0.683 [0.651, 0.729] |
ContextDream is the per-task normalization reference and equals 1 by construction. For Table R3, each method uses all available training seeds within each task (four or five seeds per task–method combination), without restricting the analysis to shared seed IDs. We first average AUC across seeds within each task and method, normalize by the same task's ContextDream mean, and then average equally across the eight tasks. The aggregate normalized AUC, reported to four decimal places, is 1.1291, corresponding to a +12.91% improvement over ContextDream (95% CI [+6.62%, +21.79%]). The interval uses 20,000 task-stratified percentile-bootstrap resamples of training seeds, with the eight tasks held fixed and both the method and ContextDream means recomputed in each resample.
Table R4: Four-task average improvement in action-effect prediction relative to each baseline (positive = lower CausalDream error or Regret).
| Baseline | Return, 95% CI | Reward, 95% CI | Image, 95% CI | Regret, 95% CI |
|---|---|---|---|---|
| DreamerV3 | +20.62% [-3.93, 30.09] | +20.07% [-31.88, 28.37] | +5.83% [2.63, 8.89] | +9.22% [1.77, 14.56] |
| ContextDream | +25.91% [-5.59, 35.88] | +17.29% [-24.28, 27.70] | +0.98% [-2.77, 4.34] | +2.36% [-11.45, 10.93] |
| no CCR | +33.54% [-5.38, 40.13] | +27.55% [-9.87, 32.81] | +11.86% [6.28, 15.98] | +17.20% [-3.32, 27.86] |
| no CFI | +15.11% [-31.49, 27.18] | +13.26% [-30.01, 23.85] | +8.93% [5.58, 11.65] | +5.41% [-16.17, 18.25] |
Clarification of W1 (1/5)
Thank you for this review. Predefined physical parameters, supervised estimation, and contextual conditioning are not unique to CausalDream. Our contribution is not a representational capability unavailable to contextual models, but a causal formulation and learning design for predicting and exploiting action effects under a fixed physical condition.
For an action intervention, CausalDream holds the initial model state and episode-level exogenous condition fixed and replaces only the action. CFI follows an episode-level Abduction–Action–Prediction procedure: infer the physical condition from the probe, replace the initial action-generation rule, and predict through the same learned transition mechanism. This defines an action comparison in which the inferred dynamics condition is held fixed. A generic contextual model can implement the same procedure, but contextual conditioning alone does not give it this counterfactual interpretation, which rests on the stated SCM assumptions. Our distinction from cRSSM therefore lies not in using named physical parameters, which cRSSM also uses, but in treating them as fixed exogenous conditions for intervention-oriented learning.
This perspective separates three requirements: estimating the physical condition, retaining its information during latent evolution, and learning action-conditioned responses under that condition. TEM addresses estimation; CCR encourages the estimated information to remain recoverable from the latent trajectory; CFI generates alternative-action rollouts under the same inferred condition and uses their model-predicted consequences for policy learning. Thus, parameter access does not ensure retention, and retention does not provide diverse action experience. CCR and CFI jointly connect information retention with action variation under a fixed condition. Neither alone guarantees correct causal prediction.
Our action-effect evaluation controls for parameter access: CausalDream and ContextDream receive the same ground-truth parameters, and paired simulator rollouts start from the same recorded state, change only the first action, and retain the subsequent action sequence. CausalDream achieves lower mean return-effect error on all four tasks. Relative to ContextDream, the four-task average reductions are 25.91% [-5.59%, 35.88%] for return-effect error and 2.36% [-11.45%, 10.93%] for candidate-selection regret; however, both bootstrap CIs include zero, so neither average advantage is statistically resolved. These comparisons concern action-effect prediction and candidate selection rather than estimator accuracy.
Q2 provides complementary controls. DirectCond+CFI injects the estimated parameters into multiple RSSM components and the policy with the same CFI and training budget, but without CCR. Its aggregate AUC ratio to ContextDream is 0.998 [0.899, 1.138], whereas CausalDream's ratio to DirectCond+CFI is 1.184 [1.077, 1.340]. DreamerV3+CFI without the estimator has a ratio of 1.013 [0.944, 1.085] to DreamerV3. Removing CCR also increases four-task average reward-effect error, image-effect error, and candidate-selection regret. These results support combining exogenous-information retention with fixed-condition action-intervention imagination, rather than attributing the gains to parameter estimation, broader conditioning, or extra rollouts alone.
Appendix F.3 distinguishes information retention from correct predictive use: CCR encourages information retention, while the action-effect experiments test the accuracy of the model’s intervention predictions. Deconfounding motivates our design: we seek to improve action-effect prediction by accounting for episode-level physical factors in the learned dynamics. Fig. 1 illustrates greater separation of latent representations across dynamics conditions, while the intervention experiments assess whether this design improves action-effect prediction under fixed physical conditions. We will distinguish this motivation from the established results and remove unqualified claims that conditioning or CCR “achieves deconfounding.” The SCM assumptions support CFI's interpretation, and the matched experiments demonstrate improved action-effect prediction and policy learning; they do not by themselves identify reduced confounding as the mechanism of improvement.