acceptodds
Under review as a conference paper at ICLR 2027

CausalDream: Causal World Models by Deconfounding of Dynamics in Episodically Non-Stationary Environments

15 views

Abstract

World models trained on observational data can conflate the effects of actions with those of unobserved physical factors, such as gravity or mass, that stay fixed within an episode but vary across episodes. When the latent state fails to retain these factors, they confound the learned dynamics and undermine policy robustness. Simply conditioning the model on an inferred context neither ensures that the dynamics use it nor exploits it to generate experience beyond the current policy. We propose CausalDream, which treats these factors as exogenous variables of a structural causal model. Its Causal World Model conditions latent transitions on exogenous variables that a label-supervised estimator infers from a short probe trajectory. A Causal Consistency Regularizer reconstructs these estimates from latent trajectories so that the dynamics retain them. Counterfactual imagination then intervenes on the first imagined action while holding the estimates fixed, broadening the experience available for policy learning. Across eight DeepMind Control tasks, CausalDream achieves the highest aggregate learning AUC among all compared methods, 12.9% above an architecture-matched baseline that conditions on the same estimates without the causal components. It also predicts the effects of action interventions more accurately on all four evaluated tasks, showing that its gains go beyond context conditioning alone.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, World Model, MBRL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_L5tHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies model-based RL with physical parameters that stay fixed within an episode and vary across episodes. CausalDream extends DreamerV3 with a supervised CNN–GRU estimator trained on simulator parameter labels. Before each target episode, the estimator processes eight observation–action pairs from a separate probe environment with the same physical parameters. Its estimate conditions the RSSM recurrent update. A Causal Consistency Regularizer (CCR) reconstructs this estimate from latent trajectories. Counterfactual Imagination (CFI) mixes policy rollouts with rollouts that start with a uniformly sampled action, masking that first step from actor–critic losses. Experiments cover eight DMC tasks and LunarLander, with contextual baselines, component ablations, action-effect prediction, and OOD tests using ground-truth parameter inputs.

Critical strengths
  1. CausalDream achieves higher learning AUC than ContextDream on all eight DMC tasks, with a 12.91% improvement in aggregate normalized AUC (Figs. 5a and 13a). ContextDream is the authors' control model: it uses the same estimator, conditioning pathway, and backbone, but omits CCR and CFI. This comparison supports the usefulness of the proposed additions beyond simply providing estimated physical parameters. The component ablations further help identify their contributions.

  2. CausalDream also has lower mean return-effect prediction error than DreamerV3 and ContextDream on all four evaluated tasks (Table 2), although the differences on Cheetah Run are small. The evaluation changes the first action from a shared simulator state while keeping subsequent actions and physical conditions fixed. CausalDream and ContextDream receive the same ground-truth parameters, so their difference cannot be attributed to estimator accuracy or access to context. These results provide evidence of better action-effect prediction alongside the gains in control performance.

Critical weaknesses
  1. CausalDream starts with predefined physical factors and learns to estimate them from simulator labels. Given existing contextual world models (Appendix B), I struggle to see what the causal formulation adds to conditioning on those estimates, retaining them in the latent state, and broadening imagined experience. The gains over ContextDream support these design choices, but do not establish that deconfounding explains the improvement. Appendix F.3 illustrates the gap: CCR can reconstruct the estimate even when predictions ignore it. This leaves me unconvinced by the claimed methodological advance.

  2. I am also not convinced that the experiments show how far the complete method generalizes. Most settings vary gravity or torso mass, and Figure 7 reports estimator accuracy on only two single-factor tasks, despite the additional two-factor experiments. The OOD tests then supply true parameters (Tables 1 and 10), bypassing the estimator’s fixed training grid. These results show how the world model and policy extrapolate when parameters are known, but leave open whether the short probe and learned estimator can identify unfamiliar dynamics well enough for control.

Questions for authors
  1. The introduction says that the policy uses context-relevant history omitted from the model state, but Equation (5) conditions the actor on that same state, (ht,zt)(h_t,z_t). For a fixed policy and representation, with independent action-sampling randomness, what creates residual action–parameter dependence after conditioning on (ht,zt)(h_t,z_t) in the training data? Please clarify how the state used to collect an action relates to the state reconstructed during replay, including any representation changes or latent-state resampling.

  2. How does CCR compare with supplying the same estimated parameters directly to the other RSSM components and policy, as in cRSSM (Appendix B), while keeping CFI and the training budget fixed? Please also test CFI on DreamerV3 without the estimator under the same imagination budget. For the main AUC and action-effect comparisons, please report confidence intervals. The seed counts for Tables 1, 2, and 10 should also be explicit.

  3. What are the per-factor and joint estimation errors in the two-factor Cheetah Run and Hopper Hop settings? Please report these alongside returns and describe how the fixed probe distinguishes the factors under randomized initial conditions. This would provide evidence for Assumption 2 beyond the single-factor examples in Figure 7.

  4. On the OOD settings in Table 8, how do CausalDream and ContextDream perform when using the learned estimator? Please report parameter errors and returns relative to oracle conditioning. Testing intermediate values between training-grid points would also help assess how the complete method handles continuous parameter variation.

5 comments
AuthorsOpenReview ↗

Clarification of W1 (1/5)

Thank you for this review. Predefined physical parameters, supervised estimation, and contextual conditioning are not unique to CausalDream. Our contribution is not a representational capability unavailable to contextual models, but a causal formulation and learning design for predicting and exploiting action effects under a fixed physical condition.

For an action intervention, CausalDream holds the initial model state and episode-level exogenous condition fixed and replaces only the action. CFI follows an episode-level Abduction–Action–Prediction procedure: infer the physical condition from the probe, replace the initial action-generation rule, and predict through the same learned transition mechanism. This defines an action comparison in which the inferred dynamics condition is held fixed. A generic contextual model can implement the same procedure, but contextual conditioning alone does not give it this counterfactual interpretation, which rests on the stated SCM assumptions. Our distinction from cRSSM therefore lies not in using named physical parameters, which cRSSM also uses, but in treating them as fixed exogenous conditions for intervention-oriented learning.

This perspective separates three requirements: estimating the physical condition, retaining its information during latent evolution, and learning action-conditioned responses under that condition. TEM addresses estimation; CCR encourages the estimated information to remain recoverable from the latent trajectory; CFI generates alternative-action rollouts under the same inferred condition and uses their model-predicted consequences for policy learning. Thus, parameter access does not ensure retention, and retention does not provide diverse action experience. CCR and CFI jointly connect information retention with action variation under a fixed condition. Neither alone guarantees correct causal prediction.

Our action-effect evaluation controls for parameter access: CausalDream and ContextDream receive the same ground-truth parameters, and paired simulator rollouts start from the same recorded state, change only the first action, and retain the subsequent action sequence. CausalDream achieves lower mean return-effect error on all four tasks. Relative to ContextDream, the four-task average reductions are 25.91% [-5.59%, 35.88%] for return-effect error and 2.36% [-11.45%, 10.93%] for candidate-selection regret; however, both bootstrap CIs include zero, so neither average advantage is statistically resolved. These comparisons concern action-effect prediction and candidate selection rather than estimator accuracy.

Q2 provides complementary controls. DirectCond+CFI injects the estimated parameters into multiple RSSM components and the policy with the same CFI and training budget, but without CCR. Its aggregate AUC ratio to ContextDream is 0.998 [0.899, 1.138], whereas CausalDream's ratio to DirectCond+CFI is 1.184 [1.077, 1.340]. DreamerV3+CFI without the estimator has a ratio of 1.013 [0.944, 1.085] to DreamerV3. Removing CCR also increases four-task average reward-effect error, image-effect error, and candidate-selection regret. These results support combining exogenous-information retention with fixed-condition action-intervention imagination, rather than attributing the gains to parameter estimation, broader conditioning, or extra rollouts alone.

Appendix F.3 distinguishes information retention from correct predictive use: CCR encourages information retention, while the action-effect experiments test the accuracy of the model’s intervention predictions. Deconfounding motivates our design: we seek to improve action-effect prediction by accounting for episode-level physical factors in the learned dynamics. Fig. 1 illustrates greater separation of latent representations across dynamics conditions, while the intervention experiments assess whether this design improves action-effect prediction under fixed physical conditions. We will distinguish this motivation from the established results and remove unqualified claims that conditioning or CCR “achieves deconfounding.” The SCM assumptions support CFI's interpretation, and the matched experiments demonstrate improved action-effect prediction and policy learning; they do not by themselves identify reduced confounding as the mechanism of improvement.

AuthorsOpenReview ↗

Clarification of Q1 (2/5)

Our citation to Lu et al. (2018) was intended as an analogy. They study observational RL in which an unobserved confounder enters both the behavior-action model and the reward model. In our setting, the physical factor enters the transition, while Eq. (5) conditions the actor on the model state st=(ht,zt)s_t=(h_t,z_t). The original passage did not make this distinction clear and could be read as saying that our actor uses history outside sts_t. We will revise the passage to distinguish the two settings explicitly.

Here, stcols_t^{\mathrm{col}} denotes the latent state actually used by the actor to select the recorded action ata_t during environment interaction, whereas streps_t^{\mathrm{rep}} denotes the latent state reconstructed from the stored sequence for the same time step during replay training.

Under the fixed policy and representation in your question, if stcol=(htcol,ztcol)s_t^{\mathrm{col}}=(h_t^{\mathrm{col}},z_t^{\mathrm{col}}) is the exact state realization used to select the action, then independent action sampling gives $ p(a_t\mid s_t^{\mathrm{col}},u^c) =\pi_\xi(a_t\mid s_t^{\mathrm{col}}) =p(a_t\mid s_t^{\mathrm{col}}), $ and hence at⊥uc∣stcola_t\perp u^c\mid s_t^{\mathrm{col}}. Therefore, nothing creates residual action–parameter dependence when conditioning on this same collection-time state.

The distinction relevant to training is that the recorded action was generated from stcols_t^{\mathrm{col}}, whereas the model is trained using a replay-reconstructed state strep=(htrep,ztrep)s_t^{\mathrm{rep}}=(h_t^{\mathrm{rep}},z_t^{\mathrm{rep}}). The buffer stores the raw transition rather than the collection-time latent state. During replay, the model reconstructs the state from the stored sequence and draws a fresh posterior sample ztrep∼qϕ(⋅∣htrep,ot)z_t^{\mathrm{rep}}\sim q_\phi(\cdot\mid h_t^{\mathrm{rep}},o_t). Thus, even with a fixed representation, ztrepz_t^{\mathrm{rep}} need not be the latent realization from which the recorded action was generated. During training, representation updates provide an additional source of mismatch.

If streps_t^{\mathrm{rep}} does not preserve action-relevant information contained in stcols_t^{\mathrm{col}}, and that information is correlated with ucu^c, then ucu^c may remain informative about the recorded action after conditioning on streps_t^{\mathrm{rep}}. For data collected by a single fixed behavior policy, with replay reconstruction based on the observation–action prefix through oto_t and fresh posterior randomness, averaging over the original collection states gives $ p(a_t\mid s_t^{\mathrm{rep}},u^c) =\int \pi_\xi(a_t\mid s_t^{\mathrm{col}}) ,p(s_t^{\mathrm{col}}\mid s_t^{\mathrm{rep}},u^c) ,\mathrm d s_t^{\mathrm{col}}. $

At the same replay state, different physical factors can imply different distributions over the original collection states. When these distributions yield different policy-weighted action probabilities, the recorded action remains dependent on ucu^c after conditioning on the replay state. Consequently, $ a_t\perp u^c\mid s_t^{\mathrm{col}} \quad\not\Rightarrow\quad a_t\perp u^c\mid s_t^{\mathrm{rep}}. $

For the fixed behavior policy considered above, conditional independence is preserved when replay reconstruction retains the information needed to recover the collection-time action distribution. We will revise the text to distinguish collection-time and replay-reconstructed states and explain how posterior resampling and representation updates can leave residual action–parameter dependence.

AuthorsOpenReview ↗

Clarification of Q2 (3/5)

The supplementary figures referenced in this response are available at this anonymous link.

Direct multi-point injection versus CCR. We implemented a cRSSM-style direct-conditioning control, denoted DirectCond+CFI. It uses the same estimator and estimated parameters, the same CFI procedure, and the same training budget as CausalDream. The estimated parameters are supplied directly to the sequence model, observation decoder, reward and continue predictors, actor, and critic, without CCR.

Figure R1 presents the results of CausalDream, DirectCond+CFI, and ContextDream in terms of AUC, learning curves on the four tasks, cross-task aggregate curves, and normalized returns. Direct multi-point injection does not yield a statistically clear improvement over ContextDream, with an aggregate AUC ratio of 0.998 [0.899, 1.138], whereas CausalDream improves over both controls.

With the estimator, CFI procedure, and training budget held fixed, CausalDream outperforms DirectCond+CFI, with an aggregate AUC ratio of 1.184 [1.077, 1.340]. Thus, directly supplying the estimated parameters to more components does not replace the benefit of CCR in CausalDream.

Testing CFI on DreamerV3 without the estimator. We implemented a DreamerV3+CFI variant under the same imagination budget. This control neither infers nor fixes physical parameters, thereby isolating the data-augmentation effect of CFI.

Figure R2 presents the results of CausalDream, DreamerV3+CFI, and DreamerV3 in terms of AUC, learning curves on the four tasks, cross-task aggregate curves, and normalized returns. Used on its own, CFI yields an aggregate AUC ratio of 1.013 [0.944, 1.085] relative to DreamerV3, showing no statistically resolved improvement. CausalDream outperforms both DreamerV3 and DreamerV3+CFI, with aggregate AUC ratios of 1.109 [1.052, 1.179] and 1.096 [1.031, 1.194], respectively. Therefore, the gains of CausalDream cannot be attributed to applying CFI to DreamerV3 alone.

Together with the component ablations, these controls support complementary roles for CCR and CFI within CausalDream’s causally motivated framework: CCR encourages latent trajectories to retain information about the estimated exogenous factors, while CFI broadens imagined experience through action interventions with those factors held fixed. We will incorporate these comparisons and conclusions into the revision alongside Figures R1 and R2.

Confidence intervals and seed counts. We report 95% bootstrap confidence intervals for the main AUC and action-effect comparisons below, with resampling over seeds stratified by task, and will add them to the revision. Tables 1, 2, and 10 use 5 seeds.

Table R3: Aggregate normalized AUC over the 8 DMC tasks, using all available seeds per method and task over 20k–500k environment steps (ContextDream = 1).

MethodNormalized AUC, 95% CI
CausalDream1.129 [1.066, 1.218]
HarmonyDream1.020 [0.948, 1.111]
ContextDream1.000 (reference)
DreamerV30.980 [0.908, 1.069]
OMPO0.811 [0.751, 0.883]
DALI-S-χ0.766 [0.726, 0.816]
TD-MPC20.683 [0.651, 0.729]

ContextDream is the per-task normalization reference and equals 1 by construction. For Table R3, each method uses all available training seeds within each task (four or five seeds per task–method combination), without restricting the analysis to shared seed IDs. We first average AUC across seeds within each task and method, normalize by the same task's ContextDream mean, and then average equally across the eight tasks. The aggregate normalized AUC, reported to four decimal places, is 1.1291, corresponding to a +12.91% improvement over ContextDream (95% CI [+6.62%, +21.79%]). The interval uses 20,000 task-stratified percentile-bootstrap resamples of training seeds, with the eight tasks held fixed and both the method and ContextDream means recomputed in each resample.

Table R4: Four-task average improvement in action-effect prediction relative to each baseline (positive = lower CausalDream error or Regret).

BaselineReturn, 95% CIReward, 95% CIImage, 95% CIRegret, 95% CI
DreamerV3+20.62% [-3.93, 30.09]+20.07% [-31.88, 28.37]+5.83% [2.63, 8.89]+9.22% [1.77, 14.56]
ContextDream+25.91% [-5.59, 35.88]+17.29% [-24.28, 27.70]+0.98% [-2.77, 4.34]+2.36% [-11.45, 10.93]
no CCR+33.54% [-5.38, 40.13]+27.55% [-9.87, 32.81]+11.86% [6.28, 15.98]+17.20% [-3.32, 27.86]
no CFI+15.11% [-31.49, 27.18]+13.26% [-30.01, 23.85]+8.93% [5.58, 11.65]+5.41% [-16.17, 18.25]
Reviewer_xhZxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets episodic non-stationarity. Physical factors such as gravity or mass stay fixed inside an episode but change across episodes, and if the latent state of a world model does not keep them, they confound the learned dynamics. CausalDream treats these factors as exogenous variables ucu_c of an episode-level SCM and adds three pieces on top of DreamerV3. A Twin Environment Mechanism (TEM) runs a short fixed probe trajectory in a parallel environment that mirrors the target one, and a GRU estimator predicts ucu_c from it, trained as classification over a discrete support with simulator parameters as labels. The Causal World Model feeds the estimate into the recurrent state, and a Causal Consistency Regularizer (CCR) reconstructs the estimate from the latent trajectory so it is not dropped. Counterfactual Imagination (CFI) replaces the first imagined action with a uniformly sampled one while holding the estimate and initial state fixed. Nine simulated control tasks are used.

Critical strengths
  • ContextDream shares the estimator, the conditioning pathway and the backbone, so the AUC gap, the OOD gaps in Table 1 and the action-effect gaps in Table 2 all isolate the same question: whether conditioning alone explains the gains. That claim is tested three ways rather than one.
  • The dependence on privileged pre-training is at least tested rather than deferred to the limitations paragraph. Sec. 5.1 perturbs the exogenous input within the training range and finds CausalDream keeps higher returns than ContextDream in most perturbed settings and loses less on Cartpole under large errors.
Critical weaknesses
  • The estimator is a classifier over a known discrete support, trained on simulator labels, with a fixed probe policy. Sec. 4.1 pre-trains it with simulator-provided parameters as privileged labels, minimizing cross-entropy over each factor's discrete support, on trajectories from a pre-defined policy. Sec. 5 samples the factors from discrete uniform distributions and states that all methods are trained and evaluated under the same task-specific parameter distributions. So "requires only observation-action trajectories at deployment" understates what is given: the range, the discretization, the sampling law, and a probe policy matching the one used to collect the training data.
  • Minor: the abstract says the model predicts action interventions more accurately on all four tasks, while Table 2 has it worse than other methods on candidate selection regret for two of them.
Questions for authors
  • What is the gain over ContextDream in the aggregate-return comparison of Sec. 5.1, where 30.06% is against the mean of five baselines?
1 comment
AuthorsOpenReview ↗

Clarification of W1, W2 and Q1

We thank the reviewer for the careful reading.

Clarification of W1

We will clarify both in Secs. 1 and 4.1. Simulator-provided parameters supervise estimator pre-training, with the following task-specific settings:

SettingSpecificationReference
Parameter rangeThe range is specified before estimator pre-training.App. C.2, Tables 6 and 7
DiscretizationEach factor has a fixed discrete support. Inference returns the support value with the highest predicted probability.Sec. 4.1; App. A.2
Sampling lawFactors are sampled uniformly from their discrete supports for estimator pre-training. The main in-distribution comparisons use the same task-specific parameter distributions across methods.App. A.2; Sec. 5; App. C.2
Probe protocolA task-specific fixed action sequence is used for both pre-training data collection and inference.App. A.2

Simulator labels align each estimated component with its physical factor in the SCM. Holding the estimate fixed in CFI therefore explicitly holds the estimated episode-level physical condition fixed (Sec. 4.1; Remark 9, App. F.2).

We will replace the deployment statement in Sec. 4.1 with: "For estimator pre-training, we specify the parameter ranges and discrete supports, sample factors uniformly over these supports, and collect trajectories with simulator-provided parameter labels under a task-specific fixed probe sequence. At deployment, the estimator maps an observation-action trajectory collected under the same probe protocol to an estimate on the predefined support, without ground-truth parameter inputs."

The corresponding statement in Sec. 1 will summarize the predefined-support and matched-probe requirements and refer to Sec. 4.1 for the full pre-training protocol.

Clarification of W2

We agree that both the outcome and the comparator should be explicit.

Return-effect MSE measures the numerical error in the predicted intervention-induced change in return. Candidate-selection regret measures the gap between the best candidate's realized return and the realized return of the candidate action selected using the model's predictions. The two metrics are related but distinct: the four-task statement concerns return-effect MSE, not regret. CausalDream achieves lower mean return-effect MSE than both DreamerV3 and ContextDream on each of the four tasks, reducing it by 3.2% to 60.6% relative to ContextDream (Sec. 5.3). It also achieves lower mean reward-effect MSE than both methods on all four tasks (Table 12). For candidate selection, CausalDream attains the lowest mean regret on Hopper Hop and Walker Run; Sec. 5.3 reports the complete four-task results and discusses the task-dependent selection gains.

The abstract sentence will read: “It also achieves lower mean return-effect prediction error than both DreamerV3 and ContextDream on each of the four counterfactual evaluation tasks, supporting gains from the combined causal design beyond context conditioning alone.”

Clarification of Q1

For the aggregate-return comparison underlying the 30.06% figure, the gain over ContextDream is 14.36%. On the four Run and Hop tasks, CausalDream achieves an average final return of 373.5 at 500k environment steps, compared with 326.6 for ContextDream (App. E.3). The 30.06% figure uses the arithmetic mean of the five world-model baselines’ group-averaged final returns, including ContextDream, as the reference (287.18). Thus, $ \left(\frac{373.5}{326.6}-1\right)\times100\%\approx14.36\%. $ $ \left(\frac{373.5}{287.18}-1\right)\times100\%\approx30.06\%. $

The two percentages therefore describe the same final-return result using different reference values.

For completeness, on the three Stand and Walk tasks, the corresponding final-return gain over ContextDream is 7.42% (722.2 vs. 672.3; Fig. 14), compared with 17.69% over the five-baseline mean. For learning AUC, CausalDream exceeds ContextDream by 12.91% in aggregate across all eight DMC tasks, with per-task gains ranging from 4.60% on Cheetah Run to 25.90% on Hopper Hop and positive gains on every task (Sec. 5.1).

We will revise Sec. 5.1 to state both comparison references explicitly: “On the Run and Hop tasks, CausalDream achieves an average final return of 373.5 at 500k environment steps, which is 14.36% higher than ContextDream (326.6) and 30.06% higher than the mean of the five world-model baselines (287.18).”


We hope that the clarified deployment assumptions, corrected abstract statement, and numerical comparisons address your concerns. We will incorporate these revisions into the manuscript. If you feel that these responses adequately address W1, W2, and Q1, we would be grateful if you would consider a higher rating.

Reviewer_qM77OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This submission presents a model-based reinforcement learning framework called CausalDream, designed to achieve robust policy optimization in episodically non-stationary environments. Conventional world models often struggle in these settings because they fail to capture unobserved physical confounders (such as varying gravity or payload mass) that stay fixed within an episode but change across episodes, leading to spurious correlations and brittle policy performance. To address this limitation, CausalDream treats physical factors as exogenous variables of a structural causal model (SCM). It features three core architectural components: i) a Twin Environment Mechanism (TEM) that extracts dynamics properties from a brief pre-task probing trajectory without acting blindly; ii) a Causal Consistency Regularizer (CCR) that forces the compressed latent state to actively retain and reconstruct these exogenous factors; and iii) Counterfactual Imagination (CFI), which holds the inferred environmental condition fixed while uniformly intervening on alternative actions to generate a diversified, unbiased trajectory distribution for policy learning. Empirical evaluations across eight continuous DeepMind Control tasks and one discrete Gymnasium task demonstrate that CausalDream significantly improves sample efficiency, generalization, and predictive accuracy.

Critical strengths
  1. Rather than relying on simple context conditioning or unconstrained latent representation vectors, CausalDream uniquely casts episodic non-stationarity into an episode-level Structural Causal Model (SCM), explicitly isolating physical exogenous factors (like gravity or mass) to block spurious action-outcome correlations. The framework also addresses a major flaw in standard world models where policies only learn from familiar trajectory branches. By introducing Counterfactual Imagination (CFI), it strategically intervenes on the initial imagined actions under fixed exogenous parameters to enrich the state distribution for robust actor-critic learning.

  2. Recognizing that powerful sequence models tend to ignore low-dimensional context signals, this submission introduces a Causal Consistency Regularizer (CCR). A formal information-theoretic proof (Proposition 10) confirms that bounded reconstruction errors guarantee explicit factor retention.

  3. The inclusion of the Twin Environment Mechanism (TEM) ensures that system identification is isolated from task execution. This prevents edge platforms from acting blindly or suffering immediate early-stage degradation during the initial tracking sequence of an unknown deployment episode.

Critical weaknesses
  1. The exogenous estimator relies heavily on simulator-provided physical parameters as privileged labels during pre-training. For complex real-world robotics tasks where ground-truth parameters (such as friction coefficients, joint damping, or center-of-mass distributions) cannot be precisely modeled in a simulator, this pre-training pipeline may break down. In addition, the SCM formulation assumes that exogenous physical factors remain strictly constant within an episode and only shift stochastically at boundary resets. This may restrict its real-world applicability to scenarios with continuously drifting environments (e.g., continuous payload leaking, fading battery voltages, or variable wind gusts during an active flight).

  2. To capture multimodal uncertainty, the exogenous parameters are heavily quantized into rigid discrete support bins to train via cross-entropy with Gaussian soft targets. This design introduces an unaddressed accuracy trade-off: a small bin increment triggers massive dimensionality expansion for the multi-task categorical cross-entropy loss, while a large increment induces high discretization error.

  3. Operating CausalDream on physical edge platforms requires a distinct pre-task calibration phase (to mirror TEM), alongside running an autoregressive sequence model, an extra GRU-based CCR decoder trajectory branch, and continuous multi-branch counterfactual action rollouts. The computational burden and latency overhead of this complete causal loop are not quantified for physical hardware deployment.

  4. As shown in the paired action-intervention analysis (Table 2), CausalDream’s candidate-action selection performance is highly inconsistent. On tasks like Cheetah Run and Cartpole Swingup, simpler models like ContextDream or variants without CFI achieve lower candidate-selection regret, suggesting that uniform alternative action sampling may introduce excessive noise or prediction errors into certain control spaces.

  5. The discussion of the literature need to consider most recent research efforts.

Questions for authors

Please check the details in the Critical Weaknesses section and clarify.

4 comments
AuthorsOpenReview ↗

Clarification of W1 (1/3)

We thank the reviewer for highlighting the requirements and scope of our approach. Our estimator uses simulator-generated physical-parameter labels only during pre-training. These labels are automatically obtained from sampled simulator parameters, requiring neither manual annotation nor real-world parameter measurements. At deployment, the estimator infers the context from a short observation–action probe, without ground-truth parameter inputs. This supervision aligns the estimated variables with the physical factors in the SCM. Accordingly, reliable estimation requires the pre-training simulator to cover the relevant physical factors and probe responses of the target system, and the probe to provide sufficient information for identifying those factors. We will make these requirements explicit in the revised manuscript.

Our context-input error experiment examines robustness to inaccurate parameter inputs. On four tasks, we supply uinc=uc+ϵu_{\mathrm{in}}^c=u^c+\epsilon, with ϵ∈{−2,−1,+1,+2}\epsilon\in\lbrace-2,-1,+1,+2\rbrace, while keeping the true environment unchanged and both the original and perturbed parameter values within the training range. CausalDream achieves higher returns than ContextDream in 14 of the 16 perturbed settings. On Cartpole Swingup, its return losses at ϵ=−2\epsilon=-2 and +2+2 are 12.1% and 22.6%, compared with 24.2% and 39.0% for ContextDream. These results demonstrate tolerance to the tested input errors, with CausalDream preserving its performance advantage in most settings.

Regarding the episode-level formulation, CausalDream targets episodic non-stationarity, a well-established setting in contextual MDPs, HiP-MDPs, and context-adaptive world models such as CaDM, cRSSM, and DALI. The SCM represents exogenous factors as constant within an episode; in practice, this formulation is suitable when their variation is negligible over a short calibration-and-control horizon, with changes occurring across episodes or trials. Examples include a fixed payload during a manipulation trial and stable terrain during a short locomotion run. The continuously drifting examples mentioned by the reviewer represent a complementary within-episode time-varying regime. As noted in our conclusion, extending the SCM to time-dependent exogenous variables utcu_t^c, together with online context updates in the world model and imagination process, is a natural direction for future work. We will supplement the discussion of the current assumptions and the case of time-varying exogenous variables, and provide more specific example applications to better illustrate the motivation for the episode-level setting. Our current formulation addresses a well-defined and widely studied problem class, and its empirical effectiveness is supported by experiments across nine tasks with episode-level dynamics shifts.

AuthorsOpenReview ↗

Clarification of W2 and W3 (2/3)

Clarification of W2

In our experimental protocol, the physical factors are sampled from finite ordered supports, and the estimator directly learns categorical distributions over the corresponding support values. The support spacing is set to Δ=0.5\Delta=0.5 or 1.01.0 depending on the task, giving 29 support values for gravity, 20 for Hopper torso mass, and 5 and 11 for the additional wind and knee-actuator factors.

The support spacing is chosen according to physical distinguishability as well as numerical resolution. If the spacing is too small relative to the sensitivity of a task, neighboring parameter values may induce nearly indistinguishable transitions and responses induced by a short probe. For example, changing the magnitude of gravity from 9.8 to 9.9 m/s² may have little observable effect over a short probe. Such extremely fine subdivisions would increase the number of categories without creating correspondingly distinct dynamics conditions. We therefore use Δ=0.5\Delta=0.5 or 1.01.0 to emphasize task-relevant dynamics variation while retaining multiple ordered support values across each parameter range.

The estimator uses a CNN--GRU feature extractor followed by categorical prediction over the support of each physical factor. For a factor with KK support values, changing KK only changes the final classifier layer; the visual encoder and recurrent backbone remain unchanged. When several factors vary jointly, their categorical outputs and losses are handled factor-wise, so the aggregate number of output logits grows as ∑iKi\sum_i K_i rather than as the joint Cartesian product ∏iKi\prod_i K_i. This avoids combinatorial expansion while preserving a separate ordered support for each factor.

For training, we construct Gaussian soft targets with σ=0.5\sigma=0.5 and optimize the KL divergence to these targets, which is equivalent to soft-label cross-entropy up to a target-dependent constant. Neighboring physical values therefore receive similar supervision, while distant values receive less. The predicted categorical distribution preserves uncertainty over the finite support, and the deterministic estimate used by the controller is the support value corresponding to the highest-probability category.

For uniformly spaced supports, any parameter value within the covered interval has a nearest support value within Δ/2\Delta/2, corresponding to 0.25 or 0.5 under our chosen spacings. This characterizes the support's representation resolution; the learned estimator's prediction accuracy is assessed empirically. We additionally evaluated six unseen within-range gravity midpoints on each of Walker Run and Cheetah Run, using estimators trained on the original Δ=0.5\Delta=0.5 grid without retraining. The per-midpoint mean absolute estimation errors ranged from 0.25 to 0.40 in gravity units on both tasks, providing direct evidence of estimation accuracy between training support values.

We will clarify these design choices and the role of physical distinguishability in the revised manuscript.

Clarification of W3

We clarify that CCR and CFI are used only during training, rather than during frozen-policy deployment. CCR provides an auxiliary reconstruction loss for world-model learning, while CFI generates factual and counterfactual imagined trajectories for actor--critic training.

In simulation, TEM uses a parallel probe environment. On a physical platform, its practical analogue is one short pre-task probe or calibration phase on the same platform, without requiring a duplicated environment. The probe trajectory is processed once by the estimator Eθ\mathcal{E}_\theta to obtain the episode-level exogenous-variable estimate. After this one-time estimation, each control step requires one RSSM state update and one actor forward pass. Frozen-policy deployment does not run the CCR decoder or loss, counterfactual imagination, or parameter updates. We will revise the pseudocode to distinguish the training-time components from the frozen-policy execution path.

AuthorsOpenReview ↗

Clarification of W4 and W5 (3/3)

Clarification of W4

This metric is a complementary finite-horizon diagnostic: it measures the return gap between the best candidate action and the action selected using the model's predicted returns under a shared action suffix. It is distinct from both action-effect prediction error and closed-loop policy performance.

During training, CFI samples one initial imagined action uniformly to broaden the experience available for policy learning, and subsequent actions are generated by the current policy. In the paired action-intervention evaluation, all methods use the same fixed, task-specific candidate sets. Cheetah Run uses Gaussian action perturbations, whereas Cartpole Swingup uses a nine-point grid; subsequent actions are fixed across methods. Therefore, the reported regret differences do not by themselves establish that CFI's uniform training sampler causes prediction noise in these tasks.

The two metrics can rank methods differently because finite-set candidate ranking is sensitive to local ordering errors, whereas effect MSE measures average intervention-effect prediction error. CausalDream achieves lower Return-effect MSE than DreamerV3 and ContextDream on all four tasks, including Cheetah Run (79.9 vs. 82.5 for ContextDream) and Cartpole Swingup (30.2 vs. 76.7). The different regret ranking on Cheetah Run therefore reflects the distinction between candidate ranking and aggregate intervention-effect prediction.

We will revise Sec. 5.3 to make this metric distinction and the candidate-selection protocol more explicit, while retaining the reported task-dependent regret results.

Clarification of W5

We will supplement the latest research work in areas such as causal world models and non-stationary dynamics.


We hope that the clarified applicability conditions, resolution analysis, deployment path, and metric distinctions address your concerns. We will incorporate these clarifications and the additional results into the revised manuscript, and will also expand the discussion of recent related work.

Thanks the authors for the response. After reviewing the responses and other reviewers' comments/rebuttal, I decided to maintain my assessment score.

Comments on the paper

1 comment

Dear authors and reviewers,

The author-reviewer discussion period closes on November 18 (AoE). Please engage in response periods so that we have our issues discussed thoroughly.

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.