acceptodds
Under review as a conference paper at ICLR 2027

Look Back: Reinforcement Learning with Counterfactual World Models

Abstract

Agents in complex, stochastic environments must contend with dynamics that can exceed their world models’ predictive capacity and randomness that can obscure the consequences of their actions. We hypothesize that, compared to imagining entirely new trajectories in such complex environments, it can be easier to transform observed trajectories with alternative actions. Counterfactual world models condition on observations to predict alternative outcomes under the same underlying environmental randomness, using experience as a reference for predicting action-dependent changes. We present a policy gradient estimator, REINFORCE Look Back (RLB), that uses counterfactual returns as baselines to distinguish action effects from environmental luck, and establish conditions for unbiasedness and variance reduction. In settings where environmental evolution is harder to predict than action-induced changes, we show that counterfactual world models are easier to learn and support better policy learning than ordinary world models. Our findings suggest that counterfactual thinking can help agents learn effective behavior without independently reproducing the full complexity of their worlds.

Primary area:
causal reasoning
Keywords:
policy gradients, counterfactuals, diffusion models, big world hypothesis

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_rF55OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this work the authors study counterfactual world models. These predict alternative outcomes under shared environmental randomness when condition on observed experiences. They introduce REINFORCE Look Back (RLB), which uses counterfactual returns as leave-one-out baselines to disentangle action consequences from environmental "noise". The authors show that RLB is unbiased with exact counterfactuals and can reduce variance under certain conditions. They test counterfactual models based on paired data or diffusion models.

Generally my view of the paper is positive - i like the idea - while the evidence for the generality and practical reliability of this approach can be broadened.

Critical strengths
  • The paper motivates a clear approach to reducing the influence of environmental randomness on policy learning.
  • The theoretical analysis identifies conditions for unbiasedness and variance reduction, complementing the empirical comparisons with "classic" world models and policy-gradient baselines.
  • The diffusion-based experiment demonstrates a route to constructing useful counterfactual predictions without paired training observations sharing environmental noise.
Critical weaknesses

The evaluation covers settings where environmental variation is easy to preserve under alternative actions. It remains unclear how the gains extend to more strongly interacting sequential environments.

Questions for authors

Can the authors evaluate RLB in an environment where alternative actions change subsequent environmental evolution or other agents' responses?

2 comments

We thank the reviewer for the encouraging assessment and the constructive comment. As the weakness and the question are closely related, we address them together.

"W1. The evaluation covers settings where environmental variation is easy to preserve under alternative actions. It remains unclear how the gains extend to more strongly interacting sequential environments."

"Q1. Can the authors evaluate RLB in an environment where alternative actions change subsequent environmental evolution or other agents' responses?"

We agree that the original evaluation was limited to such settings. We have extended both the theory and the experiments.

In theory. Our updated Proposition 8 quantifies how the gains of RLB extend to such environments, for any sequential environment and return. The proposition compares the "luck" σluck\sigma_{\rm luck}, the variation of the return that the environmental noise causes regardless of the actions, with the "interaction" σinter\sigma_{\rm inter}, which measures how much the effect of actions changes with the noise. The result is that RLB has a smaller covariance than RLOO whenever σluck≥(1+m+4)σinter\sigma_{\rm luck}\ge(1+\sqrt{m+4})\sigma_{\rm inter} for groups of mm trajectories. In strongly interacting environments, the interaction is generally larger, and the benefit of RLB depends on how much environmental luck is there relative to it.

In experiment. We added the following environments.

  • Actions with delayed effects. In Appendix D.5, we added two sequential environments, Linear Key-to-Door and Task Interleaving, used in prior work [1] to evaluate credit assignment methods. In these environments, an action changes what happens many steps later. In key-to-door, picking up the key determines whether the agent can later open the door and collect the treasure. In task interleaving, each choice determines a reward that arrives only when its context is revisited, with other contexts in between. The branches of RLB reuse the factual noise at every step, which keeps RLB unbiased (Proposition 7). In key-to-door, the observed factual also does not reveal all of the noise, as the value of a distractor reward is hidden unless the agent picks it up. With the simulator, RLB reaches a success of 95.8% in key-to-door and 99.70% in task interleaving, against 14.8% and 82.0% for RLOO. RLB+CWM reaches 98.2% and 99.99%, on par with PPO in the former (98.1%) and ahead of it in the latter (94.6%). Please see Figures 10 and 11 for a detailed comparison with additional baselines.

  • Noise that interacts with actions. In Appendix D.6, we added the RiverSwim environment, where the next state depends on the action and the noise jointly at every step, so the same noise plays out differently under an alternative action. With exact counterfactuals, RLB remains unbiased, and its variance reduction over RLOO ranges from none at the uniform policy to about twofold near the optimal policy.

  • Interaction that dominates. In Appendix D.7, in a bandit where the noise decides which action is better, we demonstrate that RLB has 3.45 times the variance of RLOO without environmental luck, but has the lower variance once environmental luck exceeds the threshold predicted by our theory (Appendix A.12).

Overall, in line with Proposition 8, we find that the gains of RLB transfer to sequential tasks with delayed effects and some degree of interactions, while they can shrink or reverse as the interaction outgrows environmental luck. We believe our new results establish a better understanding of the cases where RLB is useful, as well as demonstrating its utility in more complex environments than simple bandits.


[1] Meulemans et al., Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis, NeurIPS 2023.

Dear reviewer, thank you again for your time and for the detailed review.

As the discussion phase is ending in less than two days, we would greatly appreciate it if we could know whether our rebuttal has properly addressed your concerns. If any remains, we would be happy to provide further clarification.

Reviewer_atYPOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes counterfactual world models (CWMs). Given a factual transition, a CWM predicts the next state under an alternative action while holding that transition's noise fixed. The proposed learning algorithm RLB is RLOO in which the sampled branches are replaced by CWM counterfactuals, giving a common-random-numbers baseline.

Critical strengths
  • The SCM framework is used for discussing counterfactuals properly
  • The writing is clear and easy to follow.
Critical weaknesses
  • The novelty and the theory is thin given the strong requirement of common-random-number and MDP-SCM environment. Prop. 3.1 counts distinct models in finite classes, which says nothing about sample complexity. Prop. 3.2 is MLE consistency under realizability. Prop. 3.4 is a one-step additive common-random-numbers calculation. There is no bias or variance analysis for learned CWMs, which are the only setting RLB+CWM actually runs in.
  • Noise abducted at one step to another presumes that time-index exogenous noise is shared as a confounder. This is actually violating the MDP definition. Needs to be handled properly in the modeling part.
  • The paper assumes additive noise independent of state and action which makes the identification problem trivial and contribution overlapped with prior work. The state then splits into an endogenous part and an exogenous part, and the counterfactual is identified trivially by replaying the observed exogenous trajectory. But such conditioning baselines on that trajectory is established in prior work, e.g., input-dependent baselines (Mao et al., ICLR 2019); exogenous-reward removal (Dietterich et al., ICML 2018) and Exo-MDPs (Efroni et al., COLT 2022); hindsight learning (Sinclair et al., 2023); difference rewards and COMA (Wolpert & Tumer, 2002; Foerster et al., 2018) for the multi-agent bandit.
  • The overall experiment evaluation is too weak to draw any useful conclusion. No Atari games or popular Mujoco envs are tested. 2/3 are bandits while the definition is under MDP.
Questions for authors
  1. In Dino, can action be predicted from the inverted latent (i.e., does action information leak into it)? How large is the DDIM inversion error?
  2. How does RLB behave on a task with non-additive, state-dependent noise, where counterfactuals are not identified? Can you bound RLB's bias as a function of CWM error?
  3. How do you reconcile the "no simulator access" claim with paired training? This is simply hiding the fact that you move the common random seed replay to a world model.
10 comments

We thank the reviewer for the thoughtful feedback, which has greatly helped us improve the paper. We address each point below.

"W1-1. The novelty and the theory is thin given the strong requirement of common-random-number and MDP-SCM environment."

We first address the requirements of MDP-SCM and common random numbers, and then address the novelty and the theory.

  • MDP-SCM. Every MDP can be written as an SCM [1], so this is not a requirement on the environment. What usually makes a strong requirement is the non-identifiability of the SCM, where several different SCMs are consistent with one MDP [2]. This does not affect RLB, which is unbiased with the exact counterfactuals of any consistent SCM, so none has to be identified (Line 148, Corollary 1).
  • Common random numbers. Shared noise is needed only to collect the paired data for training a CWM, once and offline, and Proposition 2 bounds how many pairs are needed. The diffusion route needs none, as it uses ordinary interaction data (Propositions 5 and 6). Please also see Q3.

"W1-2. Proposition 3.1 counts distinct models in finite classes, which says nothing about sample complexity."

We agree. We have replaced it with a finite-sample analysis (Propositions 2 to 4):

  • Upper bound for CWMs. In Proposition 2, we show that a maximum likelihood CWM over a finite model class Ω\Omega containing the target, trained on NN paired transitions, has an expected error of at most (1+log⁡∣Ω∣)/N(1+\log|\Omega|)/N in squared Hellinger distance, whatever the distribution of the environmental noise.
  • Lower bound for WMs. In Lemma 2, we show that any WM estimated from 2N2N transitions has a worst-case error of at least that of estimating the distribution of the (visible) noise from samples. A CWM avoids this estimation, as it conditions on the factual transition.
  • When CWMs are easier. In Propositions 3 and 4, we show that the upper bound is below the lower bound when the noise takes many values (complex noise), or when its effect on the next state outweighs that of actions.

"W1-3. Proposition 3.2 is MLE consistency under realizability."

We agree that Proposition 1 (previously 3.2) is a standard consistency result. We keep it as it shows that the paired loss recovers the counterfactual distribution for any data-collecting policy which we find useful, and we added Proposition 2 which gives the corresponding sample complexity.


[1] Buesing et al., Woulda, coulda, shoulda: Counterfactually-guided policy search, 2018.

[2] Oberst & Sontag, Counterfactual off-policy evaluation with Gumbel-max structural causal models, 2019.

"W1-4. Prop. 3.4 is a one-step additive common-random-numbers calculation."

We agree that the previous result covered only a special case. We have replaced it with Proposition 8, which holds for any return and horizon, and for noise that is non-additive and partly hidden.

In the proposition, we write the return of a trajectory as G(ϵ,u)G(\epsilon,u) for action noise ϵ\epsilon and environmental noise uu. Let V(u):=Eϵ[G(ϵ,u)]V(u):=\mathbb E_\epsilon[G(\epsilon,u)] be the return that depends on the noise but not on the actions taken, with variance denoted σluck2\sigma_{\rm luck}^2, and let σinter2:=sup⁡ϵVar(G(ϵ,u)−V(u)∣ϵ)\sigma_{\rm inter}^2:=\sup_\epsilon\mathrm{Var}(G(\epsilon,u)-V(u)\mid\epsilon) measure how much the effect of actions changes with the noise. Then, for every group of mm trajectories, we show that

Cov(g^RLOO)−Cov(g^RLB)⪰(σluck−σinter)2−(m+4)σinter2m−1E[zizi⊤],\mathrm{Cov}(\hat g_{\rm RLOO})-\mathrm{Cov}(\hat g_{\rm RLB})\succeq\frac{(\sigma_{\rm luck}-\sigma_{\rm inter})^2-(m+4)\sigma_{\rm inter}^2}{m-1}\mathbb E[z_i z_i^\top],

with equality if σinter=0\sigma_{\rm inter}=0, which is the classical effect of common random numbers pointed out by the reviewer [3].

RLB thus has the smaller covariance whenever σluck≥(1+m+4)σinter\sigma_{\rm luck}\ge(1+\sqrt{m+4})\sigma_{\rm inter}, that is, when environmental luck outweighs its interaction with actions. We prove that this condition is tight up to a constant (Appendix A.12), and confirm empirically that RLB has the larger variance when luck is small, with the crossover where the theory predicts (Appendix D.7).

"W1-5. There is no bias or variance analysis for learned CWMs, which are the only setting RLB+CWM actually runs in."

We thank the reviewer for this comment. We have added both a bias and variance analysis in Propositions 9 and 10, and their measurements in Appendix D.1.

  • Bias. We show that with branches generated by a learned CWM, the bias of RLB is at most zmax⁡ηz_{\max}\eta in norm, where zmax⁡z_{\max} bounds the score and η\eta is the mean absolute error of the model's expected return of a branch. As such, the bias of the method is bounded by modeling error. For a maximum likelihood CWM trained on NN pairs, it decays as 1/N1/\sqrt N.
  • Variance. We prove that, if the returns of a branch and its factual differ by at most AA, RLB with a learned CWM has a smaller covariance than RLOO with any WM and than REINFORCE with any state-dependent baseline whenever σluck≥σinter+mA\sigma_{\rm luck}\ge\sigma_{\rm inter}+mA, that is, when environmental luck outweighs its interaction with actions.
  • Measurements. In the contextual bandit, where the exact gradient is known, the bias of RLB+CWM is at most 0.3% of the norm of the gradient at every noise level, and its variance matches that of RLB with true counterfactuals (Table 3). In Dino, the squared bias of RLB+CWM is below 2% of the squared error of a single batch (Table 4).

Appendix A.15 further extends the bias-variance analysis to sparse branching, where the branches are generated only from the first state, as in Dino and sequential environments.


[3] Glasserman & Yao, Some guidelines and guarantees for common random numbers, Management Science 1992.

"W2. Noise abducted at one step to another presumes that time-index exogenous noise is shared as a confounder. This is actually violating the MDP definition. Needs to be handled properly in the modeling part."

We believe that this is a misunderstanding caused by our overly brief description.

  • The noise does not alter the MDP. In Eq. (6), the environmental noise is indexed by step, and is independent across steps and independent of the action noise. Every MDP admits this representation, and thus, it does not violate the MDP definition [1]. As the noise of a step is independent of the action at that step given the state, it is not a confounder.
  • The noise is reused at the same step only. In our approach, a counterfactual branch reuses the noise of the factual transition at the same step, and never across steps (Eqs. (12) and (13)).
  • Each branch follows the MDP. By itself, a branch is distributed exactly as a trajectory of the MDP from its branching state, as we prove in Lemma 6. Only the joint distribution of a branch and its factual is coupled, as for two simulator runs with a shared random seed.

We have revised Line 144, Appendix A.2 to make the above points clear.

"W3-1. The paper assumes additive noise independent of state and action which makes the identification problem trivial and contribution overlapped with prior work. The state then splits into an endogenous part and an exogenous part, and the counterfactual is identified trivially by replaying the observed exogenous trajectory."

We would like to clarify that our method does not assume additive noise. Only some of our previous theoretical results did, and none of the updated ones does.

  • Non-additive noise. Propositions 2 to 4 hold for any transition function, and Proposition 8 for any return. The noise may interact with actions and be only partly revealed by the factual transition (Lines 149-151, Appendix A.2), so the state need not split into an endogenous and an exogenous part.
  • Identification is not needed. RLB is unbiased with the counterfactuals of any SCM consistent with the MDP (Corollary 1), and with a learned CWM, its bias is bounded by the error against any such SCM (Proposition 9). Please also see Q2.
  • No replay in the diffusion route. The exogenous trajectory is neither given nor replayed, as the noise is inferred from observations by diffusion inversion (Propositions 5 and 6).

Among our experiments, only the two bandits have additive noise. We kindly remark to the reviewer that the Dino environment has non-additive noise. To further expand our evaluations, in Appendix D.5 we added Linear Key-to-Door and Task Interleaving environments, which are also sequential with non-additive noise. Especially, in key-to-door, there is a distractor reward whose value is hidden unless the agent picks it up, so the counterfactual is not determined by the factual observation. Furthermore, in Appendix D.6, we added RiverSwim, an environment where the next state depends on the action and the noise jointly, so the factual transition does not determine the counterfactual here either (in fact, the SCMs themselves are not identified in this setting). Across these settings, we find that RLB works robustly, demonstrating its generality (we refer the reviewer to Figures 10-12).

Reviewer_jv54OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The authors formalize the notion of counterfactual world models in the context of RL and propose a corresponding policy-gradient framework that utilizes these world models. The authors provide theoretical and empirical results in support of their work, and show that, under some conditions, counterfactual world models are easier to learn and can yield better policies compared to ordinary world models.

Critical strengths

The overall approach is logical, sound, and well-executed. The theoretical and empirical results, taken together, support the claims made by the authors. The non-technical writing is high quality, and the figures are clear and easy to understand.

Critical weaknesses

Overall, I have no major concerns. However, there are some aspects of the paper that would need to be improved prior to publication:

  • The authors repeatedly use the notions of “luck” (e.g. lines 47-48) and “predictability” (e.g. lines 75-76) without rigorously defining them. I encourage the authors to use more precise terminology.
  • The use of structural causal models (SCMs) requires more motivation. In particular, the paper would benefit from a discussion of whether the MDP necessarily needs to be modelled via SCMs, whether alternate formulations could be possible (and under what assumptions), and what are the limitations of using SCMs.
  • While I have no concern with the theoretical results themselves, there is some discussion missing that communicates the significance of these results with respect to the broader research question. For example, in Proposition 3.1, the authors show that counterfactual world models have lower complexity than ordinary world models, but they do not show why having a lower complexity makes it easier to model. Similar connections need to be made for the other results provided.
  • More discussion is needed related to the assumptions. In particular, the authors state the assumptions required for their theoretical results, but it is not clear what the implications of these assumptions are (e.g. what they are, how restrictive they are, etc.).
  • The notation could use some sharpening. For example, pθp_{\theta} in line 91 is never formally defined.
Questions for authors
  • In line 175, the authors state that it is common to fix x_i = \hat(x}_i, but allow otherwise for convenience. What are the implications of this? Moreover, if x_i = \hat(x}_i is not fixed, would the resulting model still be considered a counterfactual world model?
  • What is the authors’ recommendation as to what mm (i.e., the number of trajectories to sample in the baseline) should be set to?
5 comments

We thank the reviewer for the positive assessment and the constructive suggestions, which have greatly helped us improve the presentation. We address each point below.

"W1. The authors repeatedly use the notions of 'luck' (e.g. lines 47-48) and 'predictability' (e.g. lines 75-76) without rigorously defining them."

Thank you for the comment. We have made both notions precise.

  • Luck (Section 3.2). Let us write the return of a trajectory as G(ϵ,u)G(\epsilon,u) for action noise ϵ\epsilon and environmental noise uu. Let V(u)=Eϵ[G(ϵ,u)]V(u)=\mathbb E_\epsilon[G(\epsilon,u)] be the return that depends on the noise but not on the actions taken. We define its variance σluck2\sigma_{\rm luck}^2 as a measure of environmental luck, as it measures variations in return that no action is responsible for. Our new variance results are stated in terms of this quantity (Propositions 8 and 9).
  • Predictability (Section 3.1). In our updated paper, "Action effects are more predictable than the environment" now refers to two rigorous conditions: the environmental noise takes many values (Proposition 3), or the effect of the environmental noise on the next state outweighs that of actions (Proposition 4). Under either of these conditions, we show that a CWM is more accurate than any WM learned from the same amount of data, in the worst case.

"W2. The use of SCMs requires more motivation. The paper would benefit from a discussion of whether the MDP necessarily needs to be modelled via SCMs, whether alternate formulations could be possible (and under what assumptions), and what are the limitations of SCMs."

We have added this discussion to Section 3.1, Appendix A.2, and Appendix B.

Is an SCM necessary? The transition probabilities of an MDP alone do not define a counterfactual, as they do not define what "the same environmental randomness" means under an alternative action. We use an SCM only to define this. Importantly, this does not require any additional assumptions, as every MDP can be written as an SCM [1] (Lines 141-142).

Alternative formulations. One alternative is another SCM, as several SCMs can be consistent with the same MDP [2], which is related to the issue of non-identifiability. This is not an issue for RLB, as it is unbiased with the exact counterfactuals of any of the SCMs consistent with the MDP, so none has to be identified, and the choice affects only the variance (Corollary 1, Proposition 8). Another is to assume that the noise is given, as exogenous inputs or exogenous state variables [3, 4]. This is a special case of our setting, and we compare with it in Appendix B (Lines 1593-1604 and 1631-1637). To summarize, prior work assumes the noise is given, or restricts the mechanisms such that counterfactuals are identified, but our approach requires neither, as it does not require identifiability.

Limitations of SCMs.

  • Transition data do not identify (uniquely determine) counterfactuals. Yet for RLB, this affects the variance but not the bias.
  • A transition may not reveal all of the noise, e.g., the outcome of a gamble that was not played. Yet, in this case, RLB remains unbiased (Proposition 7).
  • We exclude unobserved variables that persist over time, as they violate the independence of the noise across steps and may confound actions and outcomes. This is a current limitation of our framework, and we leave their handling for future work (Appendix A.2).

[1] Buesing et al., Woulda, coulda, shoulda: Counterfactually-guided policy search, ICLR 2019.

[2] Oberst & Sontag, Counterfactual off-policy evaluation with Gumbel-max structural causal models, ICML 2019.

[3] Mao et al., Variance reduction for reinforcement learning in input-driven environments, ICLR 2019.

[4] Efroni et al., Sample-efficient reinforcement learning in the presence of exogenous information, COLT 2022.

"W3-1. [...] there is some discussion missing [...] with respect to the broader research question. For example, in Proposition 3.1, the authors show that CWMs have lower complexity than ordinary WMs, but they do not show why having a lower complexity makes it easier to model."

We agree that the previous Proposition 3.1 did not show why a lower complexity makes a model easier to learn. We have replaced it with results on the error of a model (WM or CWM) learned from a given number of samples (Propositions 2 to 4). Therein, we show that a maximum likelihood CWM trained on NN pairs of data has an expected error of at most (1+log⁡∣Ω∣)/N(1+\log|\Omega|)/N for a finite model class Ω\Omega, whatever the distribution of the environmental noise. In contrast, any WM trained on 2N2N transitions has to learn this distribution. It is therefore less accurate in the worst case when the noise takes many values or outweighs the effect of actions.

"W3-2. Similar connections need to be made for the other results provided."

In our updated paper, we state the implication of each theoretical result for our research question, especially focusing on whether CWMs are easier to obtain and yield better policy gradients than WMs. We provide an overview below.

  • Propositions 5 and 6. A diffusion WM that is wrong about the environmental noise distribution can still give exact counterfactuals, as this error cancels between inversion and generation. A counterfactual model thus need not reproduce the full environment, and can be obtained without paired data.
  • Proposition 7 and Corollary 1. RLB is unbiased without identifying the SCM, whose choice affects only the variance.
  • Proposition 8. RLB reduces variance when environmental luck outweighs its interaction with actions. This result makes "distinguishing action effects from luck" precise.
  • Propositions 9 and 10. With learned models, environmental luck affects RLOO and REINFORCE even if their WM or baseline is exact (Lemma 10), whereas RLB is only affected by the effect of actions on the return and the model error.

We believe this directly connects our theoretical results to our main question.

"W4. More discussion is needed related to the assumptions. [...] it is not clear what the implications of these assumptions are (e.g. what they are, how restrictive they are, etc.)."

We have added a summary of the assumptions of all results and of how restrictive they are (Appendix A.1). In summary:

  • Propositions 1 and 2 assume that the model class contains the true counterfactual distribution, and Proposition 2 that it is finite, for simplicity. Yet, they do not restrict the form of the noise or the policy that collects the data.
  • Propositions 3 and 4 take the worst case over the distribution of the visible noise, so they concern environments whose noise distribution is not known in advance.
  • Proposition 5 assumes exact inversion and a WM that is exact up to its noise distribution. We relax both assumptions in Proposition 6.
  • Propositions 7 and 8 assume exact counterfactuals. Nevertheless, they hold for any return and horizon, also when a part of the noise is hidden.
  • Propositions 9 and 10 drop exactness of the counterfactuals. They assume bounded scores and a bound on the difference between the returns of factual and counterfactual branches. The sample complexity result in Proposition 10 further assumes that the training data pairs cover the branches, which is a comparably restrictive assumption (Appendix A.14).

“W5. The notation could use some sharpening. For example, pθp_\theta in line 91 is never formally defined.”

Thank you for the constructive feedback. We now define it below Eq. (1) as follows. τ∼pθ\tau\sim p_\theta denotes a trajectory sampled with the policy πθ\pi_\theta following Eq. (1), and τ≥i∼pθ(⋅∣xi)\tau_{\ge i}\sim p_\theta(\cdot\mid x_i) its suffix given the state xix_i. We have also improved the notation of Section 3 overall. For example, the noise variables of the SCM and their independence are now stated more explicitly below Eq. (6).

“Q1. In line 175, the authors state that it is common to fix xi=x^ix_i = \hat x_i, but allow otherwise for convenience. What are the implications of this? Moreover, if xi=x^ix_i = \hat x_i is not fixed, would the resulting model still be considered a CWM?”

Yes, the model is still a CWM. The counterfactual in Eq. (7) is defined for any state xix_i. The counterfactual defines what the next state would be from xix_i under action aia_i, with the environmental noise that produced the factual transition (x^i,a^i,x^i+1)(\hat x_i,\hat a_i,\hat x_{i+1}). Allowing xi≠x^ix_i\neq\hat x_i enables us to reuse environmental noise underlying a transition to make predictions at a different state (e.g., infer the weather at one building and use it to make an action-conditioned prediction at another building). Fixing xi=x^ix_i=\hat x_i is the special case in which only the action changes.

Allowing xi≠x^ix_i\neq\hat x_i has the following implications.

  • It makes multi-step counterfactuals possible. After its first alternative action, a branch is at a state that differs from that of the factual, and we keep reusing the noise of the factual at the same step (Section 3.2).
  • Our results cover this case. Each branch still follows the MDP (Lemma 6), so RLB remains unbiased (Proposition 7).
  • The training pairs have to cover such pairs of states. This is the coverage assumption of Proposition 10 (Appendix A.14) we have discussed in W4.

We have revised the sentence after Eq. (7) to clarify this.

“Q2. What is the authors’ recommendation as to what mm (i.e., the number of trajectories to sample in the baseline) should be set to?”

We recommend a moderate mm, and use m=4m=4 in our experiments.

  • In theory. We show in Appendix A.12 that the covariance of a group of RLB decreases with mm only down to a floor, which is determined by how much the true policy gradient varies w.r.t. the shared environmental noise. Only more factual trajectories can reduce this floor. Accordingly, the gain over RLOO in Proposition 8 decays as 1/(m−1)1/(m-1), and its condition for variance reduction becomes stricter as mm grows. Hence mm is worth increasing until the terms that decay with mm are comparable to this floor. Beyond that, the budget is better spent on more factual trajectories (e.g., more learning steps).

  • In experiment. In Appendix D.2, we added a new empirical analysis. Using the contextual bandit at a fixed budget of 256 trajectories per batch, we show that RLB has its lowest variance at m=4m=4 to 88, and its variance rises for larger mm (Table 5). It also reaches its highest final reward at m=4m=4 at both noise levels (Table 6), consistent with the theory. We note that RLOO keeps improving with mm, but stays far behind RLB even at its best mm. For example, at σ=30\sigma=30, the best final reward of RLOO is −37.30, while the worst of RLB is −18.87.

Table 5 (repeated from the paper). Gradient variance in the contextual bandit at the uniform policy, averaged over five seeds. Bold marks the lowest variance of RLB within sampling error.

σ\sigmaMethodm=2m=2m=4m=4m=8m=8m=16m=16m=32m=32m=64m=64
30RLB2,1551,5441,5391,8662,6494,242
30RLOO28,10218,76116,11315,05114,53414,394
60RLB8,3185,9645,9467,21410,24716,426
60RLOO400,882267,586229,721214,363206,629203,740

Table 6 (repeated from the paper). Expected reward after 10M interactions in the contextual bandit, as mean and 95% confidence interval over six seeds. The optimum is zero, and bold marks the highest mean of RLB.

σ\sigmaMethodm=2m=2m=4m=4m=8m=8m=16m=16
30RLB−16.26 ± 0.24−15.01 ± 0.30−16.58 ± 0.31−18.87 ± 0.11
30RLOO−41.83 ± 0.19−38.94 ± 0.14−37.83 ± 0.09−37.30 ± 0.07
60RLB−28.73 ± 0.25−25.49 ± 0.18−25.83 ± 0.19−28.00 ± 0.24
60RLOO−59.16 ± 0.48−56.92 ± 0.44−56.06 ± 0.11−55.58 ± 0.19
Reviewer_2zZVOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces counterfactual world models (CWMs), which condition on an observed transition and predict the outcome under an alternative action with the same environmental noise, and REINFORCE Look Back (RLB), a policy-gradient estimator that replaces the independent branches of RLOO by counterfactual branches generated from the factual trajectory and uses their returns as leave-one-out baselines. CWMs are learned either from paired trajectories generated under shared noise or by applying DDIM noise inversion to an action-conditioned diffusion world model. The authors argue that CWMs have lower complexity than ordinary world models (Proposition 3.1), that maximum likelihood on paired data recovers the counterfactual (Proposition 3.2), that RLB is unbiased under exact counterfactuals (Proposition 3.3), and that it reduces variance when environmental noise dominates (Proposition 3.4). Experiments on a contextual bandit, a multi-agent bandit, and Chrome Dino with a diffusion world model report faster policy learning and lower model error for CWMs.

Critical strengths

The idea is simple and well explained: coupling branch rollouts to the factual trajectory makes baselines differ from the sample only through the action, and Figures 1–3 make this argument readable. Two routes to a CWM, including the diffusion-inversion route that needs no paired data, and an explicit counterfactual-quality evaluation of the generated frames (Table 1).

Critical weaknesses

The theory is thin. Proposition 3.1 counts distinct models in finite classes and only states that the counterfactual map does not depend on the noise distribution; it is not a statement about sample complexity or learnability, and it does not cover non-additive noise. Proposition 3.4 is the standard common-random-numbers variance reduction, σ²/(m−1) E[zz^T], in a two-step additive-noise, linear-reward example; condition (ii) requires σ² ≥ mL²/(m−1), i.e., it only says RLB beats REINFORCE when noise dominates, which is the assumption. Proposition 3.3 needs exact counterfactuals and fresh policy randomness; with a learned CWM the estimator is biased, and no bias analysis or empirical bias measurement is given.

The bandit experiments are built so that the CWM has almost nothing to learn. In the contextual bandit the factual observation contains a + u, so the counterfactual a' + u is x̂ − a + a'; in the multi-agent bandit the factual cost contains the other agents' contributions additively. The model-learning panels of Figures 4–5 therefore mostly show that a shift map is easier to fit than a Gaussian with σ = 60. There is no experiment in the opposite regime (complex action effects, small environmental noise), where the paper's own claim predicts no benefit, and none with non-additive noise, where the counterfactual is not determined by the factual observation.

The Dino results do not support the claims statistically. With three seeds, RLB (118 ± 9) and RLOO (113 ± 13) with true simulations overlap, and the claim that model imaginations beat true simulations (129 ± 1 vs 118 ± 9) rests on a difference of about one standard deviation with three seeds. Baselines are REINFORCE variants only; there is no PPO/actor-critic or standard model-based RL reference. The CWM-vs-WM comparison in Table 1 measures counterfactual properties (composition, minimality) that an ordinary WM is not meant to satisfy, so it does not show that the CWM is a better world model.

Diffusion counterfactuals are not SCM counterfactuals. DDIM inversion with 10 steps and classifier-free guidance yields an approximate, model-specific coupling; the SCM implied by the diffusion model is not controlled, and over autoregressive 20-step rollouts the inversion error compounds. The paper acknowledges SCM non-uniqueness but never tests how sensitive RLB is to the coupling, for instance in a discrete environment where several consistent SCMs (e.g., Gumbel-max) can be enumerated.

Cost is not reported. RLB needs m−1 counterfactual rollouts per branch state; with a 101M-parameter diffusion model and inversion at every step this is expensive, and the Dino comparison is in trajectories rather than compute. The CWMs in the bandits are trained on only 4,096 pairs, which is cheap only because those environments are so simple.

Questions for authors

Can the authors measure the bias of the RLB gradient with the learned CWM against the exact gradient in the bandit, and show how it grows with model error?

Please add an experiment in which the action effect is harder to model than the environment noise, and one with non-additive noise, to delimit when CWMs help.

How much compute did the Dino experiments use per method, and do the differences in Table 2 hold with more seeds? Could PPO or A2C with a learned value baseline be added as a reference?

How sensitive are the Dino results to inversion depth and guidance scale, and how often does the inverted counterfactual fail the composition check along a rollout?

6 comments

We thank the reviewer for the careful and detailed feedback, which has greatly helped us improve the paper. We address each point below.

"W1-1. The theory is thin. Proposition 3.1 counts distinct models in finite classes and only states that the counterfactual map does not depend on the noise distribution; it is not a statement about sample complexity or learnability, and it does not cover non-additive noise."

We agree. We have replaced it with a finite-sample analysis (Propositions 2 to 4), which does not assume additive noise:

  • Upper bound for CWMs. In Proposition 2, we show that a maximum likelihood CWM over a finite model class Ω\Omega containing the target, trained on NN paired transitions, has an expected error of at most (1+log⁡∣Ω∣)/N(1+\log|\Omega|)/N in squared Hellinger distance, whatever the distribution of the environmental noise.
  • Lower bound for WMs. In Lemma 2, we show that any WM estimated from 2N2N transitions has a worst-case error of at least that of estimating the distribution of the (visible) noise from samples. A CWM avoids this estimation, as it conditions on the factual transition.
  • When CWMs are easier. In Propositions 3 and 4, we show that the upper bound is below the lower bound when the noise takes many values (complex noise), or when its effect on the next state outweighs that of actions.

"W1-2. Proposition 3.4 is the standard common-random-numbers variance reduction in a two-step additive-noise, linear-reward example; condition (ii) requires σ² ≥ mL²/(m−1), i.e., it only says RLB beats REINFORCE when noise dominates, which is the assumption."

We agree that the previous result covered only a special case. We have replaced it with Proposition 8, which holds for any return and horizon, and for noise that is non-additive and partly hidden.

In the proposition, we write the return of a trajectory as G(ϵ,u)G(\epsilon,u) for action noise ϵ\epsilon and environmental noise uu. Let V(u):=Eϵ[G(ϵ,u)]V(u):=\mathbb E_\epsilon[G(\epsilon,u)] be the return that depends on the noise but not on the actions taken, with variance denoted σluck2\sigma_{\rm luck}^2, and let σinter2:=sup⁡ϵVar(G(ϵ,u)−V(u)∣ϵ)\sigma_{\rm inter}^2:=\sup_\epsilon\mathrm{Var}(G(\epsilon,u)-V(u)\mid\epsilon) measure how much the effect of actions changes with the noise. Then, for every group of mm trajectories, we show that

Cov(g^RLOO)−Cov(g^RLB)⪰(σluck−σinter)2−(m+4)σinter2m−1E[zizi⊤],\mathrm{Cov}(\hat g_{\rm RLOO})-\mathrm{Cov}(\hat g_{\rm RLB})\succeq\frac{(\sigma_{\rm luck}-\sigma_{\rm inter})^2-(m+4)\sigma_{\rm inter}^2}{m-1}\mathbb E[z_i z_i^\top],

with equality if σinter=0\sigma_{\rm inter}=0, which is the classical effect of common random numbers pointed out by the reviewer [1].

RLB thus has the smaller covariance whenever σluck≥(1+m+4)σinter\sigma_{\rm luck}\ge(1+\sqrt{m+4})\sigma_{\rm inter}, that is, when environmental luck outweighs its interaction with actions. Regarding the concern that this condition is the assumption, we prove that it cannot be removed, and therefore the result establishes clearly when RLB helps:

  • In theory. In Appendix A.12, we show that without environmental luck, RLB never has a smaller covariance than RLOO, and the condition is tight up to a constant.
  • In experiment. In Appendix D.7, we show that in a bandit whose noise mostly interacts with the action, RLB has 3.45 times the variance of RLOO without environmental luck, and the two become equal at σluck=1.58\sigma_{\rm luck}=1.58, where our theory predicts the crossover.
  • In our contextual bandit. In Appendix C.1, we show that in our contextual bandit, the variance of luck grows as σ4\sigma^4 in the noise level σ\sigma, while that of its interaction with actions grows as σ2\sigma^2. Therefore, the condition for variance reduction holds for large noise, which is consistent with our results in Figure 4.

"W1-3. Proposition 3.3 needs exact counterfactuals and fresh policy randomness; with a learned CWM the estimator is biased, and no bias analysis or empirical bias measurement is given."

We have added both.

  • Analysis. In Propositions 9 and 10, we show the following bias analysis: when branches are generated by a learned CWM, the bias of RLB is at most zmax⁡ηz_{\max}\eta in norm, where zmax⁡z_{\max} bounds the score and η\eta is the mean absolute error of the model's expected return of a branch. For a maximum likelihood CWM trained on NN pairs, it decays as 1/N1/\sqrt N.
  • Measurements. We add an empirical bias measurement in Appendix D.1. For details, please see our response to Q1.

We would like to remark that fresh policy randomness is not a requirement on the environment. It only means that each branch samples its own actions from the policy, as in RLOO.


[1] Glasserman & Yao, Some guidelines and guarantees for common random numbers, Management Science 1992.

"W2-1. The bandit experiments are built so that the CWM has almost nothing to learn. In the contextual bandit the factual contains a + u, so the counterfactual a' + u is x̂ − a + a'; in the multi-agent bandit the factual cost contains the other agents' contributions additively. Figures 4-5 mostly show that a shift map is easier to fit than a Gaussian with σ = 60."

We agree that the counterfactual map is simple in both bandits. Yet, the setting follows a prior work [2] and is suitable to demonstrate our theory in a controlled manner, where our approach is expected to benefit when the noise outweighs the effect of actions (Proposition 4), with exact counterfactuals and a known optimal policy to compare against. Beyond model learning, this environment shows the consequence for policy learning. To show this, we added a bias measurement in Table 3 in Appendix D.1, where we find that RLB+CWM is nearly unbiased, whereas RLOO+WM has a bias of up to 179% of the norm of the gradient as noise increases.

Table 3 (repeated from the paper). Variance of the gradient estimate (Var.) and its bias relative to the norm of the gradient (Rel. bias) in the contextual bandit, at the uniform policy with 256 transitions per batch. A dash marks an unbiased estimator.

MethodVar. (σ=0.1\sigma=0.1)Var. (σ=30\sigma=30)Var. (σ=60\sigma=60)Rel. bias (σ=0.1\sigma=0.1)Rel. bias (σ=30\sigma=30)Rel. bias (σ=60\sigma=60)
REINFORCE95.821,122301,493–––
REINFORCE + baseline95.420,995301,304–––
CCA36.88,465121,2380.482.208.70
RLOO70.618,794268,276–––
RLOO + WM70.518,539265,2810.0040.521.79
RLB70.61,5515,989–––
RLB + CWM70.61,5515,9870.0020.0010.003

Lastly, we would like to remark that the counterfactual is not a shift map in Dino, where it is obtained from a diffusion model without paired data.

"W2-2. There is no experiment in the opposite regime (complex action effects, small environmental noise), where the paper's own claim predicts no benefit, and none with non-additive noise, where the counterfactual is not determined by the factual observation."

We thank the reviewer for this suggestion. We added experiments that delimit when our approach helps.

  • Small noise. We would like to kindly remind the reviewer that we already have the opposite regime (small environment noise) in the contextual bandit (Figure 4; also see Figure 5). At σ=0.1\sigma=0.1, we have shown that the WM is as easy to learn as the CWM, and REINFORCE learns faster than RLB and RLOO at matched interactions. In our added Table 3, we show that RLB and RLOO also have the same gradient variance in this regime.
  • Noise that interacts with actions. In Appendix D.7, we added an interaction bandit where the noise decides which action is better. Without environmental luck, RLB has 3.45 times the variance of RLOO and learns more slowly, as predicted by our theory (Appendix A.12).
  • Non-additive noise. We kindly remark to the reviewer that the Dino environment has non-additive noise. To further expand our evaluations, in Appendix D.5 we added Linear Key-to-Door and Task Interleaving environments, which are also sequential with non-additive noise. Especially, in key-to-door, there is a distractor reward whose value is hidden unless the agent picks it up, so the counterfactual is not determined by the factual observation. Furthermore, in Appendix D.6, we added RiverSwim, an environment where the next state depends on the action and the noise jointly, so the factual transition does not determine the counterfactual here either (in fact, the SCMs themselves are not identified in this setting). Across these settings, we find that RLB with CWM works robustly, demonstrating its generality (we refer the reviewer to Figures 10-12).

[2] Mesnard et al., Counterfactual credit assignment in model-free reinforcement learning, ICML 2021.

"W3-1. The Dino results do not support the claims statistically. With three seeds, RLB (118 ± 9) and RLOO (113 ± 13) with true simulations overlap, and the claim that model imaginations beat true simulations (129 ± 1 vs 118 ± 9) rests on a difference of about one standard deviation with three seeds."

Thank you for raising this point. We have increased our evaluations in Table 2 to fifteen training runs, and we have removed the claim that model imaginations beat true simulations, which did not hold with more runs. The results are shown below.

Table 2 (repeated from the paper). Policy learning in Dino, as mean and sample standard deviation over fifteen training runs. Collision is the share of encountered obstacles that cause a collision, and survival is the number of decisions before the first collision. Bold marks the better of RLOO and RLB, with the simulator and with learned models.

MethodCollision (%) ↓Survival ↑
REINFORCE7.57 ± 3.4772.4 ± 18.6
REINFORCE + baseline9.53 ± 14.6589.7 ± 31.7
RLOO7.31 ± 14.21103.6 ± 29.0
RLB (Ours)3.29 ± 0.36113.9 ± 7.9
RLOO + WM7.89 ± 14.1097.5 ± 28.2
RLB + CWM (Ours)3.77 ± 1.15109.0 ± 18.9

We find that the average performances favor RLB in both cases, and also that RLB has much smaller variation across runs, implying stability. To supplement the evaluation, we also measure the variance of the gradient directly in Table 4 in Appendix D.1. The results are below.

Table 4 (repeated from the paper). Variance and squared bias of the gradient estimate in Dino, at policies taken after 500 to 1,500 updates. The bias of RLB + CWM and RLOO + WM is estimated against simulator-based RLB and RLOO. Brackets are 95% confidence intervals, and bold marks the lower value of each pair.

UpdatesVar. (RLB)Var. (RLOO)Var. (RLB + CWM)Var. (RLOO + WM)Bias² (RLB + CWM)Bias² (RLOO + WM)
50043.859.661.260.31.05 [+0.34, +2.12]0.00 [−0.18, +0.35]
1,00023.859.154.289.20.02 [−0.10, +0.24]0.25 [−0.21, +1.19]
1,50027.063.350.579.70.05 [−0.08, +0.31]0.37 [−0.04, +1.00]

We find that RLB and RLB+CWM has lower variance than RLOO and RLOO+WM, respectively, after sufficient training.

"W3-2. Baselines are REINFORCE variants only; there is no PPO/actor-critic or standard model-based RL reference."

To address this concern, we added advantage actor-critic (A2C) and PPO to the Dino experiment (Appendix D.4), as well as our two added complex sequential environments Linear Key-to-Door and Task Interleaving (Appendix D.5). The results are below (for full results, please see the referred sections).

Table A. A2C and PPO against RLB + CWM. Mean ± standard deviation over fifteen runs in Dino, and over five runs in key-to-door and task interleaving.

MethodDino: collision (%) ↓Key-to-door: success (%) ↑Task interleaving: success (%) ↑
A2C7.07 ± 4.9249.91 ± 15.7988.1 ± 1.8
PPO3.27 ± 0.8698.10 ± 1.9194.6 ± 2.7
RLB (Ours)3.29 ± 0.3695.83 ± 1.2099.70 ± 0.36
RLB + CWM (Ours)3.77 ± 1.1598.16 ± 0.7099.99 ± 0.01

We note that PPO is a strong baseline. It competes with RLB on Dino and RLB+CWM on key-to-door, while RLB is ahead on task interleaving. Lastly, we remark that RLOO+WM is the model-based RL reference that generates independent rollouts with diffusion world model (Table 2).

"W3-3. The CWM-vs-WM comparison in Table 1 measures counterfactual properties (composition, minimality) that an ordinary WM is not meant to satisfy, so it does not show that the CWM is a better world model."

We agree for composition and minimality. They are counterfactual properties, and Table 1 uses them to test whether diffusion counterfactual inference turns a diffusion WM into a counterfactual model, which is the third question of Section 5. The WM, the same network without noise inversion, serves as a reference.

Effectiveness and realism, however, are valid criteria for any world model: a generated frame should reflect its action and look like a real frame. On these, with the same network, the CWM matches the WM in effectiveness and generates more realistic frames:

From Table 1. Effectiveness is the macro-F1 of an action predictor applied to the generated frame, and realism is the FID against recorded frames.

CriterionCWMWM
Effectiveness (macro-F1) ↑0.9990.993
Realism (FID) ↓15.9437.72

Therefore, Table 1 supports that CWM is simultaneously a better world model and a better counterfactual world model in these aspects. We have separately supported that CWMs can be easier to learn by Propositions 2 to 4 and Figures 4 and 5.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers

We thank all reviewers for their careful and constructive feedback, which has helped us significantly improve the paper. We summarize the main changes below, and respond to each reviewer in a separate comment.

Theory

  • Replaced the counting argument with a sample complexity analysis of counterfactual world models (CWMs) against ordinary world models (WMs), which holds for non-additive noise (Propositions 2 to 4). (Reviewers 2zZV, jv54, atYP)
  • Added theoretical guarantees for diffusion CWMs (Propositions 5 and 6). (Reviewers 2zZV, atYP)
  • Showed that RLB does not require the SCM to be identified (Corollary 1). (Reviewers 2zZV, jv54, atYP)
  • Replaced the variance reduction result for a special case with a general one for any return and horizon, and showed that its condition is tight up to a constant (Proposition 8, Appendix A.12). (All reviewers)
  • Added a bias and variance analysis for learned CWMs (Propositions 9 and 10), also under the sparse branching used in Dino (Appendix A.15). (Reviewers 2zZV, atYP)

Presentation

  • Defined luck and predictability precisely, and sharpened the notation (Sections 2 and 3). (Reviewer jv54)
  • Clarified the SCM formulation (Section 3.1, Appendix A.2), and added a summary of the assumptions (Appendix A.1). (Reviewers jv54, atYP)
  • Extended the comparison with prior work on exogenous inputs and states, hindsight learning, and multi-agent counterfactual baselines (Section 4, Appendix B). (Reviewer atYP)

Experiments

  • Measured the bias and variance of the gradient estimators in the contextual bandit and Dino (Appendix D.1, Tables 3 and 4). (Reviewers 2zZV, atYP)
  • Added an analysis of the group size mm, which matches our theory (Appendix D.2, Tables 5 and 6). (Reviewer jv54)
  • Added two baselines in the multi-agent bandit, difference rewards and COMA (Appendix D.3, Figure 8). (Reviewer atYP)
  • Increased the Dino evaluation to 15 training runs per method (Table 2). Added two baselines, A2C and PPO, and analyses of the wall-clock cost, the sensitivity to inversion hyperparameters, and the leak of action information (Appendix D.4, Tables 7 and 8). (Reviewers 2zZV, atYP)
  • Added two sequential credit assignment benchmarks, linear key-to-door and task interleaving, with five further baselines (Appendix D.5, Figures 10 and 11). (Reviewers 2zZV, atYP, rF55)
  • Added a sensitivity analysis in RiverSwim, where we enumerate SCMs that induce the same MDP. RLB remains unbiased under each, with a similar variance reduction (Appendix D.6, Figure 12). (Reviewers 2zZV, atYP, rF55)
  • Added a bandit in which the theory predicts that RLB loses to RLOO (Appendix A.12). The measured crossover in variance matches the prediction (Appendix D.7, Table 9). (Reviewers 2zZV, rF55)

Revisions are highlighted in blue in the updated manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.