- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this work the authors study counterfactual world models. These predict alternative outcomes under shared environmental randomness when condition on observed experiences. They introduce REINFORCE Look Back (RLB), which uses counterfactual returns as leave-one-out baselines to disentangle action consequences from environmental "noise". The authors show that RLB is unbiased with exact counterfactuals and can reduce variance under certain conditions. They test counterfactual models based on paired data or diffusion models.
Generally my view of the paper is positive - i like the idea - while the evidence for the generality and practical reliability of this approach can be broadened.
Critical strengths
- The paper motivates a clear approach to reducing the influence of environmental randomness on policy learning.
- The theoretical analysis identifies conditions for unbiasedness and variance reduction, complementing the empirical comparisons with "classic" world models and policy-gradient baselines.
- The diffusion-based experiment demonstrates a route to constructing useful counterfactual predictions without paired training observations sharing environmental noise.
Critical weaknesses
The evaluation covers settings where environmental variation is easy to preserve under alternative actions. It remains unclear how the gains extend to more strongly interacting sequential environments.
Questions for authors
Can the authors evaluate RLB in an environment where alternative actions change subsequent environmental evolution or other agents' responses?
2 comments
Dear reviewer, thank you again for your time and for the detailed review.
As the discussion phase is ending in less than two days, we would greatly appreciate it if we could know whether our rebuttal has properly addressed your concerns. If any remains, we would be happy to provide further clarification.
We thank the reviewer for the encouraging assessment and the constructive comment. As the weakness and the question are closely related, we address them together.
We agree that the original evaluation was limited to such settings. We have extended both the theory and the experiments.
In theory. Our updated Proposition 8 quantifies how the gains of RLB extend to such environments, for any sequential environment and return. The proposition compares the "luck" σluck, the variation of the return that the environmental noise causes regardless of the actions, with the "interaction" σinter, which measures how much the effect of actions changes with the noise. The result is that RLB has a smaller covariance than RLOO whenever σluck≥(1+m+4)σinter for groups of m trajectories. In strongly interacting environments, the interaction is generally larger, and the benefit of RLB depends on how much environmental luck is there relative to it.
In experiment. We added the following environments.
Actions with delayed effects. In Appendix D.5, we added two sequential environments, Linear Key-to-Door and Task Interleaving, used in prior work [1] to evaluate credit assignment methods. In these environments, an action changes what happens many steps later. In key-to-door, picking up the key determines whether the agent can later open the door and collect the treasure. In task interleaving, each choice determines a reward that arrives only when its context is revisited, with other contexts in between. The branches of RLB reuse the factual noise at every step, which keeps RLB unbiased (Proposition 7). In key-to-door, the observed factual also does not reveal all of the noise, as the value of a distractor reward is hidden unless the agent picks it up. With the simulator, RLB reaches a success of 95.8% in key-to-door and 99.70% in task interleaving, against 14.8% and 82.0% for RLOO. RLB+CWM reaches 98.2% and 99.99%, on par with PPO in the former (98.1%) and ahead of it in the latter (94.6%). Please see Figures 10 and 11 for a detailed comparison with additional baselines.
Noise that interacts with actions. In Appendix D.6, we added the RiverSwim environment, where the next state depends on the action and the noise jointly at every step, so the same noise plays out differently under an alternative action. With exact counterfactuals, RLB remains unbiased, and its variance reduction over RLOO ranges from none at the uniform policy to about twofold near the optimal policy.
Interaction that dominates. In Appendix D.7, in a bandit where the noise decides which action is better, we demonstrate that RLB has 3.45 times the variance of RLOO without environmental luck, but has the lower variance once environmental luck exceeds the threshold predicted by our theory (Appendix A.12).
Overall, in line with Proposition 8, we find that the gains of RLB transfer to sequential tasks with delayed effects and some degree of interactions, while they can shrink or reverse as the interaction outgrows environmental luck. We believe our new results establish a better understanding of the cases where RLB is useful, as well as demonstrating its utility in more complex environments than simple bandits.
[1] Meulemans et al., Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis, NeurIPS 2023.