- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates common random numbers (CRN) for reducing variance in policy-gradient reinforcement learning. Its main approach replaces the conventional state-value baseline in PPO with a noise-conditioned counterfactual baseline, computed by evaluating alternative actions from the same simulator state under the same transition-noise realization. The authors propose implementations based on simulator resets and Monte Carlo counterfactual action sampling, as well as a seed-conditioned value-function approach. Theoretical results establish unbiasedness and variance reduction under certain conditions, while experiments compare the proposed methods against standard PPO on several stochastic benchmark environments.
Critical strengths
The paper proposes a relatively simple modification to PPO that exploits reproducible simulator randomness without directly observing the noise variables. Experiments across several stochastic environments demonstrate potential improvements.
Critical weaknesses
1. Limited methodological and theoretical novelty.
The core idea of evaluating alternative actions under shared environment noise is well established, including in TRPO vine sampling (Schulman et al., 2015) [1]. Noise-dependent policy-gradient baselines have also been studied by Mao et al. (2019) [2] and Nota et al. (2021) [3]. The main theoretical results largely follow standard control-variate arguments: unbiasedness follows from the zero-mean policy-score identity, while advantage-variance reduction follows directly from the law of total variance. The paper does not convincingly establish a substantial methodological advance beyond adapting these existing ideas to PPO.
2. Insufficient validation of the practical method and its advantages.
The theoretical variance-reduction guarantee concerns an exact noise-conditioned baseline, whereas the implemented method uses a Monte Carlo approximation. The paper does not directly demonstrate that this practical estimator consistently reduces policy-gradient variance. Moreover, experiments primarily compare against standard PPO rather than existing variance-reduction methods [1–3]. The method also requires additional simulator evaluations and the ability to reset simulator and RNG states, limiting its applicability.
References
[1] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. "Trust Region Policy Optimization." Proceedings of the 32nd International Conference on Machine Learning (ICML), PMLR 37:1889–1897, 2015.
[2] H. Mao, S. B. Venkatakrishnan, M. Schwarzkopf, and M. Alizadeh. "Variance Reduction for Reinforcement Learning in Input-Driven Environments." International Conference on Learning Representations (ICLR), 2019.
[3] C. Nota, P. S. Thomas, and B. C. da Silva. "Posterior Value Functions: Hindsight Baselines for Policy Gradient Methods." Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR 139:8238–8247, 2021.
Questions for authors
-
What is the specific methodological contribution beyond TRPO vine sampling and previously established noise-dependent policy-gradient baselines?
-
Can the authors provide comparisons with existing variance-reduction methods and clarify how robust the approach is to general stochastic dynamics and different simulator noise couplings?
We thank the reviewer for their feedback. We would like to address the different concerns the reviewer brings up:
Limited methodological and theoretical novelty : We believe that the lack of novelty is overstated.
VINE runs several full (or truncated) rollouts from each state with common random numbers, and this is used for reducing the variance of the advantage only at that state. Our insight, based on the temporal difference formulation, allows two significant improvements: (1) it allows us to sample only single-step transitions per counterfactual action. More importantly, (2) the counterfactual actions for some state reduce the advantage variance of all of the states before it, through the GAE update rule. We added Figure 11 in the revised paper to show this schematically. Our algorithm is significantly more efficient than VINE. To demonstrate this, we added a comparison with VINE, see Appendix G.1, Figure 10.
Thank you for raising this question, we agree it wasn’t clear enough in the text. Another way to view our work is an extension of VINE to GAE. VINE, to the best of our knowledge, has seen very little practical use. As the experiments above show - our formulation leads to significant performance gains over VINE, which is what matters in practice.
The algorithm of Nota et al. applies to tabular POMDPs with an analytically computed belief state, and is not applicable to the experiments in our work. The algorithm of Mao et al. applies to input-driven MDPs, and trains a value conditioned on the input. It is also not applicable to the experiments in our paper. Additionally, we remark in the paper (Line 761) that both Nota et al. and Mao et al. are not compatible with the GAE.
Lastly, this paper is the first of its kind (as far as we are aware) to show a very simple algorithmic modification to plug in CRN into one of the most popular RL algorithms today - PPO with GAE. As we demonstrate, our new method leads to significant gains in several domains, which drives our main novelty claim.
Insufficient validation of the practical method and its advantages: We study the effect of different simulator noise in Figure 2, page 7.
Robustness to general stochastic dynamics: we are not sure what the reviewer means. Our evaluations across a diverse suite of tasks with diverse stochastic dynamics (Figures 2, 3, 4 domains Mujoco, Pufferlib, OR-gym) shows the robustness of our method. If the reviewer has a particular experiment in mind we are happy to add it.
Furthermore, Figure 5 in the main text compares the contribution of CFVE under different noise magnitudes across wall-clock time. The figure shows that as the noise magnitude increases, the variance reduction our method obtains is significant enough such that the performance gains outweigh the additional simulator steps (since results are reported under equal wallclock time).
In addition, per Reviewer qqf3’s suggestion, we conducted a control experiment where we remove Line 6 (the random number reset) from Algorithm 1, and verify that this hurts performance. As expected, the results show that our approach (Sampled CFVE) introduces improvement by reducing advantage variance.