acceptodds
Under review as a conference paper at ICLR 2027

Same Noise, Better Gradients: Policy Optimization with Common Random Numbers

Abstract

We study common random numbers as a variance-reduction method for policy- gradient reinforcement learning in stochastic simulators. The core idea is to decouple the random number generators for action selection and state transitions, and investigate counterfactual effects of actions under the same state transition noise. For policy gradient algorithms, we implement this idea as a control-variate that depends on the state transition noise, and prove that it is unbiased and obtains lower variance in certain conditions. Further, we propose two simple modifications of the popular PPO algorithm based on either (1) resetting the random number generator, and (2) learning a value function that depends on the random seed. We demonstrate improved performance on various stochastic domains.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Optimization, Common Random Numbers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z9DpOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates common random numbers (CRN) for reducing variance in policy-gradient reinforcement learning. Its main approach replaces the conventional state-value baseline in PPO with a noise-conditioned counterfactual baseline, computed by evaluating alternative actions from the same simulator state under the same transition-noise realization. The authors propose implementations based on simulator resets and Monte Carlo counterfactual action sampling, as well as a seed-conditioned value-function approach. Theoretical results establish unbiasedness and variance reduction under certain conditions, while experiments compare the proposed methods against standard PPO on several stochastic benchmark environments.

Critical strengths

The paper proposes a relatively simple modification to PPO that exploits reproducible simulator randomness without directly observing the noise variables. Experiments across several stochastic environments demonstrate potential improvements.

Critical weaknesses

1. Limited methodological and theoretical novelty.

The core idea of evaluating alternative actions under shared environment noise is well established, including in TRPO vine sampling (Schulman et al., 2015) [1]. Noise-dependent policy-gradient baselines have also been studied by Mao et al. (2019) [2] and Nota et al. (2021) [3]. The main theoretical results largely follow standard control-variate arguments: unbiasedness follows from the zero-mean policy-score identity, while advantage-variance reduction follows directly from the law of total variance. The paper does not convincingly establish a substantial methodological advance beyond adapting these existing ideas to PPO.

2. Insufficient validation of the practical method and its advantages.

The theoretical variance-reduction guarantee concerns an exact noise-conditioned baseline, whereas the implemented method uses a Monte Carlo approximation. The paper does not directly demonstrate that this practical estimator consistently reduces policy-gradient variance. Moreover, experiments primarily compare against standard PPO rather than existing variance-reduction methods [1–3]. The method also requires additional simulator evaluations and the ability to reset simulator and RNG states, limiting its applicability.

References

[1] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel. "Trust Region Policy Optimization." Proceedings of the 32nd International Conference on Machine Learning (ICML), PMLR 37:1889–1897, 2015.

[2] H. Mao, S. B. Venkatakrishnan, M. Schwarzkopf, and M. Alizadeh. "Variance Reduction for Reinforcement Learning in Input-Driven Environments." International Conference on Learning Representations (ICLR), 2019.

[3] C. Nota, P. S. Thomas, and B. C. da Silva. "Posterior Value Functions: Hindsight Baselines for Policy Gradient Methods." Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR 139:8238–8247, 2021.

Questions for authors
  1. What is the specific methodological contribution beyond TRPO vine sampling and previously established noise-dependent policy-gradient baselines?

  2. Can the authors provide comparisons with existing variance-reduction methods and clarify how robust the approach is to general stochastic dynamics and different simulator noise couplings?

1 comment

We thank the reviewer for their feedback. We would like to address the different concerns the reviewer brings up:

Limited methodological and theoretical novelty : We believe that the lack of novelty is overstated.

VINE runs several full (or truncated) rollouts from each state with common random numbers, and this is used for reducing the variance of the advantage only at that state. Our insight, based on the temporal difference formulation, allows two significant improvements: (1) it allows us to sample only single-step transitions per counterfactual action. More importantly, (2) the counterfactual actions for some state reduce the advantage variance of all of the states before it, through the GAE update rule. We added Figure 11 in the revised paper to show this schematically. Our algorithm is significantly more efficient than VINE. To demonstrate this, we added a comparison with VINE, see Appendix G.1, Figure 10.

Thank you for raising this question, we agree it wasn’t clear enough in the text. Another way to view our work is an extension of VINE to GAE. VINE, to the best of our knowledge, has seen very little practical use. As the experiments above show - our formulation leads to significant performance gains over VINE, which is what matters in practice.

The algorithm of Nota et al. applies to tabular POMDPs with an analytically computed belief state, and is not applicable to the experiments in our work. The algorithm of Mao et al. applies to input-driven MDPs, and trains a value conditioned on the input. It is also not applicable to the experiments in our paper. Additionally, we remark in the paper (Line 761) that both Nota et al. and Mao et al. are not compatible with the GAE.

Lastly, this paper is the first of its kind (as far as we are aware) to show a very simple algorithmic modification to plug in CRN into one of the most popular RL algorithms today - PPO with GAE. As we demonstrate, our new method leads to significant gains in several domains, which drives our main novelty claim.

Insufficient validation of the practical method and its advantages: We study the effect of different simulator noise in Figure 2, page 7.

Robustness to general stochastic dynamics: we are not sure what the reviewer means. Our evaluations across a diverse suite of tasks with diverse stochastic dynamics (Figures 2, 3, 4 domains Mujoco, Pufferlib, OR-gym) shows the robustness of our method. If the reviewer has a particular experiment in mind we are happy to add it.

Furthermore, Figure 5 in the main text compares the contribution of CFVE under different noise magnitudes across wall-clock time. The figure shows that as the noise magnitude increases, the variance reduction our method obtains is significant enough such that the performance gains outweigh the additional simulator steps (since results are reported under equal wallclock time).

In addition, per Reviewer qqf3’s suggestion, we conducted a control experiment where we remove Line 6 (the random number reset) from Algorithm 1, and verify that this hurts performance. As expected, the results show that our approach (Sampled CFVE) introduces improvement by reducing advantage variance.

Reviewer_zYibOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for reducing policy gradient variance in stochastic simulators. The method separates the randomness used for action selection from that used for environment transitions and uses common random numbers. This paper incorporates two methods into PPO. One is the Counterfactual Value Function Estimator (CFVE), which constructs a baseline by evaluating alternative actions for one step from the same state with the same environment noise. The other learns value functions conditioned on a finite set of random seeds. This paper gives conditions under which the noise-dependent value baseline reduces policy gradient variance compared with the standard state-dependent value baseline. It reports performance improvements over PPO with default settings on tasks such as inventory management and continuous control environments with added noise.

Critical strengths

This paper uses control over simulator randomness to construct a baseline without changing the objective for PPO policy updates. It shows performance improvements in several stochastic environments.

Critical weaknesses
  • W1 (Effects of randomness control are not isolated): CFVE lacks a control experiment with the same number of additional evaluations where only the environment noise is sampled independently. The seed-conditioned value functions in MuJoCo also lack a comparison with a standard value function trained using the same fixed seeds. It is difficult to assess the separate contributions of common random numbers and seed-conditioned value functions from the reported performance improvements.

  • W2 (Choice of method when states can be restored): CFVE assumes that states can be restored to try other actions. However, comparisons with existing methods are limited to PPO with default settings. This paper does not sufficiently discuss how CFVE relates to methods that train policies from search results [1, 2]. The practical benefit of choosing to improve PPO's estimator under these conditions is unclear.

References:

[1] Guo et al. Deep Learning for Real-Time Atari Game Play Using Offline Monte-Carlo Tree Search Planning. NIPS 2014.

[2] Anthony, Tian, and Barber. Thinking Fast and Slow with Deep Learning and Tree Search. NIPS 2017.

Questions for authors
  • Q1: Could you share a comparison between CFVE and a variant in which only the environment noise is sampled independently, with settings such as the number of additional evaluations kept the same? Could you also share a comparison with a standard value function trained using the same fixed seeds in MuJoCo? I would like to know the separate contributions of common random numbers and seed-conditioned value functions.

  • Q2: Could you explain the advantages of choosing CFVE over methods that use search to train policies when states can be restored? I would like to understand how CFVE compares in practice in terms of computational budget and conditions for use.

1 comment

We thank the reviewer for their review,

Control experiment with environment noise sampled independently: thank you for this interesting suggestion. If we understand the reviewer correctly, they mean to remove Line 6 (the random number reset) from Algorithm 1, and verify that this hurts performance.

We added this experiment in Appendix G.2 of the revised paper, see Figure 12. It is clear from these results that the RNG reset is critical for the performance improvement of CFVE.

Comparison to planning based methods (AlphaZero, ExIt): While CFVE requires resetting the simulator like planning-based methods, we are not sure how to conduct a meaningful comparison. Take for example AlphaZero - it is a popular method for games like Go/Chess, but it is not popular for continuous domains like robotics and the Mujoco domains we use. Running MCTS in continuous domains requires additional machinery (like sampling / progressive widening). A direct comparison with our method would depend on the sampling method, the MCTS variant, the implementation of the sampler (for the wall clock time), the MCTS budget, and other factors. We chose to compare with PPO, where the implementation is standard and the comparison is direct. PPO is actively being used in many applications (robotic locomotion, for instance), so we believe that the comparison carries practical value. Per other reviewers’ suggestions, we also added a comparison to VINE, which uses CRN in a different, less efficient manner (Appendix G.1, figure 10).

If the reviewer has a specific comparison with planning based methods in mind, we are happy to add it.

Reviewer_qqf3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a simple way to reduce noise in policy gradient reinforcement learning. The idea is to separate the random numbers used for action selection from those used for state transitions, so that alternative actions can be scored under the same transition noise. The authors turn this into two small changes to PPO, one based on resetting the simulator random number generator and one based on a value function with a separate head per random seed. They prove the first is unbiased and can reduce variance.

Critical strengths
  • The core idea is clean and easy to explain. It needs only the ability to reset the simulator random number generator. This is a small requirement compared with earlier methods that, e.g., need recurrent networks.
  • The method plugs into standard PPO code with little effort. This matters for adoption outside a specialized research group.
  • The limitations section is honest about extra simulation cost and about the restricted conditions for the theory.
Critical weaknesses
  • The exact estimator needs one extra simulator step for every action at every step. This is costly for large or continuous action spaces, and the paper's own results show that the benefit depends on how expensive the simulator is.
  • The theory does not show that policy gradient variance decreases in general. The guarantee holds only under extra conditions.
  • The argument for generalization across seeds is informal.
  • There is no direct comparison and the practical gain in comparison to opther counterfactual methods is not proved.
Questions for authors

Clarification request: The entropy gate is a heuristic with a threshold that appears to be tuned per domain.Is that correct?

1 comment

We thank the reviewer for their feedback.

Entropy gating: it is indeed a heuristic, however, it is not tuned per domain. We found that a value of 0.15 works well across all domains. We reported this in Section A.3.3.

Direct comparison to other counterfactual methods: We implemented VINE-PPO - a method incorporating the VINE algorithm from the original TRPO paper [1] into PPO, and we compare their performance in the revisioned paper (Appendix G.1, figure 10) . CFVE clearly improves upon VINE. The VINE algorithm implementation we used is detailed in Algorithm 4 in the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.