acceptodds
Under review as a conference paper at ICLR 2027

On Scaling Policy Networks in Policy-Gradient Reinforcement Learning

Abstract

Scaling network capacity has produced reliable gains in supervised learning, but scaling policy networks in reinforcement learning remains far less predictable. We study how policy depth interacts with optimization and interaction budget using a simple on-policy policy-gradient learner with Monte Carlo reward-to-go and a learned value baseline, alongside supporting bootstrapped variants. Across continuous-control experiments, we find that optimization conditions do not remain comparable as depth changes. Performance is highly sensitive to learning rate, while the region associated with strong performance shifts systematically with policy depth. As a result, transferring a learning rate selected for a shallow policy can confound additional capacity with optimization mismatch. Selecting learning rates separately by depth substantially changes the apparent scaling curve and reduces the measured penalty for 32-layer policies by about two fifths in our primary experiment. Reversing the transfer direction can also change the apparent depth ordering. Correcting this mismatch does not make deeper policies uniformly better. At moderate interaction budgets, deeper policies generally remain worse than shallow ones, while longer training produces task-dependent gains rather than a universal crossover. Our results show that policy-network scaling is jointly shaped by architectural depth, optimization protocol, and interaction budget. Shallow-rate transfer and per-depth selection answer different scientific questions and can yield materially different scaling conclusions.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Policy Gradient, Policy Networks, Network Depth, Learning Rate, Continuous Control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GrBJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is a study on the relationship between the size of the policy network and the learning rate in Reinforcement Learning. In RL, it is often the case that authors tune all the hyperparameters on smaller networks, and then report whether a bigger network, with more parameters, improves performance in the same regime. The authors argue that using a learning rate tuned for a shallow policy can confound the effects of increased network depth with optimization mismatch, and that separately tuning learning rates at each depth can lead to substantially different scaling conclusions. Specifically, they check what the gap is between policies trained with learning rates tuned for different network sizes. The experiments are repeated across a number of environments from DeepMind Control and MuJoCo, on multiple seeds and for a grid of learning rates and network depths, resulting in an impressive number of total runs. The authors find that separately tuning the learning rate for network size reduces the apparent gap in performance between shallow and deep networks. That being said, the deeper networks still underperform compared to shallower ones. This can be partially alleviated with an increase in interaction budget, but not always.

Critical strengths
  • Focus and depth: The authors pick one problem and explore it deeply. I think that this is commendable and informative. It allows the authors to isolate the problem and pick it apart precisely.
  • Experimental results: The authors conducted an impressive number of experiments to test a grid of possibilities on the number of environments, with appropriate numbers of seeds for statistical significance.
  • Shallow and Deep transfer testing. This is an interesting and clever experiment that well supports the points the authors are making.
  • Many previous works ablate their results with respect to network size without tuning learning rate (or other hyperparameters), so noticing this problem is valuable and can impact the field.
Critical weaknesses
  • All tested environments are relatively low-dimensional continuous-control tasks. I am concerned that these environments may not require sufficiently complex representations to demonstrate the benefits of additional policy depth. I think that perhaps a goal-conditioned environment with a maze could be a good study here to test the generality of the findings. Additional policy capacity may help there, since the smaller networks may have trouble understanding the layout of the maze. The authors even cite Wang et al. (2025) paper, which explicitly shows that depth unlocks new capabilities in goal reaching.
  • It is also unclear to me why the authors use this particular actor-critic setup. The authors report that “there are no trust-region constraints or clipped objectives, importance ratios, entropy bonuses, repeated actor epochs, or gradient-norm clipping.” (line 130). My intuition is that the authors wanted to test the simpler algorithm to decrease the effect of additional tricks and algorithmic improvements. These techniques can substantially affect training stability, sample efficiency, and sensitivity to optimization hyperparameters. The gap between tuned and untuned learning rates on more modern, sophisticated algorithms might be substantially smaller than what the authors show. I believe the decision to use such a simplified learner deserves further justification, given how strongly those choices can influence learning rate sensitivity.
  • Another concern I have is regarding the final scores on the DMC suite. The scores achieved by the algorithm in the paper are extremely low. In the original DMC paper (from 2018), several tasks achieve scores multiple times higher than those shown in this paper. For example, in “humanoid-walk,” the learner from the paper achieves a score below 20 after 20M interactions (on best depth). In the original DMC paper, the D4PG agent achieves a score of around 800 after 20M interactions and almost 1000 after 100M interactions. This gap can be due to the next weakness (very small number of updates), but I believe it should be addressed. This makes me question whether the algorithm studied here is appropriate or even “good”. I think that this is somewhat connected to the previous point. While I understand the motivation for using a simple learner, these low scores make me question whether results found in the paper can be applied broadly in currently used RL algorithms. I believe that including even a limited experiment with a competitive modern algorithm would substantially strengthen the practical relevance of the conclusions.
  • If I understand the hyperparameters listed in Table 2 correctly, in the base algorithm the policy gets a single update every 32 000 interactions with the environment. This seems like very infrequent updates (in total there are only 625 updates during 20M steps).
Questions for authors

The main questions I would like authors to answer are:

  1. What is the reasoning for omitting the stabilization techniques used in algorithms like PPO? Do the authors expect the gap between tuned and untuned learning rates to be lower when tested on more regularized, modern algorithms?
  2. Given the relatively low performance achieved on several DMC tasks, how confident are the authors that the observed learning-rate sensitivity would persist in stronger RL algorithms? Do they have any supporting evidence from more competitive learners?
  3. In the primary experiments, the actor receives only 625 batch updates over 20M steps. Is there a particular reason for this unusual choice? Have the authors tested more frequent updates?
Reviewer_hE6NOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents an analysis of how, in reinforcement learning (RL), making the policy network deeper changes the hyperparameters (specifically the learning rate). The submission uses several environments at several network depths to make their case, showing that in some cases simply changing the hyperparameter from a known good value to a value close by in their grid search can be detrimental to performance.

Critical strengths

None.

Critical weaknesses
  1. The presentation of the submission needs substantial re-writing and additions. There are no RL definitions. What is a policy? How is it trained? What are it's inputs? Outputs? What is on-policy? The introduction just jumps right into scaling laws. What is a scaling law? It is odd that the introduction includes equations 1 and 2. Those should be reserved for the problem definition section.

  2. This submission reads like a report on hyperparameter tuning. That is an acceptable direction (e.g. see Obando-Ceron et al 2023), but this submission is very surface level. We know that as we make networks deeper, gradients vanish and training networks becomes harder. The work of Nauman et al 2024 and Nauman et al 2025 show that to go deeper, you likely need skip connections and layer norms and a host of other tricks to make optimization stable. Shouldn't the analysis of this paper be performed with an architecture that can actually increase depth?

  3. the paper is hard to parse. I think the use of AI has clouded the writing of the authors in such a way that the actual contributions of this work are obfuscated. On line 215, what is "approximately half a decade"? Are we talking about 5 years? How is 5 years applicable to an RL paper?

"Across the primary grid, the selected learning rate decreases by about 0.20 decades per depth doubling, while nearby learning-rate mismatch"

this is just jargon that has no meaning. This trend continues throughout the paper. Terms are ill-defined, if defined at all, and there isn't really a coherent story to follow.

Questions for authors

Miscellaneous:

  1. Include McLean et al 2025, Obando-Ceron et al 2024a, in the scaling references.

  2. For the Optimization Sensitivity references, include Obando-Ceron et al, 2024b, and Obando-Ceron et al 2023.

Citations:

  1. Multi-Task Reinforcement Learning Enables Parameter Scaling. Reginald McLean, Evangelos Chatzaroulas, J K Terry, Isaac Woungang, Nariman Farsad, Pablo Samuel Castro. 2025.

  2. Mixtures of Experts Unlock Parameter Scaling for Deep RL. Johan Obando-Ceron, Ghada Sokar, Timon Willi, Clare Lyle, Jesse Farebrother, Jakob Foerster, Gintare Karolina Dziugaite, Doina Precup, Pablo Samuel Castro, 2024a.

  3. On the consistency of hyper-parameter selection in value-based deep reinforcement learning. Johan Obando-Ceron, João G.M. Araújo, Aaron Courville, Pablo Samuel Castro. 2024b.

  4. Small batch deep reinforcement learning. Johan Obando Ceron, Marc Bellemare, Pablo Samuel Castro. 2023.

Reviewer_4cr4OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses how policy depth interacts with optimization using an on-policy gradient learner. It finds that optimization conditions do not remain comparable as depth changes. Policy-depth scaling results are strongly affected by how the learning rate is chosen. As a result, transferring a learning rate selected for a shallow policy can confound additional capacity with optimization mismatch. Elaborate experiments show that policy-network scaling is jointly shaped by depth, optimization protocol, and interaction budget.

Critical strengths
  • Very detailed and elaborate experiments; they are careful and well-controlled. The primary experiment uses a complete grid over policy depths and learning rates and reconstructs different scaling curves from the same underlying runs, which makes the comparison protocol clean.
  • The conclusion is very careful. The paper claims that the learning-rate mismatch does not fully explain the poor performance of deep policies, which makes this conclusion more trustworthy.
Critical weaknesses
  • It is not clear to me whether the paper addresses a phenomenon specific to on-policy RL or a broader phenomenon that can be seen in many other methods. Of course, the scaling problem of on-policy sampling makes the paper’s setting specific, but it remains unclear how much of the observed effect can be observed in other ordinary depth-dependent optimization.
  • The concept is rather naïve. It is already well known that optimization hyperparameters can change with model scale, and how to tune hyperparameters is an important problem that has been addressed in many papers. Therefore, the main contribution here is rather the empirical quantification of this effect.
  • Many parameters other than the learning rate, such as batch size, critic capacity, initialization, and optimizer parameters, may fall within the scope of the experiments and ideally should be swept systematically.
Questions for authors
  • Do you have any fixed-data or off-policy data for comparison that can separate the on-policy feedback from ordinary depth-dependent optimization?
  • Do you have any (even small) experiments on sensitivity to other hyperparameters? Would the same conclusion still hold with the results for other hyperparameters?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.