- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper is a study on the relationship between the size of the policy network and the learning rate in Reinforcement Learning. In RL, it is often the case that authors tune all the hyperparameters on smaller networks, and then report whether a bigger network, with more parameters, improves performance in the same regime. The authors argue that using a learning rate tuned for a shallow policy can confound the effects of increased network depth with optimization mismatch, and that separately tuning learning rates at each depth can lead to substantially different scaling conclusions. Specifically, they check what the gap is between policies trained with learning rates tuned for different network sizes. The experiments are repeated across a number of environments from DeepMind Control and MuJoCo, on multiple seeds and for a grid of learning rates and network depths, resulting in an impressive number of total runs. The authors find that separately tuning the learning rate for network size reduces the apparent gap in performance between shallow and deep networks. That being said, the deeper networks still underperform compared to shallower ones. This can be partially alleviated with an increase in interaction budget, but not always.
Critical strengths
- Focus and depth: The authors pick one problem and explore it deeply. I think that this is commendable and informative. It allows the authors to isolate the problem and pick it apart precisely.
- Experimental results: The authors conducted an impressive number of experiments to test a grid of possibilities on the number of environments, with appropriate numbers of seeds for statistical significance.
- Shallow and Deep transfer testing. This is an interesting and clever experiment that well supports the points the authors are making.
- Many previous works ablate their results with respect to network size without tuning learning rate (or other hyperparameters), so noticing this problem is valuable and can impact the field.
Critical weaknesses
- All tested environments are relatively low-dimensional continuous-control tasks. I am concerned that these environments may not require sufficiently complex representations to demonstrate the benefits of additional policy depth. I think that perhaps a goal-conditioned environment with a maze could be a good study here to test the generality of the findings. Additional policy capacity may help there, since the smaller networks may have trouble understanding the layout of the maze. The authors even cite Wang et al. (2025) paper, which explicitly shows that depth unlocks new capabilities in goal reaching.
- It is also unclear to me why the authors use this particular actor-critic setup. The authors report that “there are no trust-region constraints or clipped objectives, importance ratios, entropy bonuses, repeated actor epochs, or gradient-norm clipping.” (line 130). My intuition is that the authors wanted to test the simpler algorithm to decrease the effect of additional tricks and algorithmic improvements. These techniques can substantially affect training stability, sample efficiency, and sensitivity to optimization hyperparameters. The gap between tuned and untuned learning rates on more modern, sophisticated algorithms might be substantially smaller than what the authors show. I believe the decision to use such a simplified learner deserves further justification, given how strongly those choices can influence learning rate sensitivity.
- Another concern I have is regarding the final scores on the DMC suite. The scores achieved by the algorithm in the paper are extremely low. In the original DMC paper (from 2018), several tasks achieve scores multiple times higher than those shown in this paper. For example, in “humanoid-walk,” the learner from the paper achieves a score below 20 after 20M interactions (on best depth). In the original DMC paper, the D4PG agent achieves a score of around 800 after 20M interactions and almost 1000 after 100M interactions. This gap can be due to the next weakness (very small number of updates), but I believe it should be addressed. This makes me question whether the algorithm studied here is appropriate or even “good”. I think that this is somewhat connected to the previous point. While I understand the motivation for using a simple learner, these low scores make me question whether results found in the paper can be applied broadly in currently used RL algorithms. I believe that including even a limited experiment with a competitive modern algorithm would substantially strengthen the practical relevance of the conclusions.
- If I understand the hyperparameters listed in Table 2 correctly, in the base algorithm the policy gets a single update every 32 000 interactions with the environment. This seems like very infrequent updates (in total there are only 625 updates during 20M steps).
Questions for authors
The main questions I would like authors to answer are:
- What is the reasoning for omitting the stabilization techniques used in algorithms like PPO? Do the authors expect the gap between tuned and untuned learning rates to be lower when tested on more regularized, modern algorithms?
- Given the relatively low performance achieved on several DMC tasks, how confident are the authors that the observed learning-rate sensitivity would persist in stronger RL algorithms? Do they have any supporting evidence from more competitive learners?
- In the primary experiments, the actor receives only 625 batch updates over 20M steps. Is there a particular reason for this unusual choice? Have the authors tested more frequent updates?