- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The work present Adam3, a novel stochastic line search approach that combine Adam and line search literature for RL step-size selection. The algorithm is developed to work coupled with PPO as a policy optimization algorithm. The Authors clearly define their method and present an extensive empirical evaluation against fixed and adaptive step-size baselines. Additional ablations support each design choice.
Critical strengths
- The methodology being based on the well adopted Adam optimizer result easy to implement and computationally efficient (as shown in Table 1).
- The experimental evaluation presented is extensive and convincing. The ablations show that searching along Adam’s own update direction fails, which the authors trace to that direction often being nearly orthogonal to the actor gradient. I also appreciate the Authors details on the implementation, making the results easily reproducible.
Critical weaknesses
- The experimental results do not clearly show the improvement due to the line search. Besides changing the step-size rule, Adam3 increases the minibatch size. The fixed step-size baseline, Adam-B1024, which makes only this second change, matches Adam3 on the almost every task in Fig. 3. This make me raise some doubts about how much the line search improvement give w.r.t. a simple batch size incrementation. Moreover, it would have strengthened the empirical evaluation if the fixed step-size baseline were tuned for each task (with a corresponding ablation), the choice of adopting a general good step-size value (3e-4) makes the claim that Adam3 finds good step sizes is not proven.
- The claim that Adam3 remove the step-size as a hyperparameter seems overstated in some parts. Within the experimental campaign the Authors adopts some design choice which improve the overall results and do not depend on the method itself, e.g., the usage of larger minibatch let PPO with Adam3 reach good results, but in the ablation of Fig. 4 with smaller batch size the performance drop drastically. Moreover, the methods adds some hyper parameters of its own, , , , , which need to be tuned for each task (shown in the sensitivity analysis of Section 4.3).
Questions for authors
- Can the Authors provide additional results on the fixed step-size baselines adopting linear annealing and KL-adaptive learning rate (like in Schulman, 2017)? Can the Authors provide a small grid of fixed step sizes at both minibatch sizes on a few environments, as a tuned reference?
- How does the method respond to different values of ? Can the Authors elaborate more on the values used for and ?
- It could be more informative to present the optimizer update time alone instead of the environment steps, which especially in not vectorized setting, can easily dominate in terms of computational time.
- How can the methodology be extended to other algorithms?
6 comments
Author Response (2/3)
The claim that Adam3 remove the step-size as a hyperparameter seems overstated in some parts. Within the experimental campaign the Authors adopts some design choice which improve the overall results and do not depend on the method itself, e.g., the usage of larger minibatch let PPO with Adam3 reach good results, but in the ablation of Fig. 4 with smaller batch size the performance drop drastically.
We agree, and explicitly acknowledge here and in the manuscript that we use a larger minibatch size for Adam3. It is well-known in the literature that line search methods typically need larger batch sizes to do well [1]. We explicitly show this in the ablation study, to transparently convey an explicit limitation of Adam3. Besides this, we are not aware of any other design choice of Adam3 that we adopted to improve the overall results which do not depend on the method itself. Nevertheless, we are happy to further discuss any such potential design choice that we may have missed.
Moreover, the methods adds some hyper parameters of its own, , , , , which need to be tuned for each task (shown in the sensitivity analysis of Section 4.3).
This is a good observation, and exactly why we included the sensitivity analysis of Section 4.3. We argue that this analysis shows that across the considered benchmark suites, Adam3 is relatively insensitive to its additional hyperparameters. Figure 5, right shows that performance does not change drastically when changing and . Figure 5, left shows that performance is relatively insensitive to too. Do note that the performance drop at is expected, and we only included the value of in our analysis to cover the full range of . Adam3 uses a modified Armijo line search. As , this demands that the actual improvement from the accepted step size be at least as large as the first-order Taylor approximation predicts. For a locally convex objective with positive curvature (expected near an optimum), that is impossible for any sufficiently small step size.
We recommend keeping at 1, though the algorithm should be insensitive to increases in since theoretically any larger step size will be rejected unless it makes explicit progress on the objective. We also found this to be the case empirically.
We advocate setting to 0.99, which we use across all settings, and not tuning it.
Can the Authors provide additional results on the fixed step-size baselines adopting linear annealing and KL-adaptive learning rate (like in Schulman, 2017)?
We also ran PPO with learning rate annealing, and we report it in the updated manuscript in Figure 13 in Appendix C.5. The results show that on the Brax50M suite, learning rate annealing can harm performance, and that Adam3 outperforms learning rate annealing with Adam and Adam-B1024 on this suite. Across the other benchmark problem suites, we found that learning rate annealing had relatively little impact on performance.
We were unable to find any KL-adaptive learning rate mechanism in the original PPO paper (Schulman, 2017). Do you mean the adaptive KL penalty from section 4 of the PPO paper? If you would like any further clarification from us here, let us know.
Author Response (3/3)
How does the method respond to different values of beta_eta? Can the Authors elaborate more on the values used for and ?
We mainly chose to match the momentum EMA parameter of Adam. The hyperparameter controls the Armijo condition, and theory suggested setting [2]. Please let us know if this addresses your questions about elaboration on and . We are happy to provide further details!
It could be more informative to present the optimizer update time alone instead of the environment steps, which especially in not vectorized setting, can easily dominate in terms of computational time.
Thank you for this valuable suggestion! We agree that our throughput reports in Tables 1 and 2 conflate optimization time with environment interaction time. We have now adjusted the tables to address this. Table 1 now reports across multiple different neural network architectures, including the MLP and CNN networks. Table 2 in Appendix A now reports the , the average optimization update time per step. Both tables normalize relative to the PPO with Adam baseline. An important caveat is that algorithms that use a larger batch size (Adam-B1024, Adam3) often run faster than the default in terms of average optimization time per step, simply because with PPO this translates to fewer gradient updates.
As shown by these new tables, Adam3 remains competitive across the environments tested in our paper. For the larger Atari network, Adam3 can remain competitively fast with fewer forward passes. The sensitivity analysis in Section 4.3 suggests fewer forward passes can be used with very little performance degradation.
How can the methodology be extended to other algorithms?
Adam3 can be directly applied to the actor and critic in algorithms like SAC and applied just to the action-values for algorithms like DQN. We have opted not to do so in this work, to keep the scope on on-policy optimization with PPO. We want a deeper study on this class of algorithms. Though it is straightforward to apply Adam3 to these other algorithms, there may be different interactions that require small changes to the algorithm, which we leave to future work to investigate deeply.
References
[1] Vincent Roulet, Atish Agarwala, Fabian Pedregosa. On the Interplay Between Stepsize Tuning and Progressive Sharpening. NeurIPS Annual Workshop on Optimization for Machine Learning, 2023.
[2] Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien. Adaptive Gradient Methods Converge Faster with Over-Parameterization (But You Should do a Line-Search). NeurIPS Annual Workshop on Optimization for Machine Learning, 2020.
[3] Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, Adam White. The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning. RLC, 2024.
Author Response (1/3)
We thank the reviewer for their time and effort in producing a thorough review of our work. Your feedback is invaluable to help us strengthen our paper. Due to each reviewer’s feedback, we have run numerous additional experiments and adjusted the paper text. In the updated paper, we have highlighted all new text in purple, to make it clear what we have added as additions to the original manuscript.
Before addressing your concerns, we would like to first recall that our primary goal is to develop a step size selection algorithm for problems when tuning via a grid search is not tractable. This setting arises in many real-world applications due to time and safety constraints. For example, consider learning to control a greenhouse or a factory, where episodes of experience may take weeks and testing a poorly performing learning rate could destroy equipment. Our paper and response here revolve around that primary motivation.
This is a valid concern. As correctly pointed out, Figure 3 demonstrates that Adam-B1024 performs well and often matches Adam3 when we run for the standard number of experiment timesteps. One of the main benefits of Adam3 is that it can choose good step sizes throughout learning, and Figure 2 in the paper clearly shows this. Whereas Adam-B1024 catastrophically fails on Humanoid and Ant when run for longer than the standard number of timesteps (50 million), Adam3 continues to choose good step sizes. Figure 2 primarily shows the benefit of Adam3 over the simple batch size increment.
Thank you for this valuable suggestion. Due to our problem setting – cases where tuning is impractical – we decided to use a single, known-good, hyperparameter configuration for PPO throughout our empirical study. This reflects what would happen when approaching a new problem in practice, where a reasonable strategy would be to use community-derived default hyperparameters. Furthermore, recent work has argued that using this kind of setup results in fairer algorithmic comparisons [3].
We do not claim that Adam3 can attain state-of-the-art PPO performance over a finely tuned, fixed-step-size PPO baseline, nor do we claim that Adam3 selects optimal step sizes. We only claim that Adam3 can select good step sizes for PPO – step sizes that induce reasonably good performance –without requiring problem-specific tuning. This is exactly what makes Adam3 useful for our problem setting. In such settings, a reasonable strategy is to use known default hyperparameters for an algorithm, which is why we use PPO with defaults as our performance baseline. Our paper is mainly interested in making progress on this reasonable baseline with an automatic step size selection mechanism.
Nevertheless, we strongly agree that understanding how Adam3 compares to a per-environment oracle fixed step size can be useful to quantify the performance Adam3 leaves on the table. This could further motivate future research into automatic step size methods. Because of this, we have included in Figure 12 in Appendix C.4 a comparison of PPO with Adam and Adam-B1024 with a learning rate sweep in {3e−6,3e−5,3e−4,3e−3,3e−2,3e−1} across all standard benchmark suites we considered in the paper. The results indicate that the best fixed step size is the default step size of 3e−4, which is the step size we used throughout the main paper experiments for the fixed step size PPO baselines. These results indicate that Adam3 is competitive with the best fixed step size across each environment suite and in aggregate across suites.