acceptodds
Under review as a conference paper at ICLR 2027

Beyond Fixed Learning Rates: Stochastic Armijo Line Search for On-Policy Policy Optimization

Abstract

Most reinforcement learning algorithms remain far from general purpose: they must be re-tuned for each new problem. The step size is a particularly challenging hyperparameter. Good step sizes depend on many interacting algorithmic components such as function approximators, objectives, and optimizers. Even the environment and experimental procedure can influence which step sizes work. Although the community has developed good default step sizes for many benchmark suites, the inherent dependence of the step size on the environment and experimental process often necessitates re-tuning on new problems or experiment setups. The optimization community has long relied on line search as a robust method of step size selection, and stochastic line searches are gaining popularity in supervised learning. Yet, a systematic exploration of line search strategies in reinforcement learning is lacking. In this paper, we present such a systematic empirical study, developing a line search method for on-policy policy optimization with PPO, which we call ***A**rgmin **A**rmijo **A**dam* (Adam3). We benchmark Adam3 across multiple popular benchmark suites and provide ablations to show failures in basic line search approaches and the utility of the additions in Adam3. Overall, we find Adam3 is easy to implement, does not increase wall-clock time and removes the step size as a hyperparameter, incurring little cost when the defaults work but performing well when they fail.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, ppo, policy optimization, armijo, line search, step size

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_1UPAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The work present Adam3, a novel stochastic line search approach that combine Adam and line search literature for RL step-size selection. The algorithm is developed to work coupled with PPO as a policy optimization algorithm. The Authors clearly define their method and present an extensive empirical evaluation against fixed and adaptive step-size baselines. Additional ablations support each design choice.

Critical strengths
  • The methodology being based on the well adopted Adam optimizer result easy to implement and computationally efficient (as shown in Table 1).
  • The experimental evaluation presented is extensive and convincing. The ablations show that searching along Adam’s own update direction fails, which the authors trace to that direction often being nearly orthogonal to the actor gradient. I also appreciate the Authors details on the implementation, making the results easily reproducible.
Critical weaknesses
  • The experimental results do not clearly show the improvement due to the line search. Besides changing the step-size rule, Adam3 increases the minibatch size. The fixed step-size baseline, Adam-B1024, which makes only this second change, matches Adam3 on the almost every task in Fig. 3. This make me raise some doubts about how much the line search improvement give w.r.t. a simple batch size incrementation. Moreover, it would have strengthened the empirical evaluation if the fixed step-size baseline were tuned for each task (with a corresponding ablation), the choice of adopting a general good step-size value (3e-4) makes the claim that Adam3 finds good step sizes is not proven.
  • The claim that Adam3 remove the step-size as a hyperparameter seems overstated in some parts. Within the experimental campaign the Authors adopts some design choice which improve the overall results and do not depend on the method itself, e.g., the usage of larger minibatch let PPO with Adam3 reach good results, but in the ablation of Fig. 4 with smaller batch size the performance drop drastically. Moreover, the methods adds some hyper parameters of its own, cc, δ\delta, βη\beta_\eta, ηinit\eta^{\text{init}}, which need to be tuned for each task (shown in the sensitivity analysis of Section 4.3).
Questions for authors
  • Can the Authors provide additional results on the fixed step-size baselines adopting linear annealing and KL-adaptive learning rate (like in Schulman, 2017)? Can the Authors provide a small grid of fixed step sizes at both minibatch sizes on a few environments, as a tuned reference?
  • How does the method respond to different values of βη\beta_\eta? Can the Authors elaborate more on the values used for βη\beta_\eta and cc?
  • It could be more informative to present the optimizer update time alone instead of the environment steps, which especially in not vectorized setting, can easily dominate in terms of computational time.
  • How can the methodology be extended to other algorithms?
6 comments
AuthorsOpenReview ↗

Author Response (1/3)

We thank the reviewer for their time and effort in producing a thorough review of our work. Your feedback is invaluable to help us strengthen our paper. Due to each reviewer’s feedback, we have run numerous additional experiments and adjusted the paper text. In the updated paper, we have highlighted all new text in purple, to make it clear what we have added as additions to the original manuscript.

Before addressing your concerns, we would like to first recall that our primary goal is to develop a step size selection algorithm for problems when tuning via a grid search is not tractable. This setting arises in many real-world applications due to time and safety constraints. For example, consider learning to control a greenhouse or a factory, where episodes of experience may take weeks and testing a poorly performing learning rate could destroy equipment. Our paper and response here revolve around that primary motivation.

The experimental results do not clearly show the improvement due to the line search. Besides changing the step-size rule, Adam3 increases the minibatch size. The fixed step-size baseline, Adam-B1024, which makes only this second change, matches Adam3 on the almost every task in Fig. 3. This make me raise some doubts about how much the line search improvement give w.r.t. a simple batch size incrementation.

This is a valid concern. As correctly pointed out, Figure 3 demonstrates that Adam-B1024 performs well and often matches Adam3 when we run for the standard number of experiment timesteps. One of the main benefits of Adam3 is that it can choose good step sizes throughout learning, and Figure 2 in the paper clearly shows this. Whereas Adam-B1024 catastrophically fails on Humanoid and Ant when run for longer than the standard number of timesteps (50 million), Adam3 continues to choose good step sizes. Figure 2 primarily shows the benefit of Adam3 over the simple batch size increment.

Moreover, it would have strengthened the empirical evaluation if the fixed step-size baseline were tuned for each task (with a corresponding ablation), the choice of adopting a general good step-size value (3e-4) makes the claim that Adam3 finds good step sizes is not proven.

Can the Authors provide a small grid of fixed step sizes at both minibatch sizes on a few environments, as a tuned reference?

Thank you for this valuable suggestion. Due to our problem setting – cases where tuning is impractical – we decided to use a single, known-good, hyperparameter configuration for PPO throughout our empirical study. This reflects what would happen when approaching a new problem in practice, where a reasonable strategy would be to use community-derived default hyperparameters. Furthermore, recent work has argued that using this kind of setup results in fairer algorithmic comparisons [3].

We do not claim that Adam3 can attain state-of-the-art PPO performance over a finely tuned, fixed-step-size PPO baseline, nor do we claim that Adam3 selects optimal step sizes. We only claim that Adam3 can select good step sizes for PPO – step sizes that induce reasonably good performance –without requiring problem-specific tuning. This is exactly what makes Adam3 useful for our problem setting. In such settings, a reasonable strategy is to use known default hyperparameters for an algorithm, which is why we use PPO with defaults as our performance baseline. Our paper is mainly interested in making progress on this reasonable baseline with an automatic step size selection mechanism.

Nevertheless, we strongly agree that understanding how Adam3 compares to a per-environment oracle fixed step size can be useful to quantify the performance Adam3 leaves on the table. This could further motivate future research into automatic step size methods. Because of this, we have included in Figure 12 in Appendix C.4 a comparison of PPO with Adam and Adam-B1024 with a learning rate sweep in {3e−6,3e−5,3e−4,3e−3,3e−2,3e−1}\lbrace 3e-6, 3e-5, 3e-4, 3e-3, 3e-2, 3e-1\rbrace across all standard benchmark suites we considered in the paper. The results indicate that the best fixed step size is the default step size of 3e−43e-4, which is the step size we used throughout the main paper experiments for the fixed step size PPO baselines. These results indicate that Adam3 is competitive with the best fixed step size across each environment suite and in aggregate across suites.

AuthorsOpenReview ↗

Author Response (2/3)

The claim that Adam3 remove the step-size as a hyperparameter seems overstated in some parts. Within the experimental campaign the Authors adopts some design choice which improve the overall results and do not depend on the method itself, e.g., the usage of larger minibatch let PPO with Adam3 reach good results, but in the ablation of Fig. 4 with smaller batch size the performance drop drastically.

We agree, and explicitly acknowledge here and in the manuscript that we use a larger minibatch size for Adam3. It is well-known in the literature that line search methods typically need larger batch sizes to do well [1]. We explicitly show this in the ablation study, to transparently convey an explicit limitation of Adam3. Besides this, we are not aware of any other design choice of Adam3 that we adopted to improve the overall results which do not depend on the method itself. Nevertheless, we are happy to further discuss any such potential design choice that we may have missed.

Moreover, the methods adds some hyper parameters of its own, cc, δ\delta, βη\beta_\eta, ηinit\eta^\mathrm{init}, which need to be tuned for each task (shown in the sensitivity analysis of Section 4.3).

This is a good observation, and exactly why we included the sensitivity analysis of Section 4.3. We argue that this analysis shows that across the considered benchmark suites, Adam3 is relatively insensitive to its additional hyperparameters. Figure 5, right shows that performance does not change drastically when changing δ\delta and BB. Figure 5, left shows that performance is relatively insensitive to cc too. Do note that the performance drop at c=0.99c=0.99 is expected, and we only included the value of c=0.99c=0.99 in our analysis to cover the full range of cc. Adam3 uses a modified Armijo line search. As c→1c \to 1, this demands that the actual improvement from the accepted step size be at least as large as the first-order Taylor approximation predicts. For a locally convex objective with positive curvature (expected near an optimum), that is impossible for any sufficiently small step size.

We recommend keeping ηinit\eta^\mathrm{init} at 1, though the algorithm should be insensitive to increases in ηinit\eta^\mathrm{init} since theoretically any larger step size will be rejected unless it makes explicit progress on the objective. We also found this to be the case empirically.

We advocate setting βη\beta_\eta to 0.99, which we use across all settings, and not tuning it.

Can the Authors provide additional results on the fixed step-size baselines adopting linear annealing and KL-adaptive learning rate (like in Schulman, 2017)?

We also ran PPO with learning rate annealing, and we report it in the updated manuscript in Figure 13 in Appendix C.5. The results show that on the Brax50M suite, learning rate annealing can harm performance, and that Adam3 outperforms learning rate annealing with Adam and Adam-B1024 on this suite. Across the other benchmark problem suites, we found that learning rate annealing had relatively little impact on performance.

We were unable to find any KL-adaptive learning rate mechanism in the original PPO paper (Schulman, 2017). Do you mean the adaptive KL penalty from section 4 of the PPO paper? If you would like any further clarification from us here, let us know.

AuthorsOpenReview ↗

Author Response (3/3)

How does the method respond to different values of beta_eta? Can the Authors elaborate more on the values used for βη\beta_\eta and cc?

We mainly chose βη\beta_\eta to match the momentum EMA parameter of Adam. The hyperparameter cc controls the Armijo condition, and theory suggested setting c≈0.7c \approx 0.7 [2]. Please let us know if this addresses your questions about elaboration on βη\beta_\eta and cc. We are happy to provide further details!

It could be more informative to present the optimizer update time alone instead of the environment steps, which especially in not vectorized setting, can easily dominate in terms of computational time.

Thank you for this valuable suggestion! We agree that our throughput reports in Tables 1 and 2 conflate optimization time with environment interaction time. We have now adjusted the tables to address this. Table 1 now reports env interaction time/step\text{env interaction time}/\text{step} across multiple different neural network architectures, including the MLP and CNN networks. Table 2 in Appendix A now reports the optimization time/step\text{optimization time}/\text{step}, the average optimization update time per step. Both tables normalize relative to the PPO with Adam baseline. An important caveat is that algorithms that use a larger batch size (Adam-B1024, Adam3) often run faster than the default in terms of average optimization time per step, simply because with PPO this translates to fewer gradient updates.

As shown by these new tables, Adam3 remains competitive across the environments tested in our paper. For the larger Atari network, Adam3 can remain competitively fast with fewer forward passes. The sensitivity analysis in Section 4.3 suggests fewer forward passes can be used with very little performance degradation.

How can the methodology be extended to other algorithms?

Adam3 can be directly applied to the actor and critic in algorithms like SAC and applied just to the action-values for algorithms like DQN. We have opted not to do so in this work, to keep the scope on on-policy optimization with PPO. We want a deeper study on this class of algorithms. Though it is straightforward to apply Adam3 to these other algorithms, there may be different interactions that require small changes to the algorithm, which we leave to future work to investigate deeply.

References

[1] Vincent Roulet, Atish Agarwala, Fabian Pedregosa. On the Interplay Between Stepsize Tuning and Progressive Sharpening. NeurIPS Annual Workshop on Optimization for Machine Learning, 2023.

[2] Sharan Vaswani, Issam Laradji, Frederik Kunstner, Si Yi Meng, Mark Schmidt, and Simon Lacoste-Julien. Adaptive Gradient Methods Converge Faster with Over-Parameterization (But You Should do a Line-Search). NeurIPS Annual Workshop on Optimization for Machine Learning, 2020.

[3] Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, Adam White. The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning. RLC, 2024.

Reviewer_NkWdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a line search method, Argmin Armijo Adam (Adam3), for step size selection in the Proximal Policy Optimization (PPO) algorithm in deep reinforcement learning (RL). It uses Adam3 to augment the Adam optimizers of actor and critic parameters in PPO. The paper empirically shows the effectiveness of Adam3 in environments from six suites and also presents an ablation study on the design choices.

Critical strengths

Adam3 seems to alleviate the need to tune the learning rate hyperparameter for different benchmarks. The most impressive result for me is the significantly better performance of Adam3 in Brax training with 5050M steps (Figure 22). It also shows competitive or better performance than default fixed learning rate for diverse suites (Figure 33). Based on the ablation study, it also seems that Adam3 is not very sensitive to its hyperparameters.

Critical weaknesses
  1. The Adam3 strategy is presented as a way to avoid tuning a fixed learning rate for different settings. However, the effectiveness of Adam3 is only demonstrated on different suites and training durations. The step size is influenced by different parameterizations and optimization landscapes as well, as noted in Line 4242 of the paper: "A good setting depends on the optimization landscape, itself a function of the environment, objective, and function approximator". The generality of the recipe of Adam3 for on-policy policy optimization is not established. Different architecture choices, or higher update-to-data ratio, or different optimization landscape (like PPO with entropy regularization) is not studied, thus narrowing the utility of Adam3 to just the surrogate objective of PPO on very few settings.
  2. The line search conducted by Adam3 involves B=30B=30 times more forward passes for both actor and critic to compute the objective values. While the paper shows similar wall clock time for classic PPO vs. PPO with Adam3, I think the contribution of optimization step against interactions is not clear. It would be better if the wall clock time is compared only for the optimization step. The paper also does not compare the effect on wall clock time when the architecture is scaled, which raises questions on the feasibility of using this method on benchmarks that require way larger networks (like the proper Atari setting).
  3. In the Adam-B1024 experiment in Brax 5050M steps setting, the batch size is scaled to 10241024 for PPO without appropriately adjusting the learning rate, thus disadvantaging a potentially strong baseline over heuristically selected c,δ,B,βηc, \delta, B, \beta_\eta and ηinit\eta^{\text{init}} hyperparameters of Adam3. Additonally, the hyperparameter cc is not ablated in the 0.7−0.990.7-0.99 region in this setting, where Adam3 seems to be particularly sensitive.
Questions for authors
  1. The choice of actor starting the line search from the critic's last step size would restrict the actor to have lower step sizes than the critic. However, it is unclear to me as to why the actor should be restricted by critic step sizes. What would happen when there is a high disparity in their parameterizations (like in Simba [1] PPO architecture)?
  2. The choice for varying δ\delta along with BB for the ablation study is unclear. Could you please clarify how the ablation sufficiently demonstrates the combined effect of both hyperparameters, and hence low sensitivity to both?
  3. It might be better to aggregate performances across seeds and environments using the IQM metric, as it is a more popular choice in the RL community to have robustness against outliers [2].

I also want to add a few comments related to writing and presentation of the paper:

  1. Figure 44 and Figure 77 note "Batch 32" as the experiment name, whereas text describes it as "Batch 64". I assume this is a typo and the default batch size of 6464 was used in the experiments.
  2. Line 170170 has "where δ>1\delta > 1", which should be "where δ<1\delta < 1".
  3. Line 507507 (Reproducibility Statement) ends abruptly.

References: [1] Lee et al. SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning. ICLR 2025.

[2] Agarwal et al. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS 2021.

Reviewer_VT2ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Argmin Armijo Adam (Adam3), a line search method for on-policy reinforcement learning with PPO that aims to eliminate the need to tune the step size. The authors conduct an empirical evaluation across multiple benchmark suites and provide ablation studies investigating the limitations of conventional line search approaches and the contributions of the proposed modifications. The results suggest that Adam3 can improve performance when default optimization settings are ineffective, while maintaining comparable wall-clock time and introducing limited additional computational overhead.

Critical strengths
  1. The paper addresses a relevant practical challenge in reinforcement learning by attempting to reduce the dependence on learning rate tuning in PPO through an adaptive line search approach.
  2. The empirical study includes multiple benchmark suites and ablation experiments, providing useful insights into the limitations of conventional line search methods and the proposed modifications.
Critical weaknesses
  1. The fairness of the empirical comparisons is not sufficiently established. The PPO baselines appear to rely on implementations and default configurations associated with Gymnasium MuJoCo, while some experiments are conducted in Brax environments, which may require different optimization settings. It is unclear whether the baseline configurations are appropriate for these environments. The reported performance, particularly on Ant, also requires further explanation, as PPO can perform well with standard configurations. Additional evaluations using established PPO benchmark settings would help clarify whether the improvements arise from the proposed optimizer or differences in experimental configurations.
  2. The evaluation does not sufficiently establish the advantages of Adam3 over properly tuned PPO baselines. Since the primary motivation is to eliminate step size tuning, the comparison with fixed learning rates is particularly important. The paper mentions that learning rate annealing was not used for PPO with fixed step sizes, but it remains unclear how learning rate selection and tuning affect the reported performance. A systematic comparison across learning rates and environments is necessary to determine whether Adam3 provides meaningful advantages over conventional Adam optimization. Furthermore, the applicability of the proposed method beyond PPO remains unclear.
Questions for authors
  1. Were the PPO baseline configurations specifically tuned for the Brax environments, or were default settings from Gymnasium MuJoCo implementations used?
  2. How does Adam3 compare with PPO using systematically tuned learning rates and learning rate schedules across all evaluated environments?
  3. What explains the reported PPO performance on environments such as Ant, and how does it compare with established PPO implementations under comparable settings?
  4. Is Adam3 specific to PPO, or can the proposed line search mechanism be applied to other policy optimization algorithms?
2 comments
AuthorsOpenReview ↗

Author Response (1/2)

We thank the reviewer for the thorough and insightful comments and appreciate the recognition of the real, practical challenge that hyperparameter tuning poses for RL. We believe that your feedback will help us improve our paper. Due to each reviewer’s feedback, we have run numerous additional experiments and adjusted the paper text. In the updated paper, we have highlighted all new text in purple, to make it clear what we have added as additions to the original manuscript.

Before addressing your concerns, we would like to first recall that our primary goal is to develop a step size selection algorithm for problems when tuning via a grid search is not tractable. This setting arises in many real-world applications due to time and safety constraints. For example, consider learning to control a greenhouse or a factory, where episodes of experience may take weeks and testing a poorly performing learning rate could destroy equipment. Our paper and response here revolve around that primary motivation.

The fairness of the empirical comparisons is not sufficiently established. The PPO baselines appear to rely on implementations and default configurations associated with Gymnasium MuJoCo, while some experiments are conducted in Brax environments, which may require different optimization settings.

We agree that a careful tuning of batch sizes, epochs, rollout lengths, etc could improve the implementation’s performance on Brax as has been reported in the literature. However, we do not think that this is required to support our conclusions. The setting we motivate in the work is new problems where tuning is difficult or undesirable. This motivates the use of defaults for PPO (which is a likely starting point in the absence of the ability to tune), where the defaults are chosen from popular code bases [2,3]. These codebases would likely be the implementations/hyperparameters selected if a person were to deploy PPO on a new problem. In fact, we argue that PPO’s sensitivity to hyperparameters supports our paper more, and our goal is to address this issue for environments where a grid search is not tractable. Additionally, recent work has motivated using this kind of single hyperparameter configuration across suites for more fair algorithmic comparisons [5].

We also note that the Stable Baselines 3 implementation we use shares many hyperparameters with PureJaxRL, which was specifically designed for Brax, including: step size of 3e-43\text{e-}4, γ=0.99\gamma = 0.99, GAE λ=0.95\lambda = 0.95, εclip=0.2\varepsilon_\text{clip}=0.2, no entropy regularization, maximum gradient norm of 0.50.5, hyperbolic tangent activations, etc. Similar to our setup, PureJaxRL also keeps the 37-details tricks: orthogonal initialization, observation/reward normalization, value clipping, no step size annealing, and advantage normalization [4]. Our implementation is built on PureJaxRL.

The evaluation does not sufficiently establish the advantages of Adam3 over properly tuned PPO baselines. Since the primary motivation is to eliminate step size tuning, the comparison with fixed learning rates is particularly important. The paper mentions that learning rate annealing was not used for PPO with fixed step sizes, but it remains unclear how learning rate selection and tuning affect the reported performance. A systematic comparison across learning rates and environments is necessary to determine whether Adam3 provides meaningful advantages over conventional Adam optimization.

This is a valuable point, and one that we will better address in the updated manuscript. We do not intend to attain state-of-the-art PPO performance over a finely tuned PPO baseline, and we do not make these kinds of claims in our paper. Rather, our motivation encapsulates selecting step sizes on problems where sweeping is difficult, impossible, or undesirable. In these kinds of cases, a reasonable strategy would be to use PPO’s default hyperparameters. Our paper is mainly interested in making progress on this reasonable baseline with an automatic step size selection mechanism. Our claim is that in these cases, Adam3 can recover a good enough step size.

Nevertheless, we strongly agree that understanding how Adam3 compares to an oracle fixed step size can be useful to quantify the performance Adam3 leaves on the table. This could further motivate future research into automatic step size methods. Because of this, we have included in Figure 12 in Appendix C.4 a comparison of PPO (batch sizes 64, 1024) with a step size sweep in [{3e−6, 3e−5, 3e−4, 3e−3, 3e−2, 3e−1}\{ 3e{-6},\ 3e{-5},\ 3e{-4},\ 3e{-3},\ 3e{-2},\ 3e{-1} \}] across all standard benchmark suites we considered in the paper. The figure shows that across these benchmark suites, the default step size of 3e−43e-4, that which we use in the main paper, is the best.

AuthorsOpenReview ↗

Author Response (2/2)

Furthermore, the applicability of the proposed method beyond PPO remains unclear.

Adam3 is a generic line search approach and a next step is to apply to algorithms beyond PPO. However, our focus in this paper is on the on-policy policy optimization setting with PPO, given this is such a widely used algorithm. We start with a narrower scope to provide deeper insights into this important setting. Extensions to other algorithms and settings is an important next step.

Were the PPO baseline configurations specifically tuned for the Brax environments, or were default settings from Gymnasium MuJoCo implementations used?

We use default PPO hyperparameters from Stable Baselines 3, which are also used in CleanRL [2,3]. We did not re-tune hyperparameters for Brax, to better simulate our setting of interest (an end-user applying PPO to a new problem in the real world) as explained above. However, as the SB3 hyperparameters are associated with MuJoCo, we also provide results for 50M timesteps in Gymnasium MuJoCo-v5, where we found results closely align with our result for 50M timesteps in Brax for all methods. These results are now shown in Figure 3.

What explains the reported PPO performance on environments such as Ant, and how does it compare with established PPO implementations under comparable settings?

We explicitly point out that our baseline PPO algorithm does indeed match what is reported for PPO on Brax with the hyperparameters we used [1]. The poor performance in Ant that we observed occurs when PPO is run with its default step size over 50m steps, where our proposed method Adam3 performs better. Our empirical results are also averaged across more than 20 random seeds per environment (many more than typically reported in the literature), so the high-variance nature of RL return data may also explain discrepancies with other papers in the literature that use far fewer random seeds [6].

How does Adam3 compare with PPO using systematically tuned learning rates and learning rate schedules across all evaluated environments?

Figure 12 in Appendix C.4 shows the performance attained by PPO with Adam and Adam-B1024 across all considered environment suites for each step size in [{3e−6, 3e−5, 3e−4, 3e−3, 3e−2, 3e−1}\{ 3e{-6},\ 3e{-5},\ 3e{-4},\ 3e{-3},\ 3e{-2},\ 3e{-1} \}]. The figure shows that the default step size, which is the one we report performance for throughout the main paper, is the best across environment suites.

Is Adam3 specific to PPO, or can the proposed line search mechanism be applied to other policy optimization algorithms?

Adam3 can be directly applied to the actor and critic in algorithms like SAC and applied just to the action-values for algorithms like DQN. We have opted not to do so in this work, to keep the scope on on-policy optimization with PPO. We want a deeper study on this class of algorithms. Though it is straightforward to apply Adam3 to these other algorithms, there may be different interactions that require small changes to the algorithm and is future work to investigate deeply.

References

[1] Theresa Eimer, Marius Lindauer, and Roberta Raileanu. Hyperparameters in Reinforcement Learning and How To Tune Them. ICML, 2024.

[2] Antonin Raffin and Ashley Hill and Adam Gleave and Anssi Kanervisto and Maximilian Ernestus and Noah Dormann. Stable-Baselines3: Reliable Reinforcement Learning Implementations. JMLR, 2021.

[3] Shengyi Huang and Rousslan Fernand Julien Dossa and Chang Ye and Jeff Braga and Dipam Chakraborty and Kinal Mehta and João G.M. Araújo. CleanRL: High-quality Single-file Implementations of Deep Reinforcement Learning Algorithms. JMLR, 2022.

[4] Shengyi Huang, Rousslan Fernand Julien Dossa, Antonin Raffin, Anssi Kanervisto, and Weixun Wang. The 37 Implementation Details of Proximal Policy Optimization. ICLR Blog Track, 2022.

[5] Andrew Patterson, Samuel Neumann, Raksha Kumaraswamy, Martha White, Adam White. The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning. RLC, 2024.

[6] Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, David Meger. Deep RL that Matters. AAAI, 2018.

Comments on the paper

2 comments
Reviewer_NkWdOpenReview ↗

Thanks for the Reply

I thank the authors for their detailed responses and supporting experiments to address my concerns. I will update the review by taking into consideration the authors' clarifications and edits.

AuthorsOpenReview ↗

Thank you for engaging

Thank you for engaging and if you have any further questions, we can turn around quickly with an answer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.