acceptodds
Under review as a conference paper at ICLR 2027

ALOHA: Agentic LLM-driven Online Hyperparameter Adaptation for Reinforcement Learning

Abstract

Hyperparameter optimization (HPO) for reinforcement learning is computationally expensive. Offline methods such as Bayesian optimization require hundreds of complete training runs while population-based methods demand dozens of parallel agents, making both approaches impractical when simulation costs are high relative to the available compute budget. We introduce ALOHA (Agentic LLM-driven Online Hyperparameter Adaptation), a method for online hyperparameter optimization within a single training run, that embeds a large language model directly into the RL training loop to adaptively adjust hyperparameters at fixed intervention points based on online training metrics, requiring no preprocessing and no parallel agents. At each intervention, ALOHA produces updated hyperparameter values alongside natural-language reasoning traces that explicitly justify each adjustment, making each decision interpretable. We evaluate ALOHA across three RL algorithms (PPO, DQN, SAC) and a diverse suite of environments spanning continuous locomotion, discrete control, and grid navigation, benchmarked using ARLBench. Our results demonstrate that ALOHA matches or exceeds Bayesian optimization in 9 of 14 environments and outperforms population-based training across all 14, requiring no offline search budget compared to Bayesian optimization SMAC's 400 total training runs (80 trials 5 seeds) per environment, without any preprocessing runs, parallel agents, or domain-specific tuning.

Primary area:
reinforcement learning
Keywords:
Agentic AI, Automated Reinforcement Learning, Reinforcement Learning, Hyperparameter Optimization, Automated Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_F5gyOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper introduces ALOHA, an LLM-based approach for online hyperparameter adaptation in RL. An LLM observes training diagnostics and adjusts hyperparameters during the single training run.
  • The method is evaluated with 3 algorithms (PPO, DQN, and SAC) in 14 environments, and compared against default hyperparameters, fixed LLM-suggested hyperparameters (without online adaptation), Bayesian optimisation, and population-based training.
  • The authors report that ALOHA matches or outperforms Bayesian optimisation on 9 of 14 benchmarks while requiring substantially fewer environment interactions.
Critical strengths
  • Broad empirical evaluation. The paper evaluates the method across several RL algorithms and environments, using multiple random seeds and standardised baselines. The inclusion of default LLM-suggested hyperparameters (no online adaptation) is particularly interesting.
  • Interesting idea of online adaptation based on training diagnostics.
Critical weaknesses
  • Empirical baselines: the LLM receives the environment name, which may allow it to exploit knowledge of standard benchmark environments acquired during pretraining. To isolate the contribution of online hyperparameter adaptation based on only training diagnostics, it would be necessary to see a baseline that does not pass in the environment name. The paper also only compares against offline hyperparameter optimisation methods, but not numerical online hyperparameter adaptation methods (a direct comparison to ALOHA's LLM based one).
  • Computational Cost analysis: the comparison in computational cost is only in terms of environment interactions. ALOHA repeatedly queries an LLM during training, so it would be necessary to also report LLM inference latency and end-to-end wall-clock training time.
Questions for authors
  • How does ALOHA perform when the environment name is not passed to the prompt?
  • How much do the hyperparameters vary for a single training run (for ALOHA) and between training runs for BO and PBT? It would be interesting to see plots showing how these vary/which parameters vary the most, and, for ALOHA, whether there are correlations with the training diagnostics.
  • How does ALOHA compare to online hyperparameter adaptation methods (online BO or online population methods)? [see, e.g. Parker Holder et al 2021, Provably Efficient Online Hyperparameter Optimisation with Population Based Bandits].
  • Please could the authors specify the exact baseline implementations (e.g. what acquisition function is used for BO), and clarify the parameters that are optimised by ALOHA and each baseline? (Section 3.1 lists only four optimised hyperparameters per algorithm, whereas Appendix B specifies a substantially larger search space).
Reviewer_HS2rOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes the use of LLMs for hyperparameter tuning in reinforcement learning. Instead of performing multiple trials with different hyperparameters, they propose having a single experiment and adjusting the hyperparameters within the run, without resetting. They test its effectiveness on 5 environments for PPO and DQN, 4 with SAC.

Critical strengths

This seems to be the first time an LLM is used to tune hyperparameters, on the fly, in reinforcement learning. Exploring this approach is useful, since many would want to know how well it works.

It is sensible to use the context given (statistics about learning over the run), and interesting to investigate the utility of extra context from papers.

Critical weaknesses

The paper currently is missing a key baseline and does not provide sufficient insight into the behavior of the LLM for tuning.

Misleading comparison around samples used: The biggest issue that online learning is done over 10M steps, which is quite a lot, without learning curves given. The primary conclusion is that the LLM can often match performance of offline tuners, but at a fraction of the cost because it tunes only during the online learning phase. However, if the online phase is long, and much longer than needed for the fixed hyperparameters, than this is not quite a correct claim. One way to immediately see this is by providing learning curves, which can let the reader see if many methods get to good performance within a small amount of that online phase.

Missing baseline(s): Additionally, another baseline could be included: a sequential Bayesian optimizer. Such an approach can also be run completely online, though it is common to the agent after each hyperparameter tested, unlike the LLM approach given here. This would give a more fair apples-to-apples comparison to other online tuners. Another online algorithm that is a valid baseline is to simply select random hyperparameters during the online phase, and ultimately stop at the best random one found so far. This is likely to perform poorly, but does provide a baseline. When comparing to such online tuners, it would be good also to include now the computational cost, because the LLM will likely be more expensive.

Insufficient/incorrect evidence and inconsistencies:

  • The variability is very high, and does not support the strong conclusions that are drawn.
  • The baseline result numbers (Default HP) are not convincing. For example, I see that in stablebaselines3, SAC can achieve 94.53 +/- 1.26 on MountainCarContinuous-V0, (https://huggingface.co/sb3/sac-MountainCarContinuous-v0), while the best number reported in the paper is 18.98 ± 40.01.
  • The hyperparameter search space in the appendix and main text do not match. For example, PPO has 8 hypers in appendix while it only has 4 in the main text (ln259). If this is true, then different baselines are searching over different spaces, which makes the comparison unfair.
  • The hyperparameter search space contains many lower impact, degenerate or useless settings (e.g very high learning rates, target networks=off, and so on), that could put the baselines in more of a disadvantage. This could be discussed.

Deeper investigation into the behavior of the LLM: This paper is useful if it gives a deep understanding of how the LLM tunes the hyperparameter online. Results should include some analysis of the hyperparameters selected during the run, both quantitative and qualitative. How did the stepsize change over time? What is it important that the LLM got to continually adapt the hypers, beyond just picking good static hypers? Did performance sometimes significantly degrade before good hypers were found? How did the LLM deal with hyperparameters that are only important at initialization?

Questions for authors

This issues with this paper can be addressed and I am willing to increase my score if the above concerns are addressed. I will highlight the two most important questions, though the others above should also be addressed in the discussion.

The most important question is: what do the learning curves look like and what is behavior like during the run. This will help better understand how many interventions are needed and how practical the LLM is for tuning.

It is also important to understand how a sequential optimizer would behave. I would suggest running both a sequential Bayesian optimizer that reset after each hyperparameter choice AND running the LLM tuner also with resetting, to see the difference.

Reviewer_o9QyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an agentic framework for online hyperparameter optimization of RL algorithms. The proposed method, ALOHA, uses an open-weight LLM to update the hyperparameters of an RL algorithm online given a history of performance metrics and their changes. On 14 environments across 3 RL algorithms in ARLBench, the authors compare ALOHA to SMAC and PBT and show that it matches or exceeds SMAC on 9 out of the 14 cases. Additionally, the authors ablate the use of RAG in their method by adding the ability to extract knowledge from an arXiv paper when adapting the hyperparameters.

Critical strengths
  • Paper is easy to follow.
  • ALOHA matches or exceeds performance of SMAC on 9 out of the 14 evaluation cases in ARLBench, giving evidence that an open-weight model such as gemma4-31B has satisfactory parametric knowledge to update the hyperparameters of RL algorithms online.
  • The authors compare to SMAC and PBT as standard baselines for HPO in RL.
Critical weaknesses
  • There is no reporting of computational costs of HPO runs. A figure similar to Figure 2 but with GPU time on the x-axis would be necessary.
  • PBT gets much less budget than SMAC. Increasing population size for PBT to match SMAC would level out the comparison.
  • Adding Random Search, SMAC+HB, and PB2 as baselines would further clarify the picture.
  • Adding CD diagrams of final performance of methods in terms of episodic return would help clarify the meaning of results.
  • Figure 1 does not match the method described in Section 2.2. Several elements of the figure contradict the text, which makes it unclear what was actually implemented. The text defines E, F, Ret and Train as the components, while the figure uses T, A, P, R and U. Further, the Figure mentions a sliding window of performance metrics, whereas the text describes a full history. The hyperparameter bounds in Figure 1 do not match those in Table 2 in Appendix B. I ask the authors to revise Figure 1 so that it clearly and accurately depicts the method as implemented.
  • Figure 1 depicts the retriever R retrieving a random arXiv paper, whereas Section 2.2 states that the retriever Ret "maps the current diagnostic window to relevant external context drawn from a single paper retrieved from arXiv at runtime". If the paper is random, the RAG ablation tests robustness to irrelevant information rather than the value of retrieval.
  • The paper repeatedly claims ALOHA operates at "zero tuning cost" and uses "400x less compute", but never counts LLM inference costs.
  • The authors should clarify in the text whether training is blocked during LLM calls, and report per-intervention latency.
  • Table 1 shows Hybrid RAG with a higher mean on 6 of 14 environments, which contradicts the claim in the Conclusion that Hybrid RAG outperforms zero-shot in only 2 of 14.
  • Table 1 and Table 5 disagree on HalfCheetah (7825.6 vs 7712.3).
  • Section 3.2 refers to "Table 5" when it means Table 1.
  • Figure 2's x-axis is trials, but Section 3.3 says environment steps.
Questions for authors
  • How do Random Search, SMAC+HB and PB2 perform using your evaluation protocol?
  • How do the methods perform when comparing episodic return vs GPU time?
  • What is the latency of ALOHA to make hyperparameter updates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.