The paper currently is missing a key baseline and does not provide sufficient insight into the behavior of the LLM for tuning.
Misleading comparison around samples used: The biggest issue that online learning is done over 10M steps, which is quite a lot, without learning curves given. The primary conclusion is that the LLM can often match performance of offline tuners, but at a fraction of the cost because it tunes only during the online learning phase. However, if the online phase is long, and much longer than needed for the fixed hyperparameters, than this is not quite a correct claim. One way to immediately see this is by providing learning curves, which can let the reader see if many methods get to good performance within a small amount of that online phase.
Missing baseline(s): Additionally, another baseline could be included: a sequential Bayesian optimizer. Such an approach can also be run completely online, though it is common to the agent after each hyperparameter tested, unlike the LLM approach given here. This would give a more fair apples-to-apples comparison to other online tuners. Another online algorithm that is a valid baseline is to simply select random hyperparameters during the online phase, and ultimately stop at the best random one found so far. This is likely to perform poorly, but does provide a baseline. When comparing to such online tuners, it would be good also to include now the computational cost, because the LLM will likely be more expensive.
Insufficient/incorrect evidence and inconsistencies:
- The variability is very high, and does not support the strong conclusions that are drawn.
- The baseline result numbers (Default HP) are not convincing. For example, I see that in stablebaselines3, SAC can achieve 94.53 +/- 1.26 on MountainCarContinuous-V0, (https://huggingface.co/sb3/sac-MountainCarContinuous-v0), while the best number reported in the paper is 18.98 ± 40.01.
- The hyperparameter search space in the appendix and main text do not match. For example, PPO has 8 hypers in appendix while it only has 4 in the main text (ln259). If this is true, then different baselines are searching over different spaces, which makes the comparison unfair.
- The hyperparameter search space contains many lower impact, degenerate or useless settings (e.g very high learning rates, target networks=off, and so on), that could put the baselines in more of a disadvantage. This could be discussed.
Deeper investigation into the behavior of the LLM: This paper is useful if it gives a deep understanding of how the LLM tunes the hyperparameter online. Results should include some analysis of the hyperparameters selected during the run, both quantitative and qualitative. How did the stepsize change over time? What is it important that the LLM got to continually adapt the hypers, beyond just picking good static hypers? Did performance sometimes significantly degrade before good hypers were found? How did the LLM deal with hyperparameters that are only important at initialization?