- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces DOODAT, a method for dynamically adapting the discount factor and trace-decay parameter within a single training run. The core idea is to maintain a central head and auxiliary heads corresponding to higher and lower values of each hyperparameter, defined through a neighbourhood function. All heads learn from a shared data pool, and the candidate policies are compared using a common surrogate based on the central critic. These comparisons guide hyperparameter updates, with the evaluation criterion changing as the central discount factor evolves. Overall, the paper is well written and easy to follow, and the approach is well motivated and addresses an interesting problem in online hyperparameter optimisation.
Critical strengths
The empirical results are solid. Key ones aside from the main results are the plots in Appendix J.2 (repository), showing the evolution of central parameter values over training, and Figure 20, which compares DOODAT with fixed values discovered by the method, showing that dynamic adaptation can provide gains beyond using the final parameter values from the start.
Critical weaknesses
The measurement of performance is based on the central critic, not on the returns observed from each candidate policy. As such, the comparison might favor optimistic overestimations. The authors partially address this concern in Appendix E, and I find it a natural challenge in meta-optimisation.
Another concern is the diversity of data collected by sampling from multiple heads. The authors also acknowledge that it plays a role in the results, as illustrated by Figure 5.
Questions for authors
Q1. Could the authors justify not testing the approach in vanilla DQN? While I’m not very familiar with PQN, PPO is designed to encourage conservative policy updates, which might affect the results. Moreover, it would be interesting to show whether the approach works with algorithms that do not rely on multiple epochs over a fixed batch. I don't see any constraint that would prevent adapting only the discount factor, and testing how the method interacts with replay memory and a target network would add credibility to its broader applicability.
2 comments
Dear authors,
Thank you very much for the clarifications and expanded results. In particular, I find the new results comparing DOODAT against population-based methods compelling, as well as the results showing its efficiency on top of DQN in the hard settings.
Overall, the first version of the paper already covers most of my concerns, and the new results reinforce the method's robustness. For that reason, and after seeing other reviewers' concerns, I believe my initial score reflects my assessment of the paper.
Thank you for the thoughtful and constructive feedback!
We agree that DQN is a valuable comparison, particularly since it is based on a replay buffer in contrast to PQN and PPO (this point has also been raised by reviewer JRHa). For DQN, DOODAT adapts only the discount factor, using a central value head and two neighbouring discount heads trained from shared replay. At each meta-update, independently sampled replay states are used to compare candidate policies. Meta-gradient computation is done similarly to PQN using the central critic head.
The results are presented in the following table (Figure 16 of the revised paper):
Both DOODAT and Meta-Gradients achieve comparable performance in the easy regime, with substantially overlapping confidence intervals. However, in the hard regime, DOODAT shows a strong ability to recover an IQM of 0.31, which corresponds to 60% of the easy baseline performance. In contrast, both meta-gradients and the baseline show no sign of learning for the hard initialisation.
To judge the performance of DOODAT in algorithms with single epochs, we have run PPO experiments on the DeepMind Control suite, using a single epoch and single full batch update per epoch. We appropriately adjusted hyperparameters from an A2C baseline. This comparison yields the following IQM performances (Figure 17 of the revised paper):
Overall, the performances are slightly worse compared to PPO. In the easy regime, both BMG and DOODAT outperform the baseline, with overlapping confidence intervals. In the hard regime, DOODAT is the only method that shows substantial recovery from hard hyperparameter initialisation. Overall, this shows the applicability of DOODAT also to non-epoch-based algorithms such as A2C.
Finally, we have run experiments of PPO with a single epoch, a single full-batch update per epoch and no trust-region clipping, making the algorithm comparable to A2C with generalised advantage estimates. This comparison yields the following IQM performance values (Figure 17 in the revised paper):
These performance values are very similar to those for PPO with a single epoch. However, DOODAT appears to recover less performance in the hard initialisation regime compared to PPO with a trust-region, recovering only 33% of the baseline performance compared to 46% when using trust-region clipping. While these results do not show a strong trend, it appears that DOODAT performs better in algorithms with trust regions, but overall generalises to non-epoch-based algorithms.