acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Optimisation of Discount and Trace Decay for Value-Based Reinforcement Learning

Abstract

Hyperparameter optimisation remains a central challenge in reinforcement learning (RL), particularly for the discount factor and trace-decay which directly influence objective estimation and the bias–variance trade-off in learning targets. Given the high computational cost of RL, it is desirable to dynamically adapt hyperparameters within a single training run, thereby avoiding the expense and sample inefficiency of hyperparameter optimisation over multiple independent runs. Recent approaches have explored dynamic hyperparameter adaptation during training; however, they often suffer from brittleness, limited applicability across algorithms, significant computational overhead, or reliance on costly policy evaluations. In this work, we propose a unified framework for Dynamical Optimisation Of Discount-factor And Trace-decay (DOODAT). DOODAT is simple, reliable, broadly applicable across algorithms, and incurs minimal computational and sample overhead. We evaluate our approach against state-of-the-art meta-gradient methods and standard non-adaptive baselines on PQN and PPO across multiple environment suites. DOODAT consistently improves performance and exhibits strong robustness. Notably, it recovers high performance even when initialised with suboptimal hyperparameters, far from commonly used values, achieving an IQM of whereas static training or baseline methods achieve no learning at all on the ALE.

Primary area:
reinforcement learning
Keywords:
AutoRL, AutoML, Hyperparameter Optimisation, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fehMOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces DOODAT, a method for dynamically adapting the discount factor and trace-decay parameter within a single training run. The core idea is to maintain a central head and auxiliary heads corresponding to higher and lower values of each hyperparameter, defined through a neighbourhood function. All heads learn from a shared data pool, and the candidate policies are compared using a common surrogate based on the central critic. These comparisons guide hyperparameter updates, with the evaluation criterion changing as the central discount factor evolves. Overall, the paper is well written and easy to follow, and the approach is well motivated and addresses an interesting problem in online hyperparameter optimisation.

Critical strengths

The empirical results are solid. Key ones aside from the main results are the plots in Appendix J.2 (repository), showing the evolution of central parameter values over training, and Figure 20, which compares DOODAT with fixed values discovered by the method, showing that dynamic adaptation can provide gains beyond using the final parameter values from the start.

Critical weaknesses

The measurement of performance is based on the central critic, not on the returns observed from each candidate policy. As such, the comparison might favor optimistic overestimations. The authors partially address this concern in Appendix E, and I find it a natural challenge in meta-optimisation.

Another concern is the diversity of data collected by sampling from multiple heads. The authors also acknowledge that it plays a role in the results, as illustrated by Figure 5.

Questions for authors

Q1. Could the authors justify not testing the approach in vanilla DQN? While I’m not very familiar with PQN, PPO is designed to encourage conservative policy updates, which might affect the results. Moreover, it would be interesting to show whether the approach works with algorithms that do not rely on multiple epochs over a fixed batch. I don't see any constraint that would prevent adapting only the discount factor, and testing how the method interacts with replay memory and a target network would add credibility to its broader applicability.

2 comments

Thank you for the thoughtful and constructive feedback!

Q1. Could the authors justify not testing the approach in vanilla DQN? [...] Moreover, it would be interesting to show whether the approach works with algorithms that do not rely on multiple epochs over a fixed batch.

We agree that DQN is a valuable comparison, particularly since it is based on a replay buffer in contrast to PQN and PPO (this point has also been raised by reviewer JRHa). For DQN, DOODAT adapts only the discount factor, using a central value head and two neighbouring discount heads trained from shared replay. At each meta-update, independently sampled replay states are used to compare candidate policies. Meta-gradient computation is done similarly to PQN using the central critic head.

The results are presented in the following table (Figure 16 of the revised paper):

SettingMethodScoreCI
Easy (γ=0.99\gamma = 0.99)Baseline0.4663[0.3470, 0.5760]
Easy (γ=0.99\gamma = 0.99)Meta-Gradients0.4924[0.4227, 0.5601]
Easy (γ=0.99\gamma = 0.99)DOODAT (Ours)0.4487[0.3656, 0.5383]
Hard (γ=0.5\gamma = 0.5)Baseline-0.0178[-0.0310, -0.0025]
Hard (γ=0.5\gamma = 0.5)Meta-Gradients-0.0213[-0.0321, 0.0004]
Hard (γ=0.5\gamma = 0.5)DOODAT (Ours)0.3106[0.2790, 0.3449]

Both DOODAT and Meta-Gradients achieve comparable performance in the easy regime, with substantially overlapping confidence intervals. However, in the hard regime, DOODAT shows a strong ability to recover an IQM of 0.31, which corresponds to 60% of the easy baseline performance. In contrast, both meta-gradients and the baseline show no sign of learning for the hard initialisation.


To judge the performance of DOODAT in algorithms with single epochs, we have run PPO experiments on the DeepMind Control suite, using a single epoch and single full batch update per epoch. We appropriately adjusted hyperparameters from an A2C baseline. This comparison yields the following IQM performances (Figure 17 of the revised paper):

SettingMethodScoreCI
Easy (γ=0.99\gamma = 0.99, λ=0.95\lambda = 0.95)Baseline0.3773[0.3300, 0.4273]
Easy (γ=0.99\gamma = 0.99, λ=0.95\lambda = 0.95)Meta-Gradients0.4527[0.4081, 0.4980]
Easy (γ=0.99\gamma = 0.99, λ=0.95\lambda = 0.95)DOODAT (Ours)0.4170[0.3757, 0.4603]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)Baseline0.0818[0.0734, 0.0929]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)BMG0.0795[0.0649, 0.0952]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)DOODAT (Ours)0.1720[0.1426, 0.2036]

Overall, the performances are slightly worse compared to PPO. In the easy regime, both BMG and DOODAT outperform the baseline, with overlapping confidence intervals. In the hard regime, DOODAT is the only method that shows substantial recovery from hard hyperparameter initialisation. Overall, this shows the applicability of DOODAT also to non-epoch-based algorithms such as A2C.

Finally, we have run experiments of PPO with a single epoch, a single full-batch update per epoch and no trust-region clipping, making the algorithm comparable to A2C with generalised advantage estimates. This comparison yields the following IQM performance values (Figure 17 in the revised paper):

SettingMethodScoreCI
Easy (γ=0.99\gamma = 0.99, λ=0.95\lambda = 0.95)Baseline0.3914[0.3381, 0.4462]
Easy (γ=0.99\gamma = 0.99, λ=0.95\lambda = 0.95)Meta-Gradients0.4411[0.4087, 0.4990]
Easy (γ=0.99\gamma = 0.99, λ=0.95\lambda = 0.95)DOODAT (Ours)0.4197[0.3663, 0.4716]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)Baseline0.0856[0.0705, 0.1031]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)BMG0.0795[0.0647, 0.0952]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)DOODAT (Ours)0.1276[0.1060, 0.1556]

These performance values are very similar to those for PPO with a single epoch. However, DOODAT appears to recover less performance in the hard initialisation regime compared to PPO with a trust-region, recovering only 33% of the baseline performance compared to 46% when using trust-region clipping. While these results do not show a strong trend, it appears that DOODAT performs better in algorithms with trust regions, but overall generalises to non-epoch-based algorithms.

Dear authors,

Thank you very much for the clarifications and expanded results. In particular, I find the new results comparing DOODAT against population-based methods compelling, as well as the results showing its efficiency on top of DQN in the hard settings.

Overall, the first version of the paper already covers most of my concerns, and the new results reinforce the method's robustness. For that reason, and after seeing other reviewers' concerns, I believe my initial score reflects my assessment of the paper.

Reviewer_JRHaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DOODAT adapts the discount factor and trace-decay parameter using auxiliary value heads trained with neighbouring hyperparameter settings. A shared central critic compares candidate policies and guides adaptation. Experiments with PQN on Atari and Procgen, and PPO on the DeepMind Control Suite, demonstrate improvements for PQN and recovery from poor initialisations, with more limited benefits for PPO.

===============

update1: raised score

Critical strengths

(S1: Substantial empirical evaluation) Improvements from standard PQN settings, alongside auxiliary-head ablations and comparisons against the final static configurations, provide meaningful evidence beyond recovery from poor initialisations.

(S2: Practical adaptation mechanism) The shared-head architecture avoids differentiating through optimisation trajectories. Its modest computational overhead and reported robustness to meta-hyperparameter choices make it potentially useful for reducing manual tuning.

(S3: robustness relative to BMG) Figure 5 shows substantially less sensitivity to the meta-learning rate, while Section 4.6 reports reuse of the meta-hyperparameters across training budgets and benchmarks. Following S2, this makes S2’s practical argument more concrete.

Critical weaknesses

(W1. Limited conceptual novelty) The proposed mechanism is coherent and computationally economical, but I find its conceptual advance over existing online hyperparameter adaptation relatively incremental. The case for acceptance therefore depends strongly on demonstrating empirical benefits beyond other adaptation strategies.

(W2. Practical search versus recovery) Recovery from implausible initialisations such as (γ=0.5\gamma=0.5, λ=0.1\lambda=0.1) does not clearly distinguish the effective search. On the other hand, testing plausible, pre-specified initialisations on previously untuned tasks (i.e. tasks where PQN has not a well-known discount and trace) would better reflect practical uncertainty and much more helpful in my opinion. That is, "searching" for well-established settings is less compelling than identifying effective settings for previously untuned tasks.

(W3. Persistent replay remains untested) Both evaluated algorithms learn from recent rollout batches, while PPO shows limited benefit over its standard configuration. The method’s effectiveness with persistent experience replay therefore remains untested. Since Figure 9 already evaluates PQN for discount-only adaptation, applying this variant to SAC or TD3 would be a plausible experimental extension.

(W4. Critic reliability) (not critical) In Appendix E, when the rollout oracle recommends a change, the critic selects the same direction in 35.0% and 32.1% of cases for γ\gamma and λ\lambda, respectively. To my understanding, disagreements often involve leaving the parameter unchanged, so these figures do not necessarily indicate harmful updates. Nevertheless, the claim of reliable selection needs a clearer distinction between "missed improvements" and "updates in the wrong direction".

Questions for authors

I would consider raising my score significantly if the questions are suitable responded. Q is for questions, C is for comments.

(Q1. Addresses W2) Does the proposed method works well on tuning the proposed hyperparameters for unknown tasks?

(Q2. Adresses W3) Does adaptation remain effective with persistent replay, i.e. when adapting discount for SAC or TD3?

(Q3: Addresses W4) When the critic disagrees with the rollout-based reference, how often does it leave the hyperparameter unchanged, and how often does it recommend the opposite direction? I think a confusion matrix of increase, decrease and no-change decisions would help distinguish conservative behaviour from potentially detrimental updates.

(Q4. Spacing ablation) Beyond the spacing ablation in Figure 9, could additional heads at different distances improve search? For example, raise the heads to 10 neighboring heads. Since the computational cost is small, adding more heads could be beneficial and searches faster?

(C1) Please reconcile the trace-decay labels for the standard (“easy”) PPO configuration: Figure 16 gives λ=0.65\lambda=0.65, whereas Figure 6 and Appendix K.1 give λ=0.95\lambda=0.95.

(C2) Please include representative PQN trajectories of γ\gamma and λ\lambda from standard initial settings, alongside the corresponding learning curves, within the paper. Although the plots referenced in Appendix J.2 are available in the anonymous repository, an example in the paper would make the practical behaviour of adaptation easier to assess.

6 comments

Thank you for the thoughtful and constructive feedback!

(W1. Limited conceptual novelty) The proposed mechanism is coherent and computationally economical, but I find its conceptual advance over existing online hyperparameter adaptation relatively incremental. The case for acceptance therefore depends strongly on demonstrating empirical benefits beyond other adaptation strategies.

Thank you for bringing up this point. We believe that DOODAT goes beyond an incremental improvement and yields a structural improvement over existing methods. We have described this further in Appendix A of the paper. Particularly, DOODAT is efficiently applicable to a broad range of algorithms, including epoch-based algorithms like PPO or PQN. Existing meta-gradient methods cannot be applied effectively to epoch-based algorithms. Methods that are applicable in the literature are either computationally expensive or not applicable to the discount and trace-decay (such as AdaQ). However, please let us know if there is a particular method you are thinking of, and we will be happy to add a comparison to the other methods presented in Appendix A.

(Q1. Addresses W2) Recovery from implausible initialisations such as (gamma 0.5 and lambda 0.65) does not clearly distinguish the effective search. Does the proposed method works well on tuning the proposed hyperparameters for unknown tasks?

Thank you for clarifying this distinction. We agree that recovery from poor initialisations alone does not demonstrate practical hyperparameter search. Our Procgen experiments address this more directly: we transferred the initial PQN discount and trace-decay settings from ALE (γ=0.99\gamma=0.99, λ=0.65\lambda=0.65), without tuning them specifically for Procgen, while adopting the IMPALA architecture. We are not aware of any existing tuned baseline for Procgen with PQN. Particularly, the IMPALA PPO baseline uses a discount of 0.999 and trace-decay of 0.95. From the plausible initialisation γ=0.99\gamma=0.99, λ=0.65\lambda=0.65, DOODAT improves IQM from 0.29 to 0.36. This provides evidence that DOODAT can improve transferred settings on a different task suite, beyond recovering from deliberately poor initialisations. We have highlighted this now in Section 4.5 of the revised paper.

Further, we have now run DOODAT on the Agentick benchmark (Castanyer et al. (2026) “Agentick: A Unified Benchmark for General Sequential Decision-Making Agents”). This is a novel benchmark covering diverse tasks, to be published at NeurIPS this year, for which no tuned PQN baseline is available to the best of our knowledge. Agentick comprises 37 procedurally generated gridworld tasks covering navigation, planning, reasoning, memory, generalisation, and multi-agent coordination. We are applying PQN to this environment similarly to the ALE, with no specific tuning. Performance is measured by success rate on held-out evaluation layouts and aggregated using the task-balanced arithmetic mean similar to Castanyer et al. (2026), giving each of the 37 tasks equal weight. The results of the full Agentick benchmark are given in Figure 14 in the revised paper and further in the following table:

SettingMethodScoreCI
Easy (γ=0.99\gamma = 0.99)Baseline0.19[0.18, 0.21]
Easy (γ=0.99\gamma = 0.99)BMG0.20[0.18, 0.21]
Easy (γ=0.99\gamma = 0.99)DOODAT (Ours)0.23[0.22, 0.24]
Hard (γ=0.5\gamma = 0.5)Baseline0.11[0.11, 0.12]
Hard (γ=0.5\gamma = 0.5)BMG0.11[0.10, 0.12]
Hard (γ=0.5\gamma = 0.5)DOODAT (Ours)0.17[0.16, 0.18]

Overall, the results are in a similar ballpark to the medium-difficulty task-balanced PPO success of 0.2770.277 reported by Castanyer et al. (2026) in Table 2, although lower. Furthermore, similarly to PPO in Castanyer et al. (2026), there is a long tail of environments where no learning is possible, as can be seen in Figure 14 of the revised paper.

In the easy initialisation regime, DOODAT outperforms both static PQN and BMG, with a task-balanced mean success rate of 0.2270.227 compared to 0.1940.194 and 0.1960.196, respectively. In the hard initialisation regime, static PQN and BMG achieve 0.1120.112 and 0.1100.110, respectively. DOODAT improves performance to 0.1690.169, substantially narrowing the gap to the easy-initialisation baseline. Together, these results provide evidence that the benefits of DOODAT extend beyond the benchmarks on which it was developed, improving performance under both initialisations without benchmark-specific tuning.

(Q2. Adresses W3) Does adaptation remain effective with persistent replay, i.e. when adapting discount for SAC or TD3?

We selected to test DOODAT with DQN as an algorithm with persistent replay, due to it being a more common baseline for continuous hyperparameter adaptation in the literature (e.g. Vincent et al. (2025) “Adaptive Q-Network: On-The-Fly Target Selection for Deep Reinforcement Learning”, Xu et al. (2018) “Meta-Gradient Reinforcement Learning”) and reviewer fehM also being interested in DQN.

For DQN, DOODAT adapts only the discount factor, using a central value head and two neighbouring discount heads trained from shared replay. At each meta-update, independently sampled replay states are used to compare candidate policies. Meta-gradient computation is done similarly to PQN using the central critic head.

The results are given in the following table (additionally Figure 16 in the revised paper):

SettingMethodScoreCI
Easy (γ=0.99\gamma = 0.99)Baseline0.46[0.34, 0.57]
Easy (γ=0.99\gamma = 0.99)Meta-Gradients0.49[0.42, 0.56]
Easy (γ=0.99\gamma = 0.99)DOODAT (Ours)0.44[0.36, 0.53]
Hard (γ=0.5\gamma = 0.5)Baseline-0.0178[-0.031, -0.0025]
Hard (γ=0.5\gamma = 0.5)Meta-Gradients-0.02[-0.032, 0.0004]
Hard (γ=0.5\gamma = 0.5)DOODAT (Ours)0.31[0.27, 0.34]

Both DOODAT and Meta-Gradients achieve comparable performance in the easy regime, with substantially overlapping confidence intervals. However, in the hard regime, DOODAT shows a strong ability to recover an IQM of 0.31, which corresponds to 60% of the easy baseline performance. In contrast, both meta-gradients and the baseline show no sign of learning for the hard initialisation.

(Q3: Addresses W4) When the critic disagrees with the rollout-based reference, how often does it leave the hyperparameter unchanged, and how often does it recommend the opposite direction? I think a confusion matrix of increase, decrease and no-change decisions would help distinguish conservative behaviour from potentially detrimental updates.

Thank you for pointing this out. We believe as well that confusion matrices will better help the reader to judge the proxy accuracy. Please find the respective confusion matrices below and in Table 4 of the revised paper.

γ\gamma: DOODAT \ Oracle+10-1λ\lambda: DOODAT \ Oracle+10-1
+19.0%0.9%4.7%+18.3%1.1%3.5%
014.0%37.6%17.3%015.0%37.4%20.1%
-13.0%1.5%12.0%-12.1%1.5%11.0%

We have also added clear definitions of the metrics in Table 3 together with the confusion matrices:

Gradient-direction agreement: C1,1+C0,0+C−1,−1C_{1,1} + C_{0,0} + C_{-1,-1}

Non-zero update accuracy: C1,1+C−1,−1/(C1,1+C0,1+C−1,1+C1,−1+C0,−1+C−1,−1)C_{1,1} + C_{-1,-1} / (C_{1,1} + C_{0,1} + C_{-1,1} + C_{1,-1} + C_{0,-1} + C_{-1,-1})

Opposite-direction rate: C−1,1+C1,−1C_{-1,1} + C_{1,-1}

Non-harmful decisions: C0,1+C0,0+C0,−1+C1,1+C−1,−1+C1,0+C−1,0C_{0,1} + C_{0,0} + C_{0,-1} + C_{1,1} + C_{-1,-1} + C_{1,0} + C_{-1,0}

Not considering C1,0C_{1,0} and C−1,0C_{-1,0} in the non-harmful decision updates, this is the case where DOODAT suggests a change in discount or trace decay while the oracle does not; this would still leave non-harmful decision accuracy at 89.4% for discount and 91.8%.

(Q4. Spacing ablation) Beyond the spacing ablation in Figure 9, could additional heads at different distances improve search? For example, raise the heads to 10 neighboring heads. Since the computational cost is small, adding more heads could be beneficial and searches faster?

Thank you for the suggestion! We have now conducted such a spacing ablation. We have added 4 additional auxiliary heads by considering the perturbed neighbours χ+2←Nχ(Nχ−1(χ)+2)\chi_{+2} \leftarrow N_\chi(N_\chi^{-1}(\chi) + 2) and χ−2←Nχ(Nχ−1(χ)−2)\chi_{-2} \leftarrow N_\chi(N_\chi^{-1}(\chi) - 2) for both hyperparameters. The meta-gradient is computed by selecting the best performing head in each update similarly to the previous update. This increases the overall number of auxiliary heads from 4 to 8.

The results are summarised in the following table (Figure 24 in the revised paper Appendix M.4):

SettingMethodScoreCI
Easy (γ=0.99\gamma = 0.99, λ=0.65\lambda = 0.65)Baseline1.6518[1.5229, 1.7798]
Easy (γ=0.99\gamma = 0.99, λ=0.65\lambda = 0.65)DOODAT (4 aux. heads)2.0821[1.9909, 2.1646]
Easy (γ=0.99\gamma = 0.99, λ=0.65\lambda = 0.65)DOODAT (8 aux. heads)2.1314[2.0723, 2.1915]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)Baseline0.0338[0.0125, 0.0591]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)DOODAT (4 aux. heads)0.8890[0.8233, 0.9559]
Hard (γ=0.5\gamma = 0.5, λ=0.1\lambda = 0.1)DOODAT (8 aux. heads)1.4817[1.3436, 1.6371]

In the easy regime, the additional auxiliary heads give only a marginal additional performance over DOODAT with 4 heads. However, in the hard regime, the additional heads substantially increase performance, almost achieving baseline performance in the easy regime. Examining the learning curves shows very similar hyperparameter adjustment behaviour in the easy regime for DOODAT with 4 and 8 auxiliary heads. In the hard regime, however, the additional auxiliary heads enable overcoming local optima in the hyperparameter values, where DOODAT with only four auxiliary heads is getting stuck (e.g. γ≈0.8\gamma \approx 0.8 for Amidar and Frostbite). The improvement of performance does not appear to change the speed of adjustment in the hyperparameters. We therefore conclude that, while improvement in the easy regime remains limited, it can substantially help particularly in harder optimisation regimes for overcoming local optima in hyperparameter values. We have added the individual learning curves for the environments to our anonymous repository.

We have added a corresponding discussion in Appendix M.4 of the revised paper.

(C1) Please reconcile the trace-decay labels for the standard (“easy”) PPO configuration: Figure 16 gives , whereas Figure 6 and Appendix K.1 give .

Thank you for pointing this out! We have corrected this in the revised paper. The correct trace-decay parameter for PPO is 0.95 in the easy regime.

(C2) Please include representative PQN trajectories of discount and lambda from standard initial settings, alongside the corresponding learning curves, within the paper. Although the plots referenced in Appendix J.2 are available in the anonymous repository, an example in the paper would make the practical behaviour of adaptation easier to assess.

We agree that this would increase the readability of the paper. We have included the mentioned learning curves now in Figure 21 of the revised paper.

Reviewer_6je8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper proposes a method to adapt the discount factor and the trace decay hyperparameters during the run of RL algorithms. The proposed method, Dynamical Optimisation Of Discount-factor and Trace-decay (DOODAT), augments the value functions and policies with additional heads trained with higher and lower values of discount factors and trace decay. DOODAT evaluates the policies induced by the perturbed heads under the central head and moves the hyperparameters by a finite-difference step towards the best neighbour. The paper evaluates DOODAT, BMG as a meta-gradient baseline and a static baseline with fixed discount factor and trace decay. These methods are evaluated for the PQN and PPO algorithms in the ALE, Procgen and DMC sets of environments. DOODAT outperforms the baseline methods especially in the hard regime, where initialization of the hyperparameters is far from conventional values.

Critical strengths
  • Paper is easy to follow.
  • DOODAT improves performance of PQN and PPO especially when initialized with suboptimal hyperparameters in the hard regime.
  • DOODAT has low computational overhead.
  • The authors empirically justify the components of DOODAT through ablations.
  • The method is clearly described, and Algorithms are provided on how DOODAT is integrated into PQN and PPO.
Critical weaknesses
  • The central critic as a proxy is weaker than the main text claims (Section 4.3: "the central critic can reliably select the best-performing value head"). According to Appendix E, DOODAT seems to work mostly because wrong-direction updates are rare (7.7% for discount factor and 5.6% for trace decay), not because signal is reliable (Kendall tau 0.247 and 0.195).

  • Random Search, Grid Search, PBT and PB2 are missing from the results to properly judge adaptation quality. Random Search and Grid Search would show whether good static values for these hyperparameters are sufficient. Grid Search's best per-environment configuration in Appendix K.1 shows an IQM of 0.61 vs 0.56 for DOODAT on DMC easy. Additionally, the analysis in Appendix J.5 shows that for the hard regime in the Atari-10 benchmark, the static final DOODAT hyperparameter configuration performs on par with DOODAT. If the final static configuration matches DOODAT in the hard regime, the gain comes from finding a good configuration, which is exactly what a search baseline would test.

    PBT and PB2 are other dynamic hyperparameter adaptation methods that need no modification to be run, in contrast to BMG.

    I acknowledge that these methods are considerably more expensive. The purpose is to contextualize the quality of DOODAT's adaptation, not compete on cost. A plot or table of IQM against total compute would make the trade-off visible.

Questions for authors
  • How do you reconcile the claim in Section 4.3 that DOODAT's central critic can "reliably select the best-performing value head" with the results in Appendix E?
  • What is the performance of Random Search, Grid Search, PBT, and PB2 for optimizing the discount factor and trace decay for PQN and PPO compared to DOODAT and BMG?
3 comments

Thank you for the thoughtful and constructive feedback!

The central critic as a proxy is weaker than the main text claims (Section 4.3: "the central critic can reliably select the best-performing value head").

Thank you for pointing out this inconsistency! We agree that “reliably select the best-performing value head” overstates the accuracy of individual comparisons.

To better clarify the distinction, we have now included full conclusion matrices in Appendix E (Table 4) of the revised paper. The matrices are:

γ\gamma: DOODAT \ Oracle+10-1λ\lambda: DOODAT \ Oracle+10-1
+19.0%0.9%4.7%+18.3%1.1%3.5%
014.0%37.6%17.3%015.0%37.4%20.1%
-13.0%1.5%12.0%-12.1%1.5%11.0%

The confusion matrices clarify the distinction between missed improvements and opposite-direction updates. Among disagreements, 75.6% for γ\gamma and 81.1% for λ\lambda correspond to the critic leaving the parameter unchanged when the rollout reference recommends a change. Furthermore, when both recommend a non-zero update, their directions agree in 73.2% and 77.5% of cases, respectively. Thus, the signal exhibits directional agreement beyond merely avoiding updates, although it misses many rollout-preferred changes.

As described in Section 3.3, DOODAT accumulates local updates whose magnitudes depend on estimated performance differences. Accurate selection at every individual step is therefore not a prerequisite for effective adaptation. Further evidence supports this interpretation: Section 4.4 shows that retaining the auxiliary heads and their data collection while disabling hyperparameter adaptation substantially reduces performance.

We have now described the critic as a noisy local comparison signal in Section 4.3 of the revised paper and explicitly mentioned frequently missed improvements for better transparency.

Random Search, Grid Search, PBT and PB2 are missing from the results to properly judge adaptation quality.

We agree that these experiments are relevant as a search baseline. We have now run a random search and PB2 tuning on the Atari-5 environments (NameThisGame, BattleZone, Phoenix, Qbert and DoubleDunk). Atari-5 is the subset of 5 environments that is most predictive for the full ALE suite (Aitchison et al. (2022) “Atari-5: Distilling the Arcade Learning Environment down to Five Games”). We choose random search over a grid-search for the static baseline, since random search is known to perform better with limited computational resources (Bergstra, J., & Bengio, Y. (2012). “Random Search for Hyper-Parameter Optimization.”). Random search is performed using 10 sampled configurations with the discount sampled logarithmically from the interval [0.9, 0.9999] and trace decay uniformly from [0.4, 0.8]. Each sampled configuration is trained for 5 seeds. We are choosing PB2 over PBT, since it has been shown to perform better under small computational resources as well (Parker-Holder J. & Nguyen V. (2021) “Tuning Mixed Input Hyperparameters on the Fly for Efficient Population Based AutoRL”). Following the population size and relative adaptation frequency used in that work, we used 4 members and 20 training intervals, scaled to 400M environment steps per member. We used the same search spaces for discount and trace decay with PB2 as in the random search.

We have reported the random search and PB2 comparison in Appendix K of the revised paper (Figure 15). From the easy initialisation regime, DOODAT achieves strong performance at substantially smaller sample budgets than the multi-run search baselines. After 450M environment frames, its pooled IQM recovers 92.1% of the best static tuning performance found across all ten random configurations. It exceeds the PB2 performance at the same total budget by 29.0%. After 1.125B environment frames, DOODAT exceeds the best random search performance by 2.0% and the PB2 performance by 9.3%. On Phoenix and Qbert, both BMG and DOODAT achieve substantially higher performance after 1.125B environment frames than random search and PB2 after their full optimisation. From the hard initialisation regime, DOODAT recovers 27.6% and 58.7% of the best static tuning performance at these respective budgets, despite starting substantially outside the informed search region. Please note that we have only conducted a single random search over the Atari-5 and therefore can not give statistical uncertainties over the reported values.

We have now updated our rebuttal and revised paper with the missing PB2 tuning runs. We have edited our previous answer accordingly.

Thank you for the detailed response and the additional experiments.

The revised description of the central critic in Section 4.3, together with the confusion matrices in Appendix E, resolves my concern about the overstated reliability claim.

The random search and PB2 comparison on Atari-5 helps contextualise the quality of DOODAT’s adaptation, and I appreciate the effort given the computational cost.

I have increased my soundness score and keep my overall rating.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

Thank you for your engagement with our rebuttal and your constructive feedback on our paper. We briefly summarise the revisions and discussion below:

  • 6je8: We added random-search and PB2 comparisons on Atari-5 to contextualise performance against total sample budgets, alongside new confusion matrices to clarify critic reliability. The reviewer considers their concern about overstated reliability resolved, appreciates the additional comparisons, and has increased their soundness score.
  • JRHa: We added the full 37-task Agentick evaluation, demonstrating improvements in a setting without established PQN configurations, as well as replay-based DQN experiments and an additional-head ablation. The reviewer raised their recommendation to weak acceptance, explicitly stating that their novelty concern was resolved and welcoming the new results.
  • fehM: We added DQN and single-update PPO variants, including a variant without clipping, to assess applicability beyond the original setting. On DQN, DOODAT achieves comparable performance under standard initialisation while recovering learning from poor initialisation where the baselines fail. The reviewer describes the additional PB2 comparisons as compelling and reaffirmed their acceptance recommendation, noting that the new results reinforce the method’s robustness.

All three reviewers currently recommend weak acceptance and have provided positive follow-up feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.