- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper attempts to address the issue in multi-task reinforcement learning (MTRL) where tasks are uniformly sampled from. In particular, this seems to harm the performance on the hard tasks. This submission proposes Dynamics for Adaptive Task Sampling (DynATS) that attempts to capture the inter-task relationship between tasks and models an updated sampling ratio between the different tasks. The DynATS method is trained via an evolutionary strategy to optimize the sampling ratios between tasks. Results seem to indicate that DynATS matches, if not slightly outperforms, applicable baselines methods.
Critical strengths
The method seems to be easily trainable, and didn't add nearly as much computational overload during MTRL as I expected.
Critical weaknesses
- The DynATS method only really outperforms DRATS in a single setting (XLand), but it is not clear (a) how DRATS' hyperparameters were tuned, or (b) whether the same computational budget was allocated to DRATS for tuning hyperparameters (XLand's DynATS model was trained with 65 billion interactions. DRATS needs to be tuned within a similar computational budget).
This is consequential because if we are going to say that Algorithm A is better than Algorithm B, then we need to ensure that we give Algorithm B the same set of circumstances (in this case, hyperparamater tuning budget)
- The main text mentions that "... the appropriate interpretation of the same signal can depend on the task set and its learning dynamics" in regards to what information should be used to choose the sampling ratios between tasks. However, when looking at what the DynATS method has learned, it's put all of the weight on the performance of the tasks, contradicting the motivating statement for the method.
This statement in the submission seems to suggest that performance by having additional inputs to the DynATS model outside of just performance, there will be better performance. However, the performance of DynATS only matches DRATS (except in XLand, as addressed above). Therefore, I don't particularly see the benefit of DynATS.
- Finally, I have some reservations about the generalization claims made in this work. Seeing as how training DynATS seemed to take 25 billion (grid worlds) or 65 billion (XLand) environmental interactions to train the base model (on top of once we have some new ratio from the DynATS model we then need to do RL training on top of that), I don't see how we could expand on the settings where DynATS is useful without an extremely large computational budget for some new problem setting. At that point I would imagine it would be more efficient to use DRATS because, while maybe needing some hyperparameter tuning, it does not need 25-65 billion interactions to find a better performing RL agent.
Questions for authors
-
What are the actual magnitudes of the inputs to the DynATS method? Is it possible that the performance inputs dominate the non-performance inputs? e.g. what's the scale of the relative performance input and the alloc input?
-
Did the authors consider any biologically inspired evolutionary algorithms (e.g. Particle Swarm Optimization) instead of ES?
Miscellaneous:
- Formula 3 should probably have epsilon as a subscript of the expectation. Right now it looks to be in-line.
- The specialized architectures of (Yang et al., 2020; Sodhani et al., 2021; Sun et al., 2022; Hendawy et al., 2024) were found to perform well due to the size of their networks by McLean et al, 2025b.
- The Meta-World reference should probably be McLean et al, 2025a.
Citations
-
Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, KR Zentner, Ryan Julian, Jordan K Terry, Isaac Woungang, et al. Meta-world+: An improved, standardized, rl benchmark. arXiv preprint arXiv:2505.11289, 2025a.
-
Reginald McLean, Evangelos Chatzaroulas, Jordan Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. Multi-task reinforcement learning enables parameter scaling. arXiv preprint arXiv:2503.05126, 2025b.
8 comments
Response to Reviewer T8EH (2/2)
Finally, I have some reservations about the generalization claims made in this work. Seeing as how training DynATS seemed to take 25 billion (grid worlds) or 65 billion (XLand) environmental interactions to train the base model (on top of once we have some new ratio from the DynATS model we then need to do RL training on top of that), I don't see how we could expand on the settings where DynATS is useful without an extremely large computational budget for some new problem setting.
One of the main claims of our work is that DynATS can be trained on simple, inexpensive domains and deployed zero-shot on complex, expensive one (e.g. Gridworlds -> Meta-World). Moreover, our distillation results now included in the revised paper in §5.7 demonstrate that it is also possible to learn a (more) general sampler that can be deployed zero-shot on diverse domains that may require different prioritization strategies (see GridWorlds and XLand results in Figure 5). This could be expanded in subsequent work by creating even more diverse (but cheap) training distributions that would allow the training of a general sampler. This has recently been done in the field of learned optimization, where a learned optimizer is trained on simple and small computer vision problems, while still outperforming finetuned optimizers on LLM pretraining runs [4] by improving the diversity of the training distribution and meta-learning process.
Moreover, while requiring environmental interactions, ES-training on Gridworlds corresponds to only roughly hour of runtime (see §C.9.1). The DynATS sampler can then be immediately deployed without finetuning on more complex domains.
What are the actual magnitudes of the inputs to the DynATS method? Is it possible that the performance inputs dominate the non-performance inputs? e.g. what's the scale of the relative performance input and the alloc input?
The table reports the observed input magnitudes on all held-out sets from GridWorlds and XLand. The relative performance features have a wider spread than the allocation features, as expected from standardization across tasks (Appendix B.1). In general, the allocation and performance input features have comparable scales.
| Input | GridWorlds median [p05, p95] | XLand median [p05, p95] |
|---|---|---|
| Relative Allocation | −0.11 [−0.93, 0.65] | −0.10 [−0.94, 0.70] |
| Relative Cumulative Allocation | 0.93 [0.47, 1.76] | 0.78 [0.44, 2.16] |
| Performance | 0.81 [0.30, 1.00] | 0.47 [0.00, 0.91] |
| Performance Change | 0.00 [−0.27, 0.28] | 0.00 [−0.16, 0.18] |
| Relative Performance | 0.19 [−1.76, 1.39] | 0.15 [−1.79, 1.53] |
| Relative Performance Change | 0.00 [−1.70, 1.69] | 0.00 [−1.70, 1.73] |
Did the authors consider any biologically inspired evolutionary algorithms (e.g. Particle Swarm Optimization) instead of ES?
We considered other biologically inspired evolutionary algorithms. In the end, we used ES, because they are significantly well-explored in learned optimization for Supervised [4] and Reinforcement Learning [5] and in meta-learning [6].
Formula 3 should probably have epsilon as a subscript of the expectation. Right now it looks to be in-line.
Thank you for noticing, it's updated in the revised paper.
The specialized architectures of (Yang et al., 2020; Sodhani et al., 2021; Sun et al., 2022; Hendawy et al., 2024) were found to perform well due to the size of their networks by McLean et al, 2025b.
We have added a small sentence to the first paragraph of our Related Work section (Section 2) discussing the referenced work.
The Meta-World reference should probably be McLean et al, 2025a.
That's correct, we have updated the reference in the revised paper.
Citations:
[1] Corrado, Nicholas E., Wenyuan Huang, and Josiah P. Hanna. "Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling." arXiv preprint arXiv:2605.14350 (2026).
[2] Cho, Myungsik, et al. "Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling." ICML. 2024.
[3] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.
[4] Moudgil, Abhinav, Boris Knyazev, and Eugene Belilovsky. "Celo2: Towards Learned Optimization Free Lunch." arXiv preprint arXiv:2602.19142 (2026).
[5] Goldie, Alexander D., et al. "Can learned optimization make reinforcement learning less difficult?." Advances in Neural Information Processing Systems 37 (2024): 5454-5497.
[6] Jackson, Matthew T., et al. "Discovering temporally-aware reinforcement learning algorithms." International Conference on Learning Representations. Vol. 2024. 2024.
DynATS significantly outperforms DRATS in the Synthetic, Skillchain and XLand benchmarks, while remaining competitive in the Gridworlds and continuous control benchmarks.
In Figure 2(b), the synthetic benchmark shows that DRATS and DynATS end with the same mean normalized return, so I cannot agree that it significantly outperforms DRATS. I cannot find the Skillchain results trained with DRATS, if you could kindly point to where they are I would appreciate it.
we added §C.4 and Figure 11, which shows that, for the XLand domain
Figure 11 compares a dotted line with no confidence intervals, to a line with confidence intervals. Without confidence intervals (and statistical testing) this does not show one algorithm outperforming another.
we do not claim our method to be superior to DRATS, but different
No, the submission does make that claim. From the abstract "DynATS outperforms existing sampling strategies" which states superiority about performance.
If the authors would like to make their claims statistically significant, showing that DynATS is statistically significantly better than DRATS/other methods, then I might be able to agree with claims of outperformance.
while not on the same scale as one meta-training run
Well, it's not tuned across 25 or 65 billion interactions, so I don't think I am open to this being a fair comparison. Does 25-65 billion interactions account for the amount of time tuning the hyperparameters of the transformer itself?
not because they are strictly needed for all benchmarks, but because they might be important for some.
They turned out to be important in one single case, XLand. So yes, the vast majority of the evidence in this submission contradicts the reasoning for the DynATS method. "They might be important for some" but the submission doesn't contain many examples of support that the non-performance based inputs to DynATS are providing any meaningful additions to the method, except for one example of XLand.
inexpensive domains and deployed zero-shot on complex, expensive one
The submission has evidence that clearly demonstrates that DynATS outperforms DRATS in one setting, XLand. Can it be deployed zero-shot in XLand? Absolutely. Can it be deployed zero-shot elsewhere? The submission does not support that conclusion.
My point about 25 billion (grid worlds) or 65 billion (XLand) interactions stems from the fact that the submission does not support this generalization claim. The submission has evidence that deployment zero-shot to XLand works, but that is not enough to support that this will work elsewhere. At best DynATS matches DRATS, with DynATS needing billions of interactions to be initially trained.
environmental interactions, ES-training on Gridworlds corresponds to only roughly hour of runtime
It's not about the runtime. It's about how many interactions are needed to train this model, and the limits of training on this set of gridworlds. The submission has support for only a single zero-shot generalization to XLand.
If it was about the runtime, I would make this argument. When, not if, DynATS does not perform well in a zero-shot deployment and needs to be re-trained on something that isn't a gridworld, DynATS is going to be extremely computationally expensive. Imagine if there was signal that training DynATS on Meta-World tasks was the fix instead of training on gridworlds. Each Meta-World experiment takes 6-8 hours on MT10 (dependent on hardware and hyperparameters), and 25 billion interactions is going to be one thousand two hundred fifty experiments on Meta-World alone (at twenty million total steps per experiment). Close to ten thousand hours of training DynATS on Meta-World. The method doesn't really scale well if I need to spend something like ten thousand hours training the transformer, if I could just spend that time running the RL algorihtm on the problem that I care about and tune the hyperparameters, or apply DRATS and tune it's parameters, within the same amount of time.
Response to Reviewer T8EH (1/2)
We thank the reviewer for their time and effort in reviewing our work.
DynATS significantly outperforms DRATS in the Synthetic, Skillchain and XLand benchmarks, while remaining competitive in the Gridworlds and continuous control benchmarks.
Table 8 in Appendix E.3 specifies how hyperparameters were tuned for all baselines on our experimental results (except the continual control experiments, where we reused the hyperparameters from Corrado et al. [1]). For all baselines, there are two hyperparameters: η and α. As in Corrado et al. [1], we set α=η/2 and sweep only over η. The baselines were finetuned for each individual set of tasks, using 100 PPO seeds for each value of η.
We argue that DRATS' underperformance on the XLand domain is not due to insufficient hyperparameter tuning, but instead due to a mismatch between its prioritization strategy and the sampling strategy that benefits XLand. In other words, a sampling strategy that prioritizes hard tasks is not appropriate for XLand (and neither for the Synthetic and Skillchain benchmarks). To support this claim, we added §C.4 and Figure 11, which shows that, for the XLand domain, as we increase the value of η, thereby making DRATS more aggressive in prioritizing hard tasks, performance monotonically degrades. In fact, the best performing hyperparameter was η=0.25, which corresponds to the weakest possible hard-task prioritization value we consider, since η=0 would correspond to uniform sampling. This demonstrates that DRATS' underperformance is not due to insufficient hyperparameter tuning, but to a mismatch in prioritization strategy. We also want to clarify that we do not claim our method to be superior to DRATS, but different. DRATS defines a priori what makes a task worth sampling, which works extremelly well across many MTRL benchmarks (e.g. Mujoco6 and Meta-World 10/50). Our method simply addresses the fact that different benchmarks may benefit from different strategies, and that this can be learned.
Nonetheless, for Gridworlds (a benchmark on which DRATS is well-suited), hyperparameter tuning is important. For a fairer comparison, we have now increased the range of our sweep to η={0.25,0.5,1,2,4,8,16,32} for the Gridworlds and XLand benchmarks. We believe that the DRATS tuning budget, while not on the same scale as one meta-training run, is comprehensive enough to warrant a fair comparison. Moreover, this is as large of a hyperparameter search as is reasonable for more expensive benchmarks. In the revised paper, we have therefore updated Figure 3. The results have not changed significantly on neither domain. The main difference is that DRATS on XLand now performs equivalently to uniform sampling (due to choosing η=0.25) for most sets of tasks.
We do not believe our statement to be contradicted by verifying that per-task performance is the most important input feature for learning a sampling strategy. Most sampling strategies (e.g. DRATS [1], Hard-Tasks-First [2], Learning Potential [3]) rely on performance metrics to determine what makes a task worth sampling (and they all converge to different, sometimes even opposing, interpretations!). As we show in our theoretical motivation in §4.1, the same metric can be interpreted differently. DynATS learns how it should be interpreted and converted into a sampling strategy based on inter-task dynamics. As we mention in §5.6, we include extra training signals, not because they are strictly needed for all benchmarks, but because they might be important for some.