acceptodds
Under review as a conference paper at ICLR 2027

Capturing Inter-Task Reinforcement Learning Dynamics for Adaptive Task-Sampling

Abstract

Multi-Task Reinforcement Learning (MTRL) trains a single agent on multiple tasks simultaneously, potentially improving performance and sample efficiency through positive transfer. However, MTRL suffers from *imbalanced learning*: tasks progress at different rates and may interfere with one another. Prior work addresses this problem using specialized task-sampling strategies that prescribe, *a priori*, how experience should be allocated across tasks. We instead argue that task-sampling strategies should be learned directly from data. For that end, we introduce DynATS, a learned sampler that captures inter-task learning **D**ynamics for **A**daptive **T**ask **S**ampling. DynATS is trained across sets of tasks using Evolution Strategies to maximize the performance of the multi-task agent as a whole. On MTRL problems, DynATS outperforms existing sampling strategies on average. By construction, DynATS transfers seamlessly across task distributions and domains. We transfer a learned sampler trained only on discrete, grid-based task sets to unseen task sets from the same distribution and to unseen continuous-control tasks in MuJoCo and Meta-World, demonstrating that the learned strategies can generalize beyond the environments on which they were trained.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Multi-Task Reinforcement Learning, Adaptive Task Sampling

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_T8EHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper attempts to address the issue in multi-task reinforcement learning (MTRL) where tasks are uniformly sampled from. In particular, this seems to harm the performance on the hard tasks. This submission proposes Dynamics for Adaptive Task Sampling (DynATS) that attempts to capture the inter-task relationship between tasks and models an updated sampling ratio between the different tasks. The DynATS method is trained via an evolutionary strategy to optimize the sampling ratios between tasks. Results seem to indicate that DynATS matches, if not slightly outperforms, applicable baselines methods.

Critical strengths

The method seems to be easily trainable, and didn't add nearly as much computational overload during MTRL as I expected.

Critical weaknesses
  1. The DynATS method only really outperforms DRATS in a single setting (XLand), but it is not clear (a) how DRATS' hyperparameters were tuned, or (b) whether the same computational budget was allocated to DRATS for tuning hyperparameters (XLand's DynATS model was trained with 65 billion interactions. DRATS needs to be tuned within a similar computational budget).

This is consequential because if we are going to say that Algorithm A is better than Algorithm B, then we need to ensure that we give Algorithm B the same set of circumstances (in this case, hyperparamater tuning budget)

  1. The main text mentions that "... the appropriate interpretation of the same signal can depend on the task set and its learning dynamics" in regards to what information should be used to choose the sampling ratios between tasks. However, when looking at what the DynATS method has learned, it's put all of the weight on the performance of the tasks, contradicting the motivating statement for the method.

This statement in the submission seems to suggest that performance by having additional inputs to the DynATS model outside of just performance, there will be better performance. However, the performance of DynATS only matches DRATS (except in XLand, as addressed above). Therefore, I don't particularly see the benefit of DynATS.

  1. Finally, I have some reservations about the generalization claims made in this work. Seeing as how training DynATS seemed to take 25 billion (grid worlds) or 65 billion (XLand) environmental interactions to train the base model (on top of once we have some new ratio from the DynATS model we then need to do RL training on top of that), I don't see how we could expand on the settings where DynATS is useful without an extremely large computational budget for some new problem setting. At that point I would imagine it would be more efficient to use DRATS because, while maybe needing some hyperparameter tuning, it does not need 25-65 billion interactions to find a better performing RL agent.
Questions for authors
  1. What are the actual magnitudes of the inputs to the DynATS method? Is it possible that the performance inputs dominate the non-performance inputs? e.g. what's the scale of the relative performance input and the alloc input?

  2. Did the authors consider any biologically inspired evolutionary algorithms (e.g. Particle Swarm Optimization) instead of ES?

Miscellaneous:

  1. Formula 3 should probably have epsilon as a subscript of the expectation. Right now it looks to be in-line.
  2. The specialized architectures of (Yang et al., 2020; Sodhani et al., 2021; Sun et al., 2022; Hendawy et al., 2024) were found to perform well due to the size of their networks by McLean et al, 2025b.
  3. The Meta-World reference should probably be McLean et al, 2025a.

Citations

  1. Reginald McLean, Evangelos Chatzaroulas, Luc McCutcheon, Frank Röder, Tianhe Yu, Zhanpeng He, KR Zentner, Ryan Julian, Jordan K Terry, Isaac Woungang, et al. Meta-world+: An improved, standardized, rl benchmark. arXiv preprint arXiv:2505.11289, 2025a.

  2. Reginald McLean, Evangelos Chatzaroulas, Jordan Terry, Isaac Woungang, Nariman Farsad, and Pablo Samuel Castro. Multi-task reinforcement learning enables parameter scaling. arXiv preprint arXiv:2503.05126, 2025b.

8 comments
AuthorsOpenReview ↗

Response to Reviewer T8EH (1/2)

We thank the reviewer for their time and effort in reviewing our work.

The DynATS method only really outperforms DRATS in a single setting (XLand)...

DynATS significantly outperforms DRATS in the Synthetic, Skillchain and XLand benchmarks, while remaining competitive in the Gridworlds and continuous control benchmarks.

but it is not clear (a) how DRATS' hyperparameters were tuned...

Table 8 in Appendix E.3 specifies how hyperparameters were tuned for all baselines on our experimental results (except the continual control experiments, where we reused the hyperparameters from Corrado et al. [1]). For all baselines, there are two hyperparameters: η\eta and α\alpha. As in Corrado et al. [1], we set α=η/2\alpha=\eta/2 and sweep only over η\eta. The baselines were finetuned for each individual set of tasks, using 100100 PPO seeds for each value of η\eta.

...or (b) whether the same computational budget was allocated to DRATS for tuning hyperparameters (XLand's DynATS model was trained with 65 billion interactions. DRATS needs to be tuned within a similar computational budget).

We argue that DRATS' underperformance on the XLand domain is not due to insufficient hyperparameter tuning, but instead due to a mismatch between its prioritization strategy and the sampling strategy that benefits XLand. In other words, a sampling strategy that prioritizes hard tasks is not appropriate for XLand (and neither for the Synthetic and Skillchain benchmarks). To support this claim, we added §C.4 and Figure 11, which shows that, for the XLand domain, as we increase the value of η\eta, thereby making DRATS more aggressive in prioritizing hard tasks, performance monotonically degrades. In fact, the best performing hyperparameter was η=0.25\eta=0.25, which corresponds to the weakest possible hard-task prioritization value we consider, since η=0\eta=0 would correspond to uniform sampling. This demonstrates that DRATS' underperformance is not due to insufficient hyperparameter tuning, but to a mismatch in prioritization strategy. We also want to clarify that we do not claim our method to be superior to DRATS, but different. DRATS defines a priori what makes a task worth sampling, which works extremelly well across many MTRL benchmarks (e.g. Mujoco6 and Meta-World 10/50). Our method simply addresses the fact that different benchmarks may benefit from different strategies, and that this can be learned.

Nonetheless, for Gridworlds (a benchmark on which DRATS is well-suited), hyperparameter tuning is important. For a fairer comparison, we have now increased the range of our sweep to η={0.25,0.5,1,2,4,8,16,32}\eta=\{0.25,0.5,1,2,4,8,16,32\} for the Gridworlds and XLand benchmarks. We believe that the DRATS tuning budget, while not on the same scale as one meta-training run, is comprehensive enough to warrant a fair comparison. Moreover, this is as large of a hyperparameter search as is reasonable for more expensive benchmarks. In the revised paper, we have therefore updated Figure 3. The results have not changed significantly on neither domain. The main difference is that DRATS on XLand now performs equivalently to uniform sampling (due to choosing η=0.25\eta=0.25) for most sets of tasks.

The main text mentions that "... the appropriate interpretation of the same signal can depend on the task set and its learning dynamics" in regards to what information should be used to choose the sampling ratios between tasks. However, when looking at what the DynATS method has learned, it's put all of the weight on the performance of the tasks, contradicting the motivating statement for the method.

We do not believe our statement to be contradicted by verifying that per-task performance is the most important input feature for learning a sampling strategy. Most sampling strategies (e.g. DRATS [1], Hard-Tasks-First [2], Learning Potential [3]) rely on performance metrics to determine what makes a task worth sampling (and they all converge to different, sometimes even opposing, interpretations!). As we show in our theoretical motivation in §4.1, the same metric can be interpreted differently. DynATS learns how it should be interpreted and converted into a sampling strategy based on inter-task dynamics. As we mention in §5.6, we include extra training signals, not because they are strictly needed for all benchmarks, but because they might be important for some.

AuthorsOpenReview ↗

Response to Reviewer T8EH (2/2)

Finally, I have some reservations about the generalization claims made in this work. Seeing as how training DynATS seemed to take 25 billion (grid worlds) or 65 billion (XLand) environmental interactions to train the base model (on top of once we have some new ratio from the DynATS model we then need to do RL training on top of that), I don't see how we could expand on the settings where DynATS is useful without an extremely large computational budget for some new problem setting.

One of the main claims of our work is that DynATS can be trained on simple, inexpensive domains and deployed zero-shot on complex, expensive one (e.g. Gridworlds -> Meta-World). Moreover, our distillation results now included in the revised paper in §5.7 demonstrate that it is also possible to learn a (more) general sampler that can be deployed zero-shot on diverse domains that may require different prioritization strategies (see GridWorlds and XLand results in Figure 5). This could be expanded in subsequent work by creating even more diverse (but cheap) training distributions that would allow the training of a general sampler. This has recently been done in the field of learned optimization, where a learned optimizer is trained on simple and small computer vision problems, while still outperforming finetuned optimizers on LLM pretraining runs [4] by improving the diversity of the training distribution and meta-learning process.

Moreover, while requiring 25B25B environmental interactions, ES-training on Gridworlds corresponds to only roughly 11 hour of runtime (see §C.9.1). The DynATS sampler can then be immediately deployed without finetuning on more complex domains.

What are the actual magnitudes of the inputs to the DynATS method? Is it possible that the performance inputs dominate the non-performance inputs? e.g. what's the scale of the relative performance input and the alloc input?

The table reports the observed input magnitudes on all 3232 held-out T=5T=5 sets from GridWorlds and XLand. The relative performance features have a wider spread than the allocation features, as expected from standardization across tasks (Appendix B.1). In general, the allocation and performance input features have comparable scales.

InputGridWorlds median [p05, p95]XLand median [p05, p95]
Relative Allocation−0.11 [−0.93, 0.65]−0.10 [−0.94, 0.70]
Relative Cumulative Allocation0.93 [0.47, 1.76]0.78 [0.44, 2.16]
Performance0.81 [0.30, 1.00]0.47 [0.00, 0.91]
Performance Change0.00 [−0.27, 0.28]0.00 [−0.16, 0.18]
Relative Performance0.19 [−1.76, 1.39]0.15 [−1.79, 1.53]
Relative Performance Change0.00 [−1.70, 1.69]0.00 [−1.70, 1.73]

Did the authors consider any biologically inspired evolutionary algorithms (e.g. Particle Swarm Optimization) instead of ES?

We considered other biologically inspired evolutionary algorithms. In the end, we used ES, because they are significantly well-explored in learned optimization for Supervised [4] and Reinforcement Learning [5] and in meta-learning [6].

Formula 3 should probably have epsilon as a subscript of the expectation. Right now it looks to be in-line.

Thank you for noticing, it's updated in the revised paper.

The specialized architectures of (Yang et al., 2020; Sodhani et al., 2021; Sun et al., 2022; Hendawy et al., 2024) were found to perform well due to the size of their networks by McLean et al, 2025b.

We have added a small sentence to the first paragraph of our Related Work section (Section 2) discussing the referenced work.

The Meta-World reference should probably be McLean et al, 2025a.

That's correct, we have updated the reference in the revised paper.

Citations:

[1] Corrado, Nicholas E., Wenyuan Huang, and Josiah P. Hanna. "Distributionally Robust Multi-Task Reinforcement Learning via Adaptive Task Sampling." arXiv preprint arXiv:2605.14350 (2026).

[2] Cho, Myungsik, et al. "Hard Tasks First: Multi-Task Reinforcement Learning Through Task Scheduling." ICML. 2024.

[3] Jiang, Minqi, Edward Grefenstette, and Tim Rocktäschel. "Prioritized level replay." International Conference on Machine Learning. PMLR, 2021.

[4] Moudgil, Abhinav, Boris Knyazev, and Eugene Belilovsky. "Celo2: Towards Learned Optimization Free Lunch." arXiv preprint arXiv:2602.19142 (2026).

[5] Goldie, Alexander D., et al. "Can learned optimization make reinforcement learning less difficult?." Advances in Neural Information Processing Systems 37 (2024): 5454-5497.

[6] Jackson, Matthew T., et al. "Discovering temporally-aware reinforcement learning algorithms." International Conference on Learning Representations. Vol. 2024. 2024.

DynATS significantly outperforms DRATS in the Synthetic, Skillchain and XLand benchmarks, while remaining competitive in the Gridworlds and continuous control benchmarks.

In Figure 2(b), the synthetic benchmark shows that DRATS and DynATS end with the same mean normalized return, so I cannot agree that it significantly outperforms DRATS. I cannot find the Skillchain results trained with DRATS, if you could kindly point to where they are I would appreciate it.

we added §C.4 and Figure 11, which shows that, for the XLand domain

Figure 11 compares a dotted line with no confidence intervals, to a line with confidence intervals. Without confidence intervals (and statistical testing) this does not show one algorithm outperforming another.

we do not claim our method to be superior to DRATS, but different

No, the submission does make that claim. From the abstract "DynATS outperforms existing sampling strategies" which states superiority about performance.

If the authors would like to make their claims statistically significant, showing that DynATS is statistically significantly better than DRATS/other methods, then I might be able to agree with claims of outperformance.

while not on the same scale as one meta-training run

Well, it's not tuned across 25 or 65 billion interactions, so I don't think I am open to this being a fair comparison. Does 25-65 billion interactions account for the amount of time tuning the hyperparameters of the transformer itself?

not because they are strictly needed for all benchmarks, but because they might be important for some.

They turned out to be important in one single case, XLand. So yes, the vast majority of the evidence in this submission contradicts the reasoning for the DynATS method. "They might be important for some" but the submission doesn't contain many examples of support that the non-performance based inputs to DynATS are providing any meaningful additions to the method, except for one example of XLand.

inexpensive domains and deployed zero-shot on complex, expensive one

The submission has evidence that clearly demonstrates that DynATS outperforms DRATS in one setting, XLand. Can it be deployed zero-shot in XLand? Absolutely. Can it be deployed zero-shot elsewhere? The submission does not support that conclusion.

My point about 25 billion (grid worlds) or 65 billion (XLand) interactions stems from the fact that the submission does not support this generalization claim. The submission has evidence that deployment zero-shot to XLand works, but that is not enough to support that this will work elsewhere. At best DynATS matches DRATS, with DynATS needing billions of interactions to be initially trained.

environmental interactions, ES-training on Gridworlds corresponds to only roughly hour of runtime

It's not about the runtime. It's about how many interactions are needed to train this model, and the limits of training on this set of gridworlds. The submission has support for only a single zero-shot generalization to XLand.

If it was about the runtime, I would make this argument. When, not if, DynATS does not perform well in a zero-shot deployment and needs to be re-trained on something that isn't a gridworld, DynATS is going to be extremely computationally expensive. Imagine if there was signal that training DynATS on Meta-World tasks was the fix instead of training on gridworlds. Each Meta-World experiment takes 6-8 hours on MT10 (dependent on hardware and hyperparameters), and 25 billion interactions is going to be one thousand two hundred fifty experiments on Meta-World alone (at twenty million total steps per experiment). Close to ten thousand hours of training DynATS on Meta-World. The method doesn't really scale well if I need to spend something like ten thousand hours training the transformer, if I could just spend that time running the RL algorihtm on the problem that I care about and tune the hyperparameters, or apply DRATS and tune it's parameters, within the same amount of time.

Reviewer_xvhYOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DynATS, an adaptive task sampling method for multi-task reinforcement learning that uses a transformer to map task training statistics to sampling probabilities. Critically, the sampler is trained using evolution strategies to maximize aggregate performance over complete multi-task RL training runs, rather than relying on a predefined task prioritization metric like most prior works. Experiments demonstrate that DynATS can learn effective sampling strategies, outperform existing methods on some benchmarks, and generalize to unseen task sets.

Critical strengths
  1. The paper does a very good job highlight that no fixed interpretation of a task prioritization signal is universally good across all multi-task setups, and I personally like the idea of having the agent learn what a good prioritization signal is rather than specify one a priori. This is the most interesting aspect of the paper for me.
  2. Showing transfer of the task sampler from gridwrold tasks to continuous control Mujoco tasks was compelling.
Critical weaknesses

It is somewhat unclear what the authors believe constitute a strong result for DynATS. I see two ways in which the method could demonstrate its value:

  1. DynATS discovers a task prioritization strategy that differs substantially from existing methods and achieves better performance.
  2. DynATS discovers a strategy similar to an existing method and achieves comparable performance without requiring the prioritization rule to be specified in advance.

Both would be meaningful results, but I don’t think the experiments fully explore either possibility. Current experiments provide evidence for 2; In the Chain setup, DynATS learns a strategy similar to the hardcoded curriculum and in MuJoCo and Metaworld, it learns a strategy similar to DRATS. I don’t see convincing evidence that DynATS discovers prioritization strategies that differ substantially from existing approaches. This is not necessarily a weakness of the method, but the method would be more compelling if you could show that DynATS outperforms all baselines by discovering a fundamentally different strategy. If the authors primarily aimed to show 2, I would like to see experiments designed to test this hypothesis more explicitly. For example, the authors could construct benchmarks where different existing prioritization strategies are most effective and evaluate whether DynATS can learn the appropriate strategy in each setting. I think existing experiments do this to some degree.

The cross-domain generalization experiments would be more convincing if the source and target domains favored different prioritization strategies. DynATS learns DRATS-like strategy in both GridWorld and the target domains (MuJoCo and Meta-World)---in particular, the DRATS-like strategy for GridWorld also works in MuJoCo and Metaworld. I'm not sure if the experiments clearly show that DynATS can adapt its prioritization strategy to different inter-task learning dynamics. Evaluating transfer between setups with different "optimal" prioritization strategies would provide useful info on this matter.

I find the ideas in this paper compelling, but for the reasons above, the experimental analysis strikes me as incomplete.

Questions for authors
  1. Can you construct a setup where DynATS outperforms baselines by discovering a strategy that differ substantially from baselines?
  2. Could the authors construct benchmarks where different existing prioritization strategies perform best and evaluate whether DynATS learns the appropriate strategy?
  3. Can DynATS generalize between source and target domains that favor substantially different prioritization strategies?
4 comments
AuthorsOpenReview ↗

Response to Reviewer xvhY

We thank the reviewer for their time and effort in reviewing our work.

It is somewhat unclear what the authors believe constitute a strong result for DynATS. I see two ways in which the method could demonstrate its value:

  1. DynATS discovers a task prioritization strategy that differs substantially from existing methods and achieves better performance.
  2. DynATS discovers a strategy similar to an existing method and achieves comparable performance without requiring the prioritization rule to be specified in advance.

Both would be meaningful results, but I don’t think the experiments fully explore either possibility.

We agree with the reviewer in that these are two possible ways in which DynATS could demonstrate its value. We believe our experimental results explore both.

Regarding 1:

  • The Synthetic experiment demonstrates that DynATS can learn a sampling strategy that massively outperforms existing baselines (which can only be achieved by learning a substantially different sampling strategy). Although not MTRL based, this benchmark is interesting because it allows for many complex and different inter-task relationships, see §D.1 (e.g. positive and negative transfer to different degrees and between a different number of tasks, task independency, etc.)
  • In the XLand benchmark, DynATS also learns a sampling strategy that massively outperforms existing baselines

Regarding 2:

  • The Gridworlds, Chain and continuous control experiments demonstrate that DynATS discovers a strategy similar to existing methods with comparable or slightly superior performance without requiring the prioritization rule to be specified in advance.

Regarding the remaining comments and questions, listed in the block below, we believe §5.7 of the revised paper addresses them all. Since the questions and the remaining comments are linked, we will just address the questions.

Can DynATS generalize between source and target domains that favor substantially different prioritization strategies?

In Section §5.7 of the revised paper, we analyse in-depth the extent to which the GridWorlds- and XLand-trained samplers generalize to the other domain. Figure 5 shows that a sampler trained on Gridworlds transfers poorly to XLand and vice-versa, since there is a mismatch between the prioritization strategies that are useful on those two domains. A DynATS sampler trained on the Gridworlds domain learns a DRATS-like strategy that transfers poorly to XLand; one trained on XLand learns a more "curriculum"-like strategy that transfers poorly to Gridworlds.

Could the authors construct benchmarks where different existing prioritization strategies perform best and evaluate whether DynATS learns the appropriate strategy?

Section §5.7 also verifies the performance of a student distilled from the two experts (one trained on GridWorlds and another on XLand). We show that this student matches the experts' performance on their domains. This demonstrates that a single DynATS sampler is capable of learning a mixture of different sampling strategies and of deploying the appropriate one based purely on inter-task dynamics. The student is never given a benchmark/domain identifier neither during training nor during deployment.

We also show that, when deployed on Meta-World10, the distilled student closely matches the sampling strategy (and performance) of the Gridworlds expert, demonstrating that it is capable of adapting to unseen inter-task dynamics by choosing the appropriate sampling strategy.

Can you construct a setup where DynATS outperforms baselines by discovering a strategy that differ substantially from baselines?

This is already the case for the Synthetic and XLand benchmarks. Moreover, the new results on the distilled student show that a DynATS sampler can even be trained to learn not only a single sampling strategy, but instead a combination of several, and when to use each based on inter-task dynamics.

AuthorsOpenReview ↗

Response to Reviewer xvhY

Dear reviewer, we hope that our answers have clarified your concerns. As the discussion period closes, we wanted to follow up on our rebuttal to check whether there are any issues left.

Thank you for the detailed response! I like the additional experiments in section 5.7; it's useful to see that DynATS can adapt its prioritization strategy to different inter-task learning dynamics.

It's unclear to me if the synthetic experiments are showing a fundamentally different prioritization strategy; the sampling probability curves do show that DynATS is learning a different strategy than the baselines, but the baselines in this figure are non-exhaustive. It's possible that DynATS is learning, for instance, a strategy similar to another method like PLR (just an example).

I do agree that DynATS discovers a strategy similar to an existing method and achieves comparable performance without requiring the prioritization rule to be specified in advance. This is a good result.

I am going to raise my score for now, and will finalize any additional thoughts during the AC-reviewer discussion phase.

AuthorsOpenReview ↗

Response to Reviewer xvhY

We thank the reviewer for their positive response and for raising their score.

It's possible that DynATS is learning, for instance, a strategy similar to another method like PLR.

Our baselines already include PLR's priority score as Learning Potential (LP, mean absolute GAE), alongside Learning Progress (LPR) and DRATS, all tuned per benchmark. Figure 2 shows only the best performing baseline (DRATS) for readability reasons for both the Synthetic and Chain experiments.

  • Synthetic: LPR captures none of the available headroom (performs as well as uniform sampling), DRATS 26.2%, and DynATS 66%. LP is undefined here, since this benchmark is not RL-based and therefore has no policy or value function. The oracle adapts to each task set's transfer structure, which no fixed rule can do. DynATS learns to approximate this from inter-task dynamics.
  • Chain: DynATS significantly outperforms LP and LPR (p<10−14p<10^{-14}), as well as DRATS and the hand-designed curriculum (see table in our response to Reviewer Vjj8).
Reviewer_nnBDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies adaptive task sampling in multi-task reinforcement learning and proposes DynATS, a learned sampler that uses task-level training statistics to determine how experience should be allocated. A lightweight Transformer models interactions across tasks and changes in their learning histories. Its parameters are optimized with Evolution Strategies (ES), using the area under the learning curve of the PPO agent’s mean normalized task performance as the fitness. The method is evaluated on synthetic task interactions, Chain, GridWorld, XLand, MuJoCo, and Meta-World. DynATS outperforms heuristic samplers in several settings. Its main empirical result is that a sampler meta-trained on GridWorld can be used on MuJoCo and Meta-World without target-domain sampler fine-tuning, achieving performance comparable to tuned DRATS.

Critical strengths
  1. Cross-domain transfer without retraining the sampler. The most interesting result is the transfer from GridWorld to MuJoCo6 and Meta-World10/50. The sampler is trained on five-task GridWorld suites but can be deployed on continuous-control benchmarks with up to 50 tasks without updating its parameters. It consistently improves over Uniform and performs close to DRATS, which is tuned for the target environments. GridWorld meta-training takes 1.30 hours on an H200 GPU in the reported setup. These results suggest that sampler meta-training on relatively inexpensive environments can be reused across more costly tasks, although the total cost advantage depends on how often the sampler is reused.
  2. Evidence that learned sampling can improve the training process. The controlled experiments provide useful evidence beyond the transfer results. On the synthetic benchmark, DynATS-Transformer captures 64.3% of the headroom between Uniform and the dynamic oracle, compared with 26.2% for DRATS. On Chain, DynATS achieves an AUC of 0.838, compared with 0.719 for DRATS and 0.808 for the hand-designed curriculum. The Chain results are especially useful because they show that the learned sampler can change its allocation during actual PPO training and outperform a reasonable manually specified schedule.
Critical weaknesses
  1. The transfer experiments leave an important question about what the sampler has learned. The paper motivates DynATS partly by arguing that different task distributions may favor different sampling rules. However, it is difficult to tell from the transfer experiments whether the sampler has learned to adapt to different task-learning dynamics or has learned a rule that happens to work well on the evaluated target domains. On MuJoCo and Meta-World, the transferred DynATS sampler behaves similarly to DRATS. This is not a problem by itself, since matching a tuned heuristic without target-domain sampler optimization is already useful. However, the same paper shows that DRATS works well on GridWorld but falls below Uniform on XLand. Despite this difference, GridWorld and XLand use separately meta-trained DynATS samplers, and no direct transfer between them is reported. The results therefore establish successful transfer to the selected continuous-control benchmarks, but provide less insight into the conditions under which a frozen sampler can adapt to different learning dynamics. A GridWorld-to-XLand experiment would be informative here, without requiring the method to generalize to every possible task distribution.
  2. The benefit of the full Transformer architecture is not well isolated. DynATS combines task attention and temporal attention, but the experiments do not establish how much each component contributes to the final results. The comparison with Linear and DeepSets on the synthetic benchmark is useful, yet it changes more than the network architecture. The simpler models use exponential moving averages of the training statistics, while the Transformer receives the full history. This makes it difficult to determine whether the improvement comes from attention, access to longer histories, or their combination. The issue is more important for the cross-domain results, where no simpler learned sampler is evaluated using the same transfer protocol. Since the paper's methodological contribution centers on learning inter-task dynamics, stronger evidence for the added value of its two attention components would make the work more convincing.
Questions for authors
  1. Has the GridWorld-trained sampler been evaluated directly on XLand? The difference in DRATS performance between the two domains makes this a useful test of how the frozen sampler behaves under a different task distribution. If such a transfer experiment has been conducted, what sampling behavior and performance were observed? Otherwise, could the authors clarify the expected limits of cross-domain transfer?
  2. How much do task attention and temporal attention contribute independently? Could the authors evaluate removing each attention component while keeping the historical inputs and ES training setup comparable? It would also be useful to compare a simpler learned sampler with DynATS on the cross-domain benchmarks to clarify the additional benefit of the full architecture.
  3. How stable is DynATS meta-training on XLand? Table 5 reports only one independent ES run for XLand. The 100 PPO evaluation seeds measure the performance of the selected sampler, but not the variation between independently trained samplers. Could the authors provide additional ES runs or report any observations about meta-training stability?
2 comments
AuthorsOpenReview ↗

Response to Reviewer nnBD

We thank the reviewer for their time and effort in reviewing our work.

The transfer experiments leave an important question about what the sampler has learned... A GridWorld-to-XLand experiment would be informative here, without requiring the method to generalize to every possible task distribution.

Thank you for raising these very relevant questions. In the revised paper, we added §5.7, where we analyse in-depth the extent to which the GridWorlds- and XLand-trained samplers generalize to the other domain. Figure 5 shows that a sampler trained on Gridworlds transfers poorly to XLand and vice-versa, since there is a mismatch between the prioritization strategies that are useful on those two domains. A DynATS sampler trained on the Gridworlds domain learns a DRATS-like strategy that transfers poorly to XLand; one trained on XLand learns a more "curriculum"-like strategy that transfers poorly to Gridworlds.

The main contribution of our work is in showing that it is possible to learn the prioritization strategy instead of specifying it in advance, outperforming or rivaling existing baselines (e.g. Synthetic, Chain, Gridworlds and XLand results). We then show that prioritization strategies learned on simpler inexpensive domains can transfer seamlessly to complex domains zero-shot. We now tried to go further and pave the way towards a more general DynATS sampler. We distill the two experts (one trained on GridWorlds and another on XLand) into a single student (see §D.5 of the revised paper for more details on the distillation training) and we show that this student matches the experts' performance on their domains. We also show that, when deployed on Meta-World10, the distilled student closely matches the sampling strategy (and performance) of the Gridworlds expert, demonstrating that it is capable of adapting to different inter-task dynamics by choosing the appropriate sampling strategy. We leave for future work achieving a more general sampler via larger and more diverse training distributions through end-to-end ES meta-training, and a more comprehensive analysis of the transfer capabilities of such a sampler.

The benefit of the full Transformer architecture is not well isolated. DynATS combines task attention and temporal attention, but the experiments do not establish how much each component contributes to the final results... The issue is more important for the cross-domain results, where no simpler learned sampler is evaluated using the same transfer protocol.

We modified the comparison on the synthetic benchmark such that (i.) all DynATS models receive the full history of inputs, isolating the architectural differences; and (ii.) added an ablation over the two attention mechanisms. The results show that the Linear and Deepsets models now perform better, but still underperform the transformer architecture. Moreover, removing either attention mechanism deteriorates performance. We also added a comparison between the architectures on Chain and GridWorlds (Appendix C.3, Fig. 9). On Chain, the Transformer achieves higher mean AUC than Linear and DeepSets, and removing temporal attention reduces its mean AUC. On GridWorlds, all architectures behave similarly for T=5T=5, while for T=10T=10 the linear architecture performs worse. We also evaluated zero-shot transfer of the GridWorlds-trained Linear sampler to Meta-World10, showing it achieves comparable performance to the Transformer through 20M steps (see final paragraph of §5.5 and Appendix C.6). These results show that the transformer architecture is not strictly necessary on all benchmarks (in some, a simpler linear architecture suffices), but the transformer architecture is always either the best or on-par with the best performers on all benchmarks. In benchmarks where inter-task interactions are particularly important (e.g. Synthetic and Chain), the complete transformer architecture outperforms all others. We therefore consider the transformer architecture to be the most adequate one for task sampling in MTRL, not the only meaningful one. We also note that the runtime/resources bottleneck, in both training and deployment, is in the environment interactions and PPO updates (see §C.9.2), rather than the sampler's architecture, thus justifying using the transformer.

How stable is DynATS meta-training on XLand? Could the authors provide additional ES runs or report any observations about meta-training stability?

In general, ES meta-training is stable on all domains. There is some statistical variance between seeds, but all learn effectively and converge to a similar performance level within the established ES generations. In the revised paper, we added to the Appendix a small section (§C.2 and Figure 8) with the meta-learning curves throughout training for 33 independent seeds on the GridWorlds and XLand domains, which shows that all 33 seeds converge to similar performance levels (with small statistical variance).

AuthorsOpenReview ↗

Response to Reviewer nnBD

Dear reviewer, we hope that our answers have clarified your concerns. As the discussion period closes, we wanted to follow up on our rebuttal to check whether there are any issues left.

Reviewer_Vjj8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DynATS, a learned task sampler for multi-task reinforcement learning. A Transformer maps the history of task performance and data allocation to a sampling distribution, while Evolution Strategies trains the sampler to maximize the multi-task agent’s performance over a full PPO training run. The paper evaluates DynATS in controlled experiments and on held-out GridWorld and XLand task sets. It also applies a GridWorld-trained sampler without finetuning to MuJoCo and Meta-World, where it improves over uniform sampling and remains competitive with target-tuned DRATS.

Critical strengths
  1. An interesting approach to adaptive task sampling. Rather than prescribing how a single performance signal should determine task priority, DynATS conditions sampling decisions on the history of task performance, performance changes, and data allocation. Treating the evolution of multi-task learning as input to a learned sampler is a compelling direction.

  2. Notable zero-shot transfer of the sampler. A sampler trained on discrete GridWorld task sets is applied without finetuning to unseen task sets and continuous-control benchmarks, including MuJoCo and Meta-World. This is an interesting test of whether a sampling strategy learned from generic training statistics can be reused across substantially different domains. The transferred sampler improves over uniform sampling and remains competitive with target-tuned DRATS.

  3. Clear overall presentation. The introduction motivates the task-allocation problem clearly, and the paper progresses logically from the controlled experiments to same-domain generalization and cross-domain transfer. This structure makes the main idea and the purpose of each experiment easy to follow.

Critical weaknesses
  1. Limited comparison with simpler learned samplers in actual MTRL environments. The paper compares Linear, DeepSets, and Transformer samplers in the synthetic experiment, but the RL experiments mainly compare DynATS with hand-designed sampling rules. It is therefore difficult to determine whether the gains in RL arise from learning a sampler at all or specifically from modeling temporal and cross-task learning dynamics.

  2. Limited evidence of applicability beyond PPO. The paper describes the sampler features as agnostic to the RL algorithm, yet all RL experiments use PPO. It remains unclear whether the benefits persist when DynATS is combined with other MTRL architectures or methods for mitigating task interference.

  3. Insufficient analysis of sensitivity to performance normalization. GridWorld uses exact task-specific performance ranges, whereas the continuous-control experiments use random-policy performance, observed maximum returns, and predefined initial reference values. The paper does not systematically examine how changes or errors in these references affect the learned sampler’s decisions.

Questions for authors
  1. How is the normalized evaluation performance in Eq. (8) computed? Appendix B.1 specifies a normalization formula for the sampler’s input features, while Eq. (8) uses performance from separate evaluation episodes. Are the same reference values and clipping applied in both cases? For XLand, where evaluation occurs every five PPO epochs, how is the AUC calculated?

  2. Are the initial random-policy episodes included in the reported environment-interaction budget? The Meta-World experiments use 1,000 random-policy episodes per task to estimate a reference value. Are these interactions included in the reported 100M-step budget? How are they accounted for in comparisons with Uniform, which does not require this reference?

  3. How are performance features initialized for a task that is not sampled in the first PPO epoch? Uniform sampling in the first epoch does not guarantee that every task appears in the rollout buffer. What performance and performance-change values are assigned to tasks with no observations yet?

  4. What is the practical benefit of reusing the learned sampler after accounting for its training cost? A comparison with DRATS using fixed hyperparameters and no target-task tuning would help isolate the benefit of reusing a learned sampler from that of reusing a simple sampling rule. Could the authors report total cost and performance across multiple new task sets, including sampler training and reference-estimation costs?

  5. What generalizable insight does this work offer for subsequent research? It would help to explain what new research directions become possible by treating task-level data collection as a policy conditioned on learning history, and to distinguish those possibilities from what the current experiments directly establish.

6 comments
AuthorsOpenReview ↗

Response to Reviewer Vjj8 (1/2)

We thank the reviewer for their time and effort in reviewing our work.

Limited comparison with simpler learned samplers in actual MTRL environments...

In the revised paper, we added a comparison between the architectures on Chain and GridWorlds (Appendix C.3, Fig. 9). On Chain, the Transformer achieves higher mean AUC than Linear and DeepSets, and removing temporal attention reduces its mean AUC. On GridWorlds, all architectures behave similarly for T=5T=5, while for T=10T=10 the linear architecture performs worse. We also evaluated zero-shot transfer of the GridWorlds-trained Linear sampler to Meta-World10, showing it achieves comparable performance to the Transformer through 20M steps (see final paragraph of §5.5 and Appendix C.6). These results show that the transformer architecture is not strictly necessary on all benchmarks (in some, a simpler linear architecture suffices), but the transformer architecture is always either the best or on-par with the best performers on all benchmarks. In benchmarks where inter-task interactions are particularly important (e.g. Synthetic and Chain), the complete transformer architecture outperforms all others. We therefore consider the transformer architecture to be the most adequate one for task sampling in MTRL, not the only meaningful one. We also note that the runtime/resources bottleneck, in both training and deployment, is in the environment interactions and PPO updates (see §C.9.2), rather than the sampler's architecture, thus justifying using the transformer.

Limited evidence of applicability beyond PPO. The paper describes the sampler features as agnostic to the RL algorithm, yet all RL experiments use PPO.

We limit our experiments to on-policy RL (particularly PPO), because there is a direct relationship between which tasks are sampled and which tasks contribute learning data to the next policy update. In off-policy algorithms (e.g. DQN or SAC), this relationship is more complex, since the replay buffer can contain trajectories collected under earlier task distributions and policies. This introduces the additional question of how the replay buffer should be sampled. We leave the application of DynATS to off-policy MTRL for future research. In the revised paper, we added a paragraph to §4.1 justifying our choice.

It remains unclear whether the benefits persist when DynATS is combined with other MTRL architectures or methods for mitigating task interference.

We combined DynATS with MOORE [1] and PCGrad [2] on Meta-World10 (see final sentences of §5.5 and Appendix C.6.3). The results demonstrate that coupling these two MTRL-specific methods with DynATS-based sampling outperforms coupling them with uniform sampling. This shows that DynATS can be seamlessly combined with MTRL-specific methods.

Insufficient analysis of sensitivity to performance normalization. The paper does not systematically examine how changes or errors in these references affect the learned sampler’s decisions.

When deployed on Mujoco6, DynATS uses a random-policy lower reference and a running upper reference. Nonetheless, it remains competitive with tuned DRATS (§5.5). We added to the revised paper an alternative Meta-World10 deployment that estimates both references using running statistics, skipping random-policy calibration (§C.6.2, Figure 16). Results show that both methods perform and behave similarly. These experiments (on both Mujoco6 and Meta-World10) demonstrate that DynATS is quite robust to different performance normalization strategies.

How is the normalized evaluation performance in Eq. (8) computed?

The fitness uses returns from separate evaluation episodes, while the sampler inputs use returns from the PPO training rollouts. For GridWorlds and XLand, both use the same reference bounds and clip the normalized performance to [0,1][0,1]. GridWorlds uses exact bounds, and XLand uses [0,1][0,1] references. In XLand meta-training, we evaluate every five PPO epochs and compute AUC as the mean of the normalized returns at those evaluation checkpoints. See §4.2, Eq. (8), and Appendix E.2, Table 7.

Are the initial random-policy episodes included in the reported environment-interaction budget?

The reported 100M steps are only the PPO training interactions. The 1,000 random-policy episodes per task add 5M calibration interactions per Meta-World10 seed. These are used to estimate the lower performance bounds, but they are not strictly needed. In §C.6.2, Figure 16, we verified that using running statistics on Meta-World10, skipping random policy evaluation, performs and behaves similarly.

AuthorsOpenReview ↗

Response to Reviewer Vjj8 (2/2)

How are performance features initialized for a task that is not sampled in the first PPO epoch?

In GridWorlds and XLand, the estimated return starts at zero and remains there until the task is first sampled. Once a task has been sampled, epochs without new observations carry its latest estimate forward. In the continuous-control experiments, unobserved returns and undefined sampler features are filled with zero (§4.2 and Appendix B.1).

What is the practical benefit of reusing the learned sampler after accounting for its training cost?

We establish two practical benefits. If samples are inexpensive (as is the case with many GPU-accelerated benchmarks), then learning a sampler can outperform all existing strategies (see results on the Synthetic (§5.2), Chain (§5.3) and XLand (§5.4) benchmarks). If samples are expensive (e.g. Meta-World10 and Mujoco6), we show that zero-shot deploying a sampler learned on simple inexpensive domains (DynATS was trained on GridWorlds in roughly 11 hour) can rival a method that must be tuned for the specific (expensive) benchmark.

A comparison with DRATS using fixed hyperparameters and no target-task tuning would help isolate the benefit of reusing a learned sampler from that of reusing a simple sampling rule. Could the authors report total cost and performance across multiple new task sets, including sampler training and reference-estimation costs?

For the Gridworlds and XLand domains, Appendix C.4 (Fig. 10) now compares DynATS with DRATS-F, which selects one η\eta per domain and task count on validation sets and reuses it across all 32 held-out test sets, and with DRATS-T, which selects η\eta separately for each test set. On Gridworlds, DRATS-F clearly underperforms Drats-T and DynATS (particularly for T=10T=10). On XLand, DynATS massively outperforms DRATS. Table 2 in Appendix C.9 reports ES cost, including validation, of 25.44B interactions for GridWorlds and 64.64B for XLand. This should be seen as a one-off training cost, since the sampler may then be deployed zero-shot.

What generalizable insight does this work offer for subsequent research?

The main insight our work provides is that sampling strategies that define in advance what makes a task worth sampling suffer from the no free lunch theorem i.e. they improve performance on certain benchmarks at the cost of decreasing it in others. DynATS instead learns what makes a task worth sampling, and can generalize across domains. An interesting avenue for future research (particularly in light of our distillation experiments now included in §5.7 of the revised paper) will be improving upon and learning a more general sampler by improving the training distribution diversity and the meta-training algorithm.

Citations:

[1] Hendawy, Ahmed, Jan Peters, and Carlo D'Eramo. "Multi-task reinforcement learning with mixture of orthogonal experts." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Yu, Tianhe, et al. "Gradient surgery for multi-task learning." Advances in neural information processing systems 33 (2020): 5824-5836.

Reviewer_Vjj8OpenReview ↗

Response to the author

Thank you for the detailed response and additional experiments. My questions on Eq. (8) (Q1), the calibration episodes (Q2), and the initialization of unobserved tasks (Q3) are resolved, as is my question on compatibility with MTRL-specific methods (§C.6.3). The DRATS-F comparison (§C.4) is also a useful addition. Some concerns remain:

1. Simpler learned samplers (W1). In RL, the full Transformer is clearly better only on Chain. The architectures perform similarly on GridWorlds (T=5), and the Linear sampler matches the Transformer on Meta-World10. This suggests that much of the main transfer gain comes from learning a sampler at all, rather than from modeling inter-task dynamics as the title emphasizes. Please qualify this in the main text. Also: (a) Is there an architecture comparison on XLand, where DynATS gains the most? (b) The Linear sampler is reported as comparable "through 20M steps" on Meta-World10, whose main runs use 100M steps. Was it run for the full budget?

2. Practical benefit vs. cost (Q4). DRATS-F addresses reuse within GridWorlds and XLand, but my question concerned the cross-domain setting, where DRATS uses target-tuned hyperparameters from Corrado et al. A DRATS baseline with η fixed without target information (e.g., selected on GridWorlds) would test whether reusing a learned sampler offers more than reusing a simple rule. This matters given §5.7: transfer seems to work when source and target favor similar strategies (GridWorlds → Meta-World) and to fail otherwise (GridWorlds ↔ XLand). Please also include in the paper the cost comparison given to Reviewer T8EH (DRATS tuning: 11.5B/76.8B vs. ES: 25.44B/64.64B interactions) and the 5M calibration interactions per Meta-World10 seed.

3. Normalization sensitivity (W3). The comparison of two schemes on Meta-World10 helps, but my concern is the mismatch between exact bounds during meta-training and estimated references at deployment. A controlled study that perturbs the references and reports the effects on sampling and performance would address this. Without it, I would not claim general robustness.

4. Scope beyond PPO (W2). The on-policy justification is reasonable but is a scoping argument rather than evidence. Since many Meta-World MTRL methods build on SAC, please describe DynATS as applicable to on-policy MTRL rather than agnostic to the RL algorithm. Also, the PCGrad/MOORE experiments compare only against uniform sampling. Since DRATS is comparable to DynATS on Meta-World10, DRATS + PCGrad/MOORE would show whether the benefit is specific to DynATS.

5. Statistical support. I share Reviewer T8EH's request for significance tests supporting the claim that DynATS "outperforms existing sampling strategies on average." Since several differences above appear modest, confidence intervals and significance tests for the main comparisons would clarify which are reliable.

Addressing these points, particularly 1 and 2, would clarify the paper's contributions and scope.

Comments on the paper

1 comment

Second Revision

We thank all reviewers for their constructive feedback. Below, we summarize the main changes made in the revised manuscript (highlighted in blue in the PDF), indicating the reviewers whose comments inspired each change.

  • [Vjj8,nnBD] Limited comparison with simpler learned samplers: The revised paper now contains a comparison between all architectures (and ablations over the transformer's attention mechanisms) on Chain, GridWorlds and XLand, see §C.3 and Figure 9, as well as the deployment of a linear sampler on Meta-World10 in §C.6.1 (Figure 15). We have also standardized the input features of all architectures to contain the full input history. Figure 2(a) was updated accordingly, and §5.2 now qualifies when the Transformer is necessary.
  • [Vjj8] Work is limited to PPO: The revised paper now includes a paragraph in §4.1 justifying why our work considers only on-policy RL, and describes DynATS as applicable to on-policy MTRL (contributions and §4.2).
  • [Vjj8] Combination with MTRL-specific methods: The revised paper now includes in §C.6.3 (Figure 17) a study in Meta-World10 on the coupling of DynATS and DRATS with MOORE and PCGrad, showing that DynATS can be seamlessly combined with MTRL-specific methods.
  • [Vjj8] Robustness to performance normalization: The revised paper now contains in §C.6.2 (Figure 16) a comparison on Meta-World10 of DynATS using reference performance values vs. running statistics, and in §C.10 (Table 4) a controlled study on GridWorlds that perturbs the references of DynATS and DRATS and measures the effect on sampling and performance.
  • [Vjj8,T8EH] Reusing DRATS without target tuning and computational cost: §C.4 (Figures 10 and 12) now compares DynATS with DRATS using η\eta selected on validation sets (DRATS-F), both within domain and transferred from GridWorlds to Meta-World10. §C.9 now reports the DRATS tuning cost alongside the ES training cost, as well as the random-policy calibration interactions.
  • [nnBD,xvhY] Cross-transfer analysis and research towards a general sampler: The revised paper now contains in §5.7 an analysis of the cross-transfer capabilities of DynATS trained on GridWorlds or XLand on the other domain. It also presents a sampler distilled from the experts on both domains (explained in §D.5, without domain identifiers) that matches the experts' performances on their respective domains and generalizes to Meta-World10 (Figure 5).
  • [nnBD] ES meta-training stability on XLand: The revised paper now reports, in §C.2 (Figure 8), the learning curves of ES on GridWorlds and XLand over 33 seeds, demonstrating that ES meta-training is stable.
  • [T8EH] DRATS hyperparameter tuning: The revised paper now contains in §C.4 and Figure 11 an analysis of how the performance of DRATS in XLand varies along the tested η\eta values, showing that DRATS performs best when it resembles uniform sampling the most. We increased the range of our sweep as shown in Table 8, and Figure 3 was updated accordingly.
  • [T8EH,Vjj8] Statistical support: Significance tests for the Chain, GridWorlds, and XLand benchmarks are reported in our response to Reviewer Vjj8. The abstract now specifies that DynATS outperforms existing sampling strategies in sample efficiency, on average.
  • [T8EH] Miscellaneous: Updated references and slightly modified §2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.