acceptodds
Under review as a conference paper at ICLR 2027

Which Tasks Train Adaptation? Paired Interventions for Meta-RL Task Selection

Abstract

Training a meta-RL agent requires tasks on which within-episode adaptation is useful, not merely difficult tasks. Existing curricula rank tasks by difficulty, regret, learning progress, or post-adaptation return, but these signals do not reveal how much performance depends on adaptation. To address this issue, we introduce Paired Adaptation-Contrast Ecology (PACE), which directly measures this dependence by comparing matched executions with the episode-local update enabled or disabled and evaluating both on a frozen suffix. The resulting learner-dependent return difference is used to update the training distribution in a closed loop as the learner changes. In controlled matched-Post experiments, high-AdaptGain task groups improve Post by 0.0069 more than matched low-AdaptGain groups under the same retraining budget. Across native benchmarks, PACE achieves the highest mean Post in 18 of 20 compatible benchmark–learner cells.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
meta-reinforcement learning, task selection, curriculum learning, adaptation dependence, within-episode adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_A9qzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a method for measuring the amount of local adaptation an environment benefits from. It does so through a counterfactual measure, where an agent capable of local adaptations is rolled out on an environment, then compared against the same agent being rolled out but with its local adaptation mechanism disabled. The authors show how pairing this metric with an incentive to keep it at the frontier of what is attainable leads to gains over other curriculum design approaches that prioritize regret or learning progress.

Critical strengths
  • The idea of measuring adaptation through this intervention is intuitive and well motivated, and the authors did a good job of contrasting this objective from prior work.
  • The comparison to the Post experiment shows that AdaptGain carries additional information relevant for the task and combining both can lead to improvements
Critical weaknesses
  • Each round requires rolling out adaptive and static executions. However, the gains given this additional cost do not seem to be that meaningful, and there do not seem to be seeds or anything determining if these gains are even significant. Comparing all the baselines at the same amount of environment interactions would make it clear if the extra compute is worth the moderate gains.
  • Parts of the evaluation are unclear. Freezing the initial state could push recurrent policies out of distribution, so AdaptGain could reflect errors there rather than meaningful adaptation. For agents like in-context LLMs, it’s also not clear how this approach will extend, since removing interaction history could remove information needed to track the environment state.
Questions for authors
  • How many total environment interactions does PACE use compared to the baselines? Do the results change when all baselines have the same amount of environment interactions?
  • For recurrent learners, does freezing the hidden state keep the policy within its training distribution?
  • How does the static intervention work for in-context learners like LLM agents?
  • Could you report seed counts, confidence intervals, and significance tests for all the results where applicable?
Reviewer_dCJtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose a new approach to task selection for meta-reinforcement learning. Previous studies mainly focus on task difficulty, regret, or tasks on which the model performs poorly. However, the authors argue that these criteria do not directly indicate which tasks can help the model learn to adapt more effectively. To address this problem, the authors propose a new framework called Paired Adaptation-Contrast Ecology (PACE). Specifically, they introduce a metric called AdaptGain, which evaluates each task instance through two matched executions. One execution allows episode-local adaptation, while the other keeps the learner's local state fixed. AdaptGain is calculated as the difference between the post-adaptation return and the static return. Furthermore, the authors combine AdaptGain with additional components, including a performance frontier, static-saturation penalty, and task diversity, to select tasks that benefit from adaptation while maintaining diversity. Finally, the authors conduct extensive experiments across different environments and meta-RL learners to evaluate the effectiveness of their proposed framework.

Critical strengths

1: The proposed method provides an interesting approach to directly measuring task suitability for meta-RL, which is an important problem in meta-learning.

2: The AdaptGain metric is an interesting design. The authors use paired adaptive and static executions to evaluate how much each task benefits from episode-local adaptation.

3: The experiments are comprehensive, including comparisons across different meta-RL methods, multiple baselines, and ablation studies to evaluate the effectiveness of the proposed framework.

Critical weaknesses

1: As the authors discuss in the paper, AdaptGain does not directly measure how much a task contributes to improving the learner's adaptation ability. A task's dependence on adaptation does not necessarily indicate its value for future training.

2: The necessity of the additional components in PACE is questionable. The manually designed weights lack sufficient theoretical justification, and it is unclear whether these components provide meaningful performance improvements over using AdaptGain alone.

3: The performance improvements achieved by PACE are relatively marginal in some experimental settings, raising questions about the practical significance of the proposed framework.

Questions for authors

1: Can the authors provide more details about the final PACE framework, especially how the different components are combined and how their weights are determined? The current explanation is somewhat unclear.

2: Can the authors provide a clearer analysis of how AdaptGain helps improve the learner's adaptation ability? In particular, how does adaptation dependence relate to the actual training value of a task?

3: Can the authors provide a more detailed analysis of the computational cost of PACE? The proposed method requires additional paired adaptive and static executions, which may introduce substantial overhead compared with existing task-selection methods.

Reviewer_w4KcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks which meta-RL tasks genuinely benefit from episode-local adaptation. It introduces AdaptGain, a paired intervention that measures the return gap between adaptive and static executions of the same learner on the same task instance, and embeds it in a closed-loop task selector called PACE. The method combines positive AdaptGain with a Post-frontier score, a static-saturation penalty, and descriptor diversity, and it uses graded measurement budgets with freshness penalties for variable task spaces. Experiments across several meta-RL learners and environments show that high-AdaptGain tasks yield larger post-adaptation improvements than matched low-AdaptGain tasks, and PACE often outperforms frontier-based baselines with substantially fewer paired evaluations.

Critical strengths
  1. The paired design reduces variance and comes with finite-sample guarantees, including a graded measurement budget that cuts evaluation cost. It shows that AdaptGain carries information beyond post-adaptation return.
  2. The method applies across several meta-RL architectures, including recurrent, in-context, gradient-based, and belief-style learners, demonstrating its effectiveness.
Critical weaknesses
  1. The closed-loop selector has several hyperparameters and design choices, including weighting terms, freshness penalties, mutation scales, and graded thresholds, which may require careful tuning across environments.
  2. Positive AdaptGain does not always translate into better curriculum value in high-noise settings, so the selector can still pick tasks that are noisy rather than instructive.
Questions for authors

In high-noise regimes, how should the system decide whether a positive AdaptGain is worth selecting?

Comments on the paper

1 comment

We thank the reviewers for their thoughtful and constructive feedback. Building on the original matched-Post experiment, we have added noise-stratified retraining analyses and intervention controls to further examine the training relevance of AdaptGain. In the matched-Post experiment, task groups are closely matched on initial Post and retrained from the same checkpoint under equal budgets. Across 12 learner–environment settings, the high-AdaptGain groups achieve an average held-out Post advantage of +0.00693 over the matched low-AdaptGain groups (Tables 2 and 9). The state-matched and native-suffix controls further show that the adaptation contrast persists under alternative evaluation protocols (Table 12).

We have also evaluated PACE under equal total interaction budgets, including the cost of teacher evaluation. PACE improves over the Post frontier by +0.0011 in the standard setting and +0.0012 under high drift, while the high-noise comparison shows no clear advantage (Table 6). The noise-stratified retraining analyses show smaller matched-group contrasts at higher noise levels, while the task-selection results show fewer selections from higher-noise strata in a balanced candidate pool (Tables 13, 18, and 19). These findings provide evidence for the training relevance of adaptation dependence while clarifying the modest gains and limitations of using this signal for task selection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.