acceptodds
Under review as a conference paper at ICLR 2027

Task diversity produces systematic transfer but inhibits continual reinforcement learning

Abstract

Continual reinforcement learning (RL) aims to produce agents that never stop adapting to new tasks. A key question is how this interacts with the diversity of tasks an agent experiences. Prior work has shown that training on many diverse tasks leads to agents with strong zero-shot and in-context adaptation. However, this work evaluated agents after they'd stopped learning, i.e. with frozen weights. How task diversity effects agent's ability to continue learning over a sequence of distribution shifts remains unclear. We introduce Banyan, a GPU-accelerated continual RL domain where one can parametrically control three independent axes that define a task: the map layouts an agent must navigate, the objects it must interact with, and the hierarchical structures of sub-goal dependencies. We find that increasing diversity along each axis induces systematic transfer—that is, agents begin training on a new task distribution near the performance attained on the previous one, even when the shift changes the structure of the optimal policy. While increasing diversity improves systematic transfer, we find that too much diversity inhibits a learner's ability to continue adapting to new task distributions. As diversity increases, learners plateau in the success rate they achieve on new tasks, yet continue improving on old tasks—even without further exposure to them. We find this phenomenon manifests across continual learning algorithms, memory architectures, architecture sizes, and in Kinetix—a physics-based control domain. We release Banyan as a domain for running controlled experiments that study continual RL in the many-tasks regime.

Primary area:
reinforcement learning
Keywords:
continual learning, transfer learning, systematicity, systematic generalization, continual reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CDHgOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a new benchmark suite (Banyan), which helps researchers study how task diversity and scale influence forward and backward transfer in continual RL problems.

The paper then uses the suite across off policy and on policy deep RL algorithms (PQN and PPO resp.), model architectures (RNN, Transformer), scale, and other factors, to empirically study transfer as scale grows.

The surprising result of the paper is that in the continual RL setting, simply increasing scale does not directly transfer to better specialization performance on new tasks (forward transfer), but does mildly help on the initially-trained task distribution (backward transfer). These result are then also demonstrated on other existing benchmarks for continuous control problems (PointMass and Kinetix).

Critical strengths
  • Banyan's design. Tasks with task layouts, how objects interact with each other (topology), and for each topology, which object is assigned a role needed to achieve the task. This enables two important properties for the kind of empirical research the paper proposes: (1) it is fairly easy to generate a very large pool tasks and (2) these tasks, by design share key abstractions yet are distinct, therefore requiring non-trivial generalization.
  • Except for the small number of seeds, the paper's empirical analysis and conclusions are sound and interesting.
  • Overall I enjoyed reading the paper -- the core results are relevant and novel to my understanding.
Critical weaknesses
  • Several key results include only three seeds. My estimation might be wrong, but the experiments are do not involve massive-scale training, therefore I don't see a good justification for running only three seeds. In Figure 6 for instance, it seems that some error-bars overlap quite significantly -- will running more seeds change the conclusion?
  • Some evidence that the tasks are sound would help improve the paper. While I generally tend to believe that all tasks generated by Banyan are solvable, some parts of the supplements hint otherwise. For instance, the layout paragraph in A.1 suggests that not all goals are necessarily reachable. I also believe that even if some tasks are not feasible, we would still observe the same results, however it would make the paper more transparent and Banyan more useful for future researchers.
  • On a similar note, task pools with deeper topologies ensure that shallower topologies also exist (as a "curriculum"), which could be seen as some form of a confounding factor. One interesting generalization question that arises here: given a large pool of "shallow" tasks, can agents generalize fast to "deeper" tasks?
  • Using TD-error as a way to understand better the learned representations might not be the correct metric. TD-error allow us to measure how consistent the critic with its own predictions, while not necessarily measuring the true error of the learned critic. Since you massively-parallelize in these experiments, a better metric could be to compare critic predictions with a large MC return estimate (i.e 128 rollouts). Additionally, other metrics for representation learning in this context could be relevant [1, 2]. This point also relates to
  • The point above also leads me to the conclusion, where it writes: "We hypothesize that this may be a continual learning version of the “primacy bias” (Nikishin et al., 2022), where the policy an agent learns is biased by the features it first learned to represent". I agree with the reference and think it's appropriate. At the same time, it also highlights the main weakness of the paper: it highlights a (surprising!) artifact of scaling in continual RL problems, however it doesn't: (1) go deeper in pinpointing precisely the problem or (2) proposing a solution. This is not to say that the paper's contribution is not valid on its own, however I believe additional analysis towards these questions would improve the paper. For instance, running resets as proposed by Nikishin seems like an immediate next step that could fit the paper well.

[1] Moalla, Skander, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. "No representation, no trust: connecting representation, collapse, and trust issues in ppo." Advances in Neural Information Processing Systems 37 (2024): 69652-69699.

[2] Tang, Hongyao, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Glen Berseth. "Mitigating plasticity loss in continual reinforcement learning by reducing churn." arXiv preprint arXiv:2506.00592 (2025).

Questions for authors
  • In Figure 2, how do you explain that for 1K and 10K task pool, performance seems to increase upon shifting to d2d_2, which overall seems counter-intuitive?

I'm generally leaning towards increasing the score. The following points would be useful:

  • More seeds. Seems like an easy fix. Please let me know if I'm wrong.
  • Any additional evidence that will help future research to come up with solutions to the problem.
  • Additional evidence that the tasks produced by Banyan are feasible.
4 comments
AuthorsOpenReview ↗

Response 1/3

Several key results include only three seeds. My estimation might be wrong, but the experiments are do not involve massive-scale training, therefore I don't see a good justification for running only three seeds. In Figure 6 for instance, it seems that some error-bars overlap quite significantly -- will running more seeds change the conclusion?

Thank you for raising this. We agree that three seeds is a limitation, and our cluster did not allow us to run more during the rebuttal period. Although each run is small, the runs add up: at three seeds per condition, the paper's 204204 runs took about 407407 GPU-hours, and ten seeds would take about 1,3551{,}355. We added Appendix D.4, which lists the runs and GPU-hours behind each figure.

Some evidence that the tasks are sound would help improve the paper. While I generally tend to believe that all tasks generated by Banyan are solvable, some parts of the supplements hint otherwise. For instance, the layout paragraph in A.1 suggests that not all goals are necessarily reachable. I also believe that even if some tasks are not feasible, we would still observe the same results, however it would make the paper more transparent and Banyan more useful for future researchers.

Thank you for pointing this out. Banyan's task generator rejects any task that a breadth-first search (BFS) oracle cannot solve, but we did not state this in the paper. Without this rejection, the oracle could not solve about 2%2\% of sampled tasks, all at depths 44 to 66. Every task in our experiments is therefore solvable. We added this to Appendix A.1, which now reads:

The task generator rejects any task that a breadth-first search (BFS) oracle cannot solve, so every task in our experiments is solvable.

On a similar note, task pools with deeper topologies ensure that shallower topologies also exist (as a "curriculum"), which could be seen as some form of a confounding factor. One interesting generalization question that arises here: given a large pool of "shallow" tasks, can agents generalize fast to "deeper" tasks?

Thank you for this question; it let us test whether shallow tasks alone prepare an agent for deeper ones. We ran this experiment and found that a pool of shallow tasks enables generalization to deeper tasks: PPO trained first on depths 11 to 55 and then on depth 66 ends at a success rate of 0.360.36, while PPO trained on depth 66 from scratch for the same number of steps ends at 0.000.00. We added Appendix E.3, which reads:

Shallow-to-deep transfer

Every distribution in the main text holds the shallower depths alongside the deepest one, so we ask whether shallow tasks alone prepare an agent for deeper ones. For k=1,…,5k = 1, \dots, 5, we train PPO for 10810^8 steps on a distribution holding only depths 11 to kk, then for 10810^8 steps on depth k+1k+1 alone, and compare it with PPO trained on depth k+1k+1 from scratch for 10810^8 steps. Training on the shallow depths first makes the deeper depths learnable where training from scratch fails (Figure 19): at depths 44, 55 and 66, the pool-pretrained agent ends at success 0.760.76, 0.600.60 and 0.360.36, against 0.260.26, 0.010.01 and 0.000.00 from scratch. At depth 66 it starts at 0.220.22 before any training on depth 66, above where training from scratch ends.

Success rate on the deeper depth, mean ±\pm standard deviation across 33 seeds. "At the shift" is the pool-pretrained agent run on the deeper depth with its parameters frozen, before any training on it.

Shallow pool →\to deeper depthAt the shiftEnd, after the shallow poolEnd, from scratch
{1}→2\{1\} \to 20.04±0.010.04 \pm 0.011.00±0.001.00 \pm 0.001.00±0.001.00 \pm 0.00
{1,2}→3\{1, 2\} \to 30.53±0.060.53 \pm 0.061.00±0.001.00 \pm 0.000.65±0.330.65 \pm 0.33
{1,2,3}→4\{1, 2, 3\} \to 40.61±0.040.61 \pm 0.040.76±0.030.76 \pm 0.030.26±0.240.26 \pm 0.24
{1,…,4}→5\{1, \dots, 4\} \to 50.38±0.020.38 \pm 0.020.60±0.050.60 \pm 0.050.01±0.020.01 \pm 0.02
{1,…,5}→6\{1, \dots, 5\} \to 60.22±0.030.22 \pm 0.030.36±0.070.36 \pm 0.070.00±0.000.00 \pm 0.00

We want to highlight that in every experiment of the paper, each task pool has an equal number of tasks at each depth. Section 4 states this:

Every distribution we use spans depths 11 through 66, and we require the depths to nest: every task at depth ll also appears as a sub-task inside some task at every deeper level. … A distribution with nn topologies has 6n6n tasks in total but effective diversity nn.

Appendix D.2 gives the construction:

Each distribution contains nn depth-66 task trees and nn layouts, giving n2n^2 depth-66 task–layout combinations. Each tree also contributes one nested subtask at each depth from 11 through 55, extracted along a longest root-to-leaf path.

AuthorsOpenReview ↗

Response 2/4

Using TD-error as a way to understand better the learned representations might not be the correct metric. TD-error allow us to measure how consistent the critic with its own predictions, while not necessarily measuring the true error of the learned critic. Since you massively-parallelize in these experiments, a better metric could be to compare critic predictions with a large MC return estimate (i.e 128 rollouts). Additionally, other metrics for representation learning in this context could be relevant [1, 2].

This is an excellent point, and it gave us a direct measure of the critic's zero-shot error. We re-ran the analysis using Monte-Carlo returns and found the same pattern: more diversity lowers the critic's zero-shot error on the held-out distribution d2d_2 (from 0.800.80 at one topology to 0.210.21 at 256256). For layouts and object assignments, more diversity also lowers the variance of this error across seeds (Figure 5).

We replaced Figure 5 and updated Section 4.1, which reads:

One simple hypothesis is that diversity produces better representations, and better representations produce better predictions. We test the second half: we freeze the PPO critic at the end of d1d_1 and compare its predictions with Monte-Carlo returns, averaged over 128128 rollouts run to episode termination, on d1d_1 and on d2d_2. More diversity in d1d_1 lowers the critic's zero-shot error on d2d_2 along every axis (Figure 5): from 0.450.45 at ∣L∣=1|\mathbb{L}| = 1 to 0.120.12 at ∣L∣=10|\mathbb{L}| = 10K, from 0.370.37 at ∣O∣=1|\mathbb{O}| = 1 to 0.150.15 at ∣O∣=10|\mathbb{O}| = 10K, and from 0.800.80 at ∣T∣=1|\mathbb{T}| = 1 to 0.210.21 at ∣T∣=256|\mathbb{T}| = 256. This lower zero-shot error may be why more diversity closes the transfer gap in success (Figure 3).

AuthorsOpenReview ↗

Response 3/4

The point above also leads me to the conclusion, where it writes: "We hypothesize that this may be a continual learning version of the “primacy bias” (Nikishin et al., 2022), where the policy an agent learns is biased by the features it first learned to represent". I agree with the reference and think it's appropriate. At the same time, it also highlights the main weakness of the paper: it highlights a (surprising!) artifact of scaling in continual RL problems, however it doesn't: (1) go deeper in pinpointing precisely the problem or (2) proposing a solution. This is not to say that the paper's contribution is not valid on its own, however I believe additional analysis towards these questions would improve the paper. For instance, running resets as proposed by Nikishin seems like an immediate next step that could fit the paper well.

Thank you for appreciating this connection and for suggesting resets as a next step. We arrived at the primacy bias only after ruling out four other explanations for the plateau. We rewrote the Discussion to lay these out, and its second paragraph now reads:

We considered several explanations for this plateau. One is loss of plasticity: the network gradually loses its ability to fit new data. If so, continual backprop, which re-initializes low-utility units to counter loss of plasticity, should remove the plateau, but it does not (Figure 8). Another is interference: what the agent learned on earlier distributions conflicts with what each new one requires. If so, giving each distribution its own output head, or continuing to train on earlier distributions, should remove the plateau, but neither does (Figure 8). A third is that our recurrent architecture or network size is too limited. If so, a Transformer-XL agent or a larger network should keep improving, but both plateau too (Figure 9; Figure 26). A last is that the plateau is specific to Banyan. If so, it should not appear in other domains, but in Kinetix, a domain developed independently of our study, learning also stalls at high diversity, and backward transfer from d10d_{10} to d1d_1 also improves with diversity (Figure 10).

We also added Figure 1b, which supports the primacy bias. We could not get the compute to test resets during the rebuttal. Resets would also need adapting to our agents: Nikishin et al. (2022) relearn the reset layers from a replay buffer, and our on-policy agents keep none. The third paragraph of the Discussion now reads:

The hypothesis we are left with is that the plateau is a continual RL version of the primacy bias (Nikishin et al., 2022): early experience shapes the network so strongly that later data mostly refines what it learned first. Figure 1b supports this. Across d2d_2 to d10d_{10}, the agent trained on the most diverse data keeps learning, but what it learns improves the first distribution rather than the new ones. With no further training on d1d_1, its success on d1d_1 rises from 0.770.77 after d1d_1 to 0.930.93 after d10d_{10}, against 0.690.69 to 0.720.72 at n2=162n^2 = 16^2. An agent that had lost the ability to learn would not keep improving on d1d_1. The primacy bias may also explain why continual backprop does not help: it replaces units that contribute little, but the units that encode what the agent learned first might contribute the most. Future work can test this hypothesis by shrinking and perturbing all weights (Ash & Adams, 2020), a soft reset that Juliani & Ash (2024) found effective in on-policy agents.

AuthorsOpenReview ↗

Response 4/4

In Figure 2, how do you explain that for 1K and 10K task pool, performance seems to increase upon shifting to d2d_2, which overall seems counter-intuitive?

Thank you for catching this. The increase was not real; it was an artifact of how we smoothed the curves.

We smoothed each curve with a running average that only looked backward, so on a rising curve, the drawn line lagged the data. For PPO, this drew the end of d1d_1 0.150.15 to 0.180.18 below the agent's measured final success, which created a jump at the shift that is not in the data. PPO curves are now unsmoothed. PQN logs about four times as many points and is noisier, so we smooth it only within each distribution, which leaves the values at each shift exact.

With these fixes, success no longer jumps at the shift. With 11K or 1010K layouts or object assignments, the agent starts d2d_2 within 0.050.05 of where it ended d1d_1, for both PPO and PQN. For example, with 1010K layouts PPO ends d1d_1 at 0.470.47 and starts d2d_2 at 0.480.48, and with 1010K object assignments it ends at 0.530.53 and starts at 0.530.53. We also rewrote the forward-transfer paragraph of Section 4.1 to report these numbers, which now reads:

Across all three axes and for both PPO and PQN, increasing diversity in d1d_1 closes the transfer gap (Figure 3; Figure 12). With layout count ∣L∣=1|\mathbb{L}| = 1, PPO ends d1d_1 at 0.540.54 success but begins d2d_2 at 0.110.11; with ∣L∣=100|\mathbb{L}| = 100, it ends d1d_1 at 0.500.50 and begins d2d_2 at 0.480.48. Object assignment count ∣O∣|\mathbb{O}| shows the same pattern (0.570.57 to 0.170.17 at ∣O∣=1|\mathbb{O}| = 1, 0.520.52 to 0.510.51 at ∣O∣=100|\mathbb{O}| = 100), and so does topology count ∣T∣|\mathbb{T}| (0.500.50 to 0.200.20 at ∣T∣=1|\mathbb{T}| = 1, 0.630.63 to 0.630.63 at ∣T∣=16|\mathbb{T}| = 16).

We updated Figure 3 and Figure 12, and added unsmoothed PQN curves to Appendix D.3 (Figure 11).

Reviewer_aGgXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work introduces Banyan, a symbolic grid environment designed to systematically test how generalization and continual learning under distribution shifts are affected by task diversity. Banyan has three axes across which diversity can be controlled independently: the map layout, the objects involved, and the compositional task structure. Agents first train on populations with varying diversity and are then evaluated and/or continually trained on held-out or subsequently introduced task populations. The authors find that greater diversity improves immediate transfer but does not necessarily improve continual learning: high-diversity agents begin new distributions with stronger performance but often plateau, whereas agents trained with intermediate diversity continue adapting and ultimately perform better. Meanwhile, high-diversity agents may continue improving on previously encountered tasks without revisiting them. The authors interpret this as evidence that updates from current experience increasingly favor earlier task distributions while interfering with improvement on the current one, an explanation supported by gradient-alignment analysis.

Critical strengths
  • Novelty: The paper characterizes a counterintuitive tension between systematic transfer and continual learning. It distinguishes this phenomenon from catastrophic forgetting and general loss of plasticity, and provides gradient-alignment evidence consistent with its interpretation. Banyan also serves as a useful benchmark contribution, offering independent control over three diversity axes.
  • Rigor and breadth: The findings are tested across policy- and value-based algorithms (PPO and PQN), memory architectures, network capacities, and several continual-learning interventions. The single-shift transfer result extends to PointMass and the multi-shift trade-off is reproduced in Kinetix. Means and standard errors are reported.
Critical weaknesses
  • Potentially ambiguous terminology: The paper defines a task distribution as a distribution over Banyan tasks, but in the first experiment d1 and d2 appear to be populations drawn from the same underlying task-generation process. The authors should clarify whether the procedural generators differ and distinguish an empirical support shift from a generator OOD shift. Likewise, a small gap between performance on d1 and initial performance on d2 may partly reflect small differences in baseline difficulty of the two task populations rather than transfer alone (but this does not undermine relative comparisons across diversity conditions since they evaluate on the same d2; it just makes sentences like "leads agents to begin d2 close to their performance on d1" potentially confusing).
  • The novelty claim (L57-L62 and L127-129) about the environment itself appears overstated without a fuller comparison to existing benchmarks. For example, Avalon [1] provides procedurally generated tasks with multiple controllable sources of variation for studying RL generalization. The paper would benefit from a clearer explanation of why Banyan was designed for this analysis rather than using an existing benchmark such as Avalon.

[1] https://arxiv.org/abs/2210.13417

Questions for authors
  • As a complement to the exploration-bonus experiment, did you consider a dense reward or reward shaping to test whether the observed plateau is partly attributable to long-horizon credit assignment under sparse rewards?
3 comments
AuthorsOpenReview ↗

Response 1/3

Potentially ambiguous terminology: The paper defines a task distribution as a distribution over Banyan tasks, but in the first experiment d1 and d2 appear to be populations drawn from the same underlying task-generation process. The authors should clarify whether the procedural generators differ and distinguish an empirical support shift from a generator OOD shift.

Thank you for pointing out this ambiguity. You are right that d1d_1 and d2d_2 come from the same task generator. The shift is therefore an empirical support shift, not a generator shift. Section 4 now reads:

A task distribution dd is a distribution over Banyan tasks τ=(L,T,O)\tau = (\mathbb{L}, \mathbb{T}, \mathbb{O}). […] In our experiments, task distributions come from the same task generator but differ in their empirical support, i.e. the set of tasks they contain.

What makes this an interesting transfer problem is how little of the task space each distribution covers. Just counting topologies, our generator samples fewer than one in a billion tasks in the task space. This is like learning to recognize dogs from a handful of photos. Section 4 also now has

Transfer between distributions is challenging because each distribution covers fewer than one in a billion of the possible tasks Banyan can sample (Table 1).

Likewise, a small gap between performance on d1 and initial performance on d2 may partly reflect small differences in baseline difficulty of the two task populations rather than transfer alone.

Thank you for this comment. Since this gap compares two task populations that are random draws from the same generator, we expect difficulty differences to be small. We were unclear that the "transfer gap" was comparing zero-shot performance and have updated Figure 3 and added Figure 1 to express that more clearly.

Figure 3 updated text:

Increasing task diversity in d1d_1 raises zero-shot success on held-out d2d_2 toward the success reached on d1d_1. We vary the number of layouts L\mathbb{L}, object assignments O\mathbb{O}, or topologies T\mathbb{T} in d1d_1.

AuthorsOpenReview ↗

Response 2/3

The novelty claim (L57-L62 and L127-129) about the environment itself appears overstated without a fuller comparison to existing benchmarks. For example, Avalon [1] provides procedurally generated tasks with multiple controllable sources of variation for studying RL generalization. The paper would benefit from a clearer explanation of why Banyan was designed for this analysis rather than using an existing benchmark such as Avalon.

Thank you for pointing us to Avalon, which helped us see how we could better position Banyan. Our goal with Banyan is to study continual RL over millions of tasks that share their dynamics. Shared dynamics is a familiar setting: putting a pot of water on a stove boils the water, whatever dish you are cooking.

We did not use Avalon mainly because of speed. Training on millions of tasks needs a GPU-accelerated domain, and Banyan runs about ten times faster than Avalon. Avalon also has 20 distinct tasks: procedural generation varies the conditions of each task, but there is no parameter for generating new tasks. The GPU-accelerated domains we looked at do not let us vary task diversity in a systematic, controlled way. Craftax has a single task tree, and XLand-MiniGrid redraws its dynamics for every task.

We updated the related work to compare against Avalon, Appendix F to compare speed and task variation with other GPU-accelerated domains, and Section 3 to show the size of Banyan's task space. Related work updated text:

Avalon (Albrecht et al., 2022) offers visually rich 3D worlds with 20 distinct tasks, whose conditions vary through procedural generation. Banyan trades visual richness for about ten times the speed and independent control over each kind of diversity, including the task itself, and can generate at least 8.8×10188.8 \times 10^{18} tasks (Table 1; Appendix F).

Appendix F new text on speed:

Speed. We measured simulation throughput for Banyan, Craftax, Craftax-Classic, and XLand-MiniGrid on one H100 GPU, and compare against Avalon's published rate (Table 3). Avalon's 3D worlds are far richer visually than Banyan's grid, but Avalon simulates each world in its own process, while Banyan simulates every environment in parallel on the GPU. Banyan is therefore about ten times faster than Avalon, and it varies layouts, object assignments, and topologies independently. XLand-MiniGrid is about three times faster than Banyan but expresses only 2,2222{,}222 task-tree topologies, against Banyan's 2.9×10262.9 \times 10^{26} (Table 2).

Table 3: Simulation throughput. Steps per second on one H100 GPU with 1,0241{,}024 parallel environments and random actions. Avalon's number is its published multi-process rate on one 2080 Ti GPU (Albrecht et al., 2022), so it is not measured the same way.

DomainSteps per second
Avalon≈7,000\approx 7{,}000
Craftax14,55214{,}552
Craftax-Classic35,36835{,}368
Banyan72,38972{,}389
XLand-MiniGrid215,068215{,}068

Appendix F new text on task variation:

Banyan can express 2.9×10262.9 \times 10^{26} task-tree topologies, against 2,2222{,}222 for XLand-MiniGrid (Nikulin et al., 2024) and one for Craftax (Matthews et al., 2024) (Table 2). Its trees reach depth 6, and each node transforms one object or combines two or three. Craftax also fixes its dynamics but has a single tree, of depth 11. XLand-MiniGrid samples a new tree per task but also redraws the dynamics, which suits meta-learning, where the agent infers each task's dynamics within an episode, but not continual learning, where the agent carries what it learned into new tasks.

Table 2: Number of task-tree topologies each domain can express, by depth. Depth counts the objects on the longest leaf-to-root path.

DepthCraftaxXLand-MiniGridBanyan
1011
2023
30731
40568,4018{,}401
502,1562{,}1561.0×10111.0 \times 10^{11}
6002.9×10262.9 \times 10^{26}
11100
Total12,2222{,}2222.9×10262.9 \times 10^{26}

Section 3 updated text:

Banyan is a GPU-accelerated MiniGrid-style (Chevalier-Boisvert et al., 2023) domain for studying continual RL over large compositional task spaces.

[…]

Size of the task space. Banyan can sample from at least 8.8×10188.8 \times 10^{18} tasks (Table 1). A task combines a depth-6 topology, a layout, and an object assignment, so the count is their product: 3.7×10113.7 \times 10^{11} topologies, 2.4×1072.4 \times 10^{7} layouts of the 6×66 \times 6 grid, and at least one object assignment per topology.

Unique
Depth-6 topologies3.7×10113.7 \times 10^{11}
Layouts ×\times objects ×\times topologies≥8.8×1018\geq 8.8 \times 10^{18}
AuthorsOpenReview ↗

Response 3/3

As a complement to the exploration-bonus experiment, did you consider a dense reward or reward shaping to test whether the observed plateau is partly attributable to long-horizon credit assignment under sparse rewards?

Thank you for this question. All of our experiments use reward shaping: the agent earns a small bonus the first time it picks up each object the task needs, and pays a small cost per step. We have added this to the main text.

Section 4 updated text:

All experiments used shaped reward to ease credit assignment over long horizons where the agent receives +0.1+0.1 the first time it picks up each object in the task tree and −0.001-0.001 per step (Appendix A.1).

Reviewer_XQFZOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper makes two contributions. The first contribution is a simulation environment for continual RL, which allows the user to systematically vary task diversity across 3 dimensions: task layout (that is, the navigation map), task structure, and object grounding. The second contribution is a set of experiments in this domain (augmented by experiments in other domains) that explores how task diversity impacts continual learning.

Critical strengths

A new experimental domain can be useful for increasing our scientific understanding, and this is true for the one contributed by the authors. This is a MiniGrid-style domain that allows compositional task structure. There are already a large number of simulation domains in the literature, including those that support the scientific exploration of continual learning. The specific strength of the domain developed here is that it allows users to isolate the amount and type of diversity within a relatively large number of possible tasks.

Critical weaknesses

The domain developed here is relatively simple and the task diversity is naturally limited to what can be supported within its particular structure.

The experiments on continual learning do not introduce important new insights. It is well established that training on diverse tasks improves generalization in downstream tasks; and the authors refer sufficiently to earlier findings. The main novelty claim in the paper is the exploration of the setting in which learning continues after the shift to a new task distribution. However, I do not see that this setting introduces significant new insights. The authors conclude that the highest levels of task diversity leads to reduced learning over the full sequence. I find this conclusion counterintuitive, if true. It deserves a deeper exploration than what is provided in the paper. Most importantly, alternative explanations should be ruled out. One such explanation is that the task distribution itself is more difficult under the highest levels of task diversity and that the authors' conclusion is an artifact of the difficulty of the tasks on which the agent is evaluated rather than the impact of task diversity itself.

I do not follow some of the discussion of the experimental results. These should be explained more clearly and by making explicit reference to the specific evidence that support these claims. Examples: Lines 256-259: "Layout count |L| and object assignment count |O| qualitatively have the same effect on reducing the transfer gap, whereas topology count |T| seems to have a different effect: learning seems to benefit strongly from increasing topology count. We attribute this asymmetry to [..]" Looking at the plots in figure 10, I do not follow the authors' comments: What specifically is the asymmetry? And what is the specific evidence supporting this conclusion? Lines 265-267: "Neither layout count |L| nor object assignment count |O| visibly affects backward transfer across PPO and PQN." I am not following how the authors have reached this conclusion; Figure 3 shows differences.

Questions for authors

I don't have any questions for the authors but I would welcome their clear and succinct description of how the experiments in the paper changed their own thinking about continual reinforcement learning.

2 comments
AuthorsOpenReview ↗

Response 1/2

The domain developed here is relatively simple and the task diversity is naturally limited to what can be supported within its particular structure.

Thank you for raising this. We designed Banyan's dynamics to be simple so that we could control each kind of diversity independently and train on millions of tasks quickly. Simple dynamics can still produce complex tasks. Each task is a recipe tree, and its depth is the number of objects on the longest path from a starting object to the goal. We added Figure 18, which shows that success falls as depth rises. At the end of the first distribution of 2562256^2 tasks, PPO solves every depth-1 and depth-2 task (success 1.001.00), but its success drops to 0.640.64 at depth 4 and 0.460.46 at depth 6.

Banyan's structure also supports a large range of tasks. We added Appendix F, which shows that Banyan can express 2.9×10262.9 \times 10^{26} task-tree topologies (recipe-tree shapes), against 2,2222{,}222 for XLand-MiniGrid and one for Craftax.

We also see our main result outside Banyan. In Kinetix, a physics-based domain developed independently of our study, learning also stalls at high diversity (Figure 10).

The main novelty claim in the paper is the exploration of the setting in which learning continues after the shift to a new task distribution. However, I do not see that this setting introduces significant new insights. The authors conclude that the highest levels of task diversity leads to reduced learning over the full sequence. I find this conclusion counterintuitive, if true. It deserves a deeper exploration than what is provided in the paper. Most importantly, alternative explanations should be ruled out. One such explanation is that the task distribution itself is more difficult under the highest levels of task diversity and that the authors' conclusion is an artifact of the difficulty of the tasks on which the agent is evaluated rather than the impact of task diversity itself.

Thank you for suggesting this alternative explanation! We believe our data already refute the hypothesis that continual RL is hurt because the task distribution is more difficult at higher diversity. We added Figure 1 to help illustrate this point. Figure 1 contrasts zero-shot transfer after one distribution shift with learning after nine. If harder tasks inhibited continual RL, higher diversity would lower success from the start. Instead, higher diversity leads to (1) higher success at the end of d1d_1, (2) higher zero-shot success on d2d_2, and (3) still lower success at the end of d10d_{10} (table below). Higher success on d1d_1 and on d2d_2 before any training on it means that diversity makes learning a task distribution easier and more generalizable, not harder.

I do not follow some of the discussion of the experimental results. These should be explained more clearly and by making explicit reference to the specific evidence that support these claims.

Thank you for helping us better present our results. We have rewritten several paragraphs in Section 4 to more clearly reference the numbers in the experiments.

The forward-transfer paragraph (your first example) now reads:

Across all three axes and for both PPO and PQN, increasing diversity in d1d_1 closes the transfer gap (Figure 3; Figure 13). With layout count ∣L∣=1|\mathbb{L}| = 1, PPO ends d1d_1 at 0.540.54 success but begins d2d_2 at 0.110.11; with ∣L∣=100|\mathbb{L}| = 100, it ends d1d_1 at 0.500.50 and begins d2d_2 at 0.480.48. Object assignment count ∣O∣|\mathbb{O}| shows the same pattern (0.570.57 to 0.170.17 at ∣O∣=1|\mathbb{O}| = 1, 0.520.52 to 0.510.51 at ∣O∣=100|\mathbb{O}| = 100), and so does topology count ∣T∣|\mathbb{T}| (0.500.50 to 0.200.20 at ∣T∣=1|\mathbb{T}| = 1, 0.630.63 to 0.630.63 at ∣T∣=16|\mathbb{T}| = 16).

By "neither layout count nor object assignment count visibly affects backward transfer", we meant that backward transfer did not depend on diversity along those two axes. Training on d2d_2 raises success on d1d_1 at every layout and object-assignment count, but success on d1d_1 after d2d_2 rises with diversity only when we vary topologies. We now use one y-axis across all panels so this is visible, and its caption states the result. We clarified the backward-transfer paragraph (your second example) to now read:

When we vary layout count ∣L∣|\mathbb{L}| or object assignment count ∣O∣|\mathbb{O}|, training on d2d_2 always raises success on d1d_1; only when we vary topology count ∣T∣|\mathbb{T}| does this gain depend on diversity (Figure 4; Figure 21). Success on d1d_1 after d2d_2 lands at about the same level for every ∣L∣|\mathbb{L}| and ∣O∣|\mathbb{O}| from 11 to 1010K: in the 0.630.63–0.670.67 range for ∣L∣|\mathbb{L}| and the 0.660.66–0.690.69 range for ∣O∣|\mathbb{O}| with PPO, and in the 0.270.27–0.450.45 range for ∣L∣|\mathbb{L}| and the 0.520.52–0.590.59 range for ∣O∣|\mathbb{O}| with PQN. As ∣T∣|\mathbb{T}| increases, it rises, from 0.450.45 at ∣T∣=1|\mathbb{T}| = 1 to 0.840.84 at ∣T∣=256|\mathbb{T}| = 256 for PPO, and from 0.070.07 to 0.700.70 for PQN.

AuthorsOpenReview ↗

Response 2/2

I don't have any questions for the authors but I would welcome their clear and succinct description of how the experiments in the paper changed their own thinking about continual reinforcement learning.

Thank you for asking us to clarify what we learned from this paper. We made Figure 1 to capture it, and rewrote the Discussion to make the points below, each with references to figures in the paper.

  1. We expected task diversity to improve generalization. We did not expect it to bring zero-shot success on d2d_2 close to success at the end of d1d_1 (Figure 13). Nor did we expect this along every axis of diversity we varied (layouts, object assignments, topologies) and in a continuous-control domain (Figures 3 and 6).

  2. We assumed that better zero-shot generalization would compound as the distribution kept shifting. Instead, the agent trained on the most diverse data improved least, ending with the lowest success on d10d_{10} (Figures 1 and 7). We tested and ruled out four explanations:

    1. Loss of plasticity. Continual backprop, which re-initializes low-utility units, does not remove the plateau (Figure 8).
    2. Interference between distributions. Neither a new output head per distribution nor continued training on earlier distributions removes it (Figure 8).
    3. Our recurrent architecture or network size. We were surprised to see a Transformer-XL agent trained on the same data plateau as well (Figure 9), and so do larger networks (Figure 27).
    4. An artifact of Banyan. Kinetix, a domain developed independently of our study, shows both learning that stalls at high diversity and backward transfer from d10d_{10} to d1d_1 that improves with diversity (Figure 10).

    The hypothesis we are left with is a continual RL version of the primacy bias (Nikishin et al., 2022). Figure 1b supports it: the agent trained on the most diverse data keeps learning, but what it learns improves d1d_1, whose success rises from 0.77 to 0.93 with no further training on d1d_1.

We have updated our discussion to reflect this.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Joint response

We thank the reviewers for their thoughtful and constructive feedback, which we have used to improve the clarity of our paper. We are encouraged that reviewers found Banyan a useful domain that isolates the amount and type of task diversity across a large space of tasks [R1, R2, R3], that our core result is a novel and counterintuitive tension between transfer and continual learning [R2, R3], that our findings hold across algorithms, architectures, network sizes, interventions, and domains [R2], and that our analysis and conclusions are sound and interesting [R3]. We label reviewers as follows: R1: XQFZ, R2: aGgX, R3: CDHg.

We respond to each review below. We made several improvements to the paper:

  1. A new Figure 1 that summarizes our main result. Figure 1a compares one distribution shift with ten. Higher task diversity improves performance on the first distribution and zero-shot generalization to the second, yet lowers final performance after ten shifts. Figure 1b shows that at the highest diversity, the agent keeps learning, but its gains appear as backward transfer to the first distribution rather than as progress on new ones. This figure answers whether higher diversity simply makes tasks harder [R1], what the experiments taught us [R1], and what the transfer gap compares [R2].
  2. A clearer comparison to prior work. We now explain in Related Work how Banyan differs from prior domains such as Avalon [R2]. To make this comparison precise, Section 3 now reports the size of Banyan's task space (at least 8.8×10188.8 \times 10^{18} tasks), and new tables in Appendix F compare Banyan with Avalon, XLand-MiniGrid, and Craftax [R1, R2]. Banyan can express 2.9×10262.9 \times 10^{26} distinct task structures, against 2,2222{,}222 for XLand-MiniGrid and one for Craftax, and it runs about ten times faster than Avalon.
  3. More precise experiment descriptions (Section 4). Each conclusion now cites the figure and the numbers that support it [R1]. We also state that all task distributions come from the same task generator and differ only in which tasks they contain [R2].
  4. An improved Discussion. It lists each explanation we ruled out for why learning plateaus at high diversity, with the figure that rules it out, and the hypothesis that remains: a continual RL version of the primacy bias [R1].

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.