- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a new benchmark suite (Banyan), which helps researchers study how task diversity and scale influence forward and backward transfer in continual RL problems.
The paper then uses the suite across off policy and on policy deep RL algorithms (PQN and PPO resp.), model architectures (RNN, Transformer), scale, and other factors, to empirically study transfer as scale grows.
The surprising result of the paper is that in the continual RL setting, simply increasing scale does not directly transfer to better specialization performance on new tasks (forward transfer), but does mildly help on the initially-trained task distribution (backward transfer). These result are then also demonstrated on other existing benchmarks for continuous control problems (PointMass and Kinetix).
Critical strengths
- Banyan's design. Tasks with task layouts, how objects interact with each other (topology), and for each topology, which object is assigned a role needed to achieve the task. This enables two important properties for the kind of empirical research the paper proposes: (1) it is fairly easy to generate a very large pool tasks and (2) these tasks, by design share key abstractions yet are distinct, therefore requiring non-trivial generalization.
- Except for the small number of seeds, the paper's empirical analysis and conclusions are sound and interesting.
- Overall I enjoyed reading the paper -- the core results are relevant and novel to my understanding.
Critical weaknesses
- Several key results include only three seeds. My estimation might be wrong, but the experiments are do not involve massive-scale training, therefore I don't see a good justification for running only three seeds. In Figure 6 for instance, it seems that some error-bars overlap quite significantly -- will running more seeds change the conclusion?
- Some evidence that the tasks are sound would help improve the paper. While I generally tend to believe that all tasks generated by Banyan are solvable, some parts of the supplements hint otherwise. For instance, the layout paragraph in A.1 suggests that not all goals are necessarily reachable. I also believe that even if some tasks are not feasible, we would still observe the same results, however it would make the paper more transparent and Banyan more useful for future researchers.
- On a similar note, task pools with deeper topologies ensure that shallower topologies also exist (as a "curriculum"), which could be seen as some form of a confounding factor. One interesting generalization question that arises here: given a large pool of "shallow" tasks, can agents generalize fast to "deeper" tasks?
- Using TD-error as a way to understand better the learned representations might not be the correct metric. TD-error allow us to measure how consistent the critic with its own predictions, while not necessarily measuring the true error of the learned critic. Since you massively-parallelize in these experiments, a better metric could be to compare critic predictions with a large MC return estimate (i.e 128 rollouts). Additionally, other metrics for representation learning in this context could be relevant [1, 2]. This point also relates to
- The point above also leads me to the conclusion, where it writes: "We hypothesize that this may be a continual learning version of the “primacy bias” (Nikishin et al., 2022), where the policy an agent learns is biased by the features it first learned to represent". I agree with the reference and think it's appropriate. At the same time, it also highlights the main weakness of the paper: it highlights a (surprising!) artifact of scaling in continual RL problems, however it doesn't: (1) go deeper in pinpointing precisely the problem or (2) proposing a solution. This is not to say that the paper's contribution is not valid on its own, however I believe additional analysis towards these questions would improve the paper. For instance, running resets as proposed by Nikishin seems like an immediate next step that could fit the paper well.
[1] Moalla, Skander, Andrea Miele, Daniil Pyatko, Razvan Pascanu, and Caglar Gulcehre. "No representation, no trust: connecting representation, collapse, and trust issues in ppo." Advances in Neural Information Processing Systems 37 (2024): 69652-69699.
[2] Tang, Hongyao, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, and Glen Berseth. "Mitigating plasticity loss in continual reinforcement learning by reducing churn." arXiv preprint arXiv:2506.00592 (2025).
Questions for authors
- In Figure 2, how do you explain that for 1K and 10K task pool, performance seems to increase upon shifting to , which overall seems counter-intuitive?
I'm generally leaning towards increasing the score. The following points would be useful:
- More seeds. Seems like an easy fix. Please let me know if I'm wrong.
- Any additional evidence that will help future research to come up with solutions to the problem.
- Additional evidence that the tasks produced by Banyan are feasible.
4 comments
Response 2/4
Using TD-error as a way to understand better the learned representations might not be the correct metric. TD-error allow us to measure how consistent the critic with its own predictions, while not necessarily measuring the true error of the learned critic. Since you massively-parallelize in these experiments, a better metric could be to compare critic predictions with a large MC return estimate (i.e 128 rollouts). Additionally, other metrics for representation learning in this context could be relevant [1, 2].
This is an excellent point, and it gave us a direct measure of the critic's zero-shot error. We re-ran the analysis using Monte-Carlo returns and found the same pattern: more diversity lowers the critic's zero-shot error on the held-out distribution (from at one topology to at ). For layouts and object assignments, more diversity also lowers the variance of this error across seeds (Figure 5).
We replaced Figure 5 and updated Section 4.1, which reads:
One simple hypothesis is that diversity produces better representations, and better representations produce better predictions. We test the second half: we freeze the PPO critic at the end of and compare its predictions with Monte-Carlo returns, averaged over rollouts run to episode termination, on and on . More diversity in lowers the critic's zero-shot error on along every axis (Figure 5): from at to at K, from at to at K, and from at to at . This lower zero-shot error may be why more diversity closes the transfer gap in success (Figure 3).
Response 3/4
The point above also leads me to the conclusion, where it writes: "We hypothesize that this may be a continual learning version of the “primacy bias” (Nikishin et al., 2022), where the policy an agent learns is biased by the features it first learned to represent". I agree with the reference and think it's appropriate. At the same time, it also highlights the main weakness of the paper: it highlights a (surprising!) artifact of scaling in continual RL problems, however it doesn't: (1) go deeper in pinpointing precisely the problem or (2) proposing a solution. This is not to say that the paper's contribution is not valid on its own, however I believe additional analysis towards these questions would improve the paper. For instance, running resets as proposed by Nikishin seems like an immediate next step that could fit the paper well.
Thank you for appreciating this connection and for suggesting resets as a next step. We arrived at the primacy bias only after ruling out four other explanations for the plateau. We rewrote the Discussion to lay these out, and its second paragraph now reads:
We considered several explanations for this plateau. One is loss of plasticity: the network gradually loses its ability to fit new data. If so, continual backprop, which re-initializes low-utility units to counter loss of plasticity, should remove the plateau, but it does not (Figure 8). Another is interference: what the agent learned on earlier distributions conflicts with what each new one requires. If so, giving each distribution its own output head, or continuing to train on earlier distributions, should remove the plateau, but neither does (Figure 8). A third is that our recurrent architecture or network size is too limited. If so, a Transformer-XL agent or a larger network should keep improving, but both plateau too (Figure 9; Figure 26). A last is that the plateau is specific to Banyan. If so, it should not appear in other domains, but in Kinetix, a domain developed independently of our study, learning also stalls at high diversity, and backward transfer from to also improves with diversity (Figure 10).
We also added Figure 1b, which supports the primacy bias. We could not get the compute to test resets during the rebuttal. Resets would also need adapting to our agents: Nikishin et al. (2022) relearn the reset layers from a replay buffer, and our on-policy agents keep none. The third paragraph of the Discussion now reads:
The hypothesis we are left with is that the plateau is a continual RL version of the primacy bias (Nikishin et al., 2022): early experience shapes the network so strongly that later data mostly refines what it learned first. Figure 1b supports this. Across to , the agent trained on the most diverse data keeps learning, but what it learns improves the first distribution rather than the new ones. With no further training on , its success on rises from after to after , against to at . An agent that had lost the ability to learn would not keep improving on . The primacy bias may also explain why continual backprop does not help: it replaces units that contribute little, but the units that encode what the agent learned first might contribute the most. Future work can test this hypothesis by shrinking and perturbing all weights (Ash & Adams, 2020), a soft reset that Juliani & Ash (2024) found effective in on-policy agents.
Response 4/4
In Figure 2, how do you explain that for 1K and 10K task pool, performance seems to increase upon shifting to , which overall seems counter-intuitive?
Thank you for catching this. The increase was not real; it was an artifact of how we smoothed the curves.
We smoothed each curve with a running average that only looked backward, so on a rising curve, the drawn line lagged the data. For PPO, this drew the end of to below the agent's measured final success, which created a jump at the shift that is not in the data. PPO curves are now unsmoothed. PQN logs about four times as many points and is noisier, so we smooth it only within each distribution, which leaves the values at each shift exact.
With these fixes, success no longer jumps at the shift. With K or K layouts or object assignments, the agent starts within of where it ended , for both PPO and PQN. For example, with K layouts PPO ends at and starts at , and with K object assignments it ends at and starts at . We also rewrote the forward-transfer paragraph of Section 4.1 to report these numbers, which now reads:
Across all three axes and for both PPO and PQN, increasing diversity in closes the transfer gap (Figure 3; Figure 12). With layout count , PPO ends at success but begins at ; with , it ends at and begins at . Object assignment count shows the same pattern ( to at , to at ), and so does topology count ( to at , to at ).
We updated Figure 3 and Figure 12, and added unsmoothed PQN curves to Appendix D.3 (Figure 11).
Response 1/3
Several key results include only three seeds. My estimation might be wrong, but the experiments are do not involve massive-scale training, therefore I don't see a good justification for running only three seeds. In Figure 6 for instance, it seems that some error-bars overlap quite significantly -- will running more seeds change the conclusion?
Thank you for raising this. We agree that three seeds is a limitation, and our cluster did not allow us to run more during the rebuttal period. Although each run is small, the runs add up: at three seeds per condition, the paper's 204 runs took about 407 GPU-hours, and ten seeds would take about 1,355. We added Appendix D.4, which lists the runs and GPU-hours behind each figure.
Some evidence that the tasks are sound would help improve the paper. While I generally tend to believe that all tasks generated by Banyan are solvable, some parts of the supplements hint otherwise. For instance, the layout paragraph in A.1 suggests that not all goals are necessarily reachable. I also believe that even if some tasks are not feasible, we would still observe the same results, however it would make the paper more transparent and Banyan more useful for future researchers.
Thank you for pointing this out. Banyan's task generator rejects any task that a breadth-first search (BFS) oracle cannot solve, but we did not state this in the paper. Without this rejection, the oracle could not solve about 2% of sampled tasks, all at depths 4 to 6. Every task in our experiments is therefore solvable. We added this to Appendix A.1, which now reads:
On a similar note, task pools with deeper topologies ensure that shallower topologies also exist (as a "curriculum"), which could be seen as some form of a confounding factor. One interesting generalization question that arises here: given a large pool of "shallow" tasks, can agents generalize fast to "deeper" tasks?
Thank you for this question; it let us test whether shallow tasks alone prepare an agent for deeper ones. We ran this experiment and found that a pool of shallow tasks enables generalization to deeper tasks: PPO trained first on depths 1 to 5 and then on depth 6 ends at a success rate of 0.36, while PPO trained on depth 6 from scratch for the same number of steps ends at 0.00. We added Appendix E.3, which reads:
Success rate on the deeper depth, mean ± standard deviation across 3 seeds. "At the shift" is the pool-pretrained agent run on the deeper depth with its parameters frozen, before any training on it.
We want to highlight that in every experiment of the paper, each task pool has an equal number of tasks at each depth. Section 4 states this:
Appendix D.2 gives the construction: