- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The submission proposes an evaluation protocol for continual RL using modular agents that distinguishes the loss of knowledge from the loss of capability to retrieve that knowledge. In particular, the authors study what fraction of the original performance on an earlier task is retained if using the ground-truth task-specific module, and what fraction is retained if using the learned router to select the module. Experiments on Continual World using CM-Dreamer show a significant gap between these two, and the authors conduct further analyses on the obtained data.
Critical strengths
- Distinguishing "forgetting" in terms of knowledge retention vs "forgetting" in terms of execution capability is an interesting direction in continual learning. These ideas have been studied in the literature in various forms, most notably by using linear probes to study whether learned representations maintain knowledge [1]. I am not familiar with work that has asked this question specifically in terms of fixed task-specific modules vs learned routes.
[1] Davari et al. 2022. "Probing representation forgetting in supervised and unsupervised continual learning."
Critical weaknesses
- The novelty of the contribution appears limited. It has long been understood that loss of performance does not inherently imply loss of knowledge internally in the model's weights. While I am not aware of prior instantiations of this idea that exactly match the proposed "retention gap" evaluation, it seems to me that this specific formulation on its own does not provide sufficient insight into the problem or ways to address it. While the experiments study this particular formulation exhaustively within the evaluation scope, they do not appear to yield any novel insight, but maybe rather confirmation of the well-known notion that forgetting is a complex phenomenon. I encourage the authors to expand this contribution in some direction:
- Perhaps the authors could use the results of this evaluation to identify which specific factors during training cause forgetting at various levels (representation vs retrieval)
- Or they could use their insights to propose a method to separate forgetting avoidance into representation and retrieval
- The scope of the evaluations is also quite moderate, and it is unclear whether the findings here would generalize in any way. The experiments use one benchmark (CW) and one baseline (CM-Dreamer). I do not wish to state that large-scale experiments are necessary for the contribution to be meaningful. But perhaps a larger-scale evaluation could yield deeper insights, which would strengthen the novelty of the contribution.
Questions for authors
What do the authors view as the key novelty of their submission? What meaningful insights would the community be interested in?
5 comments
W2. How far do the findings generalize? We extended the evaluation in two ways. We tested the deployment findings in SAC and examined routing in additional CM-Dreamer agents that had acquired two skills. Both extensions use Continual World, so generalization to other benchmarks remains open.
Second architecture. SAC uses hard expert selection and has no world model or LoRA adapters. When task identity was observable, the external gate matched first-task reference success in all three independent SAC seeds. Under masking, recurrent SAC retained a skill under the assigned expert that deployment could fail to retrieve. The pre-specified joint criterion was met in 2 of 3 primary seeds and 3 of 5 overall after we added two seeds upon seeing the primary results. Early routing errors also had lasting consequences: in all 10 evaluated seed–task cells, using the wrong expert for the first 16 decisions and the assigned expert for the remaining 234 reduced success to less than half its value with the assigned expert throughout (Tables D.3–D.5). The findings about soft mixture weights remain specific to CM-Dreamer.
Two-skill banks in CM-Dreamer. Seven of nine additional checkpoints met the two-skill inclusion criterion. In both evaluated cases where the shared backbone could solve both tasks on its own, deployment matched the assigned reference. Where only the assigned expert could perform the later task, deployment reduced success in four of five cases. In two of those cases, the assigned expert received the highest weight on most steps, yet deployment lost at least 50 percentage points of success. Holding the mixture fixed or changing its weights traced these losses to how the experts were mixed or how their weights varied over time (Appendix C.4, Table C.8). These cases show how retrieval can fail in agents with two usable skills. The pooled comparison remained inconclusive about how common such losses are.
The general response gives the per-seed SAC results under What transfers to a second architecture and the inclusion criteria and pooled comparison under Two-skill banks in CM-Dreamer.
Q1. Key novelty and implications. The protocol measures retention, retrieval and acquisition separately. It locates the source of a deployment loss by forcing experts, fixing or re-aiming mixtures, and delaying correction at the same checkpoint. Selective unfreezing tests acquisition during further training (Sections 3–4). Five findings illustrate how to use the protocol.
-
Evaluate the path the agent executes. At one CM-Dreamer checkpoint, the assigned expert received the highest weight on 97% of decision steps, yet task success fell from 100% with the assigned expert alone to 0% under the router’s time-varying mixture. Holding each expert’s weight fixed at its deployment average raised success to 95% at the same checkpoint. Conversely, inaccurate selection can be harmless when several experts hold the skill. Neither route accuracy nor the mean mixture alone establishes successful retrieval (Section 4.2, Table A.2).
-
Report the sign of the gap. The task-aware reference provides a controlled comparison. Deployment outperforms it in 19% of cells (Appendix C.3).
-
Measure whether the router uses task information in time. Probes decode task identity with accuracy of 0.77–0.99, yet five of six routers produce nearly constant outputs. The one router that identifies the task does so in the 33–64-step bin, after the commitment deadline of 6–8 steps (Section 4.3). Neither probe accuracy nor eventual identification alone establishes timely retrieval.
-
Measure acquisition separately from recovery of an existing skill. Frozen isolation keeps . The frozen modular arms together acquire later tasks in 3 of 63 cells on CW8, compared with 15/21 for naive training (Table A.8). Acquisition occurs in 6/13 runs with the decoder unfrozen and 1/15 with it frozen. In the cw3 head-only condition, no seed acquired the new task (0/5). First-task success still fell by 0.71–0.86 in two seeds and by 0.08–0.22 in the other three. All eighteen runs that unfreeze part of the world model lose 0.86–1.00 in first-task success (Section 4.4, Table B.6). The acquisition check distinguishes learning a new task from restoring deployment along an existing path.
-
Separate retention from expert selection. Freezing the reference path preserves its computation. With observable identity on cw3, an external gate matches reference success on all 15 CM-Dreamer seed–task pairs and on the first task in 3/3 independent SAC seeds. Under masking, the CM-Dreamer gate’s route accuracies are 0.49 and 0.66. The delayed-correction tests measure the consequences of early routing errors in each architecture (Section 4.3, Tables B.4, C.4, D.4 and D.5).
These questions can be tested in other modular agents whose routing can be overridden. Measuring retention also requires a reference with a consistent meaning across checkpoints. The component-level findings apply to the architectures in which those components were tested.
Thank you for the review and for pointing us to the probing work of Davari et al. (2022). Section 2 now cites that work. We first summarize how the protocol distinguishes retention from retrieval. In W1 we explain what the behavioural measurements add to probing and address both of your suggestions. We then discuss the scope of the evaluation in W2 and answer your question about novelty.
Protocol summary. We measure acquisition independently. F measures retention loss along the reference path. G measures the gap between that reference and deployment at the same checkpoint. Both quantities are signed differences in success rate and can take negative values. In the behavioural experiments, we fix the parameters and change only the routing by forcing slots, varying mixtures or delaying correction (Sections 3–4). The selective-unfreezing experiments require additional training and test whether the agent learns a new task (Section 3).
W1. What does the protocol reveal about forgetting? We agree that a drop in deployed performance can coexist with a retained skill. Our contribution is to test, for a particular failure, how much success the agent retains when we force the assigned expert and how much it can recover when we change only the routing. The revised Table 2 summarizes the evidence for each level.
The comparison also captures gains from deployment. Deployment exceeds the assigned reference in 28 of 150 cells (Appendix C.3, now also reported in Section 4.2). The reference therefore provides a controlled comparison and can itself be outperformed.
(i) Which training factors affect retention and retrieval? For retention, we resumed training from the same first-task checkpoint within each of five seeds. With the reference path frozen, all five controls retained their first-task success. Unfreezing the recurrent state-space model and decoder reduced first-task success by 0.97–1.00 across those same five seeds, while enabling four of them to acquire the second task, compared with none of the frozen controls (Section 4.4, Table B.6). This identifies a training choice that affected both retention of the old skill and acquisition of the new one.
For retrieval, the protocol can isolate the effect of routing at deployment. At one checkpoint, the assigned expert alone succeeded on 100% of episodes. It received the highest weight on 97% of deployment steps, yet deployed success was 0%. Holding each expert’s weight fixed at its deployment average raised success to 95%, with all model parameters unchanged (Section 4.2, Figure 2 and Table A.2). The training evidence is more limited. In configurations with retention distillation, five of six examined routers produced nearly constant outputs, even though a probe could decode task identity from their inputs with 77–99% accuracy. This is an association across configurations. The same-checkpoint routing intervention directly measures the performance cost of the deployed mixture (Section 4.3, Appendix C.5).
(ii) Prevent retrieval failures when deploying the preserved skill. The mixture experiment shows why this needs a separate intervention: a capable expert can receive the highest routing weight on almost every step and still fail under the deployed mixture. Where observations identify the task, we therefore use a separately trained gate to select the assigned expert directly. We test this choice by comparing gated success with success under the forced expert at the same checkpoint. On cw3, the gate reproduced the assigned route and matched its success on all 15 seed–task pairs. Under masking, its routing accuracy was only 0.49 and 0.66 on the two tasks, so the evidence for this retrieval strategy is limited to settings where observations distinguish the tasks (Sections 4.2–4.3, Table B.4).
Timing also matters when task identity must be inferred from interaction. On the masked task, we forced the wrong expert for the first N decisions and then switched to the assigned expert for the rest of the episode. Across six checkpoints, the last delay before success first fell below half the reference rate was 6–8 decisions. The learned router in Figure 4a first reached 70% routing accuracy in the 33–64-step bin and reached 94% from step 65 onward. Yet its deployment success was 0%, compared with 60% when the assigned expert acted from the start. A selector should therefore be evaluated on whether it identifies the task early enough to preserve success, as well as on its eventual accuracy (Section 4.3, Figure 4 and Appendix C.6).