acceptodds
Under review as a conference paper at ICLR 2027

Not Forgotten, Not Retrieved: A Same-Checkpoint Decomposition of Failure in Modular Continual RL

Abstract

An old-task performance drop in continual reinforcement learning can mean that a learned capability degraded or that the agent failed to select a capability that remains available. We introduce an architecture-agnostic diagnostic protocol for modular agents whose expert selection can be overridden, and instantiate it in CM-Dreamer, a modular DreamerV3 agent. At the same checkpoint, we compare a task aware reference that forces the expert assigned to the evaluated task with learned deployment, in which the agent’s router selects or mixes experts without a task label. Their signed difference is the retrieval gap; change in task-aware reference performance across training is retention loss. Acquisition is measured separately. In CM-Dreamer, the same final checkpoint on a three-task Meta-World stream (cw3) scores 0.66 under the task-aware reference and 0.32 under learned deployment. Another checkpoint places its largest router weight on the assigned expert on 97% of steps yet has a retrieval gap of 1.00, whereas inaccurate selection can be harmless when several experts solve the same task. Forced experts and fixed mixtures distinguish these cases. When observations reveal task identity, an observation-based selector without task labels matches the reference. When identity-revealing features are masked, a tested router identifies the task only after early actions make correction ineffective. Freezing the reference computation preserves old-task performance but often prevents new-task acquisition; unfreezing the recurrent state model and decoder improves acquisition at a retention cost. Most checkpoints contain only one usable skill because later tasks were often never acquired. The few two-skill checkpoints demonstrate selection failure but are too few to establish its frequency.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual reinforcement learning, catastrophic forgetting, modular networks, mixture of experts, world models, task routing, loss of plasticity, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jcrjOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The submission proposes an evaluation protocol for continual RL using modular agents that distinguishes the loss of knowledge from the loss of capability to retrieve that knowledge. In particular, the authors study what fraction of the original performance on an earlier task is retained if using the ground-truth task-specific module, and what fraction is retained if using the learned router to select the module. Experiments on Continual World using CM-Dreamer show a significant gap between these two, and the authors conduct further analyses on the obtained data.

Critical strengths
  • Distinguishing "forgetting" in terms of knowledge retention vs "forgetting" in terms of execution capability is an interesting direction in continual learning. These ideas have been studied in the literature in various forms, most notably by using linear probes to study whether learned representations maintain knowledge [1]. I am not familiar with work that has asked this question specifically in terms of fixed task-specific modules vs learned routes.

[1] Davari et al. 2022. "Probing representation forgetting in supervised and unsupervised continual learning."

Critical weaknesses
  • The novelty of the contribution appears limited. It has long been understood that loss of performance does not inherently imply loss of knowledge internally in the model's weights. While I am not aware of prior instantiations of this idea that exactly match the proposed "retention gap" evaluation, it seems to me that this specific formulation on its own does not provide sufficient insight into the problem or ways to address it. While the experiments study this particular formulation exhaustively within the evaluation scope, they do not appear to yield any novel insight, but maybe rather confirmation of the well-known notion that forgetting is a complex phenomenon. I encourage the authors to expand this contribution in some direction:
    • Perhaps the authors could use the results of this evaluation to identify which specific factors during training cause forgetting at various levels (representation vs retrieval)
    • Or they could use their insights to propose a method to separate forgetting avoidance into representation and retrieval
  • The scope of the evaluations is also quite moderate, and it is unclear whether the findings here would generalize in any way. The experiments use one benchmark (CW) and one baseline (CM-Dreamer). I do not wish to state that large-scale experiments are necessary for the contribution to be meaningful. But perhaps a larger-scale evaluation could yield deeper insights, which would strengthen the novelty of the contribution.
Questions for authors

What do the authors view as the key novelty of their submission? What meaningful insights would the community be interested in?

5 comments

Thank you for the review and for pointing us to the probing work of Davari et al. (2022). Section 2 now cites that work. We first summarize how the protocol distinguishes retention from retrieval. In W1 we explain what the behavioural measurements add to probing and address both of your suggestions. We then discuss the scope of the evaluation in W2 and answer your question about novelty.

Protocol summary. We measure acquisition independently. FF measures retention loss along the reference path. GG measures the gap between that reference and deployment at the same checkpoint. Both quantities are signed differences in success rate and can take negative values. In the behavioural experiments, we fix the parameters and change only the routing by forcing slots, varying mixtures or delaying correction (Sections 3–4). The selective-unfreezing experiments require additional training and test whether the agent learns a new task (Section 3).

W1. What does the protocol reveal about forgetting? We agree that a drop in deployed performance can coexist with a retained skill. Our contribution is to test, for a particular failure, how much success the agent retains when we force the assigned expert and how much it can recover when we change only the routing. The revised Table 2 summarizes the evidence for each level.

The comparison also captures gains from deployment. Deployment exceeds the assigned reference in 28 of 150 cells (Appendix C.3, now also reported in Section 4.2). The reference therefore provides a controlled comparison and can itself be outperformed.

(i) Which training factors affect retention and retrieval? For retention, we resumed training from the same first-task checkpoint within each of five seeds. With the reference path frozen, all five controls retained their first-task success. Unfreezing the recurrent state-space model and decoder reduced first-task success by 0.97–1.00 across those same five seeds, while enabling four of them to acquire the second task, compared with none of the frozen controls (Section 4.4, Table B.6). This identifies a training choice that affected both retention of the old skill and acquisition of the new one.

For retrieval, the protocol can isolate the effect of routing at deployment. At one checkpoint, the assigned expert alone succeeded on 100% of episodes. It received the highest weight on 97% of deployment steps, yet deployed success was 0%. Holding each expert’s weight fixed at its deployment average raised success to 95%, with all model parameters unchanged (Section 4.2, Figure 2 and Table A.2). The training evidence is more limited. In configurations with retention distillation, five of six examined routers produced nearly constant outputs, even though a probe could decode task identity from their inputs with 77–99% accuracy. This is an association across configurations. The same-checkpoint routing intervention directly measures the performance cost of the deployed mixture (Section 4.3, Appendix C.5).

(ii) Prevent retrieval failures when deploying the preserved skill. The mixture experiment shows why this needs a separate intervention: a capable expert can receive the highest routing weight on almost every step and still fail under the deployed mixture. Where observations identify the task, we therefore use a separately trained gate to select the assigned expert directly. We test this choice by comparing gated success with success under the forced expert at the same checkpoint. On cw3, the gate reproduced the assigned route and matched its success on all 15 seed–task pairs. Under masking, its routing accuracy was only 0.49 and 0.66 on the two tasks, so the evidence for this retrieval strategy is limited to settings where observations distinguish the tasks (Sections 4.2–4.3, Table B.4).

Timing also matters when task identity must be inferred from interaction. On the masked task, we forced the wrong expert for the first NN decisions and then switched to the assigned expert for the rest of the episode. Across six checkpoints, the last delay before success first fell below half the reference rate was 6–8 decisions. The learned router in Figure 4a first reached 70% routing accuracy in the 33–64-step bin and reached 94% from step 65 onward. Yet its deployment success was 0%, compared with 60% when the assigned expert acted from the start. A selector should therefore be evaluated on whether it identifies the task early enough to preserve success, as well as on its eventual accuracy (Section 4.3, Figure 4 and Appendix C.6).

W2. How far do the findings generalize? We extended the evaluation in two ways. We tested the deployment findings in SAC and examined routing in additional CM-Dreamer agents that had acquired two skills. Both extensions use Continual World, so generalization to other benchmarks remains open.

Second architecture. SAC uses hard expert selection and has no world model or LoRA adapters. When task identity was observable, the external gate matched first-task reference success in all three independent SAC seeds. Under masking, recurrent SAC retained a skill under the assigned expert that deployment could fail to retrieve. The pre-specified joint criterion was met in 2 of 3 primary seeds and 3 of 5 overall after we added two seeds upon seeing the primary results. Early routing errors also had lasting consequences: in all 10 evaluated seed–task cells, using the wrong expert for the first 16 decisions and the assigned expert for the remaining 234 reduced success to less than half its value with the assigned expert throughout (Tables D.3–D.5). The findings about soft mixture weights remain specific to CM-Dreamer.

Two-skill banks in CM-Dreamer. Seven of nine additional checkpoints met the two-skill inclusion criterion. In both evaluated cases where the shared backbone could solve both tasks on its own, deployment matched the assigned reference. Where only the assigned expert could perform the later task, deployment reduced success in four of five cases. In two of those cases, the assigned expert received the highest weight on most steps, yet deployment lost at least 50 percentage points of success. Holding the mixture fixed or changing its weights traced these losses to how the experts were mixed or how their weights varied over time (Appendix C.4, Table C.8). These cases show how retrieval can fail in agents with two usable skills. The pooled comparison remained inconclusive about how common such losses are.

The general response gives the per-seed SAC results under What transfers to a second architecture and the inclusion criteria and pooled comparison under Two-skill banks in CM-Dreamer.

Q1. Key novelty and implications. The protocol measures retention, retrieval and acquisition separately. It locates the source of a deployment loss by forcing experts, fixing or re-aiming mixtures, and delaying correction at the same checkpoint. Selective unfreezing tests acquisition during further training (Sections 3–4). Five findings illustrate how to use the protocol.

  1. Evaluate the path the agent executes. At one CM-Dreamer checkpoint, the assigned expert received the highest weight on 97% of decision steps, yet task success fell from 100% with the assigned expert alone to 0% under the router’s time-varying mixture. Holding each expert’s weight fixed at its deployment average raised success to 95% at the same checkpoint. Conversely, inaccurate selection can be harmless when several experts hold the skill. Neither route accuracy nor the mean mixture alone establishes successful retrieval (Section 4.2, Table A.2).

  2. Report the sign of the gap. The task-aware reference provides a controlled comparison. Deployment outperforms it in 19% of cells (Appendix C.3).

  3. Measure whether the router uses task information in time. Probes decode task identity with accuracy of 0.77–0.99, yet five of six routers produce nearly constant outputs. The one router that identifies the task does so in the 33–64-step bin, after the commitment deadline of 6–8 steps (Section 4.3). Neither probe accuracy nor eventual identification alone establishes timely retrieval.

  4. Measure acquisition separately from recovery of an existing skill. Frozen isolation keeps F=0F = 0. The frozen modular arms together acquire later tasks in 3 of 63 cells on CW8, compared with 15/21 for naive training (Table A.8). Acquisition occurs in 6/13 runs with the decoder unfrozen and 1/15 with it frozen. In the cw3 head-only condition, no seed acquired the new task (0/5). First-task success still fell by 0.71–0.86 in two seeds and by 0.08–0.22 in the other three. All eighteen runs that unfreeze part of the world model lose 0.86–1.00 in first-task success (Section 4.4, Table B.6). The acquisition check distinguishes learning a new task from restoring deployment along an existing path.

  5. Separate retention from expert selection. Freezing the reference path preserves its computation. With observable identity on cw3, an external gate matches reference success on all 15 CM-Dreamer seed–task pairs and on the first task in 3/3 independent SAC seeds. Under masking, the CM-Dreamer gate’s route accuracies are 0.49 and 0.66. The delayed-correction tests measure the consequences of early routing errors in each architecture (Section 4.3, Tables B.4, C.4, D.4 and D.5).

These questions can be tested in other modular agents whose routing can be overridden. Measuring retention also requires a reference with a consistent meaning across checkpoints. The component-level findings apply to the architectures in which those components were tested.

Reviewer_XpjTOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an evaluation protocol for modular continual reinforcement learning that distinguishes failure to acquire a task, degradation of previously learned capability, and failure to retrieve and available skill. It compares learned routing with forced expert selection at the same checkpoint, supplemented by expert and mixture interventions. Experiments with modular DreamerV3 agents show that (1) poor deployment performance does not neceessarily imply forgetting, (2) routing accuracy can misrepresent usable capability, and (3) preserving old skills can hinder new-task acquisition.

Critical strengths
  1. The paper addresses a useful problem in evaluationg forgetting. An agent can perform poorly because it has lost a skill, because is selects the wrong expert, or because it never learned the task. The proposed evaluation helps distinguish these explanations. This matters because each problem requires a different solution.

  2. The experiments give concrete examples of skills remaining available despite poor performance. In one experiment, the same trained agent achieves 0.66 success when the desginated expert is selected manually, compared with 0.32 when the agent chooses for itself. Another agent gives the designated expert the largest weight on 97% of steps but still performs much worse than when that expert is used alone. These examples show why task performance and expert-selection accuracy should be examined together.

  3. The authors test possible explanations rather than relying only on final scores. They evaluate experts individually, try fixed combinations of experts, and change when the correct expert is selected. They also check whether later tasks were learned at all. These tests help explain how routing and learning failures affect performance.

  4. The paper is open about the limitations of its results. The authors report that many agents fail to learn later tasks. They also acknowledge that the few agents with multiple usable skills are insufficient to show how often routing is the main problem. Reporting these unsucessfully and inconclusive results makes the findings easier to assess.

Critical weaknesses

Critical weaknesses

  1. Too few agents successfully learn multiple tasks. Most tested checkpoints contain only one usable skill because later tasks were never learned. The paper therefore provides limited evidence about routing in agents that have serveral skills to choose from. It demonstrates that routing can cause failure, but does not establish how common or important this problem is in effective continual learners. The experiments also use only one agent architecture.

  2. The new contribution needs to be distinguished more clearly from earlier work. Prior work such as OWL already compares learned expert selection with selection using the true task identity. Splitting the performance drop into two terms is mathematically straightfoward. The stronger contribution here is the set of experiments that checks which skills remain available and how selection errors affect behavior. The paper should explain more explicitly what these tests reveal beyond previous evaluations.

  3. Some statistical comparison treat related runs as separate observations. Several experiments start from the same trained agent and unfreeze different components. Their outcomes may therefore be correlated. The reported comparison of acquisition rates uses a statistical test that does not account for this shared starting point. An analysis that accounts for the original training seed would give more reliable evidence. This affects confidence in that comparison, rather than disproving the individual examples.

  4. The experiment setup is difficult to reconstruct. Important details are spread across the appendices. These include which observations features and hidden, what information the router recieves, which components are frozen, and the training settings. A single configuration table and a short description of the evaluation procedure would make the work easier to reproduce.

Questions for authors

Clarification requests

  1. Can the core protocol be demonstrated on a second modular architecture, or on a configuration that reliably acquires multiple distinct task skills?

  2. How stable are the qualitative conclusions to the 0.5 competence threshold and to an alternative fixed reference, such as a preselected competent route? Such checks should preserve the paper's distinction between an assigned reference and the bank's best evaluated behavior.

  3. Is the code available online? The code was unfortunately inaccessible via OpenReview.

Future work 4. Please provide how the forced-expert versus learned-routing performance difference varies across training seeds. For the unfreezing comparison, please account for the fact that several experiments start from the same trained agent.

  1. Please provide a complete configuration and instructions for reproducing the main experiments.
5 comments

Thank you for the detailed review. We appreciate your emphasis on the two same-checkpoint cases and the separate acquisition check, which remain central to the revision. We address your four concerns and five questions below, including the second architecture, runs with shared starting points, per-seed results, sensitivity analyses and reproducibility.

W1. Multi-skill banks and a second architecture. The new experiment adds multi-skill banks from a second algorithm family. On the masked stream, recurrent SAC met the pre-specified joint criterion in 2 of 3 primary seeds and 3 of 5 seeds overall. We added the two supplementary seeds after seeing the primary results (Appendix D, Table D.3).

The independent feed-forward SAC agent holds three slot-level skills in each of its three seeds on cw3. The recurrent SAC agent holds two in each of its five seeds on the masked stream (Table D.4). For example, in recurrent SAC seed 300, forcing the assigned expert gives first-task success of 1.000 and deploying the gate gives 0.140 at the same checkpoint. The frequency of retrieval failures among effective continual learners remains unknown.

Within CM-Dreamer, we extended the two-skill stratum after review using a criterion fixed before the extension (Appendix C.4, Table C.2). We evaluated nine further checkpoints from the drawer, window and CW8 streams. Seven held two skills. Appendix C.10 and Table C.8 give the results for each cell.

In the pooled analysis, the absolute gain of the better fixed mixture over learned deployment exceeded 0.2 in 4 of 15 later-task cells in two-skill banks and in 5 of 45 first-task comparison cells. The pre-registered verdict remains undecided (one-sided binomial p=0.0765p = 0.0765, treating 5/45 as fixed and cells as independent despite shared streams and configurations). The verdict is sensitive in both directions: the +0.215 cell’s interval reaches 0.155, and including one checkpoint excluded for lacking a first-task skill would give 5 of 16 (p=0.026p = 0.026).

The extension identifies the conditions under which selection reduced success in these cells. We observed no later-task routing loss where the backbone solved both tasks (2/2 cells, G=0G = 0). Where only the assigned slot held the later-task skill, deployment reduced success by 0.25–0.61 in four of five cells. One checkpoint from the eight-task stream routed correctly on 98% of steps and still lost 0.605 (Table C.8).

W2. What the protocol adds to OWL. We agree that the two-term split is mathematically straightforward. The contribution lies in the interventions and what they measure. OWL compares selection rules. It also examines performance on previously trained tasks. Section 2 distinguishes two additions to that comparison. We track retention across checkpoints on a fixed reference path, then measure the signed deployment gap at the same checkpoint. Forced experts and fixed mixtures identify which experts can perform the task and how mixture weights and timing contribute to the gap (Sections 4.2–4.3).

For example, on the same task, success is 0.455 with the assigned slot, 0.945 with the bank's best slot and 0.020 under deployment (Appendix C.4). These evaluations distinguish which slot can perform the task from whether the deployed agent uses that skill.

W3. Runs with a shared starting point. We added a seed-stratified exact permutation test. We permute labels for the unfreezing conditions only within each training seed, giving two-sided p=0.036p = 0.036 and one-sided p=0.027p = 0.027 (Table B.6 caption). With acquisition defined as peak success of at least 0.5 during periodic evaluations, acquisition occurred in 6/13 runs with the decoder unfrozen and 1/15 with it frozen (Section 4.4, Table B.6). The stratified analysis agrees with the pooled Fisher test (p=0.029p = 0.029, Section 4.4). Four of five seeds favoured the unfrozen-decoder group. When both the RSSM and decoder were unfrozen, the agent acquired the task in 4/5 seeds, compared with 0/5 for the frozen controls from the same seeds.

W4. Reconstructing the setup. The revision adds a unified configuration table and a short evaluation procedure (Appendix A.1, Table A.10). They specify the masked dimensions, router input, frozen components, LoRA configuration, routing rules, training budget and evaluation pairing. Table A.10 also points to Tables A.3–A.5 for the experimental arms, frozen paths and resources. The values come from the configuration files and training script.

Q1. Testing the protocol on a second architecture. Yes. The SAC experiments in W1 use hard expert selection and no world model. On cw3, the independent feed-forward agent has three-skill banks in all three seeds, with first-task G=0.000G = 0.000.

On the masked stream, the recurrent agent has two-skill banks in all five seeds. Its first-task gaps range from −0.065-0.065 to 0.9550.955 (Table D.4, Figure D.1). Appendix D.3 reports the shared-trunk results, undecidable combinations and the sensitivity of the joint criterion near its boundaries.

Q2. Sensitivity to the threshold and choice of reference. The qualitative findings hold at competence thresholds of 0.4, 0.5 and 0.6. Appendix C.8 and Tables C.5–C.6 report the bank's best evaluated behaviour separately from the fixed assigned reference. Recomputing the results from the stored per-cell records gives the following counts, in threshold order.

  • The counts of capable first-task cells are 52, 51 and 51. The corresponding counts of two-skill banks are 6, 6 and 4.

  • In the aimed group (route accuracy at least 0.5), 3 of 15 cells have G≥0.5G \ge 0.5 at each threshold.

  • On the masked stream, the wrong slot falls below threshold in 12/12 unanchored checkpoints and reaches threshold in 6/14 anchored checkpoints. These counts and the acquisition counts in Table B.6 (6/13 vs 1/15) hold at all three thresholds.

  • For later tasks on CW8, the acquisition counts are 4/63, 3/63 and 3/63 for the frozen modular arms. For naive training they are 16/21, 15/21 and 15/21.

For the alternative reference, we select the best evaluated forced slot in each cell (the bank-best reference). This measures the competence available in the bank. Selecting the slot after evaluation makes the rule unavailable at deployment. The best slot can also change between checkpoints, so it cannot serve as the fixed reference for FF.

Across the 51 capable first-task cells, using the bank-best reference raises the median GG from 0.025 to 0.070. The gap is negative in 10 cells under this reference and in 12 under the assigned reference. In 2 of these 51 cells, the bank-best slot outperforms the assigned slot by at least 0.2. We did not evaluate a separately preselected route. Appendix C.8 also corrects the description of 3 of the 45 banks originally classified as one-skill. These banks reach the competence threshold only under mixtures and have no slot-level skill. The original conclusion still holds.

Q3. Code availability. The anonymous code release is available at https://anonymous.4open.science/r/nfnr-protocol-F8A9. The release package contains the training and evaluation code, complete configurations, per-run results, per-episode records for SAC and shard-level aggregates for CM-Dreamer and materials for the second architecture. It also includes the scripts that generate the tables and a comparison with the numbers in the PDF. We plan to release checkpoints with the camera-ready version. We can provide the checkpoints discussed individually in the main text through the AC.

Q4.1 How does the gap between forced-expert and learned routing vary across seeds? We evaluated the first task at the end of the cw3 stream, using 200 episodes per seed and routing condition (Appendix C.9, Table C.7). For each seed, GG is the success rate with the task’s assigned expert minus the success rate under learned routing.

Training seedAssigned expertLearned routingGG
00.4900.100+0.390
1000.0000.070−0.070
2001.0000.995+0.005
3000.9750.390+0.585
4001.0000.245+0.755

Learned routing reduced success by 39.0 to 75.5 percentage points in seeds 0, 300, and 400. The two conditions performed almost identically in seed 200. In seed 100, learned routing did slightly better, although success was low under both conditions.

Q4.2 Does the unfreezing result hold after accounting for shared starting points? Within-seed comparisons favoured decoder unfreezing in four of five training seeds. An exact permutation test that shuffled condition labels only within each seed gave a two-sided p=0.036p=0.036 (Table B.6). When both the RSSM and decoder were unfrozen, the agent acquired the new task in four of five seeds, compared with zero of five for the frozen controls from the same starting models.

Q5. Configuration and evaluation procedure. Appendix A.1 and Table A.10 give the unified configuration and evaluation procedure. The release package contains the complete configuration files. The procedure records the acquisition boundary for each task. At the same checkpoint and in paired environments, it evaluates the task-aware reference, deployment, forced slots and fixed mixtures. It then computes FF and GG and rebuilds the tables (Section 3, Appendix A). The configuration specifies the number of episodes per cell, the seed formula, the frozen paths and the routing rules. Appendix D documents the training-budget difference and the departures from the original setup for the second architecture.

Reviewer_tJjgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper asks whether poor performance on an earlier task means that the agent has lost the skill or is failing to use it. The paper tests this by separating acquisition, retention, and retrieval failures. It compares task-aware reference routing with learned deployment at the same checkpoint, using a modular DreamerV3 agent on Meta-World tasks. Experiments show that high routing accuracy does not always lead to task success, and that identifying the task too late can prevent recovery from early mistakes. Allowing parts of the frozen model to update can improve new-task learning, but often reduces performance on earlier tasks.

Critical strengths
  1. Comparing routing rules at the same checkpoint directly shows that poor deployment can coexist with an effective execution path. Measuring acquisition separately also helps avoid interpreting a task that was never learned as forgetting.
  2. The mixture and delayed-correction experiments provide useful detail beyond routing accuracy. I appreciate that the authors also report negative results and clearly acknowledge the limitations of the reference route and the small number of multi-skill checkpoints.
Critical weaknesses
  1. The novelty over prior expert-selection work is somewhat unclear to me. Expert Gate already studies label-free expert selection, and OWL compares oracle, adaptive, random, and max-Q selection in continual RL. The authors acknowledge this and add mixture and timing analyses, but I would like a clearer account of what these analyses teach us beyond the earlier comparisons.
  2. The findings may depend substantially on the chosen implementation, including the frozen world model, LoRA experts, and soft routing. The paper contains many ablations and additional streams, but they remain within the CM-Dreamer family, and most capable checkpoints contain only one slot-level skill. This does not undermine the examples of retrieval failure, but leaves their relevance to other modular agents less clear.
Questions for authors
  • Given the comparisons with OWL in Section 2, which additional finding do the authors consider the strongest reason to adopt this diagnostic protocol?
  • I appreciate the multi-skill analysis in Appendix C.4. Considering those results together with the ablations, which findings do the authors expect to carry over to other modular agents, and what evidence supports that expectation?
2 comments

Thank you for highlighting the same-checkpoint comparison, the separate acquisition record, and the mixture and delayed-correction experiments. These remain central to the revision. Below we address your two concerns and two questions. Sections 1 and 5 now explain why we use the protocol and which findings transfer.

W1. What the interventions reveal beyond Expert Gate and OWL. Expert Gate uses task-trained autoencoders to select an expert without a task label (Aljundi et al., 2017). OWL compares adaptive, oracle, random and max-Q head selection and examines performance on earlier tasks (Kessler et al., 2022). Our protocol tracks the assigned reference across checkpoints, then compares forced experts and deployment at the same checkpoint (Section 2, Table A.7). On cw3, all three independent feed-forward SAC seeds have three skills in the forced-expert sweep and first-task F=G=0.000F = G = 0.000. In the three shared-trunk seeds that acquired the first task, G=0.000G = 0.000 as well, but reference and deployment success were both zero at the end of the stream and F=1.000F = 1.000 (Table D.4). The same gap thus accompanies two different retention outcomes.

In CM-Dreamer we can also compare fixed, re-aimed and time-varying mixtures at the same checkpoint to identify losses that argmax accuracy can miss (Section 4.2).

The delayed-correction experiment measures a deadline of 6–8 steps. Under masking, the router that reaches the identification threshold does so only in the 33–64-step bin (Section 4.3). Measuring when the router identifies the task therefore matters even when it eventually selects the right expert.

W2. Implementation dependence and multi-skill banks. The SAC experiments measure the deployment gap without a world model, LoRA or soft routing. On the masked stream, the recurrent SAC agent met the pre-specified criterion in 2 of 3 primary seeds and in 3 of 5 seeds in the supplementary analysis (Appendix D, Table D.3).

We verified each skill slot by slot in the independent feed-forward agent's three-skill banks and the recurrent agent's two-skill banks. Point 2 of the general response lists the recurrent agent's per-seed GG values alongside the corresponding success rates (Table D.4). We added the supplementary seeds after seeing the primary results. We report the shared-trunk arm under the corresponding primary and supplementary rules, including all undecidable combinations. Appendices D.2 and D.3 give the full criteria and their sensitivity near the decision boundaries.

Within CM-Dreamer, we fixed the inclusion criterion before extending the analysis during review. The number of later-task cells in two-skill banks rose from 8 to 15 (Appendix C.4, Table C.2, with per-cell results in Table C.8). The verdict for absolute constant-reference gain above 0.2 remains undecided (4/15 versus 5/45, p=0.0765p = 0.0765).

We ran the forced-slot sweep and path decomposition on seven two-skill banks. Deployment caused no later-task loss in either cell where the backbone solved both tasks (2/2). Where only the assigned slot held the later-task skill, deployment reduced success by 0.25–0.61 in four of five cells. In two additional aimed cells with G≥0.5G \ge 0.5, the loss arose from mixture shape or timing. One was on the eight-task stream, with Aroute=0.981A_{\rm route} = 0.981 and G=0.605G = 0.605. The general response gives the scope of this study.

Q1. What does the protocol reveal beyond an oracle–learned routing comparison? Our strongest example comes from a CM-Dreamer checkpoint where using the assigned expert alone gave 100% task success. Under learned routing, that expert received the highest weight on 97% of decision steps, yet task success was 0%. The oracle–learned comparison reveals this 100-percentage-point gap. Our additional intervention shows how much of it can be recovered: at the same checkpoint, fixing each expert’s weight at its deployment average throughout the episode raised success from 0% to 95%. Thus, in this case, allowing the mixture weights to vary from step to step had a large performance cost that routing accuracy alone did not reveal (Section 4.2, Figure 2 and Table A.2).

Q2. Which findings transfer? Section 5 distinguishes three levels of evidence.

  • By construction. The F+GF + G decomposition of the drop from acquisition-boundary reference success to current deployment success, the sign of the difference and the separate measurement of acquisition require only overridable routing and a reference whose meaning is fixed across checkpoints (Sections 3 and 5).

  • Re-tested on a second architecture. With observable identity, the gate matches the reference. Under masking, the recurrent agent can retain a skill under the reference and lose it at deployment (primary verdict, 2 of 3 seeds, and supplementary verdict, 3 of 5 seeds, Table D.3). Delayed correction tests how long the wrong expert can act before recovery fails. In eight recurrent SAC cells where the forced wrong expert scored ≤0.06\le 0.06, the last tested prefix retaining half of reference success was at most 8 decisions. The cell scoring 0.130 also crossed below half by 8 decisions. For the cell scoring 0.315, success recovered at longer prefix lengths. The last acceptable prefix before the first drop was 8, but the largest-prefix rule found no deadline in the tested range (Table D.5).

    In CM-Dreamer, success with the wrong slot forced for a full episode predicted whether a correction deadline appeared. With the borderline wrong-slot score counted as zero, this prediction matched the outcome in all 13 decidable cells (Table C.4). The two-skill extension in Appendix C.4 illustrates the same role of competence. Deployment caused no later-task loss where the backbone solved both tasks (2/2), but deployment reduced success in four of five cells where only the assigned slot held the skill. Both agent families show sensitivity to early routing, but the CM-Dreamer zero-versus-nonzero rule did not carry over unchanged to SAC. For the shared trunk on cw3, only the five-seed analysis met the usability criterion for a retention verdict (Table D.3).

  • Tested only in CM-Dreamer. These findings include near-constant learned soft routers, differences in mixture weights and timing between training and deployment, and the acquisition and retention effects of selective RSSM and decoder unfreezing (Sections 4.2–4.4). Section 5 gives the basis for each finding.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their comments. The revised manuscript PDF marks new and rewritten text in blue. We also corrected several erroneous numbers in the submission; each correction is marked in blue, and none changes a conclusion. Appendix C.8 corrects the wording of one statement. We list the changes at the end of this general response.

What the routing interventions measure

Our protocol compares routing choices at the same checkpoint. FF measures the loss of success along the task-aware reference path (forcing the assigned expert) from the task’s acquisition boundary to the evaluation checkpoint. GG is reference success minus deployed success at that checkpoint. We measure acquisition separately (Section 3). We agree that "a drop in performance is not a loss of knowledge" is a familiar observation. We test it by changing only the routing, holding the parameters fixed and recording the signed difference in success. OWL compares selection rules by their scores. Our protocol also measures whether the assigned reference retained its performance (FF), how each expert performs when forced, and how much of a routing loss comes from mixture shape, aim or timing (Eq. 2, Table A.7).

These interventions reveal three effects that aggregate scores obscure. At one checkpoint, the assigned expert receives the largest weight on 97% of deployment steps (Aroute=0.97A_{\rm route} = 0.97), yet G=1.00G = 1.00 (Section 4.2, Figure 2). Deployment outperforms the reference in 28 of 150 cells (Appendix C.3). A probe decodes task identity from the router's input with accuracy of 0.77–0.99, yet five of six routers produce nearly constant outputs (Section 4.3).

In this revision, we added Table 2 to Section 5 to distinguish the effects measured in training interventions from associations observed across configurations for retention, retrieval and acquisition. Freezing the reference computation fixes F=0F = 0 by construction. In configurations with retention distillation, which constrains the shared context path to preserve its earlier behaviour, 5 of 6 examined routers produce nearly constant outputs. This is an association across configurations. Under masking, the one router that identifies the task does so in the 33–64-step bin, after the deadline of 6–8 steps. Selective unfreezing implicates the recurrent state-space model (RSSM) and decoder in acquisition. The agent acquired the new task in 6/13 runs with the decoder unfrozen and 1/15 with it frozen.

Retention and retrieval can therefore be handled separately. Freezing the whole reference path preserves its performance. With observable task identity on cw3, the external observation gate reproduces the assigned one-hot route and its measured success on all 15 CM-Dreamer seed–task pairs. In independent SAC it gives first-task G=0.000G = 0.000 in 3/3 seeds (Section 4.3, Tables B.4 and D.4). Under masking, the CM-Dreamer gate’s route accuracy is 0.49 on the first task and 0.66 on the second.

What transfers to a second architecture

We chose soft actor-critic (SAC) agents because they address all three implementation dependencies raised in the reviews. They have no world model or LoRA adapters and use hard one-hot expert selection. In arm S, we train one independent feed-forward SAC agent per task and freeze it after that task. This design belongs to the Expert-Gate / parameter-isolation family. Its continual-learning challenge is selecting an expert without task labels at test time. Arm R uses the same design with a recurrent policy. Arm H has one head per task and a shared trunk trained throughout.

All three arms use the paper's label-free observation gate at deployment. We reused the streams, masking, success metric and seed pairing from the paper's code and fixed the primary criteria before the first main run (Appendix D). An arm–stream combination is usable when first-task reference success at the acquisition boundary reaches 0.5 in at least 2 of 3 primary seeds (3 of 5 in the supplementary analysis).

  1. Observable identity. Arm S has three-skill banks on cw3. On the first task, the gate matches the task-aware reference in 3/3 seeds, with G=0.000G = 0.000 and route accuracy 1.0001.000 (Table D.4). The pre-specified E1 band was [−0.1,0.1][-0.1, 0.1] (Table D.2).
  1. Masked identity. In arm R, the reference retains a skill that learned deployment can lose on the masked stream. All five end-of-stream first-task cells were capable (at least one channel reached 0.5). The joint criterion E2 requires 0.3≤Aroute≤0.70.3 \le A_{\rm route} \le 0.7 and G≥0.3G \ge 0.3. In the primary analysis, seeds 100 and 200 met it (2/3). We added two seeds after seeing the primary results. In the supplementary analysis, seed 300 also met E2 (3/5 overall). Seed 0 fell below the gap threshold and seed 400 fell below the route-accuracy band. Seed 100 was just inside the band at 0.698 (Tables D.2–D.3). Appendix D.2 documents the hidden-state check and the change to cell-level exclusions. The per-seed results from Table D.4 are shown below.
Seed0100200300400
Reference success0.5451.0001.0001.0001.000
Deployment success0.6100.6250.5200.1400.045
G−0.0650.3750.4800.8600.955
Route accuracy0.6500.6980.5540.6310.211
  1. Early routing errors can reduce success even after correction. As in CM-Dreamer, choosing the wrong expert early in SAC could reduce task success even after the assigned expert took over. In arm R, forcing the wrong expert for the first 16 decisions and the assigned expert for the remaining 234 reduced success to less than half the value obtained with the assigned expert throughout, in all 10 seed–task cells (Table D.5).

    In all eight cells with whole-episode wrong-expert success at or below 0.06, the longest tested prefix retaining at least half of reference success was 0 or 8 decisions. The cell with wrong-expert success of 0.130 also had a deadline of 0. Just eight wrong decisions reduced success from 1.000 to 0.000. The two deadline rules differed for seed 0 on faucet-open, where wrong-expert success was 0.315. The assigned expert itself achieved only 0.545 success on this task, compared with 1.000 for each of the other four seeds, putting the half-reference threshold at 0.2725. Success was above this threshold at N=8N=8 (0.380), fell below it at N=16N=16 (0.195), and was above it again at N=128N=128 (0.420). Thus, 8 decisions was the last tested prefix before the first drop, while the longest-prefix rule gave no deadline within the tested range.

    Even at similar routing accuracies, choosing the wrong expert may carry different costs. On faucet-open, seeds 0 and 300 had routing accuracies of 0.650 and 0.631, yet forcing the wrong expert reduced success from 0.545 to 0.315 in seed 0 and from 1.000 to 0.000 in seed 300.

Arm H illustrates a loss of retention. In the five-seed cw3 analysis, three seeds acquired the first task. For all three, F=1.000F = 1.000, so we draw no inference about retrieval from those cells. The paper's retention result relies on freezing and was not re-tested in this arm. On the masked first task, acquisition was infrequent in the feed-forward arms (S 0/3, H 1/5). We report those combinations as undecidable (Table D.3).

The study tests dependence on the implementation within a single benchmark. Section 5 distinguishes the definitions that transfer by construction, the behavioural results tested on both architectures, and the findings that require a learned soft router. Router collapse, the mismatch between training and deployment routing, and the shape term GshapeG_{\rm shape} (one-hot reference success minus success with the re-aimed constant mixture) remain specific to CM-Dreamer.

Two-skill banks in CM-Dreamer

Experiment. We evaluated nine existing checkpoints outside the original routing matrix. We used the routing channels of Appendix C.10, with 200 episodes per cell and paired seeds. The candidates were the end-of-stream checkpoints from the held-out drawer and window streams (three seeds each), plus the three CW8 checkpoints whose frozen modular arm had acquired a later task.

We specified the candidates and the rules for inclusion and pooling before this extension. The CW8 readouts and some window-stream results were already known. Inclusion required a slot-level skill (success of at least 0.5 with a slot forced alone) on both the first task and a later task in the 200-episode forced-slot sweep. Seven checkpoints met this rule. We list the two that failed the first-task requirement and exclude them from the pooled analysis (Appendix C.10, Table C.8).

Results. The constant-reference gain GselG_{\rm sel} is the success of the better of two fixed mixtures (uniform weights or the router’s mean weights within each evaluation shard) minus learned-deployment success. After pooling, ∣Gsel∣>0.2|G_{\rm sel}| > 0.2 in 4 of 15 later-task cells in two-skill banks. The registered one-sided binomial test uses the observed rate in the group originally classified as one-skill (5 of 45 first-task cells) as a fixed null rate (p=0.0765p = 0.0765, reported as 0.077 in Appendix C.4). It treats cells as independent even though cells can share a stream and training configuration. The verdict is sensitive in both directions: the +0.215 cell’s interval reaches 0.155, and including one checkpoint excluded for lacking a first-task skill would give 5 of 16 (p=0.026p = 0.026).

The shared base network with the expert contributions disabled (backbone alone) performs the first task in all seven banks (success 0.88–1.00). In both evaluated later-task cells where the backbone solved both tasks, deployed and assigned-reference success were equal (G=0G = 0). Where only the assigned slot holds the later-task skill, deployment loses 0.245–0.605 in four of five cells and shows no loss in the remaining cell.

Two of these cells are in the aimed group (Aroute≥0.5A_{\rm route} \ge 0.5) and still have G≥0.5G \ge 0.5. On the eight-task stream, the anchored checkpoint routes window-close to its assigned slot on 98.1% of steps and loses 0.605 at deployment (temporal term: fixed mean mixture minus time-varying deployment, 0.580). The window checkpoint has route accuracy 0.757 and loses 0.500 (shape term 0.675).

Conclusions. (i) The pre-registered verdict remains undecided. These data do not establish how often selection reduces success. (ii) The forced-slot sweep shows where it did so in these banks. Both backbone-solved later-task cells had G=0G = 0. Deployment reduced success when a single slot held the skill in 4/5 cells, as expected from the competence factor in Section 4.2.

(iii) The two additional aimed cells with G≥0.5G \ge 0.5 extend the three cases in Section 4.2. The eight-task checkpoint likewise shows a large gap despite high route accuracy. In both new cells, the decomposition attributes the loss to mixture shape or timing. (iv) The stratum now includes 15 later-task cells. For the seven newly included banks, the forced-slot sweeps and Eq. 2 path terms address the multi-skill question raised in the reviews.

Code. The anonymous code release is available at https://anonymous.4open.science/r/nfnr-protocol-F8A9. It includes the unified configuration (Table A.10), the evaluation procedure (Appendix A.1) and make tables, which rebuilds the script-generated tables on a CPU and compares them with the revised PDF; the README maps tables to their sources.

Changes in the revised PDF

  • Section 1 states in the third contribution how retention and retrieval can be handled separately.

  • Section 2 cites Davari et al. (2022), distinguishes probing from evaluating the agent's own forward pass, and describes the two additions to OWL's evaluation.

  • Section 4.2 reports the signed gap. G<0G < 0 in 28 of 150 cells.

  • Section 5 groups the training factors by level in Table 2, distinguishes three levels of evidence for transfer, and defines the scope of the second-architecture study.

  • Appendix A.1 and Table A.10 provide the unified configuration and evaluation procedure.

  • The caption of Table B.6 reports the seed-stratified exact permutation test.

  • The abstract and contribution statements describe the same measurements and retain all reported numbers.

  • Appendix C.4, Table C.2, Appendix C.10 and Table C.8 report the second extension of the two-skill stratum, bringing the pooled count to 15 later-task cells. The verdict remains undecided. Section 4.2 refers to this analysis.

  • Appendix C.8 and Tables C.5–C.6 report sensitivity to the competence threshold, evaluate a bank-best reference and correct one statement.

  • Appendix C.9 and Table C.7 report the retrieval gap for each training seed.

  • Appendix D, Tables D.1–D.6 and Figure D.1 report the pre-specified tests and supplementary analyses in a second architecture family.

  • We added statements on AI use and reproducibility.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.