acceptodds
Under review as a conference paper at ICLR 2027

Removing Sequentially Trained Modules: Effects on Retained Sources and Subsequent Learning

Abstract

We show that teaching a language model new information through separate, trainable components does not ensure that removing a component leaves what other components learned unaffected, and that removal can make subsequent learning easier. This matters for models that are continually learning from new datasets or environments while needing to dynamically disable some components and leave others activated. We test this by giving each dataset its own separately trainable component, called a module, while freezing the original model weights and varying which modules are active while subsequent modules learn. On synthetic tasks across five seeds we find that removing the first module trained reduces accuracy on the remaining datasets by 93.7 percentage points; when randomly disabling each earlier-trained module with probability .75 during training we avoid this observed degradation entirely at the cost of 58.3% more fitting steps. A separate three-seed experiment showed that removing four sources and their recorded dependent modules reduced the total steps needed to learn a new source to 43.6% of the no-removal baseline, although this removed all eight earlier modules.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, machine unlearning, modular neural networks, language models, parameter-efficient fine-tuning, LoRA, knowledge deletion

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_VkxxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents an empirical study of removing sequentially trained, source-specific modules from a language model. Using synthetic biographies, it trains separate residual modules while freezing the backbone and previously trained modules, and varies whether earlier modules remain active during subsequent training. The experiments examine effects on retained-source accuracy, module-level dropout, dependency-based cascade deletion, the composition of independently trained modules, and subsequent learning. The paper reports that parameter isolation does not ensure functional independence, that dropout can reduce deletion damage at additional training cost, and that deletion can reduce subsequent fitting effort in some tested settings.

Critical strengths
  1. A relevant and clearly presented practical question. Understanding how module removal affects knowledge associated with retained sources is important for modular unlearning and continual learning. I appreciate the authors' focus on the distinction between assigning separate parameters to sources and ensuring that their learned computations remain independent. This is a worthwhile issue to investigate, and the presentation makes the experimental setup easy to follow.
Critical weaknesses
  1. The central empirical contribution would benefit from a more focused explanation. The practical motivation is reasonable, but I found it difficult to identify the specific unresolved question that the study answers relative to existing modular approaches. Degradation after removing a module that was active during training is a plausible consequence of functional dependence, even when its parameters were frozen. The effects of module-level dropout, the difficulty of combining independently trained modules, and changes in subsequent fitting effort are also plausible under changes to the active computation. Empirical evidence about these effects can still be valuable. To make that value clearer, I encourage the authors to identify one or two central findings and explain what their measurements establish about the conditions, magnitude, or practical trade-offs of these effects. At present, this connection is not developed enough for me to see a substantial contribution beyond the qualitative observations.

  2. Additional validation is needed to establish practical relevance. The synthetic biographies and internal comparisons provide a controlled setting for studying module dependence. However, the evaluation does not include comparisons with established modular learning or unlearning methods on an existing benchmark or real-data task, and the principal evaluation ranks candidate answers to questions seen during training. This leaves uncertainty about how broadly the findings apply. I appreciate that the authors acknowledge these limitations. A focused comparison with a relevant existing method on an established benchmark or real-data task would help establish whether the key findings persist in a more representative setting. This evidence is important to my assessment of the empirical contribution; addressing the concern would require additional evaluation as well as clearer discussion.

Questions for authors

Clarification requests:

  1. Could the authors state the central research question more explicitly and identify the one or two findings they consider the main contribution? It would help to explain what these results add to existing understanding and how they could inform the design or evaluation of modular systems.
  2. Could the authors clarify how the internal experimental conditions relate to existing modular learning or unlearning methods, and which conclusions they expect to extend beyond the tested architecture and synthetic-data protocol? Any available evidence supporting that scope would be helpful.

Suggested additional validation: I encourage a focused evaluation of the main finding using a relevant existing method and an established benchmark or real-data task. This would materially affect my assessment by connecting the controlled observations to practical modular systems. It is a request for additional empirical support, separate from the clarification questions above.

2 comments
AuthorsOpenReview ↗

Response to Reviewer Vkxx

Thank you. We added the established-method experiment you requested and will focus the paper on what happens to retained tasks when sequentially trained modules are removed.

Central question and findings. We ask when a source-specific module can be removed without damaging retained tasks. Two findings organize the revision. First, separate frozen parameters do not ensure functional independence; our additional few-shot transfer experiment (separate follow-up comment) also shows that this dependence can accompany a useful accuracy advantage. Second, the effectiveness and cost of protection depend on the setting: dropout substantially reduces damage on the disjoint benchmark, but provides only partial protection in the related-task transfer experiment. We do not claim that co-adaptation or dropout robustness is new.

Progressive Prompts. Using the released T5-large implementation with residual prompt reparameterization, we ran DBpedia -> Amazon -> Yahoo -> AG News (16 training examples per class, three seeds, held-out test subsets), with earlier prompts and the backbone frozen. We fixed the acquisition threshold (intact accuracy at least chance + 15 points) and the deletion comparisons before seeing the first deletion results. Changes in percentage points, mean +/- sample SD:

EvaluationPublished-method implementationPrior-prompt dropout, p=.75
Yahoo: first prompt masked (primary)-3.2 +/- 3.3-0.5 +/- 1.1
AG News: first prompt masked (primary)-7.1 +/- 10.4+0.2 +/- 0.8
AG News: first two prompts masked (secondary)-74.1 +/- 21.7-0.7 +/- 1.5
Amazon acquired: seeds1/30/3

Both methods acquired Yahoo and AG News in all three seeds. Dropout kept the observed changes on these tasks small at a cost of about 0.7 points of intact accuracy, but it never acquired Amazon. We do not read this as a guarantee: three-seed uncertainty is wide, and retained utility does not show that the deleted task was forgotten.

Retraining reference. Retraining tasks 2-4 with matched data and initialization acquired Amazon in all seeds. Gaps to retraining (Yahoo / AG News) were -1.7 / +0.6 intact and -4.9 / -6.5 after masking for the published method, and -2.3 / -0.1 intact and -2.8 / +0.1 after masking for dropout. These gaps reflect the original training context as well as removal.

Output format. Deletion sometimes caused T5 to emit sentinel tokens instead of labels. Under exploratory label-likelihood ranking, two-prompt AG News losses ranged from -0.3 to -46.5 points across seeds. Ranking changes the decision rule, so it cannot cleanly separate formatting failures from classification failures.

Reproduction and selection history. Four-task intact accuracy was 70.2 +/- 2.1 (dropout: 68.5 +/- 1.4), versus 75.2 for this order in Progressive Prompts Table 1(a). We used a fixed training budget and differ in other disclosed details, so this is not an exact reproduction. The released sampler fixes the Amazon, Yahoo and AG News training subsets and all test subsets across seeds, so the SDs exclude that resampling. We chose the LayerNorm variant after earlier no-LayerNorm development runs; the appendix will report those runs separately without pooling, and we will release code, logs and per-seed results.

Thank you for the clarifications and additional real experiments. My main concern about the limited contribution beyond expected observations remains unresolved. I maintain my score.

Reviewer_RbwTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors experiment with modules, additional trainable parameters whose outputs are added to a model’s existing computation. They study what happens when multiple modules are trained sequentially, with earlier modules still active (or potentially dropped out) but frozen. They find that removing the first module severely degrades performance on subsequent synthetic tasks. These tasks involve biographies of invented people with assigned facts about them. Randomly dropping out previous modules during training can avoid this observed degradation at the cost of more training steps. If multiple modules are trained independently and then all activated simultaneously, performance collapses, despite high accuracy when each module is activated individually. The authors demonstrate that later modules can become dependent on previously trained modules through their contributions to the forward pass, even when their parameters are frozen. Module-level dropout reduces this dependence and improves robustness to removal.

Critical strengths
  • The primary experimental results are clean. The controlled experiments clearly demonstrate that there is a functional dependence of modules trained later on the ones trained earlier when modules trained earlier are removed. The main finding is consistent across seeds and model sizes; the experiments are done carefully. This work highlights the distinction between parameter isolation and functional independence.
Critical weaknesses
  • Limited conceptual novelty. It is not surprising that later adapters can become dependent on earlier adapters, as the later adapters were trained while the earlier adapters were active. Those later adapters take in as input the output of earlier adapters (from earlier layers) during training. Dropout is an expected and intuitive technique to reduce this dependence. While the magnitude of the observed degradation is surprising, the authors provide little mechanistic explanation of why this sort of dramatic dependence occurs.
  • Narrow settings. The general point of this work is about the effect of training modules sequentially on a variety of tasks, and understanding the effect of removing them. However, the tasks this work studies are narrow. The primary experiment hinges on three very small synthetic biography datasets. The authors do not attempt to study more realistic settings. The authors do not study task learning, as opposed to only fact learning.
Questions for authors
  1. Do the deletion harms and dropout benefits persist in more realistic settings when modules are trained to learn different tasks or domains, and are evaluated on held-out examples?
  2. Have the authors attempted to understand the underlying mechanism behind the dependence, even when the sources for each module are disjoint? One hypothesis I have is that the subsequent modules learned to use the outputs of earlier modules (in terms of layer and training) as input. Disabling those modules used during training is akin to input distribution shift.
  3. When all independently trained modules are activated together, how do the norms of the outputs at each layer look like, and how do they compare with the norm of the hidden states with a single-module activated? Does normalizing or rescaling outputs recover performance? This may help bring insight to the underlying mechanism, and could be explored in future work.
2 comments
AuthorsOpenReview ↗

Response to Reviewer RbwT

Thank you. We agree that input-distribution shift is a plausible account, and we tested your proposed checks on saved checkpoints. The results fit distribution-shift sensitivity but do not single out one mechanism.

Dependence. Post-deletion accuracy tracks training exposure to the deleted module's absence better than exact evaluated-mask frequency (KPYR reply). Replacing a deleted lane with its per-layer mean restores 21-47% retained accuracy, versus 96-99% intact.

Norms and rescaling. Across five seeds per size, the sum of independent lanes averages 4-6% of the residual-stream norm and 31-32% of the FFN-output norm; these are layer averages, not bounds (the first-layer ratio at 135M is about 22%). Scalar factors {1, .75, .5, 1/3, .25}, selected on half the questions and evaluated on the other half, left accuracy at about 5-11%, versus 95-97% for the correct lane alone and 2-4% for unscaled addition. This rules out only this scalar grid, and does not test normalization or merging methods in general.

Centring. Keeping the correct lane and replacing the others by their means gives 3-6% accuracy; subtracting the others' means instead gives 15% / 24% / 50% at 135M / 360M / 1.7B. So constant offsets can hurt and centring partly helps. Centring also changes output norms, leaves a large gap and needs to know the correct lane, so it is an oracle diagnostic rather than a usable fix.

External validity. We added Progressive Prompts on the standard four-task T5 benchmark (reply to Vkxx). We will restrict the composition claim to ungated addition.

Thank you for the response. I will maintain my score.

Reviewer_srmjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines what happens when an earlier module is removed from a language model that learns new data through sequentially trained modules. Each data source has its own low-rank residual module, and only the new module is updated during training. Using synthetic biographies, the authors show that later modules can rely on earlier modules even when their parameters are frozen. Removing an earlier module can therefore sharply reduce accuracy on the sources that remain. Randomly disabling earlier modules during training reduces this damage, though more training steps are needed. These results are tested across five seeds and three SmolLM2 models ranging from 135M to 1.7B parameters. Further experiments explore deleting dependent modules together, routing questions to independently trained modules, and whether shared facts remain accessible after a source’s module is removed.

Critical strengths
  1. The comparison between prior-active and independent training is convincing. Both use the same datasets and module architecture, but deleting the first module reduces retained-source accuracy by 93.7 percentage points in the former and leaves it unchanged in the latter. Similar results across five seeds and three model sizes support the finding that freezing parameters does not prevent later modules from depending on earlier ones.
  2. The paper shows both the benefit and the cost of module-level dropout. In the 135M experiment, p = 0.75 prevents the observed accuracy loss for the tested deletions, but takes 58.3% more fitting steps than p = 0. This gives a concrete measure of the extra training needed to reduce deletion damage.
Critical weaknesses
  1. The comparison with independent training and routing remains incomplete. Independent training has zero deletion collateral when each source’s own lane is selected and requires fewer fitting steps than p = 0.75 (3,080 vs. 3,800; Table 1). Activating all independent lanes together reduces accuracy to 2.53%, and the paper separately tests learned routing (Section 3.4 and Appendix D). A comparison of independent training with routing against module-level dropout under matched data and evaluation conditions, including routing costs, would clarify when dropout is preferable.
  2. The experiments leave the generality of the findings uncertain. Each synthetic source contains ten biographies, and the primary evaluation ranks candidate answers to questions seen during training (Section 2.2). The model-size comparison uses the same source sets and one model family (Section 3.7 and Table 4). These experiments establish deletion damage in the tested setting, but evaluation on unseen question formulations and additional datasets is needed to assess how broadly the damage and dropout benefits carry over.
Questions for authors
  1. Sections 3.1–3.2 show that later-trained lanes can depend on earlier lanes and that module-level dropout can reduce this dependence. Could further characterize where these dependencies arise and clarify how the proposed approach differs from related modular training methods?
  2. In Section 3.6, could repeat the comparison using the same new-module initialization and data order across deletion conditions, while including the evaluation configuration during training? Does deletion still reduce fitting steps under these controls?
1 comment
AuthorsOpenReview ↗

Response to Reviewer srmj

Thank you. We completed the matched routing comparison. For subsequent learning, the initialization/order control is complete, but we have not run your evaluation-mask control.

Independent modules with the same router. For each seed, one small router trained on frozen-backbone features supplied the same predictions to every condition. Five seeds per size; accuracy (%) and lane-training steps, +/- denotes sample SD:

SizeRouter validationIndependent: retained after deletionp=.75: retained after deletionSteps: independent / p=.75
135M100.098.2 +/- 2.697.2 +/- 1.63,080 / 3,800
360M100.096.2 +/- 2.997.8 +/- 1.92,520 / 3,360
1.7B99.795.5 +/- 3.097.5 +/- 2.91,760 / 2,240

Independent modules reach similar observed retained accuracy with 19-25% fewer lane-training steps; we did not run an equivalence test. The router's cost is 25-50 classifier updates plus feature extraction rather than language-model training steps, and source identity is unusually easy to infer in these disjoint data. We agree these results give no reason to prefer prior-active training on this benchmark.

Q1: where dependence arises. Earlier lanes change the hidden states that enter deeper layers, and later lanes are optimized on those states, so freezing the earlier weights does not make downstream computation invariant to their removal. That explains how dependence can arise; we have not localized it to particular layers. The mean-replacement diagnostic shows only that the constant replacement we tested does not restore intact performance. The revision will make this distinction explicit and compare the setup with existing frozen-component and component-dropout methods.

Subsequent learning (submitted Section 3.6). The follow-up matched new-module initialization, data order, and mask streams, and the lower fitting cost after deletion remained. It does not address your evaluation-mask control: with eight earlier modules and uniform p=.75, the all-active fit-check configuration has probability .25^8 per step, so removing modules changes the mismatch between training and evaluation. A single-seed no-dropout diagnostic went the other way (400 steps with earlier modules versus 1,000 without).

We therefore withdraw the broader plasticity interpretation; we will remove the 43.6% figure from the abstract and move the result to the appendix as a protocol-specific fitting-cost observation with an unresolved confound.

Reviewer_KPYROpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a modular architecture with per-dataset low-rank residual "lanes" that are trained sequentially on an otherwise frozen language model. It demonstrates that processing done by the later lanes can become functionally dependent on ealier frozen lanes. As a result, deleting earlier modules can cause significant performance degradation on retained datasets (~94pp without dropout). The authors show that module-level dropout can remove this damage at the cost of additional fitting steps (while independently trained lanes do not compose when summed). The authors then record structural and functional dependence graphs between the lanes, study cascade lane deletion and its cost in additionally removed datasets. They also show that facts shared between datasets survive lane deletion. The experiments utilize synthetic biographies and are studied with SmolLM2 models at 135M, 360M and 1.7B parameters.

Critical strengths
  • The paper has clean and controlled experimental design. The model backbone is frozen and the authors take special care to ensure that the observed damage can be attributed purely to the functional dependence of later lanes on earlier ones. Usage of synthetic data gives perfect control over which facts are shared, which are independent and conflicting. The authors do an in-depth analysis of task inter-dependencies, module dropout and the costs of combining and removing individual lanes following a variety of possible choices.
  • The authors use a careful evaluation strategy and thoughtfully report their findings. They discuss paired confidence intervals over seeds, a fixed seed-candidate order, disclose failed seeds, etc. Negative and contrary results all also reported. They also discuss study reproducibility and release their code.
  • The authors proposed a very useful decomposition of the deletion cost. They discussed the damage to retained sources, number of extra sources removed by the cascades, fraction of answers surviving through shared facts, and learning costs.
  • Even thought the authors consider a single model family, they replicate their main results across a wide parameter range.
Critical weaknesses
  • The paper reads like an experiment report. It does not explain the significance of its results, does not reflect on its findings in the context of earlier work (maybe contradicting or explaining it), does not hint at the future work, and only briefly mentions possible applications. In my opinion, a potential impact of this work is not adequately discussed and it could be hard for a reader to understand "why" this study had to be done.
  • While the design of experiments was very well thought through, the technique itself has very limited novelty and the findings are largely expected given prior work. Damage caused by module deletion and dropout as a remdedy against it are well established in different contexts ("Deep Networks with Stochastic Depth", "Residual Networks Behave Like Ensembles of Relatively Shallow Networks", "Reducing Transformer Depth on Demand with Structured Dropout", etc., some are cited in the paper). AdapterDrop previously discussed random dropout of per-task adapters. And the interference of independently trained adapters is well known ("Editing Models with Task Arithmetic", "TIES-Merging: Resolving Interference When Merging Models", etc).
  • Most results were not surprising or particularly revealing. Some were even too trivial and true by construction (like deletion after independent training). Perhaps the most intriguing observation in Table 1 of a positive collateral can potentially be explained by the frequency of lane combinations during training (with all 3 lanes being present being the least likely combination for p=0.75).
  • The experimental design is sufficiently thorough, but the chosen datasets are very simplistic and the authors have not attempted to extend at least some of their results to more realistic datasets.
  • Many potentially interesting extensions targeting mechanistic analysis of the interaction between the lanes, or on attempts to fix the damage were not done. For example, replacing a deleted lane by its mean output would test whether later lanes simply depend on a constant shift. Comparison with alternative deletion approaches (except for LoRA) is also missing.
Questions for authors
  • What is the main practical implication of these findings? Are there any implications for practitioners, something that they could be doing differently?
  • Do later lanes depend on the specific function learned by earlier lanes, or simply on a shift they introduce? Can the authors propose and conduct a study that would establish this? Perhaps the damage can be easily reparied by simply introducing a constant shift?
  • Can the positive collateral at p=0.75 be explained by the frequency of lane combinations during training? (see above)
  • Would it possible to conduct a more realistic experiment using real data with known dependencies, task overlap, or conflicts?
2 comments
AuthorsOpenReview ↗

Response to Reviewer KPYR

Thank you.

Q1: significance and presentation. We agree that the submission left the motivating question and its implications unclear. The revised introduction will start from the gap between freezing earlier parameters and keeping retained tasks accurate once those parameters are removed. We will group the evidence by removal damage, what protection costs and where it fails, and the difference between removing parameters and forgetting behavior. Practitioners should test the removals their deployment will perform; when modules need not share computation, independent training with routing is simpler. We now also test removal in an established method on held-out real-task examples (Progressive Prompts; reply to Vkxx). The revision will compare our setting more directly with prior modular-learning and component-dropout methods; we contribute the removal study and do not claim those mechanisms as new.

Q2: constant shift. After deleting lane 0 from p=0 runs, we replaced its output with a per-layer mean estimated on a control source never used for training, or with per-position outputs drawn at random from the lane's outputs on that source. Five seeds per size; retained accuracy (%), +/- denotes sample SD:

SizeIntactRemovedMean replacementRandom replacement
135M96.22.620.7 +/- 2.215.9 +/- 1.1
360M99.10.628.1 +/- 4.021.8 +/- 2.3
1.7B98.31.647.0 +/- 5.740.5 +/- 5.7

Constant replacement recovers part of the lost accuracy, more at 1.7B, but stays well below intact accuracy and cannot separate constant from input-dependent contributions. It also raises accuracy on questions from the deleted source from about zero to 10-18%, so this intervention does not demonstrate clean unlearning.

Q3: positive collateral and stopping. At p=.75, the all-active configuration used for lane 2's fit check occurs in about 6% of training steps; some post-deletion configurations occur 3-9 times more often. For source 2 after deleting lane 0, the exact post-deletion configuration occurred equally often at p=.25 and p=.75 (about 19% of steps), yet retained accuracy rose from 77.5-82.7% to 95.6-96.6% as exposure to lane 0's absence rose from 25% to 75%. Training duration also changes, so we report this as an association. The reported 58.3% extra steps remain conditional on the all-active stopping rule.

I would like to thank the authors for their response and additional thorough experiments. In my opinion, the new real-data experiments make the study more meaningful. As we would expect, keeping an earlier components improve the performance on a related task (removing which breaks the predictor) and training with randomly disabled components does not preserve that benefit after removal. The authors also tested replacing modules with their mean responses and showed that this helps improve performance, but just a little.

While I think that the paper became stronger, I still maintain my concern about a limited novelty and a lack of deep new insights. Because of that I will currently maintain my score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response to all reviewers and the AC

Thank you for the specific suggestions. We agree that the paper is an empirical study of module removal and does not propose a new dropout, routing, or merging method. The individual replies detail three additions:

  1. An existing method on real tasks (Vkxx). In Progressive Prompts (T5-large, DBpedia -> Amazon -> Yahoo -> AG News, three seeds), removing earlier frozen prompts sometimes caused large losses on later tasks; prior-prompt dropout kept the observed losses small on acquired tasks but never acquired Amazon.
  2. Mechanism diagnostics (KPYR, RbwT). We tested mean replacement, scalar rescaling and centring on saved checkpoints.
  3. A matched router comparison (srmj). Independent modules reached similar retained accuracy with fewer training steps.

We will remove the subsequent-learning number from the abstract and move that result to the appendix, because the mismatch between training and evaluation masks prevents us from using this result to support a general claim that deletion restores learning ability. The revised introduction and discussion will state the central question and findings explicitly. A revised PDF and change list will follow.

AuthorsOpenReview ↗

Follow-up: transfer benefit and removal cost

Our disjoint-source and four-task comparisons did not establish a matched forward-transfer benefit from keeping earlier modules active, so we ran a prespecified few-shot transfer test in the same prompt framework. We fixed the protocol, thresholds and analysis in advance and report one development seed separately. We trained a 50-token Amazon prompt (20 examples per class, T5-large) with a learned Yelp prompt present and compared it with three controls: a 100-token prompt, the untrained Yelp prompt, and training with the Yelp prompt masked. Three fresh seeds vary initialization and training subsets; the 1,250-example test set is fixed. Accuracy (%), mean +/- sample SD:

Target prompt trained withIntactYelp prompt removed
Yelp prompt present49.7 +/- 6.50.0 +/- 0.0
Yelp-prompt dropout, p=.7541.8 +/- 1.825.5 +/- 3.7
Strongest control within each seed34.6 +/- 4.2-

Prior-active training beat the strongest control by 15.1 points (9.0, 19.8, 16.4), meeting the registered transfer criterion. Removing the Yelp prompt made that predictor emit sentinel tokens for all 3,750 test inputs. Dropout avoided this format failure but cost 7.9 points of intact accuracy on average, and removal still cost 16.3 points: none of the four registered protection criteria was met. Adding the Yelp prompt only at evaluation to a prompt trained without it was not a reliable substitute (-13.4, +14.3, -5.5 points). The evidence comes from one task pair and three seeds, with wide intervals. Within that scope, the dependence was useful and dropout protected against removal only partially. We will add the experiment to Section 3.6 and the appendix.

AuthorsOpenReview ↗

Revised PDF and supplementary material uploaded

We have incorporated the additional experiments and clarified the framing and limitations. The main changes are:

  • Abstract, §1 and §§4–6: a clearer statement of the research question, main findings, relationship to prior work, practical implications and remaining limitations.
  • §§3.1–3.4 and Appendices F–G: saved-checkpoint replacement, mask-frequency, norm and rescaling diagnostics, plus the matched-router comparison. We distinguish associations from causal explanations, disclose the router's validation-split limitation, and limit the composition finding to ungated addition.
  • §3.6 and Appendix H: Progressive Prompts results on held-out classification tasks, including acquisition failures, retraining comparisons, exploratory label ranking and implementation differences.
  • §3.7, Table 5 and Appendix I: the Yelp-to-Amazon transfer-and-removal experiment, with development and confirmation results separated and all four failed protection criteria reported. Our earlier follow-up referred to this material as §3.6; it is now §3.7.
  • Appendix H.5: both earlier development configurations are disclosed separately from the final runs, including completed, interrupted and unstarted runs.
  • Submitted §3.6 is now Appendix C: we removed the subsequent-learning number from the abstract and withdrew the broader plasticity interpretation. Appendix C.5 explains the unresolved training/fit-check mismatch.
  • Supplementary material: we retained the original release and added rebuttal_additions/, containing the rebuttal experiment code, protocols, final and development results, and scripts for regenerating the reported summaries and tables. The accompanying README maps these files to the revised paper.
AuthorsOpenReview ↗

Correction revision uploaded

We uploaded a corrected PDF and supplement. The headline results are unchanged. The corrections are:

  • Dependency graphs (§3.3, Table 8): drops exactly at the stated threshold now count as edges, as the definition says. This changes the Oracle adaptive row to 7.3 edges (26.2%) and the ten-point value in §3.3 from .38 to .42; no tested joint removal changes. A script in the supplement reproduces both the old and corrected values from the recorded counts.
  • Router comparison (abstract, §1, §3.4, §5–6): we now say only that independently trained modules with routing reached similar accuracy with fewer fitting steps than dropout. Prior-active training without dropout fits in fewer steps than independent training (Table 1), so our earlier "no advantage" wording was too broad.
  • Prior work (§1, §4): we acknowledge Kunz and Holmström (2024), who removed a frozen language adapter after training with it, and cite Grimes et al. (2026) and S³T (Chowdhury et al., 2025). We also corrected our description of O³.
  • Transfer test (§3.7, §5, Appendix I): Yelp and Amazon share their labels, and we did not score the Yelp prompt alone on Amazon, so the test does not separate reuse of the Yelp prompt from improvement that required training the Amazon prompt with it. The supplement adds the exact paired McNemar tests the protocol specified.
  • Smaller fixes: the scope of the 135M-only statement (§3.8), the activation change in the development configuration (Appendix H), and the four-task result, now described as showing no evidence of a benefit.
  • Supplement v2 adds these scripts and a layout helper for the training scripts, and fixes a line-ending issue that made the table check fail on Linux. CHANGES_v2.md lists the changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.