- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3
Read review
Paper summary
This paper investigates whether a single language model trained with structured dropout can be deployed as standalone, router-free subnetworks ("members") for individual domains, arbitrary domain subsets, or narrower FFN widths simply by slicing rows and columns out of the FFN matrices. The proposed region dropout partitions every FFN layer into a shared pool and one contiguous region per training domain, leaving embeddings, attention, layer norms, and the output head shared. Within each region, the dropout rate increases linearly with the unit index so that early units form valid prefix subnetworks. Across regions, on a training batch from domain , each foreign domain region is included and updated with probability ("exposure"), which the main recipe ramps linearly from to over training. The authors evaluate this setup across 30M, 51.3M (16x widened FFN), 124M, and 1.3B parameter transformers:
- Combining Regions: Training without exposure () causes severe interference when regions are summed without a router: the all-region model suffers a 7.88–11.52 nat loss penalty on code and drops from 192/192 to 5/192 (30M) and 53/192 (124M) on held-out two-digit arithmetic. Ramping exposure () preserves 192/192 arithmetic accuracy and keeps full-width composition penalties nats on a 512-document external panel at 30M and nats at 1.3B. A matched four-arm ablation (Section 5.1, Appendix X.2) demonstrates that neither detached forward co-activation alone nor foreign gradient updates alone suffices; both are required.
- Shrinking & Physical Export: In the 51.3M 16x-FFN model, exporting the code member at 75% width reduces parameters to 32.3M and yields a 1.53–1.58x batch-16 forward speedup at a 0.042–0.049 nat loss cost.
- Removing a Region: Deleting the arithmetic region reduces accuracy on its task from 188/192 and 187/192 to 1/192 and 0/192 while changing losses on retained domains by nats, though 2M tokens of retraining on arithmetic restores 177/192 and 179/192 correct answers.
Critical strengths
- Clean mechanistic isolation of unweighted subnetwork composition: The paper poses a well-defined question—what training conditions allow domain-assigned FFN blocks to be summed with unit weights at inference without a router or posterior weighting? Across 30M, 124M, and 1.3B scales, and especially in the matched four-arm study (Appendix X.2) separating forward co-activation (with detached weights) from auxiliary foreign-gradient updates, the authors convincingly show that regions must both co-activate in the forward pass and receive cross-domain gradients to compose cleanly.
- Thorough, candid self-auditing and negative controls: The authors test and report the exact boundaries where their approach stops working. They document that pruning a dense model with just 1M tokens of repair beats their zero-shot narrower exports at every width, that 2M tokens of finetuning undoes region removal even when paired with RMU unlearning, that ramp exposure fails on arithmetic when preceded by Python context, and that standard 4x FFN transformers gain little parameter reduction or batch-1 speedup from FFN slicing.
Critical weaknesses
- Narrow practical utility of the exported models relative to standard baselines: Across extraction, shrinking, combination, and removal, the practical case for region dropout is weak:
- Single-domain extraction & shrinking: Extracted code specialists lose 0.06–0.13 nats against dense models trained on the same budget, and at 4x the code tokens, training on code alone beats extraction. Furthermore, if a tiny repair budget of 1M tokens (~1.2% of pretraining) is allowed after pruning a dense model, magnitude and activation-informed pruning outperform region dropout's ordered slices across all four widths (Appendix H.2). On standard 4x FFN architectures—and at batch size 1 or uncached generation on A100 even with a 16x FFN—physical slicing yields almost no wall-clock speedup (0.92–1.07x) because embeddings and attention are unpartitioned.
- Cross-domain context interference: While ramp exposure achieves low composition penalties on homogeneous single-domain documents, Appendix Y reveals a severe side effect on mixed-domain inputs: placing a short Python snippet before each arithmetic equation causes the ramp arithmetic and combined code+arithmetic members to score 0/192 or 1/192 (continuing the Python text instead), whereas the unexposed () arithmetic member still solves 136/192 and 134/192.
- Fragile capability removal: Structural deletion of the arithmetic region is undone within 2M tokens of finetuning on the task (recovering >92% accuracy).
- Poorly integrated 45-page appendix with legacy terminology and raw debugging logs: The 9-page main text relies heavily on Appendices A–Y for experimental details and tables. However, page 12 notes that the appendices retain old terminology from an earlier draft ("block", "product", "singleton", "parent", "", "factorized"), forcing the reader to constantly translate terms. Moreover, several appendices read like unedited internal commit logs—documenting a bug where exposure gates were applied twice () in earlier runs (Appendix K), a leaked WikiText test article (The Hustler) in the training cache (Appendix D), and unrecovered data generation scripts (Appendix J).
Questions for authors
- Clarification on the Python-prefixed arithmetic failure (Appendix Y): Why does prepending Python context cause the ramp-trained arithmetic and code+arithmetic members to drop from 192/192 to 0–1/192 while the arithmetic member retains 136/192? Does training foreign regions on code data () cause the arithmetic region and shared attention layers to prioritize continuing Python syntax whenever code appears in the context window? If a combined multi-domain member fails when a prompt mixes two of its domains, when would a practitioner deploy a combined member instead of routing between single-domain specialists?
- Clarification on the 1.3B MBPP evaluation (Appendix U.1): At 1.3B parameters (trained on 26B tokens including 6.5B StarCoder tokens), exported members solve at most 7/257 problems on sanitized MBPP. Why was MBPP evaluated at 84,000 of 99,183 updates rather than the final checkpoint, and how many problems does an unpartitioned dense 1.3B model trained on the exact same 26B-token mixture solve under your 1-shot greedy protocol?
- Suggestion for manuscript revision: Standardize the terminology across the main text and all appendices, and condense the historical audit/bug logs into a clean, self-contained supplementary section.
3 comments
(1) Routing. On documents from one domain the multi-domain model is within 0.05 nats of the single-domain one (0.11 on the external test set), so a router does not pay off; on the Python-text test the model a router would choose fails as well.
(2) Removal. Against an attacker who holds the original data, removal slows relearning: 56% against 93% of the problems after 0.4M tokens.
(3) 1.3B baseline. Over the last ten checkpoints it solves 3 to 19 of 257 MBPP problems, our network 3 to 9 (19 and 3 at the final checkpoint). The 84,000 was our error.
Multi-domain model or routing (your first question). A router pays off only if the multi-domain model is worse than the single-domain model it would route to. On documents from one domain the difference is small: at 1.3B parameters, adding the other regions changes a domain's zero-shot task accuracy by −1.5 to +2.5 percentage points when trained with exposure, so one model serves all four domains without a router. Without exposure the same step lowers accuracy by 1.3 to 6.5 points, and by 22.5 on LAMBADA.
For the Python-text test (Appendix Y of the submission, now I.1), routing would not help either. The model a router would select, the arithmetic model on its own, solves 0 or 1 of 192 problems once Python text precedes them, and so do baselines trained on the same data; all of them solve the 192 without the text. The arithmetic training data never place text before a problem, and we think the failure comes from that, not from combining regions.
Why does the model trained without exposure keep 136 and 134 answers? Its arithmetic region only ever processed arithmetic. With exposure the region is also trained on the other domains' data and, like a standard transformer, reads a line in the light of what precedes it. With exposure, the outputs after Python text are repeated punctuation or digits, empty answers and continued Python text. We think this is the same property that lets regions work together (without exposure the full model solves only 5 and 53 of the 192 problems even without added text), but we have not tested this directly. The revision states that every evaluation uses documents from one source and that mixed input can fail.
Removal. Removing the arithmetic region reduces correct answers from 188 and 187 of 192 to 1 and 0. Loss on the other domains changes by at most 0.003 nats, and no training is needed. You are right that relearning recovers the skill, so we compared removal with RMU unlearning against an attacker who holds the original data: relearning on the arithmetic training data themselves, five training runs per condition, 242 new test problems.
- After 0.4M relearning tokens, the network unlearned with RMU solves 93% of the problems again, the network without its arithmetic region 56% (95% interval across the five runs: 40 to 71), and removal followed by RMU 2%. With RMU hyperparameters tuned for our networks, the first and last figures are 92% and 10%.
- To regain 25 to 95% of the unmodified network's accuracy, the network without its region needs 1.6 to 9.7 times as many tokens as after RMU.
Removal does not prevent relearning; it slows it down, without any training.
MBPP and a 1.3B baseline (your second question). The 84,000 is our error: the table reports the final checkpoints at 99,183 updates. Re-running all 257 problems on the two code models reproduced 3 and 7 with identical outputs; the other counts are read from the stored per-problem records of the final evaluation. We trained the 1.3B baseline on the same data and 26B tokens and scored it 1-shot and greedy. Because the number of solved problems varies strongly between checkpoints, we scored the last ten saved checkpoints of both models: the baseline solves 3 to 19, our network 3 to 9 (19 and 3 at the final checkpoint). With pass/fail scoring, small differences in loss can produce large differences in solved problems (Schaeffer et al., 2023).
Terms and appendix. The revision uses one set of terms: "member" becomes "subnetwork" inside the full model and "extracted model" once it runs alone. The appendices use the same terms, so the glossary is gone. A corrections section (Appendix K) lists the corrections to the submitted version and maps its appendix letters to this version; earlier training setups are grouped in Appendix J.
Response to the authors
Thank you for a thorough and constructive rebuttal that directly resolves our main technical and presentation concerns. First, comparing extracted models and pruned dense baselines under \emph{equal} fine-tuning token budgets (Appendix F.6) changes the picture substantially: the extracted model leads by to nats without fine-tuning, wins all 32 comparisons across the four domains at 49k and 197k fine-tuning tokens (by to nats), and retains lower loss at narrow widths (168 and 336 units) through 1M--2M tokens without requiring per-subnetwork repair across a 60-model family. Second, the matched-compute comparisons clarify that the single-domain loss gap shrinks to nats in the 800M-token runs at 30M and 124M parameters, and that at total training tokens the extracted model is within nats of (or up to nats better than) a standalone specialist trained on domain tokens. Third, showing that standard dense baselines trained on the same data also score 0--1/192 on the Python-prefixed arithmetic test confirms that this behavior stems from the unprefixed arithmetic training format rather than region combination, while the new 5-seed relearning comparison against RMU unlearning ( vs.\ accuracy recovered at 0.4M tokens, and when combined with RMU; more tokens required to regain accuracy) establishes a clear advantage for structural removal over weight-only unlearning. Finally, we appreciate the 1.3B dense MBPP baseline comparison (3--19 vs.\ 3--9 solved across the final ten checkpoints, confirming that 84,000 was a typo for 99,183) and the complete unification of terminology across the main text and appendices. In recognition of these new matched controls and the manuscript cleanup, we have raised our presentation score from 2 to 3 and our overall rating from 2 to 3.
Thank you for a careful review. Your main concern is the practical case for extracted models against standard baselines. We measured each baseline you name with matched data and tokens, in all four domains.
(1) Pruning. Without fine-tuning, extracted models have 0.6 to 3.1 nats lower loss than baselines (standard transformers trained without region dropout) pruned to the same FFN width. With equal fine-tuning tokens for both, they keep the lower loss at every width up to 197k tokens, and at the narrowest width through 2M tokens. Appendix H.2 of the submission fine-tuned only the pruned models; the revision replaces it (Appendix F.6).
(2) Training on the domain alone. At full width, an extracted model has lower loss than a model of the same size trained only on that domain with the same domain tokens, in all four domains.
(3) Baselines. At 30M parameters an extracted model is within 0.04 to 0.11 nats (4 to 12% in perplexity) of a baseline of its size trained on the same data and tokens.
(4) Removal. After 0.4M relearning tokens, a network without its arithmetic region solves 56% of the test problems again; after RMU unlearning it solves 93%.
Details follow.
(a) Pruning. Our case is one training run and no training per model, so the first comparison is without fine-tuning: there an extracted model has 0.6 to 3.1 nats lower loss than a baseline pruned to the same FFN width. If fine-tuning is allowed, it is allowed for both: an extracted model is a standard transformer and can be fine-tuned, or pruned further, like any other. We therefore reran the comparison with equal fine-tuning tokens for both: five region-dropout training runs against three baselines, at 49k to 2M tokens, each model with the best of up to eight fine-tuning setups chosen on held-out data.
The extracted model is the better starting point without fine-tuning, with little fine-tuning, and at narrow widths. Pruning a baseline catches up only at half width or more and with 1M or more fine-tuning tokens per model. That is a different setting: every model is then trained on its own, and for our 60 models this would take 60M tokens, three quarters of the training run.
(b) Models trained only on the domain (validation data). At full width (672 units, 26.1M parameters) the extracted model has 0.07 to 0.24 nats lower loss than a model of the same size trained on the same 20M domain tokens, in all four domains. With four times the domain tokens, the model trained on the domain alone is better on code, WikiText and web text, by 0.47 to 0.68 nats, as you noted for code. That is a different setting: one model per domain, each trained on as many tokens as our whole run. Trained four times as long, our network is within 0.035 nats of it or better (web text 0.075 lower).
(c) Baselines of the same size, same data and tokens (validation data). An extracted model has 0.04 to 0.11 nats higher loss at 30M parameters with 80M tokens, and 0.00 to 0.05 in the 800M-token runs at 30M and 124M. On our external test set the gap is 0.47 to 0.54 nats for two models, those extracted from regions trained on preprocessed corpora (WikiText-103, Penn Treebank) and scored on raw text. Converting the documents to each corpus's format reduced the gap to 0.10 and 0.03 nats. An extracted model serves text in the format of its region's training data; the appendix gives the full table.
(d) Speed. The code model extracted at 75% width from the network with an FFN of 16 times the hidden width has 32.3M of 51.3M parameters and 1.53 to 1.58 times the throughput of the full model at batch size 16. As you note, batch-1 latency on an A100 does not improve: extracted models ran at 0.99 to 1.07 times the speed of the full model; there the benefit is throughput and memory. On an RTX 2080 Ti the extracted model is 1.3 to 1.5 times faster at batch 1.