acceptodds
Under review as a conference paper at ICLR 2027

Extracting, Removing and Shrinking Domains from One Jointly Trained Network

Abstract

Deploying language models for different sets of text domains and memory budgets usually means training a separate model for each. Dropout already trains many subnetworks inside one network but uses only their average. We ask whether single subnetworks can be deployed on their own, for one domain, several domains or a smaller budget, without a router or further training. We introduce region dropout, which assigns each domain a region of every feed-forward layer: within a region, the drop rate rises with a unit's position, so that early units form smaller models; across regions, other domains' regions are included and trained on each domain's data with a probability that rises during training, which we call exposure. Without exposure, combined models containing every unit of an arithmetic member solved 5 and 53 of 192 problems at 30M and 124M parameters that the member solved completely; with exposure, they solved all 192, and combined members at full width stayed within 0.11 nats of their single-domain members on external text and within 0.008 nats at 1.3B parameters, while the code member lost 0.06 to 0.13 nats against dense models. In a matched study, neither joint inclusion alone nor training regions on other domains' data alone kept the largest loss increase from combining below 0.10 nats. Weighting the regions of models trained without exposure by a domain posterior, as in expert mixtures, gave 0.03 to 0.12 nats lower text loss than exposure at 5.6 times the scoring time but solved at most 60 of the 192 problems. Removing a region reduced the exact answers on its task from 188 and 187 to 1 and 0 of 192 and left the other domains within 0.003 nats; retraining on the task restored most answers. A code member exported at 75% width kept 32.3M of 51.3M parameters at 0.05 nats above full width; narrowed combinations kept fewer exact answers. One network trained with region dropout thus provides models for each subset of its domains, without a domain, or for single domains at smaller width, on text from these domains.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
dropout, modular language models, subnetwork extraction, model composition, domain experts, nested dropout, elastic inference, domain removal

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_y5v7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
3
Read review
Paper summary

This paper investigates whether a single language model trained with structured dropout can be deployed as standalone, router-free subnetworks ("members") for individual domains, arbitrary domain subsets, or narrower FFN widths simply by slicing rows and columns out of the FFN matrices. The proposed region dropout partitions every FFN layer into a shared pool and one contiguous region per training domain, leaving embeddings, attention, layer norms, and the output head shared. Within each region, the dropout rate increases linearly with the unit index so that early units form valid prefix subnetworks. Across regions, on a training batch from domain dd, each foreign domain region is included and updated with probability p(t)p(t) ("exposure"), which the main recipe ramps linearly from 00 to 0.350.35 over training. The authors evaluate this setup across 30M, 51.3M (16x widened FFN), 124M, and 1.3B parameter transformers:

  • Combining Regions: Training without exposure (p=0p=0) causes severe interference when regions are summed without a router: the all-region model suffers a 7.88–11.52 nat loss penalty on code and drops from 192/192 to 5/192 (30M) and 53/192 (124M) on held-out two-digit arithmetic. Ramping exposure (0→0.350 \to 0.35) preserves 192/192 arithmetic accuracy and keeps full-width composition penalties ≤0.108\le 0.108 nats on a 512-document external panel at 30M and ≤0.008\le 0.008 nats at 1.3B. A matched four-arm ablation (Section 5.1, Appendix X.2) demonstrates that neither detached forward co-activation alone nor foreign gradient updates alone suffices; both are required.
  • Shrinking & Physical Export: In the 51.3M 16x-FFN model, exporting the code member at 75% width reduces parameters to 32.3M and yields a 1.53–1.58x batch-16 forward speedup at a 0.042–0.049 nat loss cost.
  • Removing a Region: Deleting the arithmetic region reduces accuracy on its task from 188/192 and 187/192 to 1/192 and 0/192 while changing losses on retained domains by ≤0.003\le 0.003 nats, though 2M tokens of retraining on arithmetic restores 177/192 and 179/192 correct answers.
Critical strengths
  • Clean mechanistic isolation of unweighted subnetwork composition: The paper poses a well-defined question—what training conditions allow domain-assigned FFN blocks to be summed with unit weights at inference without a router or posterior weighting? Across 30M, 124M, and 1.3B scales, and especially in the matched four-arm study (Appendix X.2) separating forward co-activation (with detached weights) from auxiliary foreign-gradient updates, the authors convincingly show that regions must both co-activate in the forward pass and receive cross-domain gradients to compose cleanly.
  • Thorough, candid self-auditing and negative controls: The authors test and report the exact boundaries where their approach stops working. They document that pruning a dense model with just 1M tokens of repair beats their zero-shot narrower exports at every width, that 2M tokens of finetuning undoes region removal even when paired with RMU unlearning, that ramp exposure fails on arithmetic when preceded by Python context, and that standard 4x FFN transformers gain little parameter reduction or batch-1 speedup from FFN slicing.
Critical weaknesses
  • Narrow practical utility of the exported models relative to standard baselines: Across extraction, shrinking, combination, and removal, the practical case for region dropout is weak:
    • Single-domain extraction & shrinking: Extracted code specialists lose 0.06–0.13 nats against dense models trained on the same budget, and at 4x the code tokens, training on code alone beats extraction. Furthermore, if a tiny repair budget of 1M tokens (~1.2% of pretraining) is allowed after pruning a dense model, magnitude and activation-informed pruning outperform region dropout's ordered slices across all four widths (Appendix H.2). On standard 4x FFN architectures—and at batch size 1 or uncached generation on A100 even with a 16x FFN—physical slicing yields almost no wall-clock speedup (0.92–1.07x) because embeddings and attention are unpartitioned.
    • Cross-domain context interference: While ramp exposure achieves low composition penalties on homogeneous single-domain documents, Appendix Y reveals a severe side effect on mixed-domain inputs: placing a short Python snippet before each arithmetic equation causes the ramp arithmetic and combined code+arithmetic members to score 0/192 or 1/192 (continuing the Python text instead), whereas the unexposed (p=0p=0) arithmetic member still solves 136/192 and 134/192.
    • Fragile capability removal: Structural deletion of the arithmetic region is undone within 2M tokens of finetuning on the task (recovering >92% accuracy).
  • Poorly integrated 45-page appendix with legacy terminology and raw debugging logs: The 9-page main text relies heavily on Appendices A–Y for experimental details and tables. However, page 12 notes that the appendices retain old terminology from an earlier draft ("block", "product", "singleton", "parent", "psubp_{\text{sub}}", "factorized"), forcing the reader to constantly translate terms. Moreover, several appendices read like unedited internal commit logs—documenting a bug where exposure gates were applied twice (p2qip^2 q_i) in earlier runs (Appendix K), a leaked WikiText test article (The Hustler) in the training cache (Appendix D), and unrecovered data generation scripts (Appendix J).
Questions for authors
  • Clarification on the Python-prefixed arithmetic failure (Appendix Y): Why does prepending Python context cause the ramp-trained arithmetic and code+arithmetic members to drop from 192/192 to 0–1/192 while the p=0p=0 arithmetic member retains 136/192? Does training foreign regions on code data (p(t)>0p(t) > 0) cause the arithmetic region and shared attention layers to prioritize continuing Python syntax whenever code appears in the context window? If a combined multi-domain member fails when a prompt mixes two of its domains, when would a practitioner deploy a combined member instead of routing between single-domain specialists?
  • Clarification on the 1.3B MBPP evaluation (Appendix U.1): At 1.3B parameters (trained on 26B tokens including 6.5B StarCoder tokens), exported members solve at most 7/257 problems on sanitized MBPP. Why was MBPP evaluated at 84,000 of 99,183 updates rather than the final checkpoint, and how many problems does an unpartitioned dense 1.3B model trained on the exact same 26B-token mixture solve under your 1-shot greedy protocol?
  • Suggestion for manuscript revision: Standardize the terminology across the main text and all appendices, and condense the historical audit/bug logs into a clean, self-contained supplementary section.
3 comments

Thank you for a careful review. Your main concern is the practical case for extracted models against standard baselines. We measured each baseline you name with matched data and tokens, in all four domains.

(1) Pruning. Without fine-tuning, extracted models have 0.6 to 3.1 nats lower loss than baselines (standard transformers trained without region dropout) pruned to the same FFN width. With equal fine-tuning tokens for both, they keep the lower loss at every width up to 197k tokens, and at the narrowest width through 2M tokens. Appendix H.2 of the submission fine-tuned only the pruned models; the revision replaces it (Appendix F.6).

(2) Training on the domain alone. At full width, an extracted model has lower loss than a model of the same size trained only on that domain with the same domain tokens, in all four domains.

(3) Baselines. At 30M parameters an extracted model is within 0.04 to 0.11 nats (4 to 12% in perplexity) of a baseline of its size trained on the same data and tokens.

(4) Removal. After 0.4M relearning tokens, a network without its arithmetic region solves 56% of the test problems again; after RMU unlearning it solves 93%.

Details follow.

(a) Pruning. Our case is one training run and no training per model, so the first comparison is without fine-tuning: there an extracted model has 0.6 to 3.1 nats lower loss than a baseline pruned to the same FFN width. If fine-tuning is allowed, it is allowed for both: an extracted model is a standard transformer and can be fine-tuned, or pruned further, like any other. We therefore reran the comparison with equal fine-tuning tokens for both: five region-dropout training runs against three baselines, at 49k to 2M tokens, each model with the best of up to eight fine-tuning setups chosen on held-out data.

  • At 49k and 197k tokens the extracted model has lower loss in all 32 comparisons, by 0.03 to 1.7 nats.
  • At 1M and 2M tokens it has lower loss at 168 units in all eight comparisons, and at 336 units in all four at 1M (unresolved at 2M).
  • At 504 and 672 units pruning catches up: the pruned model has lower loss in 4 of 16 comparisons (three on WikiText, one on web text, by 0.03 to 0.06 nats); the others are unresolved or favour the extracted model.

The extracted model is the better starting point without fine-tuning, with little fine-tuning, and at narrow widths. Pruning a baseline catches up only at half width or more and with 1M or more fine-tuning tokens per model. That is a different setting: every model is then trained on its own, and for our 60 models this would take 60M tokens, three quarters of the training run.

(b) Models trained only on the domain (validation data). At full width (672 units, 26.1M parameters) the extracted model has 0.07 to 0.24 nats lower loss than a model of the same size trained on the same 20M domain tokens, in all four domains. With four times the domain tokens, the model trained on the domain alone is better on code, WikiText and web text, by 0.47 to 0.68 nats, as you noted for code. That is a different setting: one model per domain, each trained on as many tokens as our whole run. Trained four times as long, our network is within 0.035 nats of it or better (web text 0.075 lower).

(c) Baselines of the same size, same data and tokens (validation data). An extracted model has 0.04 to 0.11 nats higher loss at 30M parameters with 80M tokens, and 0.00 to 0.05 in the 800M-token runs at 30M and 124M. On our external test set the gap is 0.47 to 0.54 nats for two models, those extracted from regions trained on preprocessed corpora (WikiText-103, Penn Treebank) and scored on raw text. Converting the documents to each corpus's format reduced the gap to 0.10 and 0.03 nats. An extracted model serves text in the format of its region's training data; the appendix gives the full table.

(d) Speed. The code model extracted at 75% width from the network with an FFN of 16 times the hidden width has 32.3M of 51.3M parameters and 1.53 to 1.58 times the throughput of the full model at batch size 16. As you note, batch-1 latency on an A100 does not improve: extracted models ran at 0.99 to 1.07 times the speed of the full model; there the benefit is throughput and memory. On an RTX 2080 Ti the extracted model is 1.3 to 1.5 times faster at batch 1.

(1) Routing. On documents from one domain the multi-domain model is within 0.05 nats of the single-domain one (0.11 on the external test set), so a router does not pay off; on the Python-text test the model a router would choose fails as well.

(2) Removal. Against an attacker who holds the original data, removal slows relearning: 56% against 93% of the problems after 0.4M tokens.

(3) 1.3B baseline. Over the last ten checkpoints it solves 3 to 19 of 257 MBPP problems, our network 3 to 9 (19 and 3 at the final checkpoint). The 84,000 was our error.

Multi-domain model or routing (your first question). A router pays off only if the multi-domain model is worse than the single-domain model it would route to. On documents from one domain the difference is small: at 1.3B parameters, adding the other regions changes a domain's zero-shot task accuracy by −1.5 to +2.5 percentage points when trained with exposure, so one model serves all four domains without a router. Without exposure the same step lowers accuracy by 1.3 to 6.5 points, and by 22.5 on LAMBADA.

For the Python-text test (Appendix Y of the submission, now I.1), routing would not help either. The model a router would select, the arithmetic model on its own, solves 0 or 1 of 192 problems once Python text precedes them, and so do baselines trained on the same data; all of them solve the 192 without the text. The arithmetic training data never place text before a problem, and we think the failure comes from that, not from combining regions.

Why does the model trained without exposure keep 136 and 134 answers? Its arithmetic region only ever processed arithmetic. With exposure the region is also trained on the other domains' data and, like a standard transformer, reads a line in the light of what precedes it. With exposure, the outputs after Python text are repeated punctuation or digits, empty answers and continued Python text. We think this is the same property that lets regions work together (without exposure the full model solves only 5 and 53 of the 192 problems even without added text), but we have not tested this directly. The revision states that every evaluation uses documents from one source and that mixed input can fail.

Removal. Removing the arithmetic region reduces correct answers from 188 and 187 of 192 to 1 and 0. Loss on the other domains changes by at most 0.003 nats, and no training is needed. You are right that relearning recovers the skill, so we compared removal with RMU unlearning against an attacker who holds the original data: relearning on the arithmetic training data themselves, five training runs per condition, 242 new test problems.

  • After 0.4M relearning tokens, the network unlearned with RMU solves 93% of the problems again, the network without its arithmetic region 56% (95% interval across the five runs: 40 to 71), and removal followed by RMU 2%. With RMU hyperparameters tuned for our networks, the first and last figures are 92% and 10%.
  • To regain 25 to 95% of the unmodified network's accuracy, the network without its region needs 1.6 to 9.7 times as many tokens as after RMU.

Removal does not prevent relearning; it slows it down, without any training.

MBPP and a 1.3B baseline (your second question). The 84,000 is our error: the table reports the final checkpoints at 99,183 updates. Re-running all 257 problems on the two code models reproduced 3 and 7 with identical outputs; the other counts are read from the stored per-problem records of the final evaluation. We trained the 1.3B baseline on the same data and 26B tokens and scored it 1-shot and greedy. Because the number of solved problems varies strongly between checkpoints, we scored the last ten saved checkpoints of both models: the baseline solves 3 to 19, our network 3 to 9 (19 and 3 at the final checkpoint). With pass/fail scoring, small differences in loss can produce large differences in solved problems (Schaeffer et al., 2023).

Terms and appendix. The revision uses one set of terms: "member" becomes "subnetwork" inside the full model and "extracted model" once it runs alone. The appendices use the same terms, so the glossary is gone. A corrections section (Appendix K) lists the corrections to the submitted version and maps its appendix letters to this version; earlier training setups are grouped in Appendix J.

Reviewer_y5v7OpenReview ↗

Response to the authors

Thank you for a thorough and constructive rebuttal that directly resolves our main technical and presentation concerns. First, comparing extracted models and pruned dense baselines under \emph{equal} fine-tuning token budgets (Appendix F.6) changes the picture substantially: the extracted model leads by 0.60.6 to 3.13.1 nats without fine-tuning, wins all 32 comparisons across the four domains at 49k and 197k fine-tuning tokens (by 0.030.03 to 1.71.7 nats), and retains lower loss at narrow widths (168 and 336 units) through 1M--2M tokens without requiring per-subnetwork repair across a 60-model family. Second, the matched-compute comparisons clarify that the single-domain loss gap shrinks to 0.00–0.050.00\text{--}0.05 nats in the 800M-token runs at 30M and 124M parameters, and that at 4×4\times total training tokens the extracted model is within 0.0350.035 nats of (or up to 0.0750.075 nats better than) a standalone specialist trained on 4×4\times domain tokens. Third, showing that standard dense baselines trained on the same data also score 0--1/192 on the Python-prefixed arithmetic test confirms that this behavior stems from the unprefixed arithmetic training format rather than region combination, while the new 5-seed relearning comparison against RMU unlearning (56%56\% vs.\ 93%93\% accuracy recovered at 0.4M tokens, and 2%–10%2\%\text{--}10\% when combined with RMU; 1.6×–9.7×1.6\times\text{--}9.7\times more tokens required to regain 25%–95%25\%\text{--}95\% accuracy) establishes a clear advantage for structural removal over weight-only unlearning. Finally, we appreciate the 1.3B dense MBPP baseline comparison (3--19 vs.\ 3--9 solved across the final ten checkpoints, confirming that 84,000 was a typo for 99,183) and the complete unification of terminology across the main text and appendices. In recognition of these new matched controls and the manuscript cleanup, we have raised our presentation score from 2 to 3 and our overall rating from 2 to 3.

Reviewer_j2GbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Authors introduce region dropout, a technique that gives each domain its own region within every feed-forward layer and, during training, mixes in other domains' regions with a probability that grows over time, a mechanism they call exposure. Thanks to exposure, the merged models perform nearly as well as models trained on a single domain alone, letting individual subnetworks run independently for one domain, multiple domains, or a reduced budget, all without a router or extra training.

Critical strengths

None

Critical weaknesses
  • Weakness 1: The main experiments rely on a small model (30M parameters) with a limited token budget (20M per domain), leaving open whether the reported effects generalize across larger model scales.
  • Weakness 2: The presentation of the results could be substantially improved. Several tables currently placed in the appendix are important for understanding the main findings and would strengthen the paper considerably if moved into the main text. Results are not discussed in the main text, they are all left to appendix.
Questions for authors
  • Datasets are not cited, cite them.
4 comments

Thank you for the review. Networks of 30M to 1.3B parameters are now in one table, and at every size regions work together without fine-tuning only when trained with exposure. All datasets are now cited.

Scale. We trained networks of 30M, 51.3M, 124M and 1.3B parameters; the revision lists them in one table (Table 2), with data, tokens and the number of training runs behind each result. At every size, regions worked together without fine-tuning only when trained with exposure. Table 3 gives the largest loss increase from combining regions for every setting: 7.6 to 13 nats without exposure and at most 0.050 with it (validation data; 0.55 and 0.0076 at 1.3B on the test split). On code, the full model trained without exposure had 7.9 to 11.5 nats higher loss than the code model extracted from it at 30M and 124M (validation data) and 0.42 at 1.3B (test split); with exposure the increase was at most 0.033 at 30M, 0.009 at 124M and −0.002 at 1.3B. At 1.3B this shows in task accuracy: adding the other regions to a single-domain model changed its zero-shot accuracy by −1.5 to +2.5 points with exposure, and lowered it by 1.3 to 6.5 points without, and by 22.5 on LAMBADA (one training run each).

Presentation. The main results and their tables are now in the main text.

Datasets. All datasets are now cited, including the six zero-shot benchmarks and the Dolma sources, with licences in the appendix.

If a concern about correctness stands behind the soundness score, we would like to address it; could you point us to it?

Thank you for your response. I will go over the updated manuscript but I have a question. Can you give the computational details for large scale trainings? In the original submission, it is stated that a single 2080Ti was used but can you elaborate the computational details of new experiments?

Thank you for taking the time to go over our revised manuscript; we really appreciate it. The compute per setting you asked for:

  • 1.3B (Appendix E.5): each of the three models was trained on eight A100-80GB GPUs, with BF16 autocast and FP32 parameters and optimiser state, for 99,183 updates of 262,144 tokens (26B tokens). Training took about 100 hours for each of the two region-dropout networks and for the baseline. The two region-dropout networks and their hardware were already in the submission; the baseline is new.
  • 30M, 51.3M and 124M networks: one GPU per training run, in float32 without autocast. The RTX 2080 Ti named in the submission refers to the 30M networks (Appendix B.1).

The current PDF gives the hardware of the baseline only as "eight GPUs" and does not state the single GPU for the 51.3M and 124M networks. The next version of the PDF adds both: a paragraph "Compute" in Appendix B.1, and the hardware and duration of the baseline in Appendix E.5.1.

If there is anything else that you need, please let us know.

Thank you for the detailed responses, the additional experiments at larger scales, and the clarification of the computational details. While I appreciate these improvements, I still find the paper difficult to follow, and I believe the presentation would benefit from further revision. I will also take into account the other reviewers’ assessments and discuss further with them before finalizing my score.

Reviewer_QnFkOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

This paper presents Region Dropout, a dropout schema where rather than uniformly dropping units within a vector (as in regular dropout), the vector is first chunked into different regions (with each region having its own dropout probability), and subsequently within a region different units are dropped out.

The key idea is that each of the regions are mapped to a subdomain in the training data, so that region corresponding to a subdomain is never dropped out (whereas all other regions are dropped out according to a pre-determined probability). Region dropout produces several specialised models with only one training akin to a mixture-of-experts, without the need for a router. This is because during inference one may keep only the regions mapped to specific subdomains.

The evaluation focuses on establishing the effects of different region dropout variants (such as constant where "foreign" regions are dropped with constant probability; decreasing ramp where "foreign" are dropped more aggressively at the beginning of training and not at all at the end of training; and, ramp where "foreign" regions are not dropped at the beginning of training, but are dropped towards the end of training). The evaluation also covers the effects of various within-region dropout settings.

Critical strengths

The idea of chunking into different regions a vector, and dropping those regions at different rates depending on the data domain is intriguing. Moreover, the authors identify many hyper-parameters of region dropout.

The paper presents many experiments: from 30M to 1.7B parameters, ablations with constant or ramp region-dropout schedules, with different data mixes, and different with-in region dropout. Table 2 in the appendix showcases the breadth of the evaluation.

Critical weaknesses

W1. The paper is hard to read. This is due to a mix of issues, for instance: a) the paper sometimes inconsistent terminology (pool vs shared pool), b) use of undefined terms, eg. "rising drop rate" in line 75 (which is only defined later in the manuscript); and, c) very long sentences which make it hard to follow the argument, eg.

It does not determine which members of a single dropout family, over domain regions and widths, can be deployed without either, because networks of variable width have no domain regions, and expert mixtures combine their experts through weights, a router, an anchor or further training rather than as one unweighted layer.

W2. Along the same topic as above, I struggled to follow the logic for the evaluation experiments. Why were the current experiments chosen? What were you trying to prove? Why did you exhaustively investigate some algorithmic choices (region dropout schedule) and not others (eg. applying Region Dropout after the attention layers rather than the feed-forward network).

W3. There is no straight comparison (in the main text) between Region Dropout and other mixture-of-experts techniques. What is the performance of DeMix, Branch-Train-Merge, Branch-Train-Mix methods with respect to Region Dropout in an apples-to-apples comparison?

Based on these weakness, I struggle to see under what circumstances (outside of those contained in the paper) could Region Dropout be deployed, and consequently, I have to recommend rejection of this paper.

Post-rebuttal update: The authors significantly improved the readability of the manuscript during rebuttal (W1), as well as reorganised the experiment section to clearly highlight their purpose and compare against baselines (W2 and W3).

I only a remaining (smaller) concern which prevents me from increasing my score beyond "weak rejection": I worry the changes to the manuscript during rebuttal may not have been well reviewed (my own review was quicker than the original).

Questions for authors

Q1 [Following from W2]. Can you apply Region Dropout post-attention?

Q2. What is the effect of removing the shared pool? I hypothesise performance drops significantly for all tasks.

Q3. Does the shared pool have to be larger than the domain regions?

Q4. Why is code almost always included along with the domain regions?

Q5. What is top-level effect of coupling in within-region dropout?

Post-rebuttal update: The authors addressed all of these questions successfully.

4 comments

Thank you for the review. The introduction now leads from the problem to our question, and the method starts from one concrete extracted model (W1, W2). We ran DEMix, BTM and BTX on the same tokens: BTM with a long seed phase gives the better full model, region dropout the fixed models for subsets, widths and removal (W3). Q1 yes; Q2 a smaller drop than expected; Q3 no; Q5 nested masks move quality from wide to narrow models.

W1, W2 (readability, logic of the experiments). The introduction now states the problem, what expert methods, structured pruning and unlearning each provide, and what they leave open. The method section starts from one concrete extracted model: which units it keeps, which it drops, and that it runs as a standard transformer. Each results section is headed by its answer. The sentence you quote now reads: "Every extracted model is thus a fixed subnetwork of one jointly trained network, selected by domain and FFN width. It runs as a standard transformer, without a router or input-dependent weights on regions and without training after extraction."

W3 (comparison with DEMix, BTM, BTX). We ran all three at the 30M scale with the same 80M training tokens and three training runs per method; the comparison is a table in the main text. With one forward pass and 30.0M parameters, our full model has lower loss on all four external test sources than Branch-Train-Merge (BTM) with a seed phase of 50% of the tokens and than Branch-Train-MiX (BTX) with small experts (4×384 units, router counted in the 80M tokens). It is within 0.03 nats of BTX with full-size experts (4×1,536 units, 51.3M parameters). Three alternatives have lower loss:

  • BTM with a seed phase of 75%: 0.04 to 0.10 nats lower for the four-domain model, at the same size. Its models for domain subsets keep all 30.0M parameters, against 26.1M to 28.7M for ours; pruned to our size without fine-tuning, they had lower loss than ours in 0 to 6 of 28 cases and higher in 13 to 26, depending on the pruning criterion.
  • BTX built on BTM's branches: 0.08 to 0.09 nats lower, with 51.3M parameters, 92M training tokens and 1.64 times our inference time.
  • Methods that mix expert outputs or ensemble branches, such as DEMix with posterior weights: up to 0.13 nats lower, at 5.2 to 6.1 times our inference time; other variants up to 0.14 nats lower, at up to 8.5 times.

The methods solve different problems. These methods give a better full model; region dropout gives several fixed models from one run: domain subsets, narrower widths, and models without a domain. On the last point, leaving out BTM's code branch raised code loss by 0.07 nats, against 0.93 nats for removing our code region, because the branches share a seed phase that was trained on every domain.

When to use it. Use region dropout when one training run has to supply models for several domain subsets and sizes, or a model without one of the domains, and little fine-tuning per model is possible. A dedicated single-domain model had higher loss than the extracted model with the same domain tokens, and lower loss on three of four domains with four times as many. When only the full model is needed, BTM with a long seed phase gives lower loss. Pruning a baseline catches up only at half width or more and with 1M or more fine-tuning tokens per model. That is a different setting: every model is then trained on its own, and for our 60 models this would take 60M tokens, three quarters of the training run.

Q1. Yes. With regions on the attention output, single-domain losses differed from the FFN version by −0.105 to +0.055 nats, and the largest composition penalty was 0.031 to 0.035 nats. An extracted model then drops only 1.7% of the parameters, against 13.3% with regions in the FFN; this is why we place them in the FFN.

Q2. The drop is smaller than you expected. Networks trained without a shared pool, at the same total FFN width, had 0.001 to 0.082 nats higher single-domain loss, and a largest composition penalty of 0.039 to 0.042 (0.033 with the shared pool). Cutting the shared pool out of a network trained with one is a different operation and does hurt: code loss 2.97 to 3.12 nats, against 2.22 to 2.28 with its shared pool.

Q3. No. A 154-unit shared pool next to 345-unit regions gave single-domain losses from 0.006 below to 0.050 above those with the 384-unit shared pool.

Q4. Code has good data and an executable benchmark, differs most in syntax from the three prose sources, and is a model one would plausibly extract. The revision reports all four domains wherever we measured them.

Q5. It moves quality from wide to narrow models. With each unit's keep probability unchanged, drawing nested masks (the first k units of each region) instead of independent unit masks lowered the loss of the 168-unit code model by 0.24 nats and raised that of the 672-unit model by 0.06. The composition penalty did not change (−0.009 nats, with an interval that includes zero).

Thank you for your detailed response to all my questions. I do need to re-read the manuscript to evaluate the highlighted changes, please bear with me while I do so, I will get back to you as soon as possible.

Thank you for your detailed rebuttal and for the updated manuscript, which I just finished reading. In what follows, I continue the different threads of our discussion:

W1, W2 The updated manuscript reads much better, the authors have done a great job of updating the paper to clearly state the motivation, and to walk the reader through the experiments and why those are important. There still some sections that could be improved in terms of readability (eg. the use of "against" in the last paragraph of the introduction → I find it easier to read with "compared with", or the settings paragraph in section 4), but overall there's a night-and-day difference in terms of readability.

W3 The authors likewise added comparisons against the baselines in the main text and trained 3 new models at 1.3B parameters to show the viability of their method.

Q1. Excellent, I assume this has been placed in the appendix?

(As an aside, with such a large appendix, it might be convenient to add a table of contents and a high-level summary of each section to help readers navigate the appendix).

Q2. Interesting. I am not sure if it is preferable to remove the shared pool for deployment of Region Dropout (since it can mean effective fewer parameters per domain as well as more flexibility with prunning).

Q3, Q4, Q5. Thanks for your answers.

Overall, based on the significant readability and methodological improvements to the manuscript, I will be increasing my score.

Thank you for rereading the revised manuscript and for these helpful comments.

Q1. Yes, the attention-output experiment is in Appendix B.2, with a pointer from Section 3.1.

Q2. You are right for the case you describe. Following your comment, we evaluated the existing networks of Appendix B.2 at equal width, without additional training and with three training runs per condition.

When one domain is delivered, that is, the network with every region but one left out (a single-domain model in the paper), training without a shared pool was better. At 384 FFN units these models had 0.05 to 0.13 nats lower loss than those from networks trained with a shared pool and shrunk to the same width, in all 12 comparisons across training runs and domains; at 288 units the difference was 0.15 to 0.27 nats. They also lost less when halved: 0.08 to 0.10 nats, against 0.18 to 0.22.

When all four domains are delivered together, the shared pool was better. Both networks then have 1,536 FFN units, and the one with a shared pool had lower loss in all 12 comparisons, by 0.009 to 0.087 nats, and a smaller composition penalty.

The size of the shared pool is thus a parameter of the method, and no shared pool is one of its settings. Our data give a direction for choosing it: the fewer domains a delivered model contains, the smaller the shared pool should be, and the more the models for several domains matter, the larger. In the table of Appendix B.2, the largest composition penalty fell with every increase of the shared pool, from 0.039 to 0.042 nats without one to 0.023 to 0.027 with 1,152 units, while a model for one domain kept more of the network. We used a shared pool of a quarter of the units because the paper delivers the full model and the models for several domains from the same network as the models for one domain. We did not tune it.

Wording and navigation. The updated PDF uses "Compared with ..." in the last paragraph of the introduction, simplifies the Settings paragraph in Section 4, and opens the appendix with a table of contents and a short description of each section. Appendix B.2 now includes the equal-width comparison above.

We also take your concern about checking the revision seriously. The supplementary material includes revision/check_claims.py and the accompanying list of claims (revision/claims.csv), which make the numerical checks against the saved result files inspectable, including the new equal-width comparison in Appendix B.2. Appendix K documents the corrections to the submitted version.

If there is something else that you consider scoring the manuscript, please let us know.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Changes Summarised

Summary of the revision, for all reviewers and the area chair

We thank the three reviewers. This comment lists what changed, so that the revision can be checked quickly.

Manuscript. The main text was rewritten for readability: the introduction leads from the problem to our question, the method starts from one worked example, each results section is headed by its finding, and the main results and their tables are in the main text. The appendix uses the terms of the main text and opens with a table of contents and one line per section. Appendix K lists the corrections to the submitted version.

Main new experiments.

  • DEMix, BTM and BTX on the same data and tokens (Section 5.4, Appendix G).
  • Pruned baselines with equal fine-tuning tokens (Appendix F.6).
  • Relearning after removal, compared with RMU, five training runs per condition (Section 5.5, Appendix H.4).
  • A 1.3B baseline on the same data and schedule, with MBPP over the last ten checkpoints (Section 5.6, Appendix E.5).
  • Regions on the attention output and other sizes of the shared pool, with an equal-width comparison with and without a shared pool (Appendix B.2).

Second upload (17 November). Wording in the introduction and in the Settings paragraph, compute details (Appendices B.1 and E.5.1), the contents page of the appendix, and the equal-width comparison in Appendix B.2.

Checking. The supplementary material contains revision/check_claims.py and revision/claims.csv, which recompute the numbers printed in the paper from the saved result files.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.