acceptodds
Under review as a conference paper at ICLR 2027

Knowledge Offloading: Decomposing LLMs into Sparse Backbones and Memory Modules

Abstract

LLMs encode both general capabilities and domain-specific knowledge in a single set of parameters. We ask whether this capacity can be reorganized: keeping broadly useful computation in a shared backbone, while moving specialized knowledge into external memory modules. We propose knowledge offloading (KOFF), a framework for decomposing a pretrained LLM into a sparse shared backbone and domain-specific memories. Starting from a frozen base model, we either jointly or sequentially learn a structured pruning mask and lightweight recovery modules, implemented as LoRA adapters and learned key-value caches. Across Llama and Qwen models from 3B to 14B, we find that 12% to 20% of capacity can be moved out of the shared backbone while largely preserving perplexity and MMLU performance (up to 90% of the full model). We show that LoRA and learned KV memories are complementary, and analyses suggest that the offloaded knowledge is specialized: mismatched memories do not yield the same gains, and language-specific neurons are preferentially removed while language-neutral neurons largely remain in the backbone. These results suggest that the knowledge of a pretrained LLM can be reallocated between a shared core and swappable external memories.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, modular neural networks, memory-augmented language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_NjM6OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the capacity of a pretrained LLM can be reorganized so that broadly useful computation remains in a shared backbone while specialized knowledge is moved into external memory modules. The authors propose Knowledge Offloading (KOFF), which jointly or sequentially learns a structured pruning mask and domain-specific recovery modules implemented as LoRA adapters and learned key-value caches. Experiments on four LLMs ranging from 3B to 14B show that attaching matched memory modules substantially reduces the perplexity increase caused by pruning and often improves MMLU performance over pruning-only. Additional analyses suggest that the memory modules are domain-specific, with language-specific neurons preferentially removed while language-neutral neurons largely remain in the backbone.

Critical strengths
  • The paper presents a well-motivated framework for reorganizing pretrained LLM capacity. KOFF combines a shared sparse backbone with swappable domain-specific memory modules and explores both joint and sequential optimization. This provides a practical approach to separating shared computation from domain-specific adaptation.
  • The experimental evaluation provides encouraging results across multiple settings. KOFF consistently reduces the perplexity increase caused by pruning across Llama and Qwen models. The ablations support the benefits of LoRA and KV memories. The memory-swapping and neuron-level analyses provide further evidence of domain specialization.
Critical weaknesses
  • The central knowledge-offloading claim is insufficiently established. The improvements over pruning-only and the memory-swapping results provide evidence of performance recovery and domain-specific behavior of the memory modules, but do not clearly establish that specialized knowledge has been relocated from the backbone to external modules.

  • The preservation of general capabilities is not consistently demonstrated. Although KOFF reduces perplexity across different settings, it substantially degrades Global MMLU performance on the instruction-tuned model compared with pruning-only (Table 2). This raises concerns about whether the proposed decomposition effectively preserves general capabilities while offloading specialized knowledge.

Questions for authors
  • The paper reports that language-specific neurons are preferentially removed from the backbone. Does the pruning-only baseline exhibit a similar pattern, or is this specific to KOFF?
  • In Table 2, KOFF substantially reduces Global MMLU performance on the instruction-tuned model compared with pruning-only. Can the authors clarify whether this degradation results from the raw-text distillation objective or the proposed decomposition itself?
  • The experiments evaluate models with matched domain-specific modules (Section 4), while inference relies on a learned router (Section 3). Can the authors report the end-to-end performance of KOFF when memory modules are selected by the learned router rather than using ground-truth domain labels?
  • KOFF trains six domain-specific memory modules, but the paper does not clarify how a module is selected for each MMLU question. Are the reported MMLU results based on the learned router, a predefined domain mapping, or oracle module selection?
3 comments

We thank the reviewer for the thoughtful review and address the points below.

Response part 1/2

W1: "The central knowledge-offloading claim is insufficiently established."

We agree that our terminology should be more precise. By knowledge offloading we mean a functional decomposition: a smaller shared backbone complemented by domain-specific external modules that recover the capacity removed by pruning. Pruning followed by domain-specific recovery is therefore not an alternative explanation of our results; it is how offloading is implemented. We do not claim to identify and physically transfer individual facts or representations. We now state this explicitly (Section 3.1) and replaced "relocation" throughout.

What distinguishes this from generic recovery is that the decomposition is specialized on both sides:

  • The modules hold domain-specific capacity. Mismatched modules perform much worse than matched ones (Fig. 3, 4), and six domain modules outperform a single shared module trained on pooled data under the same pipeline (perplexity 8.12 vs. 9.44 on topics and 10.90 vs. 11.35 on languages; Global MMLU 0.537 vs. 0.519; Table 1).

  • The backbone retains domain-general neurons more selectively when the mask is trained jointly with memory modules. Compared with pruning-only (same sparsity target, no modules), the joint mask has a 1.22x higher normalized pruning rate for the most language-specific fifth of the active pool, but only 0.55x for the most language-general fifth. For topics, the corresponding ratios are 1.18x and 0.48x (see point 3). The difference is driven mainly by stronger protection of general neurons. Post-hoc recovery cannot produce this difference in the mask, since a prune-then-recover pipeline fixes the mask before any module exists.

W2 / Q2: "Can the authors clarify whether this degradation results from the raw-text distillation objective or the proposed decomposition itself?"

The degradation is not entirely surprising: we start with an instruction-tuned model but train the modules using token-level distillation on raw text rather than instruction data (including the model's own generations). This objective does not explicitly target the preservation of instruction-following capabilities. In practice, we envision either performing the decomposition before instruction post-training, or incorporating instruction data in decomposition learning.

  • With identical data, hyperparameters and pipeline, language modules on the base model do not reduce Global MMLU; they slightly improve it over pruning-only (0.537 vs. 0.532; Table 1).
  • On the same Instruct model, the decomposition with topic modules improves MMLU over pruning-only (0.636 vs. 0.603; Table 2). The decomposition therefore does not in itself erode the instruct model's capabilities.
  • The language modules do improve what they are trained on: held-out perplexity drops from 16.75 to 11.70.

More broadly, across our 10 model/partition settings (Table 9), the better KOFF variant improves over pruning-only on (Global) MMLU in 8, is on par in one (0.598 vs. 0.599), and is worse only in this instruct/language setting, while improving perplexity in all of them.

Q1: "Does the pruning-only baseline exhibit a similar pattern, or is this specific to KOFF?"

We ran the requested comparison on Llama-3.1-8B, using 6000 articles per domain for LAPE's entropy estimation and matched pruning-only masks at 20% sparsity. We compare the masks on the same active neuron pool, ranked by LAPE entropy from most domain-specific to most domain-general.

The new analysis shows that joint training produces a much more selective mask than pruning-only. In the language setting, pruning-only is relatively flat across the specificity range: it prunes domain-specific neurons at 1.1-1.2x its average rate within the active pool and the most domain-general ones at 0.7-0.8x. KOFF: joint has a steeper profile: 1.3-1.4x for domain-specific neurons and only 0.2-0.3x for the most domain-general ones.

Comparing the same neurons and normalizing each mask by its own average pruning rate within the pool, KOFF: joint has a 1.22x (95% CI: [1.18, 1.27]) higher rate than pruning-only in the most specific fifth and a 0.55x [0.49, 0.60] rate in the most general fifth. The corresponding ratios for topics are 1.18x and 0.48x. We revised Section 6 accordingly, including the pruning-only control.

Response part 2/2

Q3 / Q4: "Are the reported MMLU results based on the learned router, a predefined domain mapping, or oracle module selection?"

The MMLU results in the submission use a predefined mapping from each MMLU subject to one of the six topic domains (available in the released code). For Global MMLU, the question's language determines the module. We have now evaluated KOFF end-to-end with the learned router selecting the module for every input (Llama-3.1-8B base, 20% sparsity):

PPL (gold)PPL (router)MMLU (predefined)MMLU (router)
KOFF: joint, languages10.9010.850.5370.536
KOFF: joint, topics8.128.060.6020.600
KOFF: sequential, languages11.9511.930.5340.534
KOFF: sequential, topics9.139.120.5920.590

Using the router instead of gold labels thus has no meaningful effect on downstream performance. We added these results to Table 7, L701 and clarified module selection in Section 4 (L259-261).

Reviewer_NjM6OpenReview ↗

Response to the Authors' Rebuttal

Thank you for the detailed response.

My questions have been largely addressed. I had hoped that the paper could show that specialized knowledge is actually transferred from the backbone to the external modules. That would be a very valuable result. However, the revision now defines knowledge offloading as a functional decomposition and no longer claims such a transfer. With this definition, the contribution is closer to structured pruning with domain-specific recovery modules. I do not have any additional questions.

I will raise my score.

Reviewer_RdqMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

KOFF decomposes a pretrained LLM into a shared pruned backbone and domain-specific modules combining LoRA adapters with learned KV caches. The pruning mask and modules are trained jointly or sequentially through distillation. Experiments show recovery of perplexity and some MMLU performance, while module swapping and neuron analyses suggest domain specialization.

Critical strengths
  • Recovery is demonstrated across several models and domain partitions. On Llama-8B, joint KOFF raises topic MMLU from 40.0% after pruning to 60.2%.

  • The module-swapping and neuron analyses go beyond aggregate scores and provide useful evidence of specialization.

Critical weaknesses
  • Domain specialization does not establish knowledge relocation. The results remain compatible with ordinary domain-specific recovery through distillation.

  • A matched shared-module baseline is missing. This would help isolate the benefit of domain-specific memories over pooled recovery.

  • The neuron analysis lacks a pruning-only comparison (Section 6). It is unclear whether preferential removal of language-specific neurons is unique to KOFF.

  • Table 3 does not report parameter-matched controls. Combining LoRA and KV memory adds capacity, so its advantage alone does not establish complementarity.

  • End-to-end routing performance is unclear. Global MMLU uses matched modules, while Appendix A.1 reports classifier accuracy without downstream routing results.

  • Capability preservation is uneven. On the instruct model, language modules reduce Global MMLU from 53.1% after pruning to 44.5–45.9% (Table 2).

Questions for authors
  • Which reported evaluations use predicted domains rather than known labels? Are downstream results with the learned router available?

  • Does the pruning-only mask show similar removal rates for language-specific and language-neutral neurons? This comparison would help assess the specialization claim.

3 comments

We thank the reviewer for the detailed and constructive review. We ran the requested controls and respond point by point.

Response part 1/2

W1: "Domain specialization does not establish knowledge relocation."

We do not propose offloading as an alternative to pruning followed by domain-specific recovery. KOFF implements offloading either sequentially (by pruning followed by domain-specific recovery) or jointly (by learning what to remove from the shared backbone together with the modules that recover it). In both cases, capacity is removed from the shared backbone and recovered through modules that live outside it and are attached per domain. "Domain-specific recovery through distillation" therefore describes our mechanism rather than a competing explanation of our results. We replaced "relocation" with more precise language: a functional decomposition in which capacity removed from a shared backbone is recovered in domain-specific external modules. We now also state explicitly (Section 3.1, L155-157) that we do not claim to transfer individual facts or representations.

The question is then whether the decomposition has the properties that distinguish it from generic recovery.

  1. The recovered capacity is domain-specific, not pooled. Domain modules outperform a single shared module trained on pooled data under the same pipeline (W2), and mismatched modules are much worse than matched ones (Fig. 3, 4).

  2. What leaves the backbone is more domain-specific under joint training than under pruning-only, mainly because domain-general neurons are better protected (W3).

We also want to be precise about the scope of our contribution. As noted in Section 2, KOFF: sequential is a domain-wise generalization of prune-then-adapt. We do not present it as a new pruning method or as substantially stronger than existing ones: pruning-only serves as a control and LLM-Pruner as an external reference point. Our contributions are (i) formulating pruning as learning a memory allocation between a shared backbone and multiple swappable domain memories, including a joint variant in which the mask is shaped by what the memories can recover, and (ii) evidence that the resulting decomposition is specialized in the sense above. That even the simple sequential variant yields domain-specific modules is, in our view, part of this finding rather than evidence against it.

W2: "A matched shared-module baseline is missing."

We trained KOFF: joint with a single module shared across all domains of a partition, trained on the pooled data, with the same size as one domain module, i.e., matching the number of active parameters at inference:

SparsityPPL topicsPPL languagesMMLUGlobal MMLU
Unpruned0%7.939.700.6670.541
Pruning only20.0%10.8814.030.4000.532
KOFF: sequential, 6 modules20.0%9.1311.950.5920.534
KOFF: joint, 6 modules20.6%8.1210.900.6020.537
KOFF: joint, 1 shared module (same size)20.1%9.4411.350.5950.519

Domain-specific modules outperform the shared module on all four metrics. While this comparison matches the active module parameters at inference rather than total stored parameters, it demonstrates the benefit of domain-specific modules under the same inference-time parameter budget. We added this baseline to Table 1 and Section 6.

W3: "The neuron analysis lacks a pruning-only comparison (Section 6)."

We ran the requested comparison on Llama-3.1-8B, using 6000 articles per domain for LAPE's entropy estimation and matched pruning-only masks at 20% sparsity. We compare the masks on the same set of prunable neurons, ranked by LAPE entropy from most domain-specific to most domain-general.

The new analysis shows that joint training produces a much more selective mask than pruning-only. In the language setting, pruning-only is relatively flat across the specificity range: it prunes domain-specific neurons at 1.1-1.2x its average rate within the active pool and the most domain-general ones at 0.7-0.8x. KOFF: joint has a steeper profile: 1.3-1.4x for domain-specific neurons and only 0.2-0.3x for the most domain-general ones.

Comparing the same neurons and normalizing each mask by its own average pruning rate within the pool, KOFF: joint has a 1.22x (95% CI: [1.18, 1.27]) higher rate than pruning-only in the most specific fifth and a 0.55x [0.49, 0.60] rate in the most general fifth. The corresponding ratios for topics are 1.18x and 0.48x. In absolute terms, of the 15% most specific neurons, KOFF: joint prunes 13.3% vs. 10.8% for pruning-only, and of the 15% most general neurons 2.6% vs. 8.3% (topics: 38.0% vs. 31.2% and 4.4% vs. 10.8%).

We revised Section 6 accordingly, including the pruning-only control.

Response part 2/2

W4: "Table 3 does not report parameter-matched controls."

We agree that the combined module has more parameters than either component, though the difference is small between LoRA and LoRA+KV. Per domain on Llama-3.1-8B, the LoRA adapter (r=128) has 134.2M parameters and the KV memory (512 tokens x 16 layers) holds 16.8M. To control for capacity directly, we trained a single-component module matched to the full module's 151M parameters (LoRA with rank 144).

MethodParametersMMLUPPL
Pruning only-0.4010.9
Pruning + LoRA (r=128)134.2M0.568.9
Pruning + LoRA (r=144)~151M0.568.8
Pruning + KV cache (512 tokens)16.8M0.588.5
KOFF: joint (LoRA + KV)~151M0.608.1

The combined LoRA + KV module outperforms the parameter-matched LoRA-only baseline on both MMLU (0.60 vs. 0.56) and perplexity (8.1 vs. 8.8; Table 3). This demonstrates that the benefit of combining LoRA and KV memory is not simply due to the increased parameter count, but that the two mechanisms provide complementary recovery capabilities. We added this control to Table 3 (L408-409).

W5 / Q1: "End-to-end routing performance is unclear."

All evaluations in the submission use known domain labels. For MMLU, each subject is assigned to a topic domain by a fixed mapping (available in the released code). For Global MMLU, the question's language determines the module. We have now evaluated KOFF end-to-end with the learned router selecting the module for every input:

PPL (gold)PPL (router)MMLU (predefined)MMLU (router)
KOFF: joint, languages10.9010.850.5370.536
KOFF: joint, topics8.128.060.6020.600
KOFF: sequential, languages11.9511.930.5340.534
KOFF: sequential, topics9.139.120.5920.590

Using the router instead of gold labels thus has no meaningful effect on downstream performance. We added these results to Table 7, L701 and clarified module selection in Section 4 (L259-261).

W6: "Capability preservation is uneven."

The degradation is not entirely surprising: we start with an instruction-tuned model but train the modules using token-level distillation on raw text rather than instruction data (including the model's own generations). This objective does not explicitly target the preservation of instruction-following capabilities. In practice, we envision either performing the decomposition before instruction post-training, or incorporating instruction data in decomposition learning.

  • With identical data, hyperparameters and pipeline, language modules on the base model do not reduce Global MMLU; they slightly improve it over pruning-only (0.537 vs. 0.532; Table 1).
  • On the same instruct model, the decomposition with topic modules improves MMLU over pruning-only (0.636 vs. 0.603; Table 2), so the degradation is not inherent to the decomposition.
  • The language modules do improve what they are trained on: held-out perplexity drops from 16.75 to 11.70.

More broadly, across our 10 model/partition settings (Table 9), the better KOFF variant improves over pruning-only on (Global) MMLU in 8, is on par in one (0.598 vs. 0.599), and is worse only in this instruct/language setting, while improving perplexity in all of them.

As the rebuttal period ends, we would appreciate your feedback on the revisions as several of the new experiments were motivated directly by your comments.

In particular, we have (1) compared the neuron specialization of KOFF:joint against pruning-only, (2) introduced size-matched LoRA-only controls, (3) evaluated downstream performance with learned rather than gold routing, (4) added the shared-module baseline to distinguish offloading from ordinary domain-specific recovery, We have also clarified the definition of offloading and the scope of our contribution, and acknowledged the limitations concerning instruction-tuned models.

We hope these additions address your main concerns.

Reviewer_CrqxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Knowledge Offloading (KOFF), which decomposes a pretrained LLM into a shared sparse backbone and swappable domain-specific memory modules. A structured pruning mask decides which neurons stay in the backbone. Per-domain modules, each a LoRA adapter plus a learned KV cache, are trained by distillation to recover what was removed, either after pruning or jointly with the mask. Across Llama and Qwen models on topic and language domains, attaching the matched module recovers most of the loss from pruning, and LoRA and KV memories turn out to be complementary. Analyses with swapped modules and language-specific neurons suggest that the offloaded knowledge is genuinely domain-specific.

Critical strengths
  1. Reframing pruning as a way to decide what stays shared and what moves to external memory is a fresh and interesting angle. It connects pruning, adapters, and memory-augmented models in a clean way.

  2. The specialization analyses go beyond simply recovering performance. Mismatched modules hurt, and language-specific neurons are preferentially removed, which gives real evidence that the split is meaningful.

Critical weaknesses
  1. No end-to-end evaluation with the router. All results use the correct-domain module. Performance with the learned router, and when routing is wrong, is not reported.

  2. Limited practical benefit. The backbone plus one module is still close to the original model in size, and keeping all modules can exceed it. The paper also notes that generation quality is not well preserved.

Questions for authors
  1. How does KOFF perform end to end when the router selects the module, including on inputs where it routes incorrectly? If routing errors are common or costly, the swappable-memory design would be much less practical.

  2. What concrete advantage does KOFF offer over keeping the dense model and adding a domain LoRA or KV cache to it, given that the compression is modest? A clear answer on memory, update cost, or editability would strongly affect how I view the contribution.

3 comments

We thank the reviewer for the thoughtful feedback and questions.

Response part 1 / 2

W1 / Q1: "No end-to-end evaluation with the router."

We re-ran the main-setting evaluations (Llama-3.1-8B base, 20% sparsity) in the end-to-end evaluation set-up, with the RoBERTa router (fine-tuning details in Appendix A.1) selecting the module for every input: each held-out perplexity text and each (Global) MMLU question. Results are essentially unchanged relative to gold domain labels. Differences are at most 0.06 perplexity and 0.002 accuracy, and router-selected perplexity is lower or equal in all four settings:

PPL (gold)PPL (router)MMLU (predefined)MMLU (router)
KOFF: joint, languages10.9010.850.5370.536
KOFF: joint, topics8.128.060.6020.600
KOFF: sequential, languages11.9511.930.5340.534
KOFF: sequential, topics9.139.120.5920.590

Language rows report Global MMLU. Using the router instead of gold labels thus has no meaningful effect on downstream performance. We added these results to Table 7, L701 and clarified module selection in Section 4 (L259-261).

W2: "Limited practical benefit. The backbone plus one module is still close to the original model in size, and keeping all modules can exceed it."

We respectfully disagree that the parameter comparison undermines the practical benefit. In our main setting (8B, 20% sparsity), the shared backbone together with all 6 domain modules is smaller than the original dense model: 7.56B vs. 8.03B for Llama-3.1-8B and 13.07B vs. 14.77B for Qwen-2.5-14B (Table 6). Thus, we obtain 6 independently manageable domain-specific modules while reducing the total parameter count. Serving an individual input requires only one module, further reducing the active model size (6.67B vs. 8.03B for Llama-3.1-8B; Table 6). We agree that these savings depend on the sparsity level. At 12% sparsity, the total can approach or slightly exceed the original model size, while the backbone plus one module is always smaller; we now state this explicitly (L293-295). However, the primary goal of KOFF is not maximizing compression, but reorganizing model capacity into a smaller shared backbone and separately manageable specialized modules. Achieving this with fewer total parameters, as in our main setting, is an additional practical benefit. Regarding generation quality, we agree that this is a limitation of the current implementation. The degradation observed in preliminary generation experiments also affects other pruning baselines, including pruning-only and LLM-Pruner. Our training objective uses token-level distillation on raw domain text, which may not sufficiently preserve richer generative behaviour. However, this does not negate the improvements observed under the evaluated objectives: KOFF improves perplexity over pruning-only in every reported setting (Table 9), and the better KOFF variant improves (Global) MMLU in 8 of 10 settings, with one additional setting effectively tied. We therefore see the generation results as an important limitation of the current training objective and implementation, rather than evidence against the proposed decomposition. The framework we introduce itself is not tied to token-level distillation on raw text, and extending it to objectives targeting generation and instruction-following is an interesting direction for future work. We clarified this distinction in Section 7 (L481-485).

Response part 2/2

Q2: "What concrete advantage does KOFF offer over keeping the dense model and adding a domain LoRA or KV cache to it, given that the compression is modest?"

Adding a domain LoRA or KV cache to a dense model only adds memory. All domain knowledge remains in the dense weights, so the module is an optional refinement and detaching it removes nothing. KOFF instead decides what the shared model keeps. Domain-specific capacity is removed from the backbone and is available only through its module, while shared capacity stays in the backbone. This makes the allocation of knowledge explicit and manageable:

  • Memory footprint. One smaller backbone serves all domains, and per-domain capacity is in small modules (151.0M parameters each; Table 6) that can be loaded, swapped, or kept off-GPU on demand. At 20% sparsity, the backbone needs 12.39 vs. 14.96 GiB of bf16 weights (Table 8). The freed 2.57 GiB exceeds what all six modules need (906.0M parameters, Table 6; 1.69 GiB in bf16), so the backbone with all modules resident still uses less memory than the dense model alone. With dense + modules, every domain still pays for the full dense backbone.

  • Separation and selective deployment. Because the backbone has lost the offloaded capacity, domain-specific performance becomes conditional on the matching module. Without a module, language perplexity is 14.03 (pruning-only) vs. 10.90 with matched modules (Table 1). With a wrong module it degrades much further: French text has perplexity 9.21 with its own module and 19.28-216.47 with the others (Fig. 3). By design, knowledge shared across domains stays in the backbone; this is why Global MMLU is largely retained without modules. What becomes module-conditional is the domain-specific part. With a dense model, withholding a module does not withhold the corresponding knowledge. This is what makes a KOFF module a meaningful unit for selective deployment, updating, or removal.

As discussed under W2, KOFF is not intended to maximize compression, and parameter savings alone do not imply proportional inference speedups. We therefore measured latency and throughput on Llama-3.1-8B (single 40GB A100, LoRA merged, mean of 7 runs; Table 8):

DenseKOFF 20%KOFF 12%
Backbone parameters8.03B6.65B7.18B
Backbone + six modules8.94B7.56B8.08B
Backbone weights, bf16 (GiB)14.9612.3913.36
Prefill, batch 8x512 (tok/s)13,26515,52014,590
Prefill latency, 512 tokens (ms)49.342.744.7
Decode, batch 16 (tok/s)421438434
Decode, batch 16, with KV memory (tok/s)344365358
Module training step (s)1.3811.1151.215

Prefill speedups are broadly consistent with the parameter reduction; decoding gains are small in our unoptimized HF implementation. We added these measurements to Appendix A.2 (Table 8, L728-731) and now state in Section 4 that KOFF is not intended as a compression method (L293-295).

Thanks for the rebuttal. The end-to-end routing and efficiency experiments address my main concerns, so I’m raising my score.

Comments on the paper

1 comment

We thank all reviewers for their feedback. We also thank reviewer NjM6 for confirming that their questions are largely addressed and for raising their score. Below we summarize the concerns and our responses; changes are marked in the revised PDF.

1. End-to-end routing (all reviewers). The submitted results used gold domain labels to select modules, as Section 4 now states. With the learned router choosing the module for every input, results in the main setting (Llama-3.1-8B, 20% sparsity) change by at most 0.06 perplexity and 0.002 (Global) MMLU accuracy for both KOFF variants and both domain partitions (Table 7). For example, KOFF:joint on topics goes from 8.12 to 8.06 PPL and from 0.602 to 0.600 MMLU.

2. Offloading vs. ordinary domain-specific recovery (RdqM, NjM6). We replaced "relocation" with a precise definition: offloading is a functional decomposition in which capacity removed from a shared backbone is recovered by swappable domain-specific modules. We do not claim to transfer individual facts or to propose a new pruning method; our contribution is casting pruning as learning how capacity is split between backbone and modules, and showing that this split is specialized on both sides. Two new experiments support this:

  • Shared-module baseline (RdqM): one module of the same inference-time size, trained on pooled data, is worse than the domain modules on all four metrics (Table 1).
  • Pruning-only neuron control (RdqM, NjM6): We demonstrated KOFF:joint prunes substantially more language-specific neurons than the pruning-only baseline (and hence KOFF:sequential, which uses the same pruning mask). Specifically, the joint mask prunes the most language-specific fifth of neurons 1.22x as often and the most language-general fifth only 0.55x as often (normalized; 95% CIs exclude 1; topics similar).

3. Benefit of KV memory beyond added capacity (RdqM). We added the size-matched comparison of LoRA-only and LoRA+KV, which shows that the gain from KV memories to LoRA is not explained by additional capacity. Namely, a size-matched LoRA-only module (rank 144) is no better than rank 128, while LoRA + KV of the same size improves both MMLU and perplexity. KV memory alone, with 8x fewer parameters, already outperforms LoRA .

4. Instruct-model Global MMLU and generation quality (all reviewers). The evidence suggests that the raw-text distillation objective, rather than the decomposition itself, is the source of the drop in generation quality for instruction-tuned models. With the same pipeline, language modules do not hurt Global MMLU on the base model, and topic modules improve MMLU on the same instruct model, whose language modules still cut perplexity substantially. Across all 10 model/partition settings, the better KOFF variant beats pruning-only on (Global) MMLU in 8, ties in 1 and is worse only here; both variants lower perplexity in all 10. In preliminary experiments, generation was less well preserved by all pruned variants, pruning-only and LLM-Pruner included. We now discuss both as limitations of the training objective (Sections 5.3, 7); natural remedies are decomposing before instruction tuning or adding instruction data.

5. Practical benefit over a dense model plus domain modules (Crqx). We added efficiency measurements of KOFF on Llama-3.1-8B (App. A.2):

  • Memory: at 20% sparsity, the backbone plus all six modules is smaller than the dense model alone.
  • Update cost and speed: module training steps take 19% less time and prefill throughput is 17% higher; decoding gains are small (≤6%) in our unoptimized implementation.
  • Separation: KOFF is not meant as a compression method (now stated in Section 4). The key benefit is reorganizing model capacity into a smaller shared backbone and separately manageable specialized modules. Unlike attaching modules to a dense model, KOFF moves domain-specific capacity out of the backbone, allowing individual modules to be deployed or removed for a specific user independently while preserving shared knowledge in the backbone (Fig. 3).

We hope these additions address the reviewers' concerns and would be grateful if they were taken into account in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.