acceptodds
Under review as a conference paper at ICLR 2027

COMPRESS FOR RECOVERABILITY: STREAMED COUPLED CORRECTIONS FOR MOE EXPERTS

Abstract

Mixture-of-Experts (MoE) models activate few experts per token, but their expert banks consume substantial GPU memory. We study recoverability as a third axis for compression, alongside resident memory and uncorrected accuracy: how much lost computation can be restored under a limited correction budget? A compressed model with poor standalone accuracy can still provide an effective base if its errors admit sparse, input-dependent correction. This perspective makes expert merging a viable compression strategy when combined with selective recovery. We introduce Delta-MoE, which keeps compressed experts on the GPU and streams selected residual channels from host memory. Each correction couples the gate, up, and down projections of a SwiGLU channel and recomputes its nonlinear contribution. Our analysis characterizes recovery through residual error spectra and explains why improving an uncorrected merge need not improve its corrected counterpart. On Qwen3-30B-A3B, the fused implementation achieves 0.810 GSM8K accuracy at 16× resident expert compression, with MATH-500 and IFEval point estimates similar to an 8× quantized base, and Mixtral-8x7B also exhibits recovery from merging.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Mixture-of-Experts, model compression, expert merging, quantization, error recoverability, residual correction, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dF1oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies recoverability as an additional consideration in MoE compression: a compressed model with relatively poor standalone accuracy may still be useful if a limited correction budget can recover its lost computation. The proposed method keeps compressed experts on the GPU and streams selected residuals from CPU memory. Each correction jointly restores the gate, up, and down weights of a SwiGLU channel and recomputes its nonlinear contribution. The paper also introduces activation-guided selection, layer-global budget allocation, and recovery-spectrum diagnostics. Evaluation covers Qwen3-30B-A3B and Mixtral-8x7B through generated-answer benchmarks, likelihood-scored diagnostics, and system-cost analyses.

Critical strengths
  1. Well-motivated recovery objective and coherent correction mechanism. Considering the error remaining after a limited correction budget is a useful perspective on compression. Jointly restoring a complete SwiGLU channel has a clear technical rationale: it accounts for changes in both nonlinear activations and output weights. The substitution identity makes the behavior of the correction unit understandable.

  2. Promising generated-answer recovery under aggressive resident-memory compression. On Qwen3, the fused implementation achieves 0.810 GSM8K accuracy at 16× resident expert-weight compression, compared with 0.868 for the uncompressed model. The MATH-500 and IFEval results provide complementary evidence that substantial generation capability can be retained. These results support the practical potential of selectively recovering expert-specific computation.

Critical weaknesses
  1. The central correction-granularity comparison lacks validation under prefix-causal evaluation. As acknowledged in Section 3, pooling activations across a teacher-forced continuation allows later supplied tokens to influence the correction set used at earlier positions. Consequently, the likelihood-scored comparison between coupled and projection-wise correction in Table 4 does not establish that the reported advantage persists during autoregressive generation. The generated-answer benchmarks remain valuable, but they do not directly isolate this central design choice. This leaves an important gap between the proposed mechanism and its empirical validation.

  2. Key matched generation controls are missing, particularly in Table 2. Table 2 includes bare pruning but omits the patched pruning control for both MATH-500 and IFEval. These results therefore demonstrate recovery relative to uncorrected pruning, without establishing an advantage over a matched alternative that also receives residual corrections. The gap also appears in Table 1: row 9a provides the strongest reported pruning-based likelihood diagnostic after adding global allocation, but its GSM8K result is unavailable. Together, these omissions leave the central generation comparison incomplete.

  3. Accuracy preservation under deep compression varies substantially across models. At 16× compression, the corrected Mixtral-8x7B model achieves only 0.222 GSM8K accuracy, compared with 0.606 for the uncompressed model (Table 3). This is a much larger relative degradation than that reported on Qwen3. Correction still improves the compressed Mixtral model, and the paper acknowledges this model dependence. Nevertheless, the second-model results support the existence of recovery more strongly than they support broadly reliable accuracy preservation. It remains unclear how much the favorable Qwen3 outcome depends on its architecture, expert structure, or compression configuration.

Questions for authors
  1. Validation of the correction mechanism: Does the advantage of coupled over projection-wise correction in Table 4 persist when selection uses only the available prefix, or in a matched generated-answer evaluation? A check on one representative merged configuration would materially strengthen the evidence for this design choice. If such evidence is already available, please identify it.

  2. Completeness of the matched comparisons: What is the GSM8K performance of the pruning-plus-patches-plus-global-allocation configuration in Table 1, row 9a? Can the authors also provide the patched pruning control missing from Table 2, or explain whether existing results support the same comparison on MATH-500 and IFEval?

2 comments

Q1: Yes. We reran Table 4's coupled vs. projection-wise comparison under causal selection at all three compression levels (Avg8, 32,946 documents per arm). "Batch" pools the prompts of a forward batch and, at each decode step, the tokens of that step (the paper's record size). "Per-example" selects from one example's own prompt and each answer token's own activation, so it is also batch-independent.

Resident baseSelectionProjection-wiseCoupledGain (pp)
2:1 bf16 merge (2.0×)Batch0.60760.6455+3.79
2:1 bf16 merge (2.0×)Per-example0.64220.6541+1.19
2:1 int4 merge (7.8×)Batch0.61010.6354+2.53
2:1 int4 merge (7.8×)Per-example0.64010.6548+1.47
2:1 int3 merge (10.3×)Batch0.59490.6426+4.78
2:1 int3 merge (10.3×)Per-example0.63300.6579+2.49

The coupled advantage persists in every row (all 95% paired bootstrap intervals exclude zero). It narrows as each record covers fewer tokens, and is absent at 2.0× when every token selects alone (−0.9 pp), a setting that streams about ten times as many records. We did not run a generated-answer comparison of the two correction types.

Q2: We ran all three missing cells. Row 9a scores 0.224 on GSM8K, and patched pruning scores 0.034 on MATH-500 and 0.235 on IFEval.

BenchmarkBare pruning, C4 list (paper)Patched pruning, C4 listBare pruning, chat-format listStacked Delta-MoE, 16×
GSM8K0.1560.224 (row 9a)0.8060.832
MATH-5000.0240.0340.4560.484
IFEval0.2240.2350.5100.599

Corrections add about 7 pp on GSM8K and 1 pp on MATH-500 and IFEval, so they do not repair the C4-pruned base. Calibration matters far more: with the keep list recomputed on GSM8K training prompts in the evaluation chat format, bare pruning comes within noise of the stacked row on GSM8K and MATH-500 and stays 8.9 pp behind on IFEval (details in our response to Reviewer TXTu). We did not run corrections on the calibrated pruned base. We will add these rows to Tables 1 and 2 and revise Section 4.1.

W3: We agree that accuracy preservation at 16× is model-dependent, as the paper states. Corrections with allocation raise Mixtral's 16× GSM8K from 0.000 to 0.222, still far below the uncompressed 0.606. Our Mixtral claim is that recovery from merging transfers (at 2×, patches raise GSM8K from 0.222 to 0.454), not that 16× preserves accuracy on every model. A likely factor is granularity (8 experts per layer against 128 in Qwen3), which we have not isolated. We will state this scope explicitly.

Thank you for the additional experiments. The causal rerun addresses W1, and the new pruning results fill the gaps in W2. The narrow gains over recalibrated pruning on math tasks and the accuracy drop on Mixtral leave my concern about a consistent practical advantage at 16× unresolved.

Reviewer_TXTuOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper provide a interesting angle about recoverability with extensive experiments and transparent results. The comparison between coupled channel correction and projection-wise correction are very valuable. Another important observation is multiple-choice tasks can hide the significant performance decline. There are also three limitations of this paper. The evaluation issue remains unresolved. The size of generation experiments is small. Quantization with expert caching can get higher accuracy and lower memory usage during decoding under the same GPU memory budget, which makes conclusions less convincing.

Critical strengths
  • Evaluates compression model by recoverability under a correctin budget. A near-chance merged base can increase to 0.81 in GSM8K
  • Coupled channel corrections performs much better than projection wise correction under merged models but it shows no significant advantage when correcting errors under quantization.
  • Generation evaluation suggest GSM8K collapse from 0.814 to 0.156 but REAP*QTIP's avg8 falling only 0.0657, which noted that discriminative and generative benchmarks can disagree.
Critical weaknesses
  • Channel selection take all tokens in a record, making likelihood evaluation non-causal and batch-dependent. The main ablations (Figure 2, Table 4, Tables 6 and 7) rely on this protocol without a prefix-causal rerun.
  • At the same 3.63 GB, a 50% QTIP-2b expert cache can get a better performance model (GSM8K 0.862) with less storage and decode traffic. Delta-MoE has advantages only in prefill and highly interleaved large-batch decode.
  • Generality is limited. On Mixtral, merging and patches is dominated by zero-traffic GPTQ-int4 and QTIP-2b, and 16× reaches only 0.222 GSM8K vs 0.606 for bf16.
Questions for authors
  1. Can you report prefix-causal Avg8/perplexity?
  2. Under what conditions does Delta-MoE beat a same-memory expert cache in throughput or latency?
  3. Do the conclusions change with neuron permutation alignment before merging, or with in-domain calibration for REAP?
2 comments

Q1: Yes for Avg8. We reran the coupled vs. projection-wise comparison of Table 4 under causal selection at all three compression levels (Avg8, 32,946 documents per arm). "Batch" pools the prompts of a forward batch and, at each decode step, the tokens of that step (the paper's record size). "Per-example" selects from one example's own prompt and each answer token's own activation, so it is also batch-independent.

Resident baseSelectionProjection-wiseCoupledGain (pp)
2:1 bf16 merge (2.0×)Batch0.60760.6455+3.79
2:1 bf16 merge (2.0×)Per-example0.64220.6541+1.19
2:1 int4 merge (7.8×)Batch0.61010.6354+2.53
2:1 int4 merge (7.8×)Per-example0.64010.6548+1.47
2:1 int3 merge (10.3×)Batch0.59490.6426+4.78
2:1 int3 merge (10.3×)Per-example0.63300.6579+2.49

The coupled advantage persists in every row (all 95% paired bootstrap intervals exclude zero). It narrows as each record covers fewer tokens, and is absent at 2.0× when every token selects alone (−0.9 pp), a setting that streams about ten times as many records. The headline generation results are generated answers, which cannot see future tokens.

Q2: We have not measured throughput or latency against an expert cache. Table 13 compares streamed bytes per token by routing-trace replay. At 3.63 GB, Delta-MoE moves fewer bytes in prefill (1.02 MB/token at p=128 and 2.05 at the 16× rows' p=256, vs 3.07 for a 2-bit expert cache at 2k tokens). At p=128 it also wins batch-64 decode, and single-stream decode once memory drops below about 1.2 GB. The cache wins small-batch decode at 3.63 GB (18.6 vs 151 MB/token), so Delta-MoE's regime is prefill-heavy, large-batch, or memory-constrained serving.

Q3: Permutation alignment: no. An optimal channel permutation between merged experts removes only 4.7% of the pair-difference energy, and a stronger merge (RegMean, +9.5 pp bare) does not improve corrected accuracy (0.6440 vs 0.6491, Section 4.4).

In-domain calibration for REAP: yes, it changes the pruning comparison. We recomputed the keep list on GSM8K training prompts in the evaluation chat format and evaluated the bare pruned 16× model (3.63 GB, zero traffic).

BenchmarkPruning, C4 list (paper)Pruning, chat-format GSM8K listStacked Delta-MoE, 16×
GSM8K0.1560.8060.832
MATH-5000.0240.4560.484
IFEval0.2240.5100.599

With this list, pruning is within noise of the stacked row on GSM8K and MATH-500 (0.418 on GSM8K with the same problems as plain text). It remains 8.9 pp behind on IFEval, and the pruned model's own Avg8 falls by 4.8 pp relative to the C4 list (0.634 to 0.586). So the pruning collapse in Tables 1 and 2 reflects calibration on plain C4 text. We will add the calibrated baseline and revise Section 4.1: Delta-MoE matches calibrated pruning on math generation and retains more instruction-following accuracy with one task-agnostic calibration, at the cost of streamed traffic.

Thank you for the dditional experiments. the new causal evaluation substantially addresses my methodological concerns. However, the lack of measured efficiency comparisons and the substantially narrowed advantage over calibrated pruning leave my main concerns unresolved, so I maintain my rating of 2 but increase soundness.

Reviewer_jZGYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Delta-MoE, a model that combines compressed GPU-resident experts with input-dependent SwiGLU residual streams fetched from CPU memory. The compression strategy is centered on recoverability. The empirical results are promising. When applied to Qwen3 utilizing fused kernels and a 16x compression ratio for the resident expert weights, the model achieves 81.0% accuracy on GSM8K, compared to 86.8% for the uncompressed version.

Critical strengths
  1. The paper is well-motivated. By jointly correcting the gating, up-projection, and down-projection layers, the method can precisely recover the contributions of selected channels prior to residual quantization, providing a solid mathematical foundation for the approach.
  2. The results suggest that poor accuracy in isolation (i.e., before correction) does not necessarily imply poor performance after correction. The finding particularly relevant in the context of expert merging.
  3. The authors compared different compression methods, which accounts for memory and data-transfer constraints, and reports both failure cases and the limitations of the evaluation.
Critical weaknesses
  1. During likelihood scoring, channel selection uses the full sequence, including tokens after the position being scored. This makes it unclear whether the gains observed in the ablation studies hold during normal generation. The authors should repeat the key ablations using only preceding tokens.
  2. For MATH and IFEval, the paper, the paper misses uncompressed model as a reference. Furthermore, these is a significant performance drop for the Mixtral model at the 16x compression ratio, which suggest limited generalizability.
  3. The paper seems written in a hurry, including different title between the ICLR title and paper title, formatting errors, strange spacings.
Questions for authors

Please see weakness above.

2 comments

W1: We reran Table 4's coupled vs. projection-wise comparison under causal selection at all three compression levels (Avg8, 32,946 documents per arm). "Batch" pools the prompts of a forward batch and, at each decode step, the tokens of that step (the paper's record size). "Per-example" selects from one example's own prompt and each answer token's own activation, so it is also batch-independent.

Resident baseSelectionProjection-wiseCoupledGain (pp)
2:1 bf16 merge (2.0×)Batch0.60760.6455+3.79
2:1 bf16 merge (2.0×)Per-example0.64220.6541+1.19
2:1 int4 merge (7.8×)Batch0.61010.6354+2.53
2:1 int4 merge (7.8×)Per-example0.64010.6548+1.47
2:1 int3 merge (10.3×)Batch0.59490.6426+4.78
2:1 int3 merge (10.3×)Per-example0.63300.6579+2.49

The coupled advantage persists in every row (all 95% paired bootstrap intervals exclude zero). It narrows as each record covers fewer tokens, and is absent at 2.0× when every token selects alone (−0.9 pp), a setting that streams about ten times as many records. We did not run a generated-answer comparison of the two correction types.

W2: Under the Table 2 protocol, the uncompressed model scores 0.520 on MATH-500 and 0.599 on IFEval.

MethodExpert HBM (GB)MATH-500IFEval
Uncompressed bf1658.10.5200.599
QTIP-2b, 8×7.260.4860.599
QTIP-2b + patches, 8×7.260.5160.595
Stacked Delta-MoE, 16×3.630.4840.599

On Mixtral, we agree that accuracy preservation at 16× is model-dependent, as the paper states. Corrections with allocation raise Mixtral’s 16× GSM8K from 0.000 to 0.222, still far below the uncompressed 0.606. A likely factor is granularity: Mixtral has 8 experts per layer against 128 in Qwen3. Our claim for Mixtral is that recovery from merging transfers (at 2×, patches raise GSM8K from 0.222 to 0.454), not that 16× preserves accuracy on every model. We will state this scope explicitly.

W3: We apologise for these issues. We will unify the title across the PDF and OpenReview and fix the formatting and spacing.

I appreciate the authors' efforts in reporting the new experimental results. However, I still have reservations about the robustness and generality. I am inclined to keep my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.