- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper studies recoverability as an additional consideration in MoE compression: a compressed model with relatively poor standalone accuracy may still be useful if a limited correction budget can recover its lost computation. The proposed method keeps compressed experts on the GPU and streams selected residuals from CPU memory. Each correction jointly restores the gate, up, and down weights of a SwiGLU channel and recomputes its nonlinear contribution. The paper also introduces activation-guided selection, layer-global budget allocation, and recovery-spectrum diagnostics. Evaluation covers Qwen3-30B-A3B and Mixtral-8x7B through generated-answer benchmarks, likelihood-scored diagnostics, and system-cost analyses.
Critical strengths
-
Well-motivated recovery objective and coherent correction mechanism. Considering the error remaining after a limited correction budget is a useful perspective on compression. Jointly restoring a complete SwiGLU channel has a clear technical rationale: it accounts for changes in both nonlinear activations and output weights. The substitution identity makes the behavior of the correction unit understandable.
-
Promising generated-answer recovery under aggressive resident-memory compression. On Qwen3, the fused implementation achieves 0.810 GSM8K accuracy at 16× resident expert-weight compression, compared with 0.868 for the uncompressed model. The MATH-500 and IFEval results provide complementary evidence that substantial generation capability can be retained. These results support the practical potential of selectively recovering expert-specific computation.
Critical weaknesses
-
The central correction-granularity comparison lacks validation under prefix-causal evaluation. As acknowledged in Section 3, pooling activations across a teacher-forced continuation allows later supplied tokens to influence the correction set used at earlier positions. Consequently, the likelihood-scored comparison between coupled and projection-wise correction in Table 4 does not establish that the reported advantage persists during autoregressive generation. The generated-answer benchmarks remain valuable, but they do not directly isolate this central design choice. This leaves an important gap between the proposed mechanism and its empirical validation.
-
Key matched generation controls are missing, particularly in Table 2. Table 2 includes bare pruning but omits the patched pruning control for both MATH-500 and IFEval. These results therefore demonstrate recovery relative to uncorrected pruning, without establishing an advantage over a matched alternative that also receives residual corrections. The gap also appears in Table 1: row 9a provides the strongest reported pruning-based likelihood diagnostic after adding global allocation, but its GSM8K result is unavailable. Together, these omissions leave the central generation comparison incomplete.
-
Accuracy preservation under deep compression varies substantially across models. At 16× compression, the corrected Mixtral-8x7B model achieves only 0.222 GSM8K accuracy, compared with 0.606 for the uncompressed model (Table 3). This is a much larger relative degradation than that reported on Qwen3. Correction still improves the compressed Mixtral model, and the paper acknowledges this model dependence. Nevertheless, the second-model results support the existence of recovery more strongly than they support broadly reliable accuracy preservation. It remains unclear how much the favorable Qwen3 outcome depends on its architecture, expert structure, or compression configuration.
Questions for authors
-
Validation of the correction mechanism: Does the advantage of coupled over projection-wise correction in Table 4 persist when selection uses only the available prefix, or in a matched generated-answer evaluation? A check on one representative merged configuration would materially strengthen the evidence for this design choice. If such evidence is already available, please identify it.
-
Completeness of the matched comparisons: What is the GSM8K performance of the pruning-plus-patches-plus-global-allocation configuration in Table 1, row 9a? Can the authors also provide the patched pruning control missing from Table 2, or explain whether existing results support the same comparison on MATH-500 and IFEval?
2 comments
Thank you for the additional experiments. The causal rerun addresses W1, and the new pruning results fill the gaps in W2. The narrow gains over recalibrated pruning on math tasks and the accuracy drop on Mixtral leave my concern about a consistent practical advantage at 16× unresolved.
Q1: Yes. We reran Table 4's coupled vs. projection-wise comparison under causal selection at all three compression levels (Avg8, 32,946 documents per arm). "Batch" pools the prompts of a forward batch and, at each decode step, the tokens of that step (the paper's record size). "Per-example" selects from one example's own prompt and each answer token's own activation, so it is also batch-independent.
The coupled advantage persists in every row (all 95% paired bootstrap intervals exclude zero). It narrows as each record covers fewer tokens, and is absent at 2.0× when every token selects alone (−0.9 pp), a setting that streams about ten times as many records. We did not run a generated-answer comparison of the two correction types.
Q2: We ran all three missing cells. Row 9a scores 0.224 on GSM8K, and patched pruning scores 0.034 on MATH-500 and 0.235 on IFEval.
Corrections add about 7 pp on GSM8K and 1 pp on MATH-500 and IFEval, so they do not repair the C4-pruned base. Calibration matters far more: with the keep list recomputed on GSM8K training prompts in the evaluation chat format, bare pruning comes within noise of the stacked row on GSM8K and MATH-500 and stays 8.9 pp behind on IFEval (details in our response to Reviewer TXTu). We did not run corrections on the calibrated pruned base. We will add these rows to Tables 1 and 2 and revise Section 4.1.
W3: We agree that accuracy preservation at 16× is model-dependent, as the paper states. Corrections with allocation raise Mixtral's 16× GSM8K from 0.000 to 0.222, still far below the uncompressed 0.606. Our Mixtral claim is that recovery from merging transfers (at 2×, patches raise GSM8K from 0.222 to 0.454), not that 16× preserves accuracy on every model. A likely factor is granularity (8 experts per layer against 128 in Qwen3), which we have not isolated. We will state this scope explicitly.