acceptodds
Under review as a conference paper at ICLR 2027

Compression Hypercube: A Unified Framework for Pruning, Quantization, and Distillation

Abstract

Model compression is critical for deploying large language models (LLMs), yet selecting an effective strategy remains largely empirical: pruning, quantization, and knowledge distillation (KD) are often studied in isolation, with limited guidance on when each method is reliable. We propose the Compression Hypercube, a structured framework that characterizes compression as an interaction between method, compression level, recovery strategy, initialization state, and data modality. We identify three key properties governing compression behavior — sequential redundancy, loss-landscape curvature, and compression-induced distributional shift — and derive local bounds linking these to compression error. Across synthetic environments and eleven real pre-trained and instruction-tuned LLMs up to 14B parameters, we observe substantial variation in outcomes: post-recovery perplexity can differ by up to depending on method choice alone. Weight-only 8-bit quantization remains near-lossless across models, while depth pruning leads to significant degradation even after recovery. Recovery behavior is regime-dependent: KD with reverse-KL or on-policy objectives dominates SFT across compression regimes, reversing earlier findings based on forward-KL only. Finally, a composite diagnostic using four base-model features (bottleneck redundancy , curvature ratio, weight kurtosis, and nominal compression level) achieves zero top-1 regret in leave-one-model-out configuration selection, providing a cheap pre-deployment protocol that requires no trial compression runs. Together, these results replace ad-hoc compression selection with a principled, theory-grounded workflow: diagnose, rank, and verify — all from a single forward/backward pass on the uncompressed model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Compression, Large language models, Efficient LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1ePSOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Compression Hypercube, a unified framework for analyzing LLM compression across five dimensions: compression method, compression level, recovery strategy, initialization state, and data distribution. The authors provide theoretical analyses based on compression-induced perturbations and loss-landscape curvature, and conduct experiments across multiple LLM families to investigate compression sensitivity, recovery effectiveness, and configuration selection.

Critical strengths
  1. The paper addresses a practically relevant problem by systematically examining the interactions among different compression methods, recovery strategies, and model characteristics.

  2. The experiments cover multiple LLM families, compression techniques, and recovery settings, providing a broad empirical comparison of compression behaviors.

  3. The paper combines theoretical analysis with empirical observations and explores a lightweight predictor for estimating compression sensitivity without performing trial compression.

Critical weaknesses
  1. Limited novel insights. Although the paper addresses an interesting problem, many findings, such as the robustness of 8-bit quantization and the severe degradation caused by aggressive pruning, are already well established. The proposed Compression Hypercube organizes existing techniques into a unified framework, but it remains unclear what fundamentally new or generalizable insights are obtained beyond prior empirical studies.

  2. Limited practical guidance from the theoretical analysis and configuration predictor. Theorems 3.1–3.2 mainly rely on standard second-order perturbation analysis, without providing clear guidance for selecting compression methods. More importantly, the proposed configuration predictor does not include the compression method as an input feature. As acknowledged in Appendix F.5, its within-model ranking depends entirely on compression level, since structural features remain constant across configurations. Consequently, it cannot distinguish Wanda, AWQ, and depth pruning at the same nominal compression level. This substantially limits its practical value for compression strategy selection.

  3. Insufficient experimental rigor in analyzing recovery strategies. The observed reversal between KD and SFT is potentially interesting: KD outperforms SFT under Wanda-50% pruning (Table 5), whereas SFT outperforms default forward-KL KD under Depth-50% pruning (Table 15). However, these experiments use different recovery objectives and training configurations, making it difficult to attribute the reversal to the compression method itself. A comparison using consistent recovery objectives, training budgets, and hyperparameter tuning protocols is needed to support this conclusion.

  4. Incorrect table numbering. The numbering of Tables 3 and 4, as well as Tables 5 and 6, appears to be reversed. The authors should correct the table numbering to ensure consistency with their order of appearance.

Questions for authors

See weaknesses

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1ePS Comments

We thank the reviewer for the careful reading. We address each concern with new experiments.

Limited novel insights

We respectfully disagree that the contribution is organizational. Three contributions are demonstrably new:

  1. Method ×\times recovery interaction. No prior work has shown that the optimal recovery reverses by compression type. Gu et al. and Agarwal et al. study KD for a single method and assume it always wins --- we show this fails under depth pruning. 55.4% of predictive signal sits in this interaction; it is the dominant factor, not a marginal effect. At 14B, scale modulates the reversal itself --- a three-way interaction no single-axis study could find.
  2. Factorial design reveals cross-axis effects invisible to single-axis studies by construction. Each prior paper holds one axis constant and varies another. The reversal was discoverable only because we varied all five dimensions simultaneously.
  3. Cross-model prediction (τ\tau=0.917, 5 families) from pre-compression diagnostics alone has not appeared in prior work. This has direct practical value --- a practitioner can estimate compressibility before spending any compute. Our experiments span 2B–14B across 3 families (LLaMA, Qwen, Gemma) --- the broadest controlled compression study we know of.

Theory and configuration predictor

On the theory: we agree Theorems 3.1–3.2 use standard perturbation techniques. The novelty is not the proof technique but the predictive power it creates: Smin⁡S_{\min} and λmax⁡\lambda_{\max} rank model families by compression robustness before any compression (τ\tau=0.917 across 5 families). No prior work connects pre-compression spectral diagnostics to post-compression quality at this level of cross-family validation.

ModelDepth-50% PPL Ratio (Π\Pi)
Gemma-2B50.77 / 14.19 = 3.58$\times$
LLaMA-8B37.95 / 10.83 = 3.50$\times$

Smin⁡S_{\min} ordering matches actual degradation. On the predictor: the reviewer is correct that structural features are constant within a model. But our ablation (leave-one-model-out CV, 18 configs, 5 families) shows the predictor's value comes from a different source entirely:

VariantFeaturesRegret ↓\downarrowKendall τ\tau ↑\uparrow
ACompression only0.6620.000
B+ method type0.0430.816
DFull (interactions)0.0430.917
ERandom2.3530.000

Compression-only (A) has τ\tau=0.000 --- no better than random. It cannot distinguish Wanda, AWQ, and depth pruning at the same compression level, exactly as the reviewer notes. But that is the point: the method ×\times recovery interaction carries 55.4% of signal (R2R^2=0.832), while compression level contributes 0.0%. Structural features (Smin⁡S_{\min}, λmax⁡\lambda_{\max}) serve a different role --- cross-model generalization, predicting how an unseen family responds to compression --- while method ×\times recovery features drive within-model selection.

KD/SFT reversal not isolated

This is a fair concern. We addressed it with a fully matched experiment where every hyperparameter is held constant, only the loss function differs (CE for SFT vs forward-KL for KD). LoRA r=8, α\alpha=16, same data subset (10M tokens SlimPajama), cosine LR 2e-4, BF16. No confounds remain.

LLaMA-3.1-8B depth-50% (baseline: PPL 10.83)

RecoveryPPL ↓HellaSwag ↑ARC-C ↑WinoGrande ↑PIQA ↑IFEval ↑
SFT37.950.3700.2260.5140.6340.148
KD49.120.3600.2280.5090.6160.139

Gemma-2-2B all configs (baseline: PPL 14.19)

ConfigRecoveryPPL ↓HellaSwag ↑ARC-C ↑WinoGrande ↑PIQA ↑IFEval ↑
WandaSFT14.420.6650.3990.6580.7620.176
WandaKD17.060.6600.4300.6700.7640.309
DepthSFT50.770.3830.2590.5240.6430.124
DepthKD60.970.3740.2670.5090.6550.131

The pattern is unambiguous: SFT wins depth pruning (22.7% lower PPL on LLaMA, 16.7% on Gemma), KD wins Wanda (ARC +3.1pp, IFEval +13.3pp on Gemma). This replicates across two distinct families under strict controls - the reversal is real and driven by how compression type interacts with the recovery objective.

Table numbering

Thank you for pointing out. We will correct this in the camera-ready version.

AuthorsOpenReview ↗

Please review our responses

Dear Reviewer 1ePS,

As we approach the end of the discussion phase, we sincerely request you to review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.

If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.

Thank you for your time, thoughtful feedback, and consideration.

Thanks

Thank you for the detailed response and additional experiments. The matched SFT/KD experiments help address my concerns about experimental consistency.

However, my main concerns remain: (1) the observed compression–recovery interaction is interesting, but the results remain largely descriptive, without explaining why different compression methods favor different recovery objectives or establishing a consistent pattern across evaluation metrics; and (2) the predictor's practical value remains unclear, particularly given the substantial improvement from adding method-specific features.

While the rebuttal improves the empirical rigor, these concerns still limit the strength of the paper's contributions.

Reviewer_fHcHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper organizes compression by method, severity, recovery strategy, initialization, and data distribution. It combines local perturbation analysis with synthetic and real-model experiments across three families, including pretrained/instruction-tuned pairs and selected configurations up to 14B. It compares compression damage, recovery methods, structural diagnostics, and systems performance, and explores a predictor for configuration selection.

Critical strengths
  1. Broad coverage of compression operators and recovery strategies.
  2. Matched pretrained/instruction-tuned comparisons across three families.
  3. Systems measurements demonstrate implementation-dependent efficiency.
Critical weaknesses
  1. Limited evaluations: mainly covers WikiText-2 and four multiple-choice benchmarks, without dedicated coding, mathematical generation, instruction-following, or agentic tasks. The paper evaluates instruction-tuned models without instruction-following benchmarks, leaving preservation of their instruction-following behavior unclear.
  2. BOS-sensitive baselines and differences between recovery experiment setups weaken some reported comparisons
  3. Findings lack generality: Not every central finding is replicated across all three families.
  4. The selector does not demonstrate that structural model features improve configuration selection beyond using compression level alone.
  5. Primary experiments cover 2B–8B; selected 14B tests do not establish larger-scale recovery behavior.
Questions for authors
  1. Can consistent tokenization/BOS evaluation and matched-budget recovery reruns reproduce the conclusions?
  2. Do operator and recovery rankings hold on generative and instruction-following tasks?
  3. Given the current selector does not establish that structural features improve within-model configuration selection - does the proposed selector outperform selection based on compression level alone?
4 comments
AuthorsOpenReview ↗

Response to Reviewer fHcH Comments

We thank the reviewer for the thorough review. We address each weakness and question with new experiments.

Limited evaluations

We added IFEval (instruction-following) and GSM8K (math reasoning) across all three families --- generation-based benchmarks that directly test instruction-following preservation. The KD/SFT interaction holds on these tasks: under Wanda on Gemma-2B, KD outperforms SFT on IFEval by +13.3pp (0.309 vs 0.176) --- the advantage is actually stronger on instruction-following than on discriminative benchmarks. Under depth on LLaMA-8B, SFT wins IFEval (0.148 vs 0.139) alongside PPL (37.95 vs 49.12). The reversal pattern is consistent across both discriminative and generative evaluation. Full 14B results are presented in W5; at that scale KD becomes universally required while SFT collapses entirely, and the interaction shifts from compression-type-dependent to scale-dependent.

IFEval and GSM8K will be added to main tables in the camera-ready.

BOS baselines and setup differences

We confirmed double-BOS via systematic tokenizer audit: LLaMA and Gemma auto-prepend BOS while lm-eval-harness also prepends it, producing [BOS, BOS, ...]. Qwen is unaffected (no BOS token). Double-BOS affects all configs equally within a family --- compressed and uncompressed, SFT and KD --- so all relative comparisons remain valid. BOS-corrected re-eval on LLaMA-8B Wanda-50% confirms: KD maintains higher HellaSwag (0.777 vs 0.769) and ARC-C (0.544 vs 0.539) across two seeds. Camera-ready will use BOS-corrected numbers throughout.

For setup differences: our matched experiment holds every hyperparameter constant (LoRA r=8, α\alpha=16, 10M tokens SlimPajama, cosine LR 2e-4, BF16) --- only the loss function differs (CE vs forward-KL). No confounds remain.

Cross-family generality

Now complete on three diverse families:

  • Gemma-2B --- SFT wins depth (PPL 50.77 vs 60.97, 16.7% lower), KD wins Wanda (IFEval +13.3pp, ARC +3.1pp).
  • LLaMA-8B --- SFT wins depth decisively (PPL 37.95 vs 49.12, 22.7% gap).
  • Qwen-14B --- KD universally required; SFT collapses.

The interaction replicates across all three. Scale modulates which recovery wins: at 2B–8B it depends on compression type; at 14B, KD is universally necessary.

Selector and structural features

Our ablation (leave-one-model-out CV, 18 configs, 5 families) directly addresses this:

VariantFeaturesRegret ↓\downarrowKendall τ\tau ↑\uparrow
ACompression only0.6620.000
B+ method type0.0430.816
DFull (interactions)0.0430.917
ERandom2.3530.000

Compression-only (A) has τ\tau=0.000, literally no better than random. Adding method type (B) cuts regret by 93.5%. The method ×\times recovery interaction carries 55.4% of signal (R2R^2=0.832); compression level contributes 0.0%. The structural features (Smin⁡S_{\min}, λmax⁡\lambda_{\max}) serve a different purpose: cross-model generalization --- predicting how an unseen family responds to compression (τ\tau=0.917 across 5 families) without running any compression.

Scale

We ran Qwen2.5-14B with all four configs (5M tokens, INT8 teacher, matched protocol):

ConfigPPL ↓\downarrowHellaSwag ↑\uparrowARC-C ↑\uparrowWinoGrande ↑\uparrowPIQA ↑\uparrow
Wanda + SFTdiverged$^\dagger$0.2630.2520.5010.515
Wanda + KD36.550.7300.4590.7070.763
Depth + SFTdiverged$^\dagger$0.2610.2540.5040.516
Depth + KD148.440.4180.2800.5290.624

†^\daggerSFT collapsed: train loss →\to 0 (PPL ≈\approx 1.0), eval PPL →∞\to \infty, all metrics at chance.

At 14B, KD becomes universally required. With 0.36 tokens/param (vs 1.25 at 8B), CE memorizes without generalizing; the teacher's soft labels provide essential regularization. This scale ×\times recovery interaction is absent at 2B–8B. 70B+ is considered as future work.

Questions

Q1: Yes. BOS-corrected re-evaluation preserves all relative orderings. Matched-budget recovery reproduces the reversal across two families and three scales.

Q2: Yes. IFEval confirms the interaction: KD advantage under Wanda is stronger on instruction-following (+13.3pp) than on discriminative tasks. SFT advantage under depth also holds on IFEval.

Q3: Yes. Compression-only selector: τ\tau=0.000 (random). Full selector with interactions: τ\tau=0.917. The selector outperforms compression level alone by every metric --- see W4 ablation.

AuthorsOpenReview ↗

Detailed discussion on rebuttal to reviewer fHcH comments

Table 1: Gemma-2B --- All Configs (10M tokens)

ConfigPPL ↓\downarrowHellaSwag ↑\uparrowARC-C ↑\uparrowWinoGrande ↑\uparrowPIQA ↑\uparrowIFEval ↑\uparrow
Wanda + SFT14.420.6650.3990.6580.7620.176
Wanda + KD17.060.6600.4300.6700.7640.309
Depth + SFT50.770.3830.2590.5240.6430.124
Depth + KD60.970.3740.2670.5090.6550.131

KD wins Wanda across all metrics except PPL. IFEval gap is +13.3pp --- larger than on discriminative tasks, showing the interaction is stronger on instruction-following. SFT wins depth on PPL (16.7% lower) and most benchmarks.

Table 2: LLaMA-3.1-8B --- Depth-50% (10M tokens)

RecoveryPPL ↓\downarrowHellaSwag ↑\uparrowARC-C ↑\uparrowWinoGrande ↑\uparrowPIQA ↑\uparrowIFEval ↑\uparrow
SFT37.950.3700.2260.5140.6340.148
KD49.120.3600.2280.5090.6160.139

SFT wins on 5 of 6 metrics. PPL gap is 22.7%. The reversal from Table 1 (where KD wins Wanda) replicates on a different family and scale.

Table 3: BOS-Corrected LLaMA-8B Wanda-50%

ConfigPPL ↓\downarrowHellaSwag ↑\uparrowARC-C ↑\uparrow
SFT seed 428.960.7680.541
SFT seed 1378.950.7690.536
KD seed 429.410.7770.542
KD seed 1379.380.7770.546

After BOS correction, KD still wins on HellaSwag (+0.008) and ARC-C (+0.006). Consistent across two seeds. PPL favors SFT but downstream metrics favor KD --- same pattern as Gemma-2B Wanda.

Table 4: Qwen2.5-14B --- All Configs (5M tokens, INT8 teacher)

ConfigPPL ↓\downarrowHellaSwag ↑\uparrowARC-C ↑\uparrowWinoGrande ↑\uparrowPIQA ↑\uparrowIFEval ↑\uparrow
Wanda + SFT2.46×10192.46 \times 10^{19}0.2630.2520.5010.5150.160
Wanda + KD36.550.7300.4590.7070.7630.155
Depth + SFT2.67×10212.67 \times 10^{21}0.2610.2540.5040.5160.145
Depth + KD148.440.4180.2800.5290.6240.145

SFT collapses for both compression types. All SFT metrics are at chance (HellaSwag $\sim$0.26 for 4-way, WinoGrande $\sim$0.50 for binary). KD retains meaningful performance under both methods.

Table 5: Qwen2.5-14B --- Train vs Eval PPL

ConfigTrain PPLEval PPL
Wanda + SFT1.00052.46×10192.46 \times 10^{19}
Wanda + KD1.41336.55
Depth + SFT1.00022.67×10212.67 \times 10^{21}
Depth + KD1.391148.44

Train PPL ≈\approx 1.0 for SFT means the model memorized training data perfectly but learned nothing generalizable. At 0.36 tokens/param (5M/14B), there is far more model capacity than data. CE on hard labels runs out of signal once memorization is complete --- gradients go to zero and training stalls. KD does not have this problem: matching the teacher's full output distribution is harder than matching one-hot labels, so gradients remain non-zero throughout training (train PPL stays at $\sim$1.4). At equal budgets, CE simply cannot regularize a 14B model the way soft labels can. The 0.36 tokens/param regime is realistic for practitioners at this scale. INT8 teacher quantization is near-lossless and does not affect this comparison.

AuthorsOpenReview ↗

Please review our responses

Dear Reviewer fHcH,

As we approach the end of the discussion phase, we sincerely request you to review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.

If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.

Thank you for your time, thoughtful feedback, and consideration.

Thanks

Reviewer_fHcHOpenReview ↗

Response to Authors

Thank you for the additional experiments. The instruction-following evaluations, matched recovery runs, and larger-model experiments improve the empirical coverage. However, substantial concerns remain:

  1. Generality: The claimed recovery pattern is not consistent across metrics, even within the same model: Gemma-2B favors SFT on perplexity and KD on IFEval under both Wanda and depth pruning. The 14B results doesn't reproduce the claimed reversal either. SFT collapse under one setting doesn't establish the new claim that KD is universally required at this scale. Moreover, prior work has demonstrated successful CE-based recovery at larger scales.

  2. The BOS-corrected Llama results are useful, but do not resolve all affected comparisons. Applying the same double-BOS protocol does not guarantee unchanged relative rankings.

  3. Method type alone and the full predictor achieve identical regret (0.043), leaving the selection benefit of structural features unestablished.

These issues continue to weaken the central conclusions. I retain my recommendation of clear rejection.

Reviewer_TuJeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how common factors in compresssion affect LLM compression. It introduces a five-dimensional Compression Hypercube framework and develops theoretical insights connecting compression damage to curvature and representation redundancy. Experiments on synthetic model and real LLMs report various performance under different setup, like quantization, depth pruning, and SFT/knowledge distillation.

Critical strengths
  • The paper addresses an interesting and important practical problem. It tries to understand when different LLM compression methods and recovery strategies are effective. The writing is also clear and easy to follow.
  • The analysis considers several relevant factors, including representation redundancy, loss-landscape curvature, etc., which are important for understanding compression behavior.
Critical weaknesses
  • My major concern is that the model provide limited explanatory contrbution. The theoretical analysis is well connected to practical compression decisions. After reading it, I still cannot determine which layers to remove, or which recovery objective to use. The paper should clarify what new, actionable understanding follows from the framework.
  • For the same LLM, the structural features change every candidate’s predicted score by the same amount, leaving their ranking unchanged. The ranking therefore depends only on compression level. As a result, the selection experiment does not demonstrate that these features help choose a better compression method.
  • The experimental analysis lacks the depth to support the framework’s scope. More controlled interventions are needed to test whether each proposed factors actually explain compression behavior. For example, the paper reports changes in the relative effectiveness of SFT and KD without clearly isolating their causes or connecting them to the previous diagnostics. The experiments therefore remain largely descriptive, leaving a substantial gap between the framework’s ambition and its empirical validation.
Questions for authors

See above

3 comments
AuthorsOpenReview ↗

Response to Reviewer TuJe Comments

We thank the reviewer for the constructive feedback. We address each concern with new experiments.

Limited explanatory contribution

We respectfully disagree that the contribution is limited. The actionable guidance from our framework is concrete: use SFT for depth pruning at 2B–8B, use KD for unstructured pruning, and use KD universally at 14B+. The selector automates this for unseen models (τ\tau=0.917). Beyond this, three specific contributions are new:

  1. Method x recovery interaction. No prior work has shown that the optimal recovery reverses by compression type. Gu et al. and Agarwal et al. study KD for a single method and assume it always wins. We show it does not. 55.4% of predictive signal sits in this interaction; it is the dominant factor, not a marginal effect. At 14B, scale modulates the reversal itself - a three-way interaction no single-axis study could find.
  2. Factorial design reveals cross-axis effects invisible to single-axis studies by construction. Each prior paper holds one axis constant and varies another. Our five-dimensional design is what made the reversal discoverable.
  3. Cross-model prediction (τ\tau=0.917, 5 families) from pre-compression diagnostics alone has not appeared in prior work. This has direct practical value - a practitioner can estimate compressibility before spending any compute on compression.

Our experiments span 2B–14B across 3 families (LLaMA, Qwen, Gemma) - the broadest controlled compression study we know of.

Selection experiment

We ran a systematic ablation under leave-one-model-out CV across 18 configs and 5 families. The results directly refute the characterization that ranking depends only on compression level:

VariantFeaturesRegret ↓Kendall τ\tau ↑KD/SFT Acc
ACompression only0.6620.00066.7%
B+ method type0.0430.81666.7%
C+ recovery type0.1310.75066.7%
DFull (interactions)0.0430.91777.8%
ERandom2.3530.00050.0%
FOracle0.0001.000100.0%

Compression-only (A) has τ\tau=0.000 - literally no better than random. It cannot distinguish between methods or recovery strategies at the same ratio. Adding method type (B) cuts regret by 93.5%, and the full model with interactions (D) reaches τ\tau=0.917 - near-oracle. Feature importance confirms: the method x recovery interaction carries 55.4% of signal (R2R^2=0.832), while compression level contributes exactly 0.0%. A practitioner using compression level alone to choose recovery would get random outcomes. The selector works because of how compression type and recovery strategy interact. We will clarify this in the camera-ready version.

KD/SFT reversal not isolated

This is a fair concern. We addressed it with a fully matched experiment where every hyperparameter is held constant, only the loss function differs (CE for SFT vs forward-KL for KD). LoRA r=8, α\alpha=16, same data subset (10M tokens SlimPajama), cosine LR 2e-4, BF16. No confounds remain.

LLaMA-3.1-8B depth-50% (baseline: PPL 10.83)

RecoveryPPL ↓HellaSwag ↑ARC-C ↑WinoGrande ↑PIQA ↑IFEval ↑
SFT37.950.3700.2260.5140.6340.148
KD49.120.3600.2280.5090.6160.139

Gemma-2-2B all configs (baseline: PPL 14.19)

ConfigRecoveryPPL ↓HellaSwag ↑ARC-C ↑WinoGrande ↑PIQA ↑IFEval ↑
WandaSFT14.420.6650.3990.6580.7620.176
WandaKD17.060.6600.4300.6700.7640.309
DepthSFT50.770.3830.2590.5240.6430.124
DepthKD60.970.3740.2670.5090.6550.131

The pattern is unambiguous: SFT wins depth pruning (22.7% lower PPL on LLaMA, 16.7% on Gemma), KD wins Wanda (ARC +3.1pp, IFEval +13.3pp on Gemma). This replicates across two distinct families under strict controls - the reversal is real and driven by how compression type interacts with the recovery objective. Prior work (Gu et al., 2024; Muralidharan et al., 2024) assumed KD universally dominates SFT; our matched experiment shows this assumption fails under depth pruning. At Qwen2.5-14B, the picture shifts further: SFT collapses entirely while KD remains functional, making scale yet another interacting dimension. Mechanistic discussion on why depth pruning severs the layer correspondence KD relies on will be added in the revised manuscript.

AuthorsOpenReview ↗

Please review our responses

Dear Reviewer TuJe,

As we approach the end of the discussion phase, we sincerely request you to review our responses. We have made every effort to carefully address all your concerns and provide detailed clarifications.

If you find our responses satisfactory, we would be grateful if you could consider reassessing your evaluation of our paper. We remain fully committed to addressing any additional questions or concerns you may have.

Thank you for your time, thoughtful feedback, and consideration.

Thanks

Thank you for the additional experiments. The matched SFT/KD comparisons strengthen the empirical findings and address part of my concern about the experimental controls. However, my main concern remains the framework’s overall explanatory contribution.

The hypercube organizes relevant experimental factors, but the paper does not sufficiently establish how the proposed diagnostics explain or predict compression outcomes across the studied settings. The SFT/KD comparison is one example: it remains unclear how redundancy or curvature accounts for the relative effectiveness of these objectives. More broadly, the local perturbation bounds provide some intuition about compression damage, but their connection to the main empirical findings is not sufficiently validated.

Overall, I see the framework’s contribution primarily as organizing evaluations and summarizing statistical patterns. The additional experiments make these observations more credible, but they do not yet establish a unified explanation connecting the proposed properties to the observed compression behavior. Therefore, I tend to keep my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Official comment by authors and paper revision

We thank the reviewers for their time and constructive feedback. Each comment has been addressed in the individual responses below, and the manuscript has been revised accordingly --- all changes are highlighted in blue. We hope the responses and revisions resolve the concerns raised, and we welcome any further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.