acceptodds
Under review as a conference paper at ICLR 2027

Prune What the Teacher Can Repair: Distillation-Aware Pruning for LLMs

Abstract

This paper proposes a novel Distillation-Aware Pruning (DAP) framework for addressing the teacher-student compatibility issue in large language models (LLMs). Unlike existing approaches to teacher-student mismatch, which typically assume a predefined student and subsequently adapt the distillation objective to bridge the resulting gap, we revisit the student construction problem through pruning, which provides a natural mechanism for preserving parameters and representations from its parent teacher. Motivated by this intuition, DAP therefore prunes for post-distillation quality rather than immediate post-pruning quality by preserving what the intended distillation process cannot recover and removing what it can repair. Across settings on Qwen3-8B and DeepSeek-R1-Distill-Llama-8B, multiple tasks, and both supervised and on-policy distillation, DAP consistently produces stronger post-distillation students than both independently constructed and conventionally pruned alternatives under matched parameter and training budgets.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pruning, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.80 over 5 reviews, above 85% of the others.
Reviewer_y2XuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks how pruning masks should be chosen when distillation comes afterward. DAP estimates post-distillation damage by subtracting a local recovery term from the immediate pruning damage. It combines reference and distillation-conditioned statistics, using layerwise Kronecker factorization and a diagonal output-gradient factor to keep the computation manageable. Experiments on Qwen3-8B and DeepSeek-R1-Distill-Llama-8B cover MATH-500, StrategyQA, and APPS under SFT and on-policy distillation (OPD). DAP beats the tested baselines at 50% unstructured and 2:4 sparsity. Teacher and procedure crossover experiments also test whether the preferred mask depends on the distillation setup.

Critical strengths
  1. The main contribution is making the downstream distillation procedure part of the pruning decision. Tables 2–3 show that a mask chosen for one teacher or procedure may be less suitable for another. I find these comparisons more persuasive than baseline gains alone, since they directly test the paper’s motivation.
  2. The gains hold across the main settings. In Table 1, DAP improves the three-task average over the strongest baseline by 3.3/3.5 points for unstructured pruning and 3.7/3.9 points for 2:4 pruning under SFT/OPD. Table 5 shows similar gains on the second model. The recovery-term ablation and coefficient sweep help explain the method, and the paper also reports where its prediction breaks down at high sparsity.
Critical weaknesses
  1. The procedure-conditioned calibration recipe is still unclear. Appendix A.4 specifies the teacher that samples targets and says gradients are evaluated through the dense parent. It does not clearly identify the policy generating OPD calibration contexts before mask selection, or the weighting used for SFT and OPD. These details matter because they determine how the score depends on the procedure. The training datasets and exact OPD loss are also missing from Appendix A.1, making Table 2 difficult to reproduce.
  2. The diagonal approximation’s practical savings are not measured. Table 4 shows the score falling from 58.9 with full-G to 55.5 with diagonal-G. The storage-complexity reduction is clear, but construction time and peak memory would show what this 3.4-point tradeoff buys in practice. Equal calibration-token budgets do not imply equal mask-construction costs.
Questions for authors
  1. Which policy generates the OPD calibration contexts before the final mask is selected? Please distinguish it from the target-sampling teacher in Appendix A.4 and specify the weighting used for SFT and OPD.
  2. Which datasets are used for calibration and downstream training, and what is the exact OPD training loss?
  3. What are the construction time and peak memory for full-G, diagonal-G, and the pruning baselines on the same hardware?
3 comments

We thank the reviewer for the feedback. We clarify the calibration procedure, datasets, OPD objective, and pruning costs.


Q1 and Part of Weakness 1

OPD-Conditioned Calibration and Weighting

OPD calibration. Since the final DAP mask is unavailable during calibration, we construct a frozen sparse proxy using the reference-only criterion (η=0\eta=0) on 128 reference sequences. The proxy πproxy=πθT⊙z0\pi _{\mathrm{proxy}}=\pi _{\theta _T \odot z _0} matches the final student's sparsity and requires no distillation or recovery statistics, avoiding circular dependence.

The proxy generates 128 OPD calibration trajectories from held-out prompts (temperature 0.7, top-pp 0.95, maximum 4,096 tokens, one rollout per prompt). At each context τt\tau _t, the dense teacher samples the target yt∼πθT(⋅∣τt)y _t\sim\pi _{\theta _T}(\cdot\mid\tau _t), and auxiliary NLL gradients are evaluated at the dense parent θT\theta _T.

Procedure-specific weighting. SFT uses fixed teacher-generated demonstration contexts. Both procedures use identical teacher-target curvature estimators, valid-token masking, sample weighting, and normalization. Following Appendix A.5.3, activation moments are averaged over valid invocations; output-gradient moments are summed within sequences and averaged over sequences. The difference arises from autoregressive context distributions, not the KL direction or rescaling.

Proxy validation. Under identical calibration and training budgets:

OPD calibration sourcePost-OPD Avg.
Dense-teacher rollouts58.9
Frozen sparse-proxy rollouts60.7

The +1.8-point improvement supports proxy-generated contexts for mask selection. The proxy approximates early student-generated contexts, not the evolving distribution throughout training.


Q2 and Part of Weakness 1

Calibration Datasets, Training Data, and OPD Loss

Datasets. We use MATH, StrategyQA, and APPS training splits, partitioned into disjoint calibration and downstream training subsets:

  • Reference calibration: 128 held-out sequences (4,096 tokens), with dense-parent-generated responses.
  • SFT calibration: 128 held-out prompt–demonstration sequences using fixed teacher-generated responses.
  • OPD calibration: 128 held-out frozen-proxy rollouts, using the SFT calibration prompt distribution.
  • Downstream training: Remaining examples from the same dataset mixture. SFT uses teacher-generated reasoning/code demonstrations; OPD uses current-student rollouts. Both use 0.1B training tokens.

Calibration, training, and evaluation examples are disjoint.

OPD objective. We use an on-policy distillation formulation based on reverse KL (student || teacher) over student-generated contexts:

LOPD(θ)=Ex∼D,,o∼πθ(⋅∣x)[1∣o∣∑t=1∣o∣DKL(πθ(⋅∣τt)∥πθT(⋅∣τt))]\mathcal{L} _{\mathrm{OPD}}(\theta)=\mathbb{E} _{x\sim\mathcal{D},\\,o\sim\pi _\theta(\cdot\mid x)}\left[\frac{1}{|o|}\sum _{t=1}^{|o|}D _{\mathrm{KL}}\left(\pi _\theta(\cdot\mid\tau _t)\Vert\pi _{\theta _T}(\cdot\mid\tau _t)\right)\right]

Here, τt=(x,o<t)\tau _t=(x,o _{<t}) denotes the student-generated context. The loss averages over valid response tokens, excluding prompt and padding positions.

The student generates one rollout per prompt (temperature 0.7, top-pp 0.95, maximum 4,096 tokens), with the dense parent as the frozen teacher. This objective differs from the teacher-sampled auxiliary NLL used only for curvature estimation.

We will add these details to Appendix A.1.


Q3 and Weakness 2

Pruning Runtime and Peak GPU Memory

We benchmark pruning costs on H100 GPUs using Qwen3-8B and 256 calibration sequences, excluding downstream distillation.

MethodTime (s)GPU-hoursPeak Mem./GPU
WANDA3000.08328 GB
WANDA++4800.13332 GB
SparseGPT7200.20030 GB
DAP (diagonal-G)1,3800.76748 GB
DAP (full-G)2,4001.33471 GB

DAP uses two H100s with FSDP2, versus one for conventional baselines; GPU-hours account for this difference. Costs include calibration, curvature collection, mask construction, and applicable proxy-generation overhead.

With two GPUs, diagonal-G reduces GPU-hours by 42.5% and peak memory by 32.4% relative to full-G, at a cost of 3.4 post-SFT points (55.5 vs. 58.9). This quantifies the accuracy–efficiency tradeoff rather than assuming equal calibration budgets imply equal computational costs.

We will incorporate these clarifications and measurements into the revision.

Dear Reviewer y2Xu:

We appreciate your time reviewing our paper. Should you have a chance to review our rebuttal, we would welcome any further feedback or updated evaluation.

Thx for the response, it resolves my concern and I hold my score unchanged.

Reviewer_YbtpOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper addresses mask selection in the prune-then-distill setting for LLMs. Instead of minimizing the immediate pruning damage DD, the authors minimize the residual damage I=D−RI = D - R that remains after distillation, where RR is the part the teacher can recover. They derive a second-order criterion I^=D^−R^\hat{I} = \hat{D} - \hat{R} and scale it with K-FAC curvature (Eq. 11). Experiments on Qwen3-8B and DeepSeek-R1-Distill-Llama-8B with 50% unstructured and 2:4 sparsity show gains of 3.3--3.9 points over the best pruning baselines, and Tables 2--3 show that the optimal mask depends on the teacher and the distillation procedure. Overall the paper is theoretically solid, and the method is simple yet efficient.

Critical strengths
  1. Novelty. The paper shifts the pruning target from immediate damage to the damage left after distillation, so the mask is chosen with the intended teacher and procedure in mind. Tables 2 and 3 show that the best mask indeed changes with the teacher and with SFT vs. OPD, which I find a new and interesting observation.
  2. Theory. The derivation is relatively complete: the criterion follows from a second-order expansion plus a masked one-step update, and the appendix covers the single-weight, diagonal-G and full-G cases. The authors are also clear about where the analysis holds and where it is only a surrogate.
  3. Experiments. DAP outperforms all pruning baselines across two parent models, two sparsity patterns and two distillation procedures under matched budgets, with average gains of 3.3 to 3.9 points.
Critical weaknesses
  1. Relation to OBD/OBS is not discussed. The immediate-damage term D^\hat{D} is the Optimal Brain Damage (OBD) saliency, and measuring residual damage after re-adjusting the surviving weights is exactly the Optimal Brain Surgeon (OBS) idea. In the matched-teacher case (HKD≈HrefH_{\mathrm{KD}} \approx H_{\mathrm{ref}}), I^\hat{I} reduces to a first-order (in η\eta) approximation of the OBS saliency, and the η=0\eta = 0 variant in Table 2 is OBD with K-FAC curvature. Neither OBD nor OBS is cited. The paper should spell out this connection and clarify what DAP adds on top of the classical framework.

  2. Figure quality is weak. Fig. 2 is rather rudimentary: it only shows numerical computation on a toy weight matrix, so the reader has to compare numbers by hand to see what DAP does differently. It does not convey the intuition behind the criterion.

Questions for authors
  1. Please clarify the relationship between DAP and OBD/OBS in the paper:
    • Is DAP with η=0\eta = 0 equivalent to OBD with K-FAC curvature?
    • In the matched-teacher case with HKD=HrefH_{\mathrm{KD}} = H_{\mathrm{ref}}, is I^\hat{I} the first-order (in η\eta) approximation of the OBS saliency? If so, why use the truncated one-step form instead of the closed-form OBS saliency, which can also be computed with K-FAC statistics (as in The LLM Surgeon)?
    • How does DAP compare empirically to an OBS-saliency mask (without the OBS weight update) followed by the same distillation?
  2. How is DD normalized in Fig. 1b? Could you also report the numbers behind Fig. 3 in a table in the appendix?
2 comments

We thank the reviewer for identifying the connection to OBD, OBS, and The LLM Surgeon. We clarify what DAP inherits from these methods, what differs, and provide the requested baseline and figure details.


Weaknesses 1 and Question 1

Relationship to OBD/OBS

1. OBD equivalence. We agree: setting η=0\eta=0 yields OBD-type saliency under our reference-curvature approximation:

I^i(0)=12wi2[Href]ii.\widehat{I}_i(0)=\frac{1}{2} w_i^2[H_{\mathrm{ref}}]_{ii}.

With diagonal-GG K-FAC, [Href](a,b),(a,b)≈γref,a[Aref]bb[H_{\mathrm{ref}}]_{(a,b),(a,b)}\approx\gamma_{\mathrm{ref},a}[A_{\mathrm{ref}}]_{bb}. Thus, Table 2's η=0\eta=0 control is OBD with factorized reference curvature. We will explicitly cite OBD.

2. OBS versus procedure-conditioned recovery. Let PP denote pruned weights, S=PcS=P^c survivors, and u=θT,Pu=\theta_{T,P}. OBS optimizes the quadratic reference loss over surviving-weight adjustments, whereas DAP approximates an update under the intended distillation objective:

\qquad m_S^{\mathrm{DAP}}=\eta H_{\mathrm{KD},SP}u.$$ The latter yields, to first order in $\eta$, $$\widehat{I}_{\mathrm{DAP}}(P)= \frac{1}{2}u^\top H_{\mathrm{ref},PP}u -\eta u^\top H_{\mathrm{ref},PS}H_{\mathrm{KD},SP}u.$$ When $H_{\mathrm{KD}}=H_{\mathrm{ref}}$, this approximates reference damage **after one masked gradient step toward the OBS quadratic optimum**, *not* a Taylor expansion of closed-form OBS saliency, which is independent of $\eta$. **Why not closed-form OBS?** OBS assumes optimal local compensation of the reference loss; DAP seeks masks recoverable under the **prescribed downstream distillation procedure**. Even with the same parent teacher, $H_{\mathrm{KD}}$ can differ from $H_{\mathrm{ref}}$ because SFT uses fixed demonstration contexts while OPD uses frozen-sparse-proxy-generated contexts. Both use teacher-sampled targets and the same dense-parent curvature estimator. Correspondingly, Table 2 shows **+2.9/+3.0 points** for procedure-matched over mismatched masks (SFT/OPD); Table 3 demonstrates teacher specificity. We acknowledge that K-FAC enables scalable OBS compensation, as in *The LLM Surgeon*. Our one-step criterion targets **procedure-conditioned recovery**, not an inability to compute OBS, and is only a local surrogate for extended training. On 15 complete masks, its Spearman correlation with final reference damage is **0.78**, versus **0.62** for immediate damage. We will add the OBD/OBS citations and explicitly delimit our novelty. **3. Requested OBS-saliency experiment.** We use the same reference-curvature approximation, mask budget, and downstream training, **without OBS weight compensation**; surviving weights remain at their original values. | Criterion | Post-SFT Avg. | Post-OPD Avg. | |---|---:|---:| | OBD-type ($\eta=0$) | 46.8 | 51.5 | | OBS saliency (no update) | 49.0 | 53.8 | | **DAP (diagonal-$G$)** | **55.5** | **60.7** | OBS improves over OBD by **2.2/2.3 points** (SFT/OPD), but DAP further improves over OBS by **6.5/6.9 points**. Thus, optimal reference-loss compensation is not necessarily the best criterion for post-distillation mask selection. As a further mechanism control, DAP's **60.7** OPD average exceeds KD-only saliency (**54.4**) and a tuned reference+KD mixture (**57.4**), supporting the value of cross-curvature interactions. --- ## Question 2 > **Figure Clarifications** **Figure 1b normalization.** Immediate damage is reference KL between the dense parent and the pruned student. We normalize by **one fixed denominator**, SparseGPT's immediate damage at 30% sparsity: $$D_{\mathrm{norm}}(m,s)= \frac{D_{\mathrm{raw}}(m,s)}{D_{\mathrm{raw}}(\mathrm{SparseGPT},30\\%)}.$$ This shared scale preserves comparisons across methods and sparsities; $R/D$ is invariant to consistent rescaling. At 50% sparsity, DAP versus WANDA++ has higher normalized immediate damage (**3.40 vs. 1.92**) but higher recovered fraction (**0.77 vs. 0.53**) and lower remaining damage (**0.782 vs. 0.902**). **Figure 3 values.** We will add an appendix table of measured $I$ and predicted $\widehat I$ for each method and sparsity (30%–75%), and clarify the logarithmic axis. At 50%, DAP's measured/predicted damage is **0.782/0.539**, versus WANDA's **0.874/0.729**. At 65%, the prediction favors DAP (**1.27 vs. 1.68**), but measurement favors WANDA (**2.25 vs. 2.55**), transparently documenting the high-sparsity limitation. --- ## Weakness 2 > **Figure 2** We agree the toy matrix obscures the intuition. We will replace it with a direct visual comparison of **immediate, recovered, and remaining damage**, using the above measured DAP–WANDA++ example. The message is that **lower immediate pruning damage does not necessarily mean lower post-distillation damage**.

Dear Reviewer Ybtp:

Thanks again for reviewing our paper. If you’re able to take a look at our rebuttal, we’d be very grateful for any additional comments or revised assessment.

Reviewer_r6ehOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Distillation-Aware Pruning (DAP), a framework that selects pruning masks by considering not only the immediate damage caused by pruning but also how much of that damage can be recovered through subsequent knowledge distillation. Unlike conventional pruning approaches that primarily optimize post-pruning quality, DAP explicitly targets post-distillation performance.

The authors formulate the pruning objective in terms of immediate damage and distillation recoverability. They derive a local second-order approximation involving reference and distillation curvature and introduce a scalable Kronecker-factored estimator with diagonalized output-gradient statistics.

Experiments on Qwen3-8B and DeepSeek-R1-Distill-Llama-8B evaluate the method under supervised fine-tuning (SFT) and on-policy distillation (OPD), with both unstructured and 2:4 semi-structured sparsity. DAP consistently outperforms conventional pruning baselines under matched sparsity and downstream training budgets. Additional experiments investigate the dependence of recoverability on the distillation procedure and teacher, as well as the limitations of the curvature approximation at higher sparsity levels.

Critical strengths
  1. A well-motivated reformulation of the pruning objective.

The main strength is the shift from minimizing immediate pruning damage to minimizing the damage remaining after distillation. This addresses a meaningful mismatch between conventional pruning objectives and the actual prune-then-distill pipeline. The proposed formulation is conceptually appealing and offers a principled motivation for incorporating downstream training dynamics into student construction.

  1. Consistent improvements supported by targeted mechanism-oriented experiments.

DAP achieves consistent improvements over established pruning baselines across different tasks, distillation procedures, and sparsity structures. For Qwen3-8B at 50% unstructured sparsity, it improves average post-distillation performance over the strongest baseline by 3.3 points under SFT and 3.5 points under OPD (Table 1).

More importantly, Tables 2 and 3 investigate whether the benefits depend on the intended distillation procedure and teacher. The matched-versus-mismatched comparisons provide meaningful evidence that the pruning mask's effectiveness is related to downstream distillation compatibility rather than simply producing a universally stronger sparse initialization. These experiments substantially strengthen the paper's central motivation.

Critical weaknesses
  1. The theoretical recovery approximation is not sufficiently validated for joint pruning.

The proposed recovery criterion is derived from a local, single-step approximation around the dense teacher, and its practical specialization considers removing one weight at a time (Section 4.1 and Appendix A.5). However, the actual pruning process removes a large fraction of weights simultaneously, and the method treats individual importance scores as approximately additive.

This introduces a potentially important discrepancy: the recovery term for a particular weight includes interactions with other parameters assumed to remain trainable, but many of those parameters may themselves be removed by the final pruning mask. Consequently, the estimated recoverability of an individual weight may not accurately represent its recoverability under the joint mask.

Furthermore, the surrogate models a local gradient update, whereas the actual distillation procedure involves extended AdamW optimization over 0.1B tokens. Although Figure 3 provides some empirical validation, the prediction breaks down at higher sparsity levels. More direct evidence connecting the proposed score to actual mask-level recoverability would strengthen the theoretical and empirical justification of the method.

  1. The construction of procedure-conditioned recovery statistics and the isolation of their contribution require further clarification.

A central claim is that recoverability depends on the intended distillation procedure, with this dependence entering through procedure-conditioned context distributions and sample weighting (Section 4.2 and Appendix A.4). However, the construction of OPD-conditioned calibration contexts is not sufficiently specified.

In particular, on-policy distillation normally involves student-generated sequences, whereas DAP must estimate recovery statistics before the final pruning mask and student have been constructed. It is unclear which model generates the OPD calibration rollouts, and how closely these contexts represent the actual trajectories encountered during downstream distillation.

Additionally, while the ablation with the recovery coefficient set to zero supports the usefulness of the recovery term, it does not fully isolate the benefit of the proposed cross-curvature interaction from simpler alternatives that use the same teacher-conditioned statistics. A stronger comparison with a teacher-aware curvature or gradient-based pruning criterion would make the claimed methodological contribution more convincing.

Questions for authors

Q1. Clarification of OPD-conditioned curvature estimation.

How exactly is the calibration distribution for OPD-conditioned recovery statistics constructed before the pruning mask is determined? Which model generates the autoregressive contexts: the dense teacher, a preliminary pruned student, or another proxy? If a proxy distribution is used, could the authors clarify why its estimated curvature should reflect the subsequent student-specific on-policy distillation dynamics?

This clarification is important because procedure-specific recoverability is one of the paper's central claims.

Q2. Validation of the joint-mask approximation.

How well does the proposed additive single-weight score predict actual post-distillation damage for complete pruning masks, particularly at 50% sparsity? Since the recovery calculation assumes that other parameters remain available for updating, what happens when many of those parameters are pruned simultaneously?

Could the authors provide a quantitative analysis of the relationship between predicted and measured mask-level remaining damage, or discuss whether accounting for interactions among jointly removed weights changes the resulting mask selection?

Q3. Comparison fairness and practical overhead.

Could the authors report the actual number of nonzero parameters in the pruned models, including the unpruned embeddings and LM head, to clarify the comparison with independently constructed dense 4B students?

It would also be helpful to report the pruning-stage computational overhead (e.g., GPU time and peak memory) relative to WANDA, WANDA++, and SparseGPT. Finally, is the improvement preserved against a simpler KD-conditioned saliency baseline using the same gradient statistics but without the proposed cross-curvature recovery term?

These results would help establish whether the additional complexity of DAP is justified by its practical benefits.

4 comments

Weakness 1 & Question 2

Joint-Mask Approximation and Recoverability

We thank the reviewer for highlighting the importance of validating recovery estimates under joint pruning. While Eq. (9) defines recovery for a complete pruning mask, Eq. (11) provides a scalable additive approximation for mask selection. We directly evaluate its predictive accuracy on complete masks, examine the benefit of joint-aware refinement, and validate its predictive value throughout extended distillation.

1. Complete-mask approximation and validation.

Eq. (9) naturally extends to a complete pruning mask. Let PP denote the pruned parameters and SS the surviving parameters. The joint-mask surrogate can be expressed as

I(P)=(1/2)uTAu−ηuTBCuI(P) = (1/2)u^T A u - η u^T B C u

where uu contains the pruned weights, AA is the reference curvature restricted to pruned parameters, BB couples pruned and surviving parameters under reference curvature, and CC couples surviving and pruned parameters under distillation curvature. Here, I(P)I(P) denotes predicted remaining damage.

Unlike the additive score in Eq. (11), this formulation restricts recovery to surviving parameters and retains interactions among jointly pruned weights.

We evaluate 15 complete masks at approximately 50% model-wide unstructured sparsity on Qwen3-8B: four conventional pruning baselines, seven DAP coefficient-sweep variants, and four independently calibrated DAP masks. All masks undergo identical 0.1B-token SFT.

For each mask, we measure final reference damage and recoverability (the reduction in reference damage during distillation). Predictions use shared frozen curvature statistics and a fixed recovery coefficient (η=1.5η0\eta=1.5\eta_0), independent of mask construction.

CriterionSpearman ρ (Final Damage)Spearman ρ (Recoverability)
Immediate damage only0.62—
Additive DAP0.780.55
Survivor-aware additive0.810.59
Joint-mask surrogate0.870.70

The practical additive DAP score substantially improves prediction of complete-mask outcomes over immediate damage alone (0.78 vs. 0.62). Restricting recovery to surviving parameters further increases the correlation to 0.81, while incorporating joint pruning interactions raises it to 0.87.

These results demonstrate that the proposed recovery criterion remains effective for ranking complete masks, with further improvements available through joint-aware modeling.

2. Does joint-aware modeling improve mask selection?

We additionally evaluate greedy joint-mask-aware refinement, initialized from the original DAP mask. The algorithm swaps pruned and surviving weights within the same output row when doing so decreases the joint surrogate, preserving the sparsity budget. Importantly, refinement uses only pre-distillation curvature statistics, without downstream evaluation feedback.

Under the same training budget, joint-aware refinement improves average post-SFT performance from 55.5 to 56.4 (+0.9 points).

This demonstrates that accounting for joint interactions can further improve mask selection, while the original additive DAP criterion provides an effective and computationally simpler starting point.

3. Predictive value throughout extended distillation.

To evaluate whether the local surrogate remains informative beyond its single-step derivation, we measure the correlation between the initial additive score and actual remaining reference damage throughout downstream AdamW training.

Training tokens10M50M100M
Spearman ρ0.830.810.78

The initial score maintains strong predictive correlations throughout training, including 0.78 after the full 0.1B-token SFT procedure. Thus, despite being derived from a local update, DAP provides useful predictions of complete-mask outcomes after extended optimization.

Together, these experiments establish the predictive utility of the practical additive criterion under simultaneous pruning, demonstrate the additional benefit of joint-aware refinement, and validate its relevance throughout downstream distillation. As noted in Fig. 3, the local approximation remains less reliable at more aggressive sparsities. We will incorporate these new analyses into the revised manuscript.

Weakness 2 & Question 1

OPD Calibration and Cross-Curvature Contribution

We thank the reviewer for these questions. We clarify how OPD-conditioned recovery statistics are constructed before mask selection and provide additional ablations isolating the contribution of cross-curvature recovery.

1. OPD-conditioned calibration. Since the final DAP student is unavailable during calibration, we construct a frozen sparse proxy using the reference-only criterion (η=0\eta=0) on 128 reference sequences. The proxy matches the final student's target sparsity and requires no distillation or recovery statistics, avoiding circular dependence on the final mask.

The proxy generates 128 OPD calibration trajectories from held-out prompts using the downstream sampling settings (temperature 0.7, top-pp 0.95, maximum 4,096 tokens, one rollout per prompt). At each autoregressive context, the dense teacher samples the targets, while gradient second moments are evaluated at the dense parent.

For SFT, we instead use fixed demonstration contexts. Both procedures use the same teacher-target curvature estimator, token weighting, and normalization, with procedure dependence captured by their different autoregressive context distributions.

Teacher-sampled targets yield a zero expected gradient at the dense parent, giving the resulting Fisher a local teacher-matching curvature interpretation on the fixed calibration distribution. Importantly, the sparse proxy provides student-like contexts at the target sparsity, approximating the initial distribution encountered during downstream OPD.

We further validate the choice of calibration policy under identical calibration and training budgets:

OPD calibration sourcePost-OPD Avg.
Dense-teacher rollouts58.9
Frozen sparse-proxy rollouts60.7

The +1.8-point improvement demonstrates the empirical benefit of using sparsity-matched, student-like contexts for procedure-conditioned mask selection.

2. Cross-curvature contribution. To isolate the contribution of cross-curvature recovery beyond teacher-conditioned information alone, we introduce two additional baselines: KD-only saliency and a tuned combination of reference and KD saliency. All methods use the same underlying curvature statistics, sparsity budget, and downstream OPD training configuration.

The combined baseline uses Si=(1−λ)S~iref+λS~iKDS_i=(1-\lambda)\widetilde S_i^{\mathrm{ref}}+\lambda\widetilde S_i^{\mathrm{KD}}, with layerwise mean normalization and λ∈{0,0.25,0.5,0.75,1}\lambda\in\{0,0.25,0.5,0.75,1\} tuned on held-out validation data. Neither baseline models cross-curvature recovery interactions.

Pruning criterionPost-OPD Avg.
Reference-only (η=0\eta=0)51.5
KD-only saliency54.4
Combined reference + KD saliency57.4
DAP (cross-curvature)60.7

DAP outperforms the tuned combined-saliency baseline by 3.3 points, demonstrating that its gains extend beyond simply incorporating or combining teacher-conditioned curvature statistics. Explicitly modeling cross-curvature recovery yields more effective masks for subsequent distillation.

We will clarify the sparse-proxy construction, procedure-conditioned context distributions, and curvature estimation in Appendix A.4, and incorporate these additional ablations into the revised manuscript.

Question 3

Comparison Fairness and Practical Overhead

We thank the reviewer for these suggestions. We provide effective parameter counts, measured pruning costs, and additional controls isolating the cross-curvature contribution.

1. Effective nonzero parameter counts and comparison fairness.

Our implementation already enforces matched effective nonzero-parameter budgets, explicitly accounting for unpruned embeddings, normalization layers, and the LM head. Rather than pruning a fixed 50% of eligible weights, we determine the required projection sparsity from the target student parameter budget.

Specifically, to obtain 4.022B nonzero parameters, matching the dense Qwen3-4B baseline, Qwen3-8B requires 60.01% sparsity within its prunable projections, corresponding to approximately 50.9% model-wide sparsity. For DeepSeek-R1-Distill-Llama-8B, achieving the same nonzero budget requires 57.42% projection sparsity, corresponding to approximately 49.9% model-wide sparsity.

Thus, the 50% sparsity reported in the main experiments refers to the approximate model-wide compression level, rather than the fraction of prunable projection weights removed. All unstructured pruning baselines use the same effective nonzero-parameter budget within each parent-model comparison, with identical downstream distillation configurations.

We will clarify this distinction in Section 5.1 and the corresponding tables, explicitly reporting the projection sparsity and effective nonzero-parameter counts to avoid ambiguity.

2. Pruning-stage computational overhead.

We benchmark all methods on H100 GPUs using the same calibration budget (256 sequences of 4,096 tokens). Conventional methods use one GPU; DAP uses two GPUs with FSDP2. We report wall-clock time, aggregate GPU-hours, and peak allocated memory per GPU, excluding downstream distillation.

MethodTime (s)GPU-hoursPeak Mem. (GB)
WANDA3000.08328
WANDA++4800.13332
SparseGPT7200.20030
DAP (diagonal-G)1,3800.76748
DAP (full-G)2,4001.33471

Diagonal-G substantially improves DAP's computational efficiency, reducing GPU-hours by 42.5% and peak memory by 32.4% relative to full-G, while retaining strong post-distillation performance (55.5 vs. 58.9). Importantly, even with this more economical approximation, DAP outperforms the strongest conventional pruning baseline by 3.3 post-SFT points (55.5 vs. 52.2).

These results highlight DAP's practical tradeoff: additional one-time computation during student construction yields a student better suited to subsequent distillation, while diagonal-G offers a more efficient alternative to the full-curvature variant.

3. Cross-curvature contribution.

To isolate DAP's recovery interaction, we introduce two baselines using identical reference/KD curvature statistics, sparsity, and downstream OPD training:

  • KD-only: KD-conditioned diagonal-Fisher saliency.
  • Ref+KD: A layerwise-normalized mixture of reference and KD saliencies, with the mixing coefficient selected from {0, 0.25, 0.5, 0.75, 1} on held-out validation data.

Neither baseline models cross-curvature recovery interactions.

Pruning criterionPost-OPD Avg.
Reference-only (eta = 0)51.5
KD-only54.4
Ref+KD57.4
DAP (cross-curvature)60.7

DAP outperforms the tuned Ref+KD baseline by 3.3 points, demonstrating that its gains extend beyond simply incorporating or combining KD-conditioned curvature statistics. Together, these results establish the additional benefit of modeling recovery interactions while quantifying the associated computational overhead.

Dear Reviewer r6eh:

We sincerely appreciate the time you spent reviewing our paper. If you have a chance to look at our rebuttal, we would be very grateful for any additional feedback or an updated assessment.

Reviewer_uDsoOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a novel approach, Distillation-Aware Pruning (DAP), for improving the teacher-student compatibility issue in LLM distillation. Existing distillation methods often suffer from a mismatch between the teacher and student models, leading to suboptimal performance. This work proposes to address the issue by pruning the teacher model to build the student model, preserving the knowledge of the teacher while reducing the model size. DAP further introduces a pruning criterion that estimates the remaining post-distillation damage by considering both immediate pruning damage and its recoverability. Comprehensive experiments demonstrate the effectiveness of DAP over existing pruning methods in terms of post-distillation performance.

Critical strengths
  1. This paper is well-motivated and addresses a significant problem in the field of LLM distillation. The authors propose to conduct LLM distillation with a pruned teacher model. To further improve the post-distillation performance, they estimate the damage caused by pruning and its recoverability. The core idea is aligned with my intuition and easy to follow.
  2. The results are impressive. The DAP demonstrates consistent improvements over existing pruning methods across various LLMs, different distillation methods, and multiple downstream tasks. Moreover, the experiments are well-designed to answer the proposed four research questions (effectiveness, mechanism, specificity, and scalability).
Critical weaknesses

Although the paper argues that DAP is an unified framework for distillation-aware pruning, it seems that the main experimental comparisons involve only conventional pruning methods. These methods are not designed for distillation, and thus the comparisons may not be fair. It would be more convincing to compare DAP with other LLM distillation methods (not necessarily pruning methods) to demonstrate its effectiveness in the context of distillation.

Questions for authors

Although the paper provides some theoretical analysis of the complexity, I am curious about the actual pruning time and computational cost of DAP. Compared to conventional distillation methods, the central contribution of DAP is to prune the teacher model to be the student model. Thus, if the pruning process is too time-consuming, it may not be practical for real-world applications.

3 comments

We thank the reviewer for the feedback. We clarify DAP's scope, provide additional distillation-aware pruning baselines, and report measured construction costs.


Weakness

Comparison with LLM Distillation Methods

1. Student construction versus distillation optimization.

DAP addresses a complementary problem to conventional LLM distillation methods. While SFT and OPD optimize knowledge transfer into a given student, DAP optimizes how the student is constructed before distillation, accounting for the intended teacher and downstream procedure.

Comparisons with Magnitude, SparseGPT, WANDA, and WANDA++ hold the dense parent, parameter budget, and downstream distillation configuration fixed within each pruning setting. This controlled setup isolates the effect of mask construction on post-distillation performance.

2. Comparison with independently constructed dense students.

Table 1 also evaluates an independently constructed dense Qwen3-4B student under the same downstream distillation procedures and training-token budgets, providing a non-pruning alternative.

Student constructionPost-SFT Avg.Post-OPD Avg.
Dense Qwen3-4B44.549.0
WANDA52.257.2
DAP55.560.7

DAP outperforms the independently constructed dense 4B student by 11.0 points under SFT and 11.7 under OPD, while exceeding the strongest conventional pruning baseline by 3.3 and 3.5 points, respectively. The unstructured comparison additionally matches effective nonzero-parameter budgets (approximately 4.022B), including unpruned parameters.

3. Additional distillation-aware pruning controls.

To further isolate the contribution of distillation-aware mask selection, we introduce KD-conditioned saliency baselines using the same reference and distillation curvature statistics as DAP, with identical sparsity and downstream OPD training budgets.

Pruning criterionPost-OPD Avg.
Reference-only saliency51.5
KD-only saliency54.4
Ref+KD saliency (tuned)57.4
DAP (cross-curvature)60.7

The Ref+KD baseline combines preservation and teacher-conditioned saliencies, with its mixing coefficient selected on held-out validation data. DAP exceeds this tuned distillation-aware baseline by 3.3 points, demonstrating the additional benefit of modeling cross-curvature recovery interactions rather than simply combining reference and KD importance scores.

Furthermore, Table 2 shows that matching DAP's mask construction to the intended SFT or OPD procedure improves post-distillation performance by 2.9 and 3.0 points, respectively, over mismatched construction.

Together, these results establish the value of distillation-aware student construction and procedure-conditioned recovery modeling, complementing existing distillation algorithms. We will incorporate the additional controls and clarify this distinction in the revised manuscript.


Question

Practical Pruning Cost

We benchmark pruning-stage costs on H100 GPUs using Qwen3-8B and a shared calibration budget of 256 sequences of 4,096 tokens. Measurements include calibration, curvature collection, mask construction, and applicable sparse-proxy generation overhead, excluding downstream distillation.

MethodTime (s)GPU-hoursPeak Mem./GPU
WANDA3000.08328 GB
WANDA++4800.13332 GB
SparseGPT7200.20030 GB
DAP (diagonal-G)1,3800.76748 GB
DAP (full-G)2,4001.33471 GB

Conventional baselines use one H100, while DAP uses two H100s with FSDP2; GPU-hours account for the different device counts. Diagonal-G substantially improves DAP's computational efficiency, reducing GPU-hours by 42.5% and peak memory per GPU by 32.4% relative to full-G. With this efficient approximation, DAP achieves 55.5 post-SFT and 60.7 post-OPD average performance, exceeding the strongest conventional pruning baseline by 3.3 and 3.5 points, respectively.

The additional computation is confined to the one-time student-construction stage, without increasing the computational requirements of the subsequent distillation procedure. These results demonstrate that diagonal-G DAP provides a practical quality–cost tradeoff, retaining substantial post-distillation improvements while reducing the resource requirements of the full-G formulation.

We will incorporate these measurements into the revised manuscript.

Dear Reviewer uDso:

Thank you once more for reviewing our paper. If you are able to take a look at our rebuttal, we would greatly appreciate any further feedback or an updated evaluation. Many thanks!

Thank you for your response to my previous questions. All my concerns have been addressed, so I will keep my score unchanged.

Reviewer_zPjaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Distillation-Aware Pruning (DAP), which selects pruning masks by estimating damage that would remain after later distillation, rather than the damage incurred immediately after pruning. The formulation separates immediate damage from post-distillation recovery and argues that optimizing pruning for post-distillation performance is beneficial. The authors derive a local curvature-based surrogate and provide an approximation using factorized activations and output-gradient second moments for the per-weight pruning criterion. They evaluate pruned students using supervised fine-tuning and on-policy distillation. On Qwen3-8B, DAP improves average post-distillation scores over the strongest conventional pruning baseline by ~3 percentage points across the reported 50% unstructured and 2:4 sparsity settings. Cross-procedure and cross-teacher experiments indicate that masks constructed for other settings underperform compared to masks constructed for the intended distillation procedure and teacher. The authors also perform analyses comparing predicted and observed post-distillation damage, showing that the approximation can degrade at higher sparsity. Overall, the work advances the idea that pruning decisions should be informed by the subsequent recovery process, supporting this with improved performance across several settings and model families.

Critical strengths

The paper introduces a well-motivated formulation of pruning that considers not just immediate post-pruning performance, but instead focuses on post-distillation performance, addressing a limitation of conventional pruning approaches. The empirical results demonstrate consistent improvements over established baselines across multiple tasks, distillation procedures, and sparsity structures. Furthermore, experiments showing that the best-performing pruning mask depends on the intended distillation procedure and teacher provide evidence that accounting for teacher–student compatibility is important for post-distillation performance.

Critical weaknesses

A central concern is the additive approximation used to construct the final pruning mask. Although DAP estimates recoverability through adjustments to surviving parameters, each weight is scored independently under a single-weight removal assumption. Consequently, the estimated recovery may rely on parameters that are also removed by the final mask, potentially leading to inaccurate estimates of joint recoverability, particularly at higher sparsity. While Section 5.3 evaluates the overall surrogate, it does not isolate the error introduced by this additive approximation, leaving an important aspect of the proposed pruning criterion insufficiently characterized.

Questions for authors

Q1. The proposed recoverability criterion is derived for removing individual weights, but the practical method uses an additive approximation to select masks containing many simultaneously removed parameters. Since recoverability depends on the surviving parameters, how does the method account for the possibility that parameters assumed available for recovery are themselves subsequently pruned? Have the authors evaluated iterative mask updates or compared predicted and observed recoverability for jointly pruned parameter groups?

Q2. Could the authors report pruning runtime and peak memory usage relative to the existing baselines, particularly for the full-G and diagonal-G variants? Given the 3.4-point performance difference reported in Table 4, these measurements would help establish whether the computational savings from diagonalization justify the reduction in post-distillation performance.

3 comments

We thank the reviewer for the constructive feedback. We provide complete-mask validation, joint-aware refinement results, and measured computational costs to clarify the accuracy and efficiency of DAP's practical approximation.


Weaknesses and Question 1

Joint-Mask Approximation and Recoverability

1. Joint-mask formulation and validation.

Eq. (9) applies to a complete mask, whereas Eq. (11) provides a computationally efficient additive approximation. Let PP denote pruned parameters, S=PcS=P^c surviving parameters, and uu the original values of pruned weights. The joint-mask surrogate is

Ijoint(P)=12u⊤Au−ηu⊤BCu,I_{\mathrm{joint}}(P)=\frac12 u^\top A u-\eta u^\top BC u,

where AA is the reference curvature over pruned parameters, BB couples pruned and surviving parameters under reference curvature, and CC couples surviving and pruned parameters under distillation curvature. Unlike the additive score, this formulation restricts recovery to surviving parameters and retains interactions among jointly pruned weights.

We evaluate 15 complete masks at approximately 50% model-wide unstructured sparsity on Qwen3-8B: four pruning baselines, seven DAP coefficient variants, and four independently calibrated DAP masks. All receive identical 0.1B-token SFT. We rescore every mask using shared frozen curvature statistics and a fixed recovery coefficient (η=1.5η0\eta=1.5\eta_0).

CriterionSpearman ρ\rho (Final Damage)
Immediate damage only0.62
Additive DAP0.78
Survivor-aware additive0.81
Joint-mask surrogate0.87

The practical additive DAP criterion achieves a correlation of 0.78 with measured post-distillation damage, compared with 0.62 for immediate damage alone. Restricting recovery to surviving parameters and incorporating joint interactions further improves the correlation to 0.81 and 0.87, respectively.

These results demonstrate that the scalable additive criterion provides a useful predictive signal for complete-mask outcomes, with further improvements available through joint-aware scoring.

2. Joint-mask-aware refinement.

We further evaluate greedy joint-mask-aware refinement initialized from the original DAP mask. The procedure swaps pruned and surviving weights within the same output row when doing so decreases the joint surrogate, preserving the sparsity budget without using downstream evaluation feedback.

Joint-aware refinement further improves post-SFT average performance from 55.5 to 56.4 (+0.9 points) under the same training budget. This shows that the additive criterion provides an effective initialization that can be further improved by accounting for joint interactions.

3. Predictive value after full distillation.

Importantly, the reported 0.78 correlation is measured after the full 0.1B-token SFT procedure, demonstrating that the local surrogate remains informative beyond its one-step derivation. As discussed in Section 5.3, its reliability is more limited at higher sparsity levels.


Question 2

Pruning Runtime and Peak Memory

We benchmark pruning-stage costs on H100 GPUs using the same calibration budget (256 sequences of 4,096 tokens). We report wall-clock time, aggregate GPU-hours, and peak allocated memory per GPU, excluding downstream distillation.

MethodTime (s)GPU-hoursPeak Mem. (GB)
WANDA3000.08328
WANDA++4800.13332
SparseGPT7200.20030
DAP (diagonal-G)1,3800.76748
DAP (full-G)2,4001.33471

Conventional baselines use one H100; DAP uses two H100s with FSDP2, as reflected in aggregate GPU-hours.

Diagonal-G reduces GPU-hours by 42.5% and peak per-GPU memory by 32.4% relative to full-G, while achieving a post-SFT average of 55.5, outperforming the strongest conventional pruning baseline by 3.3 points (55.5 vs. 52.2).

These results demonstrate that diagonal-G substantially reduces DAP's construction cost while preserving its post-distillation advantage over conventional pruning.

We will incorporate the complete-mask validation, refinement experiments, and computational measurements into the revised manuscript.

Dear Reviewer zPja:

Thank you again for your time reviewing our paper. If you have a chance to take a look at our rebuttal, we would greatly appreciate any additional feedback or updated assessment. Thank you!

Thank you for the clarifications.

Comments on the paper

1 comment

Dear Area Chair,

Thank you for coordinating the review process. We have carefully addressed all reviewer comments in our rebuttal and appreciate the reviewers' thoughtful feedback.

We remain available throughout the discussion period. If there are any additional questions, or if there are aspects of the paper or rebuttal that would benefit from further clarification, we would be happy to provide any additional information that may be helpful.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.