- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper asks how pruning masks should be chosen when distillation comes afterward. DAP estimates post-distillation damage by subtracting a local recovery term from the immediate pruning damage. It combines reference and distillation-conditioned statistics, using layerwise Kronecker factorization and a diagonal output-gradient factor to keep the computation manageable. Experiments on Qwen3-8B and DeepSeek-R1-Distill-Llama-8B cover MATH-500, StrategyQA, and APPS under SFT and on-policy distillation (OPD). DAP beats the tested baselines at 50% unstructured and 2:4 sparsity. Teacher and procedure crossover experiments also test whether the preferred mask depends on the distillation setup.
Critical strengths
- The main contribution is making the downstream distillation procedure part of the pruning decision. Tables 2–3 show that a mask chosen for one teacher or procedure may be less suitable for another. I find these comparisons more persuasive than baseline gains alone, since they directly test the paper’s motivation.
- The gains hold across the main settings. In Table 1, DAP improves the three-task average over the strongest baseline by 3.3/3.5 points for unstructured pruning and 3.7/3.9 points for 2:4 pruning under SFT/OPD. Table 5 shows similar gains on the second model. The recovery-term ablation and coefficient sweep help explain the method, and the paper also reports where its prediction breaks down at high sparsity.
Critical weaknesses
- The procedure-conditioned calibration recipe is still unclear. Appendix A.4 specifies the teacher that samples targets and says gradients are evaluated through the dense parent. It does not clearly identify the policy generating OPD calibration contexts before mask selection, or the weighting used for SFT and OPD. These details matter because they determine how the score depends on the procedure. The training datasets and exact OPD loss are also missing from Appendix A.1, making Table 2 difficult to reproduce.
- The diagonal approximation’s practical savings are not measured. Table 4 shows the score falling from 58.9 with full-G to 55.5 with diagonal-G. The storage-complexity reduction is clear, but construction time and peak memory would show what this 3.4-point tradeoff buys in practice. Equal calibration-token budgets do not imply equal mask-construction costs.
Questions for authors
- Which policy generates the OPD calibration contexts before the final mask is selected? Please distinguish it from the target-sampling teacher in Appendix A.4 and specify the weighting used for SFT and OPD.
- Which datasets are used for calibration and downstream training, and what is the exact OPD training loss?
- What are the construction time and peak memory for full-G, diagonal-G, and the pruning baselines on the same hardware?
3 comments
Dear Reviewer y2Xu:
We appreciate your time reviewing our paper. Should you have a chance to review our rebuttal, we would welcome any further feedback or updated evaluation.
Thx for the response, it resolves my concern and I hold my score unchanged.
We thank the reviewer for the feedback. We clarify the calibration procedure, datasets, OPD objective, and pruning costs.
Q1 and Part of Weakness 1
OPD calibration. Since the final DAP mask is unavailable during calibration, we construct a frozen sparse proxy using the reference-only criterion (η=0) on 128 reference sequences. The proxy πproxy=πθT⊙z0 matches the final student's sparsity and requires no distillation or recovery statistics, avoiding circular dependence.
The proxy generates 128 OPD calibration trajectories from held-out prompts (temperature 0.7, top-p 0.95, maximum 4,096 tokens, one rollout per prompt). At each context τt, the dense teacher samples the target yt∼πθT(⋅∣τt), and auxiliary NLL gradients are evaluated at the dense parent θT.
Procedure-specific weighting. SFT uses fixed teacher-generated demonstration contexts. Both procedures use identical teacher-target curvature estimators, valid-token masking, sample weighting, and normalization. Following Appendix A.5.3, activation moments are averaged over valid invocations; output-gradient moments are summed within sequences and averaged over sequences. The difference arises from autoregressive context distributions, not the KL direction or rescaling.
Proxy validation. Under identical calibration and training budgets:
The +1.8-point improvement supports proxy-generated contexts for mask selection. The proxy approximates early student-generated contexts, not the evolving distribution throughout training.
Q2 and Part of Weakness 1
Datasets. We use MATH, StrategyQA, and APPS training splits, partitioned into disjoint calibration and downstream training subsets:
Calibration, training, and evaluation examples are disjoint.
OPD objective. We use an on-policy distillation formulation based on reverse KL (student || teacher) over student-generated contexts:
LOPD(θ)=Ex∼D,,o∼πθ(⋅∣x)[∣o∣1∑t=1∣o∣DKL(πθ(⋅∣τt)∥πθT(⋅∣τt))]
Here, τt=(x,o<t) denotes the student-generated context. The loss averages over valid response tokens, excluding prompt and padding positions.
The student generates one rollout per prompt (temperature 0.7, top-p 0.95, maximum 4,096 tokens), with the dense parent as the frozen teacher. This objective differs from the teacher-sampled auxiliary NLL used only for curvature estimation.
We will add these details to Appendix A.1.
Q3 and Weakness 2
We benchmark pruning costs on H100 GPUs using Qwen3-8B and 256 calibration sequences, excluding downstream distillation.
DAP uses two H100s with FSDP2, versus one for conventional baselines; GPU-hours account for this difference. Costs include calibration, curvature collection, mask construction, and applicable proxy-generation overhead.
With two GPUs, diagonal-G reduces GPU-hours by 42.5% and peak memory by 32.4% relative to full-G, at a cost of 3.4 post-SFT points (55.5 vs. 58.9). This quantifies the accuracy–efficiency tradeoff rather than assuming equal calibration budgets imply equal computational costs.
We will incorporate these clarifications and measurements into the revision.