- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the changing reliability of component-importance estimates during structured LLM pruning. KAT-Prune first constructs a pruned model, then repeatedly reassesses components in the current sparse state and proposes exchanges that restore some removed units while deleting others at the same physical parameter budget. It selects partial corrections using teacher KL, keeping pretrained weights fixed throughout mask search. Across six LLMs, the method improves perplexity and downstream accuracy in many settings, with its strongest results at moderate pruning levels. The paper also measures search cost and inference gains from materialized models.
Critical strengths
- Competitive results with intact baseline procedures. Evaluation covers six LLMs and three pruning levels, and baselines retain their native reconstruction or weight adjustments. KAT-Prune obtains the highest seven-task average in 12/18 settings without such updates.
- Concrete validation of iterative correction. Initialization and frozen-proxy controls, a separate proxy-fidelity probe, and physically materialized inference measurements provide useful evidence beyond calibration-set improvements alone
Critical weaknesses
- The quality–search-cost trade-off is insufficiently established. A Llama-2-7B run takes 62.67 GPU-hours on one RTX 3090, with proposal construction consuming 96.6%. There is no corresponding cost–quality comparison with the strongest baseline or cheaper correction variants, so the practical value of this expensive search remains unclear.
- Repeated refresh has limited support in the end-task ablation. In Table 3, freezing the proxy changes PPL only from 10.03 to 10.49 and 8.86 to 9.03; it actually improves Llama-3-8B Avg7 from 57.14 to 58.12. These two-model results support modest PPL gains, but do not establish that repeated rebuilding is necessary for the best downstream accuracy or worth its additional cost.
Questions for authors
- Do the existing run logs provide cost and quality for frozen-proxy correction or earlier stopping, measured on the same hardware as the 62.67-hour run?
- How should the central refresh claim be interpreted given Table 3's accuracy reversal? Please distinguish improved proxy fidelity, lower PPL, and improved downstream accuracy.
- Does the full-exchange ablation retain the same joint-KL accept/reject rule? Clarifying this would separate the benefit of partial updates from that of rejecting harmful proposals.
3 comments
I thank the authors for their detailed and constructive rebuttal. The additional experiments have addressed several of my concerns and strengthened the paper. I agree that the substantial offline search cost and the limited downstream benefits of repeated proxy refresh remain concerns. Nevertheless, I appreciate the authors' efforts to provide thorough experiments under limited computational resources. I have decided to raise my rating .
Response to Reviewer o86f
Dear Reviewer o86f,
Thank you for your thoughtful feedback and for reconsidering your rating. We appreciate your recognition of our additional experiments.
Regarding the remaining concern about offline search cost, we conducted an additional experiment on Llama-2-7B at 30% pruning, reducing the calibration set from 128 to 13 blocks without modifying the KAT-Prune algorithm.
| Calibration blocks | Rounds | GPU-hours ↓ | PPL ↓ | Avg7 (%) ↑ |
|---|---|---|---|---|
| 128 | 8 | 62.67 | 7.218 | 60.37 |
| 13 | 3 | 1.67 | 7.720 | 60.10 |
| 13 | 8 | 4.52 | 7.361 | 59.91 |
With only 13 calibration blocks, the eight-round search cost decreases from 62.67 to 4.52 GPU-hours, corresponding to a 92.8% reduction, while PPL and Avg7 remain comparable to the original results. Even with a computational budget of only 1.67 GPU-hours, KAT-Prune achieves 60.10% Avg7 after three rounds, compared with 60.37% in the original eight-round setting.
These results provide further evidence that KAT-Prune can retain competitive downstream performance under a reduced computational budget. We will include these additional results in the revised manuscript and conduct a more comprehensive evaluation of the cost-quality trade-off with different calibration set sizes across models and pruning ratios.
Thank you again for your constructive comments.
Response to Reviewer o86f
Dear Reviewer o86f,
Thank you for your review. We address your concerns below.
W1 & Q1: Search cost and early stopping. All runs were conducted on a single NVIDIA RTX 3090.
Frozen-proxy correction. Both variants start from the same initialization and retain the same joint-KL acceptance rule at 30% pruning.
Freezing the proxy reduces the search cost from 53.52 to 20.04 GPU-hours on Llama-3-8B and from 90.40 to 24.74 on Qwen3-8B. Refresh reduces PPL by 4.4% and 1.9%, respectively. The additional cost is confined to offline mask optimization, and the final pruned model incurs no additional inference overhead.
Early stopping. We report the per-round results of Llama-2-7B at 30% pruning, corresponding to the 62.67 GPU-hour run in Section 4.4.
Pruning costs on Llama-2-7B at 30% pruning with the same 128 WikiText-2 calibration blocks:
After three rounds, including initialization, KAT-Prune achieves 60.23% Avg7 and 7.407 PPL, compared with 60.37% and 7.218 after eight rounds. Stopping at round 3 reduces the search cost from 62.67 to 23.50 GPU-hours, saving 62.5% with a 0.14-point drop in Avg7. Týr requires 34.08 GPU-hours and achieves 55.46% Avg7 and 7.31 PPL under the same setting. Three-round KAT-Prune improves Avg7 over Týr by 4.77 points with a slightly higher PPL.
W2 & Q2: The role of proxy refresh. Proxy refresh addresses changes in structural importance during mask correction.
Proxy fidelity. In Table 5, as the pruning state changes, the Spearman correlation of the original proxy drops from 0.908 to 0.526 on Llama-2-7B and from 0.907 to 0.706 on Llama-3.2-3B. Rebuilding the proxy at the current sparse state raises these correlations to 0.894 and 0.952, respectively.
Perplexity. Repeated refreshing reduces PPL from 10.49 to 10.03 on Llama-3-8B and from 9.03 to 8.86 on Qwen3-8B.
Downstream accuracy. Teacher KL measures discrepancies across the output distribution. A smaller discrepancy does not necessarily preserve the ranking of task-specific candidate answers. Refreshing reduces Avg7 from 58.12% to 57.14% on Llama-3-8B and improves Avg7 from 54.97% to 55.94% on Qwen3-8B.
Joint-KL screening is limited to exchanges in the current proposal. Refreshing allows KAT-Prune to construct new correction directions using importance estimates at the current sparse state. The PPL gains are consistent with our objective of preserving the teacher's output distribution. The downstream benefits of repeated refresh are model-dependent. With refresh, KAT-Prune outperforms the strongest original baselines in Avg7 by 3.44 and 0.88 percentage points on Llama-3-8B and Qwen3-8B, respectively, and achieves the highest Avg7 in 10 of the 12 settings at 20% and 30% pruning.
Q3: Partial search and KL acceptance. The original Full Exchange ablation applies the entire proposal without the joint-KL acceptance rule. We added Full Exchange + KL Accept to isolate the contribution of partial search. All three variants use the same initial mask and proposal in the first correction round on Llama-3-8B at 30% pruning.
Applying the full proposal increases Screen KL from 1.597 to 1.844. Full Exchange + KL Accept therefore rejects the proposal and retains the initial mask. With the same acceptance rule, KAT-Prune finds a partial update that reduces Screen KL to 0.652, lowers PPL from 29.45 to 10.51, and improves Avg7 from 44.66% to 58.03%.
Full Exchange increases Avg7 to 49.36% despite the higher KL. KL acceptance prevents increases in empirical teacher KL on the calibration set. Partial search identifies beneficial corrections within a proposal that would otherwise be rejected. With the acceptance rule held fixed, partial search improves Avg7 by 13.37 percentage points.