acceptodds
Under review as a conference paper at ICLR 2027

KAT-Prune: State-Anchored Path Correction for Training-Free Structured LLM Pruning

Abstract

Large language models achieve strong performance but incur high parameter and computational costs, hindering efficient deployment. Model compression is therefore essential, and structured pruning is attractive because it removes complete units such as attention heads and MLP channels to produce smaller dense models. Many training-free methods estimate structural importance under the dense model and retain these preferences as pruning changes the model, implicitly assuming they remain reliable after the state changes. Yet pruning alters hidden representations, unit interactions, and structural sensitivities, so a proxy measured at an earlier mask can lose fidelity to the current state. We term this self-induced surrogate mismatch. We introduce KAT-Prune (KL-guided Activation-ranked Transport Pruning). It revisits pruning decisions and previously removed structures at the current sparse state while preserving an exact physical parameter budget, and accepts partial corrections only when they reduce empirical teacher KL, without fine-tuning or post-pruning weight compensation. We establish the feasibility of its exact-budget correction path and derive a sufficient condition for a partial correction to decrease the population joint KL objective when its predicted gain exceeds the error from structural interactions. Across six LLMs and three pruning levels, KAT-Prune achieves the lowest WikiText-2 perplexity in 11 of 18 settings and the highest seven-task average accuracy in 12 of 18 among compared pruning methods. Relative to the strongest competing baseline in each setting, it reduces perplexity by up to 12.5% and improves seven-task average accuracy by up to 3.44 percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Structured Pruning,Training‑Free Model Compression,State‑Dependent Pruning,Self‑induced surrogate mismatch

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o86fOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the changing reliability of component-importance estimates during structured LLM pruning. KAT-Prune first constructs a pruned model, then repeatedly reassesses components in the current sparse state and proposes exchanges that restore some removed units while deleting others at the same physical parameter budget. It selects partial corrections using teacher KL, keeping pretrained weights fixed throughout mask search. Across six LLMs, the method improves perplexity and downstream accuracy in many settings, with its strongest results at moderate pruning levels. The paper also measures search cost and inference gains from materialized models.

Critical strengths
  1. Competitive results with intact baseline procedures. Evaluation covers six LLMs and three pruning levels, and baselines retain their native reconstruction or weight adjustments. KAT-Prune obtains the highest seven-task average in 12/18 settings without such updates.
  2. Concrete validation of iterative correction. Initialization and frozen-proxy controls, a separate proxy-fidelity probe, and physically materialized inference measurements provide useful evidence beyond calibration-set improvements alone
Critical weaknesses
  1. The quality–search-cost trade-off is insufficiently established. A Llama-2-7B run takes 62.67 GPU-hours on one RTX 3090, with proposal construction consuming 96.6%. There is no corresponding cost–quality comparison with the strongest baseline or cheaper correction variants, so the practical value of this expensive search remains unclear.
  2. Repeated refresh has limited support in the end-task ablation. In Table 3, freezing the proxy changes PPL only from 10.03 to 10.49 and 8.86 to 9.03; it actually improves Llama-3-8B Avg7 from 57.14 to 58.12. These two-model results support modest PPL gains, but do not establish that repeated rebuilding is necessary for the best downstream accuracy or worth its additional cost.
Questions for authors
  1. Do the existing run logs provide cost and quality for frozen-proxy correction or earlier stopping, measured on the same hardware as the 62.67-hour run?
  2. How should the central refresh claim be interpreted given Table 3's accuracy reversal? Please distinguish improved proxy fidelity, lower PPL, and improved downstream accuracy.
  3. Does the full-exchange ablation retain the same joint-KL accept/reject rule? Clarifying this would separate the benefit of partial updates from that of rejecting harmful proposals.
3 comments
AuthorsOpenReview ↗

Response to Reviewer o86f

Dear Reviewer o86f,

Thank you for your review. We address your concerns below.

W1 & Q1: Search cost and early stopping. All runs were conducted on a single NVIDIA RTX 3090.

Frozen-proxy correction. Both variants start from the same initialization and retain the same joint-KL acceptance rule at 30% pruning.

ModelVariantGPU-hours ↓PPL ↓Avg7 (%) ↑
Llama-3-8BFrozen proxy20.0410.4958.12
KAT-Prune53.5210.0357.14
Qwen3-8BFrozen proxy24.749.0354.97
KAT-Prune90.408.8655.94

Freezing the proxy reduces the search cost from 53.52 to 20.04 GPU-hours on Llama-3-8B and from 90.40 to 24.74 on Qwen3-8B. Refresh reduces PPL by 4.4% and 1.9%, respectively. The additional cost is confined to offline mask optimization, and the final pruned model incurs no additional inference overhead.

Early stopping. We report the per-round results of Llama-2-7B at 30% pruning, corresponding to the 62.67 GPU-hour run in Section 4.4.

RoundPPL ↓Avg7 (%) ↑
1 (Initialization)7.61556.66
27.48358.18
37.40760.23
47.29260.28
57.27658.37
67.24559.31
77.22260.33
87.21860.37

Pruning costs on Llama-2-7B at 30% pruning with the same 128 WikiText-2 calibration blocks:

MethodGPU-hours ↓PPL ↓Avg7 (%) ↑
Wanda-SP0.0469.8657.57
FLAP0.0568.1753.64
GRASPrune0.739.4252.18
Týr-the-Pruner34.087.3155.46
KAT-Prune (3 rounds)23.507.4160.23
KAT-Prune (8 rounds)62.677.2260.37

After three rounds, including initialization, KAT-Prune achieves 60.23% Avg7 and 7.407 PPL, compared with 60.37% and 7.218 after eight rounds. Stopping at round 3 reduces the search cost from 62.67 to 23.50 GPU-hours, saving 62.5% with a 0.14-point drop in Avg7. Týr requires 34.08 GPU-hours and achieves 55.46% Avg7 and 7.31 PPL under the same setting. Three-round KAT-Prune improves Avg7 over Týr by 4.77 points with a slightly higher PPL.

W2 & Q2: The role of proxy refresh. Proxy refresh addresses changes in structural importance during mask correction.

Proxy fidelity. In Table 5, as the pruning state changes, the Spearman correlation of the original proxy drops from 0.908 to 0.526 on Llama-2-7B and from 0.907 to 0.706 on Llama-3.2-3B. Rebuilding the proxy at the current sparse state raises these correlations to 0.894 and 0.952, respectively.

Perplexity. Repeated refreshing reduces PPL from 10.49 to 10.03 on Llama-3-8B and from 9.03 to 8.86 on Qwen3-8B.

Downstream accuracy. Teacher KL measures discrepancies across the output distribution. A smaller discrepancy does not necessarily preserve the ranking of task-specific candidate answers. Refreshing reduces Avg7 from 58.12% to 57.14% on Llama-3-8B and improves Avg7 from 54.97% to 55.94% on Qwen3-8B.

Joint-KL screening is limited to exchanges in the current proposal. Refreshing allows KAT-Prune to construct new correction directions using importance estimates at the current sparse state. The PPL gains are consistent with our objective of preserving the teacher's output distribution. The downstream benefits of repeated refresh are model-dependent. With refresh, KAT-Prune outperforms the strongest original baselines in Avg7 by 3.44 and 0.88 percentage points on Llama-3-8B and Qwen3-8B, respectively, and achieves the highest Avg7 in 10 of the 12 settings at 20% and 30% pruning.

Q3: Partial search and KL acceptance. The original Full Exchange ablation applies the entire proposal without the joint-KL acceptance rule. We added Full Exchange + KL Accept to isolate the contribution of partial search. All three variants use the same initial mask and proposal in the first correction round on Llama-3-8B at 30% pruning.

ConfigurationScreen KL ↓PPL ↓Avg7 (%) ↑
Full Exchange1.84427.8749.36
Full Exchange + KL Accept1.59729.4544.66
Partial Search + KL Accept (KAT)0.65210.5158.03

Applying the full proposal increases Screen KL from 1.597 to 1.844. Full Exchange + KL Accept therefore rejects the proposal and retains the initial mask. With the same acceptance rule, KAT-Prune finds a partial update that reduces Screen KL to 0.652, lowers PPL from 29.45 to 10.51, and improves Avg7 from 44.66% to 58.03%.

Full Exchange increases Avg7 to 49.36% despite the higher KL. KL acceptance prevents increases in empirical teacher KL on the calibration set. Partial search identifies beneficial corrections within a proposal that would otherwise be rejected. With the acceptance rule held fixed, partial search improves Avg7 by 13.37 percentage points.

I thank the authors for their detailed and constructive rebuttal. The additional experiments have addressed several of my concerns and strengthened the paper. I agree that the substantial offline search cost and the limited downstream benefits of repeated proxy refresh remain concerns. Nevertheless, I appreciate the authors' efforts to provide thorough experiments under limited computational resources. I have decided to raise my rating 2→32\to 3.

AuthorsOpenReview ↗

Response to Reviewer o86f

Dear Reviewer o86f,

Thank you for your thoughtful feedback and for reconsidering your rating. We appreciate your recognition of our additional experiments.

Regarding the remaining concern about offline search cost, we conducted an additional experiment on Llama-2-7B at 30% pruning, reducing the calibration set from 128 to 13 blocks without modifying the KAT-Prune algorithm.

Calibration blocksRoundsGPU-hours ↓PPL ↓Avg7 (%) ↑
128862.677.21860.37
1331.677.72060.10
1384.527.36159.91

With only 13 calibration blocks, the eight-round search cost decreases from 62.67 to 4.52 GPU-hours, corresponding to a 92.8% reduction, while PPL and Avg7 remain comparable to the original results. Even with a computational budget of only 1.67 GPU-hours, KAT-Prune achieves 60.10% Avg7 after three rounds, compared with 60.37% in the original eight-round setting.

These results provide further evidence that KAT-Prune can retain competitive downstream performance under a reduced computational budget. We will include these additional results in the revised manuscript and conduct a more comprehensive evaluation of the cost-quality trade-off with different calibration set sizes across models and pruning ratios.

Thank you again for your constructive comments.

Reviewer_CQC9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a training-free structured pruning method for LLMs that accounts for changes in structural importance during pruning. The authors argue that importance estimates from earlier pruning stages may become less reliable as the model structure changes. KAT-Prune addresses this issue by repeatedly updating activation rankings and conditional teacher KL measurements. It uses dynamic programming to propose a new pruning allocation and refines the pruning mask by restoring and removing structures while maintaining the same parameter budget. Partial updates are accepted only when they reduce empirical teacher KL, without modifying pretrained weights.

Critical strengths
  • The paper tackles a limitation of existing structured pruning methods, where the importance of individual components can change after other structures have been removed.
  • The proposed restore-delete mechanism allows the method to reconsider previously removed structures. Combining this mechanism with partial KL-based correction is an interesting approach to iterative pruning.
Critical weaknesses
  • Section 4.4 reports a pruning cost of 62 GPU-hours for Llama-2-7B at 30% pruning on an RTX 3090, with 96.6% of the time spent on proposal construction. Although the method does not require gradient-based fine-tuning, its extensive KL evaluations introduce substantial computational overhead. A comparison of pruning costs with existing methods would help clarify its practical efficiency.
  • Section 3.3 orders restore-delete exchanges using activation-based priorities. How sensitive are the results to this ordering? Comparisons with random ordering or KL-based greedy ordering under a similar computational budget would help clarify the effectiveness of the proposed search strategy.
  • The experiments only consider conventional dense Transformer architectures. It remains unclear whether KAT-Prune can be applied to models with linear attention or mixture-of-experts. Could the authors discuss the applicability of their method to these architectures?
  • GISP already performs iterative structured pruning by updating importance estimates as pruning progresses. Although KAT-Prune additionally allows previously removed structures to be restored under a fixed parameter budget, its practical advantages over GISP are not directly demonstrated. A direct comparison under consistent experimental settings would help establish the benefits of the proposed correction mechanism.
  • SlimLLM already employs a greedy search to refine attention head selection by exchanging heads selected for pruning with retained heads. Although KAT-Prune extends this idea through iterative mask correction under a fixed parameter budget, its advantages over SlimLLM are not directly demonstrated. A direct comparison would help clarify the benefits of the proposed approach.
Questions for authors
  • How does the substantial pruning cost compare with existing pruning methods under comparable experimental settings?
  • How sensitive are the results to the activation-based ordering of restore-delete exchanges?
  • What aspects of KAT-Prune would need to be modified to support linear-attention or mixture-of-experts architectures?
  • What additional benefits does KAT-Prune provide over GISP's iterative importance updates? Could the authors provide a direct comparison under consistent experimental settings?
  • How does KAT-Prune compare with SlimLLM's greedy head-selection strategy under matched pruning budgets and evaluation settings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer CQC9

Dear Reviewer CQC9,

Thank you for your review. We address your concerns below.

W1 & Q1: Pruning cost. We compared pruning costs for Llama-2-7B at 30% pruning on one RTX 3090 with the same 128 WikiText-2 calibration blocks.

MethodGPU-hours ↓PPL ↓Avg7 (%) ↑
Wanda-SP0.0469.8657.57
FLAP0.0568.1753.64
GRASPrune0.739.4252.18
Týr-the-Pruner34.087.3155.46
KAT-Prune (3 rounds)23.507.4160.23
KAT-Prune (8 rounds)62.677.2260.37

KAT-Prune requires more offline computation than one-shot baselines. Compared with Týr, it improves Avg7 by 4.91 points and reduces PPL from 7.31 to 7.22. Three rounds save 62.5% of the full cost with a 0.14-point drop in Avg7 and still outperform Týr in Avg7.

W2 & Q2: Exchange ordering. We fixed the initial mask, proposal and all 97,596 cost-neutral atoms in round 1 of correction on Llama-3-8B at 30% pruning. Activation ordering ranks atoms by refreshed activation percentiles. Random ordering uses the same prefix search with three seeds. Full KL-greedy search is computationally prohibitive because it requires evaluating every remaining atom at each step. We therefore use Batch KL-Greedy, generating four equal-cost candidate batches by cyclically shifting the activation-ordered remaining atoms at transport fractions 1/16, 1/8, 1/4, 1/2 and 3/4. At each stage, we select the batch with the lowest joint KL. We also evaluate the full proposal.

Exchange OrderingScreen KL ↓PPL ↓Avg7 (%) ↑
Activation Ordering0.65210.5158.03
Batch KL-Greedy0.66911.0056.70
Random (seed 0)0.90714.0748.99
Random (seed 1)0.95714.4652.33
Random (seed 2)0.86413.4653.05

Activation ordering improves Avg7 by 4.98–9.04 points over random and 1.33 over Batch KL-Greedy, and achieves the lowest Screen KL and PPL.

W3 & Q3: Other architectures. For GLA-1.3B, pruning a gated head would remove its Q/K/V dimensions, gating parameters, output-projection slices and recurrent state. MLP channel pruning could remain unchanged. For Mixtral-8x7B, each expert would have a channel mask coupling its gate, up and down projections. Exchanges could span experts and layers while keeping the top-2 router unchanged. Conditional KL, budget-constrained allocation and joint-KL acceptance could be retained under the new structural constraints. We will discuss both extensions in the revised manuscript.

W4 & Q4: GISP. GISP progressively removes structures and updates importance. Removed structures remain pruned. KAT-Prune reaches the target budget first and revisits earlier pruning decisions through cost-neutral restore–delete exchanges. We used GISP's official implementation with its original importance computation, iteration schedule and checkpoint selection, matching calibration data, removed-parameter budgets and evaluation settings.

W5 & Q5: SlimLLM. SlimLLM greedily exchanges heads within each layer using Pearson similarity and prunes FFN channels using feature-space importance. KAT-Prune instead searches partial exchanges across layers and structure types using joint teacher KL under a fixed budget, without weight recovery. We independently reproduced SlimLLM, including its non-uniform allocation and linear regression recovery, under matched pruning budgets and evaluation settings.

ModelSparsityKAT PPL ↓GISP PPL ↓SlimLLM PPL ↓KAT Avg7 ↑GISP Avg7 ↑SlimLLM Avg7 ↑
Llama-2-7B20%6.157.056.5262.4859.1763.78
30%7.229.517.8760.3752.9560.37
50%12.9228.3615.1346.5141.8750.43
Llama-3.2-3B20%9.9712.9512.0153.7953.0651.61
30%12.6419.0717.1949.7345.9546.68
50%23.3358.3762.6042.0341.6742.00
Llama-3-8B20%7.869.438.9060.4458.4061.57
30%10.0313.3811.9157.1451.7357.50
50%16.6534.6343.7940.9443.9540.13
Llama-3.1-8B20%7.839.708.7762.3661.4062.99
30%9.9813.8811.3454.3453.1058.11
50%16.5035.8543.3744.0343.6340.13
Qwen3-8B20%7.9212.3612.7564.2255.7055.79
30%8.8617.9324.3455.9447.4045.89
50%15.4056.05173.2542.6940.5335.02
Qwen2.5-14B20%6.4111.948.7867.6454.8261.05
30%8.1123.2612.6759.4648.6650.68
50%16.16180.2241.4045.9640.6738.91

Against GISP, KAT-Prune has lower PPL in 18/18 settings and higher Avg7 in 17/18. At 30%, Avg7 gains are 7.42, 8.54 and 10.80 points on Llama-2-7B, Qwen3-8B and Qwen2.5-14B.

Against SlimLLM, KAT-Prune has lower PPL in 18/18 settings and higher Avg7 in 11/18, with one tie. At 50%, Avg7 is higher in 5/6 settings. On both Qwen models, it improves both metrics at all ratios without weight recovery.

Dear Reviewer,

As the rebuttal is approaching to an end, would you mind letting the authors know if their rebuttal addressed your concerns? Thanks!

AC

AuthorsOpenReview ↗

Response to Reviewer CQC9

Dear Reviewer CQC9,

We would like to provide additional experimental results regarding W1 and Q1. We evaluated KAT-Prune on Llama-2-7B at 30% pruning using only 13 calibration blocks instead of 128, without changing the algorithm.

MethodGPU-hours ↓PPL ↓Avg7 (%) ↑
Wanda-SP0.0469.8657.57
FLAP0.0568.1753.64
GRASPrune0.739.4252.18
Týr-the-Pruner34.087.3155.46
KAT-Prune (128 blocks, 3 rounds)23.507.4160.23
KAT-Prune (128 blocks, 8 rounds)62.677.21860.37
KAT-Prune (13 blocks, 3 rounds)1.677.72060.10
KAT-Prune (13 blocks, 8 rounds)4.527.36159.91

Reducing the calibration set to 13 blocks decreases the eight-round search cost from 62.67 to 4.52 GPU-hours (92.8% reduction), with only a 0.143 increase in PPL and a 0.46 percentage point decrease in Avg7.

Compared with Týr using the original calibration setting, the eight-round configuration with 13 blocks achieves comparable PPL (7.361 vs. 7.31) and 4.45 percentage points higher Avg7, at substantially lower search cost (4.52 vs. 34.08 GPU-hours). With only three rounds, KAT-Prune achieves 60.10% Avg7 within 1.67 GPU-hours.

These results suggest that KAT-Prune can retain its accuracy advantage with substantially fewer calibration samples in this setting. We will include this comparison in the revised manuscript and further evaluate the effect of calibration set size on search cost and model quality across models and pruning ratios.

Reviewer_UUzsOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KAT-Prune to address “self-induced surrogate mismatch” during structured pruning. Under a fixed physical parameter budget, the method refreshes activations and conditional teacher KL measurements at the current sparse state, uses dynamic programming to generate an allocation proposal, and selects a partial update along a budget-neutral restore–delete exchange path. Pretrained weights remain unchanged, and only candidates that reduce joint teacher KL on the calibration set are accepted. The authors provide a proxy-transfer experiment, a sufficient condition for local KL descent, and evaluations across six models and three pruning levels.

Critical strengths
  1. The method is well motivated. The paper addresses changes in structural importance induced by pruning. Reassessing structures at the current sparse state provides a reasonable basis for revising earlier pruning decisions.

  2. Mask correction offers flexibility. Previously removed structures can be restored through equal-cost exchanges while preserving the parameter budget, allowing earlier selections to be reconsidered.

Critical weaknesses
  1. High search cost. Llama-2-7B at 30% pruning requires 62.67 GPU-hours on an RTX 3090. Matched baseline runtimes are missing, limiting assessment of whether the gains justify this cost.

  2. Limited downstream advantages at aggressive pruning. At 50% pruning, KAT-Prune achieves the best Avg7 on only two of six models. Lower perplexity does not consistently translate into higher accuracy.

  3. Mixed benefits from proxy refreshing. On Llama-3-8B, refreshing improves PPL from 10.49 to 10.03 but reduces Avg7 from 58.12% to 57.14%. Ablations cover only two models at 30% pruning.

Questions for authors
  1. How sensitive are results to exchange ordering? Can searching only path prefixes miss beneficial combinations?

  2. Do accepted updates satisfy the theoretical descent condition, or is the theorem primarily explanatory?

  3. Are the reported gains robust to different calibration sampling seeds and corpora?

3 comments
AuthorsOpenReview ↗

Response to Reviewer UUzs

Dear Reviewer UUzs,

Thank you for your review. We address your concerns below.

W1: Search cost. We compared pruning costs for Llama-2-7B at 30% pruning on one RTX 3090 using the same 128 WikiText-2 calibration blocks.

MethodGPU-hours ↓PPL ↓Avg7 (%) ↑
Wanda-SP0.0469.8657.57
FLAP0.0568.1753.64
GRASPrune0.739.4252.18
Týr-the-Pruner34.087.3155.46
KAT-Prune (3 rounds)23.507.4160.23
KAT-Prune (8 rounds)62.677.2260.37

KAT-Prune improves Avg7 by 4.91 percentage points over Týr and 2.80 over the strongest original baseline. Three rounds reach 60.23% Avg7 at this cost, with PPL of 7.41 versus 7.31 for Týr. The search is performed once and adds no inference overhead after physical pruning.

W2: High pruning ratios. Teacher KL measures distribution discrepancy and does not directly optimize downstream accuracy. At 50% pruning, KAT-Prune achieves the lowest PPL on four of six models among the original baselines and the best Avg7 on two. At 20% and 30%, it achieves the best Avg7 in 10 of 12 settings. We also compared GISP and SlimLLM at matched budgets. Results at 50%:

ModelKAT PPL ↓GISP PPL ↓SlimLLM PPL ↓KAT Avg7 ↑GISP Avg7 ↑SlimLLM Avg7 ↑
Llama-2-7B12.9228.3615.1346.5141.8750.43
Llama-3.2-3B23.3358.3762.6042.0341.6742.00
Llama-3-8B16.6534.6343.7940.9443.9540.13
Llama-3.1-8B16.5035.8543.3744.0343.6340.13
Qwen3-8B15.4056.05173.2542.6940.5335.02
Qwen2.5-14B16.16180.2241.4045.9640.6738.91

KAT-Prune achieves lower PPL on all six models and higher Avg7 on five of six against each method.

W3: Proxy refresh. Refreshing revises the correction direction as structural importance changes. Frozen proxy restricts corrections to the initial proposal. In Table 5, rebuilding the proxy raises Spearman correlation from 0.526 to 0.894 on Llama-2-7B and from 0.706 to 0.952 on Llama-3.2-3B.

In Table 3, refresh reduces PPL by 4.4% on Llama-3-8B and 1.9% on Qwen3-8B. Avg7 decreases by 0.98 points on the former and increases by 0.97 on the latter. Full KAT-Prune improves Avg7 over the strongest original baselines by 3.44 and 0.88 points, respectively. We will provide a more comprehensive evaluation of proxy refresh across models and pruning ratios in the revised manuscript.

Q1: Exchange ordering and prefix search. On Llama-3-8B at 30% pruning, we fixed the initial mask, proposal and all 97,596 cost-neutral atoms in the first correction round. Activation and random ordering use the same prefix search, with three random seeds. Batch KL-Greedy selects batches by joint teacher KL and explores non-prefix combinations. Activation and random ordering each evaluate up to 20 nonzero prefixes; Batch KL-Greedy evaluates up to 21 candidates.

Exchange OrderingScreen KL ↓PPL ↓Avg7 ↑
Activation Ordering0.65210.5158.03
Batch KL-Greedy0.66911.0056.70
Random (Seed 0)0.90714.0748.99
Random (Seed 1)0.95714.4652.33
Random (Seed 2)0.86413.4653.05

Activation ordering improves Avg7 by 4.98–9.04 points over random and 1.33 over Batch KL-Greedy. Prefix search can miss beneficial non-prefix combinations; searching arbitrary subsets of 97,596 atoms is prohibitive.

Q2: Theoretical descent. Theorem 1 gives a sufficient condition for population KL descent. Equation 9 bounds the interaction error quadratically in the transport distance. Estimating the curvature bound adds computation. We do not verify the sufficient condition for each accepted update. KAT-Prune evaluates joint teacher KL directly, accepting only strict empirical KL decreases. We will clarify the distinction between the theoretical condition and empirical descent in the revised manuscript.

Q3: Calibration robustness. We evaluated Llama-2-7B at 30% pruning with a different sampling seed and calibration corpus, and compared three baselines under both corpora.

MethodCalibrationPPL ↓Avg7 (%) ↑
Wanda-SPWikiText-29.86057.57
C415.55357.79
FLAPWikiText-28.17053.64
C49.83055.83
GRASPruneWikiText-29.42052.18
C412.35652.86
KAT-PruneWikiText-2 (seed 42)7.21860.37
WikiText-2 (seed 0)7.19160.42
C48.59160.95

Avg7 is stable across seeds 42 and 0: 60.37% and 60.42%. With C4 calibration, all four methods show higher WikiText-2 test PPL, and KAT-Prune has the smallest absolute increase. It retains the lowest PPL and highest Avg7 among the compared methods under both corpora. We will include a more detailed evaluation of KAT-Prune's robustness to different calibration corpora and sampling seeds in the revised manuscript.

I thank the authors for the additional experiments and clarifications. While the response addresses several points, two critical concerns remain unresolved:

  1. Offline Search Overhead: The computational cost remains a substantial practical bottleneck. As reported, even with only three rounds, KAT-Prune requires 23.50 hours on a single RTX 3090 to prune Llama-2-7B for a single target ratio. Requiring nearly a full day for one configuration limits the method's practical applicability.
  2. Cost-Benefit of Rebuilding the KL Table: While refreshing the table improves the proxy's ranking accuracy, the downstream results in Table 3 remain mixed compared to the frozen proxy. The current evidence fails to empirically justify whether rebuilding the table every round provides sufficient gains to warrant its cost, or whether less frequent refreshes could achieve comparable performance.
AuthorsOpenReview ↗

Response to Reviewer UUzs

Dear Reviewer UUzs,

Thank you for your follow-up comments. We conducted two additional experiments on Llama-2-7B at 30% pruning to address the remaining concerns.

1. Offline search overhead. We reduced the calibration set from 128 to 13 blocks while retaining the original KAT-Prune algorithm.

Calibration blocksRoundsGPU-hours ↓PPL ↓Avg7 (%) ↑
128862.677.21860.37
1331.677.72060.10
1384.527.36159.91

With only 13 calibration blocks, KAT-Prune reduces the full search cost from 62.67 to 4.52 GPU-hours while retaining comparable performance. Three rounds require only 1.67 GPU-hours and achieve 60.10% Avg7, compared with 60.37% in the original setting. These results suggest that a smaller calibration set can substantially reduce the search cost without changing the core algorithm.

2. Cost-benefit of rebuilding the KL table. Following your suggestion, we evaluated rebuilding the KL table every two rounds.

Refresh strategyRoundsPPL ↓Avg7 (%) ↑
Every round67.24559.31
Every round87.21860.37
Every two rounds67.22460.81

Rebuilding every two rounds achieves nearly identical PPL and slightly higher Avg7 than the original eight-round configuration.

In KAT-Prune, rebuilding the KL table improves the fidelity of individual group measurements at the current mask, but the resulting allocation remains an additive approximation that does not fully capture cross-group interactions. A more accurate conditional proxy therefore does not necessarily lead to better joint search outcomes. Moreover, partial transport limits structural changes in each round, which may allow a recently constructed table to remain useful beyond a single correction.

Our original choice of rebuilding every round ensures that each proposal is conditioned on the current mask. The new results suggest that less frequent rebuilding can retain comparable performance in this setting. Together with the proxy-transfer results in Section 4.4, these findings support state-dependent remeasurement while suggesting that the refresh frequency offers additional flexibility in controlling the search cost.

We will include both experiments in the revised manuscript and provide a more comprehensive evaluation of calibration set sizes and refresh frequencies across models and pruning ratios.

Thank you again for your suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.