KAT-Prune: State-Anchored Path Correction for Training-Free Structured LLM Pruning

Large language models achieve strong performance but incur high parameter and computational costs, hindering efficient deployment. Model compression is therefore essential, and structured pruning is attractive because it removes complete units such as attention heads and MLP channels to produce smaller dense models. Many training-free methods estimate structural importance under the dense model and retain these preferences as pruning changes the model, implicitly assuming they remain reliable after the state changes. Yet pruning alters hidden representations, unit interactions, and structural sensitivities, so a proxy measured at an earlier mask can lose fidelity to the current state. We term this self-induced surrogate mismatch. We introduce KAT-Prune (KL-guided Activation-ranked Transport Pruning). It revisits pruning decisions and previously removed structures at the current sparse state while preserving an exact physical parameter budget, and accepts partial corrections only when they reduce empirical teacher KL, without fine-tuning or post-pruning weight compensation. We establish the feasibility of its exact-budget correction path and derive a sufficient condition for a partial correction to decrease the population joint KL objective when its predicted gain exceeds the error from structural interactions. Across six LLMs and three pruning levels, KAT-Prune achieves the lowest WikiText-2 perplexity in 11 of 18 settings and the highest seven-task average accuracy in 12 of 18 among compared pruning methods. Relative to the strongest competing baseline in each setting, it reduces perplexity by up to 12.5% and improves seven-task average accuracy by up to 3.44 percentage points.