SFDA: Sparse Feature Distribution Alignment for Post-Pruning Recovery Data-selection

Structured pruning is used to reduce the inference cost of large language models. However, this degrades their capabilities, which then requires recovery fine-tuning. Under a limited recovery-data budget, existing instruction-selection methods usually rank samples by difficulty, quality or semantic diversity while pruning-aware approaches may require access to both the original and pruned models. However, these criteria do not directly characterize the representation of the model, needing repair. We propose SFDA, a Sparse Feature Distribution Alignment framework that selects recovery data directly from the internal state of the pruned model. SFDA identifies response-salient hidden coordinates using loss gradients, encodes their activations with a TopK sparse autoencoder, and selects a fixed-size subset whose projected sparse-feature distribution matches that of the candidate corpus. SFDA optimizes the subset jointly and requires only the pruned checkpoint. Across different pruned LLM configurations, SFDA achieves the highest average reasoning accuracy using only 10% of Alpaca, outperforming the strongest competing selector by 2.36-4.36 percentage points.