acceptodds
Under review as a conference paper at ICLR 2027

SFDA: Sparse Feature Distribution Alignment for Post-Pruning Recovery Data-selection

Abstract

Structured pruning is used to reduce the inference cost of large language models. However, this degrades their capabilities, which then requires recovery fine-tuning. Under a limited recovery-data budget, existing instruction-selection methods usually rank samples by difficulty, quality or semantic diversity while pruning-aware approaches may require access to both the original and pruned models. However, these criteria do not directly characterize the representation of the model, needing repair. We propose SFDA, a Sparse Feature Distribution Alignment framework that selects recovery data directly from the internal state of the pruned model. SFDA identifies response-salient hidden coordinates using loss gradients, encodes their activations with a TopK sparse autoencoder, and selects a fixed-size subset whose projected sparse-feature distribution matches that of the candidate corpus. SFDA optimizes the subset jointly and requires only the pruned checkpoint. Across different pruned LLM configurations, SFDA achieves the highest average reasoning accuracy using only 10% of Alpaca, outperforming the strongest competing selector by 2.36-4.36 percentage points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
data selection, pruning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4BHUOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SFDA, a data selection method for recovering LLM performance after structured pruning. The main idea is to select recovery examples based on the internal representations of the pruned model, rather than relying on generic data quality or difficulty scores. SFDA identifies response-salient hidden dimensions using gradients, encodes them with a TopK sparse autoencoder, and selects a subset whose projected feature distribution matches that of the full candidate dataset. The authors evaluate the method across several LLMs, pruning strategies, and downstream tasks. Using only 10% of Alpaca, SFDA consistently outperforms the evaluated data selection baselines in average reasoning accuracy.

Critical strengths

1. The problem is well motivated. Selecting recovery data based on the state of the pruned model is a reasonable direction. The observation that pruning changes sample difficulty rankings provides useful motivation for this approach. The set-level selection perspective is also meaningful, since selecting individually difficult examples may introduce redundancy and fail to preserve broader recovery-relevant information. 2. The method has potential practical value. SFDA only requires access to the pruned model and uses a limited amount of recovery data. This could be useful in practical settings where the original model is unavailable or recovery training is expensive.

Critical weaknesses

1. The necessity of the proposed sparse representation and distribution matching objective is unclear. The method combines gradient-based feature selection, TopK SAE, PCA, and distribution matching, but the ablation results do not clearly justify this complexity. In particular, Table 6 shows that dense representations achieve nearly identical recovery NLL to TopK SAE. Similarly, Table 7 suggests that simpler objectives, such as covariance matching, perform comparably to the proposed Bhattacharyya–KS combination. These results raise the question of whether the improvements mainly come from using the pruned model's representations for data selection, rather than from the specific components proposed in SFDA. 2. The practical efficiency benefits are not sufficiently evaluated. Since pruning is primarily motivated by reducing inference costs, evaluating recovery accuracy alone does not fully establish the practical value of the framework. The paper does not provide sufficient end-to-end evidence of the trade-off between model compression, inference efficiency, and recovered performance. In addition, the proposed data selection pipeline introduces extra computational costs through gradient extraction, SAE training, and subset optimization. Without these measurements, it is difficult to assess whether the additional selection complexity is justified in practical deployment scenarios.

Questions for authors

1. Could the authors compare TopK SAE with dense representations on the main downstream reasoning benchmarks? The current NLL results in Table 6 do not show a clear advantage for sparse representations.

2. Could the authors evaluate the proposed distribution matching objective against simpler alternatives, particularly covariance matching, using downstream task accuracy rather than only recovery NLL?

3. Could the authors report actual inference latency, throughput, and memory usage before pruning and after pruning with SFDA recovery? This would help quantify the practical compression–performance trade-off.

4. Could the authors compare the total computational cost of SFDA, including gradient extraction, SAE training, and subset selection, against simpler recovery data selection methods? It would be useful to understand whether the additional selection cost is justified by the resulting performance gains.

Reviewer_qJAmOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SFDA selects data for post-pruning recovery using representations from the pruned model. It identifies response-salient coordinates, encodes them with a sparse autoencoder, and selects a fixed-size subset by matching its projected feature distribution to the candidate corpus.

Critical strengths

The paper addresses a useful problem and reports consistent average downstream improvements across several models. Comparisons include relevant selectors such as PASER and MONA, with additional evaluations across pruning methods, coding and mathematics tasks. The multi-seed results and matched training-budget controls strengthen the empirical evaluation.

Critical weaknesses
  • The gains are promising, but the contribution of the main components remains unclear. In Tables 6 and 7, simpler representations and matching objectives achieve comparable recovery NLL. These controls leave open how much the sparse representation and combined objective contribute to the downstream gains. Evaluating the key simplified variants on the main benchmarks would help establish what is worth retaining in the pipeline.

  • Some results are difficult to reconcile from the reported settings. Table 1 and Figure 6(b) describe the same target model and pruning ratio, yet show substantially different results at comparable data budgets. The recovery configuration also needs more detail, including the training method, learning rate, and schedule. Please clarify which settings differ across these experiments.

  • The overall efficiency advantage needs a more direct comparison. The paper usefully reports curation time, but selection requires processing the candidate corpus before recovery. A comparison that includes both selection and training would clarify whether SFDA is preferable to spending that time on additional recovery.

  • Presentation needs attention. Table 1 is unnecessarily cramped, with small text and very tight row spacing. Please split or reformat it. The manuscript also retains the ICLR 2026 header throughout.

I am open to revisiting my score if the authors clarify the experimental discrepancies and show that the proposed design provides a meaningful benefit over the simpler variants already considered. I think this is an interesting research direction and the results look good; it just needs more revision and more clarification at the current stage.

Questions for authors
  • How do direct pruned-model representations and a simpler matching objective compare with SFDA on the main downstream tasks under the same recovery setup?

  • What explains the discrepancy between Table 1 and Figure 6(b), and which recovery settings are shared across the reported experiments?

Reviewer_npQ8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SFDA for selecting recovery data after LLM pruning. It uses gradient saliency to find important hidden features, then a sparse autoencoder and distribution matching to select training samples. The method only needs the pruned model. Experiments on LLaMA and Qwen show consistent gains over other data selection methods, using only 10% of Alpaca data.

Critical strengths
  • The motivation makes sense. Pruning changes the model representations, so selecting recovery data based on the pruned model itself is more reasonable than using generic data quality or difficulty scores

  • Experiments are quite comprehensive, with different models, pruning methods and recovery settings. The 2.36 to 4.36 point improvements are pretty convincing, especially with the matched training budget experiments.

Critical weaknesses

1, The link between sparse feature distribution matching and actual recovery performance is still not very clear. Matching feature statistics doesn't necessarily mean the selected samples can recover the capabilities lost during pruning

2, The method involves several steps, including gradient computation, SAE training, PCA and subset optimization. Not clear if the performance gains justify these extra costs compared to simpler data selection methods.

Questions for authors
  • What is the total computation cost for data selection, including SAE training? How does it compare to other methods when considering the full recovery pipeline?

  • Is SAE really necessary here? Would matching dense feature distributions with similar dimensions and compute budget give comparable results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.