- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SFDA, a data selection method for recovering LLM performance after structured pruning. The main idea is to select recovery examples based on the internal representations of the pruned model, rather than relying on generic data quality or difficulty scores. SFDA identifies response-salient hidden dimensions using gradients, encodes them with a TopK sparse autoencoder, and selects a subset whose projected feature distribution matches that of the full candidate dataset. The authors evaluate the method across several LLMs, pruning strategies, and downstream tasks. Using only 10% of Alpaca, SFDA consistently outperforms the evaluated data selection baselines in average reasoning accuracy.
Critical strengths
1. The problem is well motivated. Selecting recovery data based on the state of the pruned model is a reasonable direction. The observation that pruning changes sample difficulty rankings provides useful motivation for this approach. The set-level selection perspective is also meaningful, since selecting individually difficult examples may introduce redundancy and fail to preserve broader recovery-relevant information. 2. The method has potential practical value. SFDA only requires access to the pruned model and uses a limited amount of recovery data. This could be useful in practical settings where the original model is unavailable or recovery training is expensive.
Critical weaknesses
1. The necessity of the proposed sparse representation and distribution matching objective is unclear. The method combines gradient-based feature selection, TopK SAE, PCA, and distribution matching, but the ablation results do not clearly justify this complexity. In particular, Table 6 shows that dense representations achieve nearly identical recovery NLL to TopK SAE. Similarly, Table 7 suggests that simpler objectives, such as covariance matching, perform comparably to the proposed Bhattacharyya–KS combination. These results raise the question of whether the improvements mainly come from using the pruned model's representations for data selection, rather than from the specific components proposed in SFDA. 2. The practical efficiency benefits are not sufficiently evaluated. Since pruning is primarily motivated by reducing inference costs, evaluating recovery accuracy alone does not fully establish the practical value of the framework. The paper does not provide sufficient end-to-end evidence of the trade-off between model compression, inference efficiency, and recovered performance. In addition, the proposed data selection pipeline introduces extra computational costs through gradient extraction, SAE training, and subset optimization. Without these measurements, it is difficult to assess whether the additional selection complexity is justified in practical deployment scenarios.
Questions for authors
1. Could the authors compare TopK SAE with dense representations on the main downstream reasoning benchmarks? The current NLL results in Table 6 do not show a clear advantage for sparse representations.
2. Could the authors evaluate the proposed distribution matching objective against simpler alternatives, particularly covariance matching, using downstream task accuracy rather than only recovery NLL?
3. Could the authors report actual inference latency, throughput, and memory usage before pruning and after pruning with SFDA recovery? This would help quantify the practical compression–performance trade-off.
4. Could the authors compare the total computational cost of SFDA, including gradient extraction, SAE training, and subset selection, against simpler recovery data selection methods? It would be useful to understand whether the additional selection cost is justified by the resulting performance gains.