- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits whether sparse autoencoder representations improve physical MLP-channel pruning in Qwen3-4B. A reconstructed joint deletion-energy proxy, supplemented by a diagonal residual penalty, guides fixed-budget swaps from an RMS initialization. Matched TopK-PCA controls share the search procedure, while statistical selectors are compared under common recovery rules. Experiments include complete benchmark evaluation, local deletion-risk diagnostics, larger SAE-training studies, recovery comparisons, and an exploratory arithmetic-response objective. The results show modest perplexity improvements in some settings without establishing an SAE-specific accuracy or functional-preservation advantage.
Critical strengths
(1) The paper makes a useful methodological distinction between reconstructing activations and selecting removable physical channels. Matching retained widths, initialization, search limits, and recovery rules provides a substantially more informative comparison than contrasting unrelated pruning systems.
(2) The reporting is unusually transparent about adverse outcomes and uncertainty. The manuscript retains random-dictionary wins, recovery-induced ranking reversals, numerical sensitivity, and unsuccessful response-objective experiments. It also explicitly distinguishes conditional bootstrap uncertainty from calibration-seed uncertainty and verifies physical export for selected masks.
Critical weaknesses
(1) The end-to-end evidence is too restricted to make the audit sufficiently representative of SAE-guided pruning.
The main all-layer comparison in Section 3.2 uses a signed TopK dictionary with 512 features, 1,632 fitting activations, and 500 optimization steps for a 9,728-dimensional physical activation space. The manuscript explicitly acknowledges that this is an undercomplete learned representation rather than a semantically validated SAE. This is a legitimate experimental object, but it provides a narrow test of whether learned sparse features can improve physical deletion decisions.
The subsequent studies partially address training scale without closing this limitation. Appendix S expands the complete-mask comparison to a larger dictionary and calibration set, but remains a single-seed campaign. The million-position, longer-training experiments in Appendix B cover only three layers and local deletion-risk outcomes; they do not produce corresponding complete masks or standard-task evaluations. Consequently, the paper never combines its strongest representation-training regime with its primary end-to-end pruning outcome.
The dense-autoencoder, PCA, empirical-moment, and random-dictionary controls in Appendix S are valuable. However, their single-seed outcomes do not establish how consistently the observed differences arise from representation learning, sparsity, reconstruction quality, or calibration variation. The three-seed main comparison covers a substantially smaller representation regime and does not include all these controls.
The authors appropriately restrict their conclusions to the measured setting. Nevertheless, this leaves the scientific contribution closer to a bounded case study of a particular reconstruction-based selector than a broadly informative audit of sparse-feature-guided pruning. A negative result can be valuable, but it needs a sufficiently representative test or a clearly isolated failure mechanism. The current study provides neither at the level needed to support a stronger general lesson.
(2) The local diagnostics do not resolve the decisions made by the actual pruning procedure, and the search restrictions leave alternative explanations open.
Section 5.2 evaluates deletion of fixed 64-channel groups from individual layers of the dense model. The final pruning procedure instead removes channels across all layers and evaluates swaps within an already pruned mask. These are materially different interventions: upstream activation drift, interactions with channels already removed, and restoration of retained channels can alter which action is beneficial. The manuscript correctly acknowledges this limitation, but it leaves the central representation-to-decision link unresolved.
The complete-mask optimizer also starts every method from the same RMS mask and allows at most 64 swaps per layer. Thus, at most 64 retained channel identities per layer can be replaced relative to the initializer. This substantially restricts how far a representation can change the pruning decision. Matching this restriction makes the comparison fair within the chosen solver, but does not establish whether similar SAE/PCA outcomes reflect representation quality or a shared optimization bottleneck.
Appendix D.4 includes useful development-only swap-budget sensitivities. However, these do not establish robustness of the final representation comparison to search budget, candidate-screening width, or the rank-64 output projection. The observed differences are small enough that these restrictions could materially affect their interpretation.
The strongest direct-gradient controls appear in local risk diagnostics rather than a matched complete-mask experiment. Therefore, the paper does not establish whether the local ranking advantage of direct gradients translates into better masks, or whether the failure of the SAE proxy arises primarily from its representation, residual approximation, projection, or interaction with the solver.
Finally, the broader component and recovery comparisons are largely single-seed and use repeatedly inspected public tests or development outcomes. The paper transparently labels these results exploratory. That disclosure is appropriate, but it does not supply independent confirmation of the resulting rankings or a stable explanation of their causes.
The narrow empirical findings are credible observations. Their decision-level and explanatory significance remains insufficiently established.
Questions for authors
(1) Do the principal complete-mask findings persist with larger, well-validated SAE dictionaries trained to a documented convergence criterion? Can the stronger training regime in Appendix B be connected to an all-layer pruning result, rather than only three-layer local diagnostics?
(2) How many swaps are actually accepted, how much do the final masks differ from the RMS initializer, and how much do SAE and PCA masks differ from each other? Are the final representation rankings stable when varying the swap cap, candidate-screening width, and output-projection rank?
(3) Do the proxy scores predict the actual damage of candidate swaps inside partially pruned models, including both restoration and deletion? How do those results compare with the dense single-layer group-deletion diagnostic in Section 5.2?
(4) Does a direct-gradient selector produce better complete masks under a matched candidate-generation and search protocol? Such a comparison would help determine whether its advantage in Table 4 translates into the paper’s primary outcome.
(5) Which conclusions from the larger component audit and recovery comparisons survive additional calibration seeds and an evaluation set not previously inspected during method development? In particular, how stable are the random-dictionary accuracy advantage and the SAE perplexity advantage?
(6) Can the effects of dictionary capacity, reconstruction quality, residual approximation, and optimization restrictions be separated within one consistent experimental regime? Which of these factors is supported as an explanation of the observed failure to improve physical pruning decisions?