acceptodds
Withdrawn from ICLR 2027

Beyond Reconstruction: Auditing Sparse Features for Physical Channel Pruning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Sparse autoencoders (SAEs) describe activations, but structured pruning must choose physical channels. We audit whether SAE features improve that decision in Qwen3-4B. Matched SAE and TopK-PCA representations generate fixed-width masks with the same joint search; statistical selectors receive common recovery rules. At 35.53% MLP-channel removal, SAE improves seven-task mean accuracy by 0.285 points and lowers WikiText perplexity by 0.834 versus RMS across three calibration seeds. PCA has comparable accuracy (49.52% versus 49.50% for SAE). Larger SAEs trained on over one million activation positions improve reconstruction and some local risk rankings, but neither greater sparsity nor active decoder subspaces consistently beat direct-gradient deletion scores. Common mean recovery changes rankings; FLAP-score and RCPU-score adaptations exceed SAE accuracy in the measured setting. A separate larger-calibration audit favors SAE on perplexity but favors a random dictionary on task accuracy. These results identify a limited reconstruction-proxy benefit without establishing SAE-specific functional preservation or superior physical deletion decisions.

Primary area:
interpretability and explainable AI
Keywords:
structured pruning, sparse autoencoders, large language models, channel selection, reconstruction, recovery, model compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hTZqOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits whether sparse autoencoder representations improve physical MLP-channel pruning in Qwen3-4B. A reconstructed joint deletion-energy proxy, supplemented by a diagonal residual penalty, guides fixed-budget swaps from an RMS initialization. Matched TopK-PCA controls share the search procedure, while statistical selectors are compared under common recovery rules. Experiments include complete benchmark evaluation, local deletion-risk diagnostics, larger SAE-training studies, recovery comparisons, and an exploratory arithmetic-response objective. The results show modest perplexity improvements in some settings without establishing an SAE-specific accuracy or functional-preservation advantage.

Critical strengths

(1) The paper makes a useful methodological distinction between reconstructing activations and selecting removable physical channels. Matching retained widths, initialization, search limits, and recovery rules provides a substantially more informative comparison than contrasting unrelated pruning systems.

(2) The reporting is unusually transparent about adverse outcomes and uncertainty. The manuscript retains random-dictionary wins, recovery-induced ranking reversals, numerical sensitivity, and unsuccessful response-objective experiments. It also explicitly distinguishes conditional bootstrap uncertainty from calibration-seed uncertainty and verifies physical export for selected masks.

Critical weaknesses

(1) The end-to-end evidence is too restricted to make the audit sufficiently representative of SAE-guided pruning.

The main all-layer comparison in Section 3.2 uses a signed TopK dictionary with 512 features, 1,632 fitting activations, and 500 optimization steps for a 9,728-dimensional physical activation space. The manuscript explicitly acknowledges that this is an undercomplete learned representation rather than a semantically validated SAE. This is a legitimate experimental object, but it provides a narrow test of whether learned sparse features can improve physical deletion decisions.

The subsequent studies partially address training scale without closing this limitation. Appendix S expands the complete-mask comparison to a larger dictionary and calibration set, but remains a single-seed campaign. The million-position, longer-training experiments in Appendix B cover only three layers and local deletion-risk outcomes; they do not produce corresponding complete masks or standard-task evaluations. Consequently, the paper never combines its strongest representation-training regime with its primary end-to-end pruning outcome.

The dense-autoencoder, PCA, empirical-moment, and random-dictionary controls in Appendix S are valuable. However, their single-seed outcomes do not establish how consistently the observed differences arise from representation learning, sparsity, reconstruction quality, or calibration variation. The three-seed main comparison covers a substantially smaller representation regime and does not include all these controls.

The authors appropriately restrict their conclusions to the measured setting. Nevertheless, this leaves the scientific contribution closer to a bounded case study of a particular reconstruction-based selector than a broadly informative audit of sparse-feature-guided pruning. A negative result can be valuable, but it needs a sufficiently representative test or a clearly isolated failure mechanism. The current study provides neither at the level needed to support a stronger general lesson.

(2) The local diagnostics do not resolve the decisions made by the actual pruning procedure, and the search restrictions leave alternative explanations open.

Section 5.2 evaluates deletion of fixed 64-channel groups from individual layers of the dense model. The final pruning procedure instead removes channels across all layers and evaluates swaps within an already pruned mask. These are materially different interventions: upstream activation drift, interactions with channels already removed, and restoration of retained channels can alter which action is beneficial. The manuscript correctly acknowledges this limitation, but it leaves the central representation-to-decision link unresolved.

The complete-mask optimizer also starts every method from the same RMS mask and allows at most 64 swaps per layer. Thus, at most 64 retained channel identities per layer can be replaced relative to the initializer. This substantially restricts how far a representation can change the pruning decision. Matching this restriction makes the comparison fair within the chosen solver, but does not establish whether similar SAE/PCA outcomes reflect representation quality or a shared optimization bottleneck.

Appendix D.4 includes useful development-only swap-budget sensitivities. However, these do not establish robustness of the final representation comparison to search budget, candidate-screening width, or the rank-64 output projection. The observed differences are small enough that these restrictions could materially affect their interpretation.

The strongest direct-gradient controls appear in local risk diagnostics rather than a matched complete-mask experiment. Therefore, the paper does not establish whether the local ranking advantage of direct gradients translates into better masks, or whether the failure of the SAE proxy arises primarily from its representation, residual approximation, projection, or interaction with the solver.

Finally, the broader component and recovery comparisons are largely single-seed and use repeatedly inspected public tests or development outcomes. The paper transparently labels these results exploratory. That disclosure is appropriate, but it does not supply independent confirmation of the resulting rankings or a stable explanation of their causes.

The narrow empirical findings are credible observations. Their decision-level and explanatory significance remains insufficiently established.

Questions for authors

(1) Do the principal complete-mask findings persist with larger, well-validated SAE dictionaries trained to a documented convergence criterion? Can the stronger training regime in Appendix B be connected to an all-layer pruning result, rather than only three-layer local diagnostics?

(2) How many swaps are actually accepted, how much do the final masks differ from the RMS initializer, and how much do SAE and PCA masks differ from each other? Are the final representation rankings stable when varying the swap cap, candidate-screening width, and output-projection rank?

(3) Do the proxy scores predict the actual damage of candidate swaps inside partially pruned models, including both restoration and deletion? How do those results compare with the dense single-layer group-deletion diagnostic in Section 5.2?

(4) Does a direct-gradient selector produce better complete masks under a matched candidate-generation and search protocol? Such a comparison would help determine whether its advantage in Table 4 translates into the paper’s primary outcome.

(5) Which conclusions from the larger component audit and recovery comparisons survive additional calibration seeds and an evaluation set not previously inspected during method development? In particular, how stable are the random-dictionary accuracy advantage and the SAE perplexity advantage?

(6) Can the effects of dictionary capacity, reconstruction quality, residual approximation, and optimization restrictions be separated within one consistent experimental regime? Which of these factors is supported as an explanation of the observed failure to improve physical pruning decisions?

Reviewer_aUr8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies structured pruning of MLP channels in an LLM (Qwen3-4B). In a gated SwiGLU MLP, deleting channel j means removing row j of the gate & up projections, plus column j of the down projection. In standard structured pruning methods, each channel is first scored on its own using calibration statistics. For example, RMS-based pruning computes the root mean square of each channel's activation over calibration tokens, then prunes/removes the lowest-scoring channels. However, scoring channels independently ignores how deleted channels interact: their contributions to the output can add up or cancel out. The damage of removing a set is therefore not simply the sum of its parts, but also depends on their interactions.

Based on this motivation (the need to model channel interactions when deciding what to prune), the proposed method instead scores the whole set of deleted channels jointly, estimating how much the layer output changes when they are removed together. To capture which channels co-activate, it relies on a per-layer sparse autoencoder (SAE). The SAE is first trained to map the 9728 channels to 512 features, keep the 32 largest in magnitude, and reconstruct its input. After training, the joint damage is estimated from the SAE's reconstructed activations, plus a per-channel penalty for what the SAE fails to reconstruct. Starting from the RMS mask, the method then performs a limited number of keep/drop swaps, accepting only those that lower the estimated damage.

Experiments prune 10%, 20%, and 35% of the channels in every layer of Qwen3-4B, using WikiText-2 calibration data. The pruned models are evaluated on 7 zero-shot commonsense benchmarks and on wikitext-2 perplexity. At the largest budget, the method improves slightly but consistently over RMS (about +0.3 accuracy points and −0.8 perplexity). On the other hand, it only matches the same procedure with PCA in place of the SAE in accuracy (49.50% vs. 49.52%), with marginally better perplexity (25.31 vs. 25.47), and simpler per-channel scores reach higher accuracy. Additional diagnostics also show that the estimated damage fails to rank channel groups near the pruning boundary, and that larger SAEs do not change this. As the authors acknowledge, the paper therefore does not show that SAEs are useful for pruning.

Critical strengths

The idea of using a sparse autoencoder to model cross-channel interactions during pruning is interesting, and it is evaluated fairly, with matched controls & a common protocol for all methods.

Critical weaknesses
  • No SAE-specific gains: the SAE brings only a small improvement over RMS, and this gain is not specific to sparse features. PCA and the exact empirical second moments perform just as well, an untrained random dictionary does better, and even the simpler FLAP/RCPU scores reach higher accuracy. These results therefore do not justify the use of SAEs and their features for pruning, which is the main contribution of the paper.

  • Missing prior work and weak baselines: the core idea is not new. Pruning channels by searching for the set whose removal least changes the layer's output is the approach of OSSCAR [1] which prunes the same FFN channels in LLMs]. None of these works is cited or compared. The included baselines are also weak: FLAP and RCPU are simplified to MLP-only scores, and "Wanda-sp" picks the same channels as RMS. Finally, the simple gradient score, the best predictor in the paper's own diagnostics, is never used for pruning.

  • Degenerate setting and weak evidence: without fine-tuning, accuracy collapses at the main budget, from 66.6 to about 49. RMS then scores exactly the BoolQ always-"yes" rate (62.17%), and ARC-c and OBQA are near chance. In this regime, the small gaps between methods are hard to interpret and are about as large as numerical noise. Additionally, the paper's scope is too narrow: one model, uniform widths, larger SAEs tested on only three layers, and no speed measurements.

  • Very hard to read: the writing is very dense, and it took me multiple reads to understand parts of the paper that should have been easy to describe and explain. The main text also relies heavily on the very long appendix, which forces constant back and forth between the two. The main message is further diluted by many side experiments that are not organized into a coherent flow. The 32-page appendix itself reads like a research log rather than a polished research paper. Finally, there are many inconsistencies throughout the paper and between the main text and the appendices, for example in how the SAE setup is described.

Questions for authors

Clarifications

  • Since the method only uses the SAE to estimate how channels co-activate, them main question for me is why use any SAE over simply computing these statistics directly from the data?
  • Is there any case where the SAE clearly does better than using the raw activation statistics?
  • Why use such a small SAE (512 features for 9728 channels), when standard SAEs are usually much larger?
  • How is the method different from OSSCAR?
  • Which SAE setup was used for each table?

Suggestions

  • I think the paper needs a full rewrite to make it clearer and better structured, with a focused main message and fewer side experiments.
  • As mentioned in the weaknesses, I think the paper needs stronger baselines, such as OSSCAR, the full FLAP and RCPU methods, and pruning with the gradient score.
  • I think larger SAEs should be tested on the full pruned model, not only in small diagnostics on a few layers.
  • It would also help to run the main experiments at lower pruning ratios, with fine-tuning after pruning, and on other models.
  • Since the paper is about physical pruning, it would be good to show actual speedups.
Reviewer_eqVYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether sparse autoencoder (SAE) features can improve the selection of MLP channels for actual removal in Qwen3-4B, comparing them with PCA, statistical scoring, random dictionaries, and gradient-based methods. The results show that better reconstruction does not consistently lead to better pruning decisions, and post-pruning compensation can also change the relative ranking of methods.

Critical strengths
  1. The paper addresses a valuable question by directly evaluating actual channel removal rather than assuming that better reconstruction necessarily leads to better pruning. Matching the number of retained channels and separating channel selection from compensation help make the comparisons more interpretable.
  2. The comparisons are relatively comprehensive, and the negative results are informative. PCA, random dictionaries, gradient-based methods, larger training scales, and model export checks examine several alternative explanations. Transparently reporting results unfavorable to SAEs and numerical discrepancies is more valuable than selectively emphasizing improvements.
Critical weaknesses
  1. The conclusions should be limited to the tested reconstruction-based features. All dictionaries are undercomplete, and the quality of semantic features is not independently evaluated. Furthermore, the modest advantage over RMS reflects both representation learning and additional channel-swap search, so it cannot be attributed solely to sparse features.
  2. The local removal tests do not fully match the actual pruning procedure. The former evaluates fixed channel groups in a few layers of the unpruned model, whereas deployment involves swapping channels in an already pruned model. Reuse of the development set, changes in compensation settings, and small ranking differences depending on the export procedure further limit generalizability. Actual channel removal alone does not establish a speed advantage without throughput measurements.
Questions for authors

Please address the issues mentioned in the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.