acceptodds
Under review as a conference paper at ICLR 2027

The Feature-Space Alignment Hypothesis for Neural Network Sparsity

Abstract

Why does something as simple as magnitude pruning retain accuracy even when most weights are removed, and why does performance eventually collapse at higher sparsities? This paper shows that the answer may lie in the ability to keep intact the representation of the original dense weight matrix when viewed in feature space, despite losing many weights to pruning. We study this question in a controlled setting where a linear embedding (representing prior layers) mixes input features across coordinates before classification, enabling direct analysis of the network's effective feature-space weights. We find that accuracy collapse under standard pruning coincides with divergence of the feature-space representation from its pre-pruning values. We then construct an optimization oracle that, given access to the embedding, selects a new sparse weight matrix, independent of the original, by directly preserving the effective feature-space matrix induced by the dense model. We show that under the same retraining budget, the oracle recovers performance at sparsity levels where standard pruning techniques degrade sharply, with consistent behavior across synthetic logic and more realistic tasks. Furthermore, we show that one can achieve oracle-level pruning performance even when the true embedding is not known by using a sparse autoencoder to recover its pruning-relevant structure, providing a path toward more general feature-aware sparsification. These results suggest a mechanistic explanation of sparsity and a guiding framework for feature-aware improvements in pruning methods.

Primary area:
interpretability and explainable AI
Keywords:
Neural Network Sparsity, Feature Representations, Oracle Analysis, Sparse Topology, Pruning, Mechanistic Interpretability, Sparse Autoencoders

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KzNbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes the feature-space alignment hypothesis, arguing that successful pruning should preserve the effective computation in feature space rather than remain close to the dense model in parameter space.

Critical strengths

The resulting feature-aware masks, especially when combined with OBS-style compensation in FLO, perform well at high sparsity. The SAE experiments are also interesting because they show that much of the oracle benefit can be recovered without direct access to the true embedding.

Critical weaknesses

The core argument relies on a specially constructed setting where the feature-space mapping is explicitly available. It is unclear how directly this notion extends to deep nonlinear networks or modern transformers. The additional experiments on Digits, UCI datasets, and deeper controlled networks are useful, but still relatively far from realistic large-scale models.

The proposed MIP objective is closely related to preserving the layer output on the embedded representation. It would be important to compare against a standard activation- or layer-output reconstruction objective. Such a comparison would clarify whether the gains specifically support the proposed feature-space interpretation, rather than the more general principle of preserving layer function. The MIP formulation is NP-hard and therefore serves mainly as an oracle. In addition, MIP alone is often comparable to OBS in the one-shot setting, while the strongest method is the hybrid FLO, which combines the feature-aware topology with OBS compensation.

Questions for authors

see weakness

4 comments

We would like to thank the reviewer for your thoughtful review and for engaging seriously with the paper’s main intuition. We also appreciate how you found our recovery of oracle benefit without the true embedding interesting. We address your concerns below:

  1. Regarding scale, we agree that establishing how broadly this mechanism applies to larger nonlinear networks and transformers remains an important direction for future work. Even in a small model, however, accuracy alone does not reveal whether a sparse model preserves the dense model’s learned feature structure or achieves similar performance by reorganizing its representation. The known DNF clauses let us examine this question directly: much of the codes persist in the same neurons and for the same clauses even after pruning removes 75% of the weights in W1W_1, while their degradation across sparsity levels tracks increasing loss (Figures 2, 3). Explicitly selecting sparse weights to reconstruct the effective map retains more of these codes, even though the optimizer never receives the clause assignments (Figure 7). At this scale, we can inspect how pruning interacts with computation and use MIP to test whether preserving that computation helps maintain accuracy. We have revised the paper to make this scientific motivation and the scope of our conclusions explicit (L53, L57, L63).

  2. Regarding preserving the layer outputs, we agree that the two objectives are closely related. Our objective for Equation 1 was a deliberate experimental choice, as our original motivation concerns distortion of the full effective map, while the layer output reconstruction objective weights the changes in the map according to the effect of observed inputs. We ran this control using the weighted criterion for the feature-based mask selection. In our experiments, weighting often improved one-shot accuracy on the real data benchmarks. After finetuning, the differences largely narrowed, suggesting that both mask selection criteria support similar final accuracies (Figure 21). These results have been added to the appendix. Ultimately, both objectives are consistent with preserving layer function under pruning, a central goal in practical model compression. Our feature-space view further allows us to interpret what this preservation means for the model’s learned feature representation. Thus, our analysis connects functional reconstruction to the structure of the model’s computation. We have clarified this connection in the revision (L293, L439, Section A.12).

  3. Regarding one-shot performance, we agree that MIP and OBS are often comparable before finetuning. We find the hybrid FLO most scientifically interesting, since it combines the two criteria (loss-based and feature-based) at different stages, suggesting that they provide complementary information for pruning. Additionally, Figure 8 helps isolate the contribution of the mask itself, where the MIP mask (with the dense weights) may start at a lower accuracy than the other techniques, yet recover to higher accuracy under the same finetuning budget.

Thank you again for your engagement with our work and for your useful comments. We believe that the clarified connection to layer reconstruction and the additional control experiments strengthen the paper and make its scientific scope clearer.

AuthorsOpenReview ↗

Following up with new evidence for feature-space alignment hypothesis in Llama 3.2 1B

We would like to follow up with additional results addressing your concern about whether the feature-space perspective extends to modern and larger scale architectures. Since our initial response, we have added experiments on Llama 3.2 1B examining how feature-map preservation relates to pruning performance under magnitude pruning, Wanda [1], and SparseGPT [2], allowing us to investigate this relationship beyond our controlled setting. We show that the feature-space perspective remains informative at the scale of modern language models: better pruning performance accompanies convergence to the dense feature-map representations despite greater divergence in weight space.

We trained TopK SAEs, as we did in Section 4.4, on the dense input activations of the middle MLP layers of Llama 3.2 1B. We then compared pruning the up projection to 50%, 60%, 70%, and 80% sparsity with magnitude pruning, Wanda, and SparseGPT, pruning one layer at a time. Following pruning, we measure the perplexity (lower is better) of the resulting model, as well as the changes in the weights WW and the feature map C=WDC = WD, where DD is the SAE decoder, for each pruning condition. DD is used only for this post-pruning analysis. We used disjoint text data for SAE training, calibration, and evaluation.

In terms of language modeling performance, SparseGPT achieves the lowest perplexity, followed by Wanda and then magnitude pruning, as expected (Figure 27). Crucially, weight preservation ranks these methods in exactly the opposite order to performance: magnitude pruning preserves the weights most closely to the dense model but yields the highest perplexity, while SparseGPT changes them most and yields the lowest (Figure 10a, b). By comparison, feature-map divergence from the dense model tracks performance across pruning methods: within each examined layer, smaller changes in the feature map consistently accompany lower perplexity (Figure 10c, d). This distinction persists across the tested sparsity levels, with Wanda and SparseGPT consistently changing the weights more than magnitude pruning relative to the dense model, while changing the feature map less (Figure 28).

Across the examined layers and sparsity levels, closer preservation of the feature map consistently accompanies better language modeling performance, even though these methods do not explicitly optimize the SAE-derived map. These findings support the relevance of the feature-space alignment hypothesis to a pretrained transformer and its usefulness for understanding existing pruning methods. We have added these results, including analyses of feature preservation across calibration feature firing rates, to section 4.5 and appendix A.14. We hope this additional evidence helps address your concern about the applicability of our findings beyond our controlled models.

1 https://arxiv.org/abs/2306.11695

2 https://arxiv.org/abs/2301.00774

I thank the authors for the detailed response and the addtional experiments, I think those response have solved most of my concerns.

First, the results on Llama 3.2 1B is really valuable, it further support the hypothesis, although the author do not conduct their pruning method on LLM, I view it an open question not the significant shortcome of this paper.

then, the results on layer-output reconstruction and combinatorial OBS help clarify the relationship between the proposed method and preserving the layer output. Figure 8 and the FLO results also solved my previous concerns about MIP and OBS comparison, highlighting the value of the mask.

Overall, I think the author has addressed most of my question and make the paper more strong, thus I am willing to raise the rating by 1.

AuthorsOpenReview ↗

Thank you!

Thank you for your time and constructive discussion. We greatly appreciate your support of the work and your updated assessment. We feel that your comments have helped improve the work substantially, and agree that the remaining open questions are crucial to explore in future work.

Reviewer_QT4cOpenReview ↗
Rating
3(−1)
Confidence
4(+1)
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Pruning is framed as preserving the effective feature-space mapping, not individual weights. Controlled experiments link feature degradation to pruning failure; then feature-preserving sparse optimization is tested, with an activation-learned SAE dictionary replacing oracle information.

Critical strengths
  • Intervenes on feature preservation, not just correlation.
  • Separates mask topology from weight compensation; worse immediate mask can recover better after fine-tuning.
  • SAE dictionary replaces oracle embedding; limitations clearly acknowledged.
Critical weaknesses
  • Mathematical interpretation: objective is quadratic reconstruction involving C₀C₀ᵀ; SAE objective depends on decoder Gram matrix and is invariant to orthogonal dictionary rotations. Unclear how much benefit needs interpretable directions vs. suitable reconstruction geometry. Add covariance/PCA control.
  • MIP changes both objective and support selection; need control separating feature preservation from combinatorial search.
  • Experiments are controlled and small-scale; limits generalization but fits mechanistic contribution.
Questions for authors

What is the computational cost of SAE training and MIP pruning relative to the baselines, and which stage becomes the main bottleneck as layer width increases?

Aside from that, the scale restriction is justifiable, but I would still like to see larger data. I know we would likely lose the explainability aspect, yet I believe it is still worth pursuing beyond the synthetic dataset.

6 comments

We would like to thank the reviewer for your positive assessment and support of the work. We also appreciate that you found the feature preservation motivation as well as the separation of mask topology from weight compensation interesting. We address your main concerns below:

  1. W1: We agree that the reconstruction objective only depends on the decoder Gram matrix, and is therefore invariant to orthogonal dictionary rotations. Our primary goal was to test whether techniques from interpretability could recover geometry relevant to pruning, without explicit knowledge of the true embedding. We have now emphasized this more clearly in the revision (L459), and also added the suggested covariance control where we weigh the reconstruction error by how often features are active and how often they occur together. With this weighting, FLO often achieved higher one-shot accuracy on the real-data benchmarks compared to the unweighted version. However, following finetuning, the gap between the weighted and unweighted FLO performance largely closed, while both still outperform OBS/OBD/MP, indicating that even the unweighted FLO mask retains connections that support strong accuracy after finetuning (Figure 21). We thank the reviewer for highlighting this distinction.
  2. W2: We acknowledge that our pruning analysis changes both the objective and support selection. To separate them, we have now run a control isolating the objective from the stronger support selection. This combinatorial OBS baseline [1] selects each row r’s support by exactly solving the OBS objective min⁡m,w’(m⊙w’−w)THrr(m⊙w’−w)\min_{m, w’} (m \odot w’ - w)^T H_{rr} (m \odot w’ - w) over binary masks mm with kk nonzeros, where ww is the row’s weights and HrrH_{rr} is the row’s block of the loss Hessian. The same OBS weight update as FLO and OBS is then applied. Thus, the two methods differ only in the metric used to choose the support (loss Hessian vs. feature geometry). This combinatorial OBS baseline outperforms standard OBS, while our feature-aware methods still outperform both OBS forms across the different DNF sizes, especially after finetuning. This has now been added to the appendix (Figure 20), thank you for suggesting this additional control experiment.
  3. Q1: The SAE training and MIP are more substantial relative to the baselines. On our hardware (11GB VRAM GPU; 64 thread CPU with MIP solves running in parallel), we measure the wallclock time to be approximately as follows. For letter recognition at 75% sparsity, the MIP takes 2.9 seconds, SAE training takes about 10.5 seconds, and the Hessian computation takes 0.2 seconds. For the larger 64 literal DNF, we limit each row’s MIP to 300 seconds. All rows reach this limit, so we use the best feasible mask SCIP finds, and so the MIP therefore takes approximately 10 minutes of wall-clock time on 32 cores. For comparison, the Hessian computation takes 97 seconds. Even with the time limited masks, FLO remains the strongest method in most configurations. At scale, we find that MIP is the dominant cost. We will further emphasize how FLO is intended as a scientific tool rather than a practical pruning algorithm, with the intent of better understanding how features are preserved under pruning and how feature preservation can guide pruning. We will also continue exploring faster approximations to the MIP.
  4. W3/Q2: We agree that testing the hypothesis on larger datasets and models is an important next step. Extending this analysis to such settings, where the ground truth structure and computations is less known than it is in the DNF, remains a priority for future work.

Thank you again for your thoughtful review. We feel that the additional controls and discussion substantially strengthen the work.

1: https://arxiv.org/pdf/2203.04466

Thank you for your reply I maintain my score of 4, and I appreciate the additional details and that you considered my suggestion. I recommend removing the explicit hardware disclosure, as it closely ties the model’s/ scale limitations to the hardware mentioned. The lack of larger data and a larger model is likely due to the amount of VRAM available, which can be interpreted as so, Instead, you could simply state the VRAM of the hardware you mentioned on the comet itself. I believe a large dataset validation will be beneficial as larger model is likely not feasible.Thank you for your honesty.

Thank you for your suggestion and continued support! We have updated our comment to report only the GPU memory and CPU thread count. We agree that validation on larger datasets remains a crucial next step, and we are continuing to explore faster ways to optimize our framework.

Reviewer_wGzFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to explain the success (and failures) of magnitude-based pruning from the lens of mechanistic interpretability. Specifically, the authors propose using a controlled setting where input features are "mixed" across layers to enable analysis in the transformed basis. The mask selection problem (called the 'optimization oracle') is formulated as an MIP and solved using standard solvers for small-scale problems. Experimental evaluation is on small scale tasks.

Critical strengths
  • This work is one of the first to connect the fields of pruning (though only magnitude pruning in the unstructured/semi-structured settings) and mechanistic interpretability.
  • The controlled setting does explain the limits of sparsification for small tasks such as the DNF task.
Critical weaknesses
  • I do not think this approach is particularly interesting from a pruning standpoint. While the proposed machinery does explain sparsification in when ground truth features are well known (and yes, even in cases where one can use a sparse autoencoder), I do not think this approach can adequately explain the limits of magnitude pruning more complex models, like CNNs or transformers. Moreover, solving MIPs is certainly not scalable.
  • The improvement over baselines like OBS might be overstated. Equation 1 explictly solves the pruning problem with closed-form compensation as an MIP, whereas the baselines mentioned in the work rely on heuristics for solving optimization problems. For instance, we can use the same notation M⊙WˉM\odot \bar{W} for the compensated weights, and solve arg⁡min⁡Wˉ,M  (M⊙Wˉ−W)⊤H(W)(M⊙Wˉ−W),s.t.Mij∈{0,1},∑Mij=τ\arg\min_{\bar{W},M}\; (M\odot \bar{W} - W)^\top H(W) (M\odot \bar{W} - W), s.t. M_{ij}\in\{0,1\}, \sum M_{ij} = \tau to get the "true" OBS mask, rather than relying on heuristics. is the improvement solely down to solving the MIP vs using heuristics?
  • The MIP in equation 1 is totally agnostic to the data. The reconstruction with the data requires solving arg⁡min⁡Wˉ,M(M⊙Wˉ−W)⊤Σ(M⊙Wˉ−W),s.t.Mij∈{0,1},∑Mij=τ\arg\min_{\bar{W},M} (M\odot \bar{W} - W)^\top\Sigma (M\odot \bar{W} - W),s.t. M_{ij}\in\{0,1\}, \sum M_{ij} = \tau. Σ\Sigma is the uncentered covariance of the data. Essentially, in equation 1, the authors assume Σ≡I\Sigma \equiv I - that is, in the data, all features are uncorrelated and equally weighted (and thus equally important). To me, this is a critical issue with this work, as it fails in settings where the features are correlated.
Questions for authors

See weaknesses.

4 comments

Thank you for your thorough review, and in particular for noting the work’s connection between pruning and mechanistic interpretability, which is our primary goal and motivation. Below, we address your concerns:

  1. W1: Our central question is how accuracy is maintained when a model is sparsified. Does the sparse model preserve the dense model’s learned feature structure, or does it achieve similar accuracy by reorganizing its representation? The known DNF clauses let us examine this question directly. In Figure 3, even after magnitude pruning removes 75% of the weights in W1W_1, several codes persist in the same neurons and for the same clauses as in the dense model. Their degradation across sparsity levels tracks increasing loss (Figure 2), and selecting sparse weights to reconstruct the effective map retains more codes (Figure 7) and supports stronger recovery (Figure 8).

    We use MIP for feature-space reconstruction at this scale. We agree that its computational cost limits direct application to large models, but this controlled setting lets us examine what pruning preserves and test the usefulness of preserving it. We have made this scientific motivation explicit in the revision (L53, L57, L63). Establishing how broadly the observed mechanism applies to CNNs and transformers remains important future work.

  2. W2. Thank you for pointing this out. We agree that our pruning analysis changes both the objective and support selection. To separate them, we have now run a control isolating the objective from the stronger support selection. This combinatorial OBS baseline [1] selects each row r’s support by exactly solving the OBS objective min⁡m,w’(m⊙w’−w)THrr(m⊙w’−w)\min_{m, w’} (m \odot w’ - w)^T H_{rr} (m \odot w’ - w) over binary masks mm with kk nonzeros, where ww is the row’s weights and HrrH_{rr} is the row’s block of the loss Hessian. The same OBS weight update as FLO and OBS is then applied. Thus, the two methods differ only in the metric used to choose the support (loss Hessian vs. feature geometry). As the reviewer anticipated, this combinatorial OBS baseline outperforms standard OBS. However, our feature-aware methods still outperform both OBS forms across the different DNF sizes. Following finetuning, the mask selected by FLO yields the most recovery as well (Figure 20). Thus, a stronger search did explain part of the original advantage, but the advantage from understanding features remains under the matched comparison. This has now been added to the appendix and discussed in the main text (L344).

  3. W3. We agree that Equation 1 gives equal weight to errors along each input feature direction, though it still does depend on the trained data through the learned embedding. This was a deliberate experimental choice, as our original motivation concerns distortion of the full effective map. The reviewer’s proposed weighting measures how those same errors affect the calibration inputs actually observed. We tested this directly using a data-weighted version of Equation 1 for feature-based mask selection, evaluating changes in the effective map on calibration inputs. Weighting often improves one-shot accuracy on the real-data benchmarks. After fine-tuning, the gap largely closes, with both mask-selection criteria supporting similar final accuracies under the same training budget (Figure 21). This control shows that incorporating input statistics can improve immediate pruning performance, while full-map preservation also selects connections that support strong recovery. We have added the experiment and clarified the distinction between these preservation criteria in the appendix and discussed this in the main text (L439).

We would like to thank the reviewer once again for your insightful comments. The suggested controls show that while stronger combinatorial search improves OBS, the remaining advantage of FLO suggests that feature information also matters for mask selection. Additionally, accounting for input correlations often improves one-shot accuracy on the real-data benchmarks, while weighted and unweighted FLO reached similar accuracies after finetuning. Ultimately, the scientific value of the controlled setting is that we can identify and test how much accuracy can be recovered by selecting connections to preserve feature structure. We believe that these additional discussions and suggested controls have substantially improved the framing and scope of the work.

1: https://arxiv.org/pdf/2203.04466

Reviewer_wGzFOpenReview ↗

Response to Authors

I thank the authors for the response. However, I still have some crucial concerns.

  1. While I accept that the proposed MIP based approach is specific to very small models. While it is interesting for interpretability, the fact that this method does not scale to larger models means this approach is not applicable for pruning. Moreover, since this approach is not scalable, I do not think that it is applicable at scales where emergent phenomena or even superimposed representations are observed (which are crucial in modern deep learning).

  2. Thanks for the control experiment, it has answered some of my questions there. However, I do not think that CBS uses an MIP solver, and instead uses greedy heuristics which probably underperform the MIP solver. So while the provided experiements are helpful, they still do not completely address the question of whether the MIP solver was crucial to the performance of the proposed approach.

  3. I would first like to make clear that the formula I suggested isn't a proposed data weighting, it is exactly the expected L2L_2 reconstruction expanded out, and with constant terms removed. When you are using reconstruction error, this is the correct formula to use - and critically different from the formulation in the paper. Moreover, as per figure 21, the gap doesn't really close with fine tuning - there is still a clear advantage for using the correct reconstruction error formulation (though admittedly the gap shrinks with fine tuning). This gap, on the other hand, will likely increase significantly with model scale, even with finetuning. This in turn is an example of phenomena that would not be observed on small scale models.

AuthorsOpenReview ↗

New evidence for feature-space alignment hypothesis in Llama 3.2 1B (1/2)

Thank you for your reply and your continued engagement. We would like to address each of your concerns here as well as providing experimental evidence for relevance at scale.

1. Feature-map preservation tracks pruning performance in Llama 3.2 1B

To address your concern about whether the feature-space perspective extends to modern architectures, we have conducted additional experiments at the LLM scale on Llama 3.2 1B, a substantially larger model with emergent phenomena and superimposed representations. We show that the feature-space perspective remains informative at the scale of modern language models: better pruning performance accompanies convergence to the dense feature-map representations despite greater divergence in weight space.

We trained TopK SAEs, as we did in Section 4.4, on the dense input activations of the middle MLP layers of Llama 3.2 1B. We then compared pruning the up projection to 50%, 60%, 70%, and 80% sparsity with magnitude pruning, Wanda [1], and SparseGPT [2], pruning one layer at a time. Following pruning, we measure the perplexity (lower is better) of the resulting model, as well as the changes in the weights WW and the feature map C=WDC = WD, where DD is the SAE decoder, for each pruning condition. DD is used only for this post-pruning analysis. We used disjoint text data for SAE training, calibration, and evaluation.

In terms of language modeling performance, SparseGPT achieves the lowest perplexity, followed by Wanda and then magnitude pruning, as expected (Figure 27). Crucially, weight preservation ranks these methods in exactly the opposite order to performance: magnitude pruning preserves the weights most closely to the dense model but yields the highest perplexity, while SparseGPT changes them most and yields the lowest (Figure 10a, b). By comparison, feature-map divergence from the dense model tracks performance across pruning methods: within each examined layer, smaller changes in the feature map consistently accompany lower perplexity (Figure 10c, d). This distinction persists across the tested sparsity levels, with Wanda and SparseGPT consistently changing the weights more than magnitude pruning relative to the dense model, while changing the feature map less (Figure 28).

These results extend the central finding of our controlled experiments to Llama 3.2 1B: preserving the dense model’s feature map is associated with better performance after pruning. The consistency of this relationship across the examined layers, sparsity levels, and pruning methods supports the feature-space alignment hypothesis at this scale and demonstrates its usefulness for understanding existing scalable pruning methods.

We have now added these results, including analyses of feature preservation across calibration feature firing rates, to section 4.5 of the main text and appendix A.14.

We do agree that the computational cost of MIP in its current form limits its immediate deployment as a practical pruning algorithm. However, as discussed here and in the manuscript, its role was not meant to serve as such, but rather to provide evidence that preserving feature-space structure can improve sparse performance. The scalability of this particular optimizer is therefore distinct from the applicability of the underlying feature-space alignment hypothesis, which the evidence shows also happens at scale.

1 https://arxiv.org/abs/2306.11695

2 https://arxiv.org/abs/2301.00774

AuthorsOpenReview ↗

MIP control and objective clarifications (2/2)

2. The same MIP solver was used for both objectives

We should have been clearer regarding the control in our original reply. You are correct that CBS uses heuristics rather than an MIP solver, and we cited CBS to acknowledge the related combinatorial formulation.

In our control experiment, however, we indeed solved the Hessian-based quadratic objective using the same MIP solver and sparsity constraints as our feature-space objective method, as intended by “exactly solving” in our original response. The comparison therefore addresses whether stronger support selection alone explains the improvement: it explains part of the original advantage, but the feature-aware objective retains an advantage under this matched comparison. We apologize for the ambiguity introduced by calling this baseline “combinatorial OBS” alongside the citation, and we have made this distinction more clear in the revised manuscript when the citation appears as well (L334, L1406)

3. Feature preservation and reconstruction error

Thank you for clarifying this distinction. We agree that your expression gives the exact expected squared reconstruction error for the layer’s preactivations, and that its advantage persists after fine-tuning, although the gap narrows. Our feature-map objective remains a valid preservation objective even when features are correlated, as it measures changes to individual feature responses, and preserving those responses exactly preserves the map’s response to every feature combination, regardless of how features co-occur in the data. Our earlier wording should have distinguished this target from expected reconstruction on the observed data, and we have now amended this in the update (L431, L515, L1467). The control shows that accounting for the data distribution yields further performance gains, complementing our central finding that directly preserving feature structure can improve pruning.

The purpose of our demonstrative methods is to test the complementary prediction of the feature-space alignment hypothesis: that directly preserving learned feature structure can improve sparse performance. Our experiments provide evidence for this prediction, while leaving room for stronger objectives and algorithms, as your suggested control illustrates. Demonstrating this principle does not necessarily require uniform feature-map preservation to be the optimal pruning criterion.

Together with our new Llama 3.2 1B experiments, we connect direct feature-preserving interventions in controlled models with the behavior of existing scalable pruning methods in transformers. How the performance gap between uniform map preservation and expected reconstruction changes with scale remains an open empirical question and important future work.

Thank you once again for your continued engagement with our work. We feel that these additional results have greatly strengthened the work by broadening its empirical scope and extending the evidence for our central hypothesis to a modern pretrained transformer. We hope these results, together with our clarifications, address your remaining concerns.

Comments on the paper

1 comment

We would like to thank the reviewers and the AC for their time and engagement with our work. We would like to start by clarifying the main motivation of our paper which is reflected in its title but perhaps has not been conveyed successfully. Our goal is to answer a simple question.

When a model is sparsified, how is accuracy preserved?

  1. One hypothesis could be that the choice of sparse weights induces a change of the set of features and how they are detected in feature space, that is, a completely new representation is created. This is quite plausible, since removing many of a neuron’s weights can reduce its expressive power and thus the features it detects.
  2. An alternative is that, despite losing many weights, the sparse model preserves the dense model’s feature structure and thus its accuracy.

Our experiments provide evidence for the second possibility. We consider this quite surprising, as even after losing 75% of the weights, networks manage to keep much of the same detection structure intact in feature space (for example in Figure 3). We realize that showing this only on a DNF problem is interesting but not conclusive, and so the rest of our paper is demonstrating that this phenomena appears even in more complex cases where we do not necessarily know how to identify the feature space structure. We do so by showing that we can retain accuracy by reconstructing the dense model’s feature space, rather than pruning weights via standard baselines such as OBS. Even when the true embedding is unavailable, a feature dictionary learned from activations recovers much of the same pruning benefit.

We hope this context clarifies how our sparsification methods serve as tests of the feature-space alignment hypothesis rather than immediately practical pruning techniques, and have incorporated these clarifications in the revised manuscript. We believe that understanding feature space can help improve sparsification and, over time, guide the development of pruning methods suitable for deployment in large models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.