acceptodds
Withdrawn from ICLR 2027

HOW DO WEIGHT AND ACTIVATION SPARSITY INTERACT?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A multiply-accumulate (MAC) contributes to inference cost only if both its weight and activation operand are nonzero, yet the two are almost always regularized independently. We test whether jointly regularizing both, with a shared per-element-normalized squared-Hoyer measure, causes them to reinforce or compete, and find the answer is conditional rather than universal. On ResNet-18/CIFAR-10, joint (additive) training pushes both sparsities above what either solo penalty reaches alone (weight sparsity vs. ; activation sparsity vs. ; seeds), and the resulting effective-MAC fraction beats a naive independence estimate at a reduction and of baseline accuracy – though at matched compute a solo penalty alone reaches statistically indistinguishable accuracy, so the gain is a different sparsity split, not a proven accuracy advantage. A theoretically appealing coupled (product) objective instead underperforms the additive sum by accuracy points at matched compute, substantially a convergence-speed artifact: the gap shrinks by more than half after doubling the training budget and closes to points by 50 epochs. Reinforcement replicates in most but not all of six further settings (MNIST, CIFAR-100, SVHN, MobileNetV2, and structured sparsity) – reversing on MNIST, an order of magnitude stronger under structured sparsity, and turning outright bimodal on SVHN – and the same conditional pattern holds under an entirely different, discrete weight-sparsity mechanism on four independent production edge-AI models, where the interaction is complementary, antagonistic, sign-flipping, or accuracy-improving depending on the model. Critically, the coupled objective's failure is a fact about Hoyer specifically, not product-form coupling in general: a density-aligned surrogate closes a -point accuracy gap to points, statistically indistinguishable from additive. We report this as evidence that weight and activation sparsity interact dynamically and conditionally rather than independently.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
sparsity, weight sparsity, activation sparsity, joint regularization, compute efficiency, MAC reduction, Hoyer regularization, structured sparsity, neural network optimization, ResNet-18, conditional interactions, model compression, sparse inference, edge AI, pruning. Provide your feedback on BizChat

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NRHVOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors investigate how regularizing weights and activations jointly (either multiplicatively or additively), as compared to individually, affects the resulting performance and sparsity. In addition, they focus on wether joint regularization causes the penalties to compete with or enforce each other. As a penalty, they use a normalized squared Hoyer measure. In some settings, the additive-joint approach reinforces sparsity in most cases and yields similar test accuracy compared to the individual approaches. Furthermore, in longer training settings, the gap of the coupled-joint approach closes compared to shorter training. In addition, a density-based measure approach suggests that the failure of the coupled mechanism is Hoyer-specific in this short training setup. In other settings, these findings do not replicate. For example, outside vision, the coupled Hoyer-based approach outperforms the additive ones.

Critical strengths

Depth of analysis and evaluation: The authors investigate across a lot of settings and also provide a lot of additional material in the appendix, which is also heavily referenced in the main text itself. In addition, they thoroughly investigate failure modes (e.g., the underperformance of the coupled formulation and the related issues of the Hoyer measure itself).

Critical weaknesses
  1. Scope and payoff: The findings about reinforcement (especially w.r.t. channel sparsity) are somewhat interesting, yet I am doubtful about the downstream usefulness of these insights, as the effects are incoherent across settings. Without either a corresponding theory on when reinforcement happens (and wether/when it should be applied additively or multiplicatively) or a coherent empirical claim (that is not just "it depends", but that gives actual actionable insight for either downstream practical applications or future theoretical analyses), it is hard to see how these findings can help from a theoretical or a practical perspective.

  2. Presentation and science communication: For me, the paper was rather hard to read, as claims, explanations, empirical findings, and caveats are kind of heavily intertwined. In addition, at some points, the paper reads like an already revised version of an earlier manuscript (e.g., "Table 1 reports the five-regime comparison at n = 5 seeds per arm, upgraded from n = 3 for this headline comparison" (lines 190-192); "5 of 12 comparisons overall now survive correction" (line 200)). Whilst revision itself is not bad, the way it is included currently fragments this paper and makes it hard to read and understand the general story.

Questions for authors
  1. Why do you stick to the Hoyer measure when some of the issues you see seem to stem explicitly from using Hoyer? What is its specific appeal? If you are trying to find some patterns w.r.t. joint vs. individual regularization of weights and activations, wouldn't it make more sense to use a larger set of measures?

  2. Do you have any idea why, for the word-level Transformer LM (Table 5), the weight sparsity of the coupled approach is that much lower than for the vision-based approaches?

  3. If I understand correctly, normal GPUs can't exploit activation sparsity, and the effective sparsity thus serves as a proxy. For me, this raises the question: why might one prefer balanced weight and activation sparsification instead of only targeting the weights, if its computational advantages only hold for specific architectures? Since activation sparsity targets the computation on the input, could it maybe help in the setting of OOD or noisy inputs?

Reviewer_28QfOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper encourages weight and activation sparsity together by summing or multiplying two Hoyer penalties, mainly on CIFAR-10. The idea is a simple and intuitive combination of two existing penalties. The empirical evaluation is not convining.

Critical strengths

Covariance measurement (Appendix K) finds nonzero weights and nonzero activations are positively correlated after joint training. This could be further investigated and exploited for useful algorithms.

Critical weaknesses

Novelty is limited. The product form of effective MACs appears in SCNN, Ahmad et al.\ (2019), Hunter et al.\ (2021), Hoefler et al.\ (2021) and DuoGPT.

A 2×22\times2 factorial interaction computed from Tables 1 and 2 is −0.48-0.48 (weight) and +4.35+4.35 (activation) at 10 epochs, and −2.34-2.34 and −4.56-4.56 at 50 epochs. At matched compute, joint additive is indistinguishable from weight-only (86.33 vs 86.63, p=0.21p=0.21).

Mathematical results are not providing much non-trivialities.

Questions for authors

(1) The final conclusion is that the two sparsities reinforce in some settings and compete in others. What would a reader know after this paper that could not have been guessed before it, and which possible outcome of the experiments would have contradicted the paper's thesis?

(2) At matched compute, joint training is no more accurate than penalizing the weights alone. Given this, what is the principled reason for anyone to regularize both operands jointly?

(3) The paper motivates the product objective by the rule that a multiply-accumulate is skipped when either operand is zero, but this rule holds for each weight-activation pair, while both objectives act on whole tensors. Which principle is then being tested, and why was the objective not formulated at the level of pairs?

Reviewer_3cbwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors study weight and activation sparsity behaviour when jointly regularised. They specifically explore the concept that the multiply accumulate operation can be skipped if either the weight or the activation is zero, and where penalising both during training may cause the sparsity effects to reinforce. The authors present an evaluation of an additive joint regulariser against a coupled objective based on the squared Hoyer. The empirical evaluation tests a variety of architectures and benchmark datasets, exploring the generalisation of the approach widely to mixed results. They show that joint additive training reinforces both sparsity more that a single penalty but it does not fundamentally improve the overall accuracy to compute.

Critical strengths
  1. The insights this work provides move away from simply achieving joint sparsity to instead explaining how and when the two sparsity terms interact. This presents insightful information that the wider community may take forward and spark future research.
  2. The proposed method improves the balance of parameters, and the authors directly acknowledge that the balanced sparsity they aim to preserve is a theoretical benefit until proven on physical hardware. While the accuracy and compute benefits are not entirely present here they present work honestly and transparently, but it does introduce questions that could better leverage the findings here.
  3. Generally the empirical work is very robust including the generalisation performance (although I would like to see ViT in the main text) specifically the inclusion of edge-ai benchmarks and also the effective-mac validation section which provides a empirical metric to support the MAC proxy.
Critical weaknesses
  1. The paper is incredibly hard to read in that, the grammatical structure, and narrative flow are abrupt and disjointed in places. Significant work is needed to present the work in a more digestible and clear way.
    1. For example there is a heavy reliance on the appendices and therefore leaves critical information out. For example the exact MAC implementation is left out of the main text and arguably is an essential component to this papers message.
    2. The sentence structure is too dense in terms of terminology, sub-claims, and switches context constantly.
    3. Structure the tables and results more appropriately, in that you constantly refer to tables and results that exist throughout appendices and vice versa.
  2. For a more fair comparison, setting the MAC fraction to be uniform across all approaches to determine how accuracy is impacted by the trade-off would be more insightful. Currently, it is hard to determine the benefit of the work, as outlined in Appendix section F. the outcome is that if weight only is set to match the MAC fraction, then the accuracy is marginal under similar compute.
  3. From my understanding the proposed Effective-MAC is only suitable for non-linear activation functions that clip to 0.0, therefore the proposed approach is limited in scope and scale to more modern transformer architecture, which was demonstrated by the authors themselves. This questions the applicability of the method to modern widespread architectures.
  4. The number of seeds used is too low to claims strong statistical significance.
  5. The propositions 1 and 2 are left unproven from the authors own experimentation, this casts serious implications on the mechanistic claims made in this paper, either these need to be more appropriate
Questions for authors
  1. If setting the MAC fraction uniformly for all experimental regimes, does the proposed method improve the results? If not, is there any benefit to the method?
  2. From my understanding, the results demonstrate the theoretical MAC reduction is unstructured. Therefore how much of this compute reduction actually results in improved wall-clock speed for standard hardware? I know this is a limitation of unstructured pruning in general, but would be good to understand the practical benefit.
  3. Proposition 1 and 2 were arguably disproven from the causal test from appendix Q, you state that this does not read as evidence against reinforcement, how can you be sure that the claims you make that these propositions for reinforcement are mechanistic.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.