- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors investigate how regularizing weights and activations jointly (either multiplicatively or additively), as compared to individually, affects the resulting performance and sparsity. In addition, they focus on wether joint regularization causes the penalties to compete with or enforce each other. As a penalty, they use a normalized squared Hoyer measure. In some settings, the additive-joint approach reinforces sparsity in most cases and yields similar test accuracy compared to the individual approaches. Furthermore, in longer training settings, the gap of the coupled-joint approach closes compared to shorter training. In addition, a density-based measure approach suggests that the failure of the coupled mechanism is Hoyer-specific in this short training setup. In other settings, these findings do not replicate. For example, outside vision, the coupled Hoyer-based approach outperforms the additive ones.
Critical strengths
Depth of analysis and evaluation: The authors investigate across a lot of settings and also provide a lot of additional material in the appendix, which is also heavily referenced in the main text itself. In addition, they thoroughly investigate failure modes (e.g., the underperformance of the coupled formulation and the related issues of the Hoyer measure itself).
Critical weaknesses
-
Scope and payoff: The findings about reinforcement (especially w.r.t. channel sparsity) are somewhat interesting, yet I am doubtful about the downstream usefulness of these insights, as the effects are incoherent across settings. Without either a corresponding theory on when reinforcement happens (and wether/when it should be applied additively or multiplicatively) or a coherent empirical claim (that is not just "it depends", but that gives actual actionable insight for either downstream practical applications or future theoretical analyses), it is hard to see how these findings can help from a theoretical or a practical perspective.
-
Presentation and science communication: For me, the paper was rather hard to read, as claims, explanations, empirical findings, and caveats are kind of heavily intertwined. In addition, at some points, the paper reads like an already revised version of an earlier manuscript (e.g., "Table 1 reports the five-regime comparison at n = 5 seeds per arm, upgraded from n = 3 for this headline comparison" (lines 190-192); "5 of 12 comparisons overall now survive correction" (line 200)). Whilst revision itself is not bad, the way it is included currently fragments this paper and makes it hard to read and understand the general story.
Questions for authors
-
Why do you stick to the Hoyer measure when some of the issues you see seem to stem explicitly from using Hoyer? What is its specific appeal? If you are trying to find some patterns w.r.t. joint vs. individual regularization of weights and activations, wouldn't it make more sense to use a larger set of measures?
-
Do you have any idea why, for the word-level Transformer LM (Table 5), the weight sparsity of the coupled approach is that much lower than for the vision-based approaches?
-
If I understand correctly, normal GPUs can't exploit activation sparsity, and the effective sparsity thus serves as a proxy. For me, this raises the question: why might one prefer balanced weight and activation sparsification instead of only targeting the weights, if its computational advantages only hold for specific architectures? Since activation sparsity targets the computation on the input, could it maybe help in the setting of OOD or noisy inputs?