- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes a new regularization method called activation-spectrum regularization (ASR) which penalizes the spectrum of hidden-layer activation covariance matrices over mini-batches, with the penalty increasing as the model becomes more confident in its predictions (likelihood-gated pressure). Specifically, paper compares three objectives: a Rényi-2 participation-ratio proxy, spectral entropy, and a ridge Log-Det penalty, all intended to complement conventional weight decay by constraining representations rather than model parameters.
Proposed approach is evaluated on six task/architecture combinations: modular addition, permutation composition, and parity, including both multilayer perceptrons (MLPs) and Transformers. They report that ASR reduces the average training-data fraction required to achieve 99% test accuracy by an absolute 7.37%, or 11.40% relative to the baseline, across the six-tasks. Rényi-2 reduces the mean threshold on all six tasks, while spectral entropy performs particularly well on heterogeneous arithmetic mixtures. An additional experiment on UCI Superconductivity regression reports improvements in mean test , mean absolute error, and root mean squared error for all three ASR objectives, without tuning the regularization coefficient on that dataset.
Critical strengths
None
Critical weaknesses
- Empirical support for proposed regularizers is on quite small datasets and network sizes. For this reason, practical scalability and transfer to larger datasets and models remain uncertain.
- Primary results are for training data efficiency. However, data efficiency -- unless it is orders of magnitude -- is typically not a concern in typical model training scenarios these days.
- Missing reference to and comparison with some closely related works:
- Kazanskii (2026) — How to Tame Grokking: Representation Geometry as a Control Signal
- Yunis et al. (2024) — Grokking, Rank Minimization and Generalization in Deep Learning
- Junior et al. (2025) — Grokking Beyond the Euclidean Norm of Model Parameters
- The proposed mechanism is not established causally. The observed spectral contractions coincide with temporary accuracy drops, but the paper explicitly acknowledges that this alignment is not causal evidence. It does not establish that activation-spectrum compression causes grokking, that the model discovers a simpler algorithm, or that lower effective dimensionality is the mechanism responsible for improved generalization. The relationship between the regularizer and the learned computation remains an open question.
Questions for authors
Anything the authors can provide to address the critical weaknesses above will be helpful.