- Rating
- 1
- Confidence
- 3(+1)
- Informativeness
- 1
- Presentation
- 1(−1)
- Soundness
- 1(−1)
Read review
Paper summary
The paper studies Fourier feature sparsity in modular addition under norm-constrained margin maximization. It characterizes optimal margins and frequency configurations for interference-free product networks, proves that every optimizer of the associated spectral allocation problem uses (\Theta(\log p)) frequencies for fixed (s>1), and establishes asymptotic margin bounds for product networks allowing interference. Training experiments examine how these predictions relate to learned frequency counts.
Critical strengths
The paper goes beyond the established observation that multiplicative parameterizations induce sparsity-promoting regularization. Its characterization of interference-cancellation costs and quantitative bounds on optimal spectral support provide concrete mathematical insight within the studied setting. The authors also distinguish ideal margin maximizers from solutions reached through training.
Critical weaknesses
- The broader significance of the theoretical advance remains limited. The exact characterization concerns a restricted class of networks on modular addition, while the feature-count theorem concerns the associated spectral optimization problem. These results refine an established research question, but I do not find that the paper sufficiently establishes a transferable principle or consequential implication beyond this setting. A new task or practical algorithm is not essential for a theory paper; nevertheless, the depth and reach of the demonstrated advance are insufficient for acceptance in my assessment.
- The connection to learned feature counts is not convincing enough to compensate for this restricted scope. The exact characterization relies on interference-free global optima, whereas trained networks can contain interference and need not reach those optima. Appendix E reports that the preregistered plateau-identity test matches only 2 of 19 converged runs, and networks with the same (s) but different degrees show systematic differences. Although the experiments support qualitative trends, they provide substantially weaker support for a quantitative explanation of the number of features actually learned. This limits the explanatory contribution without invalidating the conditional theorems.
Questions for authors
- What specific insight from the new results extends beyond modular addition and its Fourier structure, beyond the established sparsity bias of multiplicative parameterization? Please identify which parts of the analysis support such a conclusion.
- How should the experimental discrepancies—particularly the plateau mismatches and degree dependence at fixed (s)—affect the interpretation of the proposed explanation of learned feature counts? What existing evidence separates the effects of interference from failure to reach a global optimum?
3 comments
Thank you for the revision. Proposition 17 extends the logarithmic feature-count result to an abstract allocation problem, but the response does not show that a trained network on a concrete task beyond modular addition satisfies its assumptions. Even on modular addition, the experiments predict the direction of change more reliably than the actual feature count, and the capacity bounds do not establish that training will find the predicted solution. I also found the proofs difficult to follow. My concerns about generality and practical value remain, I will maintain my rating.
Thanks for your response 7G2C!
Beyond modular addition
Proposition 17 is not specific to modular addition or abelian groups with a Fourier structure.
Its assumptions (and hence, its results at the optimum) hold for:
- Deep classifier heads on any N-class task (neural collapse is the s = 1 endpoint)
- Representing N items or features in a shared low-dimensional embedding, as in superposition and retrieval
Under them, Proposition 17 gives:
- The best margin is of order (ln N)^(1−s) (items 1 and 2).
- Every optimum uses O(log N) features (item 2).
Due to time constraints of the rebuttal process, we were unable to test whether Proposition 17 holds empirically for these tasks.
Training finding the predicted solution
As our paper is a theoretical analysis, the capacity bounds do not guarantee this.
As all of our bounds describe the optimum, whether gradient descent reaches it is a question of convergence to the global optimum of a non-convex problem which is open for this setting and a training dynamics problem.
The results we build on (Wei et al.) simply characterize the global minimizer and do not show that training will always find it.
Thanks a lot for the review 7G2C, the empirical observations were seriously insightful!
Q1 - Further Insights
The number of features is set by having to separate every pair of classes, and it is of order log N for N classes. This needs neither modular addition nor a group (the new Proposition 17 and Appendix H).
For any classification task with N classes, with features of bounded dimension priced by a concave power of their amplitude (s > 1), where some mix of features separates every pair of classes by a fixed amount:
On its own, the sparsity bias of multiplicative parameterization would give a single feature. That is what every optimum uses if the objective rewards the average separation rather than the worst pair of classes (Proposition 17, item 4). Having to separate the worst pair is what forces more features, and the count ends up at order log N.
Theorems 10 and 11 use the group only through how well a feature separates a pair of classes which is extensible far beyond just modular addition and its Fourier structure.
The exact optimum (Theorem 8) and the exact count K*, however, require the Fourier structure and hence, are not freely extensible outside.
Within modular addition the scope is also wider than the interference-free class: Corollary 12 holds for every network, interference included, and the new Propositions 15 and 16 extend the analysis to power networks and to networks with a shared embedding.
Q2 - Empirical Mismatches
Your concern is valid - a lot of runs don't actually hit the predicted K*. Per our diagnostics, it seems like failure to reach the optimum rather than interference is the culprit.
As before, we see that count moves with K*, b and ν in the expected directions (45 of 49 comparisons). K* has both wide and narrow plateaus - on the wide ones the count very often comes within 1 of K* (12 of 16 runs), and on the narrow ones it consistently falls below K* (7 of 8 runs).
Yet, in our observations, no run has beaten the interference-free class maximum margin (Theorem 8; 258 runs) or the best-found interference-free training loss optimum (Proposition 14; 196 settled runs, computed post hoc). Every run that comes within 2·10⁻⁷ of the best-found training loss optimum (19 of 19) has exactly its number of frequencies (which equals K* in 11 of them) and carries negligible interference. Further, amongst runs with identical configuration, runs carrying more interference are nearly always further from the optimum (211 of 233 pairs).
On the narrow plateaus, solutions with different counts are nearly tied (within 0.4% in loss), while our runs end 1% to 7% short of the best (8 of 8 runs), so the runs cannot distinguish them. This is what Proposition 13 says in general: below the optimum, the exact count is not determined.
For degree dependence at fixed s: in the settings we compare, the best-found optimum at the trained weight decay has the same number of frequencies at every degree (17 of 17 pairs), while higher-degree networks usually end further from it (16 of 17 pairs).
As our theory primarily centers on the optimum, we do not have a prediction of the exact count of a trained network.