acceptodds
Under review as a conference paper at ICLR 2027

Feature Sparsity via Margin Maximization

Abstract

Explaining the number of features in a network has been a long-desired result in mechanistic interpretability. [Morwani et al. (2024)](https://arxiv.org/abs/2311.07568v2) laid the baseline through their margin-maximization analysis, proving that quadratic networks maximizing margin under matched regularization use every frequency for modular addition. This restriction, however, excludes most models conventionally trained using regularization or a non-quadratic architecture, which are empirically sparse. Our inquiry focuses specifically on modular addition in polynomial and multilinear models. Our primary theoretical finding centers on the cost of new frequencies—a trade-off between the margin of the solution and regularization. For interference-free product networks, we use this to characterize the maximum normalized margin and an optimal number of frequencies for a particular configuration of homogeneity degree () and regularization. This recovers the dense frequency allocation seen in [Morwani et al. (2024)](https://arxiv.org/abs/2311.07568v2). Beyond the interference-free class of models, we also prove upper and lower bounds on the maximum normalized margin of product networks. We also establish matching bounds on the optimal number of frequencies, and prove that an optimal spectral allocation uses frequencies. Finally, we prove a general asymptotic scaling law for normalized margin over product models.

Primary area:
interpretability and explainable AI
Keywords:
feature learning, margin maximization, mechanistic interpretability, regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7G2COpenReview ↗
Rating
1
Confidence
3(+1)
Informativeness
1
Presentation
1(−1)
Soundness
1(−1)
Read review
Paper summary

The paper studies Fourier feature sparsity in modular addition under norm-constrained margin maximization. It characterizes optimal margins and frequency configurations for interference-free product networks, proves that every optimizer of the associated spectral allocation problem uses (\Theta(\log p)) frequencies for fixed (s>1), and establishes asymptotic margin bounds for product networks allowing interference. Training experiments examine how these predictions relate to learned frequency counts.

Critical strengths

The paper goes beyond the established observation that multiplicative parameterizations induce sparsity-promoting regularization. Its characterization of interference-cancellation costs and quantitative bounds on optimal spectral support provide concrete mathematical insight within the studied setting. The authors also distinguish ideal margin maximizers from solutions reached through training.

Critical weaknesses
  1. The broader significance of the theoretical advance remains limited. The exact characterization concerns a restricted class of networks on modular addition, while the feature-count theorem concerns the associated spectral optimization problem. These results refine an established research question, but I do not find that the paper sufficiently establishes a transferable principle or consequential implication beyond this setting. A new task or practical algorithm is not essential for a theory paper; nevertheless, the depth and reach of the demonstrated advance are insufficient for acceptance in my assessment.
  2. The connection to learned feature counts is not convincing enough to compensate for this restricted scope. The exact characterization relies on interference-free global optima, whereas trained networks can contain interference and need not reach those optima. Appendix E reports that the preregistered plateau-identity test matches only 2 of 19 converged runs, and networks with the same (s) but different degrees show systematic differences. Although the experiments support qualitative trends, they provide substantially weaker support for a quantitative explanation of the number of features actually learned. This limits the explanatory contribution without invalidating the conditional theorems.
Questions for authors
  1. What specific insight from the new results extends beyond modular addition and its Fourier structure, beyond the established sparsity bias of multiplicative parameterization? Please identify which parts of the analysis support such a conclusion.
  2. How should the experimental discrepancies—particularly the plateau mismatches and degree dependence at fixed (s)—affect the interpretation of the proposed explanation of learned feature counts? What existing evidence separates the effects of interference from failure to reach a global optimum?
3 comments

Thanks a lot for the review 7G2C, the empirical observations were seriously insightful!

Q1 - Further Insights

The number of features is set by having to separate every pair of classes, and it is of order log N for N classes. This needs neither modular addition nor a group (the new Proposition 17 and Appendix H).

For any classification task with N classes, with features of bounded dimension priced by a concave power of their amplitude (s > 1), where some mix of features separates every pair of classes by a fixed amount:

  • The best margin is of order (ln N)^(1−s) (Proposition 17, items 1 and 2).
  • Every optimum uses O(log N) features (Proposition 17, item 2).
  • An allocation on o(log N) features reaches only a vanishing fraction of the best margin (Proposition 17, item 3).

On its own, the sparsity bias of multiplicative parameterization would give a single feature. That is what every optimum uses if the objective rewards the average separation rather than the worst pair of classes (Proposition 17, item 4). Having to separate the worst pair is what forces more features, and the count ends up at order log N.

Theorems 10 and 11 use the group only through how well a feature separates a pair of classes which is extensible far beyond just modular addition and its Fourier structure.

The exact optimum (Theorem 8) and the exact count K*, however, require the Fourier structure and hence, are not freely extensible outside.

Within modular addition the scope is also wider than the interference-free class: Corollary 12 holds for every network, interference included, and the new Propositions 15 and 16 extend the analysis to power networks and to networks with a shared embedding.

Q2 - Empirical Mismatches

Your concern is valid - a lot of runs don't actually hit the predicted K*. Per our diagnostics, it seems like failure to reach the optimum rather than interference is the culprit.

As before, we see that count moves with K*, b and ν in the expected directions (45 of 49 comparisons). K* has both wide and narrow plateaus - on the wide ones the count very often comes within 1 of K* (12 of 16 runs), and on the narrow ones it consistently falls below K* (7 of 8 runs).

Yet, in our observations, no run has beaten the interference-free class maximum margin (Theorem 8; 258 runs) or the best-found interference-free training loss optimum (Proposition 14; 196 settled runs, computed post hoc). Every run that comes within 2·10⁻⁷ of the best-found training loss optimum (19 of 19) has exactly its number of frequencies (which equals K* in 11 of them) and carries negligible interference. Further, amongst runs with identical configuration, runs carrying more interference are nearly always further from the optimum (211 of 233 pairs).

On the narrow plateaus, solutions with different counts are nearly tied (within 0.4% in loss), while our runs end 1% to 7% short of the best (8 of 8 runs), so the runs cannot distinguish them. This is what Proposition 13 says in general: below the optimum, the exact count is not determined.

For degree dependence at fixed s: in the settings we compare, the best-found optimum at the trained weight decay has the same number of frequencies at every degree (17 of 17 pairs), while higher-degree networks usually end further from it (16 of 17 pairs).

As our theory primarily centers on the optimum, we do not have a prediction of the exact count of a trained network.

Thank you for the revision. Proposition 17 extends the logarithmic feature-count result to an abstract allocation problem, but the response does not show that a trained network on a concrete task beyond modular addition satisfies its assumptions. Even on modular addition, the experiments predict the direction of change more reliably than the actual feature count, and the capacity bounds do not establish that training will find the predicted solution. I also found the proofs difficult to follow. My concerns about generality and practical value remain, I will maintain my rating.

Thanks for your response 7G2C!

Beyond modular addition

Proposition 17 is not specific to modular addition or abelian groups with a Fourier structure.

Its assumptions (and hence, its results at the optimum) hold for:

  • Deep classifier heads on any N-class task (neural collapse is the s = 1 endpoint)
  • Representing N items or features in a shared low-dimensional embedding, as in superposition and retrieval

Under them, Proposition 17 gives:

  • The best margin is of order (ln N)^(1−s) (items 1 and 2).
  • Every optimum uses O(log N) features (item 2).

Due to time constraints of the rebuttal process, we were unable to test whether Proposition 17 holds empirically for these tasks.

Training finding the predicted solution

As our paper is a theoretical analysis, the capacity bounds do not guarantee this.

As all of our bounds describe the optimum, whether gradient descent reaches it is a question of convergence to the global optimum of a non-convex problem which is open for this setting and a training dynamics problem.

The results we build on (Wei et al.) simply characterize the global minimizer and do not show that training will always find it.

Reviewer_sgH1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors theoretically analyze the number of features in a network trained on modular addition. The authors characterize a trade-off between maximizing the margin and controlling the cost of regularization. Based on that, the authors provide an explanation for the sparsity of features in the network. Besides, the authors introduce a scaling law for the optimal number of features as the modulus increases in modular addition.

Critical strengths
  1. Establishing the theoretical bound on the optimal number of features in a neural network is an interesting topic, and the current understanding is still limited. This work makes valuable contributions to the literature with a new explanation of the sparsity of learned features.
  2. The paper is organized well; the main insights and the explanation of the theoretical analysis are clear.
  3. The theoretical analysis seems solid, and most of the assumptions are reasonable. The experiments cooperate well with the theoretical analysis.
Critical weaknesses
  1. The theoretical analysis in this paper is solid and interesting. However, is it possible to provide some guidance for model training in practice? For example, does the trade-off bring some additional insights?
  2. If I understand correctly, the main theoretical analysis focuses on the optimal solution of the objectives. However, we usually cannot obtain optimal solutions in real-world datasets. Consequently, can we obtain similar results when the network approaches the optimal solution?
  3. Most of the theoretical analysis is based on toy models. I understand that the theoretical analysis in a powerful model is difficult. However, I think it would be better to verify the main insights (theoretically or empirically) in more complicated models.
Questions for authors

See Weaknesses.

1 comment

Thanks so much for the review sgH1, the near optimum stuff really helped clarify a bunch of empirical stuff!

W1 - Training Guidance

Yes - for this family of models the trade-off answers the problem of representational capacity by providing a theoretical bound on the minimum capacity required to reach the best margin.

The best margin is reached with ν neurons per feature and K* = Θ(log p) features, so a width of ν·K* suffices (Theorems 8 and 11). With a shared embedding, an embedding dimension of 2·K* suffices (Proposition 16).

Beyond that size, extra width or embedding dimension cannot raise the best margin.

This is exact within the interference-free class and holds up to a constant factor for every network (Theorem 4; Proposition 16 for shared embeddings).

Well below that size the margin is lost: a network on o(log p) features reaches only a vanishing fraction of the best margin, whatever its width (Corollary 12).

This insight, however, is theoretical and does not guarantee that training will actually find the optimum when the model is sized at its minimum.

W2 - Near Optimum

Yes - similar conclusions hold for a network that only approaches the optimum, except the exact count (the new Propositions 13 and 14, Appendices I and J).

For any network close to the optimum:

  • A bounded number of large features do almost all the work while smaller features carry much less weight (Proposition 13).
  • It still needs order log p features: on fewer, a network reaches only a vanishing fraction of the optimum (Corollary 12).
  • The exact number of non-zero features is not determined. Arbitrarily small extra features cost arbitrarily little, so a count only makes sense above a size threshold (Proposition 13).

At the weight decay actually trained with, λ > 0 instead of the limit λ → 0, we now have the optimum of the training loss itself, and there small features are removed (Proposition 14).

W3 - Complex Models

Yes on the theory side - we have managed to extend our analysis to more model families.

For power networks, which include the quadratic network of Morwani et al., we have proved the exact optimum for two operands among interference-free networks whose neurons each use a single frequency.

For networks with a shared embedding layer, we have proved the exact optimum for 2 to 7 operands among interference-free networks whose embedding does not superpose frequencies.

Further, the logarithmic count of features does not need modular addition or a group at all: for any classification task with N classes whose features have bounded dimension and are priced concavely, the optimum uses Θ(log N) features (Proposition 17).

Reviewer_ddUnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper provides a detailed theoretical study on feature count and provides multiple novel results, most notably the finding that a single programme determines the feature count as a concave programme in s= v/b. the paper also proves a two-sided Θ(log p) result for the margin and feature count under the concave budget which is revealing. experiments in the paper, however limited, provide sufficient coverage for the theoretical trends.

Critical strengths
  • the paper does a great job in aggregating results/ approaches from prior work and assembling them into a coherent theoretical framework that gives several interesting results.
  • the thereotical techniques used such as averaging over inputs, quasi-norm price of a product, random cayley graph expansion, roots-of-unity cancellation blocks all exist separately in Morwani, Gunasekar, Alon-Roichman, Tian/Marchetti/He. the authors did a great job coming up with a single programme that depends on (ν, b) only through s = ν/b, optimality proof for the cancellation (lemma 7), and the two-sided Θ(log p) bounds under the concave budget
  • the theory is solid, careful, and the message is clear that the feature count is set by s, independent of task.
  • the inference-cost lemma has the equality characterization that makes it a solid technical contribution, possibly widely useful.
  • the experiments include a sweep over v, b, and p with product networks GD-trained. many trends observed here are consistent with the theory and is a good validation.
Critical weaknesses
  • whether interference can beat interference-free value for b<v remains open, and some more discussion on this would be useful
  • perhaps my biggest qualm is that the exact results are for product networks only. I’d be curious if the authors have thoughts on the lower bound for power networks (including Morwani’s quadratic network) for b<v.
  • also, networks with a shared embedding (common practice) are out of scope here to my understanding, even though the empirical sparsity arises there. I wonder if the authors have thoughts about this.
  • theorem 11’s constants are vacuous, esp at realistic p. and the log laws are only asymptotic
  • experiments only validate the trends and not the exact counts. would’ve been nice if there was some empirical validation to resolve the fine plateaus of K*
Questions for authors

see weaknesses

4 comments

Thanks so much for the review ddUn, a lot of your suggestions have been very instructive and have enabled some big extensions to the paper!

W1 - Interference vs Interference-free

Yes - the interference can beat the interference-free value: at p = 3 and b = 2, product networks carrying interference beat the interference-free value at every degree ν from 12 to 19, by a factor of up to 2.2 (Appendix D).

To us, this seems to be a small-prime phenomenon that does not generalize to larger primes.

For networks built on a single frequency, we can prove that the interference a network can carry is at most c_ν sin(π/p) ≤ (2ν−1)π/p times the clean amplitude it gives up for it, so the room for interference shrinks like 1/p (Lemma 34). Past a threshold in p it cannot pay at all: under L2 weight decay, no single-frequency network beats the interference-free value for any prime p ≥ 13, 23, 37, 59 at ν = 3, 4, 5, 6 (Theorem 36).

For networks that use several frequencies and carry interference the question is open; we so far know of no such network that beats the interference-free value at the degrees and primes of our experiments.

Either way, the order of the optimum is unchanged: Theorem 4, Theorem 10 and Corollary 12 hold for every network, so interference can raise the optimum by at most a factor ν(s−1)\nu{}^{(s−1)}, independent of p, while a network on o(log p)o(\text{log p}) frequencies still attains a vanishing fraction of it.

W2 - Power networks

Yes - similar methods apply to power networks (everything in the new Prop 15 and Appendix F).

For two operands (quadratic nets), we have proved the exact optimum among interference-free networks whose neurons each use a single frequency (analogue of Theorem 8): it is 6(−(ν−b)/b)⋅2BpΓs(p)6^{(−(ν−b)/b)} · 2B_{p} Γ_{s}(p). The optimum uses the same K* frequencies as a product network, and for b < 3 each frequency is carried by exactly six neurons. Without the single-frequency requirement, the interference-free optimum is at most a factor 2^(s−1) ≤ √2 above this.

For the quadratic network of Morwani et al. specifically (two operands, ν = 3): at their norm (b = 3) we recover their result that every frequency is used. Under plain L2 weight decay (b = 2), the optimum in our class is 6(−1/2)⋅2BpΓ3/2(p)6^{(−1/2)}·2B_{p} Γ_{3/2}(p) and uses only K* frequencies - 5 at p = 59 and 7 at p = 113 (Table 1) - not all of them.

For networks with n operands (polynomial nets), we have proved a two-sided bound on the optimum in the same class (analogue of Theorem 8 up to a constant), with the same allocation of amplitudes across frequencies. For sufficiently wide power networks, interference included, the optimum has rate Θ(p^(−ν/2)(ln p)^(1−s)); reaching a constant fraction of it needs Ω(log p) features, and O(log p) suffice (analogues of Theorem 10 and Corollary 12).

The exact optimum for three or more operands, and for neurons that mix frequencies, is still open.

Unfortunately, due to time constraints, we were not able to test these new results empirically.

W3 - Shared embeddings

Yes - again, similar methods apply to networks with a shared embedding (the new Prop 16 and Appendix G).

The shared table multiplies every operand, so it acts as extra degree: s becomes (2n+1)/2, which pushes toward fewer features.

For 2 to 7 operands, we have proved the exact optimum among interference-free networks whose embedding does not superpose frequencies (analogue of Theorem 8): the optimal network uses the same frequencies as the programme at that s, carries each on ν neurons, and its embedding places the tokens on a circle for each frequency.

For > 7 operands the same value holds up to a constant factor. Every shared-embedding product network, superposition included, has an optimum of order p^(−ν/2)(ln p)^(1−s) (analogue of Theorem 10), so superposition can change the constant but not the rate.

The exact optimum with superposition, and ReLU networks, are still open.

Again, these new results could not be empirically tested due to time constraints.

W4 - Vacuous constants

To get our constants to bite a bit harder, we've used a different proof method (Minkowski's convex-body theorem in place of the Bessel estimate) that gives us tighter bounds that bite at the primes we train on. Theorems 10, 11 and Corollary 12 now hold for every odd prime with explicit constants (they are sadly a lot uglier and dense now):

submittedrevised
lower bound on the count (s = 2)ln p / 665ln p / 4
upper bound on the count (s = 2)36 ln p1 + 8 ln p

The revised lower bound holds for every s ≤ 5/2.

Every reported K* lies squarely within these new tightened bounds.

W5 - Empirical Plateaus

Our experiments validate the trend but do not resolve the fine plateaus of K*.

On the wide plateaus the count very often comes within 1 of K* (12 of 16 runs), but on the narrow ones it consistently falls below K* (7 of 8 runs).

Per our findings, this is because trained networks get near the optimum but mostly not to it. On the narrow plateaus, solutions with different counts are nearly tied, while our runs end well short of the best so the runs cannot distinguish between them. Below the optimum, as Proposition 13 shows, the exact count is not determined.

We do have a match where a run comes within 2·10⁻⁷ of the best-found optimum of the training loss (Proposition 14): every such run (19 of 19, all at smaller primes) has exactly the number of frequencies of that optimum, which equals K* in 11 of them.

Resolving the fine plateaus would need training that reaches the optimum at those settings.

thanks for the thoughtful responses! I do believe these new results (provided the empirics are well established in the final version) make the paper much stronger. I would keep my original positive score of 3.

Hi ddUn, thanks so much for your response and help throughout this review process!

It's been great working with you on this paper and your feedback has genuinely pushed this paper so far!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes made in the revision

Thank you to all three reviewers - your inputs made a massive difference and this paper is much better for it!

Here are the major changes we've made to the paper:

Empiricals

Every run was retrained under a longer protocol with a stricter stopping rule: a second phase with a growing learning rate, and a run counts as settled only when its relative gradient is at most 10⁻⁵ (10⁻³ in the submission) and its loss and margin are stable to ensure correct counts were captured and not restricted by a training period that was too short.

We noticed divergences and a surprising number of runs that hadn't settled so we made these changes for better stability.

Every empirical number in Section 6 is hence, recomputed from the new settled runs and differ slightly from the original.

Theory

  • The conjecture that interference never beats the interference-free value for b<νb < \nu{} (Conjecture 20 of the submission) is false (Appendix D).
  • Non-vacuous constants for Theorems 10, 11 and Corollary 12.
  • Near-optimal networks (Proposition 13, Appendix I): what can and cannot be said about the count below the optimum.
  • Non-zero weight decay (Proposition 14, Appendix J): the optimum of the loss that is actually trained.
  • Power networks, including the quadratic network of Morwani et al. (Proposition 15, Appendix F).
  • Networks with a shared embedding (Proposition 16, Appendix G).
  • The count law without a group (Proposition 17, Appendix H): order log N features for N classes.

Propositions 15 to 17 form the new Section 5 with the experiments now in Section 6. Propositions 15 to 17 have not been tested empirically.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.