acceptodds
Withdrawn from ICLR 2027

We should Understand Symmetry Breaking in Sparse Training to Unmask the Role of Overparameterization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Overparameterization is a cornerstone of deep learning. When combined with weight-space symmetries, it leads to many equivalent solutions in the loss landscape, making it easier for an optimizer to find a good solution. However, breaking these symmetries — as in Sparse Neural Networks (SNNs) — reveals fundamental limitations in our training algorithms, present in both sparse and dense settings. Although in the literature sparse training research has mostly been motivated by making training efficient, in this position paper, we argue that sparse training with a fixed mask provides a valuable testbed for studying symmetry, optimization, and their interplay in a principled manner. We explore novel examples of how breaking permutation symmetry and rescaling invariance jointly affect optimization, and can inspire algorithmic advancements. Sparse training research can provide valuable insights into the role of symmetries and overparameterization in training in general, not limited to sparse training. This position paper argues that understanding how symmetry breaking can improve or hinder training could uncover deeper theoretical insights and inform the design of more robust optimizers and architectures that respect symmetries.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
overparameterization, weight-space symmetries, sparse neural networks, sparse training, position paper

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.60 over 5 reviews, above 10% of the others.
Reviewer_qejTOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper seeks to emphasize the utility of sparse neural network training as a lens for the study of overparameterization. It is framed as a position paper. AFAICT it is submitted as a main conference paper, and not to any "position paper" specific track.

The paper's goal seems to be to stimulate study in the intersection of lottery tickets, overparameterization, weight-space symmetries, and sparse neural networks. The paper synthesizes results from the existing literature to make the case that these concepts are intimately related and should be studied cohesively, esp. towards an understanding of overparatemerization.

Critical strengths

The overall motivation of the paper is sound, and the paper invokes some of the appropriate tools (orthant selection, gradient flows). I found the gradient flow argument to be most interesting. I would be keen to see a larger study of this effect in larger models, and realistic optimizers. It would be esp surprising/interesting if in fact standard overparameterized training sends the majority of weights to near zero.

I would be interested to see more work in the area of sparsity, lottery tickets, and overparameterization. Meaningful connections that can be made between these concepts would be an exciting development IMHO.

Critical weaknesses

The paper misses key related work, including ones that are central to the narrative of the paper. Two that immediately come to mind:

  1. Ainsworth et al (2023) "Git Re-Basin: Merging Models modulo Permutation Symmetries" is directly relevant to "Can we find a permutation to correct the alignment?" paragraph of section 3.1, "Weight-space symmetries." paragraph in section 2, "Permutation Symmetries." paragraph in section 3.2.
  2. Zhou et al (2019), “Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask.” directly addresses the main orthant question in this paper. It is not clear to me that this paper is contributing anything that Zhou et al has not already covered.

There are likely others, but those are the ones that stick out to me in the moment.

One of the central claims of the paper is that overparameterized NN training is a process of "orthant selection", i.e. picking whether weights should be positive or negative, more than picking any concrete value for them otherwise. But this argument has already been made in Ramanujan et al (2020), Frankle, Schwab, and Morcos (2020), and Zhou et al.

There are also substantial misunderstandings in the purported theoretical contributions. Issues with Theorem 3.1:

  1. In the theorem O_{sub, j} for j \in [N] is indexed on the N hidden units, but O_sub is introduced in the beginning of the section as being indexed over all dimensions of weight space, with no relation to hidden units. The authors confuse weights with neurons/units. LTH literature randomly masks individual weights, not hidden units.
  2. O_sub^* is not defined AFAICT
  3. Even if we charitably look past the disconnect between the theorem statement and its purported implications, the theorem is sufficiently trivial that it does not add anything useful to the conversation. Stated another way, the theorem essentially boils down to a "balls in bags" question, ie if I have B balls in a bag, R of which are red, and I draw N balls out of the bag without replacement, what is the probability that i draw all R red balls.

Furthermore, the gradient flow argument in section 3.3 is interesting but flawed. The argument is sound as far as switching the sign of a_j,t. But the argument for w_j (starting on line 354) is incorrect: gradient flow can flip the signs of entries of w_j without breaking the ||w_j|| constraint or getting stuck in a critical point. Note that a single zero in the w_j vector can still have non-zero derivative.

One final concern: part of the central messaging of the paper is that sparse masking breaks weight space symmetries and that this effect can shed light on overparameterization. But I fail to see how the paper actually "lands the plane" on this point. What is this final relationship? There is some discussion of these symmetries and there is some discussion of overparameterization, but there is not a satisfying bridge built between the two IMHO. This is unfortunate to me because I believe this may be an interesting research direction, but I find the connection made in this paper is vague and unsatisfying.

Misc.

  • line 365: \nabla_j notation is confusing and undefined afaict
  • subsection 4.3 "UNDERSTANDING THE ROLE OF SYMMETRIES IN TRANSFORMERS" does not fit with the other 2 subsections in section 4.

"Introducing sparse masks breaks symmetries in a sense that the neurons become more specialized and less likely to match specific target neurons. Furthermore, the plasticity of sparse neurons suffers more from the rescale symmetry, constraining them to their initial orthant, as they hardly flip weight signs and thus can seldom assume different roles. In summary, a combination of reduced permutation symmetry and plasticity induced by an active rescale symmetry turns sparse training with a fixed mask into a hard problem"

This paragraph is vague, imprecise, and possibly untrue. It is mathematical fact that applying a random/inconsistent weight mask breaks permutation symmetries. why not just say that directly?

line 32: "Sparse training, in contrast to dense training, uses models with sparse connections or weight matrices, and is a promising direction towards reducing the computational cost of training."

there is still no viable path towards efficient sparse training on common accelerators. any promises made to the contrary are so speculative as to be unserious imho. the focus should be on the overparameterization investigation.

"Well-known reasons to study sparse training with a fixed mask."

paragraph is mostly fluff. I am willing to accept the argument that in some hypothetical hardware future we may be able to realize advantages of a sparse NNs, but the rest of this paragraph does not actually add anything of value IMHO. it also does not support the message of the paper, i.e. that sparse NNs offer a unique method to understand overparameterization. The following "Beyond the reasons that make STFM practically relevant" paragraph is much better.

"We argue that insights from this perspective could guide the design of more trainable and efficient neural networks that rely less on massive overparameterization."

this sentence is weak and unnecessary. studying overparameterization is interesting in and of itself, and at present there is no known viable path to making fixed-mask sparse training more efficient on accelerators than standard training.

"Overparameterization exponentially increases the probability of starting in the correct orthant. Sparsity reduces this probability, which makes it difficult for the optimizer to find a good solution."

what does this mean exactly? the "correct" orthant is only relevant for sparse models. the exact takeaway is unclear.

"Gadhikar et al. (2025) proposed using the signs from a trained model that has converged to a good solution."

using them how?

"Winning tickets succeed by preserving the symmetry-breaking configuration induced by overparameterization, not by identifying intrinsically important weights."

this is confusing: what precisely is a "symmetry-breaking configuration" and how does overparameterization induce one?

"Overparameterized networks usually require less of this movement."

this is argument is made formal in eg NTK literature

(Results from (Gadhikar et al., 2025))

should be \citet

Questions for authors

what are you trying to say in this paper that has not already been done in prior work? the paper title gives us something concrete, but the contents of the paper cover a bunch of different things, loosely connected. what exactly is the research question? how does this fit together into a coherent story? what do you want readers to understand after reading this paper? what research questions do you want readers to be thinking about after reading this paper? what does this paper actually add to the conversation?

Reviewer_dMuUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This position paper argues that Sparse Training with a Fixed Mask (STFM) serves as a valuable diagnostic testbed to understand overparameterization and optimization dynamics. By examining permutation and rescaling symmetries, the authors claim that fixed masks break permutation symmetry—making initialization in the "correct orthant" exponentially rare—while the balance equation hinders necessary sign-flipping across orthants during gradient flow.

Critical strengths

Insightful Perspective: Reframing fixed-mask sparse training as an analytical tool rather than just a compute-saving trick offers a fresh angle on optimization theory.

Unified Intuition: Successfully links permutation symmetry and the gradient-flow balance equation to explain the fundamental failure of pruning-at-initialization (PaI).

Critical weaknesses
  1. The core arguments heavily rely on and synthesize recent papers from a single research line, with the paper's own derivations largely limited to simple combinatorial counting.

  2. Theoretical demonstrations and trajectories rely on scalar or single-neuron setups (N=1,d=5N=1, d=5), failing to rigorously capture real deep architectures with LayerNorm, residual branches, and stochastic noise.

  3. The future directions in Section 4 read more like a selective literature survey rather than providing concrete, falsifiable algorithmic design rules.

Questions for authors

How strictly does the rescaling balance equation impede sign-flipping in modern deep networks where LayerNorm/RMSNorm dynamically rescales layer gradients?

In Appendix Table 3, why were attention blocks frozen during DeiT retraining, and how does full attention tuning affect orthant alignment?

Does the combinatorial bottleneck in Theorem 3.1 hold under practical moderate sparsity levels (e.g., 50%–70%)?

Reviewer_CK1WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This position paper argues that sparse training can serve as a valuable testbed for understanding the interplay between symmetry, overparameterization, and optimization in neural networks. In particular, the authors suggest that the difficulty of sparse training can be partly attributed to symmetry breaking, which reduces the number of equivalent solutions and makes optimization more sensitive to initialization.

Critical strengths
  1. The paper raises an interesting question about the role of symmetries in neural network optimization and proposes sparse training as a useful setting for investigating this question.

  2. The connections drawn between permutation symmetry, rescaling invariance, and the trainability of sparse networks provide an interesting perspective on existing results.

Critical weaknesses
  1. The theoretical arguments in Section 3.2 are not sufficiently convincing. In Theorem 3.1, it is unclear why the relevant condition should involve the union of the neuron subspaces rather than their linear span. Moreover, Theorem 3.1 and Corollary 3.2 do not convincingly demonstrate how overparameterization alleviates the orthant alignment problem.

  2. The discussion of orthant transitions in Section 3.3 is incomplete. The authors argue that rescaling invariance makes sign changes difficult and thereby prevents the optimizer from moving between orthants. However, the main argument relies on the balance equation under balanced initialization. Furthermore, even under balanced initialization, the balance equation does not necessarily prevent the first-layer weights from changing orthants. A weight vector (w) can move continuously to (-w) without passing through the origin.

  3. The paper could benefit from a more detailed discussion of existing work showing that breaking parameter symmetries does not necessarily harm optimization and may even improve performance, e.g. in Lim et al. (2024). It would be interesting to discuss how such findings fit into the proposed perspective.

Questions for authors

See weakness.

Reviewer_6Go5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper aims to substantiate the position that successful sparse training with a fixed mask should not be merely an efficiency goal, but an ablative diagnostic tool for explaining the success of modern, overparametrized neural architectures. In particular, it is argued that permutation-symmetry breaking by sparse masks, in conjunction with rescaling symmetries, is a candidate cause of modern optimizers' struggle to find well-generalizing solutions during sparse training with a fixed mask.

To build the argument, the authors start from the results of Adnan et al. (2025) and Gadhikar & Burkholz (2024), which suggest that correspondence between mask alignment and weight sign at initialization with those of the "good solution" (or winning lottery ticket) is enough for optimization to recover the good solution. Further, they argue that such an alignment is more likely when losses exhibit a plethora of good solutions (and thus "good orthants"), which happens in overparametrized regimes via symmetries. A theorem and corollary claiming to quantify the probability of subspace and orthant alignment are provided for a one-hidden-layer teacher-student setup. At the same time, it is posed that rescaling symmetries can hinder correction of a bad orthant alignment at initialization. Based on these observations, the paper proposes some broad directions of research aimed at understanding the failure modes of network training and potentially improving optimization methods under sparsity.

Critical strengths
  • The paper addresses a relevant topic for the community
Critical weaknesses

To my understanding, a position paper should:

i) Perform a thorough and rigorous review of the topic's literature

ii) Coalesce the observations into a coherent thesis (optionally constructing additional formal/empirical support)

iii) Discuss possible counterarguments in the literature and show that a significant part of the thesis survives them

iv) Identify and concretely outline future explorations that can confirm/infirm the surviving part of the thesis

This sequentiality means that weakness in either i) or ii) would severely affect the paper. This is the case here, due to a) an absence of rigor in interpreting existing results in all their relevant details, often overstating their implications; b) ambiguous and/or vague use of concepts, which is particularly problematic in DL where results are highly dependent on architectural/data/algorithmic assumptions; and c) too vague/broad proposals for future exploration. See details in the next section.

Questions for authors
  • Related work treatment. A position paper's strength draws principally from a deep and rigorous of treatment of existing works, which should be apparent in the text. With this in mind,

    • Lines 87–90: The statement that "most PaI methods achieve performance comparable to random pruning" is insufficiently qualified and not reflective of the cited experimental evidence. Specifically, Frankle et al., 2021b empirically show that PaI methods, while comparable to naive random pruning at trivial and even matching sparsities, significantly outperform the latter at large sparsities, with all methods underperforming post-training pruning and dense trained networks. Complementarily, Liu et al., 2021 show that performance of several different approaches to random pruning at initialization grows with network size (width and depth), approaching dense performance. However, the setting in Frankle et al is not reproduced again here, so we do not know how the methods compare in this new regime (with the exception for Grasp in Figure 8 for CIFAR-10 and a Resnet-20 for sparsity <= 90%. However, the competitive advantage of Grasp in Frankle et al., 2021b is obtained >= 95%). Finally, Gadhikar et al., 2023 provide theoretical backing for the perfomance scaling observation of Liu et al., 2021. In short, the citations do not appear to sustain the claim and further qualification is needed.

    • Line 151: The statement "We highlight that for sparse training this prevents moving between orthants" overextends the results of its supporting citations, which address only single neuron settings. In particular, the balance invariant does not generally prevent individual input weights of neurons with multiple active coordinates from changing signs. The scope of the claim should be qualified.

    • Line 159: The statement "the balance equation forces consecutive layers to stay small introducing a bias towards sparsity or low rank" is inaccurate. Balancedness is compatibla with diverging weight norms (see classification with cross-entropy-trained networks [3]). What remains small is the difference in squared norms between consecutive weights.

    • Lines 203, 218-220: The authors refer to winning tickets as having the property that "when trained from scratch, match the dense model performance". Based on follow-up works on the LTH like [6, Figure 1 & Section 2], this observation does not extend to larger models which are the topic of this manuscript. The authors refer to this in the related work section, but break consistency in later paragraphs. The statements should be qualified accordingly.

    • Line 161: The claim "[...] we argue that it [implicit regularization] is detrimental for sparse training" is not substantiated by subsequent arguments. What the authors discuss is that a conservation law (balancedness) arising from rescaling symmetry can obsrtuct certain optimization trajectories. However, balancedness is not synonymous with implicit bias.

  • Argument construction. The main argument is covered in Section 3, with the intended logical flow being: good (subspace/orthant-aligned) initialization is important for sparse training + fixed sparse masks make good initialization less unlikely + correcting a bad initial orthant (via sign changes) can be difficult => overparametrization makes good initialization more likely, by increasing the number of symmetry-induced good minima, and thus the number of good subspaces likely to be captured by sparsity masks at initialization. This argument, unfortunately, relies on several implications that are insufficiently justified or contain errors.

    • Section 3.1: The cited experiments show that sign is sufficient for good training. However, the manuscript later phrases orthant alignment as necessary (lines 231+, as well as line 054), which is not established by the cited literature. Furthermore, this section should discuss how the seemingly contradictory results of [7] integrate with the manuscript's claims (that random PaI with high sparsity can match the performance of dense training as long as width/depth grows), and what potential experiments/developments could help settling this.

    • Section 3.2:

      • First, the theorem is stated as an optimization-related result ("we show that the probability is low that we initialize a sparse mask such that SGD is able to learn a solution", "The probability of a random global layer mask [...] to recover the target"), when it should be a statement on representational capacity. No part of the proof addresses optimization aspects (i.e., recovery).

      • Second, the event whose probability is claimed by Thm 3.1 is different from the event whole probability is computed in the proof, making the current result erroneous. The proof computes the probability of the event that each of the mm non-zero features of the target neuron is covered by at least one of the neurons of ff. However, this event is different from the subspace inclusion of the theorem statement, as follows. Take d=3d = 3, m=2m=2, N=2N = 2, mask M=(100010)M= \begin{pmatrix}1&0&0\\\\0&1&0\end{pmatrix} (and therefore ℓ=m=2\ell = m = 2) and assume the target neuron is a⋆=1a^\star = 1, w⋆=[1,1,0]w^\star = [1,1, 0]. Then, every feature of the target is covered by the union of ff's neuron's mask supports $\mathrm{supp}(w^\star) = \{1,2\} = \{1\}\cup\{2\} = \mathrm{supp}(m_1) \cup \mathrm{supp}(m_2) $ (the event whose probability is quantified in the proof holds). However, the event whose probability is claimed in the theorem does not hold, since Osub⋆=span{e1,e2}⊈span{e1}∪span{e2}=Osub,1∪Osub,2\mathcal{O}^\star_{\text{sub}} = \mathrm{span}\\\{e_1, e_2\\\} \nsubseteq \mathrm{span}\\\{e_1\\\} \cup \mathrm{span} \\\{e_2\\\} = \mathcal{O}_{\text{sub}, 1} \cup \mathcal{O}_{\text{sub}, 2}.

    • Section 3.3 dedicates most of its space to restating the single-neuron theoretical construction and results from (Gadhikar & Burkholz, 2024) and Gadhikar et al. (2025). While it is acceptable for a position paper to restate existing results, the end goal should be providing a new insight, which is lacking here. In particular, claiming that sparse networks have difficulty with sign flipping during training due to rescale symmetries, based only on a one-neuron example trained with gradient flow and a restrictive single-neuron experiment, is an overextension. A nuanced discussion is missing, including but not limited to a) a detailed account of existing empirical observations and their context when quantifying sign flips (e.g., experiments quantifying sign flips in dense vs sparse networks during training); b) a comment on the practical training regimes which often use relatively large learning rates and stochasticity, which significantly depart from gradient flow (see the edge of stability literature [8]).

      Other comments:

      • based on the comments above, the abstract statement "We explore novel examples of how breaking permutation symmetry and rescaling invariance jointly affect optimization, and can inspire algorithmic advancements" is an overclaim since neither the examples or their treatment are novel.
      • the gradient magnitude bound is misleading as a measure for optimization stalling, since it concerns the gradient of the network output rather than the training loss (and so excludes residual magnitude or evolution)
    • Finally, the implication proposed after Section 3's construction is that "more parameters => more symmetries => more good solutions => easier optimization" (e.g., line 194). However, this argument is not supported by existing literature without further qualifications. In [4] it is shown that in the mildly overparameterized regime, symmetry-induced critical subspaces outnumber the affine subspaces of the global-minimum manifold, with the relationship reversing for high overparametrization. So increasing width can increase the number of other critical structures besides global minima. Moreover, without a characterization of optimization trajectories or attraction basins, and generally without more nuance in this argument, the claim of better optimization is insufficiently justified.

  • Proposed research agenda. The directions outlined in Section 4 are stated with too high generality and fall short of outlining an actionable research agenda. In particular,

    • Section 4.1 presents DST and notes that "leveraging weightspace symmetries could guide DST [...] improving convergence and generalization by enabling efficient, structured exploration of orthants". This connection is vague and would benefit from a) a description of the basis on which it stands and b) a more precise and concise formulation (e.g., identify the step within the DST procedures that symmetries could inform, and hypothesize an approach to symmetry-informed DST).

    • Section 4.2: the first paragraph amounts to reviewing recent works aimed at improving sparse training, but outlines no research direction. The second paragraph's direction for further research is again vague: "a unified, principled framework for sparsity-aware optimization [...] is crucial, as it can reveal the fundamental role of symmetry and connectivity in optimization". What aspect of the role of symmetries do the authors hope, for example, that such a unification could help explain? Furthermore, the (Ziyin, 2023) citation seems misplaced without further explanation --- do the authors hope to import some form of those results to sparse training via unification? Can the authors make a precise statement of their hypothesis? Finally, the last paragraph is again not informative enough to be actionable.

    • Section 4.3 requires again further qualification relative to the preceding sections of the paper. In particular, the notion of overparametrization the authors seek to explore in transformers needs to be made precise, since here the classical definition of # data points < # parameters fails for language models. Furthermore, the scope of what the authors hope to understand about the role of symmetries here needs to be made precise: what experiments or concrete hypotheses do the authors think should be tested, and what would be their implications?

  • Concept usage and wording. For a position paper, wording and concept clarity are paramount, since they form the basis of the arguments. With this in mind,

    • Several places in the paper use "overparametrized" as a synonym for dense, or at least as the antonym of sparse, e.g., lines 71-75, 195-196, 204-205. If we take overparametrization to be the usual "# params > # training samples", then a sparse network can still be overparametrized, and a dense network can be underparametrized. The authors should settle on the operational definition of overparametrization they intend to use in this work and lay it out in the intro for clarity.

    • Phrases like "solve the optimization part b)" should be avoided. From the context, the authors are referring to closing the performance (learning/generalization) gap between sparse and dense training. "Solving the optimization" would generally point to finding a minimizer/stationary point of the loss. However, the two are not interchangeable (e.g., a network can be optimized to zero loss yet fail to generalize, or conversely, well-generalizing networks needn't represent a training loss minimizer [2]). Can the authors characterize precisely what they mean (in terms of metrics and citations) that "currently, we cannot solve the optimization part (b)"? In parallel, can the authors define precisely what they mean by "solving sparse training with fixed mask" (with assumptions, eval. metrics, etc.)?

  • Citation practice.

    • The intro section lacks citations for most qualifications it makes. One glaring example is in paragraph "The interplay between symmetries and trainability" at lines 112-120, where strong citations should support the claims in the 1st, 2nd, 4th and 6th sentences. Without such citations, this paragraph has little argument value. This type of reference omission is present throughout the paper and significantly weakens the construction of the perspective.
    • Citations of a method's/idea's originating work(s) should appear on first reference (e.g., LTH and PaI and others are introduced without citations)
  • Grammar/prose.

    • lines 096-103: misuse of punctuation, enumeration should be separated by ";"
    • line 155: misplaced ":" in "[...] studied extensively and is also known as: implicit regularization"
    • Line 157: incomplete, verbless sentence, "In other words, the tendency of the neural network training without explicit regularization."
    • Line 154: subject-verb agreement error
    • Lines 246, 380, 392, 408-409, and 419: referring to sections, figures, tables should be capitalized (e.g., Table 1, Section 1, etc.)
    • the explanation of orthant as a generalization of 2D quadrant is unnecessarily repeated (lines 53, 260-261, and 263-264)
    • inconsistency of sign, lines 262 vs 272, includes 0 vs doesn't
    • Lines 179-182: sentence/idea repetition in the first two sentences
    • Several places: consistency of punctuation around i.e. and e.g.

[1] Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Pruning neural networks at initialization: Why are we missing the mark? In International Conference on Learning Representations, 2021b.

[2] Zhang, Jingzhao, et al. "Neural network weights do not converge to stationary points: An invariant measure perspective." International Conference on Machine Learning. PMLR, 2022.

[3] Chizat, Lenaic, and Francis Bach. "Implicit bias of gradient descent for wide two-layer neural networks trained with the logistic loss." Conference on learning theory. PMLR, 2020.

[4] Simsek, Berfin, et al. "Geometry of the loss landscape in overparameterized neural networks: Symmetries and invariances." International Conference on Machine Learning. PMLR, 2021.

[6] Frankle, Jonathan, David J. Schwab, and Ari S. Morcos. "The early phase of neural network training." arXiv preprint arXiv:2002.10365 (2020).

[7] Liu, Shiwei, et al. "The unreasonable effectiveness of random pruning: Return of the most naive baseline for sparse training." arXiv preprint arXiv:2202.02643 (2022).

[8] Cohen, Jeremy M., et al. "Gradient descent on neural networks typically occurs at the edge of stability." arXiv preprint arXiv:2103.00065 (2021).

Reviewer_1mHVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper uses fixed-mask sparse training to study why overparameterized networks are easier to optimize. The explanation has two parts: fixed masks break permutation symmetry, while positive homogeneity constrains the dynamics. In a dense network, neurons can exchange roles, and width creates many equivalent parameterizations of the same function. A fixed sparse mask gives neurons different supports, removes most of this exchangeability, and restricts each neuron to a coordinate subspace and an initial sign pattern. Positive homogeneity also gives the gradient-flow invariant aj(t)2−∥wj(t)∥22=λja_j(t)^2-\lVert w_j(t)\rVert_2^2=\lambda_j. The paper argues that this invariant makes some sign changes difficult because they require a neuron to pass through a low-norm region. Therefore, a winning ticket is not just a useful mask, but also a neuron assignment and sign configuration from which the target is reachable.

Theorem 3.1 approximates the probability that a random global mask covers the coordinates used by a sparse teacher as (1−e−ℓ/d)m(1-e^{-\ell/d})^m, and Corollary 3.2 adds a 2−m2^{-m} penalty for matching the target orthant. The toy experiment compares dense and one-dimensional sparse dynamics under balanced initialization. In the large-scale experiments, signs copied from trained 90%-sparse solutions nearly recover the AC/DC, STR, and RigL ResNet-50 baselines, while random signs perform substantially worse.

Critical strengths
  1. A useful distinction in the paper is between finding a good mask and optimizing that mask. Most pruning-at-initialization work focuses on the first problem. The paper uses permutation symmetry, sign patterns, and the balance invariant to explain why a fixed mask may be difficult to optimize. I do not think the theory establishes this explanation, but separating mask selection from fixed-mask optimization is still useful.

  2. The sign-reinitialization experiments show that the mask alone does not explain trainability. For 90%-sparse ResNet-50 models produced by AC/DC, STR, and RigL, initialization with signs from the trained solution nearly recovers the corresponding baseline, while random signs lose several accuracy points. The experiment is oracle-based, so it does not explain the gap. Still, copied signs work across all three methods, which shows that the signs matter in addition to the mask

Critical weaknesses
  1. Theorem 3.1 talks about "recovering the target". However, this is not the same as the neurons collectively covering the dimension of the target. For example, if m=2m=2 and the target weights are given by a∗=1a^*=1, and w∗_1=w∗_2=1≠0w^*\_1 = w^*\_2 = 1 \neq 0 are the only non-zero entries in w∗\mathbf{w}^*, then having two neurons where the first one covers 11 and second covers 22 doesn't actually "cover" the target. Specifically, the target is f(x1,x2)=σ(x1+x2)f(x_1,x_2) = \sigma(x_1 + x_2) which cannot be matched with g(x1,x2)=a1σ((w1)1x1)+a2σ((w2)2x2)g(x_1,x_2) = a_1 \sigma((\mathbf{w}_1)_1 x_1) + a_2 \sigma((\mathbf{w}_2)_2 x_2). Specifically if σ\sigma is ReLU, then every student is separable, in the form of g(x1,x2)=h1(x1)+h2(x2)g(x_1, x_2) = h_1(x_1) + h_2(x_2), and we have g(1,1)+g(−1,−1)−g(1,−1)−g(−1,1)=0g(1,1)+g(-1,-1)-g(1,-1)-g(-1,1)=0. However, we have f(1,1)+f(−1,−1)−f(1,−1)−f(−1,1)=2+0−0−0=2f(1,1)+f(-1,-1)-f(1,-1)-f(-1,1)=2+0-0-0=2. In other words, the correct notion of "recovering the target" is to have at least one neuron that recovers all the dimensions in the target.

  2. The setting of Theorem 3.1 is ℓ≪Nd\ell \ll Nd. The argument works in the limit where NN going to infinity while other arguments are fixed. Naturally, this results in the ℓ\ell budget being spread out over different neurons. In fact, in the limit, the probability of having a neuron with more than 1 weight unmasked goes to zero. Therefore, this limit only considers "separable" students in the sense I described above. This is reinforced by the discussion of ℓ≈m\ell \approx m in the text. Because of this, the chance of recovery actually goes to zero, not the expression in the theorem. Note that this argument doesn't actually strengthen the claim. Instead, it undermines the assumptions. A more natural assumption is to remove ℓ≪Nd\ell \ll Nd and instead assume a fixed mask density pp, i.e., ℓ≈pNd\ell\approx pNd and then let NN and ℓ\ell go to infinity. However, in this case, the probability of a single neuron recovering the target dimensions is 1−(1−pm)N1-(1-p^m)^N which goes to 1 as NN increases. If we try to recover the orthant, like in Corollary 3.2, this becomes 1−(1−(p/2)m)N1-(1-(p/2)^m)^N which also tends to 1 as NN increases.

  3. The balance argument does not establish that sparse networks are generally trapped in the wrong incoming-weight orthant. The invariant aj(t)2−∥wj(t)∥22=λja_j(t)^2-\lVert w_j(t)\rVert_2^2=\lambda_j constrains the outgoing scalar aja_j and the total norm of the incoming vector. With balanced initialization, changing the sign of aja_j forces the entire incoming vector through zero. However, changing one coordinate sign of a multidimensional wjw_j does not require ∥wj∥2\lVert w_j\rVert_2 to approach zero, since the remaining coordinates can stay nonzero. The toy experiment focuses on cases with one incoming weight where this issue does not happen.

Questions for authors
  1. What event is Theorem 3.1 intended to describe? The proof counts the event that every target coordinate appears somewhere in the layer, while the text describes this as recovering the target. For the single-ReLU teacher, recovery generally requires one neuron to contain the full target support (See Weakness 1).

  2. What is held fixed when NN increases: the total mask budget ℓ\ell, the mask density, or something else? With fixed ℓ\ell, the probability that one neuron contains all mm target coordinates goes to zero as NN grows. With a fixed Bernoulli density pp, that probability is 1−(1−pm)N1-(1-p^m)^N and tends to one.

  3. The invariant constrains aja_j and ∥wj∥2\lVert w_j\rVert_2, but one coordinate of wjw_j can cross zero while the other coordinates keep the norm nonzero. What prevents one coordinate of wjw_j from crossing zero while the other coordinates remain nonzero?

  4. How are the correct- and random-sign initializations constructed? Were the magnitudes, normalization statistics, and neuron assignments held fixed? How many seeds were used, and what was the variation across runs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.