- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper seeks to emphasize the utility of sparse neural network training as a lens for the study of overparameterization. It is framed as a position paper. AFAICT it is submitted as a main conference paper, and not to any "position paper" specific track.
The paper's goal seems to be to stimulate study in the intersection of lottery tickets, overparameterization, weight-space symmetries, and sparse neural networks. The paper synthesizes results from the existing literature to make the case that these concepts are intimately related and should be studied cohesively, esp. towards an understanding of overparatemerization.
Critical strengths
The overall motivation of the paper is sound, and the paper invokes some of the appropriate tools (orthant selection, gradient flows). I found the gradient flow argument to be most interesting. I would be keen to see a larger study of this effect in larger models, and realistic optimizers. It would be esp surprising/interesting if in fact standard overparameterized training sends the majority of weights to near zero.
I would be interested to see more work in the area of sparsity, lottery tickets, and overparameterization. Meaningful connections that can be made between these concepts would be an exciting development IMHO.
Critical weaknesses
The paper misses key related work, including ones that are central to the narrative of the paper. Two that immediately come to mind:
- Ainsworth et al (2023) "Git Re-Basin: Merging Models modulo Permutation Symmetries" is directly relevant to "Can we find a permutation to correct the alignment?" paragraph of section 3.1, "Weight-space symmetries." paragraph in section 2, "Permutation Symmetries." paragraph in section 3.2.
- Zhou et al (2019), “Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask.” directly addresses the main orthant question in this paper. It is not clear to me that this paper is contributing anything that Zhou et al has not already covered.
There are likely others, but those are the ones that stick out to me in the moment.
One of the central claims of the paper is that overparameterized NN training is a process of "orthant selection", i.e. picking whether weights should be positive or negative, more than picking any concrete value for them otherwise. But this argument has already been made in Ramanujan et al (2020), Frankle, Schwab, and Morcos (2020), and Zhou et al.
There are also substantial misunderstandings in the purported theoretical contributions. Issues with Theorem 3.1:
- In the theorem O_{sub, j} for j \in [N] is indexed on the N hidden units, but O_sub is introduced in the beginning of the section as being indexed over all dimensions of weight space, with no relation to hidden units. The authors confuse weights with neurons/units. LTH literature randomly masks individual weights, not hidden units.
- O_sub^* is not defined AFAICT
- Even if we charitably look past the disconnect between the theorem statement and its purported implications, the theorem is sufficiently trivial that it does not add anything useful to the conversation. Stated another way, the theorem essentially boils down to a "balls in bags" question, ie if I have B balls in a bag, R of which are red, and I draw N balls out of the bag without replacement, what is the probability that i draw all R red balls.
Furthermore, the gradient flow argument in section 3.3 is interesting but flawed. The argument is sound as far as switching the sign of a_j,t. But the argument for w_j (starting on line 354) is incorrect: gradient flow can flip the signs of entries of w_j without breaking the ||w_j|| constraint or getting stuck in a critical point. Note that a single zero in the w_j vector can still have non-zero derivative.
One final concern: part of the central messaging of the paper is that sparse masking breaks weight space symmetries and that this effect can shed light on overparameterization. But I fail to see how the paper actually "lands the plane" on this point. What is this final relationship? There is some discussion of these symmetries and there is some discussion of overparameterization, but there is not a satisfying bridge built between the two IMHO. This is unfortunate to me because I believe this may be an interesting research direction, but I find the connection made in this paper is vague and unsatisfying.
Misc.
- line 365: \nabla_j notation is confusing and undefined afaict
- subsection 4.3 "UNDERSTANDING THE ROLE OF SYMMETRIES IN TRANSFORMERS" does not fit with the other 2 subsections in section 4.
"Introducing sparse masks breaks symmetries in a sense that the neurons become more specialized and less likely to match specific target neurons. Furthermore, the plasticity of sparse neurons suffers more from the rescale symmetry, constraining them to their initial orthant, as they hardly flip weight signs and thus can seldom assume different roles. In summary, a combination of reduced permutation symmetry and plasticity induced by an active rescale symmetry turns sparse training with a fixed mask into a hard problem"
This paragraph is vague, imprecise, and possibly untrue. It is mathematical fact that applying a random/inconsistent weight mask breaks permutation symmetries. why not just say that directly?
line 32: "Sparse training, in contrast to dense training, uses models with sparse connections or weight matrices, and is a promising direction towards reducing the computational cost of training."
there is still no viable path towards efficient sparse training on common accelerators. any promises made to the contrary are so speculative as to be unserious imho. the focus should be on the overparameterization investigation.
"Well-known reasons to study sparse training with a fixed mask."
paragraph is mostly fluff. I am willing to accept the argument that in some hypothetical hardware future we may be able to realize advantages of a sparse NNs, but the rest of this paragraph does not actually add anything of value IMHO. it also does not support the message of the paper, i.e. that sparse NNs offer a unique method to understand overparameterization. The following "Beyond the reasons that make STFM practically relevant" paragraph is much better.
"We argue that insights from this perspective could guide the design of more trainable and efficient neural networks that rely less on massive overparameterization."
this sentence is weak and unnecessary. studying overparameterization is interesting in and of itself, and at present there is no known viable path to making fixed-mask sparse training more efficient on accelerators than standard training.
"Overparameterization exponentially increases the probability of starting in the correct orthant. Sparsity reduces this probability, which makes it difficult for the optimizer to find a good solution."
what does this mean exactly? the "correct" orthant is only relevant for sparse models. the exact takeaway is unclear.
"Gadhikar et al. (2025) proposed using the signs from a trained model that has converged to a good solution."
using them how?
"Winning tickets succeed by preserving the symmetry-breaking configuration induced by overparameterization, not by identifying intrinsically important weights."
this is confusing: what precisely is a "symmetry-breaking configuration" and how does overparameterization induce one?
"Overparameterized networks usually require less of this movement."
this is argument is made formal in eg NTK literature
(Results from (Gadhikar et al., 2025))
should be \citet
Questions for authors
what are you trying to say in this paper that has not already been done in prior work? the paper title gives us something concrete, but the contents of the paper cover a bunch of different things, loosely connected. what exactly is the research question? how does this fit together into a coherent story? what do you want readers to understand after reading this paper? what research questions do you want readers to be thinking about after reading this paper? what does this paper actually add to the conversation?