- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper builds a framework for studying conservation laws for gradient flow of neural network parameters, which explains the difference between independent symmetries of the loss function coming from 1-dimensional connected lie groups and independent conservation laws for the parameter gradient flow, giving a number for the gap between the two and an iff condition for equality. The paper connects this to conservation laws for actual neural networks by way of network parameter identifiability. It is shown that given that the network components satisfy a "compositional identifiability" criterion, functional symmetries of the components can be lifted to functional symmetries of the full network. The paper shows that these symmetries then correspond exactly to symmetries of the loss for many reasonable loss functions, which in the framework built by the paper corresponds to finding conservation laws for the parameter gradient flow. This is then applied to a couple of actual examples. Namely, a complete set of independent conservation laws are given for grouped query attention and for networks with monomial activation. One limitation noted in the paper is that the compositional identifiability does not hold for matrix multiplication, which means that deep linear networks require a different notion of identifiability to lift component symmetries. Nevertheless, the paper also provides an upper bound for the number of independent conservation laws of a deep square linear network based on the framework.
Critical strengths
The paper provides a theoretical framework for conservation laws for parameter gradient flow. Specifically, they explain the gap between loss symmetries and conservation laws. The question of conserved quantities under various training dynamics is one that merits study, and this paper is a valuable contribution in that regard. Thus, given that all concerns/questions are met in a reasonable manner, I would recommend accepting the paper.
Critical weaknesses
I do not perceive any weaknesses in the paper as critical.
Questions for authors
line 063: the reference to Vlačić and Bölcskei 2021 is fairly vague, what do you mean by "induced by particular activation functions"? This reads slightly odd since what they essentially do is extend previous work of Fefferman 1994 (which assumed the particular activation tanh) to include a larger family of activations with certain affine symmetries.
line 078: In Related Works, the reference to Galley et al. 2026 should read something like "conservation laws under data symmetry" since their results are not data-dependent in the sense that they vary with data batches (which is the usual understanding of data-dependent, as in your reference Zhao et al. 2023). In fact, if you consider the joint group action on the data and parameters as in Sonoda et al. 2023, it is even true that the group-augmented risk from Galley et al. 2026 becomes an orbit average of the nominal risk over the parameters only and not the data (assuming FFN with equivariant activations and invariant loss).
Also, I am curious if your results explain in any way the result of Galley et al. 2026 that no new conservation laws are obtained from data symmetry when considering non-polynomial analytic margin losses? Specifically, does your Corollary 10 or Theorem 12 say anything about this case?
line 148-154: Could you be more specific about how the proof of Marcotte et al. 2025 fails? That is, explain what they get from Theorem 1 in Newns 1967, why it is not enough, and perhaps a short word about how you close the gap.
line 284: You write that the correspondence is "exact". Do you mean here that you cannot weaken the condition that the vector field should be closed in Corollary 10? Is it the first part of Prop. 8 that yields the exactness of the correspondence?
line 299 and 311: Does independent here mean infinitesimally independent or something else?
line 360-375: It would be good to include the definition of "infinitesimally generated". So, it is more clear what you want to show. You state it more clearly in the proof in the appendix, but the main text should still stand on its own in terms of readability.
line 404-407: It would be good if you could provide more details as to what is lacking in the proof of Tran et al. 2026.
line 478-485: In the conclusion, you do not mention the limitation that the Lie group has to be connected. I think it would be good to mention exactly how this assumption affects your results.
Furthermore, there is no mention of the fact that gradient flow is not how networks are actually trained in practice. I do not expect you to add results on conservation laws for discrete optimization, but it would be good to mention as a limitation of the paper since you want to apply this framework to neural networks.
line 1048: "motivates" "motivate"
line 1488: Do the functions here satisfy property (9) required by Prop. 15 only, or do they satisfy property (12) required by Theorem 17?
Why do you use the notation for a neural network? This is a bit confusing, especially since you call the group elements even though you denote the group by . Later, you also use the notation to denote subnetworks of , in Prop. 15, 16 and Theorem 17.
Do your results say anything about conserved quantities of gradient flow initialized in special configurations? Such as Lemma 2.1 from your reference Simsek et al. 2021. This question is more of a future work related question, not something that should be addressed in the paper.
The papers I referenced which are not your references:
Fefferman, 1994, "Reconstructing a neural network from its output"
Sonoda et al., 2023, "Joint group invariant functions on data-parameter domain induce universal neural networks"
6 comments
Author Response: Questions 9-15
- Q9: We thank the reviewer for this suggestion. Our infinitesimal analysis is sufficient to characterize the 1-parameter symmetries and conservation laws for connected Lie groups. Every general Lie group has a connected identity component, and the quotient by this component is a discrete group. Given our characterization of the conservation laws based on differentials, which is inherently local and does not rely on any connected Lie group argument, we do not expect these additional discrete actions to yield independent conservation laws beyond those captured by continuous symmetries. Indeed, our characterization has shown that the continuous symmetries already capture all smooth conservation laws. Nevertheless, discrete symmetries provide important global geometric information, and understanding these effects is an ongoing research direction. We have revised the conclusion to acknowledge this line of work as a promising research direction to study learning dynamics.
- Q10: We thank the reviewer for this suggestion. We have revised our conclusion to mention that our analysis does not directly apply to discrete optimization algorithms used in practice.
- Q11: Thank you for pointing this typo out, we have fixed it.
- Q12: These functions satisfy the condition (9) as mentioned at the beginning of the corresponding subsection. They do not necessarily satisfy condition (12). We have further clarified this point in the caption of this table.
- Q13: We thank the reviewer for pointing out this potential ambiguity. We made a conscious choice to use standard notations in the literature on symmetry and conservation laws (and math) so as to make it more accessible to readers from different backgrounds. We use for group elements because they are standard notations for group elements in the machine learning literature on symmetry and equivariance, as well as in group theory, differential geometry, and Lie theory. Separately, we use for subnetworks because prior works on conservation laws in neural networks have used to denote subnetworks/layers, and so using for the full network is a natural choice. Once we have focused on the one-parameter case, i.e., , it becomes natural to interpret group elements as time parameters, and hence use to denote the group element. This both synergizes with Proposition 9 showing that continuous symmetries are flows generated by symmetry vector fields and frees up the notation to denote subnetworks. We have added notes at the end of Section 2.2 and before the first inheritance principle to clarify this notational choice.
- Q14: This is another interesting question, and we thank the reviewer for raising it. Our intuition is that the answer is yes, but in a trivial way: if some parameters are initialized to be equal to each other, and they contribute identically to the output, then their trajectories will be identical at all time. Thus both the their difference and its gradient will always be zero. Under our framework, this difference is a conservation law since its gradient is the zero vector field, which is perpendicular to any distribution, including the gradient flow distribution.
- Q15: We thank the reviewer for providing these references.
Response to author's response
Thank you for answering my questions and clarifying all of the points I raised! And thank you for pointing out the error in my summary of your paper. I have a couple of follow-up questions.
Q1: Your reference to Vlačić and Bölcskei now reads Vlai and Bölcskei in your paper, I guess you forgot to format the special characters in your bib-file. This typo should be fixed.
Q4: Is there an implicit construction in Remark 2 of Newns that I am missing? As stated it does not seem to close the gap in the proof. The fact that of the components can be expressed as a function of the others does not seem to immediately imply that it is the component that can be expressed as a function of the components. I am assuming that there is an easy construction I am missing.
Author Response: Questions 1-8
We thank the reviewer for their enthusiastic review, thoughtful feedback, and for recognizing that our work is a valuable contribution to the question of conservation laws in neural networks. Before moving on with our response, we want to clarify one point in the reviewer's summary: our results do not require the compositional identifiability criterion to hold to be able to lift functional symmetries and conservation laws from components to the full network. The compositional identifiability criterion is only required to establish completeness of the lifted conservation laws. We address the reviewer's questions below.