acceptodds
Under review as a conference paper at ICLR 2027

On Parameter Symmetries and Conservation Laws in Gradient Flow

Abstract

Parameter space symmetries and conservation laws play an important role in understanding the loss landscapes and implicit biases of neural networks. Inspired by Noether's theorem in physics, prior works have sought to derive conservation laws under gradient flow from parameter symmetries, but the scope and limitations of this connection remain unclear. We develop a unified geometric framework that clarifies the precise relationship between the two notions, including the conditions under which symmetries correspond to conservation laws. We introduce a notion of compositional identifiability and use it to establish a general inheritance principle for complete characterizations of symmetries and conservation laws in multilayer networks. We apply the framework to multihead and grouped-query attention, polynomial neural networks, and square deep linear networks.

Primary area:
optimization
Keywords:
symmetry, parameter symmetry, conservation law, gradient flow, model identifiability, differential geometry, neural network

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yWq5OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper builds a framework for studying conservation laws for gradient flow of neural network parameters, which explains the difference between independent symmetries of the loss function coming from 1-dimensional connected lie groups and independent conservation laws for the parameter gradient flow, giving a number for the gap between the two and an iff condition for equality. The paper connects this to conservation laws for actual neural networks by way of network parameter identifiability. It is shown that given that the network components satisfy a "compositional identifiability" criterion, functional symmetries of the components can be lifted to functional symmetries of the full network. The paper shows that these symmetries then correspond exactly to symmetries of the loss for many reasonable loss functions, which in the framework built by the paper corresponds to finding conservation laws for the parameter gradient flow. This is then applied to a couple of actual examples. Namely, a complete set of independent conservation laws are given for grouped query attention and for networks with monomial activation. One limitation noted in the paper is that the compositional identifiability does not hold for matrix multiplication, which means that deep linear networks require a different notion of identifiability to lift component symmetries. Nevertheless, the paper also provides an upper bound for the number of independent conservation laws of a deep square linear network based on the framework.

Critical strengths

The paper provides a theoretical framework for conservation laws for parameter gradient flow. Specifically, they explain the gap between loss symmetries and conservation laws. The question of conserved quantities under various training dynamics is one that merits study, and this paper is a valuable contribution in that regard. Thus, given that all concerns/questions are met in a reasonable manner, I would recommend accepting the paper.

Critical weaknesses

I do not perceive any weaknesses in the paper as critical.

Questions for authors

line 063: the reference to Vlačić and Bölcskei 2021 is fairly vague, what do you mean by "induced by particular activation functions"? This reads slightly odd since what they essentially do is extend previous work of Fefferman 1994 (which assumed the particular activation tanh) to include a larger family of activations with certain affine symmetries.

line 078: In Related Works, the reference to Galley et al. 2026 should read something like "conservation laws under data symmetry" since their results are not data-dependent in the sense that they vary with data batches (which is the usual understanding of data-dependent, as in your reference Zhao et al. 2023). In fact, if you consider the joint group action on the data and parameters as in Sonoda et al. 2023, it is even true that the group-augmented risk from Galley et al. 2026 becomes an orbit average of the nominal risk over the parameters only and not the data (assuming FFN with equivariant activations and invariant loss).

Also, I am curious if your results explain in any way the result of Galley et al. 2026 that no new conservation laws are obtained from data symmetry when considering non-polynomial analytic margin losses? Specifically, does your Corollary 10 or Theorem 12 say anything about this case?

line 148-154: Could you be more specific about how the proof of Marcotte et al. 2025 fails? That is, explain what they get from Theorem 1 in Newns 1967, why it is not enough, and perhaps a short word about how you close the gap.

line 284: You write that the correspondence is "exact". Do you mean here that you cannot weaken the condition that the vector field should be closed in Corollary 10? Is it the first part of Prop. 8 that yields the exactness of the correspondence?

line 299 and 311: Does independent here mean infinitesimally independent or something else?

line 360-375: It would be good to include the definition of "infinitesimally generated". So, it is more clear what you want to show. You state it more clearly in the proof in the appendix, but the main text should still stand on its own in terms of readability.

line 404-407: It would be good if you could provide more details as to what is lacking in the proof of Tran et al. 2026.

line 478-485: In the conclusion, you do not mention the limitation that the Lie group Γ\Gamma has to be connected. I think it would be good to mention exactly how this assumption affects your results.

Furthermore, there is no mention of the fact that gradient flow is not how networks are actually trained in practice. I do not expect you to add results on conservation laws for discrete optimization, but it would be good to mention as a limitation of the paper since you want to apply this framework to neural networks.

line 1048: "motivates" →\to "motivate"

line 1488: Do the functions here satisfy property (9) required by Prop. 15 only, or do they satisfy property (12) required by Theorem 17?

Why do you use the notation GG for a neural network? This is a bit confusing, especially since you call the group elements gg even though you denote the group by Γ\Gamma. Later, you also use the notation gg to denote subnetworks of GG, in Prop. 15, 16 and Theorem 17.

Do your results say anything about conserved quantities of gradient flow initialized in special configurations? Such as Lemma 2.1 from your reference Simsek et al. 2021. This question is more of a future work related question, not something that should be addressed in the paper.

The papers I referenced which are not your references:

Fefferman, 1994, "Reconstructing a neural network from its output"

Sonoda et al., 2023, "Joint group invariant functions on data-parameter domain induce universal neural networks"

6 comments
AuthorsOpenReview ↗

Author Response: Questions 1-8

We thank the reviewer for their enthusiastic review, thoughtful feedback, and for recognizing that our work is a valuable contribution to the question of conservation laws in neural networks. Before moving on with our response, we want to clarify one point in the reviewer's summary: our results do not require the compositional identifiability criterion to hold to be able to lift functional symmetries and conservation laws from components to the full network. The compositional identifiability criterion is only required to establish completeness of the lifted conservation laws. We address the reviewer's questions below.

  • Q1: We thank the reviewer for pointing out this ambiguity. Our intention was to highlight Vlačić and Bölcskei's systematic study of parameter symmetries arising from affine symmetries of activation functions. We have revised the text to refer explicitly to 'networks whose activation functions admit affine symmetries or equivariances,' citing Fefferman (1994) and Vlačić and Bölcskei (2021) for the former, and Zhao et al. (2023) for the latter.
  • Q2: We thank the reviewer for the suggested wording clarification, which we have incorporated into the revised manuscript.
  • Q3: This is a very, very interesting question, and we appreciate the reviewer for raising it. Our Corollary 10 and Theorem 12 does not explain the result of Galley et al. (2026) that no new conservation laws are obtained from data symmetry when considering non-polynomial analytic margin losses. However, we think our framework can be adapted to this setting: their constraint on symmetric data distributions effectively restricts the set of possible gradient flow directions at each point, which we term the gradient flow distribution in our work. The discussion surrounding their Equation (9) can be interpreted as: a necessary condition for a new conservation law to arise is that the gradient flow distribution shrinks when we impose the symmetric-data-distribution constraint. Their Theorem 3 shows that in this setting, the gradient flow distribution does not shrink, and hence no new conservation law arise.
  • Q4: The proof of Marcotte et al. (2025) relies on Theorem 1 of Newns (1967) to show that if the Jacobian of the conservation laws h,hk+1,…,hDh, h_{k+1}, \ldots, h_{D} is not full rank then hh is a function on the other laws. However, Newns' definition of dependence functions, and thus their theorem, only guarantees that there exists a function FF such that F(h,hk+1,…,hD)=0F(h, h_{k+1}, \ldots, h_{D}) = 0, which is not yet enough to conclude that hh is a function of the other laws. Our Proposition 4 does not try to close the gap nor rely on Newns' theorem because we did not know at the time if that was the right approach. Instead, we attempted to develop a similar result directly, and in doing so, found a straightforward way to show the presented equivalence. Reviewer 1 remarked that this gap can be closed by using Remark 2 of Newns, to which we agree, and we have added a note to the text to clarify this point.
  • Q5: Yes, exactness here means that the condition that the vector field should be closed in Corollary 10 cannot be weakened, and is a consequence of the first part of Proposition 8.
  • Q6: Yes, independent here means infinitesimally independent. For brevity, we sometimes omit the adjective 'infinitesimally' and simply say 'independent' or 'complete'. We have revised the text to clarify this point.
  • Q7: We thank the reviewer for this suggestion. We have added the definition of infinitesimally generated in the main text.
  • Q8: The proof of Tran et al. (2026) shows that if hh is a conservation law of the multi-head self-attention layer, then hh must be constant on the level sets of the conservation laws hQKi=Qi⊤Qi−Ki⊤Kih^i_{QK} = Q_i^\top Q_i - K_i^\top K_i and hVOi=Vi⊤Vi−Oi⊤Oih^i_{VO} = V_i^\top V_i - O_i^\top O_i. This is weaker than our assertion that the hQKih^i_{QK} and hVOih^i_{VO} form a complete set. In particular, we cannot yet conclude that there exists a function ff such that h=f(hQKi,hVOi,∀i)h = f(h^i_{QK}, h^i_{VO}, \forall i). We think this is a nontrivial step that perhaps can be addressed with additional analysis.
AuthorsOpenReview ↗

Author Response: Questions 9-15

  • Q9: We thank the reviewer for this suggestion. Our infinitesimal analysis is sufficient to characterize the 1-parameter symmetries and conservation laws for connected Lie groups. Every general Lie group has a connected identity component, and the quotient by this component is a discrete group. Given our characterization of the conservation laws based on differentials, which is inherently local and does not rely on any connected Lie group argument, we do not expect these additional discrete actions to yield independent conservation laws beyond those captured by continuous symmetries. Indeed, our characterization has shown that the continuous symmetries already capture all smooth conservation laws. Nevertheless, discrete symmetries provide important global geometric information, and understanding these effects is an ongoing research direction. We have revised the conclusion to acknowledge this line of work as a promising research direction to study learning dynamics.
  • Q10: We thank the reviewer for this suggestion. We have revised our conclusion to mention that our analysis does not directly apply to discrete optimization algorithms used in practice.
  • Q11: Thank you for pointing this typo out, we have fixed it.
  • Q12: These functions satisfy the condition (9) as mentioned at the beginning of the corresponding subsection. They do not necessarily satisfy condition (12). We have further clarified this point in the caption of this table.
  • Q13: We thank the reviewer for pointing out this potential ambiguity. We made a conscious choice to use standard notations in the literature on symmetry and conservation laws (and math) so as to make it more accessible to readers from different backgrounds. We use gg for group elements because they are standard notations for group elements in the machine learning literature on symmetry and equivariance, as well as in group theory, differential geometry, and Lie theory. Separately, we use gg for subnetworks because prior works on conservation laws in neural networks have used gg to denote subnetworks/layers, and so using GG for the full network is a natural choice. Once we have focused on the one-parameter case, i.e., Γ=R\Gamma = \mathbb R, it becomes natural to interpret group elements as time parameters, and hence use tt to denote the group element. This both synergizes with Proposition 9 showing that continuous symmetries are flows generated by symmetry vector fields and frees up the notation gg to denote subnetworks. We have added notes at the end of Section 2.2 and before the first inheritance principle to clarify this notational choice.
  • Q14: This is another interesting question, and we thank the reviewer for raising it. Our intuition is that the answer is yes, but in a trivial way: if some parameters are initialized to be equal to each other, and they contribute identically to the output, then their trajectories will be identical at all time. Thus both the their difference and its gradient will always be zero. Under our framework, this difference is a conservation law since its gradient is the zero vector field, which is perpendicular to any distribution, including the gradient flow distribution.
  • Q15: We thank the reviewer for providing these references.
Reviewer_yWq5OpenReview ↗

Response to author's response

Thank you for answering my questions and clarifying all of the points I raised! And thank you for pointing out the error in my summary of your paper. I have a couple of follow-up questions.

Q1: Your reference to Vlačić and Bölcskei now reads Vlai and Bölcskei in your paper, I guess you forgot to format the special characters in your bib-file. This typo should be fixed.

Q4: Is there an implicit construction in Remark 2 of Newns that I am missing? As stated it does not seem to close the gap in the proof. The fact that D−k+1−(D−k)=1D-k+1-(D-k)=1 of the components can be expressed as a function of the others does not seem to immediately imply that it is the hh component that can be expressed as a function of the hk+1,…,hDh_{k+1},\ldots, h_D components. I am assuming that there is an easy construction I am missing.

Reviewer_WshLOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper looks at when symmetries of a network imply conservation laws. It restricts its attention to gradient flow symmetries and symmetries of the loss. It first states a clean characterization when every symmetry gives rise to a conseration law in the form gap=rank Lie(W)-rank W. This result was largely known but the statement is cleaner than previous ones. The paper then shows the following reduction. If a network can be shown to be built out of pieces so that the output function determines the function of each piece and if you can identify the complete set of conservation laws for each piece, then you can write down the complete set of conservation laws for the whole network. The paper carries out this program for the case of attention, where the identifiability result and the required result on the building blocks were previously known. The paper does not address the difficult question how those pieces can be found in general or how the complete set of conservation laws can be found for each piece (which is the original problem but now for a smaller part).

Critical strengths
  • the inheritance principle is likely the strongest point of the paper; this proves a reduction that makes it potentially much easier to characterize all conservation laws of a network

  • The statement gap=rank Lie(W)-rank W, which states how many symmetries do not give rise to conservation laws, and the statement that is equal to 0 iff W is involuative is very clean and clear.

Both of the above points are very nice and worth publishing.

Critical weaknesses

Post-rebuttal update -- November 14, 2026 I have raised my recommendation from weak reject to weak accept and soundness from 2 to 3. Your rebuttal and your revision has substantially addressed my concerns. My remaining reservations concern the degree of novelty and the restricted applicability: the gap characterisation largely follows from existing ideas, the inheritance theorem requires substantial restrictions, and the deep-linear bound is elementary. But overall the paper constitutes a useful and nice contribution.


Original review:

The paper somewhat overpromises and underdelivers:

  • it sets out to complete what previous papers have left unanswered in the discussion about symmetries and conservation laws but at the end it only treats Euclidean gradient flow and only symmetries of the loss; this leaves out many symmetries that previously were considered: data symmetries, symmetries of general algorithms, or symmetries of the training dynamics rather than the loss

  • if one considers only gradient flow the precise relationship is not too difficult once Marcotte’s set of all vector fields W is known; you can find various versions of this relationship in the work of Marcotte 2025, Gegenfurtner et al. 2026, Zhao et al. 2023, and Josz 2025; so this relationship was not “left unclear” as is claimed; to be fair, the formulation given here: gap=rank Lie(W)-rank W is much cleaner than previous characterizations in the literature

  • currently the inheritance principle only has a single application

  • the experiments are underwhelming and do not really serve to test something stated in the paper

  • for the deep linear networks the result does not address the actual question whether there are other conservation laws

Questions for authors

All of the below can likely be addressed.

  • there might be a problem with respect to how the inheritance principle is stated; the same goes for proposition 16; this is very likely fixable since in the one application that is shown this is already restricted; the theorem claims to hold everywhere, but the proof only works at generic points.

  • the proof of proposition 18 uses Theorem 27, but it does so wrongly; the theorem is valid for a single sum, but it is applied to a difference of sums; in this difference attention matrices can conincide; this is very likely fixable

  • the gap theorem needs second derivatives but this is not stated

  • it seems to me that proposition 20 should be dropped; it has no content

  • I could not verify your claims about missing steps in previous proofs (Marcotte and Tran); but given that there are quite some missing steps and missteps in this paper I would be careful about making this a central part of this paper.

4 comments
AuthorsOpenReview ↗

Author Response: Weaknesses 1, 2

We thank the reviewer for their thoughtful feedback and for recognizing that our inheritance principle and the symmetry-conservation law gap are nice and worth publishing. We respectfully disagree with the assessment that our work overpromises or that our symmetry-conservation law gap was largely already known. We address these and other concerns below.

  • W1: We want to clarify that our work is focused on parameter symmetries and conservation laws for gradient flow. The scope of our work is clearly stated in the abstract, introduction, and first few sections of the paper. Although other types of symmetries can certainly be interesting to study as well, we never claim to give an answer to unanswered questions about them. We sometimes omit the word parameter later on in the paper for readability once the scope has been established, this omission is intended to avoid unnecessary repetition and awkward phrasing, and not to suggest a broader scope. We have re-added the word parameter in the conclusion to clarify this point.

  • W2: The reviewer claims that the precise relationship is largely known or not too difficult to derive once Marcotte's (2023) distribution is known. Although recent works have recognized that symmetries and conservation laws can be related through their underlying vector fields (as we have documented in our Appendix B.3), we argue that our framework is nevertheless novel and nontrivial. Our one-to-one correspondence for partial symmetries and our one-to-one correspondence for conservation laws are not implied by any of the previous works. These correspondences are what allows us to establish the symmetry-conservation law gap. We further provide a few examples indicating that the correspondence has not been largely known, contrary to what the review suggests:

    • The survey paper by Zhao et al. (2025) states that 'parameter space symmetry implies the existence of conserved quantities in gradient flow' (see the beginning of their Section 5). The survey also states: 'While many conserved quantities can be derived from parameter space symmetries, it remains an open question whether all conserved quantities originate from known symmetries.'
      This indicates that the correspondence was not largely known.
    • Please note that none of the previous works considered partial symmetries, which is necessary since there exists vector fields that are not complete, and hence do not generate a global one-parameter group of transformations. The closest prior work we are aware of that considered a notion of local symmetry is Gegenfurtner et al. (2026), but their local restriction is due to the non-smoothness of the ReLU activation, and not due to the inherent incompleteness of the vector field. This is a crucial point, as it allows us to establish a one-to-one correspondence between partial symmetries and the symmetry distribution, and hence to the dimension counting argument that yields the symmetry-conservation law gap. We have revised our Appendix B.3 to add an explicit example of a partial symmetry that is not complete.
    • The four works that the reviewer mentioned recognized that the vector-field-based construction doesn't always yield a conservation law. Three of these papers recognize that if the infinitesimal generator of a parameter symmetry is a closed vector field, then this symmetry induces a conservation law. But, as discussed in our Appendix B, previous works did not show that there is a bijection between the set of partial symmetries and the symmetry distribution. Previous works also did not show that there is a bijection between the set of closed vector fields in the symmetry distribution and conservation laws. For example, in Gegenfurtner et al. (2026), the authors conclude that a symmetry arising from layer composition does not induce a conservation law because its infinitesimal generator is not the gradient of any function. However, while this rules out the prescribed generator as a conserved gradient, it does not by itself rule out the possibility that there exist other constructions associating a conservation law with that symmetry.
AuthorsOpenReview ↗

Author Response: Weaknesses 3, 4, 5

  • W3: We thank the reviewer for this feedback, which prompted us to clarify that our framework establishes three distinct inheritance principles. The first inheritance principle is general and applies to any compositional model that relies only on the output of its subcomponents, not just neural networks. In deep learning, it genuinely covers many ways of embedding a subnetwork, such as sum, concatenation, composition, graph aggregation, etc. (see Appendix F.1) and gives a very direct motivation to study layerwise conservation laws, an active area of research in the field. For the third inheritance principle, the criticism that it applies to only one model is fair, and we look forward to future work that either relax the assumptions or use it in other contexts, which we did not exhaustively explored. Nevertheless, it produced a concise, instructive, and complete proof for grouped-query attention, which itself extends the results of Marcotte et al. (2025) and Tran et al. (2026) on single-head and multi-head attention, and hence should not be considered as a 'critical weakness'.

  • W4: Thank you for this helpful feedback, which prompted us to add two experiments that directly test predictions stated in the paper:

    • Conservation laws (New Section 6 and Appendix I): For polynomial neural networks and grouped-query attention, the predicted laws remain approximately conserved under SGD. Their drift decreases approximately linearly with step size at a fixed effective training horizon, consistent with the theoretical error bound.
    • Endpoint spectra (Appendix H): For the learning-to-scale problem, small-step-size SGD closely matches the initialization-based predictions for singular values and condition numbers across predicted condition numbers from 1 to 100, among runs satisfying the fit criterion.
  • W5: We thank the reviewer for raising this concern. Completeness of conservation laws for deep linear networks has been an open question in the field at least since 2023 (see Marcotte et al. (2023)) without any new progress that we know of in the literature. We do not claim to address this question in full. Instead, we provide a non-trivial upper bound on the number of conservation laws. This is the first improvement on this question since then. Even though our proof for this improved upper bound is short, this is the case only because it relies on the machinery that we have developed in the paper. Furthermore, the result illustrates how our framework allows an automatic transfer of completeness of symmetries to an upper bound on the number of independent conservation laws for neural networks, including deep ones. For these reasons, we believe that this result is a nontrivial contribution and would prefer to keep it in the paper.

AuthorsOpenReview ↗

Author Response: Questions

  • Q1: We thank the reviewer for raising this concern. We double-checked the statements and the proofs, and we believe that they are correct as written. The requirement for generic points in later results is to avoid degenerate cases where the dimension of the distributions considered have a disconinuity, which often occurs only on a set of measure zero. Prior works have also considered a complete characterization of symmetries and conservation laws in a generic sense (see Marcotte et al. (2023, 2025) and Tran et al. (2026)). We intentionally included this assumption in our results whenever necessary to avoid confusion. Another point to note is that we can always restrict the parameter space Θ\Theta to a subset of the full parameter space, such as to an open subset containing only generic points. This does not affect the validity of our results, as partial symmetries and conservation laws are inherently local objects.

  • Q2: We thank the reviewer for identifying this point. The proof of this result has an implicit grouping argument that we have now made explicit in the revised manuscript. We originally omitted this argument for brevity as it seemed natural to us, but we agree that it is better to include it for clarity. The technique used here was developed by Hoang et al. (2025) to prove the maximal symmetric group of multi-head attention, and adapting it to our setting was straightforward. A key observation is that the minus sign in the difference of sums does not affect the grouping argument.

  • Q3: We thank the reviewer for suggesting this assumption, but the regularity assumption that we make is already stronger than what the reviewer mentioned. Our gap theorem explicitly states the assumptions that WW and Lie⁡(W)\operatorname{Lie}(W) are smooth distributions of constant rank. Here, 'smooth' is used in the differential-geometric sense of C∞C^\infty: these distributions admit smooth local frames, whose iterated Lie brackets have the regularity required by the proof. Functions in the class C∞C^\infty are infinitely differentiable.

  • Q4: We do not think that this result should be dismissed. Please see our response to W5 above.

  • Q5: We thank the reviewer for this caution. We want to clarify that we do not want to make the missing steps in previous proofs a central part of our paper, and that we take the utmost care in ensuring the rigor of our arguments. We were diligent to read prior proofs and learn from them, and through this process, we found that some aspects were left implicit (please see our response to specific questions raised by Reviewer 3). We then found ways to prove these results ourselves, and included them in our paper in service to the completeness of our work and to the community. Based on Reviewer 1's feedback, we have also revised our text to clarify that Proposition 4 is a self-contained and direct proof of the equivalence, and acknowledge one method to close the gap in Marcotte et al. (2025) (please see our response to Reviewer 1). We hope that we have satisfactorily clarified all technical details raised by the reviewer in our previous comments. If there are any other specific concerns about the rigor of our proofs, we would be happy to address them.

Thank you for taking the time to revisit the paper and update your assessment, and for recognizing that our work constitutes a useful and nice contribution. We’re glad that our rebuttal and revision substantially addressed your concerns. We’re grateful for your thoughtful comments, which helped us refine the paper and clarify the scope of our results.

Reviewer_VEGmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies and clarifies the link between symmetries and conservation laws under GF dynamics. In particular, the authors rigorously show that each conservation law correspond to a partial one-parameter symmetry (and not the other way). The authors then give inheritance principles under identifiability conditions, and apply this to one-layer grouped query attention networks.

Critical strengths

S1. The paper clarifies the link between conservation laws and symmetries under gradient flow dynamic (cf Proposition 9 & corollary 10). This is a good thing as there is a lot of confusion in the literature about this link. Several papers were expecting that "symmetry of the loss   ⟹  \implies conservation", mostly because they were focused on "generalizing" Noether theorem in a setting (gradient flow) which is actually less constrained.

S2. The paper is mathematically rigorous, I've checked a good proportion of the proofs.

S3. Proposition 18 is nice

Critical weaknesses

W1. The definition 11 should be motivated. This definition allows to match what is needed for point i) of theorem 12, but its geometric motivation should be explained. For example if I take the translation Tt(a,b)=(a,b+t)T_t(a, b) = (a, b+t) and the dilation Dt(a,b)=(a,exp(t)b)D_t(a, b) = (a, exp(t) b), then χT(a,b)=(0,1)\chi_T(a, b) = (0, 1) and χD(a,b)=(0,b)\chi_D(a, b) = (0, b) so within definition 11, translation and dilatation are dependent symmetries.

W2. The possibility to deal with multi-layer networks (introduction, and beginning of section 4) is a bit overclaimed. The proposition 16 & theorem 17 cannot be applied to classical deep architectures such as deep linear networks, deep MLP, ResNets, transformers, etc. At the beginning of their Section 4, the authors should also mention the section 4 of Marcotte et al 2025, that allows to characterize all conservation laws of deep networks depending on a subset of parameters (see eg their theorem 4.6).

W3. The claim that theorem 12 (point ii) ) do provide a generalization of theorem 3.3 of Marcotte et al 2023 is incorrect. Although the latter is presented with WϕW_\phi in Marcotte et al 2023, their proof does not rely on that. Theorem 2.11 of Marcotte et al 2025 is the same theorem presented without such parameterization, and the proof of the point ii) of the present paper actually corresponds to the proof of Marcotte et al 2025.

W4. about line 148-154: the authors mention that the proof of Prop 2.12 of Marcotte et al 2025 is insufficient as it relies on Theorem 1 of Newns 1967, which is not enough to conclude. However the conclusion follows from the remark 2 of theorem 1 of Newns 1967. The proposition 4 introduced by the authors essentially corresponds to this remark 2 Newns 1967, but states it more clearly. The proposition 4 could be kept for pedagogical purpose, but its introduction should be rephrased so as to avoid misrepresentation of the actual contributions of the present paper.

W5. The authors should also discuss in their appendix B.3 the section 3.1 of Marcotte et al 2024 that is dedicated to this question

Questions for authors

please see the weaknesses above

9 comments
AuthorsOpenReview ↗

Author Response

We thank the reviewer for their thoughtful feedback and for recognizing that our work clarifies the link between symmetries and conservation laws, is mathematically rigorous, and includes a nice Proposition 18. We address the main concerns below.

  • W1: We thank the reviewer for the suggestion to add more motivation for definition 11. Originally, we viewed the underlying distribution as a crucial object to understand, and defined independent symmetries so as to capture all directions in this distribution. Prompted by the reviewer's comment, we have added a new proposition right after this definition to show that any new partial symmetry can locally be expressed as a composition of an infinitesimally complete set of partial symmetries. The geometric motivation is that since no other symmetry direction exists (and also that the symmetry distribution is involutive), any new symmetry can be factored into the already existing ones. The two symmetries that the reviewer mentioned travel in the same symmetry direction, and hence are dependent in our definition. This can be seen by thinking about their trajectories in 2D space: both move along a vertical direction with different speeds.

  • W2: As suggested, we have added a sentence at the beginning of Section 4 acknowledging Marcotte et al. (2025) and their results on conservation laws depending on subsets of parameters in deep networks. We agree that the assumption needed for Proposition 16 and Theorem 17 is strong, and future work is needed to relax it so that the framework can be applied to deep networks in general. Nevertheless, the results of Section 4, including Proposition 15, provide general criteria for transferring symmetries and conservation laws from component models to their composition under identifiability conditions. This is what we want to convey through our paper, and the abstract and conclusion already point to this limitation. To make the scope clear, we have now also stated this limitation in the introduction and near the beginning of Section 4 as suggested.

  • W3: Thank you for pointing this latter reference out; we were not aware of this subsequent improvement to the original result. We have revised our statement to say that part (ii) of our gap result can be seen as a reformulation of Marcotte et al. (2025)'s Theorem 2.11, where we remove another unnecessary assumption, namely Assumption 2.4 in that paper.

  • W4: Thank you for letting us know about this. As suggested by the reviewer, we have revised the text to note that the additional argument left implicit in the published proof can be supplied using Remark 2 of Newns (1967). We note that Remark 2 does not have a self-contained proof, and it takes some additional work to translate it into the argument needed. For our part, we did not rely on Newns (1967), but rather used a more straightforward approach in our development. Thus, we believe that our direct formulation and proof provide additional clarity, and we agree that it should be kept in the paper.

  • W5: As suggested, we have added a discussion of the relevant section of Marcotte et al. (2024) in Appendix B.3. In particular, we mention that this work relates loss invariance to the orthogonality between infinitesimal generators and loss gradients, and use this relation to recover balancedness conservation laws for two-layer linear and ReLU networks under Euclidean gradient flow.

I thank the authors for their careful rebuttal. They've already addressed my points W2, W4 & W5. I still have questions and comments about W1 and W3.

About W1 and the notion of partial symmetry: all conservation laws that I know about during the training of a neural network are associated to a specific symmetry, do you have an example of a conservation law h(θ)h(\theta) which could not be associated to a specific symmetry but only to a partial one (and such that it does not correspond to F∘h0F\circ h_0 with h0h_0 that could work but not F∘h0F \circ h_0)?
In particular, I am not sure to get the motivation of line 1097-1106 of the two-layer NN abxabx. The vector field χ:(a,b)↦(a2,−ab)\chi:(a,b)\mapsto (a^2, -ab) is such that for all (a,b)∈R⋆×R(a, b) \in \mathbb{R}^\star \times \mathbb{R}, χ(a,b)\chi(a, b) is proportional to (a,−b)(a, -b) which corresponds to the gradient of the classical conservation law (a2−b2)/2(a^2- b^2)/2 that is known to be associated to the symmetry (exp(s)a, exp(-s)b): so in that particular example, I would just think that χ\chi is a "bad" choice, and in that example a "good" choice is possible.

Just a clarification about W3: To clarify, Assumption 2.4 (which is satisfied for all classical losses) is not needed for the counting argument itself, either in the proof of Theorem 2.11 of Marcotte et al. 2025 or in that of Theorem 3.3 of Marcotte et al. 2023. The same argument applies directly to your distribution WW. You just need to apply it to WW instead of Wg,ℓ\mathbb{W}^{g, \ell} . Assumption 2.4 is only used in the preceding characterization of W(θ)W(\theta) with a finite number of known vector fields (corollary 2.8). Knowing that a distribution has a constant rank allows you to have the existence of such a family, but does not give you a specific one. However, knowing such a family is actually central to characterize dimLie(W(θ))dim Lie(W(\theta)) and thus the number of independent conservation laws (for example to obtain the recalled result lemma 3.1). (It is also central to characterize dimW(θ)dim W(\theta) and thus the number of independent partial symmetries under your definition.)

Minor: you should add a label on the equation line 2073 and call it to justify eq 22

(I still need to finish to read some parts (section 6, updated proof of prop 20, and appendix H.3 and I.1) of the updated version so I might ask further questions)

AuthorsOpenReview ↗

Author Response 2

We thank the reviewer for their follow-up questions and comments. The first follow-up question, corresponding to the second paragraph, raises an additional and distinct point about the role of partial symmetries, rather than W1's focus on the role of infinitesimally independent/complete symmetries. We will call this new question Q1. Along with addressing Q1, we discuss two related issues: whether alternative associations between symmetries and conservation laws could eliminate a positive symmetry-conservation law gap (Q2), and why partial symmetries are necessary for the exact correspondence established in our framework (Q3). In particular, Q2 asks whether a different association could restore the naive expectation that independent symmetries yield independent conservation laws. We explain below why a positive gap precludes this. We hope this discussion clarifies the motivation for our definitions.

  • W1: the concern raised in the initial review asked for a clearer motivation for our Definition 11 of independent and complete infinitesimal symmetries. Our previous response described our geometric motivation, presented a new proposition in a similar spirit to Marcotte et al. (2025)'s Proposition 2.12, showing that any new partial symmetry can locally be expressed as a composition of an infinitesimally complete set of partial symmetries, and gave our interpretation of the example mentioned in the review. We believe that our discussion and result offer a good motivation for our definition, independently of its use in our framework. This is because they describe the geometric structure of the underlying distribution and corresponding flows. We hope this has addressed the reviewer's concern about our Definition 11, and we will be happy to provide further clarification if needed.

  • Q1: If we allow for a generous notion of 'good choice' of the infinitesimal generator, then our intuition is that one can always find a global symmetry 'associating' to any conservation law. Specifically, the intuition is that a suitable rescaling may slow down an incomplete generator sufficiently to avoid finite-time blow-up and yield a global flow. Given a conservation law hh and some function F∘hF \circ h, the gradient ∇(F∘h)=F′(h)∇h\nabla (F \circ h) = F'(h) \nabla h is effectively a rescaling of the gradient ∇h\nabla h. Thus, from the intuition above, we suspect that it should generally be possible to find FF such that ∇(F∘h)\nabla (F \circ h) is complete. For example, in the two-layer scalar network abxabx, instead of associating the conservation law h(a,b)=(a2−b2)2h(a,b) = (a^2 - b^2)^2 with the incomplete gradient ∇h=(4a(a2−b2),−4b(a2−b2))\nabla h = (4a(a^2 - b^2), -4b(a^2 - b^2)), we can associate it with the rescaled vector field χ(a,b)=(a,−b)\chi(a,b) = (a, -b), which is complete and is the gradient of the conservation law 12(a2−b2)\frac 1 2 (a^2 - b^2). Since there isn't a concrete definition of 'good choice' of the infinitesimal generator, we do not attempt to formalize this intuition.

  • Q2: A natural follow-up question to Q1 is whether a better way to associate independent conservation laws with infinitesimally independent symmetries can be found so that the gap is zero. We answer this question in the negative. The gap as formulated by our framework is determined by WW and Lie⁡(W)\operatorname{Lie}(W), independently of any particular association between symmetries and conservation laws. Consequently, when the gap is positive, no alternative construction can associate functionally independent conservation laws with all infinitesimally independent symmetry directions. We also note that our framework associates symmetries and conservation laws to vector fields in a natural way: that symmetries are associated to their infinitesimal generators, and conservation laws are associated to their gradients.

  • Q3: Within our framework, partial symmetries are necessary, as there exists conservation laws whose gradients are not complete vector fields, and hence do not generate a global one-parameter group of transformations. Without it, we would not be able to construct the one-to-one correspondence between partial symmetries and vector fields in the symmetry distribution, which motivates our formalization of the symmetry-law gap An example for a conservation law whose gradient isn't complete has already been given above: for the two-layer scalar network abxabx, the conservation law h(a,b)=(a2−b2)2h(a,b) = (a^2 - b^2)^2 has an incomplete gradient ∇h=(4a(a2−b2),−4b(a2−b2))\nabla h = (4a(a^2 - b^2), -4b(a^2 - b^2)). We have added this example to our Appendix B.3 to illustrate the necessity of partial symmetries within our framework.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to All Reviewers

We thank the reviewers for their careful feedback and their recognition of the paper's mathematical rigor [VEGm], clear symmetry-conservation law characterization [VEGm, WshL], useful inheritance principles [WshL], and a valuable contribution to the study of conserved quantities, which is a question that merits study [yWq5]. Based on the reviewers' feedback, we have revised the manuscript in three main directions:

  • Contribution and scope. We distinguish our framework from existing approaches connecting symmetries and conservation laws. We clarify that our inheritance results are general in the sense that they don't make any assumptions about the structure of the network, and that basic inheritance applies broadly, while compositional identifiability is needed for completeness. We revise our text to make clear what is new and what is known, and to clarify the scope of our results.

  • Mathematical exposition. We add a new result to justify and provide geometric motivation for our definition of infinitesimally complete symmetries, make the grouping argument in the attention identifiability proof explicit, and clarify definitions and assumptions throughout.

  • Empirical support. We add new experiments that directly examine theoretical predictions: conservation-law drift decreases approximately linearly with SGD step size for polynomial networks and grouped-query attention, while the learning-to-scale experiment supports the predicted endpoint spectra and condition numbers among runs that converged.

We hope the revisions address the concerns raised by the reviewers. We believe they have strengthened both the presentation and empirical support for the paper's theoretical contributions. We highlight them in blue in the revised manuscript, and provide a detailed point-by-point response to the reviewers below.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.