acceptodds
Under review as a conference paper at ICLR 2027

Discovering Symmetries in Neural Network Parameter Spaces

Abstract

Parameter space symmetries are important for understanding neural networks' loss landscape, training dynamics, and generalization. However, systematically identifying these symmetries remains a challenge. In this paper, we formalize data-dependent parameter symmetries and characterize loss invariance and the group-action axioms through infinitesimal conditions, which provide objectives for jointly learning group generators and nonlinear action maps. Our framework systematically uncovers parameter symmetries, including previously unknown ones. To study larger networks, we establish conditions under which subnetwork symmetries extend to the full model. The same construction gives an explicit family of finite-batch symmetries, providing both analytical examples and a foundation for discovery through small subnetworks. Using the infinitesimal characrization and subnetwork construction, we implement a framework for automated discovery of parameter symmetries, and successfully uncovered symmetries in various architectures, including pretrained transformer models.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Parameter space symmetry

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UYs1OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds a framework for automatically discovering parameter symmetries of neural networks. The framework is also applied theoretically in the paper to find data-dependent parameter symmetries in shallow neural networks with smooth row-wise activation. The paper also provides theoretical results for when symmetries of smaller subnetworks yields symmetries of the full network. Error bounds for approximate symmetries are provided and several experiments to validate their findings are conducted, showing that they can reproduce known symmetries in, for example, ReLU networks, as well as discover new ones in larger architectures.

Critical strengths

Builds a framework for discovering symmetries in parameter space and verifies it experimentally. Since parameter symmetries are relevant for things like network identifiability, the construction of new architectures, and conservation laws for training dynamics, discovering unknown symmetries automatically has the potential to be quite useful both theoretically and practically.

I would thus recommend this paper be considered for acceptance to ICLR 2027, given that critical weaknesses are dealt with in a reasonable manner.

Critical weaknesses

Section 6: I find the section detailing the experiments quite hard to follow as it stands. It would be good to clearly state, for each experiment, what exactly is the experimental setup. This is something that I believe needs to be fixed. It also not clear to me if the symmetries learned for the Pythia model are data-dependent or not. The section also introduces some terminology that is never really explained in the text:

line 393: "direct-map",

line 395: the meaning of "retain" as it is used here is not clear to me. Is it preservation of network output that is being referred to here?

line 415: "moving unit"

line 455: "boundary output"

line 463: "protected-output"

line 464: "coefficient radius"

I also find it hard to read from Appendix D: Experimental Setup and Validation exactly what the setup is. One example is that the paragraph under the "Examining the learned structure" header, reads like instructions for what would be desirable. For example, "Numerical dimension estimates should report the rank tolerance and its sensitivity," reads like this is something that would be a desirable property, not that its something you did.

I believe that any paper accepted to ICLR 2027 should be an easy read, it is therefore of great importance that this section of the paper be revised. Should my concerns about this section be alleviated, I would consider raising my score to at least a 3.

Questions for authors

line 21: "characrization" →\to "characterization"

line 22: "uncovered" →\to "uncover" since everything else is written in the present tense.

line 233-242: The text in the figure does not quite match the caption text. In the Figure c) it reads "not a subnetwork", but in the caption it says that c) is an example of when the symmetries of a subnetwork are not symmetries of the full network.

line 289: I think it would be nice to include the example from Appendix C in the text. I also think that it would be nice if the example included the very short calculation which shows that the output is preserved.

line 286: Should "subgraph" be "subnetwork?

line 366-368: How big is the LHS and RHS of (13) and (14) empirically in your experiments? Is it possible to include this information in the experiment section? Perhaps it is included and I missed it.

line 474: "Discussion" →\to "Conclusion"

line 485: A couple of good references for functional/parameter symmetries used to create new equivariant architectures are Agrawal and Ostrowski 2022 and Agrawal and Ostrowski 2023. Perhaps one of these references should be included.

line 1417-1422: In Fig. 4 in Appendix E, the caption does not seem to match the figure. For example, the caption mentions shading, but no shading is visible in the figure.

It is unclear to me after reading the paper why exactly the data-dependent symmetries are of interest. I think it would benefit the paper if some good examples could be given throughout.

References I use, which are not your own references:

Agrawal and Ostrowski, 2022, "A classification of GG-invariant shallow neural networks".

Agrawal and Ostrowski, 2023, "Densely connected G-invariant deep neural networks with signed permutation representations".

5 comments

We thank the reviewer for the detailed read and helpful suggestions on improving the writing of the experiment section.

Section 6: I find the section detailing the experiments quite hard to follow as it stands. It would be good to clearly state, for each experiment, what exactly is the experimental setup. …

Thank you for the detailed list; we agree with it. We have rewritten Section 6. The section now begins with a paragraph stating the shared setup and a paragraph defining the output error, composition error, coefficient radius, and motion. Each subsection then states the network, the transformed parameters, the preserved function, and the data before giving results. Appendix D.2 defines all test quantities in one place, and Table 1 lists the setup of each small-network experiment.

It also not clear to me if the symmetries learned for the Pythia model are data-dependent or not.

The symmetries learned for the Pythia models are data-dependent. The action network takes in a batch of data as part of its input. We added clarification and the data used in the main text.

Undefined terminology (lines 393, 395, 415, 455, 463, 464).

We moved all their definitions into the main text, and removed irrelevant terms (“direct-map” and “retain”). "Moving unit" and "compensating units" are defined in Section 6.2, following Corollary 4.6: the moving unit changes its incoming weights, and the k compensating units change their outgoing weights to cancel the resulting output change. "Boundary output" is renamed "subnetwork output", defined in Section 6.3 as the vector the selected unit passes to the next layer. "Protected output" is replaced by "protected positions", the k token positions at which the block output is preserved, defined in Section 6.3. The "coefficient radius" ρ\rho is defined in the Section 6 setup.

I also find it hard to read from Appendix D: Experimental Setup and Validation exactly what the setup is.

We rewrote Appendix D, which now directly describes what we did.

Response to questions

We fixed all typos. We appreciate the careful reading. We also make the following changes in the revised version.

line 233-242: The text in the figure does not quite match the caption text. In the Figure c) it reads "not a subnetwork", but in the caption it says that c) is an example of when the symmetries of a subnetwork are not symmetries of the full network.

We removed the text in the figure for the subnetwork example to avoid confusion, and used different colors for the example and nonexample instead.

line 289: I think it would be nice to include the example from Appendix C in the text. I also think that it would be nice if the example included the very short calculation which shows that the output is preserved.

We moved the example from Appendix C into the main text and included the calculation. We agree that the example is helpful for understanding the construction of the action and appreciate the suggestion.

line 366-368: How big is the LHS and RHS of (13) and (14) empirically in your experiments? Is it possible to include this information in the experiment section? Perhaps it is included and I missed it.

We measured both sides and added the result as Appendix F, with a pointer after Proposition 5.1. The measured error is below the bound in every case. On the finest grid, the composition error is at most 0.31 of the right-hand side of (13) and the output error at most 0.52 of the right-hand side of (14).

line 485: A couple of good references for functional/parameter symmetries used to create new equivariant architectures are Agrawal and Ostrowski 2022 and Agrawal and Ostrowski 2023. Perhaps one of these references should be included.

Thanks for the pointer. We included the two references for using symmetries to create new equivariant architectures.

line 1417-1422: In Fig. 4 in Appendix E, the caption does not seem to match the figure. For example, the caption mentions shading, but no shading is visible in the figure.

The spread across the five seeds is smaller than the line width, so the band that marks it cannot be seen. We added this clarification to the caption.

It is unclear to me after reading the paper why exactly the data-dependent symmetries are of interest. I think it would benefit the paper if some good examples could be given throughout.

There are many applications using known data symmetries as mentioned in the introduction; data dependent symmetry generates a much larger set of symmetries, or a larger search space, for these applications to use. Examples include searching for equivalent models that are easy to optimize early in the training, searching for models with equivalent performance on a dataset but easier to compress, aligning models for better merging, fine-tuning on new data while preserving model behavior on a specific dataset, efficient sampling in Bayesian neural networks, and symmetry-aware architectures in weight space learning. We refer to the opening paragraph in the manuscript as well as a recent survey paper [1] for more areas of applications.

We added an illustrative example where learned data dependent symmetry is used to improve quantization, a model compression technique (Appendix K). Symmetry can be used to search for equivalent models with the same loss but smaller quantization error. When the architecture has no continuous functional symmetries, using data dependent symmetry successfully improves quantization. When the architecture has known functional symmetries, augmenting them with learned data dependent symmetry improves quantization over using known functional symmetries alone.

Reviewer_UYs1OpenReview ↗

Response to the author's response

Thank you for rewriting the experiment section and appendix, it now reads much more clearly. Thank you also for clarifying the points I raised in my questions, especially the explanation for why the data-dependent symmetries are of interest. I also think that Appendix F and the added example after Corollary 4.6 were good additions.

One follow-up comment: The reference to the papers by Agrawal and Ostrowski in the conclusion simply appear on their own in parenthesis. Either there should be an accompanying sentence describing the context in which they use the parameter symmetries, or the reference should be removed. But it should not appear as it now does.

If this last minor point is resolved, I would be happy to raise my score to a 3.

Reviewer_F5ryOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a formal definition and an automated, framework-style approach for discovering parameter-space symmetries in neural networks. It formalizes symmetries and how to derive network characteristics from them. In addition to the theoretical work, it includes experiments.

Critical strengths
  • Extensive formal approach
  • Practical trainable implementation
Critical weaknesses
  • Experimentation is incomplete, with much detail missing
  • Unclear if and how the approach scales
  • Limitations of the approach are not discussed
Questions for authors

The introduction suggests that this gap is being tackled: "While several frameworks have been developed to unify known symmetries, whether the symmetries in current literature are complete remains an open question (Zhao et al., 2026)" with "an automated approach for discovering parameter space symmetries and their group actions in neural networks." but concludes with "Mathematically, the existence and number of symmetries in neural network parameter spaces remain open questions."

One feels that this is at least misleading; it should be clearer what one can expect from the framework.

  • This connects to a missing Limitations section. What can the framework do and what not?

The theoretical framework appears to be well set up. But the empirical work lacks detail and appears to be incomplete.

  • Details like running times and memory consumption are not reported, which makes one wonder about the scalability of the approach. After all the networks analyzed are small.

  • The reporting is incomplete, as much of the relevant information is "outsourced" to the appendix; e.g., Section 6.1 requires Appendix F.

  • Or the Pythia experiments (what layers are used, what data, etc). Only page 30 in the appendix reveals that Pythia was considered only on a specific layer (Pythia-160M layer 6 with eight tokens and Pythia-1B layer 3 with 16 tokens).

  • What about the other layers? Which tokens were used- the first, etc. One wonders if similar observations can be made on other layers where FF blocks may be used differently by the language model etc.

Overall, a general statement like the one in the conclusion “In this work, we introduce a framework for formalizing and discovering parameter space symmetries in neural networks.” is too strong and broad. Yes, there is a formalization; yes, there is a framework. But no, it does not appear to be applicable tor any neural network, but comes with important certain bounds, which are not discussed.

2 comments

We thank the reviewer for the constructive suggestions. We address missing experimental details, scalability, and limitations below.

The introduction says the framework addresses whether known symmetries are complete, while the conclusion says existence and number remain open; a Limitations section is needed.

We do not claim to fully close the question of whether the known symmetries are complete, and the conclusion now states this explicitly. We instead propose a way to search for symmetries, test this method on a few architecture classes, and find symmetries that have not appeared in previous literature. We have also extended the discussion on limitations in the conclusion section. In particular, our method fixes the generator count and size in advance, and the learned actions are approximate, with errors bounded by Proposition 5.1.

The theoretical framework appears to be well set up. But the empirical work lacks detail and appears to be incomplete.

In the revised version, we moved all main experimental setup and results to the main text.

Details like running times and memory consumption are not reported, which makes one wonder about the scalability of the approach. After all the networks analyzed are small.

We added a summary of running time and memory in Section 6, and detailed tables for each experiment in the appendix (Tables 10 and 11).

On one NVIDIA H200, learning an action takes 376-461 seconds for the small networks and 395-407 seconds for the pretrained LLMs, with peak GPU memory of 70-73 MB and 116-130 MB during training. The cost of learning an action is set by the subnetwork, which is customizable, instead of the full model. The procedure transfers across model sizes, depths, and block designs with the same recipe. The complete pipeline for a pretrained model, which includes evaluating logits of the full model after installing the transformed weights, peaks at 3.91 GB for Pythia-1B, 5.04 GB for OPT-1.3B, and 5.87 GB for Qwen2.5-1.5B.

The reporting is incomplete, as much of the relevant information is "outsourced" to the appendix; e.g., Section 6.1 requires Appendix F.

We agree. We moved the main setup and more detailed results from appendix to Section 6. We retain the full hyperparameters and results in the appendix for reproducibility purposes.

Or the Pythia experiments (what layers are used, what data, etc). Only page 30 in the appendix reveals that Pythia was considered only on a specific layer (Pythia-160M layer 6 with eight tokens and Pythia-1B layer 3 with 16 tokens).

Section 6.3 now states the complete Pythia setup, including the exact layers, the data (the block inputs at the first k tokens of a WikiText-103 article, k = 8 and k = 16), the weights that change, how parameter points are sampled, and how the installed weights are evaluated. The protected positions are the first k tokens of the article.

What about the other layers? Which tokens were used- the first, etc. One wonders if similar observations can be made on other layers where FF blocks may be used differently by the language model etc.

To verify that our method works for other layers and architectures, we added the same experiment on twelve further blocks spread over the depth of three models: Pythia-1B (blocks 4, 7, 11, 15), Qwen2.5-1.5B (blocks 6, 13, 20, 27), and OPT-1.3B (blocks 5, 11, 17, 23), each with three seeds (Appendix J, Table 9, Figure 6). At every block, seed, and radius, the learned action changes the protected logits less than a random change with the same motion, by 57–259 times.

Overall, a general statement like the one in the conclusion “In this work, we introduce a framework for formalizing and discovering parameter space symmetries in neural networks.” is too strong and broad. Yes, there is a formalization; yes, there is a framework. But no, it does not appear to be applicable tor any neural network, but comes with important certain bounds, which are not discussed.

Thanks for the suggestion. The claim is now scoped by the sentences that follow it in the conclusion, including what the framework provides, within which class, and what remains open, followed by the limitations paragraph. The two bounds the framework comes with are stated there: the search class fixed in advance, and the approximation errors, which Proposition 5.1 bounds in terms of the training residuals and which Appendix F measures against those bounds.

Reviewer_F5ryOpenReview ↗

Response to authors' response

Thank you for clarifying my questions. I adapted my score.

Reviewer_GTToOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper rigorously investigates data-dependent parameter symmetries as mathematical objects and develops objectives for jointly learning their group generators and nonlinear actions. It establishes conditions under which subnetwork symmetries extend to the full model, provides an explicit construction of finite-batch symmetries, and demonstrates the framework empirically across several architectures, including pretrained transformers.

Critical strengths
  • The paper is easy to follow. The problem framing is clear, and the approach based on infinitesimal generators is clean and well suited to the problem.
  • The proposed training objective is well motivated, and the experiments demonstrate its effectiveness in discovering approximate parameter symmetries in subnetworks.
Critical weaknesses
  • The finite-data symmetries appear to arise from parameter redundancy in the network. Although their formulation as group actions is theoretically meaningful, their existence under the stated redundancy and rank conditions is somewhat unsurprising. The paper would benefit from clarifying what these symmetries reveal beyond this compensation mechanism.

  • The large model experiments are limited to the Pythia family. The paper would be stronger with experiments across a broader range of practical architectures and a more thorough characterization of the discovered symmetries.

Questions for authors
  • Could the authors clarify the main practical implications of data-dependent parameter symmetries? In particular, can these symmetries support concrete applications or provide insights into network behavior beyond parameter redundancy?

  • Could the authors extend the evaluation to a broader range of practical architectures and systematically characterize the symmetries they discover? What types of symmetries arise in modern architectures, and how might they be classified?

2 comments

We thank the reviewer for the constructive suggestions. We extended the experiments to a broader range of architectures (Qwen2.5-1.5B and OPT-1.3B), and added a downstream application of the learned symmetries to demonstrate their potential in practice. We address each comment below.

The finite-data symmetries appear to arise from parameter redundancy in the network. Although their formulation as group actions is theoretically meaningful, their existence under the stated redundancy and rank conditions is somewhat unsurprising.

The stated rank conditions imply the existence of loss-preserving directions, but these local directions do not establish the existence of symmetry in a trivial way. The fields have to integrate into transformations that satisfy the group action axioms. Similarly, in networks with redundancy, the transformations among equivalent networks may not readily form a group. Formulating them as group actions is useful beyond theoretically too: it allows us to define parameterized families of equivalent models, which helps downstream applications by making it much more efficient to sample and optimize on the set of equivalent models.

What types of symmetries arise in modern architectures, and how might they be classified? The paper would benefit from clarifying what these symmetries reveal beyond this compensation mechanism.

We analyzed the learned symmetries in more depth in Appendix J.3. We found that most of these symmetries can be explained by the compensation mechanism, which is expected, as a loss-preserving transformation is defined by changes in different parameters whose effects can cancel out. Theorem 4.5 also provides an example where all output-preserving deformation of a network decomposes into such compensating transformations. A group theoretic characterization of symmetries in general architectures is nontrivial and among the open questions in this field.

The large model experiments are limited to the Pythia family. The paper would be stronger with experiments across a broader range of practical architectures and a more thorough characterization of the discovered symmetries.

We apply the same discovery method on twelve additional feedforward blocks of three model families: Pythia-1B (blocks 4, 7, 11, 15), Qwen2.5-1.5B (6, 13, 20, 27, a gated SiLU block), and OPT-1.3B (5, 11, 17, 23, ReLU) (Appendix J). At every block, seed, and radius, the learned action preserves model performance, changing the protected logits less than a random change with the same motion by 57–259 times (Table 9, Figure 6).

Could the authors clarify the main practical implications of data-dependent parameter symmetries? In particular, can these symmetries support concrete applications or provide insights into network behavior beyond parameter redundancy?

There have been many applications using known data symmetries; data dependent symmetry generates a much larger set of symmetries, or a larger search space, for these applications to use. Examples include searching for equivalent models that are easy to optimize early in the training, searching for models with equivalent performance on a dataset but easier to compress, aligning models for better merging, fine-tuning on new data while preserving model behavior on a specific dataset, efficient sampling in Bayesian neural networks, and symmetry-aware architectures in weight space learning. We refer to the opening paragraph in the manuscript as well as a recent survey paper [1] for more areas of applications.

Data dependent parameter symmetries also provide insights into many aspects of network behaviors. For example, as UYs1 also notes, they give rise to additional conservation laws for training dynamics, and are directly related to network identifiability, which has implications on interpretability and explainability.

We added an illustrative example where learned data dependent symmetry is used to improve quantization, a model compression technique (Appendix K). Symmetry can be used to search for equivalent models with the same loss but smaller quantization error. When the architecture has no continuous functional symmetries, using data dependent symmetry successfully improves quantization. When the architecture has known functional symmetries, augmenting them with learned data dependent symmetry improves quantization over using known functional symmetries alone.

[1] Zhao, Walters, Yu. Symmetry in Neural Network Parameter Spaces. TMLR 2026.

Reviewer_GTToOpenReview ↗

Reply for the authors

Based on the authors’ rebuttal, I think my questions have been adequately addressed. I especially appreciate the extension of the empirical results to a broader range of model families, which addresses what I considered the weakest aspect of the original submission. In light of these changes, I have decided to update my evaluation to weak accept and thank the authors for their rebuttal work.

My weak accept rating reflects the fact that I have not kept up with the latest work on neural network symmetry, rather than any clear weakness in the paper. Among the papers in my review batch, I consider this one the strongest, with clear, easy to follow and human written content, and recommend its acceptance.

Comments on the paper

1 comment

We thank the reviewers for their helpful comments. We are encouraged that the reviewers find our problem framing clear and well motivated and recognize both our theoretical contribution and practical implementation. We are also glad that the paper is easy to follow outside the experiment section, which we improved significantly.

We made the following changes on the revised paper.

  • We moved the main experimental setup and results into the main text. We also added full details on runtime and memory use in the appendix.
  • We repeated the pretrained-model experiment on a wider range of models and layers (twelve feedforward blocks of Pythia-1B, Qwen2.5-1.5B, and OPT-1.3B) to demonstrate the applicability on modern architectures.
  • We added a downstream application of the learned data-dependent symmetries -- reducing accuracy loss in post-training quantization (Appendix K).
  • We revised the conclusion section to clarify what the framework provides and what remains open, and expanded the discussion of limitations.

We address each reviewer's comments in detail in individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.