- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds a framework for automatically discovering parameter symmetries of neural networks. The framework is also applied theoretically in the paper to find data-dependent parameter symmetries in shallow neural networks with smooth row-wise activation. The paper also provides theoretical results for when symmetries of smaller subnetworks yields symmetries of the full network. Error bounds for approximate symmetries are provided and several experiments to validate their findings are conducted, showing that they can reproduce known symmetries in, for example, ReLU networks, as well as discover new ones in larger architectures.
Critical strengths
Builds a framework for discovering symmetries in parameter space and verifies it experimentally. Since parameter symmetries are relevant for things like network identifiability, the construction of new architectures, and conservation laws for training dynamics, discovering unknown symmetries automatically has the potential to be quite useful both theoretically and practically.
I would thus recommend this paper be considered for acceptance to ICLR 2027, given that critical weaknesses are dealt with in a reasonable manner.
Critical weaknesses
Section 6: I find the section detailing the experiments quite hard to follow as it stands. It would be good to clearly state, for each experiment, what exactly is the experimental setup. This is something that I believe needs to be fixed. It also not clear to me if the symmetries learned for the Pythia model are data-dependent or not. The section also introduces some terminology that is never really explained in the text:
line 393: "direct-map",
line 395: the meaning of "retain" as it is used here is not clear to me. Is it preservation of network output that is being referred to here?
line 415: "moving unit"
line 455: "boundary output"
line 463: "protected-output"
line 464: "coefficient radius"
I also find it hard to read from Appendix D: Experimental Setup and Validation exactly what the setup is. One example is that the paragraph under the "Examining the learned structure" header, reads like instructions for what would be desirable. For example, "Numerical dimension estimates should report the rank tolerance and its sensitivity," reads like this is something that would be a desirable property, not that its something you did.
I believe that any paper accepted to ICLR 2027 should be an easy read, it is therefore of great importance that this section of the paper be revised. Should my concerns about this section be alleviated, I would consider raising my score to at least a 3.
Questions for authors
line 21: "characrization" "characterization"
line 22: "uncovered" "uncover" since everything else is written in the present tense.
line 233-242: The text in the figure does not quite match the caption text. In the Figure c) it reads "not a subnetwork", but in the caption it says that c) is an example of when the symmetries of a subnetwork are not symmetries of the full network.
line 289: I think it would be nice to include the example from Appendix C in the text. I also think that it would be nice if the example included the very short calculation which shows that the output is preserved.
line 286: Should "subgraph" be "subnetwork?
line 366-368: How big is the LHS and RHS of (13) and (14) empirically in your experiments? Is it possible to include this information in the experiment section? Perhaps it is included and I missed it.
line 474: "Discussion" "Conclusion"
line 485: A couple of good references for functional/parameter symmetries used to create new equivariant architectures are Agrawal and Ostrowski 2022 and Agrawal and Ostrowski 2023. Perhaps one of these references should be included.
line 1417-1422: In Fig. 4 in Appendix E, the caption does not seem to match the figure. For example, the caption mentions shading, but no shading is visible in the figure.
It is unclear to me after reading the paper why exactly the data-dependent symmetries are of interest. I think it would benefit the paper if some good examples could be given throughout.
References I use, which are not your own references:
Agrawal and Ostrowski, 2022, "A classification of -invariant shallow neural networks".
Agrawal and Ostrowski, 2023, "Densely connected G-invariant deep neural networks with signed permutation representations".
5 comments
Response to questions
We fixed all typos. We appreciate the careful reading. We also make the following changes in the revised version.
line 233-242: The text in the figure does not quite match the caption text. In the Figure c) it reads "not a subnetwork", but in the caption it says that c) is an example of when the symmetries of a subnetwork are not symmetries of the full network.
We removed the text in the figure for the subnetwork example to avoid confusion, and used different colors for the example and nonexample instead.
line 289: I think it would be nice to include the example from Appendix C in the text. I also think that it would be nice if the example included the very short calculation which shows that the output is preserved.
We moved the example from Appendix C into the main text and included the calculation. We agree that the example is helpful for understanding the construction of the action and appreciate the suggestion.
line 366-368: How big is the LHS and RHS of (13) and (14) empirically in your experiments? Is it possible to include this information in the experiment section? Perhaps it is included and I missed it.
We measured both sides and added the result as Appendix F, with a pointer after Proposition 5.1. The measured error is below the bound in every case. On the finest grid, the composition error is at most 0.31 of the right-hand side of (13) and the output error at most 0.52 of the right-hand side of (14).
line 485: A couple of good references for functional/parameter symmetries used to create new equivariant architectures are Agrawal and Ostrowski 2022 and Agrawal and Ostrowski 2023. Perhaps one of these references should be included.
Thanks for the pointer. We included the two references for using symmetries to create new equivariant architectures.
line 1417-1422: In Fig. 4 in Appendix E, the caption does not seem to match the figure. For example, the caption mentions shading, but no shading is visible in the figure.
The spread across the five seeds is smaller than the line width, so the band that marks it cannot be seen. We added this clarification to the caption.
It is unclear to me after reading the paper why exactly the data-dependent symmetries are of interest. I think it would benefit the paper if some good examples could be given throughout.
There are many applications using known data symmetries as mentioned in the introduction; data dependent symmetry generates a much larger set of symmetries, or a larger search space, for these applications to use. Examples include searching for equivalent models that are easy to optimize early in the training, searching for models with equivalent performance on a dataset but easier to compress, aligning models for better merging, fine-tuning on new data while preserving model behavior on a specific dataset, efficient sampling in Bayesian neural networks, and symmetry-aware architectures in weight space learning. We refer to the opening paragraph in the manuscript as well as a recent survey paper [1] for more areas of applications.
We added an illustrative example where learned data dependent symmetry is used to improve quantization, a model compression technique (Appendix K). Symmetry can be used to search for equivalent models with the same loss but smaller quantization error. When the architecture has no continuous functional symmetries, using data dependent symmetry successfully improves quantization. When the architecture has known functional symmetries, augmenting them with learned data dependent symmetry improves quantization over using known functional symmetries alone.
Response to the author's response
Thank you for rewriting the experiment section and appendix, it now reads much more clearly. Thank you also for clarifying the points I raised in my questions, especially the explanation for why the data-dependent symmetries are of interest. I also think that Appendix F and the added example after Corollary 4.6 were good additions.
One follow-up comment: The reference to the papers by Agrawal and Ostrowski in the conclusion simply appear on their own in parenthesis. Either there should be an accompanying sentence describing the context in which they use the parameter symmetries, or the reference should be removed. But it should not appear as it now does.
If this last minor point is resolved, I would be happy to raise my score to a 3.
We thank the reviewer for the detailed read and helpful suggestions on improving the writing of the experiment section.
Thank you for the detailed list; we agree with it. We have rewritten Section 6. The section now begins with a paragraph stating the shared setup and a paragraph defining the output error, composition error, coefficient radius, and motion. Each subsection then states the network, the transformed parameters, the preserved function, and the data before giving results. Appendix D.2 defines all test quantities in one place, and Table 1 lists the setup of each small-network experiment.
The symmetries learned for the Pythia models are data-dependent. The action network takes in a batch of data as part of its input. We added clarification and the data used in the main text.
We moved all their definitions into the main text, and removed irrelevant terms (“direct-map” and “retain”). "Moving unit" and "compensating units" are defined in Section 6.2, following Corollary 4.6: the moving unit changes its incoming weights, and the k compensating units change their outgoing weights to cancel the resulting output change. "Boundary output" is renamed "subnetwork output", defined in Section 6.3 as the vector the selected unit passes to the next layer. "Protected output" is replaced by "protected positions", the k token positions at which the block output is preserved, defined in Section 6.3. The "coefficient radius" ρ is defined in the Section 6 setup.
We rewrote Appendix D, which now directly describes what we did.