- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This submission studies how parameter-space symmetries, a kind of transformations that modify a model’s parameters without changing its outputs, affect the properties of the kernel. It relates gradient-norm maximization to a modified kernel alignment, with experiments on FashionMNIST MLPs.
Critical strengths
-
Using the tool of neural tangent kernel to analyze parameter-space symmetries seems to be interesting, which may help the understanding of neural network training behavior.
-
This submission presents useful background on parameter-space symmetries and their applications to neural network optimization.
Critical weaknesses
-
The key concern is that the contribution of this submission appears limited relative to the extensive background. Majority of the main text is about background (section 2&3 occupy page 2-6). Theorem 4.1 shows that NTK limit remains valid after applying a linear parameter-space symmetry, and Appendix F.2 provides a guideline for designing nonlinear symmetries. However, from my part, they do not sufficiently broaden the optimization conclusions. Together with the chain-rule identity in Proposition 5.1 these analysis leaves me unconvinced that this submission provides sufficient new insight beyond the background it develops.
-
The main focus of this paper is confused for me. I think this paper is to study parameter-space symmetries as the title and abstract highlight, but it jumps to talk about neural teleportation in the main body, i.e., section 4. However, in introduction, authors explained neural teleportation is ONE approach of using parameter-space symmetries to accelerate the optimization of neural networks. This presents the inconsistency between the main focus of this paper and the main body analysis. Besides, the empirical evidence of acceleration is not sufficient: Figure 6 shows iteration improvements, but the comparison uses a fixed learning rate and requires 500 preliminary Adam steps (Appendix G, p.22, lines 1159–1160 and 1173–1176). Without a tuned learning-rate comparison or runtime including this overhead, the optimization benefit, along with what the proposed NTK interpretation adds to its explanation, remains insufficiently established.
Questions for authors
- Under what conditions does increasing initial alignment improve subsequent convergence relative to the untransformed network? Proposition H.1 currently provides only a lower bound on remaining loss.
- Do the Figure 6 improvements persist against a tuned learning-rate baseline and after including the cost of the 500 preliminary Adam steps (Appendix G, p.22, lines 1159–1160)?