acceptodds
Withdrawn from ICLR 2027

Neural Tangent Kernel Perspective on Parameter Space Symmetries

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-space symmetries are transformations that modify a model's parameters without changing its outputs. These transformations can be leveraged to accelerate optimization and enhance generalization. Remarkably, applying a single transformation, either before or during training, often suffices to realize these benefits. While the effectiveness of this approach is promising, its underlying mechanisms remain poorly understood. In this paper, we provide an explanation within the Neural Tangent Kernel (NTK) framework, by analyzing how such transformations affect the properties of the kernel. We show that maximizing the loss gradient norm via a symmetry is equivalent to maximizing the alignment between the NTK and the network. Since kernel alignment is known to correlate with the optimization rate in the NTK limit, this result elucidates how loss gradient optimization facilitates faster training. To establish the validity of this approach, we prove that parameter-space symmetries preserve the NTK limit.

Primary area:
learning theory
Keywords:
Neural Tangent Kernel, NTK, Parameter Space Symmetries, Neural Teleportation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_7BrvOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This submission studies how parameter-space symmetries, a kind of transformations that modify a model’s parameters without changing its outputs, affect the properties of the kernel. It relates gradient-norm maximization to a modified kernel alignment, with experiments on FashionMNIST MLPs.

Critical strengths
  • Using the tool of neural tangent kernel to analyze parameter-space symmetries seems to be interesting, which may help the understanding of neural network training behavior.

  • This submission presents useful background on parameter-space symmetries and their applications to neural network optimization.

Critical weaknesses
  • The key concern is that the contribution of this submission appears limited relative to the extensive background. Majority of the main text is about background (section 2&3 occupy page 2-6). Theorem 4.1 shows that NTK limit remains valid after applying a linear parameter-space symmetry, and Appendix F.2 provides a guideline for designing nonlinear symmetries. However, from my part, they do not sufficiently broaden the optimization conclusions. Together with the chain-rule identity in Proposition 5.1 these analysis leaves me unconvinced that this submission provides sufficient new insight beyond the background it develops.

  • The main focus of this paper is confused for me. I think this paper is to study parameter-space symmetries as the title and abstract highlight, but it jumps to talk about neural teleportation in the main body, i.e., section 4. However, in introduction, authors explained neural teleportation is ONE approach of using parameter-space symmetries to accelerate the optimization of neural networks. This presents the inconsistency between the main focus of this paper and the main body analysis. Besides, the empirical evidence of acceleration is not sufficient: Figure 6 shows iteration improvements, but the comparison uses a fixed learning rate and requires 500 preliminary Adam steps (Appendix G, p.22, lines 1159–1160 and 1173–1176). Without a tuned learning-rate comparison or runtime including this overhead, the optimization benefit, along with what the proposed NTK interpretation adds to its explanation, remains insufficiently established.

Questions for authors
  1. Under what conditions does increasing initial alignment improve subsequent convergence relative to the untransformed network? Proposition H.1 currently provides only a lower bound on remaining loss.
  2. Do the Figure 6 improvements persist against a tuned learning-rate baseline and after including the cost of the 500 preliminary Adam steps (Appendix G, p.22, lines 1159–1160)?
Reviewer_n8QuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies parameter space symmetries of neural networks using the neural tangent kernel (NTK) in the infinite-width limit. In particular, the paper seeks to understand why applying parameter symmetries during training (neural teleportation) can improve performance. To this end, the authors show that the NTK-limit survives applying the parameter symmetry. Then, they introduce a novel alignment measure for the NTK that measures alignment of the NTK not with the labels but with the derivative of the loss evaluated on the training data. They show empirically that this measure correlates with performance and analytically that it is proportional to the squared magnitude of the gradients which is the quantity that is maximized in neural teleportation.

Critical strengths
  • The proposed alignment measure is novel and could be an interesting object to study, even outside the application in this paper.
Critical weaknesses
  1. The total contributions seem thin for an ICLR paper. The first 5 pages (more than half) are spent on preliminaries and background. The main result of Section 4 seems to be a straightforward application of the chain rule combined with an assumed bound on the magnitude of the parameter transformation. Section 5 does not discuss parameter symmetries directly but concerns the magnitude of the gradients. The proposed alignment measure is interesting but follows directly from the expression of the gradient norm in Equation (26). The final item in the list of main contributions is relegated to the appendices and not mentioned in the main part of the paper.
  2. The proposed alignment measure seems to be the main result of the paper, but no interpretation is provided. The power of the standard measure in (24) is that it explains the mechanism: the eigenfunctions of the NTK align with the target function. In the new measure, the same interpretation would mean that the eigenfunctions of the NTK align with the derivative of the loss at the training points. Why should this increase performance? Without this interpretation, the conceptual insight is limited, since the mechanism is explained via a new unknown measure: saying that neural teleportation increases this measure which is correlated with performance is not much more insightful than saying that neural teleportation increases performance.
Questions for authors
  1. In l. 189 you mention that nn could may represent any measure of system scale. However, it seems like you only considered the width in the rest of the paper. Could you give an example of other system scales that you mean here? Would depth be such a scale?
  2. I was a bit confused by the discussion in Section 3.2 about how CC is chosen. It is introduced in (15) as a bound which could be arbitrarily loos if nothing else fixes it, so I do not understand how it could interact with the learning rate. What is the "largest useful value of CC" you refer to in l. 252?
  3. What transformation did you apply in the experiments in Figure 2?
  4. It was unclear to me where the NTK limit is needed in Section 5. Could you clarify?
  5. In l. 352 you mention that the symmetry transformations might completely unravel the independence assumption. Can you clarify how?
  6. In l. 485, you mention a clear example of a wide network breaking independence, could you point to which example you mean here?
Reviewer_4UaMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Symmetries in a network are transformations of the parameters that keep the network function unchanged. Prior work has established that one can use symmetries during the training process to speed up convergence and to enhance generalization. No complete theory of how this happens is known to date. The paper looks at the wide network regime in which the NTK framework appears. In this regime the process is easier to analyze. The paper first observes that if one maximizes the gradient norm over the orbit of a given symmetry, then optimizing the Rayleigh quotient of the error signal wrt the NTK is the same as optimizing the norm of the gradient. This is a relatively simple rewriting. The actual claim is then that a large such Rayleigh quotient implies a fast decay. The paper does not in fact show this analytically (there is some numerical evidence shown in Figure 3) but it shows instead that a small such quotient means a slow decay. Some experiments are carried out, but they test the theory only weakly. There are two further results shown in the paper. First, if one teleports a single time this corresponds to preconditioning the original gradient. Second, the paper proves that a single teleportation step still allows for an NTK analysis, assuming this teleportation step is linear and obeys a boundedness condition. Whether “real” transportation steps fulfill those conditions is left open.

Critical strengths
  • the paper shows exactly what a single teleportation step for a linear symmetry corresponds to, namely preconditioning of the gradient; it is simple and direct and does not require the NTK regime

  • the paper identifies the Rayleigh quotient as an important object; maximising it is equivalent to maximising the gradient norm; further this quantity upper bounds the training speed; and according to numerical evidence it in fact predicts training speed

Critical weaknesses
  • Theorem 4.1 requires a boundedness condition and the paper does not show that any "real" teleportation fulfils this condition; in fact, Appendix D cannot test it and the experiments in Appendix G shows entries of the Jacobian around 7

  • the central claim, namely that the Rayleigh quotient is the crucial quantity is not shown in the paper; or at least the important direction is not shown in the paper and only experimental evidence is given; worse, this is somewhat hidden from the reader by an ambiguous framing

  • I am not an expert in this area but a search found several related papers that have not been cited but some citations likely would have been appropriate; to give just a few I found that seemed very relevant: Mishkin et al. 2025, Dinh et al. 2017, Kristiadi 2023, and Shan et al. 2021;

  • the experiments in Appendix G are quite weak and cannot really resolve the question of the supposed speedup

Questions for authors
  • your main Theorem 4.1 requires a boundedness condition; this quantity is computable for both of your cases; why did you not measure it? please do so and report ∥Γ∥\Vert \Gamma\Vert as a function of the width

  • rather than using random initialisation as you did in Figure 3 you should test directly whether a single teleportation step speeds things up; pick a random network and train with and without teleportation; see if the speedup corresponds to the difference of A values; in teleportation the quantity A is not randomly chosen; so the results of Figure 3 cannot bear on the fundamental question

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.