acceptodds
Withdrawn from ICLR 2027

Understanding Representation Alignment in Neural Networks through Normalized Gradient Dynamics with Periodic Resets

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Understanding how neural networks develop useful representations during training remains an open problem. We study this question by introducing feature-target alignment, a scale-invariant quantity that measures how well a learned feature map captures the target function. We show that normalized feature learning initially induces an alignment-improving direction in the training. Motivated by this observation, we propose normalized gradient descent with periodic resets, which repeatedly returns training to a regime that promotes target-aligned representation learning. We prove that the resulting updates monotonically improve feature-target alignment for a broad class of parameterized feature maps over any given finite horizon, in both population and finite-sample settings. We further study a full-matrix representation model and show that the same learning principle strictly decreases the misspecified risk, providing a concrete example in which representation learning improves prediction. We also provide empirical support for our theoretical results.

Primary area:
learning theory
Keywords:
feature learning, feature-target alignment, periodic reset, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ASaEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper writes the hidden layers as a feature operator PθP_\theta and introduces a scale-invariant feature–target alignment Q=∥Pθf∥2/∥PθPθ⊤∥HSQ = \lVert P_\theta f\rVert^2 / \lVert P_\theta P_\theta^\top\rVert_{HS}, which is an uncentered population kernel–target alignment. The alignment is connected to Source Condition. It proposes normalized gradient descent in which the linear head β\beta is periodically reset to zero. The main results are as follows. Starting from β=0\beta=0, the first normalized step increases QQ (Thm 3.3). QQ increases monotonically over any fixed finite number of blocks for small step sizes (Thm 3.7), and a finite-sample version also holds (Thm 3.12). For a full-matrix parameterization, the best-fit risk EoptE_{\rm opt} decreases strictly (Thm 4.1). Experiments use a one-parameter toy model and six small benchmarks.

Critical strengths
  • I think viewing the network as an operator gives a clean formulation. QQ is basis-independent, scale-invariant, and an explicit function of θ\theta, which makes it convenient for analyzing training dynamics.
  • The results extend the silent-alignment result of Atanasov et al. (2022) from whitened two-layer linear networks to general differentiable feature maps, including ReLU networks.
  • The finite-sample analysis handles the empirical normalizer and its gradient, which seems non-trivial.
Critical weaknesses
  1. The theory is local and does not show that resetting helps. Resetting seems to be motivated from an empirical point of view.
  2. I think the paper could benefit more from articulating why alignment is a good target. It might be because I don't know well about the deep learning theory, so I don't understand what is "Source Condition". Is it an established term in the literature? What does a better alignment mean for DL training?
  3. Sort of related: Section 2.2 is unclear.
    • Please define the source condition before Eq. (8): ∑ifi2/λis<∞\sum_i f_i^2/\lambda_i^s<\infty, with larger ss giving a faster rate.
    • What is Corollary 2.3 a corollary of? No proof is given. Eq. (9) needs ζ(ν+α)/ζ(α)\zeta(\nu+\alpha)/\zeta(\alpha) to be increasing in α\alpha, which is not obvious from Eq. (7).
    • What is the takeaway? The equivalence holds only within this one-parameter family.
  4. The empirical evidence seems weak. The main text has only a one-parameter toy. In Table 1 most gains are within one std. Only UCI HAR improves clearly. There are no significance tests, and QQ is never measured on real data.
  5. I think some results on unnormalized update does not improve alignment in general would help.
Questions for authors
  1. Under what conditions (readout, nn, stopping time) does a larger QQ imply lower excess risk? Given the final head fit (JJ steps), why is QQ rather than EoptE_{\rm opt} the right target?

  2. Do we know how QQ evolves on the real datasets of Appendix B.3 under each of the four methods? I think this analysis would help reader understand the advantages of the proposed method.

Reviewer_f8i8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on learning useful features in a neural network by treating the network as a learned feature map followed by a linear output layer. It introduces a scale-invariant measure of feature-target alignment, QQ and shows that when the features are normalized, the first feature update increases QQ when the output layer is initialized to zero. The paper further proposes normalized gradient descent with periodic resets of the output layer. Given that the update step size is sufficiently small, it proves that QQ increases monotonically over any fixed finite number of steps, both at the population level and with finite samples. The authors also prove that the same procedure strictly decreases the best-refit risk in a full-matrix feature parameterization problem. Experiments on a toy regression model, a full-max model, and six real datasets are provided.

Critical strengths
  1. The idea of decomposing the negative gradient of the loss with respect to θ\theta into an ascent direction of Q plus a negligible correction term with small β\beta (starting from zero) is interesting and insigntful. The algorithm that builds upon this observation with normalized gradient descent and periodic resets of β to zero is well-grounded.

  2. The analysis covers a broad class of feature maps and provides finite-sample guarantees, including an explicit verification for finite ReLU networks.

Critical weaknesses
  1. No convergence guarantee, and conditions degrade as training converges. Theorems 3.7 and 3.12 only show that QQ increases over a fixed finite horizon, but no convergence is guaranteed. The proofs require β\beta remaining small and approximately proportional to Pˉθf\bar{P}_\theta f within each block (eqs. 32 and 37), which is ensured only by taking η1,η2\eta_1, \eta_2 sufficiently small, with constants depending on TT. For example, in Theorem 3.12, the step sizes and the sample size must be chosen according to ∥a(θ)∥2\|a(\theta)\|_2 at the start of each block. As training approaches a critical point of QQ, ∥a(θ)∥2→0\|a(\theta)\|_2 \to 0, so the required step sizes go to zero and the required sample size grows without bound: the guarantee degrades exactly as training converges. In addition, the paper also does not provide practical guidance for choosing η1,η2\eta_1, \eta_2.

  2. Prediction performance of the returned model is neither analyzed nor properly measured. As β\beta is reset every block, the final predictor depends on the fitting phase of the output layer (lines 10–12 of Algorithm 1, JJ steps on fixed features), which is not analyzed: no result relates the returned (θ,β)(\theta, \beta) to prediction error for general feature maps. In the experiments, Figures 2, 4, and 6 assess the error at the end of blocks, i.e., with β\beta trained only TT steps after a reset, while the traditional baseline is trained continuously. No final fit for β\beta (with θ\theta fixed) is reported. Also, the "generalization error" in Figure 2 is not defined.

Questions for authors
  1. Can you provide a convergence result for fixed step sizes (e.g., convergence to a stationary point of (Q))? If not, could you explain how η1\eta_1 and η2\eta_2 should change with T and with ∣a(θ)∣2|a(\theta)|_2 at the start of each block to guarantee the convergence of the algorithm? Also, can you provide some practical advice on how to choose them?

  2. What value of J is used in the experiments? In Table 1, when is the validation performance measured? At the end of a block or after this final fit of β\beta?

  3. Could you report the prediction error of each method using its final features, with β\beta fitted until convergence? Also, how is the ``generalization error" shown in Figures 2, 4, and 6 defined?

  4. Do the backbones (e.g., ResNet-20) already use BatchNorm? If so, does combining resets with a standard normalization layer on the feature layer give similar improvements? This would help show whether the specific normalization by ∣Σθ∣HS1/2|\Sigma_\theta|_{\mathrm{HS}}^{1/2} is important in practice.

Reviewer_yj1XOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to explain how neural networks learn useful internal representations, going beyond the fixed-kernel picture of the neural tangent kernel regime. It views a network as a learned feature map followed by a linear output layer, and measures the quality of the features by a feature-target alignment QQ: roughly, how much of the target function lies along the directions that the features emphasize, normalized so that rescaling the features does not change it.

Concretely, the features are written as Φθ=PθΨ\Phi_\theta=P_\theta\Psi, where θ\theta denotes the network parameters, Ψ\Psi is a fixed orthonormal basis of functions and PθP_\theta is the matrix of coefficients of the features in that basis. If ff is the vector of coefficients of the target function in the same basis, then Q=∥Pθf∥2/∥PθPθ⊤∥HSQ=\Vert P_\theta f\Vert^2/\Vert P_\theta P_\theta^\top\Vert_{HS}, where ∥⋅∥\Vert\cdot\Vert is the Euclidean norm and ∥⋅∥HS\Vert\cdot\Vert_{HS} the Hilbert-Schmidt (Frobenius) norm. The paper proposes a training procedure that alternates gradient steps on the head and on the features, normalizes the overall scale of the features, and resets the head to zero every TT steps. It proves that, for small enough step sizes, every step of this procedure increases QQ, both with infinite data and with finite samples, and that this covers ReLU networks. In a simpler model where the feature coefficients form an unconstrained matrix, each step also lowers the error of the best linear predictor built on the features. Experiments on a one-parameter toy regression problem, a simulation of the matrix model, and six small benchmarks provide empirical support.

Critical strengths

The paper has the following strengths:

  • QQ does not depend on the choice of basis or on the scale of the features, and its gradient with respect to θ\theta, written ∇Q\nabla Q, has an explicit form (L263-264). Right after the head is reset to zero, the first normalized feature step points along ∇Q\nabla Q (Thm. 3.3, Eqs. 24-27), which is a neat way to see why a freshly initialized output layer pushes the features toward the target. The theory is stated for general parameterized feature maps, the finite-sample analysis handles the estimated normalization and uniform concentration, and App. A.5 verifies the assumptions for finite ReLU networks. I did not find errors in the main proofs, with the limited reading of the Appendix.

  • In the matrix model, each update strictly decreases the error of the best linear predictor on the current features (Thm. 4.1). The ablations separate the effect of resetting from that of normalization (Figs. 3-6, Table 1), use 20 random seeds shared across methods (L2128-2130), and report unfavorable outcomes openly (L2069-2073, L2157-2159).

Critical weaknesses

There are some major concerns with regard to the techniques used in the paper:

  • It seems the guarantees amount to small-step gradient ascent on QQ, and they do not require resets. Let η1\eta_1 and η2\eta_2 be the step sizes for the head and the features, TT the number of steps between resets, and KK the number of reset blocks. In the regime of the theorems, the head stays of order η1T\eta_1 T within a block (Eqs. 31-38), so each feature step is gradient ascent on QQ up to higher-order terms, and the monotonic increase follows from a first-order expansion. All constants depend on TT (L702), so "over any given finite horizon" (L20-22) means that the step sizes must shrink as the horizon grows, and nothing bounds the total improvement in QQ or describes where the iterates converge (acknowledged at L482-483). Moreover, with K=1K=1 Algorithm 1 is normalized gradient descent without any reset, and Thm. 3.7 applied to a single block of any finite length gives the same per-step guarantee. The only theoretical role left for resets is to make the step-size requirement depend on TT rather than on the total number of steps (L356-359), and this requires the gradient of QQ to be nonzero at the start of every block (L296-297), which is assumed rather than shown. The experiments do not probe this regime either: B.1 uses η1=0.1\eta_1=0.1 with T=100T=100 or 400400 (L1951-1954), so η1T=10\eta_1 T=10 or 4040; the authors observe QQ decreasing at T=400T=400 (L2069-2073); and all methods share the same nominal step sizes although normalization changes the effective step size. In Table 1, the gains over standard training are within two standard errors on CIFAR-10 and SVHN and absent on Fashion-MNIST and Wine Quality, normalization alone is best on MNIST, and only UCI HAR shows a clear gain beyond normalization.
  • A higher QQ need not mean a better representation, and the results do not address standard training. QQ is the uncentered alignment (L158), whereas Cortes et al. (2012, Sec. 2.3), whose results the paper uses to motivate QQ (L160-161), note that without centering, alignment "does not correlate well with performance". A simple example within the paper's setting: take three orthonormal functions ψ1,ψ2,ψ3\psi_1,\psi_2,\psi_3, a scalar parameter θ\theta, the two features Φθ=(ψ1,θψ2+ψ3)\Phi_\theta=(\psi_1,\theta\psi_2+\psi_3), and the target f∗=ψ1+ψ2+0.1ψ3f^{\ast}=\psi_1+\psi_2+0.1\psi_3. Then Q(θ)=1+(θ+0.1)21+(1+θ2)2Q(\theta)=\frac{1+(\theta+0.1)^2}{\sqrt{1+(1+\theta^2)^2}}, and the error of the best linear predictor on these features is 1.01−(θ+0.1)21+θ21.01-\frac{(\theta+0.1)^2}{1+\theta^2}. Both decrease in θ\theta on the interval [2.2,10][2.2,10] (QQ peaks near θ=2.2\theta=2.2, while the error vanishes at θ=10\theta=10), so starting from θ=3\theta=3, every QQ-increasing step guaranteed by Thm. 3.7 moves θ\theta toward 2.22.2 and increases the error, from 0.0490.049 to about 0.100.10. Outside the matrix model of Section 4, the paper therefore gives no reason to expect a higher QQ to mean better prediction. Separately, Remark 3.4 (L270-275) states that Thm. 3.3 proves the "silent alignment" phenomenon on a broader class of models, but silent alignment (Atanasov et al., 2022) concerns the early phase of standard, unnormalized gradient descent from small initialization, whereas Thm. 3.3 is a one-step statement for a normalized loss. In the paper's own toy model, QQ under standard gradient descent rises only temporarily (Fig. 4b), and QQ is never measured in the trained networks of B.3.

Additional, less decisive concerns:

  • Positioning: kernel-target alignment is due to Cristianini et al. (2001) rather than Cortes et al. (2012) (L64-65); periodically re-initializing the output layer or later layers is an established technique (e.g., Li et al., 2020, RIFLE; Nikishin et al., 2022; Zhou et al., 2022), yet resets are compared only to dropout (L220-221); and Imani et al. (2022), "Representation Alignment in Neural Networks", is not cited.
  • The finite-sample condition (19) grows linearly in the number of parameters DD, and each block uses fresh samples (L356-357).
  • B.3 omits the optimizer, learning rates, batch size, data augmentation and the unit of TT, reports only validation performance (L2130), and uses cross-entropy for five of the six tasks (L2120), which is outside the theory.
Questions for authors

I have the following questions for the authors:

  1. (Additional evidence; could change my rating.) Can you show that resets are necessary, for example that QQ stops improving without them, or give step sizes that do not depend on the number of blocks together with a lower bound on the total gain? How does the method compare with normalized gradient descent without resets but with a smaller head step size (or weight decay on the head), tuned for each method?

  2. (Clarification; could change my rating.) Beyond Section 4, under what conditions does increasing QQ reduce the error of the best linear predictor, and do your experimental models satisfy them? How does the example above fit your interpretation of QQ? Do the results extend to the centered alignment?

  3. (Additional evidence; could change my rating.) Can you report QQ for the penultimate-layer features during training in B.3, and rerun the toy model with a small η1T\eta_1 T so that the experiments probe the regime of the theorems?

  4. (Clarification.) Does the leading-order decrease in Thm. 4.1 also hold for plain (unnormalized) gradient descent from a zero head, given that the scalar normalization leaves the row space of the feature matrix unchanged (L1721)?

  5. (Clarification.) Which result establishes condition (103) for ReLU networks of depth two or more? Remark A.8 only states that it "can be easily verified" in related works.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.