- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper theoretically investigates how the power-law exponent of the representation spectrum evolves during training. For two-layer linear networks, the authors show that, the local power-law exponent of the representation spectrum evolves non-monotonically in the feature-learning regime, and find similar behaviors in the training of non-linear networks in the teacher-student setting.
Critical strengths
To the best of my knowledge, this is the first work to analytically characterize the evolution of the weight/representation power-law exponents during learning. I find the problem interesting and the results novel, particularly the identification of different exponent regimes and their transitions over training.
Critical weaknesses
-
Limited technical novelty. The main theoretical results are built upon the exact learning dynamics of deep linear networks established by Saxe et al. (2013).
-
Clarification of the phase diagram. Figure 3(a) could be improved by specifying the boundaries between different exponent regimes, at least up to scalings. In particular, it would be helpful to characterize the size of the hatched regions and explain how these regions depend on the model parameters.
-
Visualization of the power-law exponents. It is difficult to directly identify the predicted changes in the power-law exponents from Figures 2 and 4. The authors could consider adding reference lines corresponding to the theoretical slopes , , , and , where applicable. This would make the comparison between theoretical predictions and numerical results more convincing.
-
Validation on real datasets. Although the authors conduct experiments with nonlinear networks, these experiments are still restricted to a synthetic teacher–student setting. It would be interesting to investigate whether qualitatively similar spectral dynamics can be observed in more realistic training scenarios. For example, could one estimate the power-law exponents of the input and effective teacher spectra from a real dataset and compare the observed evolution of the representation spectrum with the theoretical predictions?
Questions for authors
See weakness. It is especially important if the authors could compare their theory with existing empirical observation about how the weight spectrum changes over training, at least qualitatively.
6 comments
Thank you for the detailed response and the revisions. I have one remaining question regarding Figure 3(a). Could the authors clarify whether the relative ordering of the three boundaries, (k_{\mathrm{cross}}(t)), (k_{\mathrm{learn}}(t)), and (k_{\mathrm{layer}}(t)), is fixed or can change during training? In particular, do any of these boundaries intersect or merge, and under what parameter conditions?
Thank you for your question. (1) The ordering of and remains fixed during training, whereas (2) can intersect either of the other two boundaries.
We first restate the definitions of these boundaries. For each mode, the mode strength increases monotonically from zero toward during training. The boundaries are implicitly defined by
k_{\mathrm{cross}}(t):\quad& g_k^{aw}(t)=\sqrt{\sigma_w^2(2\sigma_w^2+\sigma_a^2)}, \end{aligned}$$ $$\begin{aligned} k_{\mathrm{learn}}(t):\quad& g_k^{aw}(t)=0.9\gamma_0\theta_k, \end{aligned}$$ $$\begin{aligned} k_{\mathrm{layer}}(t):\quad& g_k^{aw}(t)=\sigma_w^2+\sigma_a^2. \end{aligned}$$ **1. Relationship between $k_{\mathrm{cross}}$ and $k_{\mathrm{layer}}$** The thresholds defining these two boundaries are independent of both $k$ and $t$. Moreover, $g_k^{aw}(t)$ decreases monotonically with $k$. Therefore, their ordering cannot reverse during training. Specifically, solving $$\sqrt{\sigma_w^2(2\sigma_w^2+\sigma_a^2)} < \sigma_w^2+\sigma_a^2$$ gives $$\sigma_a^2>\frac{\sqrt5-1}{2}\sigma_w^2.$$ Under this condition, \$ k_{\mathrm{cross}}(t)>k_{\mathrm{layer}}(t) \$ at all times. If the opposite inequality holds, their ordering is reversed. In the equality case, the two thresholds are identical, and the boundaries coincide. In the feature-learning setting primarily considered in the manuscript, $\sigma_a\gg\sigma_w$, so $k_{\mathrm{cross}}$ lies above $k_{\mathrm{layer}}$. **2. Relationship between $k_{\mathrm{learn}}$ and the other two boundaries** The threshold defining $k_{\mathrm{learn}}$ depends on $k$ through $\theta_k=\theta_0k^{-\beta}$, allowing this boundary to intersect the other two. Equating the respective thresholds gives the following intersection ranks: $$k_{\mathrm{cross,learn}}^* =\left( \frac{0.9\gamma_0\theta_0} {\sqrt{\sigma_w^2(2\sigma_w^2+\sigma_a^2)}} \right)^{1/\beta},$$ $$k_{\mathrm{layer,learn}}^* =\left( \frac{0.9\gamma_0\theta_0} {\sigma_w^2+\sigma_a^2} \right)^{1/\beta}.$$ If an intersection rank lies within the analyzed mode range $1<k<D$, the corresponding pair of boundaries intersects at a finite training time.
Response to VVjb
We thank the reviewer for carefully reading and evaluating our manuscript. Below, we describe the revisions made in response to the identified weaknesses.
The main novelty of our study is the identification and analysis of a previously uncharacterized phenomenon: the nonmonotonic evolution of power-law exponents, revealed by applying the learning dynamics established by Saxe et al. (2013) to data with power-law spectra. Contributions 1 and 2 in the original manuscript could have misleadingly suggested technical novelty in the derivation of the learning dynamics themselves. We have therefore rewritten the “Our contributions” section to make the novelty described above clear.
We have clarified these points in Appendices C.2 and C.4. The regime boundaries are defined implicitly and are difficult to express in a simple form, but their intersections and asymptotic values can be obtained explicitly. Adding these expressions directly to Fig. 3(a) would make the figure cluttered. We therefore explain how the boundary locations and the sizes of the regimes depend on the model parameters in Appendix C.4 (line 957 onward), with a reference to this discussion in the main text (line 362).
We have added reference lines to Fig. 2 for the four asymptotic exponents, which are α, 3α+2β, α+2β, and α+β in the revised notation. We have also indicated the input exponent α in Fig. 4; please see our response to the next question. These additions make the agreement between the theoretical exponents and the spectra in Fig. 2 easier to assess visually.
Result 3.3 in the original manuscript considered the limited setting of a nonlinear student learning from a linear teacher using synthetic data. In the revised manuscript, we have replaced this experiment with a more general setting in which a nonlinear MLP learns to denoise real images from CIFAR-10. This is an intrinsically high-dimensional problem, unlike the solvable model in which the modes evolve independently. Because this task has no explicitly defined teacher exponent β, we cannot quantitatively compare the evolution of the exponents with our theoretical predictions. Nevertheless, the representation exponent reflects the input exponent α at initialization and evolves nonmonotonically during training, reproducing the qualitative behavior predicted by the theory. This provides empirical support for the phenomenon in a more general setting beyond the assumptions of our theory.
Best regards,
Authors