- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies one gradient step in a two-layer network with Gaussian inputs and a spiked covariance. It identifies two quantities with different roles: A controls the learning-rate scale for an order-one change in task-related overlap, while H/(A^2) gives a sufficient batch-size scale for reproducing the population overlap. At the critical batch scale, the authors derive a joint fluctuation law for coherent alignment and squared overlap. Synthetic experiments test these predictions.
Critical strengths
-
I think separating the strength of the expected update from the batch size needed to estimate it is a useful question. The experiments that hold one geometric quantity approximately fixed while varying the other help make this distinction pretty clear.
-
The joint critical law seems to be an interesting part of the analysis. It distinguishes coherent alignment from squared overlap, which can also increase because of noise. I also appreciate that the authors report finite-dimensional deviations and do not claim that these results establish better downstream prediction.
Critical weaknesses
-
I found the introduction difficult to follow. Terms such as “population onset,” “resolution,” and “coherent alignment” appear before I had a clear understanding of the experiment or what counts as learning. I would have preferred the motivation, model, and observables to come first. In particular, it took me some time to understand that “resolution” concerns fluctuations around an unbiased gradient estimate.
-
I'm not yet sure how much recovering the chosen projection (i.e. q_i) tells us about feature learning. The observables measure projection onto v, but do not control movement orthogonal to v or normalized directional alignment. A batch could reproduce the population projection while producing a large update elsewhere. The authors acknowledge that this is not a prediction-risk result, which is fair, but I would like a clearer explanation of why this observable captures an important learning phenomenon.
Questions for authors
- Could the authors clarify the relationship between recovering Q_a or Q_2 and recovering a useful feature direction? In particular, can their recovery condition hold while the empirical update has poor directional alignment because of orthogonal noise?
2 comments
Thank you again for the thoughtful review. We have added the full-matrix experiment directly addressing the projected-versus-full-gradient recovery question and clarified the role of the task-selected projection. We value the discussion during the rebuttal period and would be very happy to hear any remaining concerns or questions.
We thank the reviewer for the careful assessment and for isolating the distinction between projected recovery and full-gradient directional recovery.
Yes. The new experiment shows that projected task recovery and full-gradient directional recovery are genuinely different notions. This is consistent with the theorem's intended scope: Theorem 4.1 controls the population-selected task projection, not the angle of the complete high-dimensional update.
Coherent projected recovery is already near its population value while most empirical-update energy remains orthogonal. Near (χFL≈0.05), Figure 3(a) gives projected-recovery ratios of 0.994−1.023 with every bootstrap interval containing one. Yet at �0�, the clean and intermediate geometries have
\qquad R_\parallel=0.124,\ 0.119.$$ **Most of the empirical update energy—about $88\%$ in these $n=800$ cells—remains outside the task-selected direction.** Equivalently, the representative full-update cosine is about $0.35$, while the task-directed energy fraction is about $0.12$. Here $�5�$ compares the complete empirical and population update matrices, and $�6�$ is the exact rowwise energy fraction in $�7�$. Appendix M.1 verifies $�8�$ to numerical precision. Figure 3(a) uses 500 independent initializations per realized cell, and the full-matrix panels use 100. The comparison uses matched $\text{tanh}, �9�$, three predeclared geometries, bootstrap 95% intervals, and the complete population update including self-prediction. > Why the projection onto $v=\frac{\Sigma\xi}{\sqrt B}$ is meaningful? **The projection is not arbitrary:** $v$ is selected analytically by the population label gradient. The theorem therefore asks whether the empirical batch resolves the task-relevant component selected by the population dynamics themselves. The leading row update inherits the auxiliary sign\$$a_i$\$. Weighting the row overlap by $�12�$ removes that sign, so $�13�$ measures coherent task-directed movement across rows. By contrast, $�14�$ measures total projected energy and may include incoherent movement. Recovery of $�15�$ is therefore more specific than an increase in projected energy alone. > Scope and presentation The Introduction now distinguishes population onset, finite-batch resolution, and coherent alignment before presenting the results. Sections 2.4--2.5 define the population-selected direction and observables, while Section 5.3 and Figure 3 expose the projected/full-gradient distinction directly. The revised paper therefore makes **projected statistical recovery and full-gradient directional recovery explicitly distinct notions**. **Revision:** Introduction; Sections 2.4--2.5 and 5.3; Figure 3; Appendix M.1; Discussion. --- We hope this clarification of the projected scope, together with the full-matrix evidence, addresses the reviewer's concern about orthogonal noise and directional recovery.