- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper writes the hidden layers as a feature operator and introduces a scale-invariant feature–target alignment , which is an uncentered population kernel–target alignment. The alignment is connected to Source Condition. It proposes normalized gradient descent in which the linear head is periodically reset to zero. The main results are as follows. Starting from , the first normalized step increases (Thm 3.3). increases monotonically over any fixed finite number of blocks for small step sizes (Thm 3.7), and a finite-sample version also holds (Thm 3.12). For a full-matrix parameterization, the best-fit risk decreases strictly (Thm 4.1). Experiments use a one-parameter toy model and six small benchmarks.
Critical strengths
- I think viewing the network as an operator gives a clean formulation. is basis-independent, scale-invariant, and an explicit function of , which makes it convenient for analyzing training dynamics.
- The results extend the silent-alignment result of Atanasov et al. (2022) from whitened two-layer linear networks to general differentiable feature maps, including ReLU networks.
- The finite-sample analysis handles the empirical normalizer and its gradient, which seems non-trivial.
Critical weaknesses
- The theory is local and does not show that resetting helps. Resetting seems to be motivated from an empirical point of view.
- I think the paper could benefit more from articulating why alignment is a good target. It might be because I don't know well about the deep learning theory, so I don't understand what is "Source Condition". Is it an established term in the literature? What does a better alignment mean for DL training?
- Sort of related: Section 2.2 is unclear.
- Please define the source condition before Eq. (8): , with larger giving a faster rate.
- What is Corollary 2.3 a corollary of? No proof is given. Eq. (9) needs to be increasing in , which is not obvious from Eq. (7).
- What is the takeaway? The equivalence holds only within this one-parameter family.
- The empirical evidence seems weak. The main text has only a one-parameter toy. In Table 1 most gains are within one std. Only UCI HAR improves clearly. There are no significance tests, and is never measured on real data.
- I think some results on unnormalized update does not improve alignment in general would help.
Questions for authors
-
Under what conditions (readout, , stopping time) does a larger imply lower excess risk? Given the final head fit ( steps), why is rather than the right target?
-
Do we know how evolves on the real datasets of Appendix B.3 under each of the four methods? I think this analysis would help reader understand the advantages of the proposed method.