- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper provides an analysis of gradient flow in neural networks, followed by identifying how the learning direction changes as time increases. The authors then comment on how weight decay can numerically cause better generalisation.
Critical strengths
S1: The paper offers a perspective on the dynamics under gradient flow, and identifies that in the overparameterized setting the instantaneous update to the weights is rank constrained.
Critical weaknesses
W1: The paper does not offer a stated theorem or result with derivation.
W2: The in-subspace rotation remark is automatic from the fact that V^T V = I
W3: The paper doesn’t reach a conclusion in the main body, but offers a lengthy introduction to gradient flow. It would be better to cut the derivation from first principles, and expand on the novel part
W4: Limited experimentation
Questions for authors
Q1: What is the main takeaway from the paper? The authors state that from the experiments weight decay increases generalisation – is there an argument for this based on the analysis in the main paper?