- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies the difference between SGD and GD training trajectories in parameter space. More specifically, they show that if we train a deep linear network to fit a linear function, while the SGD and GD training trajectories are close in function space, SGD introduces drift in parameter space: GD approximately preserves the layer balance; SGD with no label noise redistributes the norm toward the last layer; and SGD with label noise shifts the norms of different layers toward an equilibrium.
Critical strengths
The paper is quite clean. Within the studied setting, they do provide convincing theoretical analysis and experimental results.
Critical weaknesses
This paper considers only linear networks, which have very strong structural properties.
In particular, for linear networks in the analyzed setting, different singular values and subspaces are essentially decoupled. The theoretical analysis heavily relies on these properties, and it is not clear whether the intuition obtained this way carries over to nonlinear networks.
Questions for authors
Can you (at least partially) reproduce the observed phenomena in nonlinear networks, empirically or theoretically?
For example, if we train a fully connected bias-free ReLU network to fit this linear target function, can we observe similar drift? In this setting, the layers are still homogeneous, so we can still shift norms between layers and study how they change during training.