- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In this work, the authors investigate the effective rank in deep residual networks, discussing how architectural components such as skip connections, normalization and width expansion impacts rank collapse.
Critical strengths
- The work proposes an interesting spectral interpretation of different commonly used architectural choices.
- The work contains a large number of experimental results to showcase the analysis presented.
Critical weaknesses
- My main concern is regarding the implicitly positive treatment of rank collapse in the paper presentation. More precisely, it is not clear (and I am personally not even convinced it's true) that effective rank preservation should be something we want to aim in the first place. In fact, given that linear maps are the only learnable parts of modern networks, rank collapse is the only general mechanism which allows to discart and compress information.
- While intuitive, the discussed tradeoff between rank preservation and compositional depth given by residual layers seems to be largely speculative. Moreover, the claimed universal relationship between effective rank and mean path length is established only at initialization, without any further discussion on how this might evolve during training.
- The current presentation mainly explains and reinterprets empirically observed effect of certain architectural choices, but it remains unclear what is the conclusion from these observations.
- The scale of the experiment is small to moderate, and they do not seem to include multiple seeds per run, which might be important in the proposed type of analysis. Moreover, some experiments (such as the one presented in figure 5), seem to be somehow detached from the main discussion, in particular the main focus seems to be on the trainability-rank collapse relationship, which is in fact not deeply discussed in the current version of the manuscript.
Questions for authors
- Is it possible for the authors to justify (even numerically) that mean path length is a useful predictor for compositional ability?
- Can the proposed framework lead to novel practically useful architectural recommendations?