- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
In this work, the authors present an empirical analysis which suggests three possible trajectories of the rank of weight matrices. The authors also identify what they name as Destructive Representation Collapse (DRC), namely a persistent low-rank state which do not increment during training. The authors claim that identification of the category of rank trajectory during training can be used as an effective indicator for the success of training. The core of the work is given by a large series of numerical experiments, mainly concentrated on the exploratory side of the rank dynamics.
Critical strengths
The problem investigated by the authors, namely feature learning and representational rank, is an important problem which deserves attention because of the consequences on our understanding about how deep networks act on intermediate representations.
Critical weaknesses
- The presentation is hasty and unclear, in my opinion far from being acceptable at ICLR. I believe this level of presentation could already be sufficient to justify immediate rejection, as it would require a complete restructuring to be brought at an acceptable state.
- The main model proposed is a linear teacher and a general student (is it general? no details are given). This assumption can already explain some part of the results, as linear networks can be exactly represented by certain kinds of architectures (as residual networks) and not by others. No discussion of this kind is included in the work.
- The main corpus of the work contains a disproportionate amount of technical details on the experimental setting, without any substantial substance on the research question, results and hypothesis to explain what is observed.
- Literature review is fairly shallow: there is an enormous body of work studying rank collapse in deep feedforward and residual networks which has been almost completely ignored by the authors.
The main criticisms I want to move regarding this work is that, on its entirety, it reads overall as a very large collection of numerical results without a specific aim or a specific conclusion
Questions for authors
See "Critical weaknesses".