- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper tries to connect rank collapse with the phenomenon of grokking.
Critical strengths
The authors address an important research question, that is trying to understand whether rank collapse is necessary for grokking.
Critical weaknesses
The paper is very badly written and, because of that, very difficult to understand. Many things are not defined properly, the jargon introduced for the first time by the authors is left for granted, without explaining it. In this way, I have barely identified the main topic and question that the authors try to address in their work, but I cannot definitely understand which is the main finding of this paper, nor how their results compare with the existing literature, since a "related works" section is completely missing.
Because of that, I am forced to reject the paper and encourage the authors to resubmit their work after a consistent rewriting of the paper.
Questions for authors
The questions are many since the paper is very hard to read.
For instance, in the abstract, the authors write:
"We ask whether preventing it changes generalisation, using a rank pin on the token embedding that is magnitude-matched online to the force driving the natural collapse, and we run the same pin on both sides of the comparison."
What does "rank pin" means? "magnitude" of what? which "force"? "both sides of comparison"...which comparison are we talking about?
In the introduction, the authors write:
"We answer with a measured contrast, obtained by running one causal intervention on both sides of a dissociation. The intervention is a pin that holds a chosen rank statistic near its initial value, magnitude-matched on each task to the force that drives its natural collapse, and replicated penalty-free by hard projection."
What does "dissociations" means? What "rank statistics" are we talking about? That is, rank of what? "hard projection" of what?
I could keep on going for the whole paper. In this way, it is impossible to understand the author's main results.