- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper studies how two-layer quadratic neural networks coordinate Fourier features across separate left and right input weight blocks when learning modular addition.
Critical strengths
Clear conceptual distinction between tied and untied input streams Well-controlled surgical interventions Good empirical control by comparing grokking vs non-grokking using same hyper parameters
Critical weaknesses
While the central question (how features from distinct input streams co-locate on the same hidden units to form multiplicative interactions) is natural and well-posed, the theoretical contributions do not really explain grokking, and the empirical decomposition inadvertently reveals that the paper's headline concept ("cross-block relative alignment") plays only a minor ~10% role compared to standard within-block neuron sparsity (~70%).
Questions for authors
Table 4(c) shows that between grokking and non-grokking runs in matched configurations, endpoint overlap comes only in minor part from cross-block relative alignment. Doesn't this imply that cross-block alignment occurs robustly even in non-grokking (memorizing) runs, and that the true bottleneck for grokking is within-block neuron sparsity/concentration rather than cross-block alignment?
3 comments
Increasing score
Good answers. I am willing to increase my score
Thank you
Dear Reviewer 1D1V,
Thank you very much for your reply and for your willingness to raise your score. We are glad that our responses addressed your concerns. Your question about the roles of alignment and concentration led to one of the main improvements of the revision, and we are sincerely grateful for it.
If it is not too much trouble, we would be very grateful if you could update the score in the system at your convenience. We fully understand if this takes some time, and we truly appreciate your support. We remain happy to answer any further questions.
Best regards,
The Authors
Response to Weakness 1 and Question 1
Dear Reviewer 1D1V,
We sincerely thank you for the encouraging assessment, in particular for recognizing the conceptual distinction between tied and untied input streams, the well-controlled surgical interventions, and the empirical control within matched configurations. Your question led to one of the main additions of the revision and sharpened its central message. We respond to the weakness and the question together.
Response to W1 and Q1.
Yes, your reading is correct, and the revision turns it into a quantitative result. The central quantity of the paper is cross-block overlap A=C⋅S, the product of concentration C and alignment S. We prove that alignment has bounded range: S≤1 and S≈1/2 at initialization, so it can add at most about log2 to log-overlap, whereas concentration can add far more (Appendix B.8, Proposition 14). Overlap growth must therefore be carried mainly by concentration: the bound gives a concentration share of at least 72%, against 77% measured. Alignment rises in every run, grokking or not, and on average uses up most of its available range, which is why it does not discriminate between outcomes. In the continued runs, alignment is already close to its ceiling before generalization, and over 90% of the subsequent growth in overlap comes from concentration (Appendix D.9, Table 10).
Pairing and concentration are two parts of one mechanism.
Theory and grokking. In a fixed-energy projected gradient flow, overlap grows monotonically and full overlap is the only attracting state, for every convex loss of the task coefficient (Propositions 16, 18, and 22). From a spread-out state this growth is slow, with a logarithmic rate of order 1/K, which gives a structural reason for the wait before generalization (Proposition 21). The margin decomposition shows that generalization then requires the task margin to overtake the residual and gives a closed-form waiting time (Appendix B.11). Both terms are measured in the new runs: between memorization and generalization, the task margin grows and the residual advantage shrinks in every run with a measurable wait (Table 16). In the continued runs, the remaining wait decreases with overlap within each optimizer (Appendix D.9).
In summary, your observation is now a proven result of the paper, pairing and concentration appear as two parts of one mechanism supported by an intervention, and the theory gives a structural reason for the wait before generalization. We hope that these additions address your concerns, and we would be grateful if you would take them into account in your final assessment. We are glad to provide any further clarification.
Best regards,
The Authors