- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper uses prior work to classify the interaction between pairs of gradients, corresponding to two training examples, into positive interactions (positive inner product) and negative interactions (negative inner product). It then proposes a regularizer (GCAM) that depends on disagreement between pairs of training example gradients and shows how it works on grokking and cifar10. The main motivation is to speed-up grokking.
Critical strengths
- Comprehensive experiments.
- The proposed regularizer seems quite interesting.
- The method should be generally applicable.
Critical weaknesses
- Overuse of AI for writing the paper to the point where it's very hard to understand it or even read.
- Unclear what is section 2 trying to say.
- Some sections are extremely short.
- Conclusion is missing.
- Generally writing is very poor.
- In the appendix, table 3, the authors try adding positive gradient interactions into the loss, instead of negative ones. The results seem similar. doesnt that undermine the main narrative of the paper? The discussion on this point doesnt help.
Questions for authors
- How are ODE solves used instead of AdamW for training a model? is that conventional or even possible?
- Do you backpropagate through the gradients in the regularizer or do you have a "stop grad" on them?
- Does the method work on larger datasets than CIFAR10? ImageNet? Language pretraining? Self supervised learning? etc.
- Why would the proposed regularizer accelerate grokking? How is it even related to grokking?
3 comments
My main concern of the paper being highly unclear still remains. The paper requires major rewriting. The biggest scale experiment is still cifar10. I cannot recommend acceptance.
Thank you for the follow-up. It does not say which of our answers fell short, so we list where the revised paper answers each point of your review.
- What GCAM penalizes: not pairwise inner products. For each weight it multiplies the batch's summed positive and summed negative gradient; this equals the conflict summed over all pairs of examples on that weight, obtained without forming the matrix of pairwise products, and it also counts opposition that cancels inside a pairwise dot product (§5).
- Section 2, short sections, conclusion: rewritten; sections merged; Conclusion added (§7).
- Table 3 (agreement works similarly): expected, since both coordinate penalties contain the opposition that a dot product cancels within each pair (§5). At the dot level the sign decides: kept on, conflict 12, agreement 4, full norm 0 of 16 seeds (Table 2). With less data or a harder task, GCAM beats a gradient-size penalty at the same learning rate: 15 vs 1 of 16 seeds keep ≥99% accuracy with 15% of the data, and 7 vs 1 of 8 held-out seeds reach 90% on a polynomial task (Figure 6).
- Q1, ODE solvers: in those runs AdamW is the optimizer; Heun and RK4 only probe the gradient passed to it (Figure 2).
- Q2, stop-gradient: none (Algorithm 1).
- Q3, scale: CIFAR-100 added: GCAM 51.87%, SAM 51.80%, AdamW 49.07% (Table E.2).
- Q4, relation to grokking: grokking can be delayed when examples get in each other's way. In runs that fail, attention gives the MLP nearly the same input for every example, and examples with different answers pull the same MLP weights in opposite directions; the batch gradient hides this because the pulls cancel. GCAM penalizes these opposing pulls weight by weight; on attention it makes the MLP input example-specific, and the run generalizes (§6.1, Figure 4).
The updated general response at the top of the forum gives the argument in five steps. If a sentence still reads unclearly, we would be grateful if you could point it out, and we will revise it.
Response to Reviewer 3xG9
We thank the reviewer for the constructive feedback. Beyond speed, the paper explains the delay — the interaction hidden in the batch-gradient norm and where it sits in the model — and rescues runs that do not generalize under AdamW within the training budget (general response, results 1–4).
Writing. The abstract and Section 2 are rewritten, a Conclusion (Section 7) replaces the former Discussion, the former Section 6 is merged into Section 5, and the results are regrouped into four subsections.
Table 3 (now Table 2). You are right that Table 3 shows rescue is not specific to the conflict penalty. The revision explains why and where the sign does matter (§6.3). The positive penalty there is the coordinate agreement Pc, and Pc−Nc=P−N exactly: both contain X, the coordinate opposition that the dot product cancels within a pair (Pc=P+X, Nc=N+X, with X 10–57 times N at V,O). In width-128 AdamW runs at V,O, Pc/Nc is 1.00–1.06 (Appendix C), consistent with their similar rescue, while the signed part P−N alone rescues 0 of 8. By Proposition 2(b), Nij+Xij is the exact maximum first-order interference per unit step over diagonal factors in [0, 1] for that pair, so X can become interference. The two separate under class-blocked CIFAR-10 batching: with about four classes per batch, Nc improves on Pc at a common coefficient on 4 of 5 seeds (64.37 vs 63.26), while their accuracies are similar with shuffled batches (69.34 vs 69.19; §6.3, Appendix E). At the dot level, where X is absent, the sign also separates kept-on runs (general response).
Q1. AdamW remains the optimizer; Heun and RK4 evaluate the batch gradient at intermediate points and pass it to AdamW; Figure 2 labels them "AdamW + Heun probe" and "AdamW + RK4 probe".
Q2. There is no stop-gradient: the penalty is differentiated through both per-sample gradients (Algorithm 1, step 3).
Q3. Small algorithmic tasks isolate delayed generalization, as in the grokking studies we build on; within them the effect holds as width grows to 256 and with three layers (reply to PCJy, Scale), and we add validation-tuned CIFAR-10. Larger datasets are future work.
Q4. At update 500 in AdamW's failing width-128 runs, one vector common to all examples makes up 92% of the mean squared MLP input at the answer position, so gradients overlap across different answers. Writing the MLP input-weight gradient as Gi=uixi⊤, with xi the layer input and ui its backpropagated derivative, Gi⋅Gj=(ui⋅uj)(xi⋅xj) (Appendix A.2). GCAMVO applied from update 500 reduces the shared fraction from .92 to .16 by update 1,000, the MLP coupling drops before the fit, and generalization follows (Figure 4). Penalizing the MLP-gradient overlap Ω directly rescues failing runs, and rewarding it leaves six successes below 99% within the budget (§6.1, Appendix Figure C.2).