acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Improving Grokking through Cross-Sample Gradient Interactions

Abstract

Grokking exposes a long delay between fitting training data and generalizing. Gradient-norm regularization can alter this delay, but the norm of the averaged gradient does not distinguish small per-sample gradients from cancellation between them. We decompose the squared batch-gradient norm into individual squared norms and positive and negative pairwise interactions. Negative interactions correspond to first-order increases in one sample’s loss under another sample’s update, yet enter norm penalties with a minus sign. We introduce Gradient Conflict-Aware Minimization (GCAM), which penalizes a coordinate-wise upper bound on these interactions using per-coordinate sign aggregation. In modular-addition Transformers, GCAM applied to attention value and output projections accelerates generalization, enables runs that fail under baseline training to generalize within the same budget, and nearly eliminates the memorization–generalization delay at narrow width. Beyond grokking, GCAM improves a vision Transformer’s CIFAR-10 accuracy without augmentation from 66.6% to 72.0% under the evaluated learning-rate choices.

Primary area:
optimization
Keywords:
grokking, cross-sample gradient interactions, gradient regularization, optimization, generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3xG9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper uses prior work to classify the interaction between pairs of gradients, corresponding to two training examples, into positive interactions (positive inner product) and negative interactions (negative inner product). It then proposes a regularizer (GCAM) that depends on disagreement between pairs of training example gradients and shows how it works on grokking and cifar10. The main motivation is to speed-up grokking.

Critical strengths
  • Comprehensive experiments.
  • The proposed regularizer seems quite interesting.
  • The method should be generally applicable.
Critical weaknesses
  • Overuse of AI for writing the paper to the point where it's very hard to understand it or even read.
    • Unclear what is section 2 trying to say.
    • Some sections are extremely short.
    • Conclusion is missing.
    • Generally writing is very poor.
  • In the appendix, table 3, the authors try adding positive gradient interactions into the loss, instead of negative ones. The results seem similar. doesnt that undermine the main narrative of the paper? The discussion on this point doesnt help.
Questions for authors
  • How are ODE solves used instead of AdamW for training a model? is that conventional or even possible?
  • Do you backpropagate through the gradients in the regularizer or do you have a "stop grad" on them?
  • Does the method work on larger datasets than CIFAR10? ImageNet? Language pretraining? Self supervised learning? etc.
  • Why would the proposed regularizer accelerate grokking? How is it even related to grokking?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 3xG9

We thank the reviewer for the constructive feedback. Beyond speed, the paper explains the delay — the interaction hidden in the batch-gradient norm and where it sits in the model — and rescues runs that do not generalize under AdamW within the training budget (general response, results 1–4).

Writing. The abstract and Section 2 are rewritten, a Conclusion (Section 7) replaces the former Discussion, the former Section 6 is merged into Section 5, and the results are regrouped into four subsections.

Table 3 (now Table 2). You are right that Table 3 shows rescue is not specific to the conflict penalty. The revision explains why and where the sign does matter (§6.3). The positive penalty there is the coordinate agreement PcP_c, and Pc−Nc=P−NP_c - N_c = P - N exactly: both contain X, the coordinate opposition that the dot product cancels within a pair (Pc=P+XP_c = P + X, Nc=N+XN_c = N + X, with X 10–57 times N at V,O). In width-128 AdamW runs at V,O, Pc/NcP_c/N_c is 1.00–1.06 (Appendix C), consistent with their similar rescue, while the signed part P−NP - N alone rescues 0 of 8. By Proposition 2(b), Nij+XijN_{ij} + X_{ij} is the exact maximum first-order interference per unit step over diagonal factors in [0, 1] for that pair, so X can become interference. The two separate under class-blocked CIFAR-10 batching: with about four classes per batch, NcN_c improves on PcP_c at a common coefficient on 4 of 5 seeds (64.37 vs 63.26), while their accuracies are similar with shuffled batches (69.34 vs 69.19; §6.3, Appendix E). At the dot level, where X is absent, the sign also separates kept-on runs (general response).

Q1. AdamW remains the optimizer; Heun and RK4 evaluate the batch gradient at intermediate points and pass it to AdamW; Figure 2 labels them "AdamW + Heun probe" and "AdamW + RK4 probe".

Q2. There is no stop-gradient: the penalty is differentiated through both per-sample gradients (Algorithm 1, step 3).

Q3. Small algorithmic tasks isolate delayed generalization, as in the grokking studies we build on; within them the effect holds as width grows to 256 and with three layers (reply to PCJy, Scale), and we add validation-tuned CIFAR-10. Larger datasets are future work.

Q4. At update 500 in AdamW's failing width-128 runs, one vector common to all examples makes up 92% of the mean squared MLP input at the answer position, so gradients overlap across different answers. Writing the MLP input-weight gradient as Gi=uixi⊤G_i = u_i x_i^\top, with xix_i the layer input and uiu_i its backpropagated derivative, Gi⋅Gj=(ui⋅uj)(xi⋅xj)G_i\cdot G_j = (u_i\cdot u_j)(x_i\cdot x_j) (Appendix A.2). GCAMVO\mathrm{GCAM_{VO}} applied from update 500 reduces the shared fraction from .92 to .16 by update 1,000, the MLP coupling drops before the fit, and generalization follows (Figure 4). Penalizing the MLP-gradient overlap Ω\Omega directly rescues failing runs, and rewarding it leaves six successes below 99% within the budget (§6.1, Appendix Figure C.2).

My main concern of the paper being highly unclear still remains. The paper requires major rewriting. The biggest scale experiment is still cifar10. I cannot recommend acceptance.

Thank you for the follow-up. It does not say which of our answers fell short, so we list where the revised paper answers each point of your review.

  • What GCAM penalizes: not pairwise inner products. For each weight it multiplies the batch's summed positive and summed negative gradient; this equals the conflict summed over all pairs of examples on that weight, obtained without forming the B×BB \times B matrix of pairwise products, and it also counts opposition that cancels inside a pairwise dot product (§5).
  • Section 2, short sections, conclusion: rewritten; sections merged; Conclusion added (§7).
  • Table 3 (agreement works similarly): expected, since both coordinate penalties contain the opposition that a dot product cancels within each pair (§5). At the dot level the sign decides: kept on, conflict 12, agreement 4, full norm 0 of 16 seeds (Table 2). With less data or a harder task, GCAM beats a gradient-size penalty at the same learning rate: 15 vs 1 of 16 seeds keep ≥99% accuracy with 15% of the data, and 7 vs 1 of 8 held-out seeds reach 90% on a polynomial task (Figure 6).
  • Q1, ODE solvers: in those runs AdamW is the optimizer; Heun and RK4 only probe the gradient passed to it (Figure 2).
  • Q2, stop-gradient: none (Algorithm 1).
  • Q3, scale: CIFAR-100 added: GCAM 51.87%, SAM 51.80%, AdamW 49.07% (Table E.2).
  • Q4, relation to grokking: grokking can be delayed when examples get in each other's way. In runs that fail, attention gives the MLP nearly the same input for every example, and examples with different answers pull the same MLP weights in opposite directions; the batch gradient hides this because the pulls cancel. GCAM penalizes these opposing pulls weight by weight; on attention it makes the MLP input example-specific, and the run generalizes (§6.1, Figure 4).

The updated general response at the top of the forum gives the argument in five steps. If a sentence still reads unclearly, we would be grateful if you could point it out, and we will revise it.

Reviewer_PCJyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper explores grokking through cross‑sample gradient interactions. It decomposes batch‑gradient norm and reveals flaws in conventional gradient‑norm regularization. Gradient Conflict‑Aware Minimization (GCAM) penalizes coordinate‑bounded gradient conflicts and GCAM‑VO targets Transformer attention projections. It accelerates generalization, rescues failed grokking runs and narrows memorization‑generalization gaps with extra overhead from per‑sample gradients.

Critical strengths
  1. This work decomposes the squared batch‑gradient norm into per‑sample magnitude, agreement and conflict terms. It reveals that existing gradient‑norm regularizers fail to suppress cross‑sample gradient interference, because conflict terms receive negative weights in their objectives.

  2. It proposes Gradient Conflict‑Aware Minimization (GCAM), an efficient regularizer that penalizes a coordinate‑wise upper bound of gradient conflicts. Selective regularization on Transformer attention value/output projections (GCAM‑VO) accelerates grokking‑style generalization, rescues failed training trails, and brings gains on CIFAR‑10 vision tasks.

Critical weaknesses
  1. Unsupported mechanistic claim for grokking understanding. Though titled as understanding grokking via cross‑sample gradient conflict, performance gains cannot be causally attributed to conflict suppression. Rescuing failed runs is not exclusive to penalizing surrogate conflict term NcN_c. NcN_c is only a loose upper bound instead of true gradient conflict NN, which remains unmeasured. Superior results of GCAM‑VO cannot disentangle contributions from conflict penalty versus selective VO‑layer regularization, invalidating its mechanistic “understanding” narrative.

  2. Flawed empirical evaluation with limited practical value. Only width‑48 results are robust. Other comparisons suffer from hyperparameter‑selection bias without held‑out validation. The metric t99t_{99} ignores severe post‑peak accuracy collapse. Reported speed‑ups use update steps rather than wall‑clock time, but GCAM bears 3.7× runtime and 5× memory overhead over AdamW. Evaluations are restricted to small‑scale models and some baselines are partial re‑implementations.

Questions for authors
  1. With matched gradient‑correction magnitude or equal compute budget, do penalties for D,Pc,NcD, P_c, N_c and full norm differ in t99t_{99}, sustained end accuracy and trajectories of true dot‑product conflict NN?

  2. When coefficients and learning rates in Table 1 are tuned on held‑out validation seeds and evaluated on fully unseen test splits, do GCAM’s advantages persist for setups besides width‑48?

  3. Given equal wall‑clock budget, does GCAM still surpass cheaper baselines (AdamW, SAM, EGR, MSQ) on t99t_{99}, final accuracy and post‑99% retention?

3 comments
AuthorsOpenReview ↗

Response to Reviewer PCJy

We thank the reviewer for the constructive feedback.

W1. We narrowed the claim: at width 128, observations and interventions support the sharing of the MLP input between examples as one major factor of the delay (§6.1). We now track true dot-product interactions (Figure 4; Appendix Figure C.1), and the gap Nc−N=XN_c - N = X has a stated meaning (Proposition 2; general response). On VO-layer selection: early GCAM (updates 0–1,000) rescues 33–34/34 width-48 failures on V,O, all parameters, the MLP or the embedding, but 19–21/34 on Q,K or the output head (Appendix Table C.3). In a separate update-500 checkpoint test, changing only the V,O learning rate, with no penalty, rescues 14–25/34 and loses 11–12/30 successes (Appendix Table C.1).

W2 / Q2. Coefficients and learning rates were fixed on development seeds before evaluation on new seeds and unseen train/test splits (Appendix Table C.2). On 64 new seeds per width GCAMVO\mathrm{GCAM_{VO}} reaches 99% on 64, 64 and 63 vs AdamW's 52, 49 and 30 (widths 128, 64, 48), and on each of two unseen splits on every evaluated seed (Table 1); on a width × data grid it reaches at least as many seeds as AdamW in 11 of 12 cells (§6.2; Appendix Figure D.2). On post-peak collapse: at lr 2e-3 Grokfast falls below 50% after 99% on 18 and 35 of 64 runs (widths 48, 64), GCAMVO\mathrm{GCAM_{VO}} on none, and applying GCAMVO\mathrm{GCAM_{VO}} only in updates 0–1,000 removes the falls of the always-on penalty at width 128 (21/64 → 0) (§6.2). MSQ is the only adapted baseline (Appendix B).

Scale. As in the grokking studies we build on (Power et al., 2022; Nanda et al., 2023; Liu et al., 2023), algorithmic tasks separate fitting from generalization in time. Within them the effect holds as width grows to 256 and with less data (general response, result 4) and with three layers (GCAMVO\mathrm{GCAM_{VO}} in updates 0–1,000: median 3,000 vs 3,750 updates, 16 seeds; §6.2). On cost: for one-layer models at widths 48 and 128, GCAMVO\mathrm{GCAM_{VO}} in updates 0–1,000 gives an estimated 20k-run cost of 1.16–1.27 times AdamW's (Q3); at width 128 and depths 2–8, block-local GCAMVO\mathrm{GCAM_{VO}}, computed one block at a time, peaks at 163–268 MiB per penalized step against 259–942 MiB for GCAMVO\mathrm{GCAM_{VO}} over all blocks (AdamW 36–86 MiB; Appendix D).

Q1. At matched displacement every component rescues early: median t99 4,350–4,900 updates against AdamW's 12,750 (GCAMVO\mathrm{GCAM_{VO}} 4,500, D 4,350), and 53–62 of 64 runs end at ≥ 99%, the full norm lowest (Appendix C, Table C.1); Appendix Figure C.1 shows the normalized dot conflict, (NV+NO)/(DV+DO)(N_V + N_O)/(D_V + D_O), under each penalty. Kept on through the fit, they separate by sign, and GCAMVO\mathrm{GCAM_{VO}} separates from D at a shared learning rate (general response; §6.3, Figure 6).

Q3. The decomposition holds at every update; in practice we apply GCAMVO\mathrm{GCAM_{VO}} only early, where the MLP coupling that separates the outcomes forms (§6.1). A 20k-update run with GCAMVO\mathrm{GCAM_{VO}} in updates 0–1,000 costs 1.16–1.27 times AdamW's. Width 48 (Appendix Table C.6 and Figure C.5):

AdamW failures reaching 99% at equal estimated wall-clockof 34
AdamW (3–7 across timing calibrations)3–7
GCAMVO\mathrm{GCAM_{VO}}, updates 0–1,00034
MSQ / SAM / EGR kept on, at 25,365 AdamW-update times34 / 17 / 1
D / full norm / SAM in updates 0–1,000, each at its own 20k-run cost34 / 34 / 20

At width 128, GCAMVO\mathrm{GCAM_{VO}}, D, the full norm and SAM in updates 0–1,000 each bring all 12 AdamW failures to 99%; AdamW reaches 2 at the largest of their estimated time budgets (Appendix Figure C.5; Table C.6). At width 48, over runs reaching 99%, the median time to 99% is 9,600 AdamW-update times for GCAMVO\mathrm{GCAM_{VO}} vs 10,400 (MSQ), 15,500 (SAM) and 24,200 (EGR), and post-99% retention (share of later evaluations ≥ 99%) .97 vs .85, .98 and .43; median final accuracy over all 64 seeds is 1.000 vs 1.000, .983 and .924 (AdamW .988). At shared learning rates the cheaper rivals lose ground: at width 64 with 15% of the data (lr 1e-2) late accuracy stays ≥ 99% on 8/8 seeds for GCAMVO\mathrm{GCAM_{VO}}, 5 for SAM, 1 for AdamW and 0 for Grokfast and MSQ; on (4x+y2)3(4x + y^2)^3 mod 97 at lr 3e-3, MSQ, Grokfast and SAM reach 90% on 0, 2 and 3 of 8 seeds, GCAMVO\mathrm{GCAM_{VO}} on 8 (§6.2). On the 30 seeds AdamW solves, GCAMVO\mathrm{GCAM_{VO}} is earlier on 22 (median 9,700 vs 12,750). With per-sample gradients for V,O only, a penalized update took 4.3 times AdamW's time and 146 vs 28.5 MiB at width 128 (the submitted 3.7× was one single-step benchmark; Appendix C).

While the authors' response resolves most of the points I raised, the mechanistic account of cross-sample gradient conflict and the substantive novelty of the work remain underdeveloped. I therefore lean toward keeping or lowering my score.

Thank you; we are glad that our response resolves most of your points. Our first reply's "we narrowed the claim" described the scope of the evidence: at width 128, measurements and interventions implicate the overlap between examples' MLP gradients in the delay. Within that scope the revision develops the account and tests it by lowering and by raising that overlap (§§6.1, 6.3). The updated general response at the top of the forum gives the whole argument in five steps; here we expand on the mechanism and on what is new.

Mechanism. In failing runs the attention output varies little from input to input, so a component shared by all examples makes up 92% of the MLP input, and examples with different answers pull the same MLP weights in opposite directions; the coupling between their MLP gradients predicts failure at update 200, long before the training set is fitted (ROC-AUC .92). Lowering the overlap of examples' MLP gradients (the overlap index in GCAM's penalty, Proposition 3) rescues all 12 failing runs tested; raising it leaves 6 of 12 successful runs below 99% (Appendix Figure C.2).

What GCAM changes. On attention, GCAM raises the input-specific part of the attention output (RMS 1.67 to 4.00); the shared part of the MLP input falls from 92% to 16%, and different answers share fewer active MLP units (70% to 12%) while more units are active, so this is not sparsity (Figure 4; Appendix C).

Why not just smaller gradients. A per-example weight gradient is uixi⊤u_i x_i^\top, so two examples conflict on a weight only when both send input through that weight and pull it in opposite directions. At attention's output projection (error-signal path fixed), GCAM's force on input coordinate kk is −∂Nc/∂xik=−sik∑j≠i∣xjk∣(∣ui∣⊤∣uj∣−siksjkui⊤uj)-\partial N_c / \partial x_{ik} = -s_{ik} \sum_{j \ne i} |x_{jk}| (|u_i|^\top |u_j| - s_{ik} s_{jk} u_i^\top u_j), sik=sign(xik)s_{ik} = \mathrm{sign}(x_{ik}). This force is zero on a coordinate that no other example uses, whereas a penalty on gradient size shrinks every coordinate. At failing checkpoints, the two penalties' gradients are nearly parallel (cosine .95), yet a step on GCAM lowers the overlap between examples' gradients, while a step on the size penalty raises it. Their outcomes differ where data is scarce or the task is harder: at the same learning rate, 15 vs 1 of 16 seeds keep ≥99% accuracy with 15% of the data, and 7 vs 1 of 8 held-out seeds reach 90% on a polynomial task (Figure 6).

Novelty. Prior work measures gradient conflict between examples; this paper adds three things: (i) the squared batch-gradient norm contains conflict with a minus sign, so norm penalties do not penalize it; (ii) a penalty equal to the coordinate conflict summed over all pairs of examples, obtained from two per-weight batch sums without forming the B×BB \times B matrix of pairwise products, with a proved meaning (per pair, the largest first-order interference when each weight's step is rescaled by a factor in [0, 1]; Proposition 2); (iii) its mechanism: because a weight's gradient is error signal times input, penalizing conflict weight by weight separates the routes of conflicting examples, so attention becomes input-specific and different answers share fewer MLP units without the MLP becoming sparser.

If any of these steps still seems underdeveloped, we would be grateful if you could point it out, and we will address it.

Reviewer_Hh4POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies grokking through interactions between per-sample gradients. It decomposes the squared batch-gradient norm into individual gradient magnitudes, positive interactions, and negative interactions. Motivated by the local interference associated with negative inner products, it proposes Gradient Conflict-Aware Minimization (GCAM), which penalizes a coordinate-wise upper bound on this interference. The penalty can be aggregated through per-coordinate sign sums and applied to selected parameter groups. Experiments on modular arithmetic show earlier generalization and a substantially reduced memorization-generalization gap in narrow models. The paper also evaluates a CIFAR-10 extension.

Critical strengths
  1. The paper develops a clear and well-motivated exploratory idea. The decomposition makes explicit that a small batch-gradient norm can arise from cancellation rather than small per-sample gradients. The local loss-change calculation gives a concrete meaning to negative interactions, and the sign-aggregation identity makes the proposed coordinate penalty tractable without constructing every pairwise inner product. These pieces fit together naturally.

  2. The experiments show interesting effects within the studied settings. In particular, the nearly vanishing memorization-generalization gap at width 48 is a useful finding. The paper examines several widths and data fractions, includes a modular-multiplication extension, and provides component and placement comparisons. I also appreciate the explicit discussion of computational cost, instability after first reaching the accuracy threshold, and the limits of the causal interpretation. Even if the method's practical advantage remains uncertain, I find this a valuable focused study.

Critical weaknesses

1. The evidence does not yet isolate the particular role of conflict in the improvement.

Table 3 shows that penalties on individual gradient magnitudes, positive coordinate interactions, and the full gradient norm also rescue all eight baseline-failing runs. The authors acknowledge this, but it remains unclear how much of GCAM's benefit comes specifically from changing conflict, as opposed to changes in gradient scale or the training trajectory more generally. Since GCAM penalizes the coordinate upper bound N_c, understanding how the actual dot-product conflict N changes during training would also help assess this interpretation.

2. The connection between the interference interpretation and the actual optimizer needs clarification.

Equation (2) concerns an ordinary per-sample negative-gradient step, whereas the main experiments use AdamW. With a common frozen preconditioner M, the corresponding first-order interaction involves giTMgjg_i^T M g_j rather than giTgjg_i^T g_j. Momentum and the evolving preconditioner introduce further differences. Although the appendix and Discussion acknowledge this boundary, the relationship between raw-gradient conflict and interference under actual adaptive updates remains unclear.

3. The CIFAR-10 performance comparison has unequal tuning budgets.

Table 2 reports a search over multiple regularization coefficients for GCAM, while EGR, SAM, and Grokfast each use one coefficient. Considering two learning rates does not remove this asymmetry. The result establishes an advantage among the evaluated configurations, but the advantage over comparably tuned alternatives remains uncertain.

Questions for authors
  1. What evidence connects the gains specifically to changes in conflict? Could the authors compare N, N_c, and gradient magnitudes at comparable training states? Short interventions starting from the same checkpoint, with comparable correction strengths, could help distinguish the effects of conflict reduction from other forms of gradient regularization.

  2. How does the interpretation change with the optimizer? How does raw-gradient conflict relate to interference under AdamW's preconditioning and momentum, and what changes for an optimizer such as Muon? A small optimizer comparison or a diagnostic of update-induced loss changes would help clarify the scope of the argument.

  3. Does the CIFAR-10 advantage persist under comparable hyperparameter search? I wonder what happens if the baseline experiments have a similar coefficient search. This will make the comparison more fair. Please also clarify how learning rates and coefficients were selected, including whether validation data were separate from the reported test data.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Hh4P

We thank the reviewer for the constructive feedback; we ran the same-checkpoint test you proposed.

W1 / Q1. From each seed's update-500 AdamW state, seven V,O penalties matched on parameter displacement rescue alike in a 1,000-update window (33–34/34 failures at width 48, 12/12 at width 128), whereas changing only the V,O learning rate, with no penalty, rescues 14–25/34 and loses 11–12/30 successes (Appendix Table C.1). At update 1,500, after branching from the same update-500 states (width 48), every penalty lowers V,O D to 1–3% of AdamW's, but only the coordinate penalties (GCAMVO\mathrm{GCAM_{VO}}, PcP_c, X) lower Nc/DN_c/D (.82 times AdamW's) and only the dot-level ones lower the dot conflict N/DN/D (.79 times); D and the full norm leave both at AdamW's level (Appendix Figure C.1). Kept on through the fit, the sign separates the penalties (general response). In saved states on the polynomial task (four seeds), the routes differ: under D the normalized coordinate overlap at V,O stays at AdamW's level (.40), whereas GCAMVO\mathrm{GCAM_{VO}} lowers it to .08 while its per-example gradients grow (§6.3, Appendix Figure D.1). Since NcN_c contains X (10–57 times N at V,O), which can obscure the sign balance, Figure 4 tracks dot-level coupling and conflict share.

W2 / Q2. With Adam's frozen preconditioner M normalized to mean diagonal one (width 128), ∑i≠jmax⁡(0,−gi⊤Mgj)\sum_{i\neq j}\max(0,-g_i^\top M g_j) at V,O is .52–.88 times the raw N: preconditioning changes conflict without removing it. In the actual next AdamW update (width 48), about one third of the examples have a positive first-order loss change, and among them the conflicting cross-example part averages 3.5 times the momentum part (Appendix C). Muon and SOAP reach 99% on all 64 seeds at widths 48 and 128; adding GCAMVO\mathrm{GCAM_{VO}} reduces the median updates to 99% to .84–.86 times those of the optimizer alone (Appendix Table C.7), and with 15% of the data at width 64 it shortens Muon's time to 99% on 16 of 16 seeds (§6.4).

W3 / Q3. You are right. Every method was re-tuned over 7–17 configurations on 5,000 held-out training images and tested on the full test set (5 seeds, accuracy %; Appendix Table E.1). GCAM improves on AdamW in every setting: with flip-and-crop, GCAM on the classifier head reaches 78.32 (all parameters 77.78, AdamW 76.75), level with SAM (78.37) and below MSQ and EGR (78.82, 78.68); without augmentation GCAMall\mathrm{GCAM_{all}} reaches 67.44 (AdamW 65.09), below SAM and MSQ (72.22, 68.49). In a separate unaugmented protocol whose batches hold about four classes each, the conflict penalty improves on the agreement penalty at a common coefficient on 4 of 5 seeds (64.37 vs 63.26), with similar accuracy under shuffled batches (Appendix E).

Thanks for the response and additional experiments. My concerns have been partially addressed, but after reading other reviewers' comments, I agree that the paper is still difficult to follow (some of these issues are overlooked during my initial review). I'll keep my score.

Thank you for the follow-up and for noting that your concerns were partly addressed. Your main question (W1) was how much of GCAM's benefit comes from conflict rather than from smaller gradients. Our first reply showed that, applied briefly from the same checkpoints, every penalty rescues; the second revision adds where they part and why (§6.3):

  • Outcomes part where data is scarce or the task is harder. Kept on at the same learning rate, with 15% of the data 15 of 16 seeds keep ≥99% accuracy under GCAM and 1 under a gradient-size penalty; on a polynomial task 7 and 1 of 8 held-out seeds reach 90% (Figure 6).
  • They act differently even where outcomes agree. At failing checkpoints the two penalties' gradients are nearly parallel (cosine .95), yet a step on GCAM lowers the overlap between examples' gradients and a step on the size penalty raises it.
  • The sign of the interaction matters. Both coordinate penalties contain the opposition that a dot product cancels within each pair (§5), so the sign is tested at the dot level: kept on, penalizing conflict lets 12 of 16 seeds reach 99%, agreement 4, the full gradient norm none (Table 2).

W2 and W3 stand as in our first reply (§6.4; Appendix E). On clarity, §§6.1 and 6.3 are rewritten as one argument, which the updated general response at the top of the forum sets out in five steps.

If a part still reads unclearly, we would be grateful if you could point it out, and we will revise it.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

Updated for the second revision. Section, table and figure numbers follow the revised PDF; the earlier version of this response remains under Revisions.

We thank the reviewers for their feedback. As submitted, the paper relates the grokking delay to how the gradients of individual examples interact and introduces GCAM to act on it. The revision makes this argument explicit and adds where the interaction acts in the network, through attention and the MLP during training, and how GCAM changes it.

1. The gradient norm hides conflict between examples (§§3–4).

B2∥gˉ∥2=D+P−NB^2\lVert\bar g\rVert^2 = D + P - N: the examples' own gradient sizes (DD), agreement between pairs (PP), and conflict between pairs (NN), where a step that helps one example hurts another. Conflict enters with a minus sign, so the batch gradient does not show it and a penalty on the norm does not penalize it.

2. In failing runs, attention feeds the MLP nearly the same input for every example (§6.1).

The attention output varies little from input to input, so a component shared by all examples makes up 92% of the MLP input, and examples with different answers pull the same MLP weights in opposite directions. The coupling between examples' MLP gradients already predicts failure at update 200, long before the training set is fitted (ROC-AUC .92; Appendix C).

3. GCAM separates the routes of conflicting examples through attention and the MLP (§§5, 6.1).

GCAM penalizes opposing pulls weight by weight: Nc∝∑kakbkN_c \propto \sum_k a_k b_k, with ak=∑imax⁡(0,gik)a_k = \sum_i \max(0, g_{ik}) and bk=∑imax⁡(0,−gik)b_k = \sum_i \max(0, -g_{ik}) summed over the batch. ∑kakbk\sum_k a_k b_k equals the coordinate conflict summed over all pairs of examples, yet it is obtained from two sums per weight, without forming the B×BB \times B matrix of pairwise products. Since an example's gradient on a linear layer is uixi⊤u_i x_i^\top (error signal times input), two examples conflict on a weight only when both send input through that weight and pull it in opposite directions; lowering NcN_c therefore discourages examples whose pulls oppose from sharing a route. On the attention weights, GCAM makes attention more input-specific: the shared part of the MLP input falls from 92% to 16%, examples with different answers share fewer MLP units, and the run generalizes (Figure 4).

4. Results (§6.2).

AdamWGCAM
narrow models: seeds that generalize at widths 128 / 64 / 48 (of 64)52 / 49 / 3064 / 64 / 63
width 128, 15% of the training data: seeds that generalize (of 16)016
width 64, 15% of the data, high learning rate: seeds with ≥99% late accuracy (of 8)18
failing runs rescued at equal estimated wall-clock cost, GCAM in the first 1,000 updates only (of 34)4–734

5. Is it the interaction between examples, or just smaller gradients? (§6.3)

Applied briefly, GCAM and a penalty on gradient size (DD) both rescue failing runs, but they part when data is scarce or the task is harder: with 15% of the data at the same learning rate GCAM keeps ≥99% accuracy on 15 of 16 seeds and the size penalty on 1, and on a harder task 7 versus 1 of 8 held-out seeds reach 90% (Figure 6). At failing checkpoints, the two penalties' gradients are nearly parallel (cosine .95), yet a step on GCAM lowers the overlap between examples' gradients and a step on the size penalty raises it. The sign is tested at the dot level, because both coordinate penalties contain the opposition that a dot product cancels within each pair (§5): kept on, penalizing dot-product conflict lets 12 of 16 seeds reach 99%, agreement 4, the full gradient norm none (Table 2).


Changes in the revised PDF

  • Mechanism and novelty (PCJy): §6.1 rewritten as a mechanistic account; §6.3 compares GCAM with the size penalty.
  • Is the gain specific to conflict? (Hh4P): new Table 2.
  • Clarity (3xG9, Hh4P): §§6.1 and 6.3 rewritten as one argument, in the order above; abstract, Section 2, contributions and Conclusion rewritten.
  • Scale (3xG9): CIFAR-100 added; GCAM matches SAM and improves on AdamW (51.87% and 51.80% against 49.07%; Appendix Table E.2).
  • Also: Propositions 2–4 with proofs; new Figures 4–6; held-out seeds and unseen splits; equal-cost comparisons; CIFAR-10 re-tuned on a validation split. The submitted appendix Table 3 is now Appendix Table C.8.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.