- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how the order of precursor tasks affects grokking in small transformers and which parameters carry the resulting benefit into target-task training. The main setting is symbolic solution of first-order linear ODEs, preceded by negation, addition, and coefficient-extraction tasks. Across 16 model seeds, the ordered curriculum increases mean held-out accuracy from approximately 0.48 to 0.88 and substantially reduces variation across seeds, whereas reversing the same precursor tasks largely removes the benefit. The principal intervention partitions the hand-off parameters into token identity parameters—the input embedding and output readout—and the remaining network body. In the ODE family, transplanting only the identity into an otherwise freshly initialized model recovers approximately 97% of the curriculum improvement, while transplanting the body performs substantially worse than cold start. Disjoint-alphabet and donor-transfer experiments further investigate the role and portability of this initialization. In a second family involving evaluation of symbolic derivatives, the benefit is instead carried primarily by the network body. The paper also reports controls for training budget, weight norms, answer-key overlap, and causal masking, among other factors. The authors interpret their findings as evidence that curricula control grokking through the initialization left by the last precursor, rather than through transferred computation, and propose a basin-selection interpretation.
Critical strengths
-
The transplantation experiments provide concrete evidence about the location of a transferable curriculum benefit. The identity-only and body-only interventions are substantially more informative than a simple curriculum-versus-cold-start comparison. In the ODE family, identity transfer nearly reproduces the full curriculum’s performance while body transfer does not. The disjoint-alphabet experiment, the frozen-body adaptation control, and transfer across recipient seeds strengthen this finding by addressing several straightforward alternative explanations. This is the paper’s most consequential contribution.
-
The paper studies reliability across seeds and includes a useful counterexample to an identity-specific explanation. The reduction in outcome variability—from approximately to in the main family—is practically meaningful within this setting. The second family, where body transfer is effective and identity transfer is not, is also valuable: it shows that the observed benefit cannot be reduced to a universal embedding-only mechanism. Reporting this reversal makes the empirical contribution more informative.
Critical weaknesses
-
The central contrast between transferring an initialization and transferring computation is not established by the interventions. A learned representation or computation is itself encoded in parameters and can influence subsequent learning when those parameters are transplanted. Consequently, showing that a parameter subset is sufficient to reproduce a curriculum benefit does not, by itself, distinguish a favorable optimization starting point from transferable representational or computational structure within that subset. This issue is particularly apparent in the second family, where the trained network body carries the improvement.
I recognize that the frozen-body experiment and its positive control provide meaningful additional evidence in the ODE family. They support the narrower conclusion that the curriculum body does not provide a usable generalizing solution under the tested adaptation protocol. They do not establish the broader claim that curricula transfer initialization “rather than computation.” Similarly, the double-well model in Appendix D.1 illustrates how basin selection could produce the observations, but does not independently establish that the actual transformer dynamics have the proposed structure. Since the manuscript presents this interpretation as a central explanation, the gap is consequential to the assessment.
-
The contribution beyond effective precursor-task pretraining remains insufficiently delineated. Reversing the same tasks provides convincing evidence that order matters in the main setting. However, several results also show that a full progressive curriculum is not required: a single Add or Coefficients precursor reaches approximately 0.83, removing Coefficients from the curriculum can retain approximately 0.88, and in the second family the compatible precursor alone reaches approximately 0.90, exceeding the longer ladder’s approximately 0.77. These findings are useful, but they leave unclear how much of the contribution concerns curriculum order specifically, as opposed to selecting a favorable pretraining task and transferring its parameters.
Moreover, the carrier reversal occurs across families that differ in task, depth, vocabulary, and weight decay. The paper acknowledges that it measures the carrier rather than predicts it. I also recognize the depth control in Appendix C.5; because that condition does not exhibit the relevant cold-start lottery, it does not resolve why the carrier changes. Thus, the manuscript provides a compelling set of conditional observations, but a more limited general explanation than its framing suggests.
Questions for authors
-
Clarification of the central hypothesis. What operational distinction separates “initialization transfer” from “computational or representational transfer encoded in the transferred weights”? Which observation would distinguish these explanations in the present experiments? In particular, how should the body-transfer success in the second family be interpreted under the claim that curricula transfer initialization rather than computation? A precise, narrower statement of the conclusion would materially affect my assessment.
-
Quantification of the value of the multi-stage curriculum. Using the existing paired runs where possible, what is the effect size and uncertainty of the full ODE curriculum relative to the strongest single-precursor condition under a matched total training budget? The manuscript already controls for the possibility that extra training alone explains the cold-start gap; my question is whether the multi-stage ordering provides a reliable benefit beyond an appropriately chosen precursor. How does that comparison inform the interpretation of the second family, where the longer ladder performs worse?
-
Interpretation of the carrier reversal. Is there evidence that identifies which change between the two families causes the identity/body reversal—for example, task structure, model depth, or regularization? If the current experiments cannot distinguish these factors, would the authors restrict the claim to localization of a transferable benefit in each tested configuration, rather than a general mechanism explaining curriculum-induced grokking? I am asking about the interpretation supported by the current evidence, rather than requesting a large-model extension.
Thanks for the review.
W1 and Q1. Narrowing the claim is fair, but rather than computation or initialization, the key is that we transfer only one parameter group, the token identity in the ODE family and the body in the second. Also, the transformer dynamics itself is the evidence for that structure. Which change flips the carrier is indeed future work.