acceptodds
Withdrawn from ICLR 2027

Curriculum Order Controls Grokking Through Its Hand-off Weights

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A neural network can generalize long after memorizing its training data; this late generalization is called grokking. A curriculum often helps by training the model on a chain of simpler tasks first, each starting from the weights the last one left. It is not known whether this chain passes on skills, or only a better initialization that another model could simply copy. We study a small transformer that solves a linear differential equation symbolically after a chain of negation, addition and reading off the solution's two coefficients. On the target alone, runs differing only in their random starting weights sometimes generalize and sometimes fail. Ordered easiest to hardest, the chain lifts average test accuracy from 0.48 to 0.88. And nearly every run now generalizes. But run in reverse order, ending on the easiest, the same tasks do no better than the target alone. To see which it is, we copy part of the chain's final weights into a fresh model trained only on the target. We split those weights in two: the token identity, which maps tokens in and out, and everything else. On its own, the token identity recovers 97% of the gain, although the fresh model has never seen the simpler tasks. And copying only the rest does worse than no curriculum at all. We repeat this on a second problem, symbolic differentiation, with its own chain, model and vocabulary. There the rest of the network carries the gain instead. So we find that a curriculum passes on an initialization, set by the last task before the target. The benefit of a curriculum is thus something one model can hand to another rather than earn by training.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, grokking, curriculum learning, training order, model stitching, token embedding, symbolic mathematics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xGVVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how the order of precursor tasks affects grokking in small transformers and which parameters carry the resulting benefit into target-task training. The main setting is symbolic solution of first-order linear ODEs, preceded by negation, addition, and coefficient-extraction tasks. Across 16 model seeds, the ordered curriculum increases mean held-out accuracy from approximately 0.48 to 0.88 and substantially reduces variation across seeds, whereas reversing the same precursor tasks largely removes the benefit. The principal intervention partitions the hand-off parameters into token identity parameters—the input embedding and output readout—and the remaining network body. In the ODE family, transplanting only the identity into an otherwise freshly initialized model recovers approximately 97% of the curriculum improvement, while transplanting the body performs substantially worse than cold start. Disjoint-alphabet and donor-transfer experiments further investigate the role and portability of this initialization. In a second family involving evaluation of symbolic derivatives, the benefit is instead carried primarily by the network body. The paper also reports controls for training budget, weight norms, answer-key overlap, and causal masking, among other factors. The authors interpret their findings as evidence that curricula control grokking through the initialization left by the last precursor, rather than through transferred computation, and propose a basin-selection interpretation.

Critical strengths
  1. The transplantation experiments provide concrete evidence about the location of a transferable curriculum benefit. The identity-only and body-only interventions are substantially more informative than a simple curriculum-versus-cold-start comparison. In the ODE family, identity transfer nearly reproduces the full curriculum’s performance while body transfer does not. The disjoint-alphabet experiment, the frozen-body adaptation control, and transfer across recipient seeds strengthen this finding by addressing several straightforward alternative explanations. This is the paper’s most consequential contribution.

  2. The paper studies reliability across seeds and includes a useful counterexample to an identity-specific explanation. The reduction in outcome variability—from approximately 0.48±0.310.48\pm0.31 to 0.88±0.050.88\pm0.05 in the main family—is practically meaningful within this setting. The second family, where body transfer is effective and identity transfer is not, is also valuable: it shows that the observed benefit cannot be reduced to a universal embedding-only mechanism. Reporting this reversal makes the empirical contribution more informative.

Critical weaknesses
  1. The central contrast between transferring an initialization and transferring computation is not established by the interventions. A learned representation or computation is itself encoded in parameters and can influence subsequent learning when those parameters are transplanted. Consequently, showing that a parameter subset is sufficient to reproduce a curriculum benefit does not, by itself, distinguish a favorable optimization starting point from transferable representational or computational structure within that subset. This issue is particularly apparent in the second family, where the trained network body carries the improvement.

    I recognize that the frozen-body experiment and its positive control provide meaningful additional evidence in the ODE family. They support the narrower conclusion that the curriculum body does not provide a usable generalizing solution under the tested adaptation protocol. They do not establish the broader claim that curricula transfer initialization “rather than computation.” Similarly, the double-well model in Appendix D.1 illustrates how basin selection could produce the observations, but does not independently establish that the actual transformer dynamics have the proposed structure. Since the manuscript presents this interpretation as a central explanation, the gap is consequential to the assessment.

  2. The contribution beyond effective precursor-task pretraining remains insufficiently delineated. Reversing the same tasks provides convincing evidence that order matters in the main setting. However, several results also show that a full progressive curriculum is not required: a single Add or Coefficients precursor reaches approximately 0.83, removing Coefficients from the curriculum can retain approximately 0.88, and in the second family the compatible precursor alone reaches approximately 0.90, exceeding the longer ladder’s approximately 0.77. These findings are useful, but they leave unclear how much of the contribution concerns curriculum order specifically, as opposed to selecting a favorable pretraining task and transferring its parameters.

    Moreover, the carrier reversal occurs across families that differ in task, depth, vocabulary, and weight decay. The paper acknowledges that it measures the carrier rather than predicts it. I also recognize the depth control in Appendix C.5; because that condition does not exhibit the relevant cold-start lottery, it does not resolve why the carrier changes. Thus, the manuscript provides a compelling set of conditional observations, but a more limited general explanation than its framing suggests.

Questions for authors
  1. Clarification of the central hypothesis. What operational distinction separates “initialization transfer” from “computational or representational transfer encoded in the transferred weights”? Which observation would distinguish these explanations in the present experiments? In particular, how should the body-transfer success in the second family be interpreted under the claim that curricula transfer initialization rather than computation? A precise, narrower statement of the conclusion would materially affect my assessment.

  2. Quantification of the value of the multi-stage curriculum. Using the existing paired runs where possible, what is the effect size and uncertainty of the full ODE curriculum relative to the strongest single-precursor condition under a matched total training budget? The manuscript already controls for the possibility that extra training alone explains the cold-start gap; my question is whether the multi-stage ordering provides a reliable benefit beyond an appropriately chosen precursor. How does that comparison inform the interpretation of the second family, where the longer ladder performs worse?

  3. Interpretation of the carrier reversal. Is there evidence that identifies which change between the two families causes the identity/body reversal—for example, task structure, model depth, or regularization? If the current experiments cannot distinguish these factors, would the authors restrict the claim to localization of a transferable benefit in each tested configuration, rather than a general mechanism explaining curriculum-induced grokking? I am asking about the interpretation supported by the current evidence, rather than requesting a large-model extension.

1 comment

Thanks for the review.

W1 and Q1. Narrowing the claim is fair, but rather than computation or initialization, the key is that we transfer only one parameter group, the token identity in the ODE family and the body in the second. Also, the transformer dynamics itself is the evidence for that structure. Which change flips the carrier is indeed future work.

Reviewer_9xEHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how curriculum order affects grokking in small transformers on two restricted symbolic-math tasks. Transplant experiments show that much of the curriculum benefit can be recovered by copying selected pretrained parameters. The embedding and readout carry the benefit in the ODE task, while the transformer body carries it in the second task.

Critical strengths

The component transplants directly test which parameters can transfer the benefit through curriculum training.

Critical weaknesses

Clarity. Very poorly written paper. It is difficult to read with non-technical terms used without proper definition (or scattered definitions). The task is not well defined at all. Does the model have to predict the full sequence or just the tokens after the separator ? What tokens are used for these separators? What happens to the cases for which a+b = 0 (and for which the solution as stated is undefined). The tone and style is also informal, and not suited for a scientific paper.

Central question unanswered. The main question that the work poses in the abstract itself is to distinguish between initialization and computation. However, learnt embeddings can encode reusable computations directly (see for instance [1]). Transferring them confounds these two questions. More importantly, it does not address which properties of the transferred parameters lead to improvement in performance.

Possible overclaim. The paper proposes to understand “symbolic differential equation solution” but it considers an extremely narrow case. The solution can be read off by just noting the coefficient of yy and ebxe^{bx}. The target already creates that template for the model. Claims of symbolic ODE solution should involve general or multiple different classes of differential equations, where the solution does not belong to the same family. Even their differentiation experiment is similarly limited, where they test a fixed arithmetic expression, rather than differentiation across different expression structures. It is not immediately clear if the zero validation accuracy reported in Section 4.2 is because of unavailability of a generalizing solution. Several confounding factors like tokenization, split, etc. are not carefully reported.

Internal inconsistencies.

Section 5; Table 5 : Cold start does not consistently clear the threshold above decay 0.3: at 0.5, mean accuracy is 0.65. Section 4.2 : Both task order and initialization vary. Their effects are not fully separated. Appendix A.3 : Only seven of thirty “confirmatory” tests were registered before the runs. The timeline needs clarification. Sections 4.3–4.4; Appendix C.6 : A nonsignificant difference does not establish equivalent performance. Section 3 : Equal training steps do not ensure equal compute or target exposure.

[1] He, T., Doshi, D., Das, A., & Gromov, A. (2024, June). Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks. In ICML 2024 Workshop on Mechanistic Interpretability.

Questions for authors

I do not have any questions that could materially affect my assessment. I urge the authors to fix their presentation, clarity and novelty issues.

1 comment

Thanks for the review.

Clarity. Some points are fixed in the revision. But the task definition is in the appendix in detail and referenced from the main text, and the separator token and predicting only after it are both now stated.

Central question. It is right that we did not answer the core question in general. But the direction the conclusion points to is which parameter group carries the gain.

Section 4.2. The initialization is the same for each seed. What differs between the schedules is the order, and the state it leaves for the target.

Reviewer_JcdwOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how the curriculum of the data impacts grokking. In particular, the paper focuses on symbolic controlled tasks that sometimes generalize perfectly and sometimes don't generalize based on the seed of the training. Then the authors investigate how imposing a curriculum of tasks during training changes this pattern, i.e., making generalization more or less likely across the seeds, showing that a proper curriculum can make generalization much more likely. There are interesting experiments where only some part of the weights are transferred between curriculum stages, e.g., only the embedding weights or only the body weights of the model, and it is shown that generalization is heavily affected by which weights are transferred in this process.

Critical strengths
  • The paper proposes simple symbolic tasks which are nice as the their generation can be totally controlled. The experiments and methodology of the paper are interesting, especially the implant experiments where only a particular set of the weights is transferred between models.
  • Relatively extensive experimentation (trying different curricula and seeds) as well as having statistical tests makes the results more robust.
Critical weaknesses
  • I do not see what is the general "message" the paper tries to convey. There are several interesting experiments in the paper, but I'm not sure what is the expected takeaway from the paper.
  • I am not generally convinced by the question of the paper asking about initialization transfer and skill transfer; I'm not sure they are even different let alone distinguishable. I think grokking happens due to implicit biases of the model, where the models learn a "nice" (e.g., symmetrical) solution on the training set. Now this "niceness" for some tasks may be embedded in the body weights and for others in embedding weights (e.g., encoding particular symmetries). I don't see this as two different phenomena, but instead two instantiations of the same thing.
Questions for authors
  • Q1. Regarding to the weakness mentioned above, can you define what would you call a skill transfer and how is that distinguished from the transfer that happens with the initialization?
  • Q2. Does the seed lottery setting depend on the optimizer setting or is it relatively robust to the choice of hyper-params?
  • Q3. Do we have any evidence that such grokking observations also happen in larger settings and real-world tasks? (e.g., language model training.)
  • Q4. How would you summarize the main message of your work?
1 comment

Thanks for the review.

W1 and Q4. The message is that stage order sets the outcome through only one parameter group of the weights, which differs by family, and that it can be copied into another model.

W2 and Q1. In the ODE family it is a starting point, and the core is which parameter group of the weights is passed on.

Q3. Scaling to LLMs is future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.