acceptodds
Under review as a conference paper at ICLR 2027

Not Order but Support: Curriculum Effects on Grokking Are a Transient-Support Phenomenon

Abstract

Grokking is when a network generalizes abruptly, long after it has already fit (memorized) its training set. It is sensitive to how the training data is presented—but “presentation” quietly bundles two different things, because a curriculum that shows hard examples first also shows them more often. We separate the two with a pair of controlled experiments. In the first, we fix the set of examples so that every example is seen equally often over the whole run, and vary only the order. Sorting by difficulty, in either direction, then prevents grokking outright on four of six small algorithmic tasks (validation accuracy stays at chance for the whole run even though the training set is fully memorized) and delays it 6–54× on the other two. So how often each example is seen is not what matters. In the second experiment, on two of the tasks, we go further and make every epoch (one full pass through the data) contain the whole difficulty range, so exposure is matched not just overall but at every epoch boundary, and again vary only the within-epoch order. The orders-of-magnitude delay then disappears: sorted and shuffled schedules reach the threshold within one evaluation interval. Thus, fine-grained order is not the primary lever either. The leading explanation is how long the optimizer is kept on a narrow, unrepresentative slice of the difficulty range. Curriculum effects on grokking are therefore a transient-support phenomenon; calling them ordering effects is imprecise. Two smaller results accompany this finding. First, a two-feature linear predictor built from early weight-matrix singular-value statistics forecasts the transition time with fully nested cross-validated between 0.66 and 0.81, requires no validation signal from the target run, and generalizes to held-out curricula. Second, at transformer scale, a sharp grokking-like transition appears on 2WikiMultiHopQA and reproduces across an 8× range of model sizes, but is absent or gradual on three other datasets. Because those experiments use an uncontrolled growing-pool curriculum, we present them as a boundary map rather than causal evidence about ordering. We release all 505 training runs.

Primary area:
optimization
Keywords:
grokking, curriculum learning, data ordering, transient support, training dynamics, generalization, spectral diagnostics, transformers

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qFWMOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies curriculum effects on grokking by separating example order, exposure, and support. Across six algorithmic tasks at two model scales, global difficulty sorting delays or prevents grokking within the training budget despite matched exposure, while sorting within full-data epochs largely removes the effect on the two tasks tested. The results suggest sustained restriction to an unrepresentative data subset as the main explanation.

Critical strengths
  • The research question and gap are clear, and the paper's structure is logical.
Critical weaknesses
  • The sampling schemes are specified, but support and confinement duration need a clearer formal definition, including the time window over which support is measured. The growing-pool schedules use linear pacing, leaving it unclear whether the conclusions hold under non-linear pacing or discrete schedules that expand the pool in stages [1].
  • The experiments use six tasks at two scales, and the prefix-matched control covers only two of them, which limits the evidence that fine-grained order has little effect on grokking. Each task uses a single difficulty metric, so sensitivity to alternative metrics on the same task remains unclear. Appendix J.4 tests different difficulty signals for QA, but those experiments are not exposure-controlled.

[1]. Wang, X., Chen, Y. and Zhu, W., 2022. A survey on curriculum learning, IEEE transactions on pattern analysis and machine intelligence.

Questions for authors
  • How are support and confinement duration defined over time, and is there evidence that the conclusions hold under non-linear pacing or discrete schedules that expand the pool in stages?
  • For the same task, do the results in Tab 2 and 3 hold when a different difficulty metric changes the example order?
6 comments

Thank you. All four requests are now in the paper.

Definitions of support and confinement duration over time, including the window (§2, Box 1).

  • Windowed support with timescale τ is the distribution of recently seen examples: pt(τ)(i)∝∑s≤t(1−1/τ)t−sci(s)p^{(\tau)}_t(i)\propto\sum_{s\le t}(1-1/\tau)^{t-s}c_i(s). A boxcar window W uses ni(t)−ni(t−W)n_i(t)-n_i(t-W) instead.
  • From it we compute:
    • support bias Γ, the total-variation distance between the recent difficulty mix and the full set;
    • relative effective support φ;
    • coverage;
    • confinement duration Tθ=∣t:Γ(t)>θ∣T_\theta=|\\{t:\Gamma(t)>\theta\\}|.
  • Windows. We report every statistic at three windows: Adam's second-moment timescale (50 steps), one epoch, and the weight-decay timescale 1/(ηλ).
  • Box 2 lists what every experiment fixes and what it varies.

Non-linear and staged pacing (§5.8, Appendix I). We ran square-root, quadratic, exponential and staged pacing on mod-97 addition, GCD and S5, three seeds each.

  • Predictions made in advance. Before these runs we predicted the mod-97 transition times from the linear-pacing runs, at threshold 0.7. The assumption was that the pool fraction at the transition does not depend on the pacing law.

    • Quadratic and exponential pacing match within 2% in both directions.
    • Square-root pacing crosses earlier than predicted, by 9% easy-first and 28% hard-first. Part of this is bookkeeping: the prediction used the competence function of Platanios et al. rather than the sampler's square-root law.
  • Staged schedules cross shortly after the first stage whose pool reaches that fraction, in 6 of 8 schedules.

  • Pool fraction at 0.7. It varies little across laws on mod-97 addition and GCD. Appendix I also gives it at 0.9.

  • Where the account fails.

    • On S5, six of eight hard-first schedules never reach 0.7, even with the pool full from mid-run.
    • On mod-97 addition, admitting examples in random order at the same pacing is as slow as admitting by difficulty.
    • Fresh random subsets cross from 40% of the mod-97 training set, whereas growing pools cross only at 76–90%.

    So a growing pool's history adds to its size. We report this as a limitation of the simple support account (Limitation 8).

The matched-prefix control beyond two tasks (§5.2, Table 2). All six tasks, 5 seeds, evaluation every 25 steps.

  • mod-97 addition and multiplication and S5: epoch-sorted runs stay within 0.87–1.3× of shuffled.
  • sort-8: unchanged.
  • GCD and 4-digit addition: residuals of 1.3× and up to 1.9×. sort-8, with the same 156-step epochs, shows none.
  • For contrast, the global sort blocks the transition on four tasks.

Alternative difficulty metrics on the same task, re-running Tables 1 and 2 (§5.8). We repeat the global and the epoch sort with one to three alternative structured keys per task, plus a key that permutes the difficulty values across examples. For S5 these include input-side keys.

  • Global sort. It delays the transition at least 2× in 24 of 26 structured-key cells.
  • Epoch sort. It stays within 1.5× of shuffled in 18 of 26 structured-key cells and in all 12 permuted-key cells.
  • The largest exception is GCD sorted by its label. One class, about 60% of the data, then fills a contiguous stretch of every epoch, and even the epoch sort fails. We therefore state the practical rule in terms of contiguous stretches confined to a narrow slice, not epochs (§6).

Do these address your concerns? We would be glad to add further variants.

Thank you again for your review. As the discussion period closes on Nov 18, we wanted to check whether the four additions you asked for address your concerns: the windowed definitions of support and confinement duration, the non-linear and staged pacing runs, the matched-prefix control on all six tasks, and the alternative difficulty metrics. We would be glad to answer any further questions. If the revision addresses your concerns, we would be grateful if you would consider updating your score.

Thank you for your response and additional work, which address several of my concerns through clearer support definitions and expanded pacing, control, and difficulty-metric experiments. However, the revision does not yet isolate support from training history and forgetting, establish a grokking-specific explanation, and resolve the remaining statistical and theoretical issues. So, I keep my score.

Reviewer_xJxWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper investigates why curriculum learning delays grokking by disentangling example order, exposure, and data support. Experiments on six algorithmic tasks suggest that equalizing total exposure does not eliminate the delay, whereas ensuring that each epoch covers the full dataset largely removes it. The authors attribute this effect to sustained confinement to a narrow data support.

Critical strengths
  1. The paper addresses an interesting and important question about the mechanisms underlying curriculum-induced grokking delays.
  2. The experimental design attempts to disentangle example order, exposure, and data support, providing useful insights beyond conventional curriculum comparisons.
  3. The authors evaluate multiple algorithmic tasks and difficulty definitions, and release training logs to support reproducibility.
Critical weaknesses

1. Insufficient formalization of the key variables. The central contribution is to disentangle exposure, order, and support, yet these concepts lack precise mathematical definitions. The authors should formally define each variable as a function of training step or training prefix, and explicitly characterize, for every experiment, which variables are held fixed and which are varied according to these definitions. Without this, it is difficult to verify whether the experiments genuinely disentangle the intended factors or support the paper’s central conclusion. If this concern can't be solved, it would be hard to convince the reader.

2. Clarification of Figure 1. In the “Epoch sort, matched prefix” panel, the point patterns appear to differ across epochs. However, if every epoch contains the same dataset and examples have fixed difficulty values, the difficulty distribution should be identical across epochs. Is Figure 1 purely schematic, or does it reflect the actual implementation? If the latter, how are these differences produced?

3. Insufficient causal evidence for support confinement. The experiments suggest that longer confinement correlates with greater grokking delays, but do not fully isolate support confinement from other temporal effects, such as gradient autocorrelation. A more direct control that varies confinement duration while holding exposure fixed would strengthen the mechanistic claim. Otherwise, the authors should state this claim more cautiously.

4. Limited coverage of the matched-prefix control. This control is evaluated on only two of the six tasks, with evaluations every 250 steps. This limits the strength of the conclusion that within-epoch order does not matter. Could the authors extend this control to additional tasks or evaluate more frequently around the grokking transition?

Questions for authors
  1. Could the author define the basic term “support confinement” precisely and explain how it is quantified?
  2. Reporting epoch lengths alongside grokking times would help clarify the temporal scale of the effect.
  3. How could the author distinguish cases that fail to grok within the training budget from cases where grokking is genuinely absent?
6 comments

Part 1/2: Definitions, Figure 1, and the direct control

Thank you for the careful reading. Each point is now addressed in the PDF.

W1. Formal definitions (§2, Box 1). A scheme is a slot stream σ over T steps of batch size B.

  • Exposure. ni(t)=∑s≤tci(s)n_i(t)=\sum_{s\le t} c_i(s), where ci(s)c_i(s) counts the copies of example i in batch s. Two schemes are exposure-matched on a grid G if ni(t)n_i(t) agrees for every i and every t ∈ G:
    • G = {T} for the global-sort control;
    • G = every epoch boundary for the epoch sort;
    • G = every block boundary for a block sort.
  • Order. The arrangement of each multiset between consecutive grid points. "Only the order varies" always refers to a stated grid.
  • Windowed support with timescale τ: pt(τ)(i)∝∑s≤t(1−1/τ)t−sci(s)p^{(\tau)}_t(i)\propto\sum_{s\le t}(1-1/\tau)^{t-s}c_i(s).
  • Support bias Γ. The total-variation distance between the recent difficulty mix and that of the full set.
  • Confinement duration. Tθ=∣t:Γ(t)>θ∣T_\theta = |\\{t: \Gamma(t)>\theta\\}|. Tθ/TT_\theta/T, averaged over θ ∈ [0, 1], equals the run's mean Γ.

Box 2 states, for every experiment, which of these it fixes and which it varies: exposure, grid, order, bias, size, and repetition/dwell.

W2. Figure 1. You are right: the submitted figure was schematic. It is now drawn from the actual sampler streams. In the implementation every epoch of the epoch sort is the same full-range ramp, and the new bottom panels show every slot of the first five epochs.

W3. A direct control of confinement duration at fixed exposure (§5.3). We sort within blocks of K epochs. Each block holds every example exactly K times, so exposure is matched at every block boundary and only the length of each sorted stretch changes.

  • On all six tasks t* rises with K (trend tests in Appendix E). Shuffling inside the same blocks stays within 1.1× at every K.
  • On mod-97 addition the delay passes 2× at sorted stretches of about 22 steps (95% CI 21–25), from a fine sweep with 5 seeds.
  • Random level order, with no easy-to-hard direction, delays the transition too.

Gradient statistics as the alternative (§5.7). We logged two quantities in 3,000-step replicas, each paired with its full-length run:

  • the cosine between minibatch gradients 1 and 8 steps apart (39 arms);
  • the alignment of an exponential average of recent gradients with the full-training-set gradient, at timescales from 1 to 512 steps (34 arms, mod-97 addition and GCD).

What they show:

  • Some arm pairs dissociate the cosine from the outcome:
    • in the mod-97 block sweep, K = 1 and K = 4 have nearly the same lag-1 cosine (0.40 and 0.37), but very different Γ, and only K = 1 crosses in these two seeds;
    • repeating each batch 16 times raises the lag-1 cosine above that of 16-epoch sorted blocks, yet costs only 1.2–1.3×.
  • Held out by family, Γ ranks these runs better than the lag-1 cosine (0.73 vs. 0.57). In sample, each adds to the other.
  • We had predicted that harmful arms align less with the full-data gradient. They do not. At the timescale we had fixed in advance, alignment gives C = 0.44 against 0.77 for Γ on the same runs.

In our runs, these gradient statistics do not account for the support effect. We measured lags of at most 8 steps, and Limitation (2) states that the two are partly confounded in our designs.

Part 2/2: Matched prefix, censoring, and your questions

W4. Matched prefix on all six tasks, evaluated densely (§5.2, Table 2). All six tasks, 5 seeds, evaluation every 25 steps, with epoch lengths reported.

  • mod-97 addition and multiplication (9.2-step epochs) and S5 (14.1): epoch-sorted runs stay within 0.87–1.3× of shuffled runs.
  • sort-8: unchanged.
  • GCD and 4-digit addition (156-step epochs): a residual of 1.3× and up to 1.9× remains.

The same sort applied globally blocks the transition on four of the six tasks and delays it 7.5–61× on sort-8 and GCD (Table 1).

Q. How is support confinement quantified? By the support bias Γ over a window, and by the confinement duration TθT_\theta (Box 1). The threshold average of Tθ/TT_\theta/T is the run's mean Γ, which is the statistic we use in §5.7.

Q. Epoch lengths alongside grokking times. Table 2 gives the epoch lengths (9.2, 9.2, 14.1 and 156 steps). Table 3 gives each onset as a sorted-stretch duration in steps, with bootstrap intervals in Appendix E.

Q. Failing within the budget vs. genuinely absent (§5.6). We report survival curves and a stratified log-rank test (p < 0.001; §5.1, Appendix D), and the peak accuracy of every censored arm (Appendix G). Three further results bear on this:

  • Switch to shuffled. After the budget T we switch each never-crossing stream to shuffled epochs. 22 of the 24 runs then reach the transition, in a median of 0.62× the shuffled t*. The other two, one seed in each hard→easy modular arm, stay below 0.9 for a further T steps. A sorted phase can leave the network harder to move than a fresh start.
  • Longer budget. Rebuilding the sort over 3T, no run crosses. This test is weaker than the switch, because holding each level longer also narrows the support of every window.
  • Training fit. At the end of the budget most censored runs do not fit the full training set either: the median final training accuracy over censored cells is 0.09. For them the delay is not memorization without generalization: sorted, block-sorted and replay arms mostly fit only a small part of the training set (Appendices D, G).

As Limitation (1) states, censoring cannot exclude a transition far beyond the budget.

Q. Soundness. We reran every main-text experiment with full per-step logs. The supplementary scripts regenerate every number, table and figure from these logs. Appendix R lists the corrections this produced, including:

  • the evaluation threshold of the growing-pool table, now 0.9 throughout;
  • one matched-exposure row, now 5 seeds run to the full budget.

Do the definitions in Box 1 and the controls in §5.3 and §5.7 address your soundness concern? We would welcome any further control you would find convincing.

Thank you again for your review. As the discussion period closes on Nov 18, we wanted to check whether our two-part response addresses your points: the formal definitions of exposure, order and support (now in Box 1), Figure 1 redrawn from the actual sampler streams, the direct block-length control at fixed exposure, and the matched-prefix control on all six tasks. We are happy to answer any follow-up questions. If the response addresses your concerns, we would be grateful if you would consider revisiting your score.

Reviewer_1Ro5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines whether curriculum-induced delays in grokking arise from example order, exposure counts, or sustained restriction to a narrow subset of the training data. Global sort of the data can strongly delays or prevents generalization.

Critical strengths

A useful experimental distinction. Separating total exposure from its temporal arrangement is important, and comparing global sorting with within-epoch sorting is an informative control. Large, consistent effects. Globally sorted streams fail to reach the generalization threshold on four of six tasks, despite matched total exposure. This is more compelling than a small improvement on one benchmark. Both curriculum directions are tested. Easy-first and hard-first both cause substantial delays, challenging a simple explanation based on learning easy examples first. Transparent limitations. The authors acknowledge censoring, limited seeds, confounding with gradient autocorrelation, and the fact that their language-model experiments are not classical grokking. They also provide explicit falsifiable predictions and describe releasing detailed logs.

Critical weaknesses
  1. Many things not clearly defined (see below)
  2. Not many direct experiments. The authors explicitly propose the decisive experiment about varying block length, slice width, and replay but do not run them.
  3. No connections to important prior works on data repetition inducing useful feature learning

Dandi et al., “The Benefits of Reusing Batches…” (ICML 2024): Reusing batches can produce feature alignment that single-pass learning cannot achieve on the same timescale, with a DMFT characterization of the dynamics. proceedings.mlr.press Arnaboldi, Dandi et al., “Repetita Iuvant”: Even repeated gradient evaluations on the same example can substantially improve feature learning in high-dimensional multi-index models. Importantly, the analyzed mechanism can involve immediate reuse, not waiting until another epoch.

Questions for authors

How does your support explanation relate to results showing that data reuse creates new feature-learning signals? Can you compare repeated random subsets with difficulty-selected subsets, matching subset size, repetition count, and dwell time, to distinguish biased support from reuse dynamics?

What precisely is “effective support,” and over what timescale is it measured? Does it predict transition time across schedules and optimizer settings?

Does difficulty restriction matter beyond subset size? Compare random and difficulty-selected subsets with matched size, dwell time, and exposure.

4 comments

Thank you. You are right that the submission proposed its decisive experiments without running them. We have now run them, together with the comparisons you asked for.

The three proposed experiments, each at fixed exposure (§5.3–5.5).

  • Block length. Sorting within blocks of K epochs delays the transition more as K grows, on all six tasks. Shuffling inside the same blocks stays within 1.1× at every K. Random level order also delays it, so an easy-to-hard direction is not needed. On mod-97 addition the delay passes 2× at sorted stretches of about 22 steps (95% CI 21–25).
  • Slice width and dwell. See Q3.
  • Replay. Replacing a fraction ρ of a global sort with examples from the whole training set restores the transition on every task where the sort blocks it, and earlier as ρ grows.

Q1. Support vs. data reuse (Dandi et al., 2024; Arnaboldi et al., 2024). We discuss both papers (§3) and test an analogue of their setting (§5.4).

  • Immediate reuse. Both papers concern immediate reuse, where the same batch enters consecutive gradient steps while every new batch comes from the full distribution. We carried this over to our tasks and AdamW: every batch of a shuffled, exposure-equal stream is repeated k ≤ 16 times in a row.
    • No k blocks the transition.
    • On GCD, t* rises to 6.8× at k = 16, less than k-fold. On mod-97 addition and S5 it does not rise.
    • Immediate reuse alone therefore does not reproduce the block of a sorted stream.
  • Dwell vs. repetition. Within subsets, the repetition count grows with the dwell. To separate them, we held 5% subsets of mod-97 addition and varied the batch size.
    • Subsets held 5 steps cost at most 1.85×, up to 22 repetitions.
    • Subsets held 10 or 20 steps cost at least 34.8×, from 10.9 repetitions on.
    • On mod-97 addition the dwell decides.
  • Dwell vs. repetition on GCD. Every tested cell delays the transition (2.6–78×). Post hoc, the repetition count orders the delays at least as well as the dwell, so we claim the dwell result for mod-97 addition only.

Q3. Does difficulty restriction matter beyond subset size? Yes, and size matters too (§5.4). At equal subset size m and dwell D, random and difficulty-selected subsets match in size, dwell, repetition count, within-visit order and total exposure.

  • Size. Small random subsets held long delay the transition with no difficulty bias. On mod-97 addition, random 10% subsets held 1,000 steps cross in only 1 of 3 seeds.
  • Bias. Difficulty slices are at least as slow as random subsets in all 26 cells where either arm crosses, with one GCD tie. The pooled seed-paired ratio is 2.7× (95% CI 2.2–3.7) on mod-97 addition and 1.4× (1.2–1.6) on GCD.
  • Growing pools. Admitting examples in random order at the same pacing costs no less than admitting by difficulty on mod-97 addition. On GCD, hard-first, difficulty admission multiplies the delay by a further 2.8×. In our growing pools, size and history carry much of the delay, and bias adds to it on GCD hard-first.

Q2. What is effective support, over what timescale, and does it predict across schedules and optimizer settings?

  • Definitions (Box 1). Windowed support with timescale τ is the distribution of recently seen examples. From it we compute support bias Γ, relative effective support φ and coverage.
  • Across schedules (§5.7). We fit a one-statistic model on all scheme families but one and use it to rank the held-out family's transition times (9 families).
    • Over the weight-decay window we named before the runs, Γ gives Harrell's C = 0.65.
    • Over one epoch, a window we chose after the block-length runs, it gives 0.72.
    • Γ cannot see size. A coverage statistic does better (0.77), but it was chosen post hoc.
  • Across optimizer settings (§5.3).
    • A 100-fold change of Adam's momentum timescale does not move the onset.
    • Stronger weight decay shortens the onset, on two tasks.
    • At a fixed product ηλ the onset still differs, so η and λ both matter.
    • We do not test the statistic across optimizer settings: it is computed from the data stream and cannot see the optimizer.

Do these experiments answer your questions? We would be glad to run further controls.

Thank you again for your review. As the discussion period closes on Nov 18, we wanted to check whether our response addresses your main concerns: the three experiments you noted were proposed but not run (block length, slice width and dwell, and replay), which we have now run at fixed exposure; the formal definitions, now in Box 1; and the connection to Dandi et al. and Arnaboldi et al. on data reuse, which the revision discusses and tests directly. We would be glad to answer any follow-up questions. If the response addresses your concerns, we would be grateful if you would consider revisiting your score.

Thank you for the substantial additional experiments. I am impressed by the block-length, replay, and matched-subset controls, which directly address my main concerns and considerably strengthen the empirical case. While some questions about the underlying mechanism remain, these results make the contribution more convincing, and I am willing to increase my score.

Thank you for reading the revision and for updating your assessment. We agree that questions about the underlying mechanism remain, and the limitations section states where the account falls short. We appreciate your time.

Comments on the paper

1 comment

We thank the reviewers. All three asked for the same three things:

  • precise definitions of exposure, order and support over time;
  • direct tests that vary confinement at fixed exposure, in place of converging evidence;
  • broader controls: the prefix-matched control beyond two tasks, other difficulty metrics and pacing laws, and how our account relates to data reuse.

The revised PDF responds to each, with about 2,500 new runs, all with per-step logs.

ConcernWhat we didWhere
Definitions (all)Box 1 defines exposure ni(t)n_i(t), order relative to a stated grid, windowed support, support bias Γ, size φ, coverage and confinement duration. Box 2 gives what every experiment fixes and varies.§2
Direct test at fixed exposure (xJxW, 1Ro5)The three experiments the submission proposed, now run: block length on all six tasks, slice width and dwell, and replay.§5.3–5.5
Reuse vs. support (1Ro5)Random vs. difficulty-selected subsets matched in size, dwell, repetition and exposure; dwell vs. repetition at fixed size; batch repetition at full support.§5.4
Timescale and optimizer (1Ro5)Sweeps over Adam's β1 and β2; weight decay on two tasks; λ and η at a fixed product.§5.3
Gradient statistics (xJxW)Gradient autocorrelation and alignment with the full-data gradient, both compared with support bias.§5.7
Matched prefix (xJxW, qFWM)All six tasks, 5 seeds, evaluation every 25 steps, epoch lengths reported.§5.2
Censoring (xJxW)Kaplan–Meier curves, log-rank tests, a 3× budget, a switch to shuffled data, and the training fit of censored runs.§5.1, §5.6; App. D, G
Pacing and metrics (qFWM)Square-root, quadratic, exponential and staged pacing; random-admission pools; alternative difficulty keys and a permuted-key control.§5.8

Main results.

  • Dose response at fixed exposure. On all six tasks, t* rises with the length of the sorted stretches, while shuffling inside the same blocks stays within 1.1×. On mod-97 addition the delay passes 2× at stretches of about 22 steps (95% CI 21–25).
  • Size and bias. Both matter. Small random subsets delay the transition when held long enough. Difficulty-selected subsets are at least as slow as random subsets of the same size, dwell and repetition in all 26 cells where either arm crosses, with pooled ratios of 2.7× and 1.4×.
  • Immediate reuse. Repeating every batch up to 16 times at full support never blocks the transition.
  • Replay. Replay from the whole training set restores the transition, and earlier the more is replayed. It is faster than replaying only examples already seen in 32 of 33 comparable pairs; seen-only replay restores the transition late.
  • Timescale. The onset does not move under a 100-fold change of Adam's momentum timescale. It shortens with stronger weight decay, and it also depends on the learning rate at a fixed product. How these set it is open.
  • What the gradients do not explain. The support bias ranks held-out transition times better than either gradient autocorrelation or alignment with the full-data gradient.

We report predictions that failed, among them the onset window we expected, the momentum hypothesis and the gradient-alignment hypothesis, and we state where the account falls short (Limitations 7–8).

Corrections. While rerunning the main-text experiments with full logs, we found and corrected several errors in the submitted version. Appendix R lists all of them. The main ones are:

  • the growing-pool table used a 0.7 threshold; at the stated 0.9, the delays on mod-97 addition are 71–84×;
  • one matched-exposure row rested on truncated logs, and is now 5 seeds run to the full budget, with the same conclusion;
  • "validation stays at chance" overstated two tasks;
  • the data-scale sweep and two forecaster rows used the 0.7 threshold; they are withdrawn and the sweep is rerun at 0.9.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.