acceptodds
Withdrawn from ICLR 2027

When Grokking Fails: Phase Boundaries under Data Skew and Low-Dose Rescue

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Grokking is usually studied under fixed training distributions, leaving unclear how data imbalance changes whether delayed generalization is reached at all. We study this question by continuously varying data skew and weight decay in an autoregressive counting task. Increasing skew systematically contracts the region in which models grok and reveals two qualitatively different failure regimes: a boundary regime where additional training can recover generalization, and a deep-trap regime that remains ungeneralized even after 10^6 training steps. This separation shows that skew can change the accessibility of generalizing solutions, rather than merely delaying their emergence. We then ask whether this loss of accessibility can be reversed through small changes to the training distribution. Replacing fewer than 1% of minibatch slots with samples drawn from a broader distribution can restore grokking, while sample-count-matched interventions that preserve the original sampling distribution or target a narrow rare-state pool are substantially less effective. Rescue strengthens as the support of the injection distribution broadens and remains possible even when putative bottleneck states are excluded, arguing against a simple missing-rare-example explanation. The same intervention ordering transfers to modular addition. On Dyck-1, skew again produces a structured failure region and broader injection remains preferable, although complete rescue does not transfer and longer training can expose late-stage collapse. Overall, these results identify training-distribution coverage as an empirical control axis for both the accessibility and recovery of grokking under data skew.

Primary area:
learning theory
Keywords:
Grokking, Delayed generalization, Training distribution coverage, Distribution injection, Generalization failure

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_LLHaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Paper is beyond limits and figures can be compressed. Not meeting ICLR submission guidelines.

Critical strengths

none

Critical weaknesses

none

Questions for authors

none

Reviewer_khNROpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how training-data skew affects grokking in algorithmic tasks and whether small changes to the training distribution can recover generalization. The authors introduce a Zipf-based sampling distribution and examine the interaction between data skew and weight decay, constructing empirical phase diagrams on an autoregressive binary counting task. They identify a boundary regime where additional training can improve generalization and a deep-failure regime where models remain unsuccessful within the tested computational budgets. The paper further investigates low-dose sample injection, finding that replacing a small fraction of minibatch samples with uniformly sampled examples can substantially improve generalization. Comparisons with alternative sampling strategies suggest that broad distributional coverage is more effective than targeting a narrow subset of rare states.

Critical strengths

The paper investigates the interaction between data skew and weight decay across a range of settings, rather than relying on a small number of selected configurations. The finite-budget experiments and additional training-length comparisons provide useful observations about how generalization success changes under different sampling distributions.

Critical weaknesses
  1. The experiments do not directly establish the core grokking and recovery phenomena claimed by the paper. The paper introduces grokking as delayed generalization occurring after a model has fitted its training data. However, its operational criterion for a successful "GROK" outcome is simply a final uniform evaluation accuracy above 0.9. The experiments do not report training accuracy or training-loss trajectories establishing that the models first fit the training data before exhibiting delayed generalization.

  2. Limited mechanistic explanation and insufficient evidence for the broader interpretation of the empirical boundaries.

  3. The paper exceeds the page limits significantly, has clear typos.

Questions for authors

The submitted manuscript exceeds the ICLR 2027 nine-page main-text limit, with the main text extending onto page 12 before the references. Additionally, there are unresolved cross-references (e.g., "Appendix ??" on page 11) and other formatting issues. I strongly encourage the authors to strictly adhere to the conference's submission guidelines, including page limits, required sections, and manuscript formatting requirements, and to carefully review and correct these issues to ensure the submission is complete and meets professional standards.

Reviewer_PqhyOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission should be desk-rejected due to format violation without going through the full review.

The submitted manuscript appears to grossly violate the ICLR 2027 formatting requirements. The main text extends onto page 12, exceeding the 9-page limit for initial submissions. Additionally, the left and right margin has been reduced from 1.5-inch to 1 inch.

(This is a placeholder review to prevent the system from marking this review assignment as incomplete.)

Critical strengths

N/A

Critical weaknesses

Format violation.

Questions for authors

N/A

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.