acceptodds
Withdrawn from ICLR 2027

Representability Decides Whether Grokking Can Occur

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A flat test curve has several possible causes. The target may lie outside the architecture's function class, or the training sample may permit memorisation without constraining the population solution, or the optimiser may need a long time to find a generalising representation. Learning curves do not tell these apart, and existing grokking analyses begin only once memorisation is feasible. We separate them before training. Our instrument is an exactly solvable two-layer network with holomorphic activation on roots-of-unity-encoded modular inputs. Its expressible class is an algebraically fixed -dimensional subspace of characters of , independent of width, and a task is representable exactly when its discrete Fourier support lies in . For linear-phase targets this becomes the arithmetic criterion , an identity in on the canonical residues rather than a congruence modulo . From the class we extract two quantities computable in advance, the spectral representability defect and the conditioning of the class restricted to the training sample, and prove two consequences that do not follow from the definitions. The global obstruction reaches the finite sample as a width-independent training-loss floor with an explicit rate, so a non-representable target cannot be memorised even though a flexible model would interpolate it. For representable targets under a well-conditioned restriction, training and population losses are sandwiched together at every iterate, so the prolonged separation that defines grokking is structurally excluded. Across 585 runs the pre-training prediction matches the observed regime with accuracy. Two experiments carry the diagnostic beyond the solvable model. A bottlenecked one-hot ReLU network, given no Fourier basis, passes through four regimes as the bottleneck widens. At test accuracy stops at a ceiling that a fourfold step budget does not lift, and at the outcome stops being a function of width: six seeds of eight escape after delays spanning two orders of magnitude, and two never reach the chance rate at all. Swapping AdamW for vanilla gradient descent at fixed model and target reverses the degree-dependence of the acquisition delay in all sixteen cells of a four-target grid crossing vanilla gradient descent with three AdamW weight decays, so an observed scaling of grokking time can be optimiser-induced and not a property of the landscape. Which learning regime exists at all is set by the geometry of the effective function class, not by parameter count.

Primary area:
learning theory
Keywords:
grokking, representability, learning theory, function class geometry, modular arithmetic, exactly solvable models, optimization dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_QKTaOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how representational capacity influences grokking. For a restricted two-layer complex-valued polynomial network on modular arithmetic, it characterizes the conditions under which target functions can or cannot be fit. The basic idea is that if a setting is sufficiently restricted, such that the targets are representable and the training samples are informative enough, then low training loss forces low error on unseen examples, and a prolonged grokking loss gap does not occur.

Critical strengths

Interest in Grokking: Grokking is a consequential topic for understanding neural network generalization. For this, I appreciate the authors’ interest in the topic.

Critical weaknesses

1. Failure to generalize is not the same as grokking: Grokking concerns delayed generalization after a model has already fit its training data. A substantial part of the paper focuses on representational impossibility (cases where the model cannot fit even its training data). Saying that “a network fails before grokking becomes possible” is like saying “acceleration is impossible whenever a car doesn't start”. It's honestly quite a tautological observation whose relevance to explaining grokking is unclear. Theorem 4 has an even odder argument that basically says “if we consider a sufficiently constrained setting with a representable target and sufficiently informative training examples, then low training error forces low error on unseen examples”. This is, of course, again quite limited as an explanation, as the definition of grokking is specific to when there is a delay in generalization. Therefore, simply telling us that a prolonged grokking loss gap cannot occur under some restrictive set of assumptions about the network and dataset setting, does not help identify the underlying cause of grokking, which occurs specifically not in these settings.

2. Severely limited novelty: Grokking has already been studied across a wide variety of settings, with prior work linking it to training hyperparameters [1,3], weight decay [1,2,6], representation learning [2,3], and embedding optimization dynamics [4]. Studies also extend beyond algorithmic tasks [2], including delayed generalization and delayed robustness in CNNs and ResNets on natural images [5]. Yet, this paper's main analysis uses a polynomial network with fixed input representations, which is an even more restrictive setting than Power et al. [1], who used Transformers with learned embeddings. It remains unclear what studying this restricted setting adds to our understanding of grokking, especially given the broader models and tasks already studied.

References

[1] Power et al., "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets", 2022.

[2] Liu et al., "Omnigrok: Grokking Beyond Algorithmic Data", ICLR 2023.

[3] Liu et al., "Towards Understanding Grokking: An Effective Theory of Representation Learning", NeurIPS 2022.

[4] AlQuabeh et al., "Mechanistic Insights into Grokking from the Embedding Layer", 2025.

[5] Humayun et al., "Deep Networks Always Grok and Here is Why", ICML 2024.

[6] Nanda et al., "Progress Measures for Grokking via Mechanistic Interpretability", ICLR 2023.

Questions for authors

I understood the work sufficiently.

Reviewer_Gnf9OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a fixed function class with monomial (holomorphic) activation functions to explain whether grokking can occur or not.

Critical strengths

None.

Critical weaknesses

Clarity: The paper is extremely poorly written. For example, the paper introduces, directly in the abstract itself. technical terms or symbols without definition (“canonical residues”; “at d = 4”, what is d ?; etc.) There are internal inconsistencies. For example, around line 155,

Output and target both lie on the unit circle, so the loss is a squared chord distance. The target is unimodular while \hat{y} ranges over C with no normalisation imposed.

They claim that the grokking time follows 1/k21/k^2 but the actual numbers disagree quite a lot (0.27 v/s theoretical value of 0.11 on line 421.)

The motivation in the introduction also fails to discuss Doshi et. al. [1] and Gromov [2] which directly consider analytic solutions and the representability hypothesis, central to the goals of this work. (While [1] is referred to in the main text, [2] is omitted completely in the main text)

Limited relevance and novelty: The paper considers an unusually constrained model of the data. Failure to represent the training data is not a big surprise in that case. However, its relevance to actual grokking is not well-established. There is no description if actual neural networks show the features assumed by the function class. In that case, comparison with representations in prior work like that of [2], [3], should be better emphasized.

Mischaracterization of previous work: The fixed encoding and function class is materially different from the training-based learnability hypothesis in [1]. [1] questions whether real neural networks can grok on arbitrary modular polynomials, and provide a conjecture based on representation of the polynomial. They then provide empirical support for this conjecture. This paper asks whether the assumed function class can represent certain modular functions. However, the correspondence between this fixed function class and actual neural networks is missing.

Insufficient connection between experiments and theory: The ReLU bottleneck experiment is not well-explained. It is difficult to understand what the experiment is meant to show, especially in relation to the solvable model that the authors considered. For example, does this standard architecture also show the same features as that of the toy model ? it does not compute the proposed representability diagnostics or establish that the theoretical obstruction explains these regimes. The numbers reported are scattered and fail to establish a coherent picture.

[1] Doshi, D., He, T., Das, A., & Gromov, A. (2024). Grokking modular polynomials. arXiv preprint arXiv:2406.03495.
[2] Gromov, A. (2023). Grokking modular arithmetic. arXiv preprint arXiv:2301.02679.
[3] Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217.

Questions for authors

I don't have any questions which could materially affect my assessment. I urge the authors to fix the presentation and re-consider the relevance of their work to the broad grokking literature.

Reviewer_ArMAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies a two-layer complex network with activation σ(z)=zk\sigma(z) = z^k, trained on modular arithmetic tasks. The inputs a,ba, b are encoded as roots of unity (ωa,ωb)(\omega^a, \omega^b), where ω=e2πi/p\omega = e^{2\pi i/p}. At any width, the network's outputs span the same k+1k+1 Fourier functions ωℓa+(k−ℓ)b\omega^{\ell a + (k-\ell) b}, ℓ=0,…,k\ell = 0, \dots, k (Proposition 1), so a target ma+nbma + nb is learnable exactly when m+n=km + n = k (Theorem 2).

Critical strengths
  • The paper gives a network whose function class is known exactly. That makes it a clean teaching example.
  • The settings are fully reported (App. F), the experiments are cheap, and several limitations are stated openly (Section 7, Remark D.22).
Critical weaknesses
  • I think the title is a bit confusing from a machine learning standpoint, since a hypothesis class that is not learnable, is by definition, not learnable? and on the other hand, I don't think the paper provides evidence that if a hypothesis class is learnable, grokking will happen, independent of the task? Moreover, the model covered by the theorems never groks (Section 5.2). The networks that do grok (App. E.4, Section 5.3, App. E.6) are not analysed.

  • I think the abstract has claims with undefined terms and statements: what's a flat test curve? what is considered "flat"?

Here's some more concerns:

  1. The theorems are elementary. By Proposition 1, the network at any width is a linear model on at most k+1≤7k+1 \le 7 fixed features, trained on 3,763 points. Theorems 2, 4 and 6 are standard facts about such a model. The m+n=km + n = k pattern in Figure 2 comes from Eq. (1) having no bias term.
  2. The bottleneck result is misread (Section 5.3). The paper attributes the d≤2d \le 2 failures on rank, but width d=1d = 1 can represent a+b mod 97a + b \bmod 97: the ReLU layer can map each of the 193 possible sums to its residue. The failure therefore comes from training, not from what the network can represent.
Questions for authors
  • What's a holomorphic activation function? I think the f(x)=xaf(x) = x^a activation function is called polynomial in the machine learning community.
  • What's the well-conditioned restriction that the authors mention in abstract but never elaborate on?

Here's some more questions:

  1. With a bias term, (W1x1+W2x2+b)k(W_1 x_1 + W_2 x_2 + b)^k, every target with m+n≤km + n \le k becomes learnable. How much of Figure 2 remains?
  2. Can δ2\delta^2 or the Gram-matrix conditioning (cS,CS)(c_S, C_S) be computed or estimated for any network that groks, such as the ReLU network of Section 5.3? If not, how does the diagnostic apply to grokking?
  3. A bottleneck of width d=1d = 1 can represent a+b mod 97a + b \bmod 97. Do the outcomes at d≤4d \le 4 change with smaller weight decay?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.