acceptodds
Withdrawn from ICLR 2027

What a Grokking Plateau Can Be Rescued By Is Not What It Is Spent Building: Component Surgery in a Glass-Box Network

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A grokking plateau poses two questions usually conflated: which component, supplied, ends it, and which one training spends it building. We separate them — exploitable from deferred — in a glass-box network whose hidden units all compute one fixed primitive, , and learn only routing and gains, so its component boundaries are architectural. It groks modular addition in 17/19 runs. Implanting a run's 100k-step embedding into its own 30k-step checkpoint reaches 0.9 test accuracy in 1.2k steps against 12.7k for a fresh-optimizer fine-tune-only baseline; a capacity-matched readout head is slower, a random direction at matched norm does not rescue on this task, and rescaled host structure ends below baseline at 20k. An untrained cosine bank rescues if its frequencies are position-shared. A registered control refuses the missing-component reading: the host already spans 87% of the donor embedding's norm, and that in-span part reads higher at 2k (0.823–0.834 against the orthogonal part's 0.289–0.527, three hosts, though it crosses 0.9 first on only two of them) — yet 72% of it is not the host's own structure rescaled. The orthogonal part rescues too, so the in-span part is better early, not necessary: amplification fits as well as supply. Either way the rescue works only in a host short of the solution. Exploitable and deferred come apart on both tasks — embedding exploitable, wiring deferred — but the wiring's exploitability inverts: harmful early on modular addition, immediately useful on parity. What parity's embedding defers is its allocation of power to the support bits, not its convergence. All results are small-scale, on two task families.

Primary area:
interpretability and explainable AI
Keywords:
grokking, mechanistic interpretability, delayed generalization, training dynamics, representation learning, science of deep learning, causal intervention, sparse parity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_pwrPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

see weaknesses.

Critical strengths

N/A

Critical weaknesses

Weaknesses

  • This paper is very clearly fully AI generated and not in a good way. The authors didn't even go through the process of re-writing paragraphs to be easily readable (its just a wall of text) or fix the plots e.g. Figure 2 (d) / Figure 3 to not have legends overlap half of the plots. Research is a means to communicate findings in an easy and understandable way and unfortunately the authors didn't prioritize this goal. The appendix and the main portion read like a research log rather than an actual paper and even some figures are missing even though they are referenced in the text (Figure 5, fig. A-E2). Clearly the authors aren't even sure what their contributions are "Our contributions are three" is followed 13 numbered items (l. 89–120). The paper has 12 matches for "an earlier draft" which shouldn't really appear in any camera ready paper and the text even mentions run IDs (i0_d1, emltree_wd0.1_cpu, parity403_i1), script names (rung3_transplant.py, scratch_crossrun_t90.py) and log paths (runs/transplant.log, runs/registration_timestamps.txt).
Questions for authors

N/A

Reviewer_6GPROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The authors introduce a small “glass-box” network for modular addition. Its units use a fixed arithmetic primitive; learning is concentrated in a dense input embedding, discrete routing (“wiring”), and a dense readout. The central experiment transplants components from a later, grokked checkpoint into an earlier checkpoint from the same training run. On the main modular-addition example, transplanting the embedding leads to 0.9 test accuracy in about 1.2k fine-tuning steps, compared with 12.7k for the fresh-optimizer fine-tuning baseline. Controls indicate that norm alone does not explain this rescue, and that some structured cosine embeddings can also rescue the host. The paper’s training-time analyses point to a more nuanced picture: the embedding is highly exploitable, while the wiring is deferred; on sparse parity, the wiring is both deferred and immediately useful when transplanted. The authors also report that later wiring layers remain unsettled at their training cutoff, so the strongest commitment-ordering claim is limited to an early embedding rung and a core layer-0 wiring rung.

Critical strengths
  1. Transplanting a component from a run’s own future checkpoint into its past is a compelling way to probe what helps a plateaued host, and differs from simply correlating internal measures with generalization.

  2. The paper tests norm-matched alternatives, random directions, synthetic embeddings, component ablations, and multiple host checkpoints. It also distinguishes the embedding transplant from a capacity-matched readout transplant, which helps address the possibility that any dense parameter block would work equally well.

  3. The embedding result is tested across multiple donors and hosts; the paper also examines sparse parity and includes narrower transfer experiments in an MLP and a transformer. These broaden the evidence beyond one successful modular-addition run, even if they do not establish broad generality.

Critical weaknesses
  1. The donor embedding is not shown to be necessary, and the in-span analysis favors an amplification interpretation at early measurement times. The paper appropriately acknowledges this, but the distinction between supplying missing structure and amplifying host structure is central to what the transplant means.

  2. The central findings come from small models and two synthetic task families, with parity lineages sharing the same support bits. Several important controls or transfer cells are single-host or single-measurement experiments. This limits confidence that the reported dissociation generalizes beyond these setups.

  3. The headline compares fine-tuning conditions using freshly constructed optimizer states, rather than inherited host optimizer state. Since all arms receive the reset, it does not explain the between-arm difference, but it leaves unclear how the effect compares with uninterrupted training or an intervention that preserves optimizer state. Also, first crossing of 0.9 test accuracy can be sensitive to evaluation noise and does not alone characterize stability or final performance.

Questions for authors

Please address points raised in Weaknesses.

Reviewer_kMvVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The reviewer is unfortunately not able to provide a clear summary of this paper, as it was not understandable from a human perspective.

Critical strengths

I was not able to understand this paper, nor to extract strengths from it.

Critical weaknesses

This paper may contain interesting ideas. However, the reviewer was not able to access them, as the current writeup does not seem human-readable. I understand it as a collection of AI generated lab notes, and the general ideas (if any are present) are difficult to grasp (I was not able to), even at the level of the abstract that contains too many details to be useful for a human reader.

I recommend a full human-readable rewrite separating the essential ideas from the very fine experimental details.

Questions for authors

A human-readable rewrite of the paper would be a first step. The appendix may contain AI generated content useful for reproduction, but the core of the paper should be understandable to a human.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.