acceptodds
Withdrawn from ICLR 2027

Interpretability-Guided Design from a Head-Binding Failure in Grokking

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformers often generalise long after fitting their training data, a delayed transition known as grokking. Yet on some tasks the transition never comes, and it has been unclear what fails inside a model that does not generalise. We study two-layer transformers trained on math tasks, such as taking a derivative, whose answer combines several inputs. There the failure lies in variable binding: which input each first-layer head attends to. On the derivative task, most models that fail have one head bound to the constant term, which differentiation ignores, and no model that succeeds has one. Ablating that head costs far less than ablating the others, because the input it reads plays no part in the answer. Transplanting only a generalising model's embedding and readout layers, 12.5% of its weights, repairs all six failed models we tried. With the inputs shuffled, all 19 bound heads read their input through a learned tag instead of its position. The wrong binding is usually set within the first few hundred of fifty thousand steps, as each head's attention sharpens onto one input. SoftBind keeps the first layer's attention soft over the first 500 steps, so no head locks onto an input before the task can steer it. On a sum task this more than doubles the grok rate, and keeping the softening for the whole run raises it from 26.2% to 85.2%. Interpretability here does more than explain a failure: it can tell us where to intervene.

Primary area:
interpretability and explainable AI
Keywords:
grokking, mechanistic interpretability, variable binding, attention heads, training dynamics, generalization, induction head

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_imiBOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on the phenomenon of grokking, that is when a neural networks starts generalizing on fresh data many epochs later interpolation on the training set. The authors aim to understand what fails when a model never generalizes compared to when it groks. The authors claim that this can be due to the fact that some heads focuses on portion of the inputs that are not relevant to solve the task and, therefore, training get stuck. Moreover, they show that, when transferring the embeddings and the readout layers of models capable of generalizing, the models that were failing to generalize now exhibit grokking. They also show that, when a model get stuck because of an head focusing on useless information, this typically happens in the first stage of the training. They thus introduce SoftBind, a technique preventing the head to specialize on a specific part of the input, before having understood the general task to accomplish.

Critical strengths

The paper addresses an important question, that is trying to understand why certain models trained on math tasks fail to generalize while some others exhibits grokking.

The paper shows some potentially interesting phenomenology and empirical findings, such as that transferring embeddings or specific layers of a model capable to generalize on a given task save some seeds of another model that are failing to generalize, to instead start grokking. They also propose a mechanics to avoid early specialization of heads on useless parts of the input, before the model has a general understanding of the task.

Critical weaknesses

I think the writing of the paper as well as the plots should be consistently improved. It is especially hard to understand the novelty of this works since a "related works" section is missing. Moreover, the level of details to describe the experiments and the eccessive terminology and keywords most of the time obscures the logic behind the experiments themselves.

I think that the main weakness of the paper is that it seems more a report of a series of experiments, with many phenomenon detected but not a clear explanation and identification of the underlying mechanism generating them.

Questions for authors
  • How standard is the grokking criterion established by the authors? Because in Fig. 2 it seems that if we wait a little bit longer, also the validation loss of those seeds the authors say they never grok would instead grok.

  • The math operations "det", "dot" and "poly" do not contain "traps", so why they do not grok?

  • Why H > K has just been investigated for some tasks and not all of them? Same for H <= K.

Reviewer_hLokOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies grokking in arithmetic tasks with 2 layer networks. The paper finds that overfitting is related to attention heads attending to irrelevant terms, e.g. constant offsets +c in differentiation. The paper examines attention heads, transplating the parameter weights between models, and eventually proposes to soften the attention in the beginning of training, which is found to substantially improve the rate of grokking.

Critical strengths

The paper attempts to showcase an example where mechanistic understanding leads to positive intervention in the training dynamics.

The experimental results in the paper are presented transparently and discussed in depth.

Critical weaknesses

Given the claim that interpretability produces a valuable intervention here, the SoftBind intervention is only evaluated in a fairly a synthetic setting and it is not studied whether this intervention will generalize to deeper models and more realistic datasets.

The paper is near-unreadable with sentences such as "Ablating that head costs far less than ablating the others, because the input it reads plays no part in the answer. Transplanting only a generalising model’s embedding and readout layers, 12.5% of its weights, repairs all six failed models we tried." Similarly, the paper is littered with unreadable claims: "But the sixth, the softening held for the whole run rather than annealed, is the only arm that goes further than SoftBind: it groks 104 of the same 122 seeds against 66, and 5 of 32, 24 of 32 and 24 of 31 when held at L1 only, at both layers, or only after step 1000 (Tab. 11)." Since otherwise the impact and scope of the paper is limited in my opinion, this warrants a clear rejection.

Questions for authors

n/a

Reviewer_v5EtOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors train an algorithmic transformer with 2 layers and 4 heads on arithmetic functions of arity 4, like determinant or partial sum, with a custom vocabulary expressing integers between -100, 100 and a special answer output token. They select the functions to never use at least one of their four inputs, and show that models cannot grok when a layer one attention head fixates early on an unused input. In fact, they show that having number of heads equal to or less than the number of used inputs never groks, which makes sense as with a single separator token there is not enough space for a transformer to copy what it needs from the input. They show that if the heads exceeds the number of used inputs, then it sometimes groks. They investigate translating some of the learned weights, and find in their setting that transplanting embeddings can induce grokking whereas transplanting other weights does not help. On the task of summing three integers, they also show that the number of input positions visible to the query at the token before the answer must be greater than the number of heads (“V>k”) for the embedding transplantation to work.

Critical strengths
  • The authors provide all the relevant details around architecture, vocabulary, and training parameters to fully reproduce the training experiments.
  • Standard causal ablations and validations are provided to corroborate that the “mis-bound” head is indeed functionally relevant to failing the task.
Critical weaknesses
  • The reviewer found the paper to be poorly written. For example, in the Contributions outlined on L070-082, what is the “seven time as many” referring to and what is its relevance? In the third contribution, what does “it” refer to in the first sentence? The word “grok rate” is used in the abstract, introduction and throughout the paper without a clear definition. Similarly, a “single-trap rule” is never defined and it is not clear what it refers to. It would be helpful to remind the reader of the definition of V on L315. The label on Table 2 is particularly hard to read, as the use of “anneal” does not appear earlier in the paper and it is not clear what the effect on prodsum is referring to. The label on Figure 6 refers to “Sophia” and “AdEMAMix” which are not further explained or motivated as to their inclusion ini the chart. The discussion makes a conclusion about “NormSoftmax, whose rows cannot sharpen,” but this ablation is not discussed earlier in the paper.
  • The reviewer's main objection is about the generalizable value and applicability of the results. The authors run many experiments to investigate when grokking occurs in their algorithmic transformer setup, but it is not clear how their result can generalize to real training data and architectures. The architecture is constrained by head count, the vocabulary is constrained, and even the number of tokens allowed for computation between input and output is constrained to a single token. Each of these axes should be directionally ablated towards a more realistic transformer, or some justification provided on why these results may generalize in explaining grokking in other settings.
Questions for authors
  • If donating the embedding on a successful grokked run is required, as you suggest in your results, can you share more details about what happened to the embedding in failed runs? For example, do all of the integers between -100 and 100 separate and have adequate distance between them as measured by something like cosine similarity? Or might the unsuccessful runs be due to poor learning of embeddings? And if there is no discernible structural difference in the geometry of the embedding in grokked and non-grokked runs, what other mechanistic justification is there for why embeddings of grokked runs are privileged? Or maybe it is just the failure to learn positional embeddings given their criticality on mapping to formula inputs?
  • Can the authors clean up the presentation of the paper as discussed in the Weaknesses section, ensuring that all terms are clearly defined before they are used, and that experimental variations are motivated in the main body before being included in discussion results or charts/figures?
  • The boundary between grokking and not grokking, as investigated in the paper, seems heavily determined by the constraints of the architecture and task setup, namely restricting the number of heads to about as many as the number of inputs, while simultaneously only giving the model only a single separator token to compute the result. This apparently makes it rather artificial to the setup, rather than something that can generalize to transformers trained on real-world data. Can the authors discuss future work and these limitations, and how their results can have any type of generalization to more than this setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.