acceptodds
Under review as a conference paper at ICLR 2027

Built First, Kept: Task Symmetry Selects What a Recurrent Network Represents

Abstract

Several different sets of features can solve one task, and a trained network holds one of them. We ask what selects that set, and whether it is kept. We study the word problem of a finite abelian group, written as modular counters, with one bit of supervision per position. A recurrent network solves it, yet in most runs one counter is nearly missing from its features. A symmetry is a relabelling of the symbols that leaves the task unchanged. We prove that updates treating alike the symbols a symmetry exchanges can raise only a feature that cannot tell them apart. At counters of equal size, a feature that ignores which symbol advanced it is built first in every run that built a feature before its symmetry broke. Rewriting the task in a new alphabet, with the counters fixed, moves the symmetry and the first feature with it. Making that feature cost more to learn does not move the choice to cost while the symmetry lasts. Whatever feature the network builds first it keeps, in almost every run that solves, so the start of training settles the choice. We prove why under an idealisation of the network. At counters of equal size, the kept feature is one the task could do without, and it takes one counter’s place. Thus, what a trained network holds depends on how its task was written down.

Primary area:
interpretability and explainable AI
Keywords:
learning dynamics, feature learning, symmetry, recurrent neural networks, state tracking, word problem of finite abelian groups, Fourier features, mechanistic interpretability, representation selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vRbhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper addresses an interesting question: how does a model select one set of features out of the many possible options, and whether the initial selection is kept or replaced. The authors focus on a specific, somewhat narrow setting: a word problem of a finite abelian group, and studies 3 RNNs for the task. Several theoretical and empirical results are then presented, in particular that models select their feature set early on and tend to keep them.

Critical strengths

The question the authors focus on is of interest. It is also addressed rigorously.

Critical weaknesses

I would say upfront that this paper is far from my research expertise. Having said that, I found the paper quite hard to follow. The structure was hard to decipher; I could not tell how the different parts connected, what was the role of each part, and how they all came together to address the paper's main research questions. I would advise the authors to add introductory paragraphs or sentences to each of the sections/sub-sections that map the current part in context, and a summarizing text outlying what we found out, how it matters, and how it relates to the next part. Another example is Figure 1. It took me some time to understand what 1(a) is showing, and I still don't understand what I'm seeing in 1(b). Again, this could be my lack of expertise speaking, but some discussion of the meaning of the graph (e.g., is higher or low better?), what does it tell us/teach us would be helpful. And finally - even the conclusion is a list of statements that make no coherent story.

Somewhat related, an important question that authors do not address is what can we learn from these experiments about more realistic problems? Do we have reason to believe these findings transition to real models? Are there ideas on how to test this?

Questions for authors

N/A

3 comments

We thank the reviewer for the careful reading and for the concrete suggestions on the structure, Figure 1 and the conclusion. We agree that the submitted version was hard to follow, and we have revised its presentation throughout. The revised version has been uploaded.

The main changes are:

  1. Sections 2 to 5 now open by stating what each section does and how it connects to the others, and Sections 4 and 5 end with a summary. Each section and subsection title now states what that part shows.

  2. The abstract, introduction, and conclusion have been rewritten. The conclusion now follows one line of argument: the question, what we found, and what it implies.

  3. Section 3 now starts each step from the example of Figure 1(a) and states in plain words what each result means. Two lemmas have been moved to the appendix, and our own terms have been replaced with standard ones (see Table 2 in Appendix A).

  4. Figure 1(b) now shows the energy of four main features, in place of the shares of all features and a strip for the symmetry measure. Its three marks are labelled, and the caption explains how to read the curves.

Figure 1(b) summarises the entire paper in a single network, so we also explain it here: the panel shows the story of a single network during training. Each curve shows how much of the hidden state follows one feature, so a higher curve means more of that feature, and neither height is better. At first, the network’s input weights for a, b, and c stay nearly equal, so it cannot tell the three symbols apart. It can then learn only the total-count feature, and at epoch 23 this curve rises first (i). The total count alone cannot solve the task, so at epoch 53 the weights move apart and the symmetry breaks (ii). The network then learns the features of counters 2 and 3 and retains the total-count feature. These three together determine counter 1, so counter 1’s curve stays near zero to the end, and we say that this counter is left out (iii). Section 1 (page 2) walks through the same story, and Sections 4 and 5 show how often each step occurs across all networks.

On realistic problems. We have not tested our findings beyond our tasks, but we can explain why we expect some of them to carry over and how to check.

The mechanism uses nothing specific to recurrent networks. When the data treat two tokens identically, gradient descent moves their embeddings in the same direction, and only the small random differences at initialisation tell them apart. For the same reason, two hidden units with identical initial weights stay identical. Realistic tasks may contain such tokens: the names of entities in a story or variables in a program can be exchanged without changing the answer. We therefore expect any model trained from small random weights to learn first the features that treat such tokens alike, for example, how many events have happened, before features that tell the entities apart. A practical consequence for interpretability is that a trained model may hold such a pooled feature alongside, or in place of, some per-entity features, as our networks do.

Our conclusions may not apply in every setting, for two reasons. First, natural data are rarely exactly symmetric, so in a realistic task the symmetry may break much earlier than in ours. Second, our evidence that the first feature is kept comes only from our tasks, and our explanation of it holds in an idealised model.

To test this in larger models is a direction for future work. For any task with interchangeable tokens, one can track how far their embeddings move apart during training, as our measure R does, and record which features remain close while they are still close.

We hope that the revised paper is now easier to follow, and we would be glad to answer any further questions.

Thank you for the detailed response. Unfortunately the changes made would require a new, complete review process, and are out of scope for a rebuttal. I hope these changes will make the paper easier to digest for the next set of reviewers.

Thank you for reading our response. For the record: the theorem, the symmetry lemma and the runs behind the main results are those of the submission, and the revision mainly changes how they are presented. The additions were requested in the reviews: one corollary (Section 3.3), a weight-decay experiment (Section 5.1) and counts moved from the appendix into the main text. We understand your position, and we thank you for the comments on presentation, which shaped the revision.

Reviewer_o8Z4OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies which features a recurrent network learns when several feature sets can solve the same task, focusing on which feature is learned first and whether it persists through the rest of training. The authors study this on the word problem of a finite abelian group, written as modular counters, with a binary accept label at every position. They show that, with counters of equal size, several minimal feature sets are tied in how fast they are learned. They prove that, as long as training treats symbols exchanged by a symmetry of the task identically, only features invariant to that symmetry can grow. The authors empirically show that, on LSTM, GRU and Elman networks, the symmetry-invariant “symbol-blind” feature is learned first whenever the symmetry has not yet broken, even when another feature would be faster to learn. The first-learned feature is then kept in almost all successful runs, which the authors explain with an idealized gradient-flow model. The authors conclude that what a network learns depends on how the task is presented, not only on what it computes.

Critical strengths

The research question is interesting and, if the clarity issues are thoroughly addressed, the paper could be a valuable contribution to the community.

Critical weaknesses

Clarity: the paper material is buried under poor writing. For instance:

  • Abstract: "A recurrent network solves it, yet in most runs one counter is nearly missing from its features." Is it expected that all counters should be part of the network features? "We prove that updates treating alike the symbols a symmetry exchanges can raise only a feature that cannot tell them apart." This sentence is really hard to read. "At counters of equal size, a feature that ignores which symbol advanced it is built first in every run that built a feature before its symmetry broke." Symmetry breaking is undefined, and the claim is restricted to an unspecified subset of runs. "Rewriting the task in a new alphabet, with the counters fixed, moves the symmetry and the first feature with it." What does "counters fixed" mean? "Making that feature cost more to learn does not move the choice to cost while the symmetry lasts" I am not sure I understand this sentence. Cost is not defined in the abstract, so the sentence cannot be interpreted without reading Section 3.2. "Thus, what a trained network holds depends on how its task was written down." What is the logical connection with how the task was written down, why not seeds, parametrization, etc.?

  • The paper relies on a private vocabulary of ordinary words used in a technical sense, defined only in passing and often after they are first used: build, break, cost, tier, carried, protected, etc.

  • Several key sentences are very hard to parse. Examples:

Section 1 (lines 82–85): “The proof needs only a label the relabelling leaves unchanged, as the accept bit is and a label that names the state is not, and holds for any network that reads each symbol through its own input column, transformers and state-space models included.”

Section 3.2: “So cost leaves several complete sets of the same size exactly tied: the symbol-blind class and a fundamental tie by an identity, and the fundamentals tie with one another by the symmetry of Definition 4.”

Section 3.3: “They agree where the tier holds a protected class and separate where it holds none, as at (3, 3, 3) at w = 2 and at signed (4, 6), where cost names a counter’s class and the prediction the symbol-blind class.”

Section 4.3: “Every run that built a counter first had lost the symmetry before its build (section 4.2), so before the break the higher cost does not move the choice; the architectures differ in how often the break comes first (section 4.4).” The reader cannot tell from this that the symbol-blind feature comes first in only about half of the GRU and Elman runs (Table 7).

Section 5.2: “Of the four losses of section 5.1, those at (4, 6) are declines where the lemma’s model has the class grow, though it never left the code, and the others lie outside the lemma’s condition (appendix B).” This sentence reports evidence against the paper own lemma. It is so compressed that a reader would likely miss that it is an anomaly.

These clarity problems make it very hard to assess the overall quality and significance of the contributions.

Substance: Key quantities are buried in the appendices, and once surfaced they substantially narrow the claims: the symmetry lasts until the first feature is built mostly in LSTM runs, and the “symmetry decides” result depends on the threshold chosen for the break (Tables 5 and 7). The persistence result relies on training without weight decay and on a strongly idealized lemma, and the missing counter appears in only a minority of runs at (2,2,2,2) and (2,2,2,2,2) (Table 9).

Questions for authors

My assessment is driven primarily by clarity. The question that could materially change my score is whether the authors can provide a substantially revised manuscript that addresses the issues listed under Weaknesses:

  • using standard or clearly defined terminology introduced before use,
  • rewriting key sentences clearly, with explicit logical connections between claims,
  • stating in the main text, for each claim restricted to a subset of runs, the size of that subset relative to all runs,
  • moving the key evidence (Tables 5, 7 and 8) from the appendices into the main text, with a discussion of what they imply for the scope of the claims.
2 comments

I have read the revision submitted by the authors, and now find the paper understandable and properly written. The lack of clarity was the reason behind my score of 1 (clear rejection), which is now resolved, so I will update my score to a 2 (weak rejection).

The authors have responded to my comments on substance mainly by bringing the relevant quantities into the main text and narrowing the claims accordingly, which I appreciate. I still have reservations regarding the scope. As I read the experimental results, it seems that the symmetry lasts until the first feature emerges mainly in LSTMs. At the same task, (3,3,3) at w = 2, it is unbroken at emergence in 28/29 LSTMs, against 8/24 GRUs and 8/23 Elman networks. Can the authors provide a mechanistic explanation of this difference? If it is a limitation of the results, it should be discussed within the main text.

Thank you for reading the revision and for your follow-up. We agree that at (3,3,3) and at w = 2, the symmetry is unbroken at the first feature, mainly in LSTMs. Our result is conditional: where the symmetry was unbroken when the first feature emerged, that feature was the total-count feature in all three architectures (Table 1). The architecture changes how often this condition holds. We can say where this difference comes from, and we have a partial account of when the symmetry breaks. Why the first feature emerges later in the GRU and the Elman network is a limitation, which we now discuss in the main text.

Which feature comes first depends on which of two events comes first: the first feature emerges, or the symmetry breaks (Section 4.4). For the networks of Table 1 at this task:

& & \text{symmetry unbroken} & \text{first feature} & \text{break} \\\\ \text{architecture} & \text{networks} & \text{at the first feature} & \text{(median epoch)} & \text{(median epoch)} \\\\ \hline \text{LSTM} & 29 & 28 & 34 & 80 \\\\ \text{GRU} & 24 & 8 & 65 & 57 \\\\ \text{Elman} & 23 & 8 & 155 & 102.5 \end{array}$$ The break is the median over the networks whose symmetry broke, which is all of them except one Elman network. Across the architectures, the median epoch of the break varies by less than a factor of two, and that of the first feature by nearly five. The quartile ranges of the first feature do not overlap between the architectures, while those of the break overlap (Table 8). The Elman network breaks the symmetry latest of the three, and its first feature emerges later still. The difference therefore lies mostly in when the first feature emerges. This timing also orders Table 7: the later the first feature emerges, the lower R is at that time. How often the condition holds also changes within one architecture. In the GRU, on the same 24 seeds, the symmetry was unbroken at the first feature in 8 networks from the standard initialisation, in 15 when the input weights of the three symbols were tied at initialisation, and in 22 when the training set was also completed by the swapped copies of its strings (Section 4.4, Table 11). In each setting, all of these networks learned the total-count feature first. How often the condition holds therefore depends on how symmetric the start of training is. The architectures differ in this start. At initialisation, the biases are zero, and the weights are small, so the input to each gate is set mainly by the current symbol's own input weights. In the LSTM's input and output gates and in the GRU's update gate, the gradient is multiplied by a quantity computed from those weights, so these blocks receive different updates for different symbols from the first step, and Adam gives these small gradients steps of the same size as the others. The Elman network has no such product, which explains why it breaks the symmetry the latest. The account does not order the LSTM and the GRU. We checked each step on the networks of Table 1 with matched seeds (Appendix B.1). Our present results do not explain why the first feature emerges later in the GRU and the Elman network, which is the larger part of the difference, and we leave this to future work. Section 4.4 now states this limitation. Appendix B.1 gives the timing with quartiles (Table 8), its link to Table 7, and the analysis above. We would welcome your view on whether this addresses your concern.
Reviewer_J1YcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies what kind of equivalent representations useful for a task is selected by the model from training. They then make a connection between task symmetry (relabelling of the symbols that leaves the task unchanged), what kinds of features are first learnt, and what kinds of features are eventually kept. Rigorous theoretical analysis are done in idealized setting.

Critical strengths
  • The problem of which representation sufficient for a task is actually learnt by a trained network is a very important question in the field and, to the best of my knowledge, the authors have come up with an interesting and novel task-based perspective. The setting is also clean enough that the authors can meaningfully separate task requirements from different representations and analyze them directly. The methods used to study them (e.g. character-based Fourier analysis) is also principled and can potentially be applied to other problems.
  • The experiments directly investigate the proposed mechanism. Beyond measuring final accuracy, the paper tracks feature emergence, approximate symmetry breaking, and subsequent retention. Interventions on the alphabet, symbol probabilities, initial input columns, and training-data symmetry provide evidence that the observed feature choices are connected to the presentation’s symmetry. The consistency of retention across many successful runs is also notable.
Critical weaknesses
  • The paper is very hard to read, due to dense formal definitions. Sentences also do not flow into each other, making reading arduous (partly because of the necessary definitions to make things well-defined). To give an example, on page 5 "Cost ranks the classes and, at equal moduli, leaves a tie. The symmetry of the instance limits which classes can rise. The symmetries split the advance symbols into orbits, the sets of symbols they carry into one another, and we count the no-op as an orbit of its own." These are 3 consecutive sentences that support a point but is not tied together. "The symmetry" is also repeated as the main subject for the last 2 sentences, which is extremely abnormal for human writing.
  • I would not describe this work as accessible to general ML audiences and would suggest that the authors cut down on the technical formalities in the main paper, provide much more general intuition before any technical definition/lemmas/theorems
Questions for authors
  • Since the paper deals with a certain feature learning phenomenon, how closely do the randomly initialized networks remain to the tied-input-column subspace during training?
  • To what extent is this built first, kept phenomenon hold? Or do we expect this to hold even on very general, practical tasks?
3 comments

We thank the reviewer for the careful reading and for the concrete example.

We agree that the submitted version was hard to read. To improve the presentation and make the work more accessible to a general ML audience, we revised the paper as follows. We rewrote the abstract, the introduction and the conclusion. Each section now opens by stating what it does, and each definition and result in Section 3 starts with the example in Figure 1(a) and is followed by an explanation in plain words. The group-theory background and two lemmas were moved to the appendix, and our own terms were replaced with standard ones (see Table 2 in Appendix A). The passage quoted on page 5 is rewritten as follows in Sections 3.1 and 3.4 (pages 4 and 6). The revised version has been uploaded.

Q1: How closely do the networks stay to the symmetric subspace?

The tied-input-column subspace is now called the symmetric subspace (Definition 6). We measure closeness with R (equation 4, Section 4.1), the ratio of the part of the input-weight updates that the advance symbols share to the part in which they differ. R is infinite when the updates are equal and 1/2 at chance at (3, 3, 3), and we call the symmetry unbroken when R is at least 3. Following this question, the revision adds Table 7 to Appendix B, which gives R at the checkpoint where the first feature emerged:

& \text{networks} & \text{smallest} & \text{lower quartile} & \text{median} & \text{upper quartile} & \text{largest} \cr \hline \text{unbroken, LSTM} & 99 & 5.92 & 209 & 346 & 559 & 1168 \cr \text{unbroken, GRU} & 12 & 3.43 & 11.5 & 34.1 & 76.2 & 273 \cr \text{unbroken, Elman} & 8 & 3.25 & 4.75 & 5.64 & 12.4 & 31.7 \cr \text{unbroken, all} & 119 & 3.25 & 155 & 294 & 525 & 1168 \cr \hline \text{broken before it} & 53 & 0.61 & 1.18 & 1.45 & 1.87 & 2.99 \end{array}$$ In the 119 networks whose symmetry was unbroken, the median of R is 294, so the updates of the advance symbols differ from their mean by about 6% of the mean’s length. The LSTMs are closest, and the GRUs and the Elman networks are closer to the threshold. This closeness is enough for the prediction. At thresholds 5 and 10, all networks with unbroken symmetry still learned the total-count feature first (Table 6). Setting the input weights of the advance symbols equal at initialisation raises the GRUs with unbroken symmetry from 8 to 15 of 24 (Table 10). In short, networks trained from small random weights stay close to the symmetric subspace until their first feature emerges, most closely in the LSTM, and this is close enough for the prediction to hold. Q2: To what extent does the phenomenon hold? The phenomenon has two parts, and our evidence for them differs. The symmetry constrains the first feature. This part is proved (Lemma 2), and the proof needs only a symmetry of the task and separate input weights for each symbol. It therefore also holds for a transformer or a state-space model that reads its input through an embedding (Section 3.3). The proof applies while the symmetry lasts, and how long the symmetry lasts depends on the architecture, as the table above shows. We have not tested this part on realistic tasks. Since the theory does not rely on the recurrent architecture, we expect this part to be general wherever the data contain exchangeable tokens, such as the names of entities in a story. Natural data are rarely exactly symmetric, however, so the symmetry may break early. Testing this in larger models is a direction for our future work. The first feature is kept. This part builds on the first: because the task can be solved by several sets of features, a network can complete a solution around the first feature it learned. Whether it does so is a question about the learning dynamics after the symmetry breaks, and the answer does not follow from the symmetry alone. In our tasks, the networks almost always kept the first feature (Section 5.1), and an idealised model explains why. So far, we have established this only in our tasks, and whether it holds for other tasks and models remains open. Taken together, we expect the first part to hold in any model with an input embedding, wherever the data contain exchangeable tokens. The second part, and with it the effect of the symmetry on the trained network’s representation, has so far been established only in our tasks.

I thank the authors for the rebuttal and appreciate the revision. While the texts are mostly understandable now, the writing is still rather 'peculiar', for instance the second sentence of "We then prove that, until a network distinguishes the counters, it can learn only a feature that counts the total number of advances, as if there were a single counter. This is because gradient flow respects the symmetry of the task." Since peculiar writing is not ground for rejection, I am keeping my score.

Thank you for reading the revision and for the concrete example. We have rewritten that sentence and revised similar ones throughout the paper in the updated PDF.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.