- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper addresses an interesting question: how does a model select one set of features out of the many possible options, and whether the initial selection is kept or replaced. The authors focus on a specific, somewhat narrow setting: a word problem of a finite abelian group, and studies 3 RNNs for the task. Several theoretical and empirical results are then presented, in particular that models select their feature set early on and tend to keep them.
Critical strengths
The question the authors focus on is of interest. It is also addressed rigorously.
Critical weaknesses
I would say upfront that this paper is far from my research expertise. Having said that, I found the paper quite hard to follow. The structure was hard to decipher; I could not tell how the different parts connected, what was the role of each part, and how they all came together to address the paper's main research questions. I would advise the authors to add introductory paragraphs or sentences to each of the sections/sub-sections that map the current part in context, and a summarizing text outlying what we found out, how it matters, and how it relates to the next part. Another example is Figure 1. It took me some time to understand what 1(a) is showing, and I still don't understand what I'm seeing in 1(b). Again, this could be my lack of expertise speaking, but some discussion of the meaning of the graph (e.g., is higher or low better?), what does it tell us/teach us would be helpful. And finally - even the conclusion is a list of statements that make no coherent story.
Somewhat related, an important question that authors do not address is what can we learn from these experiments about more realistic problems? Do we have reason to believe these findings transition to real models? Are there ideas on how to test this?
Questions for authors
N/A
3 comments
Thank you for the detailed response. Unfortunately the changes made would require a new, complete review process, and are out of scope for a rebuttal. I hope these changes will make the paper easier to digest for the next set of reviewers.
Thank you for reading our response. For the record: the theorem, the symmetry lemma and the runs behind the main results are those of the submission, and the revision mainly changes how they are presented. The additions were requested in the reviews: one corollary (Section 3.3), a weight-decay experiment (Section 5.1) and counts moved from the appendix into the main text. We understand your position, and we thank you for the comments on presentation, which shaped the revision.
We thank the reviewer for the careful reading and for the concrete suggestions on the structure, Figure 1 and the conclusion. We agree that the submitted version was hard to follow, and we have revised its presentation throughout. The revised version has been uploaded.
The main changes are:
Sections 2 to 5 now open by stating what each section does and how it connects to the others, and Sections 4 and 5 end with a summary. Each section and subsection title now states what that part shows.
The abstract, introduction, and conclusion have been rewritten. The conclusion now follows one line of argument: the question, what we found, and what it implies.
Section 3 now starts each step from the example of Figure 1(a) and states in plain words what each result means. Two lemmas have been moved to the appendix, and our own terms have been replaced with standard ones (see Table 2 in Appendix A).
Figure 1(b) now shows the energy of four main features, in place of the shares of all features and a strip for the symmetry measure. Its three marks are labelled, and the caption explains how to read the curves.
Figure 1(b) summarises the entire paper in a single network, so we also explain it here: the panel shows the story of a single network during training. Each curve shows how much of the hidden state follows one feature, so a higher curve means more of that feature, and neither height is better. At first, the network’s input weights for a, b, and c stay nearly equal, so it cannot tell the three symbols apart. It can then learn only the total-count feature, and at epoch 23 this curve rises first (i). The total count alone cannot solve the task, so at epoch 53 the weights move apart and the symmetry breaks (ii). The network then learns the features of counters 2 and 3 and retains the total-count feature. These three together determine counter 1, so counter 1’s curve stays near zero to the end, and we say that this counter is left out (iii). Section 1 (page 2) walks through the same story, and Sections 4 and 5 show how often each step occurs across all networks.
On realistic problems. We have not tested our findings beyond our tasks, but we can explain why we expect some of them to carry over and how to check.
The mechanism uses nothing specific to recurrent networks. When the data treat two tokens identically, gradient descent moves their embeddings in the same direction, and only the small random differences at initialisation tell them apart. For the same reason, two hidden units with identical initial weights stay identical. Realistic tasks may contain such tokens: the names of entities in a story or variables in a program can be exchanged without changing the answer. We therefore expect any model trained from small random weights to learn first the features that treat such tokens alike, for example, how many events have happened, before features that tell the entities apart. A practical consequence for interpretability is that a trained model may hold such a pooled feature alongside, or in place of, some per-entity features, as our networks do.
Our conclusions may not apply in every setting, for two reasons. First, natural data are rarely exactly symmetric, so in a realistic task the symmetry may break much earlier than in ours. Second, our evidence that the first feature is kept comes only from our tasks, and our explanation of it holds in an idealised model.
To test this in larger models is a direction for future work. For any task with interchangeable tokens, one can track how far their embeddings move apart during training, as our measure R does, and record which features remain close while they are still close.
We hope that the revised paper is now easier to follow, and we would be glad to answer any further questions.