- Rating
- 1(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors define a family of data-generating mechanisms for token sequences, that are supposed to imitate (and motivated by) structures in natural language.
They then train small transformers on this data and analyze how well/easily the data is learned, under which conditions, and look at properties of the models with mechanistic interpretability tools.
Critical strengths
(S1) The topic of characterizing and understanding the learning and generalization properties of transformer LMs is a topic of broad interest.
(S2) The data used is generated with simple formulae, so it is controlled, and speciffic aspects can be varied (e.g. controllable recurrence order ).
(S3) The analysis is quite thorough (variations in the data, in model depth and width, use of mech. interp. tools). And some of the findings are novel and non-obvious, as far as I can tell (in particular those with curriculum learning).
Critical weaknesses
(W1) I'm not sure how interesting the proposed family of data-generating mechanisms is. In the introduction, it is briefly claimed to be motivated by the structures of natural language, but there is no elaboration (I have no background in linguistics so I can't readily evaluate how relevant this proposed data is to natural language).
The most essential definitions of the paper ("linear recurrence") in Eq. (1) and (2) are not very rigorous. is not defined. I don't understand the notations even after some headscratching. Are and scalars? What is ? And ? Is the whole paper just about modular arithmetic?
My core concern is that the paper claims to study how transformers learn and generalize on a generic family of data (relevant to natural language, cf. the abstract) while it is, like papers in the literature on mechanistic interpretability, focusing entirely on modular arithmetic. As far as I can tell, this setting has limited connections to other practical tasks and data that transformers are applied to in practice.
As a constructive suggestion, tf this is indeed a fair concern with the current version of the paper, one improvement that might broaden its appeal would be to extend the experiments with other data-generating processes, which could bridge some of the gap with realistic tasks, and show that some findings from the current experiments still hold (this would involve a substantial revision of the paper though, not just a promise to extend its final version after acceptance).
(W1b) As a possible example of the limited applicability of the chosen data and setting to realistic data like language: most experiments are conducted on order =2 and 3. Experiments on order 4 already show that the models struggle. Isn't this quite different from realistic data like natural language? It seems that handling natural language like transformer LMs do is quite beyond predictions from 2- and 3-gram(/tokens) Let me know if I am mistaken.
(W2) Experimental setup: I understand from the description L146 that the model is provided with tokens (2 in the main experiments with data of order 2) and then asked to predict autoregressively -2 tokens. I looked throughout the paper and appendix but cannot find the value of . This setting seems odd to me, since any mispredicted token will propagate for the rest of the autoregressively-generated sequence up to . I would have thought that the setting would be instead a prediction of every token in a length- sequence with causal masking and teacher forcing.
(W3) "Shortcut" setting. I understand that the original and "shortcut" rules are exactly identical, in the sense that they generate the exact same data? If this is correct, I think that this differs from the typical cases in the literature on "shortcut learning", where the data typically contains a mixture of examples that each supports either of several hyptotheses, and one of the hypotheses of preferred by the model because of its iinductive biases. Here, if the whole data is perfectly described by the so-called "shortcut", there is truly no reason for any learner to hypothesize a higher-order rule than necessary. Let me know if I am mistaken!
(W4) Significance of the findings. Overall it's not clear what this paper contributes to. From the introduction, I cannot tell that this paper is addressing open research questions (understanding learnability and generalization is interesting, but what exact research question is this adressing? This is tied to W1). And from the conclusions, I also cannot tell what the observations/analysis mean for the broader field (beyond modular arithmetic).
To be more precise, here are the stated 3 main observations:
- "First, the generalization capability is not determined by model size alone: rule organization, algebraic structure, masking, and training procedure": this is hardly a new finding. It essentially means that generalization depends on both the model and the data.
- "successful prediction does not imply that the model learns the intended computation, as models may instead use algebraic shortcuts": this is again the topic of a whole existing line of research.
- "depth affects both learning and representation": ditto.
(Relegating the discussion of the related work to the appendix is not an issue in itself, but it did not help given the above issues.)
Detailed or minor comments
-
L46 (please number all equations!): the symbol is not defined. So I don't understand what or are. Is just a product of two scalars? Why is in parentheses then?
-
The names "inter-trajectory" and "intra-trajectory" seem unncessarily fancy and cryptic. I'm not sure what a "trajectory" refers to here. Why not just call them just something like "single-rule sequences" and "multiple-rule sequences"? At least the reader can directly understand what they mean.
-
A figure illustrating these settings and the format of the data would be helpful to make things concrete and quick to grasp by the reader (maybe I could have undersood what "trajectory" means then).
-
Typo: "Revealing underlying mechanism" (need plural "mechanisms" I guess).
-
In the title, I'm not sure what "multi-task recurrence" means. It might seem like a minor point, but it's tightly related to the main weakness in that I couldn't initially tell what setting the paper was addressing. The title should help set the right expectations of the reader for what's to be found in the rest of the paper.
Questions for authors
No need to address every weakness above.
The critical ones are W1/1b/4, and things I may have misunderstood.
If W1 and/or W4 are fair concerns with the paper, fixing either of them would require a substantial rewrite of the motivation, analysis, and connection to the existing literature (W1/4), and possibly extension of the experments to realistic srttings (W1). This goes beyond what can be done as a promise of updating the final version of the paper.
4 comments
# Response to Reviewer SnZM(2/3): Regarding motivation
Regarding W1: Motivation, relation with natural language.
I'm not sure how interesting the proposed family of data-generating mechanisms is. In the introduction, it is briefly claimed to be motivated by the structures of natural language, but there is no elaboration (I have no background in linguistics so I can't readily evaluate how relevant this proposed data is to natural language).
....
My core concern is that the paper claims to study how transformers learn and generalize on a generic family of data relevant to natural language, cf\. the abstract while it is, like papers in the literature on mechanistic interpretability, focusing entirely on modular arithmetic. As far as I can tell, this setting has limited connections to other practical tasks and data that transformers are applied to in practice.
We believe there are strong correlations between multi-task recurrence and some natural language tasks.
We want to provide this example as an evidence of the correlation: the grammar in "we have to insist on paying for the meal", "have to" licenses the base-form verb, "insist on" licenses the -ing form verb; and "pay for" introduces a nominal complement. Each newly generated element satisfies rules of the preceding context while adding another rule to subsequent generation.
Modular arithmetic provides a finite vocabulary, exact targets, and analytically testable alternative strategies. We start with simple linear recurrence as a starting point to analyze the mechanism behind them. Appendix F then extends the experiments to nonlinear rules.
As a constructive suggestion, tf this is indeed a fair concern with the current version of the paper, one improvement that might broaden its appeal would be to extend the experiments with other data-generating processes, which could bridge some of the gap with realistic tasks, and show that some findings from the current experiments still hold (this would involve a substantial revision of the paper though, not just a promise to extend its final version after acceptance).
Thank you for your constructive suggestion. In this work we concern the recurrence tasks on modular arithmetics. Our scaling results and mechanism analysis are based on the certainty and controllability permitted by modular arithmetics. Therefore establising a meaningful connection to realistic tasks is beyond the scope of the present study, but could be considered in the future work.
Regarding W1b: Models struggle in order=4 experiments.
As a possible example of the limited applicability of the chosen data and setting to realistic data like language: most experiments are conducted on order =2 and 3. Experiments on order 4 already show that the models struggle. Isn't this quite different from realistic data like natural language? It seems that handling natural language like transformer LMs do is quite beyond predictions from 2- and 3-gram Let me know if I am mistaken.
The fact that models struggle in these experiments does not result from the context limit. In contrast, for , we test 4 groups of order-4 rules , models with two layers and width-1024 can exceed 99% fresh-test accuracy. So the struggle of p=127 cannot be explained as "context limit". The conclusion we would like to draw is that "order 4 is significantly harder than order 3, fixing model size and p".
# Response to Reviewer SnZM(3/3): Regarding other concerns.
Regarding W2: Sequence length and autoregressive evaluation.
Experimental setup: I understand from the description L146 that the model is provided with tokens (2 in the main experiments with data of order 2) and then asked to predict autoregressively tokens. I looked throughout the paper and appendix but cannot find the value of . This setting seems odd to me, since any mispredicted token will propagate for the rest of the autoregressively-generated sequence up to . I would have thought that the setting would be instead a prediction of every token in a length- sequence with causal masking and teacher forcing.
Thanks for your help comment, we have added the specific value of in the revised main text. should be for unmasked experiments, for masked ones $$T.
We hope to point out the misunderstandings respectfully. Training and the reported token-level learning curves already use causal masking and teacher forcing. Earlier predictions are not fed back into subsequent inputs.
Regarding W3: Algebraic shortcuts.
"Shortcut" setting. I understand that the original and "shortcut" rules are exactly identical, in the sense that they generate the exact same data?
...
Here, if the whole data is perfectly described by the so-called "shortcut", there is truly no reason for any learner to hypothesize a higher-order rule than necessary.
The "shortcut" we mention is that model uses fewer operands but requires a longer history, so it is not identical to the original rule at every supervised position. The recurrence implies , but the latter cannot predict , which is included in the loss. In this sense 3-order is necessary.
Our understanding is that the model has two competing preferences: prefers "shortcut" because it involves less operands. prefers "original" because it decreases the loss more. We have discussed in Sec 3.3 Figure 4 that whether or dominant the learning process is related to its delay.
Regarding W4: Significance of the findings.
Overall it's not clear what this paper contributes to. From the introduction, I cannot tell that this paper is addressing open research questions (understanding learnability and generalization is interesting, but what exact research question is this adressing? This is tied to W1). And from the conclusions, I also cannot tell what the observations/analysis mean for the broader field (beyond modular arithmetic).
We would like to emphasize the significance of our work in these aspects.
-
Our concern is beyond modular arithmatic. As the example in the "motivation" part above shows, multi-rule recurrence tasks have close connections with natural language.
-
Our research question is to understand when and how Transformers learn multi-task recurrence. Our empirical results could specify whether using extra layers, extra width, or turning to curriculum learning improves the performance on different types of tasks. Our conclusions also reveal some mechanisms on how Transformers work in related language scenarios.
Regarding other minor points
Typo: "Revealing underlying mechanism" .
Thanks for correction.
In the title, I'm not sure what "multi-task recurrence" means. It might seem like a minor point, but it's tightly related to the main weakness in that I couldn't initially tell what setting the paper was addressing. The title should help set the right expectations of the reader for what's to be found in the rest of the paper.
We hope our response to W1 could explain that multi-task recurrence is the core experimental setting in our work.
Reviewer's response
Thanks to the authors for the lengthy response. It essentially confirmed my initial concerns with the paper, hence I have updated my rating to a recommendation for strong rejection.
I reiterate the main issues, in short.
-
Motivation, context, and defintion of the research questions not clear; the stated connections with the processing of language are tenuous and hand-wavy and do not match the toy setting actually covered in the paper.
-
The significance and implications of the findings are vague, so the study comes short of providing practically-useful results, or other new findings that future research could build upon. The authors acknowledged that "the connection to realistic tasks is beyond the scope of the present study" (it should not be, I think it is essential to establish/verify some connections with realistic settings).
-
General lack of clarity and rigour in the presentation/writing.
I appreciate the efforts in the review to patch things up by providing missing definitions and motivating elements, but the paper is still below the bar for publication at the main conference (as mentioned in the original review). These corrections can make it a decent workshop paper about the study of a toy setting. I already provided constructive suggestions in the original review to build up future work on this paper, which I hope will be helpful.
# Response to Reviewer SnZM(1/3): Regarding definitions and notations.
Thank you for your helpful comments.
Regarding W1: Definitions and notations.
Thanks for your suggestion, we have revised our definitions accordingly. qi∈{0,1,…,p−1} are the values in the sequence, the subscripts i's are their indices. {F}p={Z}/p{Z} is the finite field with p elements. a=�5� are vectors representing rules. In linear recurrence, q_t=f_a�6�=\sum_\{i=1\}^\{k\}a_iq_\{t-i\} \pmod p\. Since rules may change by step t, so we add inner subscript t when necessary$(a_t) = ((a_t)_1,\dots,(a_t)_k)$. The parentheses are used to distinguish inner and outer subscripts.
The topic of our paper is recurrence tasks. "Modular" is a simple tool to keep vocabulary finite.
Thanks for your suggestions, but these are two multiple rule settings. We use "intra" to prompt that "rules may change within one sequence", and use "inter" to show "rules persist in one sequence, but can be different from another sequence". If we repeatedly apply the recurrence, the state will return to its initial value, which forms a circle, or trajectory. Changing rules in the sequence is like jumping through different trajectories.
Thanks for the suggestion. We have added Figure 1 to illustrate our settings.