acceptodds
Under review as a conference paper at ICLR 2027

Capability and Interpretability of Transformers on Multi-task Recurrence

Abstract

Transformers are widely used in various language tasks, with limited understanding of their internal mechanisms. Motivated by the recurrent structure of many linguistic rules, we study the capability and interpretability of Transformers through a controlled family of linear recurrence tasks over finite cyclic groups. These tasks range from executing a single fixed rule to identifying a latent rule from context and switching between explicitly specified rules, with and without input masking. Through extensive experiments across different configurations, we find that switching specified rules is simpler than identifying latent rules for Transformers in general. We also identify how several key factors affect the generalization capability of Transformers, including the depth, width, and the algebraic structure of the linear recurrence. Curriculum training is extremely useful when the number of rules is large. With standard tools in mechanistic interpretability like linear probing and input patching, our study reveals how Transformers represent, identify, and execute multiple rules.

Primary area:
interpretability and explainable AI
Keywords:
Transformer, linear recursion, interpretability, grokking

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SnZMOpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors define a family of data-generating mechanisms for token sequences, that are supposed to imitate (and motivated by) structures in natural language.

They then train small transformers on this data and analyze how well/easily the data is learned, under which conditions, and look at properties of the models with mechanistic interpretability tools.

Critical strengths

(S1) The topic of characterizing and understanding the learning and generalization properties of transformer LMs is a topic of broad interest.

(S2) The data used is generated with simple formulae, so it is controlled, and speciffic aspects can be varied (e.g. controllable recurrence order kk).

(S3) The analysis is quite thorough (variations in the data, in model depth and width, use of mech. interp. tools). And some of the findings are novel and non-obvious, as far as I can tell (in particular those with curriculum learning).

Critical weaknesses

(W1) I'm not sure how interesting the proposed family of data-generating mechanisms is. In the introduction, it is briefly claimed to be motivated by the structures of natural language, but there is no elaboration (I have no background in linguistics so I can't readily evaluate how relevant this proposed data is to natural language).

The most essential definitions of the paper ("linear recurrence") in Eq. (1) and (2) are not very rigorous. ff is not defined. I don't understand the notations even after some headscratching. Are aa and qq scalars? What is Z/pZ\mathbb{Z}/p\mathbb{Z}? And F\mathbb{F}? Is the whole paper just about modular arithmetic?

My core concern is that the paper claims to study how transformers learn and generalize on a generic family of data (relevant to natural language, cf. the abstract) while it is, like papers in the literature on mechanistic interpretability, focusing entirely on modular arithmetic. As far as I can tell, this setting has limited connections to other practical tasks and data that transformers are applied to in practice.

As a constructive suggestion, tf this is indeed a fair concern with the current version of the paper, one improvement that might broaden its appeal would be to extend the experiments with other data-generating processes, which could bridge some of the gap with realistic tasks, and show that some findings from the current experiments still hold (this would involve a substantial revision of the paper though, not just a promise to extend its final version after acceptance).


(W1b) As a possible example of the limited applicability of the chosen data and setting to realistic data like language: most experiments are conducted on order kk=2 and 3. Experiments on order 4 already show that the models struggle. Isn't this quite different from realistic data like natural language? It seems that handling natural language like transformer LMs do is quite beyond predictions from 2- and 3-gram(/tokens) Let me know if I am mistaken.


(W2) Experimental setup: I understand from the description L146 that the model is provided with kk tokens (2 in the main experiments with data of order 2) and then asked to predict autoregressively TT-2 tokens. I looked throughout the paper and appendix but cannot find the value of TT. This setting seems odd to me, since any mispredicted token will propagate for the rest of the autoregressively-generated sequence up to TT. I would have thought that the setting would be instead a prediction of every token in a length-TT sequence with causal masking and teacher forcing.


(W3) "Shortcut" setting. I understand that the original and "shortcut" rules are exactly identical, in the sense that they generate the exact same data? If this is correct, I think that this differs from the typical cases in the literature on "shortcut learning", where the data typically contains a mixture of examples that each supports either of several hyptotheses, and one of the hypotheses of preferred by the model because of its iinductive biases. Here, if the whole data is perfectly described by the so-called "shortcut", there is truly no reason for any learner to hypothesize a higher-order rule than necessary. Let me know if I am mistaken!


(W4) Significance of the findings. Overall it's not clear what this paper contributes to. From the introduction, I cannot tell that this paper is addressing open research questions (understanding learnability and generalization is interesting, but what exact research question is this adressing? This is tied to W1). And from the conclusions, I also cannot tell what the observations/analysis mean for the broader field (beyond modular arithmetic).

To be more precise, here are the stated 3 main observations:

  • "First, the generalization capability is not determined by model size alone: rule organization, algebraic structure, masking, and training procedure": this is hardly a new finding. It essentially means that generalization depends on both the model and the data.
  • "successful prediction does not imply that the model learns the intended computation, as models may instead use algebraic shortcuts": this is again the topic of a whole existing line of research.
  • "depth affects both learning and representation": ditto.

(Relegating the discussion of the related work to the appendix is not an issue in itself, but it did not help given the above issues.)


Detailed or minor comments

  • L46 (please number all equations!): the symbol F\mathrm{F} is not defined. So I don't understand what aa or (a)q(a)q are. Is (a)q(a)q just a product of two scalars? Why is aa in parentheses then?

  • The names "inter-trajectory" and "intra-trajectory" seem unncessarily fancy and cryptic. I'm not sure what a "trajectory" refers to here. Why not just call them just something like "single-rule sequences" and "multiple-rule sequences"? At least the reader can directly understand what they mean.

  • A figure illustrating these settings and the format of the data would be helpful to make things concrete and quick to grasp by the reader (maybe I could have undersood what "trajectory" means then).

  • Typo: "Revealing underlying mechanism" (need plural "mechanisms" I guess).

  • In the title, I'm not sure what "multi-task recurrence" means. It might seem like a minor point, but it's tightly related to the main weakness in that I couldn't initially tell what setting the paper was addressing. The title should help set the right expectations of the reader for what's to be found in the rest of the paper.

Questions for authors

No need to address every weakness above.

The critical ones are W1/1b/4, and things I may have misunderstood.

If W1 and/or W4 are fair concerns with the paper, fixing either of them would require a substantial rewrite of the motivation, analysis, and connection to the existing literature (W1/4), and possibly extension of the experments to realistic srttings (W1). This goes beyond what can be done as a promise of updating the final version of the paper.

4 comments
AuthorsOpenReview ↗

# Response to Reviewer SnZM(1/3): Regarding definitions and notations.

Thank you for your helpful comments.

Regarding W1: Definitions and notations.

The most essential definitions of the paper ("linear recurrence") in Eq. 11 and 22 are not very rigorous. ff is not defined. I don't understand the notations even after some headscratching. Are aa and qq scalars? What is {Z}/p{Z}\mathbb\{Z\}/p\mathbb\{Z\}? And {F}\mathbb\{F\}? Is the whole paper just about modular arithmetic?

L46 pleasenumberallequations ⁣please number all equations\!: the symbol FF is not defined. So I don't understand what aa or a�3�a�3� are. Is �4�q�4�q just a product of two scalars? Why is aa in parentheses then?

Thanks for your suggestion, we have revised our definitions accordingly. qi∈{0,1,…,p−1}q_i \in \lbrace 0,1,\dots,p-1 \rbrace are the values in the sequence, the subscripts ii's are their indices. {F}p={Z}/p{Z}\mathbb\{F\}_p=\mathbb\{Z\}/p\mathbb\{Z\} is the finite field with p elements. a=�5�a=�5� are vectors representing rules. In linear recurrence, q_t=f_a�6�=\sum_\{i=1\}^\{k\}a_iq_\{t-i\} \pmod p\. Since rules may change by step tt, so we add inner subscript tt when necessary$(a_t) = ((a_t)_1,\dots,(a_t)_k)$. The parentheses are used to distinguish inner and outer subscripts.

The topic of our paper is recurrence tasks. "Modular" is a simple tool to keep vocabulary finite.

The names "inter-trajectory" and "intra-trajectory" seem unncessarily fancy and cryptic. I'm not sure what a "trajectory" refers to here. Why not just call them just something like "single-rule sequences" and "multiple-rule sequences"? At least the reader can directly understand what they mean.

Thanks for your suggestions, but these are two multiple rule settings. We use "intra" to prompt that "rules may change within one sequence", and use "inter" to show "rules persist in one sequence, but can be different from another sequence". If we repeatedly apply the recurrence, the state will return to its initial value, which forms a circle, or trajectory. Changing rules in the sequence is like jumping through different trajectories.

  • A figure illustrating these settings and the format of the data would be helpful to make things concrete and quick to grasp by the reader.

Thanks for the suggestion. We have added Figure 1 to illustrate our settings.

AuthorsOpenReview ↗

# Response to Reviewer SnZM(2/3): Regarding motivation

Regarding W1: Motivation, relation with natural language.

W1W1 I'm not sure how interesting the proposed family of data-generating mechanisms is. In the introduction, it is briefly claimed to be motivated by the structures of natural language, but there is no elaboration (I have no background in linguistics so I can't readily evaluate how relevant this proposed data is to natural language).

....

My core concern is that the paper claims to study how transformers learn and generalize on a generic family of data relevant to natural language, cf\. the abstract while it is, like papers in the literature on mechanistic interpretability, focusing entirely on modular arithmetic. As far as I can tell, this setting has limited connections to other practical tasks and data that transformers are applied to in practice.

We believe there are strong correlations between multi-task recurrence and some natural language tasks.

We want to provide this example as an evidence of the correlation: the grammar in "we have to insist on paying for the meal", "have to" licenses the base-form verb, "insist on" licenses the -ing form verb; and "pay for" introduces a nominal complement. Each newly generated element satisfies rules of the preceding context while adding another rule to subsequent generation.

Modular arithmetic provides a finite vocabulary, exact targets, and analytically testable alternative strategies. We start with simple linear recurrence as a starting point to analyze the mechanism behind them. Appendix F then extends the experiments to nonlinear rules.

As a constructive suggestion, tf this is indeed a fair concern with the current version of the paper, one improvement that might broaden its appeal would be to extend the experiments with other data-generating processes, which could bridge some of the gap with realistic tasks, and show that some findings from the current experiments still hold (this would involve a substantial revision of the paper though, not just a promise to extend its final version after acceptance).

Thank you for your constructive suggestion. In this work we concern the recurrence tasks on modular arithmetics. Our scaling results and mechanism analysis are based on the certainty and controllability permitted by modular arithmetics. Therefore establising a meaningful connection to realistic tasks is beyond the scope of the present study, but could be considered in the future work.

Regarding W1b: Models struggle in order=4 experiments.

W1bW1b As a possible example of the limited applicability of the chosen data and setting to realistic data like language: most experiments are conducted on order =2 and 3. Experiments on order 4 already show that the models struggle. Isn't this quite different from realistic data like natural language? It seems that handling natural language like transformer LMs do is quite beyond predictions from 2- and 3-gram/tokens/tokens Let me know if I am mistaken.

The fact that models struggle in these experiments does not result from the context limit. In contrast, for p=31p=31, we test 4 groups of order-4 rules whosecoefficientsareverifiedtobeshortcut−freewithinlagT=16whose coefficients are verified to be shortcut-free within lag T=16, models with two layers and width-1024 can exceed 99% fresh-test accuracy. So the struggle of p=127 cannot be explained as "context limit". The conclusion we would like to draw is that "order 4 is significantly harder than order 3, fixing model size and p".

AuthorsOpenReview ↗

# Response to Reviewer SnZM(3/3): Regarding other concerns.

Regarding W2: Sequence length and autoregressive evaluation.

W2W2 Experimental setup: I understand from the description L146 that the model is provided with kk tokens (2 in the main experiments with data of order 2) and then asked to predict autoregressively T−2T-2 tokens. I looked throughout the paper and appendix but cannot find the value of TT. This setting seems odd to me, since any mispredicted token will propagate for the rest of the autoregressively-generated sequence up to TT. I would have thought that the setting would be instead a prediction of every token in a length-TT sequence with causal masking and teacher forcing.

Thanks for your help comment, we have added the specific value of TT in the revised main text. TT should be T=16T=16 for unmasked experiments, T=64T=64 for masked ones $$Tisthenumberofvalues,soruletagsinintra−doesnotcount is the number of values, so rule tags in intra- does not count.

We hope to point out the misunderstandings respectfully. Training and the reported token-level learning curves already use causal masking and teacher forcing. Earlier predictions are not fed back into subsequent inputs.

Regarding W3: Algebraic shortcuts.

"Shortcut" setting. I understand that the original and "shortcut" rules are exactly identical, in the sense that they generate the exact same data?

...

Here, if the whole data is perfectly described by the so-called "shortcut", there is truly no reason for any learner to hypothesize a higher-order rule than necessary.

The "shortcut" we mention is that model uses fewer operands but requires a longer history, so it is not identical to the original rule at every supervised position. The recurrence qt=q{t−1}+q{t−2}+q{t−3}q_t=q_\{t-1\}+q_\{t-2\}+q_\{t-3\} implies qt=2q{t−1}−q{t−4}q_t=2q_\{t-1\}-q_\{t-4\}, but the latter cannot predict q4q_4, which is included in the loss. In this sense 3-order is necessary.

Our understanding is that the model has two competing preferences: aa prefers "shortcut" because it involves less operands. bb prefers "original" because it decreases the loss more. We have discussed in Sec 3.3 Figure 4 that whether aa or bb dominant the learning process is related to its delay.

Regarding W4: Significance of the findings.

Overall it's not clear what this paper contributes to. From the introduction, I cannot tell that this paper is addressing open research questions (understanding learnability and generalization is interesting, but what exact research question is this adressing? This is tied to W1). And from the conclusions, I also cannot tell what the observations/analysis mean for the broader field (beyond modular arithmetic).

We would like to emphasize the significance of our work in these aspects.

  1. Our concern is beyond modular arithmatic. As the example in the "motivation" part above shows, multi-rule recurrence tasks have close connections with natural language.

  2. Our research question is to understand when and how Transformers learn multi-task recurrence. Our empirical results could specify whether using extra layers, extra width, or turning to curriculum learning improves the performance on different types of tasks. Our conclusions also reveal some mechanisms on how Transformers work in related language scenarios.

Regarding other minor points

  • Typo: "Revealing underlying mechanism" needplural"∗mechanisms∗"Iguessneed plural "*mechanisms*" I guess.

Thanks for correction.

  • In the title, I'm not sure what "multi-task recurrence" means. It might seem like a minor point, but it's tightly related to the main weakness in that I couldn't initially tell what setting the paper was addressing. The title should help set the right expectations of the reader for what's to be found in the rest of the paper.

We hope our response to W1 could explain that multi-task recurrence is the core experimental setting in our work.

Reviewer_SnZMOpenReview ↗

Reviewer's response

Thanks to the authors for the lengthy response. It essentially confirmed my initial concerns with the paper, hence I have updated my rating to a recommendation for strong rejection.


I reiterate the main issues, in short.

  • Motivation, context, and defintion of the research questions not clear; the stated connections with the processing of language are tenuous and hand-wavy and do not match the toy setting actually covered in the paper.

  • The significance and implications of the findings are vague, so the study comes short of providing practically-useful results, or other new findings that future research could build upon. The authors acknowledged that "the connection to realistic tasks is beyond the scope of the present study" (it should not be, I think it is essential to establish/verify some connections with realistic settings).

  • General lack of clarity and rigour in the presentation/writing.


I appreciate the efforts in the review to patch things up by providing missing definitions and motivating elements, but the paper is still below the bar for publication at the main conference (as mentioned in the original review). These corrections can make it a decent workshop paper about the study of a toy setting. I already provided constructive suggestions in the original review to build up future work on this paper, which I hope will be helpful.

Reviewer_HXuTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how Transformers learn and execute multiple linear recurrence rules over finite fields. It compares fixed-rule prediction, latent-rule inference across trajectories, and explicitly tagged rule switching within trajectories, with additional experiments involving masking and higher-order recurrences. The authors investigate depth, width, curriculum learning, and algebraic shortcuts, using linear probes and controlled input interventions to characterize learned behavior. An explicit two-layer construction establishes representational sufficiency for unmasked latent-rule prediction under specified architectural assumptions.

Critical strengths
  1. A useful controlled task family. The recurrence setting allows systematic variation of rule count, missing observations, and algebraic structure. It extends earlier multi-task modular-arithmetic studies, such as He et al. (2024), by coupling successive predictions through temporal dynamics.

  2. Interesting evidence of algebraic shortcut learning. The distinction between the intended recurrence and equivalent sparse formulas is valuable. The loss-reweighting experiments provide an actionable demonstration that training objectives can influence which solution is learned.

  3. Interventions strengthen the behavioral analysis. Randomizing historical values helps distinguish formulas that are indistinguishable on valid trajectories. The paper appropriately treats attention patterns as candidate evidence and supports its conclusions with input interventions.

  4. Theoretical support for representational capacity. The finite-softmax construction provides a concrete sufficient architecture and distinguishes unique continuation from unique rule identification.

Critical weaknesses
  1. The comparison between inter- and intra-trajectory learning has important confounds. These settings differ simultaneously in rule observability, rule-switching structure, sequence format, and data construction. The intra-trajectory setting supplies the information that the inter-trajectory setting must infer, so its advantage does not isolate the effect of switching versus identifying rules. Controlled tagged and untagged variants, with matched training examples and optimizer steps, would support a sharper conclusion.

  2. The trajectory split needs a more explicit overlap audit. The main text states that test length-kk subsequences never occur in training, while Appendix B.1 permits splitting one orbit into contiguous portions. With sliding-window examples, this requires an explicit treatment of windows crossing the boundary. The paper should describe boundary purging and report actual overlap counts. It should also clarify how the same guarantees apply to the randomly sampled higher-order settings.

  3. The mechanistic conclusions exceed what the probes establish. Linear decodability demonstrates that rule information is accessible at a particular activation site, but does not establish that the model uses that representation to produce its answer. Appendix G.2 also draws probe examples from the full state space without excluding Transformer training examples or enforcing trajectory-disjoint probe splits, and reports no repeated-seed uncertainty. The input interventions establish useful functional properties; internal activation patching or targeted ablations would strengthen claims about rule identification and execution mechanisms.

  4. Capacity and optimization effects are not sufficiently separated. Several comparisons use different training horizons, and many capability boundaries lack systematic uncertainty estimates across seeds and rule pools. Epoch counts also become difficult to compare when dataset sizes change. Furthermore, the theoretical construction omits normalization and uses an independent output matrix, whereas the experimental models use LayerNorm and tied embeddings. The theorem provides an expressivity result, but its connection to the observed learning thresholds requires further explanation.

Questions for authors

See weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer HXuT

Response to Reviewer HXuT

Thank you for your suggestive comments.

(Weakness 1) Regarding confounds in the inter/intra comparison:

The comparison between inter- and intra-trajectory learning has important confounds. ... its advantage does not isolate the effect of switching versus identifying rules.

Comparing these two settings is not our concern. The subsection has been revised and focuses on how models can be trained to execute more rules.

We want to clarify that the motivition to use inter- and intra- settings is to represent two different scenarios in natural language, not to compare them. Inter abstracts adaptation to an implicit, persistent domain or style, a model must infer its linguistic conventions from context and maintain them. Intra abstracts successive explicit instructions, a model must execute them using the growing context.

(Weakness 2) Regarding trajectory splitting and higher-order sampling

... describe boundary purging and report actual overlap counts. ... the randomly sampled higher-order settings.

Splitting orbit is necessary but the overlap between test and train samples is controlled to a minimum scale. We have revised the "never occur" expression in the main text.

We reconstructed the clean single-rule baseline, k=2,p=127,T=16k=2, p=127, T=16, rule (1,1)(1,1), using the historical generator. We call kk continuous tokens a state, then the overlap can be explained as "how many state transitions are in the train dataset". In the table below, we show all three audited seeds (0,17996,18318)(0, 17996, 18318) give the same 0.269% overlap rate.

Despite the overlap in experiments using uniform initial-state sampling is not controlled, we have verified our relevant converged models, all achieved 99% conservative accuracy on unseen transitions. We believe as the state space is growing exponentially, it is more appropriate to use fresh-build random test samples, instead of filtering never-seen states. We will clarify this in our paper.

(Weakness 3) Further strengthen the claims about the linear probe: multi-seed uncertainty and internal activation edit

Linear decodability ... does not establish that the model uses that representation to produce its answer. ... no repeated-seed uncertainty. ... Internal activation patching or targeted ablations ...

We agree that decodability alone does not establish causal use. We revised our claims and added the following checks:

  1. Both probe-training and probe-test data now come from each checkpoint’s held-out pool, excluding Transformer training orbits and enforcing trajectory-disjoint splits. We report means and standard deviations across three model seeds (Section G.2).

  2. We added internal steering experiments using mean-difference and probe-weight directions (Section G.3). Changing probe readouts does not reliably switch the executed rule, showing that linear separability does not imply linear controllability by these directions:

The table summarizes steering at α=1\alpha=1. Each cell reports Readout (by probe) / Switch (to the other rule) / Neither (%), averaged across three model seeds. Sites are selected on validation data. Standard deviations and strength sweeps are provided in the appendix.

(Weakness 4) Regarding experimental robustness:

Capacity and optimization effects are not sufficiently separated. ... uncertainty estimates ... Epoch counts ... when dataset sizes change.

Our findings are supported by repeated runs across seeds. We have added a figure of the variability across random seeds in Appendix B.3.

When dataset sizes differ, we only compare the capability. We have proper criteria to claim failure: either the test accuracy remains at random-guessing standard, or the best test accuracy stops updating in considerably long period (like 3000 epochs for the baseline setup, or double the time of a normal converge).

... omits normalization and uses an independent output matrix ... its connection to the observed learning thresholds ...

Omitting LayerNorm is a simplifying convention used in prior constructive analyses of Transformer expressivity (see, e.g., [1], [2] as examples). We adopt this simplification to make the information routing and arithmetic operations in our construction explicit. As for the independent output matrix, our expressiveness results also apply here: the construction can be modified to use tied input and output embeddings while preserving its depth, number of attention heads, residual width, and FFN hidden width. We agree that expressivity and successful learning should be distinguished more explicitly. Our theorem is an auxiliary expressivity result that provides conceptual motivation for studying two-layer architecture, but not a main contribution.

[1] Liu et al., Transformers Learn Shortcuts to Automata. ICLR 2023.

[2] Feng et al., Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective. Neurips 2023.

Reviewer_sz9sOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies how transformer models learn sequences generated by discrete linear dynamical systems. They train models on sequences that are generated by a single or a mix of recurrence rules and study how model capacity, recurrence order and input masking affect learning and generalization. They further study the effects of curriculum learning and interventions on the input to investigate how the model learns the rules.

Critical strengths

The main strength of the paper is its use of a synthetic setup to study different factors affecting generalization, such as model capacity, task difficulty, and training strategy like curriculum learning, in a systematic way.

Critical weaknesses

I have three main concerns with the paper.

  1. Limited novel insights: While the paper studies several aspects of transformer training, I do not find the main empirical observations particularly novel. The benefits of depth and width are well-established, as are the benefits of curriculum learning (even on controlled synthetic tasks (e.g., [1])).

    The mechanistic analysis also does not seem particularly deep. Perhaps the most interesting observation in Section 5 is the non-monotonic effect of changing the data ratio in Figure 8, which is not itself a mechanistic analysis. The linear probing experiments provide limited insight into how the model applies the rules, while the input perturbation analysis is not sufficiently contextualized (see points 2 and 3 below). Overall, it is unclear to me what additional understanding of transformers these experiments provide beyond reproducing previously observed phenomena in another synthetic setting.

  2. Experimental design and interpretation:

    • The comparison between the inter- and intra-trajectory settings conflates two factors: (1) whether the rule is explicitly specified by a rule token or remains latent in the sequence, and (2) whether the sequence is generated by a single recurrence rule or a mixture of recurrence rules. I think it is clear that explicitly providing the rule identity in the sequence would make training substantially easier. Also, the points established through this comparison could have been made by simply comparing sequences generated by a single rule, with and without a token explicitly indicating which rule was used.

    • In the shortcut experiment, both the original recurrence rule and the alternative "shortcut" are valid prediction rules. While it is interesting to investigate which rule the model learns, the paper does not sufficiently motivate why learning the original recurrence rule should be preferred or encouraged during training, given that both rules can generalize. I have a similar concern about the analysis in Section 5.3, which investigates whether the model infers the rule from the beginning of the sequence or from the most recent local tokens. It is unclear why this distinction matters when both approaches lead to correct predictions.

  3. Presentation and clarity: The discussions throughout the paper are not always coherent and are, in some places, rather vague.

    • The methodologies are not clearly explained in the main body. Almost all methodological details, from identifying the rules learned by the model and the shortcuts in Sections 3.2 and 3.3 to the input interventions in Section 5.3, are deferred to the appendix. As a result, the main text often presents only high-level claims that are difficult to understand or descriptions about the experiment that do not make sense.

    • The discussions also feel fragmented. For example, the benefits of depth are discussed repeatedly across several sections rather than being presented in a unified discussion.

Overall, I find the presentation of the results confusing, and the findings largely appear to replicate already known phenomena in another synthetic setting, without providing substantial additional insights.

[1] Bhasin, H., et al. How Does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes.

Questions for authors

See above

2 comments
AuthorsOpenReview ↗

Response to Reviewer sz9s

Response to Reviewer sz9s

Thank you for your helpful comments.

(Weakness 1) Regarding the lack of novel insights:

The benefits of depth and width are well-established, as are the benefits of curriculum learning (even on controlled synthetic tasks (e.g., [1])).

....

Overall, it is unclear to me what additional understanding of transformers these experiments provide beyond reproducing previously observed phenomena in another synthetic setting.

The multi-task and recurrence properties enable us to investigate the mechanism of the Transformer from new perspectives. New insights include:

  1. The competition and cooperation between tasks/rules (Section 5.1).

  2. There is no task/rule vector formed for the single-layer Transformer (Section 5.2) in the inter-trajectory setting, significantly different from models with more layers. This conclusion is different from those on the in-context learning setup, where task vectors are observed and can be transferred to facilitate zero-shot learning [1][2][3].

  3. The local recurrent algorithm learned by the models (Section 5.3). In the inter-trajectory setting, the models leverage only the most recent tokens to predict xtx_t. For a 2-rule inter-trajectory setting, (xt−3,xt−2,xt−1)(x_{t-3}, x_{t-2}, x_{t-1}) is sufficient for the transformer to infer the rule of the current trajectory, with (i) no more redundant information and (ii) no further tokens like (xt−4,xt−3,xt−2)(x_{t-4}, x_{t-3}, x_{t-2}) (although this tuple also provides sufficient information to infer xtx_t).

[1] Roee Hendel, Mor Geva, and Amir Globerson. In-Context Learning Creates Task Vectors. Findings of EMNLP, 2023.

[2] Eric Todd, Millicent L. Li, Arnab Sen Sharma, Aaron Mueller, Byron C. Wallace, and David Bau. Function Vectors in Large Language Models. ICLR, 2024.

[3] Liu Yang, Ziqian Lin, Kangwook Lee, Dimitris Papailiopoulos, and Robert Nowak. Task Vectors in In-Context Learning: Emergence, Formation, and Benefit. arXiv preprint arXiv:2501.09240, 2025.

(Weakness 2a) Regarding experimental design: Inter- versus intra-trajectory learning.

I think it is clear that explicitly providing the rule identity in the sequence would make training substantially easier. Also, the points established through this comparison could have been made by simply comparing sequences generated by a single rule, with and without a token explicitly indicating which rule was used.

Comparing these two settings is not our concern. The subsection has been revised and focuses on how models can be trained to execute more rules.

We want to clarify that the motivition to use inter- and intra- settings is to represent two different scenarios in natural language, not to compare them. Inter abstracts adaptation to an implicit, persistent domain or style, a model must infer its linguistic conventions from context and maintain them. Intra abstracts successive explicit instructions, a model must execute them using the growing context.

(Weakness 2b) Regarding the problem of shortcuts:

While it is interesting to investigate which rule the model learns, the paper does not sufficiently motivate why learning the original recurrence rule should be preferred or encouraged during training, given that both rules can generalize. I have a similar concern about the analysis in Section 5.3, which investigates whether the model infers the rule from the beginning of the sequence or from the most recent local tokens. It is unclear why this distinction matters when both approaches lead to correct predictions.

A shortcut uses fewer operands, but is unavailable for the first few predictions.

To clarify our point, we divide supervised positions into two parts:

  1. The first n=delayn=delay tokens (recall that delay is ...), where only the origin rule can apply.

  2. The latter tokens, where origin rule and shortcut can both apply.

Correspondingly, the model has three behaviors:

a. using origin rule for both 1 and 2,

b. using shortcut for 2 first, then either stuck here, or turn to c:

c. keeping shortcut for 2 and further learning origin rule for 1.

We have no preference for a or c, as they both reach 99%+ test accuracy. We add loss weights on 1 to report some changes from b to a.

Section 5.3 reveals that the difference between equivalent rules lies on the robustness to context perturbations.

(Weakness 3) Regarding the unclear presentation and lack of clarity:

The methodologies are not clearly explained in the main body. ... methodological details ... are deferred to the appendix. ... The discussions also feel fragmented.

Thanks for pointing out the problem, we have revised the paper accordingly by adding a methodology paragraph at the end of Section 2, and a summary paragraph at the beginning of Section 3. However, since the page limit is quite tight, we have to keep some detailed discussions in the appendix.

Thank you for the response. I will maintain my score, as my main concerns about the novelty and scope of the paper remain.

First, the contribution still seems marginal to me. Two of the contributions in the updated paper focus on the benefits of increasing model depth and width, which are already established and marginal. I also find the mechanistic study limited. In your response, you draw a connection to task vectors, but the evidence you point to is linear probing, which alone does not establish such a connection. As far as I understand, task vectors involve more than task information being linearly decodable from hidden activations. The relevant question is whether and how this information is used by the model to perform the task, which the paper does not establish. Plus, I am not convinced that the probing analysis provides much new insight beyond what we know from previous work on synthetic settings.

I also find the motivation for the synthetic setup weak. I agree with Reviewer SnZM that the connections drawn between the inter- and intra-trajectory settings and natural language are quite loose and hand-wavy. That said, I am not against synthetic setups, nor do I think they necessarily need to have direct applicability to natural language. Previous work studying modular arithmetic or Markov chains, for example, has provided useful insights without making strong connections. My concern here is that the paper uses the natural language analogy to motivate the setup, but the connection is not sufficiently grounded. At the same time, the experiments provide limited new insight beyond what has already been established in the literature on synthetic settings. I believe these two issues together are a weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.