acceptodds
Under review as a conference paper at ICLR 2027

Dual-Mod Attention: Token Recurrence Beats Looping

Abstract

Transformers have shown incredible gains in reasoning performance by scaling parameter count and test time compute, but together they create an immense increase in computation budget and context lengths. Latent-thinking approaches grow computation without emitting tokens, along two paths: looped transformers, which iterate a weight-tied block, and recurrent hybrids, which carry a compact evolving state. Both of these methods have their own failure modes. Looped transformers can solve some problems outside of , but their per-iteration update shrinks and accuracy saturates, after which further iterations add nothing. Their reach is capped by the training-time iteration count and length, and we find that extra test-time iterations do not extend it. Past the training length, looping rapidly collapses to chance and stays within a transformer's length-generalization boundary. Recurrent methods are better at length generalization and also solve tasks outside of , but carry a fixed state that provably bounds them as the number of tracked states grows. Linear recurrences commonly used in hybrids cannot perform -complete state tracking: diagonal ones remain in outright (Illusion of State), and single-step delta-rule ones confine their transition spectra to [0,1]. We present a new path to scaling computation using a non-linear recurrence within softmax attention. Our method uses an append-only -tape where the entries are computed recurrently from the tape instead of raw tokens. We call this class of architectures token recurrences. Our arch is an exact superset of transformers whose chunk sweeps have a unique fixed point, the sequential recurrence itself, reached in finitely many steps, and whose state moves with every token. Since AR decoding is already sequential, the recurrence adds no decoding cost. Empirically, a single layer of our arch solves instances of the P-complete circuit value problem up to depth 1024. It also solves the -complete word problem, and on a variant with interleaved registers tracks up to 1024 registers at full depth, where single-pass attention and fixed-state baselines stop by 32 and 128 registers. Token recurrences cross the boundary that loops cannot while keeping a growing cache, and ours allows chunk-parallel training at a fraction of a layer per sweep. On language modeling tasks, it beats parameter matched baselines on loss and is competitive with SOTA arches in each eval suite.

Primary area:
foundation or frontier models, including LLMs
Keywords:
state tracking, expressivity, looped transformers, latent reasoning, linear RNNs

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_phcdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to improve sequential computation and multi-state tracking without generating additional reasoning tokens. It proposes Dual-Mod attention, which first reads the key-value cache and then uses the retrieved information and current input to construct contextual keys and values. This creates a recurrence along the token sequence. Chunk-wise iterative sweeps enable parallel training.

Experiments cover the A5 word problem, interleaved-register tracking, circuit value prediction, and language modeling at 340M and 1.3B parameters. Comparisons include Transformers, LSTMs, gated DeltaNet, looped Transformers, and full-bandwidth Transformers. With progressive training, a one-layer model with approximately 12M parameters tracks 1,024 registers at composition depth eight and reaches the paper's criterion of at least 95% accuracy at every depth on 1,024-gate circuits. Language-modeling experiments show improvements in loss and several retrieval metrics; the 1.3B sliding-window model achieves a retrieval average of 45.64.

Critical strengths
  1. The architectural design is closely connected to the core experiments. Conditioning future key-value writes on the attention read provides sequential computational depth and an addressable cache that grows with the sequence. CVP and keyed A5 separately probe serial computation and the maintenance and updating of multiple independent states. The one-layer model's results on both dimensions are a meaningful contribution.

  2. The task protocols and training-operator checks are carefully considered. The authors remove an input format that exposes intermediate states, use independently generated evaluation instances, and share the training token stream across the 340M language-modeling controls. Appendix C.4 also clarifies that committed cache entries remain connected to the gradient and checks agreement between the exact chunk computation and sequential scan in loss, outputs, and gradients.

Critical weaknesses
  1. Inconsistency. I used an LLM Agent to recompute the averages in your reported results and cross-check the tables, figures, and text (This audit was restricted to the manuscript and did not rerun training or evaluation). The consequential issues concern success criteria, seed accounting, and comparison conditions. In Figure 6, the red frontier follows the any-seed minimum depths reported in Table 2: 2, 3, 6, and 4 layers. The caption instead describes an all-seed success criterion. Moreover, the cell at T=15 and L=6 displays 1.00, while the accompanying text reports “best seed 0.89, 0/3” for that row without explaining whether these numbers refer to different sets of runs. These discrepancies affect the claimed minimum depth and cross-seed reliability of the Transformer baseline. Table 18 also specifies two DM seeds on A5, whereas the text reports one successful seed out of three, leaving the denominator of the training success rate unclear. Finally, Appendix G describes Figure 8 as comparing all models on 64-gate circuits, while its caption and legend show each model at its largest trained circuit size, including 1,024 gates for DM and 256 for FBT. This obscures the distinction between a comparison at a common task size and a display of the largest sizes reached.

There are also smaller aggregation discrepancies. From the displayed task scores, the Avg9 values for GDN-2 recurrent and KDA hybrid in Table 1 are approximately 53.22 and 52.74, compared with the reported 53.11 and 52.68. Equal-weight averaging of the four MQAR loads in Table 8 gives 22.365 for Transformer++ and 34.6175 for GDN, compared with 22.46 and 34.64. These averaging discrepancies do not change DM’s ranking in the affected comparisons. The inconsistencies in success criteria, seed accounting, and comparison conditions have a more direct bearing on the strength of the claims.

  1. Baseline Incompletion. The manuscript explicitly states that FBT’s 32-register keyed A5 experiment and 512-gate CVP experiment are still in progress. Its largest completed experiments therefore do not establish its capability limits. Meanwhile, the 1,024-gate DM result has been completed by only one training lineage and required an additional 26k steps beyond the original 60k-step budget; the second lineage currently satisfies the success criterion only through depth 794. These results demonstrate that DM can reach 1,024 gates under the reported training conditions. They provide insufficient evidence to establish the size of its advantage over FBT or the reproducibility of the largest-scale result.

p.s.: The font sizes in Table 1. and Table 3. are too small to read. In addition, the abstract is too long, which I recommend to the range from 150 to 250 words. These two issues are just a reminder, not a attribution to any of the following ratings.

Questions for authors

Question 1: Can the largest-scale advantage be reproduced under a common training budget?

The most important additional experiment is a matched-budget, multi-seed comparison of DM and FBT on 512- and 1,024-gate CVP. Can you use a common cumulative training-compute budget and size-continuation policy, and report per-depth accuracy, whole-circuit exact-match accuracy, and the fraction of seeds satisfying the paper’s success criterion? For DM at 1,024 gates, please also report results at both 60k and 86k steps to establish how much the additional training contributes to the headline result.

Question 2: How many A5 seeds were run, and how were the reported aggregates computed?

Which runs correspond to the two DM seeds specified in Table 18 and to the one successful seed out of three described in the text? Please provide the seed identifier, in-distribution accuracy, and accuracy at ten times the training length for each run, and explain how unsuccessful and unrun experiments enter the statistics. Please then reconcile the success rate and the aggregation used in Table 18. In addition, please provide the averaging formulas or aggregation script for Tables 1 and 8 and explain the averages that cannot be reproduced from their displayed component scores.

Question 3: Which experiments and success criteria do Figures 6 and 8 represent?

Does Figure 6’s red frontier use an any-seed or an all-seed success criterion? How should the 1.00 entry at T=15 and L=6 be reconciled with “best seed 0.89, 0/3” in the text, and how should the panel title “1 seed per cell” be reconciled with “best of 1–3 seeds” in the caption? Please regenerate Figure 6 and Table 2 from the same run-level results, specify the seed count for every cell, and separately identify the minimum depths under both success criteria. Please also reconcile the description of Figure 8 as a 64-gate comparison with its use of each model’s largest trained circuit size, and clarify which comparisons support conclusions at a common task size and which only display the largest sizes reached.

Reviewer_zrNtOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Dual-Mod Attention, an attention-based recurrent architecture that uses each token’s attention output to modify the key and value cached for subsequent tokens. This enables sequential computation within a layer while retaining a growing, addressable memory. Its contributions include the KV-update mechanism, a chunk-parallel iterative training scheme, and comparisons with standard Transformers, fixed-state recurrent models, and looped Transformers. On synthetic state-tracking tasks, a roughly 12M-parameter single-layer model reaches circuit depth 1,024 and tracks 1,024 interleaved registers with eight updates each, using progressive training. Language-modeling experiments at 340M and 1.3B parameters show competitive performance, particularly on retrieval. However, length extrapolation remains inconsistent and sometimes trails LSTMs, while the language-model comparisons match parameters rather than training compute; the reported 1.3B setup uses approximately 3.3 times the training FLOPs of its Transformer counterpart. The results demonstrate a promising implementation of recurrent attention, although its advantages over earlier attention-based recurrent memory architectures remain less clearly established.

Critical strengths
  • The paper proposes a compact and technically coherent way to introduce token-wise recurrence through contextual KV writes, while retaining the standard cache structure and recovering ordinary attention as a special case.
  • The chunk-parallel training scheme addresses a central implementation challenge of the architecture. The finite-step equivalence to sequential evaluation and the empirical agreement between approximate and exact evaluation provide useful support.
  • The synthetic evaluation distinguishes sequential computation depth from state capacity. The results achieved by a small, single-layer model are compelling within the reported progressive-training settings.
  • Experiments at 340M and 1.3B parameters demonstrate applicability beyond synthetic tasks, with competitive language-modeling results and particularly encouraging retrieval performance.
  • Extensive ablations and failure analyses provide useful evidence about optimization, memory preservation, and length extrapolation, making the paper informative beyond its headline results.
Critical weaknesses
  1. The distinction from prior attention-based recurrent memory architectures is insufficiently established. The central mechanism (retrieving previous latent states, updating them nonlinearly, and storing the result for future retrieval) belongs to an established family of recurrent memory models. The paper identifies a specific implementation through contextual KV writes, but the comparisons primarily establish advantages over standard Transformers, fixed-state recurrent models, and selected looping recipes. They do not sufficiently isolate what this particular design contributes beyond recurrent attention with a growing memory. A more explicit architectural comparison and a representative baseline from this family would strengthen the novelty and attribution claims.

  2. The language-modeling comparisons do not establish gains under a matched training-compute budget. The reported comparisons are parameter-matched, while the 1.3B DM configuration requires approximately 3.3× the training FLOPs of its Transformer counterpart. Consequently, the results demonstrate competitive performance at a given parameter count, but do not establish whether the proposed architecture offers a better use of training compute. A comparison allowing the baseline to use the additional budget would be important for assessing the practical significance of the gains.

  3. The broad conclusions about looping exceed the scope of the evidence. The experiments provide useful evidence that the tested looping recipes saturate and often fail to benefit from additional evaluation iterations. However, these observations do not establish a general limitation of looped Transformers. Likewise, shrinking state updates are consistent with saturation but do not, by themselves, identify its underlying cause. The “token recurrence beats looping” conclusion should be qualified by the evaluated tasks, architectures, and training procedures.

  4. The strongest synthetic results demonstrate trained capacity more clearly than robust extrapolation. The largest circuit and register results rely on progressive training, and the paper reports sensitivity to initialization, optimization, and continuation schedules. These results are meaningful, but should be distinguished from generalization to substantially larger instances without further training. Length extrapolation remains inconsistent and sometimes trails LSTMs. More systematic reporting of success rates and cumulative training budgets across the progressive-training chains would clarify the reliability and cost of reaching the headline results.

Some headline terminology obscures the conditions under which the claims hold. In particular, “exact” denotes an accuracy threshold in the synthetic experiments, whereas it denotes computational equivalence when discussing the recurrence solver. Similarly, “no extra decoding cost” can be read more broadly than the absence of additional full-model iterations, since contextual KV writes still require computation and parallel prompt processing presents a separate trade-off. These qualifications should appear alongside the corresponding claims, with architectural expressivity, optimization success, and extrapolation treated separately.

Questions for authors
  1. What benefit is specific to the proposed KV-update mechanism, beyond recurrent attention with a growing memory? Could you clarify this distinction relative to earlier attention-based recurrent memory architectures and identify a controlled comparison that would isolate the contribution of your design?
  2. Do the language-modeling gains persist under an equal training-compute budget? Given the reported approximately 3.3× training FLOPs at 1.3B, could you compare against a Transformer trained with a comparable budget, or provide performance-versus-compute curves at a smaller scale?
  3. How reliably are the headline synthetic results obtained? Could you summarize success rates across seeds, cumulative training costs—including preceding curriculum stages—and performance on larger instances without additional training? This would help distinguish representational capacity, optimization reliability, and extrapolation.
  4. What explains the observed saturation of the looped baselines? Can the existing evidence distinguish a limitation of the tested training recipes from a broader architectural limitation? Which narrower conclusion about looping do you believe the experiments establish most directly?
  5. What is the practical efficiency trade-off between chunk-parallel training, prompt prefill, and autoregressive decoding? Could you report measured throughput, latency, and memory usage against representative Transformer and looped baselines, separating these phases and stating whether evaluation uses approximate sweeps or exact recurrence?
  6. Which failure mode is the main obstacle to making the method reliable? Your analyses identify initialization sensitivity, attention dilution, and loss of verbatim information in contextual writes. Which of these best explains the remaining performance gaps, and do the ablations suggest a targeted modification that improves it without sacrificing the other benefits?

[Post-Rebuttal Edit] The authors have meaningfully addressed my weaknesses and questions, I thus lean weakly towards acceptance.

Reviewer_7ki2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce dual-mod attention, a way to introduce non-associative recurrent attention into transformers. Specifically, authors introduce an LSTM-style gate to update historical K,V pairs, upon which fresh attention is applied. To correct for the obvious inefficiencies, authors introduce a chunked training scheme where subsequences are scanned in multiple parallel passes until finite depth, providing an approximation for full recurrence. Authors then examine their model's performance across standard natural language generation and inference tasks and find marginal gains over baseline, while finding interesting positive performance for the keyed A5 task and CVP tasks. Authors advocate for this architecture as a means to break the standard TC0 gap while still allowing the carried state size to grow as a function of sequence length (unlike, for instance, LSTMs.)

Critical strengths
  • The paper introduces a convincing and reasonably efficient method for training non-associative attention systems; arguably, the systems contribution is the strongest gain in this paper and can be used to build future scanning-based attention methods.
  • The authors also introduce interesting results in Table 3, left, where they show that unlike traditional transformers their method can (with warm-starts) solve nested A5 style problems where transformers fail.
Critical weaknesses
  • The notion of a non-associative scan should be needed in a transformer is not new [1]. Although this may be concurrent work, the relatively lack of insights through the rest of this work makes it difficult to see the novelty with existing projects in mind.
  • Table 1 does not show really meaningful gains beyond most baselines, and it seems like the recurrent approach has tradeoffs in performance across most task categories. Its unclear whether there is a strong gain to be found in any general task, especially concerning relative to the performance overhead it creates.
  • Table 3, which is one of the stronger contributions of the work, is very confusing. In addition to not finishing the sweep and leaving many gaps (even in some cases where the models did not degrade, like Full Bandwidth at 9L or authors contribution at 294M), the metric of "solved" seems relatively arbitrary at 95% gains, it also leaves open some question about the performance of baselines (see below in questions).
  • Lastly, the writing quality of the paper makes it extremely difficult to parse. For instance, the bottom paragraph in page 7 provides a a expansive surface form description of hyperparameters used in the sweep after for each setup which is already in the table but does not actually make clear, for instance, how the actual dataset is generated, how it is setup in terms of OOD generalization, and memorization effects.
  • As a nitpick, figure 1 did not help me understand the work at all.

[1] Oncescu C., et al., "The Recurrent Transformer: Greater Effective Depth and Efficient Decoding." ArXiv preprint, 2604.21215 (2026).

Questions for authors

I'm very curious about the "solved" metric in Table 3, left, especially as it supports the strongest empirical contribution in the work. Some questions relating to the construction of the figure itself:

  • How many runs that are "unsolved" actually strattles this metric barely?
  • What are the behaviors of baselines beyond what is examined?
  • When a run terminates without the whole grid, what statistical evidence across seeds indicate that future runs will be worse?
  • What are the experimental properties of the task? How many synthetic samples are generated? What's the IID/OOD split, in what way? What types of generalization can be memorized, and what must be learned?

Additionally, I'm curious about the surprising underperformance of LSTMs. Arguably, authors' statement about the complexity bounds of recurrent systems should indicate that they can solve the A5 task at the depths examined. Specifically, authors indicate that LSTMs have limited ability to store and track state through its one recurrent vector. While this is true, I don't see how LSTMs would struggle on this examined task specifically:

There are 5!2=60\frac{5!}{2}=60 possible even permutations in A5. Each state is thus one of 60 possibilities, and each application is Markovian in that it only depend on the previous state. Now consider the case of K=16K=16 registers. Our 16 registers require 601660^{16} states; this gives log⁡(6016)=16log⁡(60)≈29\log(60^{16}) = 16 \log(60) \approx 29 bits to encode the unique state, which surely can be stored in an LSTM of 53 million parameters. Thus, I wonder if the authors have any insights in terms of why LSTMs would fail at this task?

I'm happy to raise scores if the understandability or scientific rigor of the work can be improved.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.