- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to improve sequential computation and multi-state tracking without generating additional reasoning tokens. It proposes Dual-Mod attention, which first reads the key-value cache and then uses the retrieved information and current input to construct contextual keys and values. This creates a recurrence along the token sequence. Chunk-wise iterative sweeps enable parallel training.
Experiments cover the A5 word problem, interleaved-register tracking, circuit value prediction, and language modeling at 340M and 1.3B parameters. Comparisons include Transformers, LSTMs, gated DeltaNet, looped Transformers, and full-bandwidth Transformers. With progressive training, a one-layer model with approximately 12M parameters tracks 1,024 registers at composition depth eight and reaches the paper's criterion of at least 95% accuracy at every depth on 1,024-gate circuits. Language-modeling experiments show improvements in loss and several retrieval metrics; the 1.3B sliding-window model achieves a retrieval average of 45.64.
Critical strengths
-
The architectural design is closely connected to the core experiments. Conditioning future key-value writes on the attention read provides sequential computational depth and an addressable cache that grows with the sequence. CVP and keyed A5 separately probe serial computation and the maintenance and updating of multiple independent states. The one-layer model's results on both dimensions are a meaningful contribution.
-
The task protocols and training-operator checks are carefully considered. The authors remove an input format that exposes intermediate states, use independently generated evaluation instances, and share the training token stream across the 340M language-modeling controls. Appendix C.4 also clarifies that committed cache entries remain connected to the gradient and checks agreement between the exact chunk computation and sequential scan in loss, outputs, and gradients.
Critical weaknesses
- Inconsistency. I used an LLM Agent to recompute the averages in your reported results and cross-check the tables, figures, and text (This audit was restricted to the manuscript and did not rerun training or evaluation). The consequential issues concern success criteria, seed accounting, and comparison conditions. In Figure 6, the red frontier follows the any-seed minimum depths reported in Table 2: 2, 3, 6, and 4 layers. The caption instead describes an all-seed success criterion. Moreover, the cell at T=15 and L=6 displays 1.00, while the accompanying text reports “best seed 0.89, 0/3” for that row without explaining whether these numbers refer to different sets of runs. These discrepancies affect the claimed minimum depth and cross-seed reliability of the Transformer baseline. Table 18 also specifies two DM seeds on A5, whereas the text reports one successful seed out of three, leaving the denominator of the training success rate unclear. Finally, Appendix G describes Figure 8 as comparing all models on 64-gate circuits, while its caption and legend show each model at its largest trained circuit size, including 1,024 gates for DM and 256 for FBT. This obscures the distinction between a comparison at a common task size and a display of the largest sizes reached.
There are also smaller aggregation discrepancies. From the displayed task scores, the Avg9 values for GDN-2 recurrent and KDA hybrid in Table 1 are approximately 53.22 and 52.74, compared with the reported 53.11 and 52.68. Equal-weight averaging of the four MQAR loads in Table 8 gives 22.365 for Transformer++ and 34.6175 for GDN, compared with 22.46 and 34.64. These averaging discrepancies do not change DM’s ranking in the affected comparisons. The inconsistencies in success criteria, seed accounting, and comparison conditions have a more direct bearing on the strength of the claims.
- Baseline Incompletion. The manuscript explicitly states that FBT’s 32-register keyed A5 experiment and 512-gate CVP experiment are still in progress. Its largest completed experiments therefore do not establish its capability limits. Meanwhile, the 1,024-gate DM result has been completed by only one training lineage and required an additional 26k steps beyond the original 60k-step budget; the second lineage currently satisfies the success criterion only through depth 794. These results demonstrate that DM can reach 1,024 gates under the reported training conditions. They provide insufficient evidence to establish the size of its advantage over FBT or the reproducibility of the largest-scale result.
p.s.: The font sizes in Table 1. and Table 3. are too small to read. In addition, the abstract is too long, which I recommend to the range from 150 to 250 words. These two issues are just a reminder, not a attribution to any of the following ratings.
Questions for authors
Question 1: Can the largest-scale advantage be reproduced under a common training budget?
The most important additional experiment is a matched-budget, multi-seed comparison of DM and FBT on 512- and 1,024-gate CVP. Can you use a common cumulative training-compute budget and size-continuation policy, and report per-depth accuracy, whole-circuit exact-match accuracy, and the fraction of seeds satisfying the paper’s success criterion? For DM at 1,024 gates, please also report results at both 60k and 86k steps to establish how much the additional training contributes to the headline result.
Question 2: How many A5 seeds were run, and how were the reported aggregates computed?
Which runs correspond to the two DM seeds specified in Table 18 and to the one successful seed out of three described in the text? Please provide the seed identifier, in-distribution accuracy, and accuracy at ten times the training length for each run, and explain how unsuccessful and unrun experiments enter the statistics. Please then reconcile the success rate and the aggregation used in Table 18. In addition, please provide the averaging formulas or aggregation script for Tables 1 and 8 and explain the averages that cannot be reproduced from their displayed component scores.
Question 3: Which experiments and success criteria do Figures 6 and 8 represent?
Does Figure 6’s red frontier use an any-seed or an all-seed success criterion? How should the 1.00 entry at T=15 and L=6 be reconciled with “best seed 0.89, 0/3” in the text, and how should the panel title “1 seed per cell” be reconciled with “best of 1–3 seeds” in the caption? Please regenerate Figure 6 and Table 2 from the same run-level results, specify the seed count for every cell, and separately identify the minimum depths under both success criteria. Please also reconcile the description of Figure 8 as a 64-gate comparison with its use of each model’s largest trained circuit size, and clarify which comparisons support conclusions at a common task size and which only display the largest sizes reached.