- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes dyadic attention, a mechanism designed specifically for prefix-scan problems such as parity, and . The DST model reuses a single Transformer layer iteratively to imitate the prefix-scan algorithm. In each iteration, the attention mask is determined solely by the current iteration index. The model generalizes perfectly to these prefix-scan problems. For where and are two positions, is the number of trailing zeros in the binary representation of , and is . For language modeling, each attention head learns a bias term for every clipped pair. This dyadic model shows slightly better length generalization than YaRN on some tasks.
Critical strengths
None
Critical weaknesses
-
Despite DST's perfect performance on synthetic tasks, the dyadic model improves the perplexity only marginally over a RoPE + YaRN baseline on language modeling tasks. Moreover, the model used for the synthetic tasks is highly customized for these tested tasks and substantially different from the model used in language modeling tasks.
-
FoX [1] learns a bias term using the forget gate. RelAttn (used in Inkling [2] from Thinking Machines Lab) learns a query-dependent relative-position bias term. Both methods are closely related to the method proposed in this paper. However, this paper does not cite or discuss either method.
[1] Forgetting Transformer: Softmax Attention with a Forget Gate, ICLR 2025
[2] Inkling: Our Open-Weights Model, blog
Questions for authors
Can dyadic model outperform FoX or RelAttn in language modeling tasks?
4 comments
Thank you for the additional experiments. My main concerns remain.
-
The new experiments show that dyadic+PI outperforms PI-only, but do not establish an advantage over YaRN under matched conditions. A controlled ablation against basic linear PI cannot replace a comparison against strong baselines.
-
I acknowledge DST’s theoretical and empirical contributions on synthetic tasks. However, these tasks already have exact, efficient algorithms. I expect their study to provide explanations or improvements relevant to practical applications, not necessarily language modeling. Sharing a representation does not transfer DST’s results to the language model, which requires independent evaluation. Its practical advantages over strong baselines remain unestablished.
-
The paper already trains RoPE and dyadic models from scratch, so I do not find architectural differences a sufficient reason to omit FoX and RelAttn comparisons. These methods may achieve competitive length extrapolation without PI or YaRN. Whether dyadic can additionally be combined with them is separate from their relevance as competing methods. Discussion alone cannot replace controlled experiments.
I will maintain my score of clear rejection.
Thank you for clarifying your remaining concerns. We would like to emphasize three points.
1. LM baselines. We pretrained matched 1.3B and 3B RoPE and dyadic models from scratch. Our experiments include RoPE-FT, PI-FT, and full-budget YaRN-FT, as well as smaller-scale NoPE and ALiBi controls.
Importantly, perplexity is not our only LM metric. Our dyadic+PI model achieves 94%/75% retrieval accuracy at 8k/16k, compared with 60%/22% for YaRN. These finetunes use the same token budget and context schedule, although some training settings differ. The new matched PI-only experiment directly tests whether PI alone explains the retrieval gain. It does not replace the YaRN comparison.
2. Certified DST result. We do not propose a new prefix-scan algorithm. Our contribution is showing that a weight-shared Transformer can learn this computation from short sequences, and that the correctness of the trained model can be verified for arbitrary lengths, including numerical errors, under the stated inference conditions.
This is a formal guarantee for a trained neural network, not simply high accuracy on long test sequences. What is more, we do not claim that this guarantee transfers to the LM, which we evaluate separately.
3. FoX and RelAttn. We have added both methods to Related Work and explained their differences. They are relevant alternatives, but our evaluation already covers several established approaches. A fair comparison with these different architectures would require new training at comparable scales, which is beyond the rebuttal period. Additionally, Inkling was introduced in a July 2026 technical blog post, shortly before our submission, and falls within the four-month window for contemporaneous work.
Taken together, our paper provides both a verified theoretical result for a trained Transformer and experimental evidence from 1.3B and 3B language models. We believe these contributions should be assessed on their own merits, rather than mainly on the absence of two additional architectural comparisons.
Thank you for considering these clarifications.
-
Your retrieval comparison against YaRN uses different optimizers, learning rates, and data shards, without explaining why. It therefore does not provide reliable evidence of dyadic’s superiority. Why not compare dyadic+YaRN against YaRN under matched conditions? I see no justification for omitting this experiment.
-
I acknowledge that the certified DST result is interesting, but I do not see its practical utility. Stating that the LM is evaluated separately does not address this concern: that evaluation still does not convincingly establish an advantage over strong baselines.
-
The contemporaneous-work argument may apply to Inkling, but not to FoX, published at ICLR 2025. I agree that achieving SOTA is not mandatory. However, YaRN is an established, widely used baseline, with subsequent methods such as LongRoPE2 (ICML 2025) reporting stronger results. You have not convincingly established an advantage even over YaRN under matched conditions. Arguing that SOTA is unnecessary does not answer this concern.
-
The discussion period has lasted at least 13 days. Training a FoX baseline at one scale during this period is not obviously infeasible. Despite the repeated replies, the requested experiment remains missing. I therefore maintain my rating of clear rejection.
Thank you for the review and the related-work suggestions. We have added a controlled experiment and clarified the relation between our two models.
Weakness 1: LM performance.
Our original results show competitive perplexity with YaRN: 18.4 vs. 18.5 at 65k. Retrieval shows a larger improvement, but the original finetuning configurations differ.
To isolate the contribution beyond PI, we trained four 1.3B models: PI-only and dyadic+PI, each with two finetuning seeds. Both methods use identical training data and order, optimizer, learning-rate and context schedules, PI factor, and 1.573B-token budget.
Table 13(a)-(b) (updated manuscript, page 34, upper half) reports the results:
Passkey accuracy (%)
Same-target PPL
At 16k, dyadic+PI improves retrieval by 29.6 percentage points. The paired 95% confidence intervals are positive for both seeds: [13.6, 29.6] and [28.8, 46.4] percentage points. At 32k, dyadic+PI also achieves 34.2% lower perplexity than PI-only. These results provide direct evidence of benefits from dyadic pretraining and finetuning beyond PI alone.
Weakness 1: relation between the two architectures.
We clarified the shared geometry in Section 5 (updated manuscript, page 5, lower part). DST uses shifted binary coordinates to select scan partners. The LM uses the same coordinates to share positional-bias parameters. These are two applications of the same representation.
Importantly, the DST contribution includes a verified numerical guarantee. For S5, we check 14,520 initialization cases and 14,520 transitions, with certified margins of 9.830 and 10.540. Projection and readout are also verified with floating-point bounds. Appendix E (pages 22-23) describes the certified deployment.
Weakness 2 and Question 1.
We thank the reviewer for these references. We added a comparison in Section 7 (page 9) and Appendix V (page 47).
FoX uses input-dependent cumulative forgetting, while RelAttn uses query-dependent relative-position biases. Our method instead groups offsets by their binary scale and divisibility, using a learned scalar for each cell. This structure also enables our occupancy analysis.
We selected PI and YaRN as established baselines for extending RoPE-based pretrained models. They allow us to evaluate context extension within a shared architecture and tokenizer. FoX and RelAttn introduce different attention mechanisms, so a direct architectural comparison would require a separate controlled training study.
Our new matched experiment directly tests the benefit of the dyadic-trained model beyond PI under identical finetuning conditions. Together with the verified scan result, this provides evidence for the two main contributions of our paper.
We hope the new evidence supports reconsideration of the overall assessment.