- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies length extrapolation in small transformers trained on synthetic copying with additive harmonic positional encodings. Its main intervention is “component dropout”: whole sine/cosine bands are randomly removed during training and all bands are restored at evaluation.
Critical strengths
The experimental controls are careful: paired seeds, seed-level bootstrap intervals, some predictions frozen before the runs, and failed cohorts reported rather than dropped.
The PE-gap probe (fixed tokens, shifted positional coordinates) cleanly separates positional effects from content effects.
Critical weaknesses
1. Limited novelty; key related work is not discussed. The central finding is that distant attention mass interferes with context construction and that restricting it restores extrapolation (Sec. 5, App. L). This overlaps substantially with prior work that the paper neither cites nor discusses:
- Distant-attention restriction: LM-Infinite (Han et al., NAACL 2024) attributes length-generalization failure to attention-entropy growth and proposes a Λ-shaped mask; the paper's Local243 intervention (local window plus BOS/separator) is essentially this mask. StreamingLLM (Xiao et al., ICLR 2024) similarly shows that initial/marker tokens must be kept.
- Attention dilution and scaling: Chiang & Cholak (ACL 2022), Scalable-Softmax (Nakanishi, 2025) and the attention temperature in YaRN (Peng et al., ICLR 2024).
- Frequency-band-specific methods: PI (Chen et al., 2023), YaRN/NTK-by-parts, p-RoPE (Barbero et al., ICLR 2025; cited but not discussed in this respect) and DoPE (Xiong et al., 2025).
- Positional schemes for length generalization: FIRE (Li et al., ICLR 2024). Randomized PE (Ruoss et al., 2023) is cited but not compared against.
The paper's own equal-mass control (App. N) finds no advantage for forecast-selected keys over diffusely removing the same distant mass. This supports a generic attention-dilution explanation rather than one specific to learned harmonic comparisons, and leaves band-level dropout as an incremental contribution.
2. The experimental scale is insufficient.
- Setting: 2-layer, width-64 models (~10^5 parameters), trained at length ≤120 with vocabulary 4 and tested at length 480. This says little about length extrapolation in practice. The tiny vocabulary also makes repeated n-grams pervasive, which inflates exactly the ambiguity the paper analyzes.
- Gains do not hold where it matters: they are concentrated in additive sinusoidal encodings. With RoPE (Table 3) they are negligible or negative, except +4.25 points at V=8, where absolute accuracy is only 17%. Under greedy decoding (Table 2), most grids gain only a few points.
- Missing baseline: there is no working generic-dropout baseline; App. G acknowledges the comparator failed. The benefit is therefore not separated from ordinary regularization.
- Minimum expected evaluation:
- RoPE models of ≥100M parameters trained on natural text with ≥4K context.
- Evaluation well beyond the training length (e.g., 16K–64K) on PPL and RULER/NIAH.
- Baselines: generic dropout, Randomized PE, PI/YaRN, attention scaling and Λ-shaped masking.
Since the setting does not support the central claim, I do not evaluate the mechanistic analyses or the theorem in detail.
Questions for authors
-
Does band dropout outperform standard dropout at matched short-range loss?
-
Do the gains hold for RoPE models at ≥100M parameters with ≥4K training context on natural-language benchmarks?
-
How does the method compare with Λ-shaped masking or attention logit scaling, which directly target the distant-interference mechanism the paper identifies?