- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces QDecay, a lightweight controller that can be added to any additive decay-based positional encoding, such as ALiBi or FoX. For each query, it predicts a nonnegative coefficient from the current token’s hidden state to strengthen, preserve, weaken, or fully remove the underlying decay, allowing token-level switching between ALiBi-style recency bias and NoPE-style attention without a distance penalty. The experiments show promising results at the 250M and 800M model scales, with a training context length of 4K.
Critical strengths
The idea is simple and directly addresses the problem: giving the current query control over decay allows the model to switch between favoring nearby information and reading distant information as needed.
The experiments cover language modeling, long-context recall, length extrapolation, and hybrid architectures, while ablation studies test the roles of query dependence and zero decay.
Critical weaknesses
The main results cover only the 250M and 800M model scales, and the paper reports only one training seed, without repeated runs or error bars. It remains unclear whether the gains are stable across runs and hold at larger scales.
The paper does not report comparisons with RoPE using common context-extension methods, such as position scaling or YaRN. The results therefore support an advantage over the tested RoPE baseline, but not necessarily over widely used long-context alternatives.
Long-context recall is mainly tested on RULER, while the real-world recall tasks use short contexts. More results on real long-document tasks, such as question answering, summarization, and code understanding, would strengthen the evaluation.
In Table 17, QDecay’s S-NIAH-1 accuracy at 8K (78.7) is much lower than at 16K (94.3). The authors should explain this change and provide repeated evaluations or confidence intervals to help identify its cause.
The FlexAttention training implementation takes 22.5 hours, compared with 16.8 hours for RoPE, so training cost should be clearly separated from claims about fast inference. The authors claim that optimized FlashAttention training kernels reduce the slowdown to below 10%, but state, “We plan to release these kernels after the deadline,” making this claim difficult to independently verify from the current submission.
Questions for authors
See above
4 comments
QDecay’s S-NIAH-1 accuracy at 8K (78.7) is much lower than at 16K (94.3)
We re-evaluated the 800M QDecay model on S-NIAH-1 with five evaluation seeds. The table reports the mean ± standard deviation.
| 1K | 2K | 4K | 8K | 16K | 32K | 64K | |
|---|---|---|---|---|---|---|---|
| S-NIAH-1 | 100.0 ± 0.0 | 100.0 ± 0.0 | 100.0 ± 0.0 | 84.6 ± 6.9 | 94.4 ± 1.5 | 96.6 ± 1.5 | 96.2 ± 1.3 |
Averaged over 5 evaluation seeds, the gap between 8K and 16K shrinks but still persists. We believe this could be a training artifact of the particular training initialization.
Indeed, a QDecay model trained with a different seed shows no dip at 8K:
| 1K | 2K | 4K | 8K | 16K | 32K | 64K | |
|---|---|---|---|---|---|---|---|
| S-NIAH-1 | 100.0 | 100.0 | 100.0 | 100.0 | 99.3 | 98.0 | 97.0 |
hold at larger scales
We report results at two scales, 250M/10B and 800M/30B. Training larger models is unfortunately beyond our compute budget.
To make it easy for the community to scale QDecay further, we plan to release our efficient FlashAttention-based implementation.
QDecay training kernels
We plan to release the training kernel in the next one or two days. Thanks for your patience!
It remains unclear whether the gains are stable across runs
We are training additional seeds, and we hope to share the results soon. Thanks for your patience!
It remains unclear whether the gains are stable across runs
We thank the reviewer for their patience.
We trained two additional seeds each for QDecay, PaTH and Wall at the 800M/30B scale. Here are the results over three total seeds per model (mean ± standard deviation). Due to limited time and compute, we were not able to train RoPE at more seeds, but we include its performance for reference comparison.
Language modeling and recall:
| Model | LM ppl. ↓ | Recall ↑ |
|---|---|---|
| RoPE | 18.0 | 54.1 |
| Wall | 17.1 ± 0.0 | 56.6 ± 1.9 |
| PaTH | 16.7 ± 0.2 | 56.3 ± 0.9 |
| QDecay | 16.3 ± 0.1 | 57.6 ± 1.8 |
RULER accuracy, averaged across tasks:
| Model | 1K | 2K | 4K | 8K | 16K | 32K | 64K |
|---|---|---|---|---|---|---|---|
| RoPE (YaRN) | 69.7 | 57.1 | 50.4 | 40.1 | 24.3 | 17.0 | 8.4 |
| Wall | 79.7 ± 4.9 | 70.8 ± 2.9 | 59.4 ± 11.5 | 62.3 ± 1.5 | 55.9 ± 0.5 | 16.2 ± 16.7 | 0.1 ± 0.2 |
| PaTH | 82.3 ± 3.3 | 74.3 ± 1.7 | 62.0 ± 6.4 | 53.7 ± 19.6 | 41.0 ± 16.1 | 14.8 ± 3.1 | 1.7 ± 0.6 |
| QDecay | 81.6 ± 4.9 | 74.3 ± 0.1 | 63.8 ± 8.8 | 52.3 ± 11.6 | 37.2 ± 12.0 | 25.5 ± 9.6 | 15.3 ± 5.0 |
Within the 4K pretraining length, QDecay is on par with or better on both RULER and language modeling. At extremely long contexts of 32K and 64K, QDecay outperforms all encodings by at least ~10 and ~15 absolute points respectively across three seeds.
This stable performance at extreme contexts of 32K and 64K agree with language modeling on long books. On pg19, QDecay has the lowest perplexity at every length and stays stable up to 64K across seeds, whereas the perplexity of Wall and PaTH rises at 64K with a large variance across seeds.
Further, Wall and PaTH lose performance at these extreme lengths, consistent with the contraction of the attention logits of old keys that we discuss in Section 7 and measure for Wall in Appendix D.
Here's PPL on pg19 long books:
| Model | 4K | 8K | 16K | 32K | 64K |
|---|---|---|---|---|---|
| RoPE | 16.80 | 68.35 | 295.44 | 546.13 | 737.12 |
| RoPE (YaRN) | 16.80 | 16.51 | 16.81 | 18.19 | 20.59 |
| Wall | 16.77 ± 0.20 | 16.24 ± 0.18 | 15.84 ± 0.17 | 16.09 ± 0.31 | 26.81 ± 14.29 |
| PaTH | 16.57 ± 0.08 | 15.99 ± 0.05 | 15.53 ± 0.02 | 15.62 ± 0.09 | 17.02 ± 1.02 |
| QDecay | 16.22 ± 0.11 | 15.70 ± 0.12 | 15.29 ± 0.11 | 15.24 ± 0.10 | 15.24 ± 0.06 |
results after post training i.e. long context extension
Moreover, following standard practice before deployment at long contexts, we post-train all models.
Specifically, we perform a long-context extension phase on long 128K-token documents from the train split of the pg19 dataset
(note that the base models were pretrained on 4K).
For RoPE, we long-context extend using two methods and report both: Adjusted Base Frequency (ABF) and YaRN.
We grid search the best for RoPE.
Here RULER S-NIAH average (over three tasks) after post training (i.e. long context extension to 128K):
| Model | 8K | 16K | 32K | 64K | 128K |
|---|---|---|---|---|---|
| RoPE (ABF θ=1M) | 64.0 | 49.3 | 42.7 | 21.3 | 14.7 |
| RoPE (YaRN) | 42.7 | 37.3 | 33.3 | 30.7 | 20.0 |
| Wall | 28.4 ± 20.2 | 8.0 ± 10.6 | 1.8 ± 2.0 | 1.8 ± 3.1 | 4.4 ± 5.4 |
| PaTH | 80.4 ± 6.0 | 72.4 ± 5.4 | 43.1 ± 30.6 | 23.1 ± 13.7 | 12.4 ± 5.6 |
| QDecay | 80.4 ± 14.7 | 64.9 ± 13.4 | 50.2 ± 20.2 | 44.0 ± 20.2 | 37.3 ± 13.5 |
QDecay performs great again at extreme contexts of 64K and 128K. Multiplicative decays like Wall (and contractions like PaTH) suffer at long contexts despite being trained on these long sequences, due to their fundamental nature of contraction, which results in "confusion" in attention (see Sec. 7 and App. D). Hence, even RoPE outperforms these decays once one does training on long sequences.
Interestingly, Wall does not improve despite training on long documents, possibly because its accumulated decay pushes the attention logits of old keys toward zero (App. D), resulting in near-uniform attention scores and poor optimization at long contexts.
This suggests that not all position encodings may be suited for post-training, including long-context extension or RL.
Here's pg19 PPL after post training (i.e. extension to 128K):
| Model | 4K | 8K | 16K | 32K | 64K | 128K |
|---|---|---|---|---|---|---|
| RoPE (ABF θ=1M) | 14.16 | 13.63 | 13.26 | 13.00 | 12.96 | 12.85 |
| RoPE (YaRN) | 14.30 | 13.74 | 13.36 | 13.10 | 13.05 | 12.93 |
| Wall | 23.11 ± 2.87 | 24.51 ± 3.65 | 25.78 ± 3.98 | 26.22 ± 3.93 | 24.92 ± 3.60 | 22.72 ± 2.84 |
| PaTH | 13.57 ± 0.07 | 13.11 ± 0.07 | 12.79 ± 0.08 | 12.59 ± 0.07 | 12.61 ± 0.09 | 12.59 ± 0.09 |
| QDecay | 13.05 ± 0.02 | 12.60 ± 0.02 | 12.26 ± 0.02 | 12.03 ± 0.01 | 12.00 ± 0.01 | 11.92 ± 0.01 |
Similar patterns hold for language modeling after post-training. QDecay has the lowest perplexity at every length and keeps improving up to 128K.
We will add these results to the final revision.
QDecay training kernels
Forward + Backward time (ms) of FlashAttention-3 (FA3) + RoPE and of the QDecay kernel, measured on a sm_90 (H100) at a fixed 32K tokens per batch.
Using 12 heads, head dimension 64:
| Batch size | Sequence length | RoPE (ms) | QDecay (ms) | QDecay / RoPE |
|---|---|---|---|---|
| 32 | 1K | 0.799 | 0.732 | 0.92× |
| 16 | 2K | 1.107 | 1.109 | 1.00× |
| 8 | 4K | 1.783 | 1.858 | 1.04× |
| 4 | 8K | 3.073 | 3.339 | 1.09× |
| 2 | 16K | 5.737 | 6.250 | 1.09× |
Using 32 heads, head dimension 64:
| Batch size | Sequence length | RoPE (ms) | QDecay (ms) | QDecay / RoPE |
|---|---|---|---|---|
| 32 | 1K | 2.111 | 1.982 | 0.94× |
| 16 | 2K | 2.913 | 2.976 | 1.02× |
| 8 | 4K | 4.732 | 5.040 | 1.07× |
| 4 | 8K | 8.275 | 8.937 | 1.08× |
| 2 | 16K | 15.262 | 16.812 | 1.10× |
Using 16 heads, head dimension 96:
| Batch size | Sequence length | RoPE (ms) | QDecay (ms) | QDecay / RoPE |
|---|---|---|---|---|
| 8 | 4K | 3.364 | 3.265 | 0.97× |
| 2 | 16K | 11.018 | 10.804 | 0.98× |
Using 8 heads, head dimension 128:
| Batch size | Sequence length | RoPE (ms) | QDecay (ms) | QDecay / RoPE |
|---|---|---|---|---|
| 8 | 4K | 2.081 | 1.985 | 0.95× |
| 2 | 16K | 6.626 | 6.760 | 1.02× |
In summary, at the smaller head dimension of 64, the QDecay kernel is within 10% of FA3 + RoPE. At the larger head dimensions of 96 and 128, which are used when models are scaled up, QDecay matches FA3 + RoPE.
We will add these timings to the paper.
That being said, the training kernel code is not yet in a state ready for release, and we are still polishing it. We have already released our efficient decode kernels for QDecay in the supplementary material (Appendix G.2). We plan to release the training kernels upon publication.
We thank the reviewer for their patience and for the helpful feedback.
We appreciate all the careful and insightful feedback on our work by the reviewer. Please find our replies below.
We have added this comparison in Appendix E of the revised paper (Figure 13 and Table 9). We apply YaRN to the 800M RoPE model at inference time, without fine-tuning.
Here's a gist of the results.
log PPL on
pg19books:RULER S-NIAH accuracy averaged over three tasks:
YaRN stops RoPE's perplexity from diverging beyond the pretraining length, but QDecay still has lower perplexity and substantially better long-context recall.
We evaluated the 800M/30B models on QA and summarization tasks from LongBench.
Each table averages over the listed datasets and groups examples by context length in tokens.
Single-doc QA (
qasper,multifieldqa_en)Multi-doc QA (
hotpotqa,2wikimqa,musique)Summarization (
gov_report,qmsum,multi_news)Beyond the 4K pretraining length, QDecay is the best model in 8 of the 9 settings. Within 4K it is the best or within 0.2 points of the best.
We have added these results to the revised paper (Section 4.5, Table 2), with per-dataset results in Appendix H.6 (Tables 21 and 22).
We unfortunately do not evaluate on code understanding, since our pretraining data contains no significant amount of code data (i.e. FineWeb-Edu).