acceptodds
Under review as a conference paper at ICLR 2027

Forgetting without Forgetting: Query-Dependent Decays for Effective Long-Context Modeling

Abstract

Position encodings are an integral part of transformers, heavily influencing language modeling performance as well as capabilities like recall and extrapolation to long contexts. Yet no existing encoding excels at all three aspects. The ubiquitous encoding RoPE performs well within training lengths, but degrades beyond. Decay-based encodings (like ALiBi or FoX) can extrapolate better, as they favor recent tokens more strongly, but this hurts long-range recall. Removing position encodings entirely (NoPE) can improve recall, but hurts language modeling. To resolve this, we introduce Query-Dependent Decay (QDecay): a lightweight component that can be added on top of any decay-based encoding, enabling each query to independently control the intensity of the underlying decay. A query can choose to either favor recent tokens (like ALiBi), or "turn off" the decay completely when distant information matters (like NoPE). Critically, this choice is made independently for each query in the sequence – favoring recent tokens for one query does not affect the decay for the next. Attaching QDecay on top of the simple ALiBi decay, the resulting position encoding achieves strong performance on all three desiderata. On standard language modeling benchmarks, QDecay matches or outperforms other position encodings, including the popular RoPE. Next, QDecay achieves the best recall on RULER at 32K+ token context, improving on other encodings by 18+ accuracy points, and remains competitive at shorter lengths. Third, QDecay extrapolates well: on Books3, it matches or outperforms other encodings on validation perplexity at all context lengths, maintaining and even improving performance through 64K tokens (16× training length), whereas all other encodings degrade. Our interpretability ablations identify QDecay's ability to completely "turn off" the underlying decay as a key driver of recall. QDecay adds negligible (< 0.1%) overhead in parameters and is compatible with fast FlashAttention-based algorithms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
position encoding, decay-based position encoding, attention, long-context modeling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BKnyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces QDecay, a lightweight controller that can be added to any additive decay-based positional encoding, such as ALiBi or FoX. For each query, it predicts a nonnegative coefficient from the current token’s hidden state to strengthen, preserve, weaken, or fully remove the underlying decay, allowing token-level switching between ALiBi-style recency bias and NoPE-style attention without a distance penalty. The experiments show promising results at the 250M and 800M model scales, with a training context length of 4K.

Critical strengths

The idea is simple and directly addresses the problem: giving the current query control over decay allows the model to switch between favoring nearby information and reading distant information as needed.

The experiments cover language modeling, long-context recall, length extrapolation, and hybrid architectures, while ablation studies test the roles of query dependence and zero decay.

Critical weaknesses

The main results cover only the 250M and 800M model scales, and the paper reports only one training seed, without repeated runs or error bars. It remains unclear whether the gains are stable across runs and hold at larger scales.

The paper does not report comparisons with RoPE using common context-extension methods, such as position scaling or YaRN. The results therefore support an advantage over the tested RoPE baseline, but not necessarily over widely used long-context alternatives.

Long-context recall is mainly tested on RULER, while the real-world recall tasks use short contexts. More results on real long-document tasks, such as question answering, summarization, and code understanding, would strengthen the evaluation.

In Table 17, QDecay’s S-NIAH-1 accuracy at 8K (78.7) is much lower than at 16K (94.3). The authors should explain this change and provide repeated evaluations or confidence intervals to help identify its cause.

The FlexAttention training implementation takes 22.5 hours, compared with 16.8 hours for RoPE, so training cost should be clearly separated from claims about fast inference. The authors claim that optimized FlashAttention training kernels reduce the slowdown to below 10%, but state, “We plan to release these kernels after the deadline,” making this claim difficult to independently verify from the current submission.

Questions for authors

See above

4 comments

We appreciate all the careful and insightful feedback on our work by the reviewer. Please find our replies below.

comparisons with RoPE using YaRN

We have added this comparison in Appendix E of the revised paper (Figure 13 and Table 9). We apply YaRN to the 800M RoPE model at inference time, without fine-tuning.

Here's a gist of the results.

log PPL on pg19 books:

Model1K2K4K8K16K32K
RoPE2.9102.8542.8214.2255.6886.303
RoPE + YaRN2.9102.8542.8212.8042.8222.901
QDecay2.8772.8202.7802.7462.7192.717

RULER S-NIAH accuracy averaged over three tasks:

Model8K16K32K64K
RoPE0.00.00.00.0
RoPE + YaRN55.230.822.016.7
QDecay78.977.759.040.0

YaRN stops RoPE's perplexity from diverging beyond the pretraining length, but QDecay still has lower perplexity and substantially better long-context recall.

More results on real long-document tasks, such as question answering, summarization, and code understanding

We evaluated the 800M/30B models on QA and summarization tasks from LongBench.

Each table averages over the listed datasets and groups examples by context length in tokens.

Single-doc QA (qasper, multifieldqa_en)

Model0–4K4–8K8–16K16K+
RoPE6.72.22.52.0
FoX13.58.56.76.9
Wall9.56.86.87.9
PaTH11.26.97.38.1
QDecay13.49.09.712.5

Multi-doc QA (hotpotqa, 2wikimqa, musique)

Model0–4K4–8K8–16K16K+
RoPE8.80.70.80.3
FoX10.97.94.43.5
Wall10.33.72.42.1
PaTH11.47.15.63.7
QDecay11.29.65.85.4

Summarization (gov_report, qmsum, multi_news)

Model0–4K4–8K8–16K16K+
RoPE9.54.54.55.1
FoX6.610.06.67.0
Wall9.212.010.410.5
PaTH9.38.37.49.6
QDecay11.011.711.211.2

Beyond the 4K pretraining length, QDecay is the best model in 8 of the 9 settings. Within 4K it is the best or within 0.2 points of the best.

We have added these results to the revised paper (Section 4.5, Table 2), with per-dataset results in Appendix H.6 (Tables 21 and 22).

We unfortunately do not evaluate on code understanding, since our pretraining data contains no significant amount of code data (i.e. FineWeb-Edu).

QDecay’s S-NIAH-1 accuracy at 8K (78.7) is much lower than at 16K (94.3)

We re-evaluated the 800M QDecay model on S-NIAH-1 with five evaluation seeds. The table reports the mean ± standard deviation.

1K2K4K8K16K32K64K
S-NIAH-1100.0 ± 0.0100.0 ± 0.0100.0 ± 0.084.6 ± 6.994.4 ± 1.596.6 ± 1.596.2 ± 1.3

Averaged over 5 evaluation seeds, the gap between 8K and 16K shrinks but still persists. We believe this could be a training artifact of the particular training initialization.

Indeed, a QDecay model trained with a different seed shows no dip at 8K:

1K2K4K8K16K32K64K
S-NIAH-1100.0100.0100.0100.099.398.097.0

hold at larger scales

We report results at two scales, 250M/10B and 800M/30B. Training larger models is unfortunately beyond our compute budget.

To make it easy for the community to scale QDecay further, we plan to release our efficient FlashAttention-based implementation.

QDecay training kernels

We plan to release the training kernel in the next one or two days. Thanks for your patience!

It remains unclear whether the gains are stable across runs

We are training additional seeds, and we hope to share the results soon. Thanks for your patience!

It remains unclear whether the gains are stable across runs

We thank the reviewer for their patience.

We trained two additional seeds each for QDecay, PaTH and Wall at the 800M/30B scale. Here are the results over three total seeds per model (mean ± standard deviation). Due to limited time and compute, we were not able to train RoPE at more seeds, but we include its performance for reference comparison.

Language modeling and recall:

ModelLM ppl. ↓Recall ↑
RoPE18.054.1
Wall17.1 ± 0.056.6 ± 1.9
PaTH16.7 ± 0.256.3 ± 0.9
QDecay16.3 ± 0.157.6 ± 1.8

RULER accuracy, averaged across tasks:

Model1K2K4K8K16K32K64K
RoPE (YaRN)69.757.150.440.124.317.08.4
Wall79.7 ± 4.970.8 ± 2.959.4 ± 11.562.3 ± 1.555.9 ± 0.516.2 ± 16.70.1 ± 0.2
PaTH82.3 ± 3.374.3 ± 1.762.0 ± 6.453.7 ± 19.641.0 ± 16.114.8 ± 3.11.7 ± 0.6
QDecay81.6 ± 4.974.3 ± 0.163.8 ± 8.852.3 ± 11.637.2 ± 12.025.5 ± 9.615.3 ± 5.0

Within the 4K pretraining length, QDecay is on par with or better on both RULER and language modeling. At extremely long contexts of 32K and 64K, QDecay outperforms all encodings by at least ~10 and ~15 absolute points respectively across three seeds.

This stable performance at extreme contexts of 32K and 64K agree with language modeling on long books. On pg19, QDecay has the lowest perplexity at every length and stays stable up to 64K across seeds, whereas the perplexity of Wall and PaTH rises at 64K with a large variance across seeds.

Further, Wall and PaTH lose performance at these extreme lengths, consistent with the contraction of the attention logits of old keys that we discuss in Section 7 and measure for Wall in Appendix D.

Here's PPL on pg19 long books:

Model4K8K16K32K64K
RoPE16.8068.35295.44546.13737.12
RoPE (YaRN)16.8016.5116.8118.1920.59
Wall16.77 ± 0.2016.24 ± 0.1815.84 ± 0.1716.09 ± 0.3126.81 ± 14.29
PaTH16.57 ± 0.0815.99 ± 0.0515.53 ± 0.0215.62 ± 0.0917.02 ± 1.02
QDecay16.22 ± 0.1115.70 ± 0.1215.29 ± 0.1115.24 ± 0.1015.24 ± 0.06

results after post training i.e. long context extension

Moreover, following standard practice before deployment at long contexts, we post-train all models. Specifically, we perform a long-context extension phase on long 128K-token documents from the train split of the pg19 dataset (note that the base models were pretrained on 4K). For RoPE, we long-context extend using two methods and report both: Adjusted Base Frequency (ABF) and YaRN. We grid search the best θ\theta for RoPE.

Here RULER S-NIAH average (over three tasks) after post training (i.e. long context extension to 128K):

Model8K16K32K64K128K
RoPE (ABF θ=1M)64.049.342.721.314.7
RoPE (YaRN)42.737.333.330.720.0
Wall28.4 ± 20.28.0 ± 10.61.8 ± 2.01.8 ± 3.14.4 ± 5.4
PaTH80.4 ± 6.072.4 ± 5.443.1 ± 30.623.1 ± 13.712.4 ± 5.6
QDecay80.4 ± 14.764.9 ± 13.450.2 ± 20.244.0 ± 20.237.3 ± 13.5

QDecay performs great again at extreme contexts of 64K and 128K. Multiplicative decays like Wall (and contractions like PaTH) suffer at long contexts despite being trained on these long sequences, due to their fundamental nature of contraction, which results in "confusion" in attention (see Sec. 7 and App. D). Hence, even RoPE outperforms these decays once one does training on long sequences.

Interestingly, Wall does not improve despite training on long documents, possibly because its accumulated decay pushes the attention logits of old keys toward zero (App. D), resulting in near-uniform attention scores and poor optimization at long contexts.

This suggests that not all position encodings may be suited for post-training, including long-context extension or RL.

Here's pg19 PPL after post training (i.e. extension to 128K):

Model4K8K16K32K64K128K
RoPE (ABF θ=1M)14.1613.6313.2613.0012.9612.85
RoPE (YaRN)14.3013.7413.3613.1013.0512.93
Wall23.11 ± 2.8724.51 ± 3.6525.78 ± 3.9826.22 ± 3.9324.92 ± 3.6022.72 ± 2.84
PaTH13.57 ± 0.0713.11 ± 0.0712.79 ± 0.0812.59 ± 0.0712.61 ± 0.0912.59 ± 0.09
QDecay13.05 ± 0.0212.60 ± 0.0212.26 ± 0.0212.03 ± 0.0112.00 ± 0.0111.92 ± 0.01

Similar patterns hold for language modeling after post-training. QDecay has the lowest perplexity at every length and keeps improving up to 128K.

We will add these results to the final revision.

QDecay training kernels

Forward + Backward time (ms) of FlashAttention-3 (FA3) + RoPE and of the QDecay kernel, measured on a sm_90 (H100) at a fixed 32K tokens per batch.

Using 12 heads, head dimension 64:

Batch sizeSequence lengthRoPE (ms)QDecay (ms)QDecay / RoPE
321K0.7990.7320.92×
162K1.1071.1091.00×
84K1.7831.8581.04×
48K3.0733.3391.09×
216K5.7376.2501.09×

Using 32 heads, head dimension 64:

Batch sizeSequence lengthRoPE (ms)QDecay (ms)QDecay / RoPE
321K2.1111.9820.94×
162K2.9132.9761.02×
84K4.7325.0401.07×
48K8.2758.9371.08×
216K15.26216.8121.10×

Using 16 heads, head dimension 96:

Batch sizeSequence lengthRoPE (ms)QDecay (ms)QDecay / RoPE
84K3.3643.2650.97×
216K11.01810.8040.98×

Using 8 heads, head dimension 128:

Batch sizeSequence lengthRoPE (ms)QDecay (ms)QDecay / RoPE
84K2.0811.9850.95×
216K6.6266.7601.02×

In summary, at the smaller head dimension of 64, the QDecay kernel is within 10% of FA3 + RoPE. At the larger head dimensions of 96 and 128, which are used when models are scaled up, QDecay matches FA3 + RoPE.

We will add these timings to the paper.

That being said, the training kernel code is not yet in a state ready for release, and we are still polishing it. We have already released our efficient decode kernels for QDecay in the supplementary material (Appendix G.2). We plan to release the training kernels upon publication.

We thank the reviewer for their patience and for the helpful feedback.

Reviewer_9x9HOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes QDecay, a context-aware modification of ALiBi which allows the model to dynamically tune the strength of the decay by multiplying the ALiBi bias by a factor dependent on the query (ReLU(w^T x_i + c)). The method is evaluated on the RULER benchmark and perplexity, managing to outperform or match performance of the selected baselines.

Critical strengths
  1. Improvements on long-range capabilities: QDecay meaningfully extends the usable context window of ALiBi-based models at modest computational cost. Unlike most of the compared methods, it maintains non-trivial performance on long-range recall tasks well beyond its training length, while remaining cheaper to train than the other strong long-context baselines (PaTH, Wall). This is the most consequential result in the paper, as it addresses a real practical limitation of the original ALiBi design.
  2. Simple, natural extension: Although similar to several published works, letting the model switch dynamically between local attention and retrieval is a simple and natural extension of ALiBi, which makes it easy to adopt.
Critical weaknesses
  1. Lack of comparable baselines / missing ablations: The method is a modification of ALiBi that makes it context-aware and extends it with the ability to attend to arbitrarily long contexts (via NoPE). However, the experiments do not compare against other methods with similar behavior (e.g., CABLE, TRA, DAPE, ...). Most critically, it is missing an ALiBi + NoPE baseline, which is the most direct and natural comparison for the proposed method.
Questions for authors
  1. 32K/64K reporting: Please report per-task RULER results at 32K/64K for all 800M models. QDecay trails PaTH and Wall on most tasks at 8K/16K and leads only at these two lengths, which appear only in Fig. 5b. Is Wall’s drop to zero between 16K and 32K a model failure or an evaluation artifact?
  2. Missing ALiBi + NoPE baseline: How does QDecay compare to a static ALiBi + NoPE baseline? Could you provide this comparison to disentangle the benefit of dynamic, per-query gating from simply having heads with zero decay?
7 comments

We appreciate all the careful and insightful feedback on our work by the reviewer. Please find our replies below.

Most critically, it is missing an ALiBi + NoPE baseline ...

We thank the reviewer for this suggestion. We trained the requested ALiBi + NoPE baseline at the 250M/10B scale of Table 1, interleaving ALiBi and NoPE layers at a 1:1 ratio.

ModelLM ppl. ↓Recall ↑RULER 1K ↑RULER 2K ↑RULER 4K ↑RULER 8K ↑RULER 16K ↑
ALiBi + NoPE (1:1)31.239.947.755.451.21.00.0
QDecay-ALiBi29.142.860.857.356.149.433.6

QDecay outperforms the ALiBi-NoPE baseline, with the largest gains beyond the 4K training length. We will add this baseline in the paper.

In fact, we already report a stronger baseline in the paper: the "learnt ALiBi ablation" in Appendix A.1 (Table 4) -- where each head learns its own fixed slope instead of using a hand-picked mix of ALiBi and NoPE. That variant also fails to extrapolate on RULER beyond the pretraining length.

Is Wall’s drop to zero between 16K and 32K a model failure or an evaluation artifact?

We thank the reviewer for the insightful question. This drop is a model failure.

Wall's accumulated decay contracts the attention logits of old keys toward zero rather than toward negative infinity, as we discuss in Section 7, and measure empirically in Figure 12 in the revised paper.

This means that old keys are never truly "forgotten". Instead of being discarded, they keep participating in attention with similar, nonzero weights.

As a result, at long contexts of 32K tokens, old keys end up with nearly the same logit and the query can no longer tell them apart, so recall fails.

Once the context is long enough at 64K tokens, these stale keys together may start consuming the majority of the attention mass, so the attention output becomes a near-uniform average over them and the model breaks down. The RULER generations turn into gibberish and repeating phrases, and at the same time, the perplexity on long books blows up as well.

We have added this analysis as Appendix D of the revised paper.

We note that a similar argument may apply to PaTH attention, whose accumulated Householder transformations are also contractive. This contraction acts more slowly, which is consistent with PaTH retaining RULER accuracy for longer than Wall before eventually collapsing as well (Section 7).

Finally, we evaluate all models with the same evaluation harness, and other encodings such as QDecay, PaTH and FoX retain non-zero RULER accuracy at 32K and 64K under it. This rules out an evaluation artifact.

Please report per-task RULER results at 32K/64K for all 800M models

We apologize for our oversight -- we now have added the per-task RULER results at 32K and 64K for all 800M models to Tables 19 and 20 in the appendix of the revised paper (new numbers are shown in blue).

comparison with CABLE

We thank the reviewer for pointing us to CABLE. It is closely related to QDecay. Both methods let the current query adjust the strength of the decay.

However, CABLE uses a softplus weight to scale a learned cumulative bias.

On the other hand, QDecay uses a ReLU weight that can reach exactly zero, letting the query turn off decay completely and recover a “pure” NoPE attention in that head.

The analysis in the paper (Section 5) explains why this difference in parameterization is crucial for long-context recall. Turning off decay completely lets a query attend to very distant tokens without any distance penalty. Our comparison with CABLE below emphasizes this importance yet again.

On 250M parameters, 10B training tokens, and a 4K training context:

ModelFineWeb val ppl. ↓RULER 1K ↑RULER 2K ↑RULER 4K ↑RULER 8K ↑RULER 16K ↑
CABLE11.0966.355.441.834.226.4
QDecay11.2960.857.356.149.433.6

QDecay-ALiBi wins on RULER from 2K through 16K, as we expect at longer context lengths. CABLE has lower validation perplexity and higher RULER accuracy at 1K. We attribute CABLE’s advantage on these in-distribution metrics mainly to its more flexible FoX-style decay compared with ALiBi’s fixed linear decay.

More broadly, QDecay controller lets the current token control the strength of any additive bias:

sij=qi⊤kj+δi,bij,δi≥0.s_{ij} = \boldsymbol{q}_i^\top \boldsymbol{k}_j + \delta_i \\, b_{ij}, \qquad \delta_i \ge 0.

As a result, one can apply the QDecay controller to ALiBi, or FoX (see Appendix A.6), or any future additive encoding. We also show that QDecay complements multiplicative encodings by alternating QDecay layers with Wall or PaTH layers (Appendix B). Further, the QDecay controller can be applied directly to a multiplicative decay, and we provide exploratory results for QDecay on a scalar version of Wall in the revised paper (Appendix F).

In summary, two things set QDecay apart from CABLE:

  • Understanding of why to turn off decay. The CABLE paper mentions that its query weight is added for flexibility, to "dampen or amplify" the bias, without any motivation or intuition for why this helps. In contrast, our analysis and understanding experiments (Section 5) show that turning off decay is what drives long-context recall. This understanding is what motivates read-time control of a decay in the first place, and the ReLU parameterization in particular. Further, this understanding is not tied to one encoding, and we believe it can help the community develop better encodings.
  • Generality. QDecay is thus a general idea that can be instantiated with any decay, be it additive or multiplicative. As one example, we show how to extend it to multiplicative encodings, with new exploratory results in Appendix F of the revised paper. CABLE, in contrast, is presented as one specific additive encoding rather than a general idea, and does not discuss how its query weight could extend to other decays, such as multiplicative ones.

We will add CABLE to the related work and clarify how it compares with the broader QDecay controller.

CABLE: https://arxiv.org/abs/2503.08067

comparison with TRA and DAPE

TRA and DAPE both make the position encoding depend on both the query and the key jointly, that is, on each (i, j) pair. Specifically, the bias is computed from the query-key attention scores themselves. However, this joint dependence comes at a significant cost for both methods, adding substantial work across query-key pairs. Further, the bias is computed from the attention scores, and existing implementations of both methods materialize the full N x N score matrix, which makes them hard to scale.

TRA sets the decay in an attention block using a running count of keys that pass some threshold in newer blocks, and carrying this count across blocks forces sequential processing for each query, which limits parallelism. Further, the authors themselves describe a FlashAttention style implementation as "highly speculative" and "not trivial" (Appendix D), and their official implementation materializes the full N x N score matrix (Appendix A). All of the above is reflected in their experiments as well: the largest model has about 80M parameters and is trained on a tiny context length of 128 tokens.

DAPE runs an MLP for every query-key pair attention score, adding substantial compute because each new query needs a fresh bias for every past key. Moreover, applying the MLP on the attention scores requires the full N x N scores, and the paper does not discuss FlashAttention compatibility at all. Its training length is at most 1K tokens. QDecay on the other hand, predicts one decay weight per query and head, and reuses it across all past keys, so each block can apply the decay weight independently – making it fully compatible with a FlashAttention-style algorithm.

We will discuss both in our related work, and thanks a lot for bringing these works to our notice!

TRA: https://arxiv.org/abs/2503.23174

DAPE: https://arxiv.org/abs/2405.14722

Reviewer_D7B1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a novel decaying position embeddings which try to improve on language modelling, recall and length extrapolation all at once. The paper's proposal is a modification on top of Alibi which learns the position wise forget at "read time" based on Q matrix.

The paper has extensively evaluated its proposal. First, through dedicated benchmark evaluations, the paper confirms that the proposed method improves on the target metrics. The paper also presents other experiments that are relevant for the paper: 4.5. investigates if the proposal still holds when integrating into hybrid Transformers; 4.6. investiates the effect of the proposed mechanisms on the multiplicative position embeddings; section 5 presents what the resulting positional embeddings become at different positions for different tasks.

Critical strengths

As said in the paper summary, the paper has performed extensive experiments to investigate into paper's claim. The experiments cover various state of the art techniques, various benchmarks and different architectures. 4.5 and 4.6 offers additional insights on the applicability for the proposed method.

I found the insights on design on the query helpful and the comparison with FOX gives good support on this claim.

Critical weaknesses

I think a high level objective for LLMs can be to propose the best mechanims for general tasks for positional embedding designs. The paper has not offered how to unify multiplcative settings and additional settings (admittedly not the paper's main focus); the paper proposed a slightly more complex mechanism on top of existing techniques. I would appreciate if the paper can also discuss the simplification and unification aspects.

Questions for authors

None

3 comments

We appreciate the positive feedback on our work by the reviewer. Please find our reply below.

The paper has not offered how to unify multiplcative settings and additional settings (admittedly not the paper's main focus); the paper proposed a slightly more complex mechanism on top of existing techniques. I would appreciate if the paper can also discuss the simplification and unification aspects

Thank you for raising this point.

QDecay is a generic controller: each query predicts one scalar δi≥0\delta_i \ge 0 that sets how strongly an underlying decay is applied.

For any additive bias bijb_{ij}, QDecay rescales it:

sij=qi⊤kj+δi,bij.s_{ij} = \boldsymbol{q}_i^\top \boldsymbol{k}_j + \delta_i \\, b_{ij}.

This is the setting we study extensively in the paper.

The same controller also extends to multiplicative decays, that we briefly discuss future work (Section 7). We will add following extension to the revised paper. For any multiplicative decay Mij\boldsymbol{M}_{ij}, QDecay can control the decay through:

sij=qi⊤(Mij)δikj.s_{ij} = \boldsymbol{q}_i^\top (\boldsymbol{M}_{ij})^{\delta_i} \boldsymbol{k}_j.

Taking Wall as an example, where Mij=diag⁡(⨀t=j+1iαt)\boldsymbol{M}_{ij} = \operatorname{diag}\big(\bigodot_{t=j+1}^{i} \boldsymbol{\alpha}_t\big) is an accumulated channel-wise decay, QDecay raises each channel's decay to the power δi\delta_i:

(Mij)δi=diag⁡(⨀t=j+1iαtδi).(\boldsymbol{M}_{ij})^{\delta_i} = \operatorname{diag}\Big(\bigodot_{t=j+1}^{i} \boldsymbol{\alpha}_t^{\delta_i}\Big).

As in the additive case, δi=0\delta_i = 0 turns off Wall, and δi=1\delta_i = 1 restores it. Values between 0 and 1 weaken it. We take the first steps in this direction, and provide exploratory results for QDecay on a scalar version of Wall (Section 7 and Appendix F). An exhaustive study of QDecay on multiplicative encodings requires further research, which we leave for future work.

QDecay can thus control both additive and multiplicative decays.

This is the unification and simplification: one simple controller that can be applied on top of any decay-based encoding of choice, be it additive or multiplicative.

Please let us know if we can improve this simplication and/or unification discussion further. Thanks!

Thanks for the explanations. I also quickly looked at the other reviews. The authors have addressed my concerns and answered properly for other required experimental settings which should strengthen the paper. My position of weak acceptance thus remains.

Thank you for helping us strengthen the paper. Please let us know if you have any lingering concerns or if further revisions would help you consider raising the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.