acceptodds
Under review as a conference paper at ICLR 2027

FUSED BUT NOT RELATIVE: LOW-RANK ATTENTION BIASES SILENTLY DROP TRANSLATION INVARIANCE

Abstract

A rank- additive attention bias written as , with , can be folded into the queries and keys, allowing compatible unmodified FlashAttention kernels to compute biased attention without materializing the bias. Alternative fused solutions require bias-specific kernel support or compilation. Low-rank factorization supports fused training but does not guarantee relativity: may depend on rather than only on . % Prior work proposes learning these factors from initialization but does not evaluate this variant; its reported instantiations use closed forms, factorize a pretrained table, or fit an existing bias, thereby enforcing or inheriting relativity. When trained freely in vision, only of the learned bias energy remains displacement-dependent, compared with for the unfactorized relative bias it replaces. % We restore relativity by generating factors from learned frequencies, whose inner product reduces to and therefore depends only on displacement by algebraic identity. A learned phase lifts the resulting even-symmetry restriction, still depending only on displacement, without increasing factor width. Kernel and computational cost are unchanged, while positional parameters become independent of window size and context length; in Swin-T at window , they fall from to . % At matched factor width, architecture, and training recipe, restoring relativity reduces language-model perplexity degradation at the training context by roughly across ranks –, while using fewer positional parameters and improving in-distribution perplexity. In vision, it improves transfer to semantic segmentation and detection, and out-of-domain generalization at the larger window, while the phase variant strengthens robustness at both. These results provide a practical route to learned relative biases within fused attention across vision, language, and other position-structured domains.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
FlashAttention, relative positional bias, low-rank attention, translation invariance, vision transformers, resolution generalization, context-length extrapolation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_3ggmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies low-rank additive attention biases that can be appended to queries and keys and computed using standard fused-attention kernels. The authors observe that freely learned low-rank factors do not guarantee that the resulting bias depends only on relative displacement. They propose generating the factors from learned sinusoidal frequencies, which guarantees relative structure through the cosine-difference identity. Experiments are conducted on a tiny language model and Swin-T.

Critical strengths
  1. The core idea is straightforward. The proposed parameterization is mathematically valid and compatible with standard fused-attention kernels.
Critical weaknesses
  1. The presentation must be improved. The current manuscript appears to be a simple combination of different paragraphs, with loosely connected discussions of many different concepts like kernel fusion, relativity, parameter efficiency, robustness, and extrapolation.

  2. The research significance and novelty are limited. The paper does not establish why an attention bias must depend exclusively on relative position. Absolute-position information may be useful, and the necessity of enforcing exact relativity remains insufficiently validated. The method itself closely resembles the standard RoPE construction, except that positional features are appended to queries and keys to produce an additive bias. This also enlarges the Q/K dimension and introduces additional computation, weakening the claimed efficiency motivation.

  3. The experiments are too toy to support the claims. The language evaluation uses only a 51M model on WikiText-103 trained at context 512, while the vision evaluation is restricted to Swin-T. This evidence is inadequate for the broad claims.

  4. The improvements are marginal or negative. The proposed variants remain below full SwinV2 in the available comparisons. In language modeling, the method extrapolates substantially worse than ALiBi. Its efficiency is also not clearly better than modern kernel-based alternatives such as FlexAttention, while the appended positional dimensions introduce extra QK computation.

Questions for authors

N/A

7 comments
AuthorsOpenReview ↗

Response To Reviewer 3 concerns Part 1

We thank the reviewer for the candid and constructive critique, which prompted us to clarify the paper's positioning, strengthen the experimental evidence, and better distinguish our contribution from existing alternatives. We address each concern in detail below.

W1. Presentation. We are reorganizing the manuscript to improve and address all identified issues of flow and clarify the scope of each claim. We will upload the revised manuscript in the next few days.

W2.1. Why enforce relativity?. We do not claim that every attention bias should be relative; absolute position can also be useful. However, prior controlled comparisons demonstrate benefits from relative structure in several settings. Shaw et al.\ (2018) report gains of 0.30.3 BLEU with the base Transformer and 1.31.3 BLEU with the big Transformer over absolute representations on WMT14 English--German. Liu et al.\ (2021) report that Swin's relative bias improves ImageNet-1K accuracy by 0.80.8 points and ADE20K segmentation by 2.92.9 mIoU over absolute embeddings, while Zhao et al.\ (2021) report an improvement from 66.566.5 to 70.470.4 mIoU with relative rather than absolute position encoding in Point Transformer.

Our experiments support this scoped claim rather than a universal preference for relativity. On the training distribution, different factor generators often perform similarly. The clearest benefit appears when position-specific structure must transfer beyond the training regime: freely learned factors degrade by 1111--21×21\times under context extension, compared with 1.41.4--3×3\times for ours. In Swin-T, at comparable ImageNet-1K accuracy, both relative variants improve robustness over freely learned factors. The phase variant improves all five robustness measures, including mean gains of 2.02.0 points on ImageNet-A, 1.51.5 on ImageNet-R, and 1.41.4 on ImageNet-Sketch. Thus, we do not claim that relativity always improves performance; we show that preserving relative structure prevents specific failures under context and distribution shift.

Enforcing relativity helps when displacement or geometry matters and may extend to video, scientific meshes, and molecular structures; systematic evaluation is left future work.

W2.2. Relationship to RoPE. We agree that the geometric construction is closely related to RoPE, and the revision states this explicitly. The distinction is that RoPE rotates content-dependent queries and keys, whereas our method applies the same geometry to appended constant channels, producing a content-independent additive bias. To test whether the two mechanisms are redundant or complementary, we trained RoPE alone, RoPE with freely learned factors, and RoPE with our relative bias using the same recipe and three seeds. All three have nearly identical perplexity at context 512512 (24.25±0.1024.25\pm0.10, 24.21±0.0324.21\pm0.03, and 24.21±0.0324.21\pm0.03, respectively). At 8,1928{,}192 tokens, however, degradation falls from 18.8±0.9×18.8\pm0.9\times for RoPE alone and 16.6±0.4×16.6\pm0.4\times with freely learned factors to 9.7±1.2×9.7\pm1.2\times with our bias. Thus, the learned relative bias complements RoPE rather than replacing it. More broadly, our framework provides a fused way to encode relative relationships without modifying token content, which may be useful when geometry should influence attention independently of the features being compared.

W2.3. Additional Q/K computation and efficiency. Appending bias factors adds RR query and key channels; at R=16R=16, the head dimension grows from 3232 to 4848. This overhead applies to all factorized fused biases, including FlashBias and RIB, and is included in our measurements.

On an H200 at window 3232 (Swin-T, 256256px, batch 6464), our method retains 87%87\% of no-bias throughput. Relative to eager SwinV2, it is 2.8×2.8\times faster at patch 44 during inference (3,0423{,}042 versus 1,0841{,}084 img/s). During training, it is 2.1×2.1\times faster at patch 44 (836836 versus 394394 img/s) and 2.6×2.6\times faster at patch 22 (229229 versus 8787 img/s), while using only 1717--19%19\% of the memory. It also outperforms dense-bias streaming through memory-efficient SDPA, reaching 836836 versus 699699 img/s at patch 44 and 229229 versus 177177 img/s at patch 22. Throughput is comparable to FlexAttention: ours is approximately 2%2\% faster at patch 44 and 7%7\% faster at patch 22 during training.

Unlike FlexAttention, which compiles bias-specific score logic, our method expresses the bias through the standard query--key interface and runs on compatible, unmodified FlashAttention-style kernels. It requires no bias-specific compilation, keeps positional parameters independent of window and context size, and achieves comparable throughput while supporting configurations for which FlexAttention did not compile. FlexAttention supports arbitrary score programs; ours provides a compact, learnable relative bias portable across fused-attention backends.

AuthorsOpenReview ↗

Response To Reviewer 3 concerns Part 2

W3. Experimental scale and domain breadth. The submitted 5151M language model and Swin-T were chosen to support controlled comparisons and test the identified failure and proposed solution across language and vision. They were not intended to establish behavior at large model scales. The effects within these settings are nevertheless large: ours degrades by 1.41.4--3×3\times under context extension, compared with 1111--21×21\times for free factors and 6×6\times for RIB for the unconstrained alternatives.

We expand experiments and analysis along three axes. First, at GPT-2-small scale (124124M parameters; one seed per method), the ordering persists. At context length 512512, freely learned factors, RIB, and our method achieve perplexities of 22.3122.31, 22.3722.37, and 22.0822.08, respectively; at 8,1928{,}192, their degradation factors are 21.1×21.1{\times}, 8.2×8.2{\times}, and 3.4×3.4{\times}. We treat this as an ordering check and do not interpret the small in-distribution differences.

Second, the advantage persists at longer training contexts. When trained at length 1,0241{,}024, freely learned factors degrade by 7.55×7.55{\times} at 8,1928{,}192, compared with 2.15×2.15{\times} for ours. When trained at length 2,0482{,}048 using the same tokens per step and total training budget, freely learned factors achieve 24.124.1 perplexity in distribution and degrade by 15.3×15.3{\times} at 32,76832{,}768. Ours achieves 23.123.1 and degrades by only 3.0×3.0{\times}. Thus, under the same 16×16{\times} context extension, our degradation remains 3.0×3.0{\times} whether trained at length 512512 or 2,0482{,}048.

Third, we add Point Transformer V3 on S3DIS Area~5 (4646M parameters, two seeds per arm; Table Below. Our fused relative bias improves the stock model by 0.60.6--1.31.3 mIoU at patch sizes 128128--1,5361{,}536 and remains feasible at substantially larger patches. At the default patch size of 1,0241{,}024, freely learned factors do not improve the stock model (70.670.6 versus 70.970.9 mIoU), whereas our symmetric bias reaches 72.272.2 and the phase variant performs best at 72.772.7. RIB also reaches 72.272.2, showing that generating positional factors is important in this in-distribution setting, while the additional gain from phase indicates value in representing directional relative geometry. These results extend the applicability of our fused parameterization to three-dimensional point clouds; the matched language control provides the direct isolation of exact relativity.

The dense relative-position table performs slightly better where it fits (71.771.7 versus 71.171.1 mIoU at patch 128128), but does not fit on a 141141,GB GPU at patch 1,0241{,}024 and the standard batch size. Our factorized bias remains feasible there, requiring 15.815.8,GB and reaching 72.272.2 mIoU. Thus, the factorized form enables learned relative geometry at neighborhood sizes where the dense bias is impractical.

Patch sizePTv3 baseline (xCPE)Ours (symmetric), R=16R=16Gain
12870.3671.08+0.72
51271.5872.19+0.61
1,02470.8672.18+1.32
1,53671.1172.25+1.14
2,04871.7471.69−0.05
4,09671.4871.58+0.10
ParameterizationmIoUGain over PTv3 baseline
Freely learned factors70.61−0.25
RIB72.15+1.29
Ours + phase72.70+1.84

These experiments demonstrate applicability across model size, training length, patch scale, and a third domain, while also identifying settings in which exact relativity provides limited additional benefit. Evaluation on substantially other larger models and datasets remains future work.

AuthorsOpenReview ↗

Response To Reviewer 3 concerns Part 3

W4. Performance and efficiency trade-offs. Our method is designed to provide a learnable relative bias through standard fused-attention interfaces, not to dominate every specialized bias or kernel. The relevant comparison depends on the setting: SwinV2 provides a dense vision bias, ALiBi supplies a fixed monotone language bias, and FlexAttention compiles arbitrary score modifications. Our method instead provides a compact, task-specific relative bias when the appropriate function must be learned and materializing a dense bias is impractical.

Comparison with SwinV2. The factorized variants do not uniformly outperform full SwinV2, and we will clarify this positioning. Their ImageNet-1K accuracy is close to SwinV2-T (82.5682.56 and 82.7382.73 for ours and 82.76/82.6982.76/82.69 with phase, versus 82.8082.80), while the robustness comparison is mixed: ours with rank 32 is higher on ImageNet-V2 and mCE but lower on ImageNet-A, ImageNet-R, and ImageNet-Sketch. This is not a fully controlled comparison because the SwinV2 reference also differs in window size and use of the shift mask.

The controlled comparison is against freely learned factors at matched factor width and training recipe. The phase variant improves all five robustness measures, including mean gains of 2.02.0 points on ImageNet-A, 1.51.5 on ImageNet-R, and 1.41.4 on ImageNet-Sketch. The factorized representation also enables larger windows and denser tokenization where the dense SwinV2 bias becomes memory-limiting. Our claim is therefore an improved factorized alternative with a different accuracy--efficiency trade-off, not universal superiority to SwinV2.

Comparison with ALiBi. ALiBi is the stronger specialized choice for one-dimensional extrapolation when its monotone form is appropriate. Our bounded cosine bias is learnable across spatial and geometric domains but, by itself, extrapolates less effectively. To test whether this gap arises from factorization or functional form, we append FlashBias's existing exact rank-22 factorization of ALiBi's linear term to our learned cosine channels. The combined bias achieves 0.91×0.91\times degradation at 8,1928{,}192 tokens for every tested cosine width, comparable to ALiBi's 0.92±0.01×0.92\pm0.01\times, with comparable in-distribution perplexity. The original gap therefore comes from the absence of a monotone component rather than from relative factorization. Our framework can incorporate such a component when its form is known while retaining learned relative structure for other applications.

Comparison with FlexAttention and additional Q/K computation. Appending the factors increases the query and key dimension from 3232 to 4848 at R=16R=16, and this overhead is included in all reported measurements. Despite the additional computation, our method retains 87% of no-bias throughput. It is 2.8×2.8\times faster than eager SwinV2 attention while using 17% of its memory, and 15%15\% faster than streaming the dense bias through memory-efficient SDPA.

Our throughput is within 22--3%3\% of FlexAttention, so we do not claim a substantial speed advantage over it. The distinction is how the bias reaches the kernel. FlexAttention compiles a specified score modification into bias-specific kernel logic, whereas our bias uses ordinary augmented queries and keys and runs through compatible, unmodified FlashAttention-style kernels. It therefore requires no compilation for a particular bias form, keeps positional parameters independent of window size and context length, and can be used across supported fused-attention backends.

Taken together, these results position our method as a practical option within the accuracy--efficiency design space. It trades the unrestricted form of a dense or compiled bias for a compact, learnable relative parameterization that works through standard fused-attention interfaces. This enables task-specific relative biases at window, context, and neighborhood sizes where dense materialization is impractical, while remaining compatible with specialized components such as ALiBi when their form is appropriate.

Reviewer_SaZdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the loss of relative positional structure when attention biases are represented using freely learned low-rank factors for efficient attention computation. The authors propose a deterministic parameterization based on learnable frequencies and phases, using sine–cosine factors to guarantee that the resulting attention bias depends only on relative displacement while remaining compatible with FlashAttention. Through language-model context extrapolation, visual robustness, and downstream transfer experiments, the paper analyzes the loss of relative positional structure in freely learned factors and its relationship to generalization. The proposed method demonstrates improved performance over freely learned factors and RIB in several experimental settings.

Critical strengths
  1. A clearly motivated and practically relevant problem. The paper highlights that freely learned low-rank positional factors do not guarantee relative positional consistency. Its quantitative analysis of the relative structure retained by learned attention biases provides useful empirical insights into positional representations in efficient attention.

  1. Broad experimental evaluation. The authors evaluate the proposed approach across language and vision tasks. Comparisons with RIB and additional experiments controlling for positional interpolation help address several alternative explanations and strengthen the empirical findings.
Critical weaknesses
  1. Insufficient differentiation from prior work and unclear methodological novelty. Liutkus et al. [1] introduced sineSPE, which already constructs low-rank relative positional representations using learnable frequencies, amplitudes, and phases. Its mathematical formulation is highly similar to Equations (4)–(6) of this paper. Choromanski et al. [2] also proposed a Fourier-based decomposition of relative positional biases and incorporated the resulting factors into Query–Key dot products. Furthermore, FlashBias [3] established a mechanism for fusing low-rank attention biases into attention computation, while RIB [4] explicitly recognized that low-rank positional factors do not necessarily preserve relative positional consistency. Although this paper applies deterministic relative positional factors to exact softmax attention using FlashAttention and provides new empirical analyses, it does not discuss its direct relationship to [1, 2] or sufficiently clarify the substantive methodological advances over these closely related approaches. Consequently, the paper's independent methodological contribution remains unclear.

  1. Insufficient experimental isolation of the proposed mechanism. The paper demonstrates that freely learned low-rank attention biases lose relative positional structure and that the proposed parameterization can improve generalization. However, comparisons with freely learned factors and RIB [4] change not only the relativity constraint but also the functional form, parameter sharing, and representational capacity of the positional bias. It is therefore difficult to determine how much of the observed improvement can be attributed specifically to enforcing exact relativity. Although the RIB comparisons and interpolation-free experiments in the supplementary material address several alternative explanations, they do not directly isolate the effect of the relativity constraint within an otherwise matched parameterization. Thus, while the experiments demonstrate the effectiveness of the proposed method, they do not fully substantiate the claim that restoring relative positional structure is the primary source of its generalization advantages.

References

[1] Liutkus, A., et al. (2021). Relative Positional Encoding for Transformers with Linear Complexity. Proceedings of the 38th International Conference on Machine Learning (ICML), PMLR 139, 7067–7079. https://proceedings.mlr.press/v139/liutkus21a.html [2] Choromanski, K., et al. (2024). Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers. Proceedings of the 27th International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 238, 2278–2286. https://proceedings.mlr.press/v238/choromanski24a.html [3] Wu, H., et al. (2025). FlashBias: Fast Computation of Attention with Bias. Advances in Neural Information Processing Systems 38 (NeurIPS 2025). https://proceedings.neurips.cc/paper_files/paper/2025/hash/1dc3d70df51a218497529df998a8a8ce-Abstract-Conference.html [4] Lee, D., et al. (2026). Rank-Factorized Implicit Neural Bias: Scaling Super-Resolution Transformer with FlashAttention. arXiv:2603.06738. https://arxiv.org/abs/2603.06738

Questions for authors
  1. What are the substantive mathematical or algorithmic differences between Equations (4)–(6) of this paper and the learnable frequency, amplitude, and phase parameterization of sineSPE [1], as well as the Fourier decomposition and Query–Key integration mechanism of FourierLearner [2]? Could the authors clarify the specific methodological contributions beyond these prior approaches?

  1. Could the authors provide a comparison that varies only the relative positional constraint within the same parameterization framework, while keeping parameter count and optimization settings as closely matched as possible? This would help determine whether the generalization improvements arise primarily from exact relativity rather than other properties of the chosen functional form.
5 comments
AuthorsOpenReview ↗

Response To Reviewer 2 concerns Part 1

We thank the reviewer for carefully reading the paper, recognizing its strengths, identifying the connections to sineSPE and FourierLearner, and requesting a direct test of the relativity constraint. In response, we clarify our methodological contribution relative to both approaches and add a matched experiment that directly isolates the effect of enforcing relativity. We hope that these revisions and results address the reviewer's questions and clarify the contribution of our work.

W1 / Q1. Relationship to sineSPE and FourierLearner. SineSPE constructs stochastic positional features for linear attention. For each content dimension, correlated positional processes multiplicatively modulate the queries and keys so that their product approximates a learned relative kernel in expectation. Our construction instead produces a deterministic, content-independent additive logit bias whose factor product equals the parameterized relative function exactly.

FourierLearner is closer to ours because it also expresses an additive relative-position mask through factors concatenated to queries and keys. The computational objective, however, is different. FourierLearner targets linear-time Performer attention. It learns the Fourier transform of an underlying relative function, samples frequencies from a proposal distribution, and uses random features to construct an unbiased low-rank estimate of the relative mask. It then applies a second random-feature approximation to the exponentiated query--key product to obtain Performer attention. Its output therefore approximates both the learned relative mask and softmax attention, with accuracy depending on the numbers of random features. Our method instead learns a finite discrete spectrum directly: the frequencies, amplitudes, and phases deterministically define

$ B_{ij} = \sum_k a_kb_k \cos\bigl(\mathbf w_k^\top(\mathbf p_i-\mathbf p_j)+\theta_k\bigr). $

The resulting factors reproduce this parameterized bias exactly, and appending them to the queries and keys allows FlashAttention to compute exact softmax attention for that bias. FourierLearner therefore pursues asymptotically linear attention through stochastic approximation, whereas our method preserves standard softmax attention and its IO-aware fused implementation while avoiding materialization of the bias.

Our contribution is therefore not Fourier factorization itself, but its use and analysis in a different setting. Unlike sineSPE, our factors produce a content-independent additive bias deterministically rather than modulating content through stochastic positional features. Unlike FourierLearner, we learn a finite discrete spectrum whose factors reproduce the parameterized bias exactly and run through exact FlashAttention rather than approximating both the bias and softmax attention with random features. Building on this formulation, we:

  • measure how much relative structure is lost when FlashBias-style factors are trained freely;
  • provide a deterministic relative-by-construction parameterization for exact fused softmax attention;
  • introduce a phase term, as in sineSPE, for directional relative structure without increasing factor width; and
  • isolate when enforcing the constraint affects robustness and context extension. Under our corrected softmax-invariant metric, freely learned factors retain only 39% relative structure in vision and 31% in language.

CONTINUED IN NEXT COMMENT.

AuthorsOpenReview ↗

Response To Reviewer 2 concerns Part 2

W2 / Q2. Isolating the relativity constraint. We agree that comparisons with freely learned factors and RIB change several properties simultaneously. We therefore introduce a matched control within our frequency-generated parameterization that changes only whether the query- and key-side frequencies remain equal.

Our model uses the same frequency wkw_k on both sides, yielding cos⁡(wk(i−j)),\cos\bigl(w_k(i-j)\bigr), which depends only on displacement. The {untied-frequency} control instead learns separate frequencies, initialized equally: cos⁡(wkqi−wkkj).\cos\bigl(w_k^q i-w_k^k j\bigr). The two models are otherwise identical in factor width, sinusoidal features, amplitudes, attention computation, architecture, optimization, and token budget. Untying adds only KK scalars per head---6464 parameters model-wide at R=16R=16---and therefore slightly favors the control in capacity.

Tying has negligible in-distribution cost but substantially improves extrapolation. At R=16R=16, tied and untied models obtain perplexities of 24.52±0.0624.52\pm0.06 and 24.61±0.1224.61\pm0.12 at context 512512, but degrade by 2.98±0.74×2.98\pm0.74\times and 15.0±0.7×15.0\pm0.7\times at 8,1928{,}192. At R=32R=32, their training-context perplexities are 24.51±0.3224.51 \pm 0.32 and 24.44±0.0524.44\pm0.05, while degradation is 1.68±0.21×1.68\pm0.21\times and 15.0±0.2×15.0\pm0.2\times. Increasing rank therefore does not repair the untied model.

The untied bias nevertheless scores ρ⊥=98.4±0.5%\rho_\perp=98.4\pm0.5\% at context 512512. The reason is that $w_k^q i-w_k^k j=w_k^k(i-j)+(w_k^q-w_k^k)i. $ The first term is relative, whereas the second depends on absolute position and grows with sequence length. A small mismatch can therefore appear negligible over 512512 positions but dominate at 8,1928{,}192. Thus, ρ⊥\rho_\perp measures relativity over the evaluated range; tying guarantees it algebraically at every length.

Four additional controls clarify why tying the frequencies improves extrapolation.

Gradually enforcing the constraint. We first penalize disagreement between the query- and key-side frequencies: $ \lambda,\operatorname{mean}_k(w_k^q-w_k^k)^2. $

As λ\lambda increases from 00 to 10510^5, mean frequency detuning falls from 8×10−38{\times}10^{-3} to 6×10−66{\times}10^{-6} radians per token, while degradation decreases monotonically from 15.0±0.7×15.0\pm0.7\times to 2.7±1.2×2.7\pm1.2\times, matching the tied model's approximately 3.0×3.0\times. Training-context perplexity remains between 24.524.5 and 24.724.7. This dose response connects the extrapolation improvement directly to the strength of the relativity constraint rather than to in-distribution fit.

Training does not recover relativity automatically. When the two frequency sets are initialized differently, the model obtains the same training-context perplexity (24.69±0.0724.69\pm0.07) and nearly the same degradation (15.1±0.6×15.1\pm0.6\times) as the untied model initialized with equal frequencies. The frequency gap also persists during training. Thus, the training objective does not independently recover the tied relative solution.

Enforcing relativity only at evaluation is insufficient. We also re-tie the trained untied model by copying one learned frequency set onto the other at evaluation. This substantially worsens training-context perplexity and does not recover extrapolation, which remains at 66--11×11\times degradation. The rest of the network therefore adapts to the absolute-position structure introduced during training; the constraint must be present while the model is learned.

The relative function must remain fixed across lengths. Exact relativity at one context is not sufficient if the displacement scale changes with context length. RIB normalizes its coordinates by sequence length and is nearly relative at context 512512, but evaluates a rescaled function at longer contexts. We reproduce this effect by rescaling the input displacement to our trained bias:

Bn(Δ)=B(Δ512n).B_n(\Delta)=B\left(\Delta\frac{512}{n}\right).

This leaves the training context unchanged but increases degradation at R=16,32,128R=16,32,128 from 3.0×3.0\times, 1.7×1.7\times, and 1.4×1.4\times to approximately 9×9\times at every rank. Thus, even an exactly relative bias extrapolates poorly if its meaning in token space changes with context length.

Together, these controls identify two distinct requirements. First, the bias must depend only on displacement; frequency detuning violates this property. Second, it must preserve the same function of displacement across context lengths; length normalization violates this property. Our tied-frequency construction uses unnormalized positions and satisfies both requirements by design. The vision experiments evaluate the complete factor generator rather than isolating these two properties individually, which we will clarify in the revision.

AuthorsOpenReview ↗

Summary of the Response For Reviewer SaZd

Reviewer SaZd,

Please find a brief summary of the response above:

  1. W1/Q1: Relation to prior work and contribution.

(a) We now cite sineSPE and FourierLearner in Section~2 with a direct comparison. These methods realize relative kernels stochastically within linear attention, whereas ours is a deterministic additive logit bias for softmax attention, has zero approximation error, and runs unchanged in fused kernels.

(b) Our primary contribution is measuring what unconstrained factor learning actually recovers. FlashBias introduces free factors and RIB observes that they need not be relative, but neither quantifies their learned structure. Under the corrected metric, free factors retain only 39%39\% relative structure in vision and 31%31\% in language.

Therefore, the contribution is not the Fourier identity itself, but its exact fused realization and the empirical and theoretical characterization of when learned factors become relative. The updated manuscript with the changes above has been uploaded for your review.

  1. W2/Q2: Isolating the source of extrapolation.

(a) We untie the query- and key-side frequencies within our parameterization, allowing the two frequency sets to be learned independently, while keeping the functional form, rank, initialization, architecture, and training recipe fixed. The control adds only 6464 scalar parameters and produces identical outputs at initialization. At R=16R=16, the tied and untied models obtain in-distribution perplexities of 24.5224.52 and 24.6124.61, respectively, but degrade by approximately 3×3\times and 15×15\times under extension; the same separation appears at R=32R=32.

(b) Four controls refine this result. Penalizing the frequency gap reduces degradation monotonically from 1515 to 13.713.7, 8.78.7, and 2.7×2.7\times while in-distribution perplexity remains flat. Initializing the frequencies apart still gives 15×15\times degradation, and re-tying them only after training does not repair it. Finally, applying RIB's length normalization to our exactly relative weights increases degradation from 3×3\times to approximately 9×9\times across ranks.

Therefore, extrapolation requires both exact displacement dependence and preservation of the same displacement function across lengths; our parameterization enforces both by construction. The updated manuscript has this discussion in the main paper.

If there are any more questions or queries, we would be happy to clarify them.

Reviewer_C7KAOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that freely learned low-rank attention biases can lose the translation invariance of relative positional biases. It introduces a learnable Fourier-based factorization that preserves this property by construction, together with a phase extension for directional asymmetry. Experiments in language modeling and vision demonstrate improvements in length extrapolation, robustness, and downstream transfer, with little additional computational cost.

Critical strengths
  1. A useful observation and an elegant solution. The paper highlights a structural limitation of freely learned low-rank attention biases that is easy to overlook when focusing on computational efficiency. The proposed construction is simple, guarantees translation invariance, and retains compatibility with standard FlashAttention kernels without increasing the factor width.

  2. Convincing empirical evidence across different settings. The language experiments show substantial improvements in context-length extrapolation, while the vision experiments demonstrate gains in robustness and downstream transfer. The comparisons with RIB, the rank ablations, and the analysis of directional bias provide useful insights beyond the performance numbers. The substantial reduction in positional parameters is another practical advantage.

Critical weaknesses
  1. Interpretation of the relativity metric. The proposed relative energy ratio ρ\rho is not invariant to row-wise shifts of the attention logits, even though such shifts leave the softmax attention unchanged. For example, a bias of the form Bij=g(i−j)+ciB_{ij}=g(i-j)+c_i can have ρ<1\rho<1 while producing exactly the same attention weights as the relative bias g(i−j)g(i-j). Appendix A.3 acknowledges this issue, but the row-centering analysis does not fully resolve it, since even exactly relative biases can obtain ρ<1\rho<1 after centering. This makes the interpretation of the reported 25.2% and 37.2% relative energy less straightforward.

  2. Limited evidence for practical language-modeling settings. The language experiments use a relatively small Transformer in which the attention bias is the only source of positional information. This is useful for studying the proposed mechanism, but its benefit in models already equipped with positional encodings such as RoPE remains unclear. Moreover, ALiBi still achieves substantially better length extrapolation, which limits the practical advantage of the proposed cosine-based bias in this setting.

  3. Minor presentation issue. The main paper and supplementary material lack clickable hyperlinks for citations and cross-references, making navigation between sections, tables, and references unnecessarily inconvenient, particularly with the appendix provided separately.

Questions for authors

Main concerns presented in weaknesses.

5 comments
AuthorsOpenReview ↗

Response To Reviewer 1 concerns Part 1

We thank the reviewer for the careful reading, recognizing its strengths and constructive comments, which helped us strengthen both the analysis and the positioning of the paper. Within the next few days, we will upload a revised manuscript incorporating the new results described below, the suggested edits, and corrected hyperlinks.

W1. Interpretation of the relativity metric. The reviewer is correct that our original metric is not invariant to per-query offsets: it can distinguish g(i−j)g(i-j) from g(i−j)+cig(i-j)+c_i, although they produce identical attention weights. Row-centering does not fully resolve this issue because causal and windowed rows contain different displacement sets. Consequently, even exactly relative biases score below 100% after centering (97.4% in vision and 82.% in causal language).

Following the reviewer's observation, we now measure distance to the full softmax-equivalence class g(Δij)+cig(\Delta_{ij})+c_i, where Δij\Delta_{ij} is the displacement between positions. For biases with nonzero energy after removing per-query offsets, we define

ρ⊥(B)=1−min⁡g,{ci}∑(i,j)∈Ω[Bij−g(Δij)−ci]2min⁡{ci}∑(i,j)∈Ω[Bij−ci]2,\rho_\perp(B)=1-\frac{\displaystyle\min_{g,\{c_i\}}\sum_{(i,j)\in\Omega}\left[B_{ij}-g(\Delta_{ij})-c_i\right]^2}{\displaystyle\min_{\{c_i\}}\sum_{(i,j)\in\Omega}\left[B_{ij}-c_i\right]^2},

where Ω\Omega contains the visible attention entries. The denominator is the bias energy remaining after the best per-query offsets are removed, while the numerator is the residual energy after additionally fitting the best displacement function. If BB consists entirely of per-query offsets, the denominator is zero; we define ρ⊥(B)=1\rho_\perp(B)=1 because such a bias is softmax-equivalent to the zero relative bias.

The score lies in [0,1][0,1], is invariant to Bij↦Bij+ciB_{ij}\mapsto B_{ij}+c_i, and assigns 100% to every bias functionally equivalent to a relative bias. We solve the joint least-squares problem by conjugate gradients; synthetic relative biases with arbitrary row offsets score 100.0%, while i.i.d. random biases attain chance.

Under the corrected metric, freely learned factors score 38.6-39.3% in vision (previously 37.1-37.2%) and 31.2±4.231.2\pm4.2 % in causal language (previously 25.2±4.025.2\pm4.0)%. Relative-by-construction controls score 100100%, while random factors score 0.40.4%. Row-only offsets explain only 66--77% of the non-relative energy in vision and 1010--1717% in language, leaving our central finding unchanged: most of the learned bias remains pairwise absolute-position structure.

The corrected metric also clarifies RIB. It scores 99.4%99.4\% at context 512512 but degrades by 6×6\times at 8,1928{,}192 because its normalized coordinates rescale the displacement function as context length changes. We therefore present ρ⊥\rho_\perp as a fixed-context structural diagnostic, not a standalone predictor of extrapolation.

W2.1. Language models with RoPE. The bias-only setting deliberately isolates the parameterization proposed by FlashBias without a second positional signal. To test the practical setting raised by the reviewer, we additionally trained RoPE alone, RoPE with freely learned factors, and RoPE with our relative bias using the same recipe, three seeds, and R=16R=16.

At context 512512, all three perform similarly: RoPE obtains 24.25±0.1024.25\pm0.10 perplexity, while both augmented variants obtain 24.21±0.0324.21\pm0.03. At 8,1928{,}192 tokens, RoPE degrades by 18.8±0.9×18.8\pm0.9\times, RoPE with free factors by 16.6±0.4×16.6\pm0.4\times, and RoPE with ours by 9.7±1.2×9.7\pm1.2\times. Thus, free factors provide smaller improvement, whereas the relative bias nearly halves RoPE's degradation. Because the augmented models have identical factor width and in-distribution perplexity, the gain reflects the structure of the bias rather than additional capacity or better training-context fit. The remaining gap to our bias-only model, which degrades by approximately 3×3\times, suggests that RoPE becomes the limiting positional component at long contexts. Thus, the relative bias remains beneficial alongside RoPE, reducing degradation by nearly 2×2\times without affecting in-distribution perplexity.

AuthorsOpenReview ↗

Response To Reviewer 1 concerns Part 2

W2.2. Comparison with ALiBi and fixed-form relative biases. Our aim is not to outperform ALiBi, but to provide a general framework for learning relative biases within fused attention. ALiBi is a strong reference for language extrapolation, where its one-dimensional monotone form is well suited to the task. That fixed form, however, is less suited to represent the task-specific spatial or geometric relationships required in other domains. For example, Zhao et al.\ (2021) report 70.470.4 mIoU with relative position encoding in Point Transformer, compared with 66.566.5 using absolute encoding.

To test whether a predetermined distance function is sufficient in vision, we add a fused control, denoted FB-Fixed, that replaces our learned frequency bias with a frozen monotone function of spatial distance. Relative to our learned frequencies, FB-Fixed loses 1.31.3 points on ImageNet-1K, 2.22.2 on ImageNet-V2, and 5.15.1--6.66.6 on ImageNet-A, ImageNet-R, ImageNet-Sketch, and mCE (single run). Thus, while a monotone distance bias is effective for language extrapolation, vision benefits from learning the appropriate relative function. The same distinction may apply in other domains with richer relative geometry.

We also test whether our remaining language gap to ALiBi arises from relative factorization or from the bounded cosine basis. FlashBias already shows that ALiBi has an exact rank-22 factorization under causal attention. Using this existing construction, we append ALiBi's linear term to our RR learned cosine channels within the same fused kernel. The combined bias achieves 0.91×0.91\times degradation at 8,1928{,}192 tokens for every R∈{16,32,128}R\in\{16,32,128\}, comparable to ALiBi (0.92×0.92\times), with comparable in-distribution perplexity. This shows that the original gap comes from the absence of a monotone component, not from relative factorization itself.

Fusing ALiBi is not our contribution. Our contribution is enabling relative biases to be learned during fused training when the appropriate relative function is not known in advance, while remaining compatible with specialized components such as ALiBi when their form is appropriate.

W3. Clickable citations and cross-references. We will correct this presentation issue in the revision. The appendix will be included after the references in the same PDF, and all citations and cross-references to sections, equations, tables, figures, and appendix material will be clickable.

I would like to thank the authors for the detailed response. I'll keeping my original score since it's already positive.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Revised Manuscript Uploaded Addressing Reviewer Comments and Concerns. Summary of Edits below.

Summary of Revisions in the updated Manuscipt Uploaded Above.

Edits are marked in blue for ease of reading.

Main manuscript

  • Softmax-invariant relativity metric. (Reviewer C7KA) Replaced the original relative-energy ratio and row-centering diagnostic with ρ⊥, which treats biases g(Δᵢⱼ) + cᵢ as equivalent. All language and ImageNet measurements, including RIB, are reported under ρ⊥.

  • Relationship to prior spectral methods . (Reviewer SaZd) Added explicit discussion of sineSPE and FourierLearner in related works, stating that sinusoidal factorization is not itself new. We distinguish our deterministic additive bias in exact fused softmax attention from their random-feature approximations for linear attention.

  • Matched isolation of relativity . (Reviewer SaZd) Added a tied-versus-untied frequency control. Both variants have comparable training-context perplexity, but degradation at 8,192 tokens rises from about 3× to 15× when frequency tying, and with it guaranteed relativity across lengths, is removed.

  • Compatibility with RoPE . (Reviewer C7KA, Reviewer 3ggm) Added RoPE experiments: our bias reduces degradation at 8,192 tokens from 18.8× to 9.7×, while freely learned factors reach 16.6×, with unchanged in-distribution perplexity.

  • Comparison with ALiBi. (Reviewer C7KA; Reviewer 3ggm) Clarified that ALiBi's advantage comes from its monotone functional form. Appending its existing rank-2 linear factorization to our learned cosine channels closes the gap (0.91× versus 0.92×).

  • Additional vision controls .(Reviewer C7KA; Reviewer SaZd) Added RIB and FB-Fixed at window 32. RIB matches ours on robustness, so vision does not separate relativity from factor generation; the matched language control isolates relativity. FB-Fixed shows that a predetermined monotone relative function lacks the flexibility required in vision.

  • Corrected vision claims . (Reviewer SaZd, Reviewer 3ggm) Robustness gains are now attributed to generated factors rather than to relativity, and the contribution statement no longer claims that vision isolates relativity.

  • Point-cloud evaluation .(Reviewer 3ggm) Added Point Transformer V3 on S3DIS. Our symmetric bias improves the baseline by 0.6–1.3 mIoU at patch sizes 128–1,536, and the phase variant reaches 72.70 mIoU at the default patch size of 1,024.

  • Presentation and positioning .(Reviewer C7KA, Reviewer 3ggm) Improved transitions, enabled clickable references, and placed the appendix in the same PDF. The limitations now include point-cloud evaluation and state the remaining limits in scale and domain coverage.

Appendix

  • Added the definition, computation, and validation of ρ⊥ for causal and two-dimensional supports, explained why row-centering fails, and replaced analytical chance levels with empirical ones (Reviewer C7KA).
  • Updated the language and ImageNet RIB relativity results under ρ⊥; the analysis of the released DF2K checkpoint is labelled as the uncentered share (Reviewer C7KA).
  • Added full tied/untied results and four controls: detuning penalties, different initialization, post-training re-tying, and length-dependent coordinate scaling (Reviewer SaZd).
  • Added the FB-Fixed construction and the RIB/FB-Fixed training setup (Reviewer C7KA, Reviewer SaZd).
  • Added the Point Transformer patch-size sweep, parameterization comparison, and memory/throughput analysis (Reviewer 3ggm).
AuthorsOpenReview ↗

Summary of the Discussion Phase For all Reviewers and Area Chairs.

As the discussion period closes, we briefly summarize the reviewers' assessments and our revisions.

Strengths recognized by reviewers

  • Novel problem: We identify a structural limitation of low-rank attention biases that is easy to overlook (Reviewer C7KA) and practically relevant (Reviewer SaZd). No prior work measured it: freely learned factors retain only 39% relative structure in vision and 31% in language, causing failures under context extension and distribution shift.

  • Method: Elegant and simple, guarantees relativity, and stays compatible with standard FlashAttention kernels (Reviewer C7KA and 3ggm). Unlike sineSPE, FourierLearner, and RoPE, ours is an exact, content-independent additive bias inside fused softmax attention.

  • Analysis and evaluation: Useful insight into retained relative structure (Reviewer SaZd); convincing gains in extrapolation, robustness, and transfer, with a large reduction in positional parameters (Reviewer C7KA).

Key new evidence

  • Our formulation enables extrapolation (Reviewer SaZd): Untying only the query- and key-side frequencies leaves in-distribution perplexity unchanged but raises degradation at 8,192 tokens from about 3× to 15×. Extrapolation needs a bias that depends only on displacement and keeps the same function across lengths; untied frequencies and RIB's length normalization each break one, and our construction guarantees both.

  • Our bias complements RoPE (18.8× → 9.7× degradation), and adding ALiBi's linear term matches ALiBi (Reviewer C7KA, 3ggm).

  • On Point Transformer V3, our bias adds 0.6–1.3 mIoU where dense RPE runs out of memory (Reviewer 3ggm).

Most of the reviewers' concerns were addressed with new experiments and analyses. We hope the reviewers, AC, and SAC will consider these in their overall evaluation.

I appreciate the authors' efforts.

However, I believe the key is to further improve your manuscript, rather than repeatedly rephrasing the rebuttal to get your paper accepted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.