- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies low-rank additive attention biases that can be appended to queries and keys and computed using standard fused-attention kernels. The authors observe that freely learned low-rank factors do not guarantee that the resulting bias depends only on relative displacement. They propose generating the factors from learned sinusoidal frequencies, which guarantees relative structure through the cosine-difference identity. Experiments are conducted on a tiny language model and Swin-T.
Critical strengths
- The core idea is straightforward. The proposed parameterization is mathematically valid and compatible with standard fused-attention kernels.
Critical weaknesses
-
The presentation must be improved. The current manuscript appears to be a simple combination of different paragraphs, with loosely connected discussions of many different concepts like kernel fusion, relativity, parameter efficiency, robustness, and extrapolation.
-
The research significance and novelty are limited. The paper does not establish why an attention bias must depend exclusively on relative position. Absolute-position information may be useful, and the necessity of enforcing exact relativity remains insufficiently validated. The method itself closely resembles the standard RoPE construction, except that positional features are appended to queries and keys to produce an additive bias. This also enlarges the Q/K dimension and introduces additional computation, weakening the claimed efficiency motivation.
-
The experiments are too toy to support the claims. The language evaluation uses only a 51M model on WikiText-103 trained at context 512, while the vision evaluation is restricted to Swin-T. This evidence is inadequate for the broad claims.
-
The improvements are marginal or negative. The proposed variants remain below full SwinV2 in the available comparisons. In language modeling, the method extrapolates substantially worse than ALiBi. Its efficiency is also not clearly better than modern kernel-based alternatives such as FlexAttention, while the appended positional dimensions introduce extra QK computation.
Questions for authors
N/A
7 comments
Response To Reviewer 3 concerns Part 2
W3. Experimental scale and domain breadth. The submitted M language model and Swin-T were chosen to support controlled comparisons and test the identified failure and proposed solution across language and vision. They were not intended to establish behavior at large model scales. The effects within these settings are nevertheless large: ours degrades by -- under context extension, compared with -- for free factors and for RIB for the unconstrained alternatives.
We expand experiments and analysis along three axes. First, at GPT-2-small scale (M parameters; one seed per method), the ordering persists. At context length , freely learned factors, RIB, and our method achieve perplexities of , , and , respectively; at , their degradation factors are , , and . We treat this as an ordering check and do not interpret the small in-distribution differences.
Second, the advantage persists at longer training contexts. When trained at length , freely learned factors degrade by at , compared with for ours. When trained at length using the same tokens per step and total training budget, freely learned factors achieve perplexity in distribution and degrade by at . Ours achieves and degrades by only . Thus, under the same context extension, our degradation remains whether trained at length or .
Third, we add Point Transformer V3 on S3DIS Area~5 (M parameters, two seeds per arm; Table Below. Our fused relative bias improves the stock model by -- mIoU at patch sizes -- and remains feasible at substantially larger patches. At the default patch size of , freely learned factors do not improve the stock model ( versus mIoU), whereas our symmetric bias reaches and the phase variant performs best at . RIB also reaches , showing that generating positional factors is important in this in-distribution setting, while the additional gain from phase indicates value in representing directional relative geometry. These results extend the applicability of our fused parameterization to three-dimensional point clouds; the matched language control provides the direct isolation of exact relativity.
The dense relative-position table performs slightly better where it fits ( versus mIoU at patch ), but does not fit on a ,GB GPU at patch and the standard batch size. Our factorized bias remains feasible there, requiring ,GB and reaching mIoU. Thus, the factorized form enables learned relative geometry at neighborhood sizes where the dense bias is impractical.
| Patch size | PTv3 baseline (xCPE) | Ours (symmetric), | Gain |
|---|---|---|---|
| 128 | 70.36 | 71.08 | +0.72 |
| 512 | 71.58 | 72.19 | +0.61 |
| 1,024 | 70.86 | 72.18 | +1.32 |
| 1,536 | 71.11 | 72.25 | +1.14 |
| 2,048 | 71.74 | 71.69 | −0.05 |
| 4,096 | 71.48 | 71.58 | +0.10 |
| Parameterization | mIoU | Gain over PTv3 baseline |
|---|---|---|
| Freely learned factors | 70.61 | −0.25 |
| RIB | 72.15 | +1.29 |
| Ours + phase | 72.70 | +1.84 |
These experiments demonstrate applicability across model size, training length, patch scale, and a third domain, while also identifying settings in which exact relativity provides limited additional benefit. Evaluation on substantially other larger models and datasets remains future work.
Response To Reviewer 3 concerns Part 3
W4. Performance and efficiency trade-offs. Our method is designed to provide a learnable relative bias through standard fused-attention interfaces, not to dominate every specialized bias or kernel. The relevant comparison depends on the setting: SwinV2 provides a dense vision bias, ALiBi supplies a fixed monotone language bias, and FlexAttention compiles arbitrary score modifications. Our method instead provides a compact, task-specific relative bias when the appropriate function must be learned and materializing a dense bias is impractical.
Comparison with SwinV2. The factorized variants do not uniformly outperform full SwinV2, and we will clarify this positioning. Their ImageNet-1K accuracy is close to SwinV2-T ( and for ours and with phase, versus ), while the robustness comparison is mixed: ours with rank 32 is higher on ImageNet-V2 and mCE but lower on ImageNet-A, ImageNet-R, and ImageNet-Sketch. This is not a fully controlled comparison because the SwinV2 reference also differs in window size and use of the shift mask.
The controlled comparison is against freely learned factors at matched factor width and training recipe. The phase variant improves all five robustness measures, including mean gains of points on ImageNet-A, on ImageNet-R, and on ImageNet-Sketch. The factorized representation also enables larger windows and denser tokenization where the dense SwinV2 bias becomes memory-limiting. Our claim is therefore an improved factorized alternative with a different accuracy--efficiency trade-off, not universal superiority to SwinV2.
Comparison with ALiBi. ALiBi is the stronger specialized choice for one-dimensional extrapolation when its monotone form is appropriate. Our bounded cosine bias is learnable across spatial and geometric domains but, by itself, extrapolates less effectively. To test whether this gap arises from factorization or functional form, we append FlashBias's existing exact rank- factorization of ALiBi's linear term to our learned cosine channels. The combined bias achieves degradation at tokens for every tested cosine width, comparable to ALiBi's , with comparable in-distribution perplexity. The original gap therefore comes from the absence of a monotone component rather than from relative factorization. Our framework can incorporate such a component when its form is known while retaining learned relative structure for other applications.
Comparison with FlexAttention and additional Q/K computation. Appending the factors increases the query and key dimension from to at , and this overhead is included in all reported measurements. Despite the additional computation, our method retains 87% of no-bias throughput. It is faster than eager SwinV2 attention while using 17% of its memory, and faster than streaming the dense bias through memory-efficient SDPA.
Our throughput is within -- of FlexAttention, so we do not claim a substantial speed advantage over it. The distinction is how the bias reaches the kernel. FlexAttention compiles a specified score modification into bias-specific kernel logic, whereas our bias uses ordinary augmented queries and keys and runs through compatible, unmodified FlashAttention-style kernels. It therefore requires no compilation for a particular bias form, keeps positional parameters independent of window size and context length, and can be used across supported fused-attention backends.
Taken together, these results position our method as a practical option within the accuracy--efficiency design space. It trades the unrestricted form of a dense or compiled bias for a compact, learnable relative parameterization that works through standard fused-attention interfaces. This enables task-specific relative biases at window, context, and neighborhood sizes where dense materialization is impractical, while remaining compatible with specialized components such as ALiBi when their form is appropriate.
Response To Reviewer 3 concerns Part 1
W1. Presentation. We are reorganizing the manuscript to improve and address all identified issues of flow and clarify the scope of each claim. We will upload the revised manuscript in the next few days.
W2.1. Why enforce relativity?. We do not claim that every attention bias should be relative; absolute position can also be useful. However, prior controlled comparisons demonstrate benefits from relative structure in several settings. Shaw et al.\ (2018) report gains of 0.3 BLEU with the base Transformer and 1.3 BLEU with the big Transformer over absolute representations on WMT14 English--German. Liu et al.\ (2021) report that Swin's relative bias improves ImageNet-1K accuracy by 0.8 points and ADE20K segmentation by 2.9 mIoU over absolute embeddings, while Zhao et al.\ (2021) report an improvement from 66.5 to 70.4 mIoU with relative rather than absolute position encoding in Point Transformer.
Our experiments support this scoped claim rather than a universal preference for relativity. On the training distribution, different factor generators often perform similarly. The clearest benefit appears when position-specific structure must transfer beyond the training regime: freely learned factors degrade by 11--21× under context extension, compared with 1.4--3× for ours. In Swin-T, at comparable ImageNet-1K accuracy, both relative variants improve robustness over freely learned factors. The phase variant improves all five robustness measures, including mean gains of 2.0 points on ImageNet-A, 1.5 on ImageNet-R, and 1.4 on ImageNet-Sketch. Thus, we do not claim that relativity always improves performance; we show that preserving relative structure prevents specific failures under context and distribution shift.
Enforcing relativity helps when displacement or geometry matters and may extend to video, scientific meshes, and molecular structures; systematic evaluation is left future work.
W2.2. Relationship to RoPE. We agree that the geometric construction is closely related to RoPE, and the revision states this explicitly. The distinction is that RoPE rotates content-dependent queries and keys, whereas our method applies the same geometry to appended constant channels, producing a content-independent additive bias. To test whether the two mechanisms are redundant or complementary, we trained RoPE alone, RoPE with freely learned factors, and RoPE with our relative bias using the same recipe and three seeds. All three have nearly identical perplexity at context 512 (24.25±0.10, 24.21±0.03, and 24.21±0.03, respectively). At 8,192 tokens, however, degradation falls from 18.8±0.9× for RoPE alone and 16.6±0.4× with freely learned factors to 9.7±1.2× with our bias. Thus, the learned relative bias complements RoPE rather than replacing it. More broadly, our framework provides a fused way to encode relative relationships without modifying token content, which may be useful when geometry should influence attention independently of the features being compared.
W2.3. Additional Q/K computation and efficiency. Appending bias factors adds R query and key channels; at R=16, the head dimension grows from 32 to 48. This overhead applies to all factorized fused biases, including FlashBias and RIB, and is included in our measurements.
On an H200 at window 32 (Swin-T, 256px, batch 64), our method retains 87% of no-bias throughput. Relative to eager SwinV2, it is 2.8× faster at patch 4 during inference (3,042 versus 1,084 img/s). During training, it is 2.1× faster at patch 4 (836 versus 394 img/s) and 2.6× faster at patch 2 (229 versus 87 img/s), while using only 17--19% of the memory. It also outperforms dense-bias streaming through memory-efficient SDPA, reaching 836 versus 699 img/s at patch 4 and 229 versus 177 img/s at patch 2. Throughput is comparable to FlexAttention: ours is approximately 2% faster at patch 4 and 7% faster at patch 2 during training.
Unlike FlexAttention, which compiles bias-specific score logic, our method expresses the bias through the standard query--key interface and runs on compatible, unmodified FlashAttention-style kernels. It requires no bias-specific compilation, keeps positional parameters independent of window and context size, and achieves comparable throughput while supporting configurations for which FlexAttention did not compile. FlexAttention supports arbitrary score programs; ours provides a compact, learnable relative bias portable across fused-attention backends.