acceptodds
Under review as a conference paper at ICLR 2027

MINKOWSKI ATTENTION

Abstract

This paper introduces Minkowski Attention, a novel formulation for attention and self-attention layers through a generalization of Minkowski spacetime. Attention layers must reason over two distinct representations per token, for content and position. The canonical fix, where content and position embeddings are summed before computing attention, entangles these two signals into cross terms that don't cleanly separate, hence a growing body of work advocates for decoupled attention. Decoupled attention however requires computing separate attention matrices for each pair of representations, inducing considerable compute and memory cost per layer. In this work, we show that with a Minkowski spacetime perspective, it becomes possible to recover a single attention score from decoupled representations. We propose to view token and positional embeddings as multi-dimensional space and time vectors, where token pairs are scored with a Minkowski inner product instead of a Euclidean dot product. The signed structure of Minkowski spacetime keeps content and positions contributions separable under one scalar. We generalize the (3+1)D Minkowski metric to arbitrary (p+q)-dimensional signatures and to the corresponding generalized Lorentz boosts. Our Minkowski transformer consistently outperforms coupled and decoupled transformers across various tasks and modalities, demonstrating strong attention diversity over attention heads and robustness to non-discriminative tokens.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Disentangled Attention, Pseudo-Euclidean Transformer, Minkowski Space

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_79DiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose Minkowski attention, an extension of standard dot-product attention to separate the processing of content and position when positional information is present. The method replaces the euclidean inner product QK⊤QK^\top in attention with an Minkowski inner product and preprocesses Q,K,VQ,K,V with individual learned Lorentz transformations. The authors find that the method systematically outperforms baselines on language, graph, and vision benchmarks. However, the method has many seemingly arbitrary design choices and central aspects are not ablated, and compute cost is not discussed.

I rated the paper a weak reject due to the missing formal justification and the poor ablation of design choices, although I think the method is promising. In 2026 with agents and so on I believe that strong empirical performance alone is not sufficient for a good paper, and the understanding of why the method works and to rule out alternatives is increasingly important. I look forward to the rebuttal as I think this paper can be improved significantly. Also, the paper is missing the required AI use statement; I did not include this in my rating and leave this to the AC to handle.

Critical strengths
  • The method outperforms all baselines across language, graph, and vision tasks. I find this impressive because the method is simple and it should be easy to apply it to a wide variety of tasks that use transformers on tasks that require positional information. The experimental setup seems credible as the authors report uncertainties and explain their hyperparameters in detail, although they do not provide their implementation.
Critical weaknesses
  • The paper does not include an AI use statement. From my understanding of the author guidelines this is a strict requirement. I did not include this aspect in my rating below, and leave it to the AC to decide how to handle this.
  • The method lacks a sound justification for the design choices in Section 3. Also, there are several unexplored design choices that should be ablated to properly understand the method, see the second point in "Questions For Authors". Currently Section 3 reads like the authors found this by trying and it just works well in experiments, but the physics motivation used to introduce the method is weak and there are just so many open questions. I think this is a missed opportunity and the interpretability of the method would benefit significantly from finding something better (and potentially modifying the method without losing performace). For instance, a starting point could be the tensorial message-passing of https://arxiv.org/pdf/2505.20280, where the authors apply learned Lorentz transformation to QKV such that attention is performed in a global frame, whereas all other operations are performed in per-particle local frames; I think the proposed Minkowski attention is equivalent to this approach, but with different local frames for each attention block.
Questions for authors
  • How expensive is the method? I believe this is the central open question required to decide if the method is useful. The authors include GFLOPs for the language study in Table 2, but no timings. If the method is more expensive, the authors should consider to compare performance at fixed compute cost. I suspect that the method requires a custom GPU implementation to be useful, and in 2026 with coding agents I think its feasible to expect the authors to ship this with the paper. In any case, this aspect should be discussed in more detail.
  • Equation 6, the construction of the velocity, deserves more explanation and ablations. Enforcing v∈[0.5,1]v\in [0.5,1] is a clear restriction, the natural choice is v∈[−1,1]v\in [-1,1], and the authors provide no explanation or ablation. It also seems unnatural to me to construct the velocity from the same token and context embeddings that enter attention without a learned modification; the natural choice for me would be to have a separate weight matrix multiplying the context and position encodings before taking their norms in Equation 5.
  • The authors do not discuss their choice of spacetime metric, i.e. pp and qq for Rp,q\mathbb{R}^{p,q}. This is a central hyperparameter of the method and should be ablated. What happens in the regimes p=0p=0 and q=0q=0?

Minor points:

  • What does "FLN" in Table 4 mean?
  • In Table 3, the authors report two digits after the comma for their method, but only one digit for other methods. The extra digit is in uncertainty, so I think they should just always give one digit.
  • Typos: L323 "... exciting future research (direction?).", L364 "... vanil(l?)a ..."
  • Suggestion to improve clarity: In Section 3 both the values and the velocity are denoted by vv. You could instead write uu for the velocity to avoid confusion.
  • Suggestion for related work: Lorentz-equivariant transformers (https://arxiv.org/pdf/2405.14806, https://arxiv.org/pdf/2505.20280) use attention with Minkowski inner products to construct Lorentz-invariant attention logits.
2 comments
AuthorsOpenReview ↗

Response to Reviewer 79Di: cost, custom kernel, velocity, signature

AI use statement: Thank you for pointing this out. We added an LLM Usage Statement to the revised paper (after the conclusions).

Q1 (computational cost, fixed cost, GPU kernel): We thank the reviewer and now report actual runtime and memory for both inference and training. All models have 20.5M parameters and are implemented with identical unfused PyTorch operations.

  • Inference (single A40 GPU, bf16, batch size 128): forward-pass latency, mean ± std over 20 runs after 3 warmup passes.
  • Training (2 A40 GPUs): total pre-training time and memory per GPU.
MethodIL-512 (ms)IL-1024 (ms)TTMem/GPU (MB)
Standard79.85 ± 0.23249.62 ± 0.376h 18min25,171
DeBERTa222.75 ± 0.30 (+179%)835.21 ± 0.69 (+235%)10h 05min27,761
Minkowski137.07 ± 0.48 (+72%)413.29 ± 0.81 (+66%)8h 58min28,201

IL-512 / IL-1024: inference latency per batch at sequence length 512 / 1024 (single A40 GPU, bf16, batch size 128, mean ± std over 20 runs after 3 warmup passes). TT: total training time on 2 A40 GPUs. Mem/GPU: training memory per GPU.

More detailed results are in the new Appendix "Computational Cost". Minkowski attention is more expensive than standard attention but substantially cheaper than DeBERTa, the decoupled baseline it is designed to replace, and its relative overhead does not grow with sequence length.

On a custom kernel: none is required. The score is a single dot product over concatenated head vectors, −Mij=[qi′,tq,i′]⊤[−kj′,tk,j′]-M_{ij} = [q_i', t_{q,i}']^\top [-k_j', t_{k,j}'], and the aggregation is ∑jPij(Vj′−Tj)\sum_j P_{ij}(V_j' - T_j). Minkowski attention can therefore be computed with one standard fused attention call (e.g. FlashAttention / PyTorch SDPA) of head dimension 2dh2d_h, with the boosts applied beforehand as a cheap elementwise operation. With PyTorch's fused attention and torch.compile, Minkowski attention trains at 8192 tokens, with a 1.4–1.8× overhead over equally optimized standard attention (Appendix "Computational Cost").

Q2 (velocity, Eq. 5–6, domain of uu):

Ablation Study on the choice of Sigmoid function for uu,

MethodSQuAD v1 EMSQuAD v1 F1MNLI-mMNLI-mmSST-2 AccParams
Minkowski (TanH)60.19 ± 0.4171.62 ± 0.3771.00 ± 0.2072.75 ± 0.0683.91 ± 0.5220.5M
Minkowski (1−e−x1 - e^{-x})59.95 ± 0.6471.44 ± 0.5870.64 ± 0.2072.00 ± 0.1884.52 ± 0.5420.5M
Minkowski60.62 ± 0.2772.02 ± 0.3171.01 ± 0.2872.53 ± 0.1784.17 ± 0.4520.5M

Q3 (choice of p and q as hyperparameters Rp,q\mathbb{R}^{p,q}, cases p=0 and q=0): Minkowski attention is defined for p=qp=q; we now make this explicit in the abstract ("(d+d)-dimensional signatures") and in Section 3.3 (each head operates in Rdh,dh\mathbb{R}^{d_h,d_h}). Although the Minkowski product itself would allow any (p,q)(p,q), the pseudo-Lorentz transformation pairs every time-like coordinate with a space-like one (x′=γ(x−ut)x'=\gamma(x-ut), t′=γ(t−ux)t'=\gamma(t-ux)), which requires p=qp=q. Each head therefore lives in Rdh,dh\mathbb{R}^{d_h,d_h}, so pp and qq are not an additional hyperparameter, they act as an ordinary embedding dimension hyperparameter. Consequently, the cases p=0p=0 and q=0q=0 coincide (p=q=0p=q=0) and are improper, they correspond to a model with embedding dimension 00.

Minor points. All corrected in red. FLN is the final LayerNorm before the classification head (now defined in the caption of Table 4; NA for Post-LN models). Table 3 now uses one decimal for our method, the two typos are fixed, and the velocity is now denoted by uu. We added L-GATr and LLoCa to the related work: they use the Minkowski product to obtain Lorentz-invariant logits for physical four-vectors, whereas we apply it to learned content and positional embeddings.

Thank you for the responses. My conceptional concerns about the compute cost aspect are settled. However, I believe that there is still room to further reduce the overhead, as 1.3x slower training and 2x slower inference feels a bit too much for the quite small change to the method. My understanding is that you did not use torch.compile for these measurements, maybe that helps. Thank you also for the ablation on the velocity design, it's good to see that the tanh approach gives similar but slightly worse performance.

My main concern about the missing motivation for the design choices still stands, and I will therefore keep my score at "weak reject". I will keep track of the discussion with the other reviewers and might increase my score later on, but currently I don't see it going above "weak accept". To be explicit, my concerns are not about missing experiments or performance or so, but about a better understanding of why the concrete choice of the authors works so well. There is probably a wide range of other design choices that yield similar performance, and I am confident that some of them have a clear motivation (and probably work better), which would make it much easier for others to understand why the method works and to convince them to adapt them. I am aware that figuring this out takes more time than the two weeks for the rebuttal. The current paper reads a bit like a wasted opportunity to me.

Reviewer_r5v6OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Minkowski Attention, an attention mechanism based on generalized Minkowski space, designed to address the issue of content-position coupling found in traditional Transformers. The authors model token content and positional representations as spatial and temporal components, respectively; they compute attention scores using an inner product defined by an indefinite metric and introduce an input-dependent Pseudo-Lorentz Transformation (PLT) to facilitate adaptive interaction between content and position. This approach aims to achieve information aggregation via a single attention matrix while maintaining the structural separation of the two representation types, thereby enhancing expressive power and computational efficiency. The authors conducted experiments across language understanding, graph node classification, and image classification tasks, validating the method through component ablation, attention visualization, and optimization stability analysis. However, the paper has not yet fully demonstrated the distinct advantages of Minkowski geometry over general signed bilinear attention; furthermore, some experimental comparisons involve architectural differences, and the analyses regarding computational efficiency and attention behavior require further verification.

Critical strengths
  1. This paper proposes an attention mechanism based on the generalized Minkowski inner product; it models content and position as distinct representation components and achieves adaptive interaction through a pseudo-Lorentz transformation. This design offers a research perspective on joint content-position modeling that differs from traditional Euclidean attention.
  2. The paper conducts extensive experiments on tasks involving language understanding, graph node classification, and image classification.
Critical weaknesses
  1. The Minkowski Attention Score can essentially be implemented via a concatenated Euclidean dot product combined with a fixed sign flip, while the input-dependent PLT can be interpreted as a form of dynamic feature mixing. Although Table 5 validates the roles of the PLT and aggregation modules, the absence of direct comparisons with simple dual-stream Euclidean attention, dynamic gating, and signed bilinear attention means it has not yet been fully demonstrated that Minkowski geometry itself is the key factor driving performance improvements.
  2. When the Query and Key employ different Boost Velocities, the final attention score still incorporates content-position cross-terms. Consequently, the proposed method primarily maintains a separation between content and position in terms of representation, rather than achieving strict interaction decoupling.
  3. Token-wise Minkowski Interval Invariance does not directly guarantee corresponding invariance in attention scores between different tokens, and the practical significance of the associated geometric interpretation requires further clarification.
  4. In Table 4, the Vanilla ViT utilizes a Pre-LN architecture, whereas the Minkowski ViT employs a Post-LN architecture; thus, the observed improvements in visual classification performance may stem from a combination of factors, including the normalization placement and changes to the attention mechanism.
  5. Minkowski Attention incurs approximately 16.7% higher GFLOPs compared to standard attention and demonstrates no significant computational advantage over DeBERTa configurations. While the paper emphasizes that using a single attention matrix reduces the computational cost of decoupled attention, it fails to report actual inference latency, training throughput, or peak GPU memory usage across varying sequence lengths; therefore, the efficiency advantage lacks sufficient validation.
Questions for authors
  1. Could the authors compare their method against simple dual-stream Euclidean attention, dynamic gating, or signed bilinear attention—using the same parameter count and computational budget—to further demonstrate the independent contributions of Minkowski geometry and PLT?
  2. Could the authors provide a direct comparison between Vanilla ViT and Minkowski ViT using identical Pre-LN or Post-LN architectures to rule out the potential impact of normalization placement on the performance gains shown in Table 4?
  3. When different "Boost Velocities" are applied to Queries and Keys, the final score still includes content-position cross-terms. Could the authors further explain the substantive advantages of this mechanism over existing decoupled attention methods, and how token-wise geometric invariance influences the final attention behavior?
  4. Could the authors provide additional data on actual inference time, training throughput, and peak GPU memory usage across different sequence lengths to validate the claimed computational efficiency advantages?
  5. Why do the attention weights in Figure 2 include negative values ​​and values ​​greater than 1? Could the authors clarify the specific calculation method and further validate the improvements in head diversity and semantic localization capabilities using quantitative metrics?
  6. Please provide further elaboration on the issues identified in W2.
2 comments
AuthorsOpenReview ↗

Response to Reviewer r5v6 (1/2): baselines, cross terms, invariance

Part 1/2

We thank the reviewer for the careful reading of our paper and the constructive feedback, which helped us improve its clarity and strengthen the experimental evaluation.

W1 / Q1 (dual-stream, gating and signed bilinear baselines). All models below use the same 20.5M-parameter architecture and training protocol as Table 2.

MethodSQuAD v1 EMSQuAD v1 F1MNLI-mMNLI-mmSST-2 AccParams
Dual-stream (MA (Parabolic), Table 2)48.45 ± 1.1059.63 ± 1.1669.11 ± 0.2070.57 ± 0.1882.93 ± 0.6520.5M
Signed bilinear (Table 5: STA ✓, PLT ✗)56.11 ± 2.9867.24 ± 3.0670.06 ± 0.2171.61 ± 0.1283.64 ± 0.9420.5M
Minkowski60.62 ± 0.2772.02 ± 0.3171.01 ± 0.2872.53 ± 0.1784.17 ± 0.4520.5M
  • Dual-stream is our model, boosts included, with a plus instead of the minus sign in Eq. 10. It loses 12.2 EM and 1.9 MNLI-m points, so the indefinite sign is essential.
  • Signed bilinear attention is already in the paper. It is Minkowski attention without the pseudo-Lorentz transformation (PLT): the score is the concatenated Euclidean dot product with a fixed sign flip that the reviewer describes, QiS⊤KjS−QiT⊤KjTQ_i^{S\top}K_j^{S} - Q_i^{T\top}K_j^{T}, with the same spatio-temporal aggregation (STA). This is the STA✓/PLT✗ row of Table 5. It differs from our model only in the PLT, which adds 4.5 EM, 4.8 F1 and 1.0 MNLI-m points and reduces the variance across seeds (EM std 2.98 → 0.27). The sign flip alone therefore does not explain the gains; the input-dependent boost adds to it. Without the STA as well (STA✗/PLT✗ row of Table 5), signed bilinear attention drops further, to 53.14 EM.
  • Dynamic gating is training with the same protocol; we will add its row and post an update as soon as it finishes. It mixes the two scores with an input-dependent gate inside a single softmax, as in ConViT: [(1−gi) QiS⊤KjS+gi QiT⊤KjT]/dh[(1-g_i)\,Q_i^{S\top}K_j^{S} + g_i\,Q_i^{T\top}K_j^{T}]/\sqrt{d_h}, with gi=σ(wh⊤[QiS;QiT]+bh)g_i=\sigma(w_h^\top[Q_i^{S};Q_i^{T}]+b_h) per head.

W2 / Q3 / Q6 (content-position cross-terms):

We agree with the reviewer's characterization and now state it explicitly in the revision: Minkowski attention keeps content and position separate in representation and makes their interaction structured, rather than strictly decoupled. With rapidities tanh⁡ϕ=u\tanh\phi = u, the score can be written exactly as (new Eq. 12 in Section 3.3)

⟨qi′,kj′⟩L=cosh⁡(Δϕij) (qi⊤kj−tq,i⊤tk,j)+sinh⁡(Δϕij) (qi⊤tk,j−tq,i⊤kj),Δϕij=ϕQi−ϕKj.\langle q_i', k_j'\rangle_L = \cosh(\Delta\phi_{ij})\,\big(q_i^\top k_j - t_{q,i}^\top t_{k,j}\big) + \sinh(\Delta\phi_{ij})\,\big(q_i^\top t_{k,j} - t_{q,i}^\top k_j\big), \quad \Delta\phi_{ij} = \phi_{Q_i} - \phi_{K_j}.

The cross-terms are therefore not free additive components, as in DeBERTa (c2p + p2c), but a single term whose weight is set by the content-dependent rapidity gap between query and key. It vanishes exactly when uQi=uKju_{Q_i} = u_{K_j} and grows smoothly as the two tokens' content/position balance diverges. This brings two substantive advantages over existing decoupled methods: (i) the strength of the content-position coupling is adaptive per token pair rather than fixed by architecture, and (ii) all four interactions come from a single attention matrix, with no relative-position embedding tables or extra projection weights. In practice, at equal parameter count (20.5M), this costs +63-73% inference latency over standard attention vs. +167-252% for DeBERTa (see W5), while outperforming DeBERTa on all five metrics of Table 2 (SQuAD EM/F1, MNLI-m/mm, SST-2). How token-wise invariance shapes this behavior is discussed in W3.

AuthorsOpenReview ↗

Response to Reviewer r5v6 (2/2): Pre/Post-LN, efficiency, Figure 2

W3 (Minkowski interval's invariance):

We agree, and Remark 1 (App. A.3) states it explicitly: Theorem 1 guarantees invariance of each token's own interval I=∥x∥2−∥t∥2I = \|x\|^2 - \|t\|^2, not of cross-token scores. Its practical significance is twofold. (i) A conserved content-position budget: for any velocity, the PLT redistributes energy between a token's content and positional parts without changing their signed difference. In particular, the sign of II is preserved, so a content-dominated token stays content-dominated. Additive positional encodings provide no such guarantee, since ∥E+P∥2\|E+P\|^2 shifts with position. (ii) Only relative boosts affect attention: token-wise invariance is the Δϕij=0\Delta\phi_{ij} = 0 case of the equation above. A token's own boost never alters its own geometry, and the score departs from the unboosted Lorentzian product only through the rapidity gap between query and key. The boost thus acts as a relative, content-driven modulation of attention rather than as an arbitrary re-scaling of each token. We moved this discussion from the appendix into Section 3.3 and softened the geometric wording accordingly.

W4 / Q2 (Pre-LN vs Post-LN in ViT): Here is the updated table with ViT post-LN with exact same number of parameters [FLN stands for Final Layer Norm],

MethodFLNCIFAR-10 ↑CIFAR-100 ↑TinyImageNet ↑ImageNette ↑ImageWoof ↑
ViT (Vanilla)✓90.53 ± 0.3766.25 ± 0.5851.89 ± 1.0182.41 ± 0.3158.51 ± 0.35
ViT (Vanilla)✗91.33 ± 0.3867.53 ± 0.1653.54 ± 0.2080.98 ± 0.8158.18 ± 0.70
ViT (Post-LN)NA91.42 ± 0.2270.63 ± 0.0656.54 ± 0.5283.41 ± 0.0960.15 ± 0.31
MViT (Post-LN)NA93.78 ± 0.3175.50 ± 0.3158.80 ± 0.7584.64 ± 0.2262.11 ± 1.21

W5 / Q4 (efficiency: latency, throughput, memory):

The latency, throughput and memory during inference,

BatchSeq. lenMethodLatency (ms/batch)Samples/sPeak mem. (GB)Latency Delta
64512Standard40.80 ± 0.931568.72.02—
64512DeBERTa117.21 ± 1.04546.02.10+187.3%
64512Minkowski66.65 ± 0.36960.32.18+63.4%
128512Standard79.85 ± 0.231603.04.09—
128512DeBERTa222.75 ± 0.30574.64.09+179.0%
128512Minkowski137.07 ± 0.48933.84.09+71.7%
256512Standard160.30 ± 0.461597.07.91—
256512DeBERTa427.66 ± 0.36598.67.91+166.8%
256512Minkowski276.59 ± 0.25925.67.91+72.5%
641024Standard125.16 ± 0.29511.34.09—
641024DeBERTa440.09 ± 0.35145.45.50+251.6%
641024Minkowski206.69 ± 0.30309.64.09+65.1%
1281024Standard249.62 ± 0.37512.87.91—
1281024DeBERTa835.21 ± 0.69153.39.72+234.6%
1281024Minkowski413.29 ± 0.81309.77.91+65.6%
2561024Standard495.78 ± 1.53516.415.55—
2561024DeBERTa1636.97 ± 0.53156.418.15+230.2%
2561024Minkowski823.07 ± 0.83311.015.55+66.0%

All measurements use bf16 autocast on a single A40 GPU, mean ± std over 20 runs after 3 warmup passes.

Comparison on training time, peak memory during training.

MethodParamsTraining timeTraining time vs StandardMemory per GPU (MB)
Standard20.5M6h 18min—25,171
DeBERTa20.5M10h 05min+60.1%27,761
Minkowski20.5M8h 58min+42.3%28,201

All models trained on 2 A40 GPUs.

Q5 (negative values and values > 1 in Figure 2): We apologize for the unclear label. Figure 2 plots the pre-softmax attention scores (logits), not softmax probabilities: qi⊤kj/dhq_i^\top k_j/\sqrt{d_h} for standard attention and −Mij/dh-M_{ij}/\sqrt{d_h} for Minkowski attention, averaged over all query-key pairs with the same relative offset j−ij-i. Logits are unbounded, so negative values and values above 1 are expected. Only the shape of each curve matters: a higher score at offset j−ij-i means the head prefers keys at that distance and direction. Standard heads share a similar, symmetric, local profile, while Minkowski heads specialize (local, long-range, directional). We relabelled the axis as "mean attention logit". Head diversity is quantified in App. A.5 via per-head attention entropy across all layers (Figures 6-9).

Reviewer_bXKvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This submission introduces Minkowski Attention, which represents token content and position as space-like and time-like components in a generalized Minkowski space. The mechanism incorporates pseudo-Lorentz transformations to compute attention scores, followed by spatio-temporal aggregation (STA) to combine the transformed value representations and positional embeddings using the resulting attention weights. The authors further build a Minkowski Transformer using the proposed attention layer and conduct extensive experiments on language understanding, node classification, and image classification. The results demonstrate improvements over several evaluated baselines.

Critical strengths
  1. The paper introduces an interesting and broadly applicable attention formulation that uses standard tensor operations without constraining representations to a curved manifold. I find the motivation compelling and the proposed solution creative.
  2. The authors evaluate the method across various tasks, modalities, and language-model sizes, providing clear comparisons of performance and parameter counts. The experiments offer strong empirical support for the advantages of the proposed method.
Critical weaknesses
  1. Although Table 2 reports parameter counts and FLOPs, the practical efficiency evaluation could be more complete. Reporting peak GPU memory usage and inference latency would help clarify the computational cost in practice.
  2. It would be helpful to discuss whether Minkowski Attention can scale to substantially larger models, particularly at the LLM scale. Preliminary results, if available, or a discussion of potential computational and training stability challenges would clarify its applicability beyond the evaluated model sizes.
Questions for authors

Please address my concerns in Critical Weaknesses section

1 comment
AuthorsOpenReview ↗

Response to Reviewer bXKv: latency, memory and scaling

We thank you for your review, for finding the motivation compelling and the experiments convincing. Below we address the concerns of the reviewer.

Memory and latency. We now report wall-clock latency, peak memory and training time (new Appendix "Computational Cost", which covers several batch sizes and sequence lengths). For the 20.5M language model of Table 2, with the implementation used in the paper:

Table: Inference latency for batch × sequence length (ms) and peak memory of the full 20.5M language model on one A40 GPU (bf16), and pre-training wall-clock time (125k steps, 2 A40 GPUs); percentages are relative to standard attention.

MethodLatency, 64×512 (ms)Latency, 256×1024 (ms)Peak mem., 256×1024 (GB)Pre-training time
Standard40.8495.815.66h18
DeBERTa117.2 (+187%)1637.0 (+230%)18.210h05 (+60%)
Minkowski66.7 (+63%)823.1 (+66%)15.68h58 (+42%)

Minkowski attention computes a single attention matrix per head, so its overhead is well below DeBERTa's and its inference memory matches standard attention.

Scaling to LLM size. We have not trained at LLM scale. We state this as a limitation. Our results in smaller models are encouraging. In Table 1, Minkowski attention matches or beats standard attention on every metric across all scales from 20.5M to 110M parameters, on tasks fine-tuned after masked language modeling pre-training. The models trained stably. Given this evidence we can speculate that on the model side there would be no reason to think it would not scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.