- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors propose Minkowski attention, an extension of standard dot-product attention to separate the processing of content and position when positional information is present. The method replaces the euclidean inner product in attention with an Minkowski inner product and preprocesses with individual learned Lorentz transformations. The authors find that the method systematically outperforms baselines on language, graph, and vision benchmarks. However, the method has many seemingly arbitrary design choices and central aspects are not ablated, and compute cost is not discussed.
I rated the paper a weak reject due to the missing formal justification and the poor ablation of design choices, although I think the method is promising. In 2026 with agents and so on I believe that strong empirical performance alone is not sufficient for a good paper, and the understanding of why the method works and to rule out alternatives is increasingly important. I look forward to the rebuttal as I think this paper can be improved significantly. Also, the paper is missing the required AI use statement; I did not include this in my rating and leave this to the AC to handle.
Critical strengths
- The method outperforms all baselines across language, graph, and vision tasks. I find this impressive because the method is simple and it should be easy to apply it to a wide variety of tasks that use transformers on tasks that require positional information. The experimental setup seems credible as the authors report uncertainties and explain their hyperparameters in detail, although they do not provide their implementation.
Critical weaknesses
- The paper does not include an AI use statement. From my understanding of the author guidelines this is a strict requirement. I did not include this aspect in my rating below, and leave it to the AC to decide how to handle this.
- The method lacks a sound justification for the design choices in Section 3. Also, there are several unexplored design choices that should be ablated to properly understand the method, see the second point in "Questions For Authors". Currently Section 3 reads like the authors found this by trying and it just works well in experiments, but the physics motivation used to introduce the method is weak and there are just so many open questions. I think this is a missed opportunity and the interpretability of the method would benefit significantly from finding something better (and potentially modifying the method without losing performace). For instance, a starting point could be the tensorial message-passing of https://arxiv.org/pdf/2505.20280, where the authors apply learned Lorentz transformation to QKV such that attention is performed in a global frame, whereas all other operations are performed in per-particle local frames; I think the proposed Minkowski attention is equivalent to this approach, but with different local frames for each attention block.
Questions for authors
- How expensive is the method? I believe this is the central open question required to decide if the method is useful. The authors include GFLOPs for the language study in Table 2, but no timings. If the method is more expensive, the authors should consider to compare performance at fixed compute cost. I suspect that the method requires a custom GPU implementation to be useful, and in 2026 with coding agents I think its feasible to expect the authors to ship this with the paper. In any case, this aspect should be discussed in more detail.
- Equation 6, the construction of the velocity, deserves more explanation and ablations. Enforcing is a clear restriction, the natural choice is , and the authors provide no explanation or ablation. It also seems unnatural to me to construct the velocity from the same token and context embeddings that enter attention without a learned modification; the natural choice for me would be to have a separate weight matrix multiplying the context and position encodings before taking their norms in Equation 5.
- The authors do not discuss their choice of spacetime metric, i.e. and for . This is a central hyperparameter of the method and should be ablated. What happens in the regimes and ?
Minor points:
- What does "FLN" in Table 4 mean?
- In Table 3, the authors report two digits after the comma for their method, but only one digit for other methods. The extra digit is in uncertainty, so I think they should just always give one digit.
- Typos: L323 "... exciting future research (direction?).", L364 "... vanil(l?)a ..."
- Suggestion to improve clarity: In Section 3 both the values and the velocity are denoted by . You could instead write for the velocity to avoid confusion.
- Suggestion for related work: Lorentz-equivariant transformers (https://arxiv.org/pdf/2405.14806, https://arxiv.org/pdf/2505.20280) use attention with Minkowski inner products to construct Lorentz-invariant attention logits.
2 comments
Thank you for the responses. My conceptional concerns about the compute cost aspect are settled. However, I believe that there is still room to further reduce the overhead, as 1.3x slower training and 2x slower inference feels a bit too much for the quite small change to the method. My understanding is that you did not use torch.compile for these measurements, maybe that helps. Thank you also for the ablation on the velocity design, it's good to see that the tanh approach gives similar but slightly worse performance.
My main concern about the missing motivation for the design choices still stands, and I will therefore keep my score at "weak reject". I will keep track of the discussion with the other reviewers and might increase my score later on, but currently I don't see it going above "weak accept". To be explicit, my concerns are not about missing experiments or performance or so, but about a better understanding of why the concrete choice of the authors works so well. There is probably a wide range of other design choices that yield similar performance, and I am confident that some of them have a clear motivation (and probably work better), which would make it much easier for others to understand why the method works and to convince them to adapt them. I am aware that figuring this out takes more time than the two weeks for the rebuttal. The current paper reads a bit like a wasted opportunity to me.
Response to Reviewer 79Di: cost, custom kernel, velocity, signature
AI use statement: Thank you for pointing this out. We added an LLM Usage Statement to the revised paper (after the conclusions).
Q1 (computational cost, fixed cost, GPU kernel): We thank the reviewer and now report actual runtime and memory for both inference and training. All models have 20.5M parameters and are implemented with identical unfused PyTorch operations.
IL-512 / IL-1024: inference latency per batch at sequence length 512 / 1024 (single A40 GPU, bf16, batch size 128, mean ± std over 20 runs after 3 warmup passes). TT: total training time on 2 A40 GPUs. Mem/GPU: training memory per GPU.
More detailed results are in the new Appendix "Computational Cost". Minkowski attention is more expensive than standard attention but substantially cheaper than DeBERTa, the decoupled baseline it is designed to replace, and its relative overhead does not grow with sequence length.
On a custom kernel: none is required. The score is a single dot product over concatenated head vectors, −Mij=[qi′,tq,i′]⊤[−kj′,tk,j′], and the aggregation is ∑jPij(Vj′−Tj). Minkowski attention can therefore be computed with one standard fused attention call (e.g. FlashAttention / PyTorch SDPA) of head dimension 2dh, with the boosts applied beforehand as a cheap elementwise operation. With PyTorch's fused attention and
torch.compile, Minkowski attention trains at 8192 tokens, with a 1.4–1.8× overhead over equally optimized standard attention (Appendix "Computational Cost").Q2 (velocity, Eq. 5–6, domain of u):
Ablation Study on the choice of Sigmoid function for u,
Q3 (choice of p and q as hyperparameters Rp,q, cases p=0 and q=0): Minkowski attention is defined for p=q; we now make this explicit in the abstract ("(d+d)-dimensional signatures") and in Section 3.3 (each head operates in Rdh,dh). Although the Minkowski product itself would allow any (p,q), the pseudo-Lorentz transformation pairs every time-like coordinate with a space-like one (x′=γ(x−ut), t′=γ(t−ux)), which requires p=q. Each head therefore lives in Rdh,dh, so p and q are not an additional hyperparameter, they act as an ordinary embedding dimension hyperparameter. Consequently, the cases p=0 and q=0 coincide (p=q=0) and are improper, they correspond to a model with embedding dimension 0.
Minor points. All corrected in red. FLN is the final LayerNorm before the classification head (now defined in the caption of Table 4; NA for Post-LN models). Table 3 now uses one decimal for our method, the two typos are fixed, and the velocity is now denoted by u. We added L-GATr and LLoCa to the related work: they use the Minkowski product to obtain Lorentz-invariant logits for physical four-vectors, whereas we apply it to learned content and positional embeddings.