acceptodds
Withdrawn from ICLR 2027

Nexus: Higher-Order Attention Mechanisms in Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Transformers have achieved broad success across language and vision, using self-attention to model dependencies among input elements. Yet the relevance of one position to another often depends on evidence distributed across a wider context. Standard attention forms each matching score from a query and a key projected at the current layer, whose inputs may already carry context from earlier layers. Context-aware transformations and multi-token interactions enrich this process, motivating further study of how context can enter the matching operation itself. We introduce Nexus, a higher-order attention mechanism that refines queries and keys through separate causal attention branches before their final comparison. The construction extends recursively while reusing the original projections. Our analysis shows how other prefix tokens can influence comparisons within a single attention head. Experiments across four from-scratch model scales yield higher six-task averages than standard attention, and adapting pretrained Qwen2.5 models with Nexus improves reasoning-task averages after supervised fine-tuning. These results support recursive query/key refinement as a design for context-dependent matching in Transformer decoders.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Transformer Architecture, Self-Attention, Higher-Order Matching

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jF4DOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Nexus, an attention mechanism that refines queries and keys through separate causal self-attention operations before computing their final matching scores. Each refinement uses the prefix visible at the corresponding position, while the final attention aggregates the original values. The construction can be nested to obtain higher-order attention and reuses the existing projection parameters. The paper evaluates Nexus through language-model pretraining at multiple scales and supervised fine-tuning of pretrained models.

Critical strengths
  1. Clear architectural design; Nexus introduces contextual information into query–key matching while preserving causality and reusing existing projections.
  2. Useful theoretical analysis; the paper demonstrates how additional visible tokens can change relative attention weights within a Nexus operation, distinguishing it from the specified class of token-local scoring functions.
  3. Evaluation across multiple model scales; experiments include both training from scratch and adapting pretrained models, with results averaged over three independent training runs.
  4. Relevant architectural controls; Nexus achieves slightly higher average accuracy than the 11-layer Standard model at approximately matched analytical computation and with fewer parameters. It also outperforms the tested shared hidden-state-loop baseline at lower analytical cost.
Critical weaknesses
  1. Comparisons with alternative attention mechanisms are limited; MTA and Differential Transformer are evaluated only at 70M. Other relevant methods, including context-aware self-attention, DCMHA, and Tensor Product Attention, are discussed but not evaluated. Comparisons with suitable causal implementations would strengthen the empirical assessment.
  2. The advantage over MTA is mixed; both methods achieve a reported average accuracy of 0.406, while MTA requires 1.07× Standard’s analytical forward computation compared with 1.44× for Nexus. Nexus achieves lower NLL, but its additional computation does not produce a higher displayed task average.
  3. The parameter-efficiency evidence could be strengthened; the shared hidden-state-loop baseline repeats only the attention sublayer. Reusing complete Standard Transformer blocks would provide another way to increase computational depth without adding parameters. Comparing these alternatives under similar parameter and computational budgets would better assess the benefit of query/key refinement. The existing architectural controls are also limited to 70M.
  4. Uncertainty around the improvements is insufficiently characterized; although results are averaged over three training runs, standard deviations or confidence intervals are not reported. Cross-run win counts do not establish whether small differences, such as the 0.3-percentage-point gain over the 11-layer Standard model, are statistically reliable.
Questions for authors
  1. Can the authors evaluate additional relevant attention mechanisms, particularly contextual query/key methods, and extend the strongest baseline comparisons beyond 70M?
  2. How do Nexus and MTA compare under matched total training computation? Does Nexus’s lower NLL translate into downstream improvements at larger scales or with further training?
  3. How does Nexus compare with a Standard Transformer that reuses complete blocks, with both parameter count and computation approximately matched?
  4. Can the authors provide the individual training-run results and uncertainty estimates for the differences against the 11-layer Standard, shared hidden-state-loop, and MTA baselines?
Reviewer_xrEYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces Nexus, an attention mechanism that constructs context-dependent matching by refining queries and keys. The default second-order variant aggregates context through separate causal query–query and key–key attention operations, then performs the final query–key comparison using the refined states while aggregating the original values. The construction reuses the original projections and extends to higher orders through recursive refinement.

The paper analyzes prefix dependence relative to token-local scoring, information beyond the base cross-score, and local matching corrections associated with recursive order. Experiments cover four from-scratch model scales, architectural ablations at the 70M scale, and retrofitting Qwen2.5 models followed by supervised fine-tuning, reporting improvements in average performance over standard attention.

Critical strengths
  1. The paper presents a clearly specified contextual matching mechanism that separates query/key refinement from final value aggregation, reuses existing projections, and fits within Transformer decoders. The analyses of prefix-conditioned preferences, information beyond the base cross-score, and local properties of recursion provide a theoretical characterization of the construction.

  2. The experiments cover multiple from-scratch model scales and pretrained-model adaptation, with controls involving increased depth, hidden-state recurrence, and other attention mechanisms. Improvements in average performance across several settings, together with architectural comparisons at approximately matched computational budgets, support the effectiveness of the method.

Critical weaknesses
  1. Nexus achieves a lower NLL than MTA, but further evidence would help establish whether the additional computation associated with two independent self-similarity maps offers advantages over simpler contextualization schemes. Targeted comparisons with a shared relation map or simple causal mixing would help clarify the benefits and costs of this design.

  2. Important related methods, including JAT and CASTLE, are currently compared primarily through structural analysis, while the considerations underlying their exclusion from the experimental comparisons remain unclear. Further explanation of the baseline selection criteria and the reasons for omitting these methods would help clarify the scope of the experimental comparisons.

Questions for authors
  1. Have the authors considered or evaluated lower-cost contextualization schemes, such as shared relation maps or simple causal mixing? In light of the accuracy, NLL, and computational costs reported for MTA and Nexus, could the authors further clarify whether the benefits of two independent self-similarity maps justify their additional computational cost? Targeted comparisons would help clarify the value of this central design choice.

  2. Have the authors considered including important related methods discussed in the paper, such as JAT and CASTLE, in the experimental comparisons? Are there specific considerations regarding adaptation, implementation, or computational budget that motivated the baseline selection? Such clarification would help assess whether the experimental comparisons cover the principal related approaches.

Reviewer_SoHnOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a modification to the attention mechanism with the goal of helping contextualize better. Prior to the query-key matmul, they propose a series of operations designed to make each query or key representation more contextual. From what I can tell, the approach is roughly softmax(QQ^T, dim=1) @ Q, and would essentially smooth the queries (and keys) before query-key attention. The authors then conduct experiments on models of varying scales (from 70M to 1B) on some reasoning / knowledge datasets, finding that the approach improves performance on average under parameter-matched comparisons.

Critical strengths
  • The idea represents an elegant and targeted modification to attention.
  • The architectural modification does not require new parameters, but permits additional computation within the architecture.
  • The reported results improve over the baselines, when measured on a per-parameter basis.
Critical weaknesses

First, the writing of the paper is very hard to follow. The abstract is unclear and does not immediately establish the area of work this paper falls into, or the approach the authors intend to take. Prefix tokens are mentioned out of the blue, and from what I recall never addressed later the the paper. The introduction as well is lacking in both its summary of the existing research context, and the clarity with which it discusses the contribution. It does not provide intuition for why this approach is needed, given that self-attention allows for contextualization across multiple layers.

Second, a large amount of space in the paper is devoted to proofs which do not address fundamental questions about the proposed architecture. Proofs establish properties / advantages of NEXUS in an extremely unfair comparison to existing self-attention. Rather, I suggest the authors focus on explain why query-to-query and key-to-key smoothing allows the model to perform better.

Third, the paper does not provide documentation for the actual model architectures, nor reasoning for why the evaluation datasets were selected. The experimentation section mentions "Pythia-syle" models, as well as various ablations, but none of the models are every precisely specified. Datasets are referenced, but never justified. How do the selected datasets help analyze the claims made about contextualizing queries against queries?

Finally, I find that the proposed idea is quite unintuitive and this is never addressed by the authors. They make bold claims about the inability of attention to contextualize, yet multiple layers of attention very clearly can express global receptive fields. Perhaps a comparison of local attention + NEXUS would have been warranted, and then the authors could FLOP-match better. A priori, I would expect the proposed architecture to reduce expressiveness and really function more like regularization, given that they seem to be smoothing query and key matrices using softmax'd similarities, which intuition suggests would have the effect of (approximately) reducing the variance across the query and key matrices, and I imagine this narrows the information bottleneck somewhat. I'd like to see proofs or experiments exploring this.

Questions for authors
  1. Why is NEXUS needed when multi-layer attention achieves contextualization?
  2. What are the theoretical advantages and disadvantages in expressivity of NEXUS?
  3. What are the exact model architectures that were tested in the experiments section?
  4. What was the exact nature of AI usage in manuscript preparation? Which parts of the manuscript were AI written, and which parts were drafted by the authors?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.