acceptodds
Under review as a conference paper at ICLR 2027

MoSAR: Mixture of Semantic Attention Regimes for Learning Adaptive and Approximable Attention Geometries

Abstract

The quadratic complexity of dense self-attention remains a central bottleneck for long-context language modeling. Many efficient alternatives address this cost by deciding in advance where attention should be sparse or local. We argue that attention approximation should instead be approached as a geometric problem, with the relevant interaction geometry learned from data: natural-language dependencies are input-dependent and difficult to prescribe in advance, so the model should learn where positional relevance can decay and where broader interactions must be preserved. We introduce Mixture of Semantic Attention Regimes (MoSAR), which learns such an adaptive, controlled-decay geometry over query–key interactions. Input-conditioned query and key routers, applied after positional encoding, select mixtures over short, medium, and global regimes, inducing a continuous distance-dependent attention field rather than a fixed sparsity pattern. This geometry is learned during training and can subsequently be discretized through top-1 routing. In controlled pre-training experiments with matched 500M-parameter models, MoSAR learns a substantially lower-reach attention geometry without degrading language-modeling quality, improving perplexity over dense RoPE at the training context length. Under length extrapolation, MoSAR achieves the best perplexity among all evaluated variants, including strong baselines such as ALiBi. Moreover, the learned geometry remains stable under deterministic top-1 discretization, suggesting that it is not only adaptive, but also amenable to low-cost approximation at inference time.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Transformers, Attention Mechanisms, Positional Encodings, Efficient Attention, Sparse Attention, Long-Context Language Modeling, Learned Routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PLKUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MoSAR, an attention mechanism that uses separate query and key routers to select mixtures of predefined short-, medium-, and global-range decay regimes. The resulting content- and distance-dependent field is incorporated as an additive attention bias. Controlled pretraining experiments with approximately 500M-parameter models show competitive perplexity at the training context length and improved length extrapolation.

Critical strengths
  1. The method is clearly explained, and routing on both the query and key sides is an intuitive design.
  2. The experiments are well controlled and include useful fixed-decay and hard-window baselines.
Critical weaknesses
  1. Downstream evaluation is missing. Perplexity alone does not show whether the model preserves useful long-range dependencies. Benchmarks such as RULER would make the long-context claims more convincing.
  2. The comparison is too narrow. Comparisons with more related methods, such as Inkling, would help establish where MoSAR stands beyond positional-encoding and fixed-locality baselines.
  3. The benefit of adaptive routing needs stronger evidence. Fixed-M essentially matches MoSAR at 2k, and the advantage over ALiBi at 8k is small. Single-sided routing ablations and repeated runs would help.
Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Paper update

Dear reviewer,

thank you for appreciating the intuitive idea of ​​MoSAR and finding the experimental section useful.

We have rewritten the paper and reported new inference results, which you might find interesting.

Even though you didn't ask questions, we saw the weaknesses, and we will certainly take them into account for future work.

In particular, we believe that addressing these weaknesses requires resources we don't currently have.

But if the paper were accepted, it would certainly be easier for us to apply for hardware resources, and obtain them.

Best regards

Reviewer_FhK3OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoSAR, an attention mechanism in which token-conditioned query and key routers select mixtures of distance-decay profiles at different scales. The aim is to learn an attention geometry that preserves useful interactions while allowing less relevant interactions to be approximated or omitted. The authors evaluate the method through controlled pretraining of 500M-parameter models and length extrapolation from 2K to 8K tokens.

Critical strengths
  1. The central idea is reasonable. Allowing contextual representations to influence attention range offers greater flexibility than assigning every token the same locality pattern.
  2. The controlled comparisons are useful. Fixed decay profiles and a hard-window baseline help distinguish the effects of smooth decay, locality, and adaptive routing.
  3. The authors acknowledge several limitations, including the absence of an efficient sparse implementation. The soft-routing results provide encouraging evidence that distance-dependent decay can improve language-modeling perplexity.
Critical weaknesses
  1. The main approximation claim is not directly validated. Section 3 states that the top-1 evaluation still computes all causal query–key similarities and retains the finite decay floor outside the nominal support. This evaluates router discretization, rather than the removal of interactions required for sparse execution. Before concluding that the learned geometry supports accurate sparse approximation, the authors should evaluate the proposed support with excluded interactions actually masked out. This experiment does not require an optimized sparse kernel.
  2. The advantage of adaptive routing over simpler decay profiles is modest. At the training length, Fixed-M slightly outperforms soft MoSAR, while Fixed-S performs similarly. More importantly, Fixed-S achieves lower perplexity than top-1 MoSAR at every evaluated length: 12.313 versus 12.703 at 2K, 11.864 versus 12.216 at 4K, and 12.453 versus 12.872 at 8K. Although the reported reach metrics do not constitute a complete computational comparison, these results leave the benefit of adaptive routing after discretization unclear. Comparisons at matched support density would better establish its value.
  3. The additional hyperparameters introduce an unresolved practical trade-off. MoSAR requires choices concerning regime scales, plateau fractions, decay shape, routing configuration, and, for MoSAR-Cost, the auxiliary-loss coefficient. The paper provides limited evidence about sensitivity to these choices or their transfer across datasets, model scales, and context lengths. Without such analysis, it is difficult to assess whether the gains justify the added tuning burden.
  4. The novelty relative to existing adaptive-attention methods needs sharper justification. The relevant comparison extends beyond positional embeddings to learned attention spans, content-dependent attention selection, and sparse routing methods discussed in Section 2. Combining query- and key-conditioned routing with multiple decay profiles is a plausible contribution, but the experiments do not sufficiently isolate why this combination is necessary or what it achieves beyond simpler alternatives.
Questions for authors
  1. What happens to perplexity and retrieval accuracy when interactions outside the proposed top-1 support are actually removed? How much attention mass is discarded?
  2. Does adaptive routing outperform fixed decay profiles or learned input-independent mixtures at matched support density and quality?
  3. How robust are the gains across seeds, datasets, and hyperparameter choices? What evidence shows that the routers preserve task-relevant long-range interactions?
  4. Given that existing RoPE-based sparse-attention systems already support strong long-context performance, what specific limitation does MoSAR address, and how would the authors demonstrate an advantage over those approaches?
1 comment
AuthorsOpenReview ↗

Paper update

Dear reviewer,

thank you for appreciating the idea of ​​MoSAR and recognizing its potential. Here are brief answers to your questions:

  1. We've created several paragraphs throughout the paper that answer the question. Please start with the experiments section.
  2. We've included some very comprehensive tables describing what happens; we kindly invite you to look at them.
  3. We've run several experiments with different floor parameters to see how the approximation changes as the parameter varies. We're currently unable to conduct experiments of large scale, but surely, in the future, especially if this paper is accepted, it would be easier for us to obtain resources to conduct larger-scale experiments, which is essentially our number one concern.
  4. We believe the strength of MoSAR lies in its underlying idea: that is, if the model is trained well, which requires massive resources, it can, in principle, learn whether a similarity is useful or not, unlike baselines with fixed geometries. This seems possible in the paper. But we need to work on approximation, but given the existing literature on MoEs, which is now socially accepted, we believe significant progress can be made. This is because, while MoEs activate parts of FFN during inference, the idea of ​​MoSAR is to activate only the parts of attention deemed most relevant for inference.

Best regards

Reviewer_CDWfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MoSAR, which adds a distance-decaying bias to attention, with the goal of making attention amenable to sparse approximation and thereby alleviating its quadratic complexity. The authors predefine three decay curves (short, medium, and global) and use small router networks to select a curve separately for the query and the key of each token, so that the decay range adapts to the input content. At inference time, the routing can be discretized via top-1 selection, so that the query–key pairs to be computed can be determined before the attention scores are calculated. The authors pretrain models of about 500M parameters from scratch on English Wikipedia and compare perplexity against RoPE, ALiBi, and other methods, both at the training length and when extrapolating to 4k and 8k tokens.

Critical strengths
  • The proposed idea is relatively novel. By assigning regime labels to queries and keys separately, the method can determine which query–key pairs need to be computed before computing q⊤kq^⊤k, which in principle has the potential to reduce the complexity of attention.
  • The experimental results show a consistent improvement over dense RoPE.
Critical weaknesses
  • The paper's positioning is inconsistent with its experimental evidence. The paper is motivated by alleviating the quadratic computational cost of attention, yet the method actually increases computation (through the additional routers and bias matrix). Even in the top-1 discretization setting, the computation remains dense: positions outside the support still participate in the computation with weight e−6e^{-6}, and the paper does not report perplexity when these positions are masked out. Moreover, under the top-1 setting, MoSAR has worse perplexity than Fixed-S at 2k, 4k, and 8k, even though Fixed-S has a smaller reach.
  • The experiments are limited in scope: the models are trained and evaluated only on Wikipedia, with a single model size and a single training length.
  • The paper has several presentation issues, especially in the method section, which make it difficult to follow. Several symbols are not clearly defined when they first appear, and some symbols carry multiple meanings. For example, gGGg_{GG} is used before S, M, and G are defined, and the meaning of S, M, and G is not explained when they first appear. In addition, the same symbol is used for different quantities: B denotes the barrier depth of the decay floor, the attention bias, and the upper bound on the value vectors in the lemma. Such issues make the method section considerably hard to read.
Questions for authors
  • Under the top-1 discretization setting (the only setting that corresponds to the paper's motivation of alleviating the quadratic complexity), what is the perplexity of MoSAR at 2k, 4k, and 8k if positions outside the support are masked out and dense computation is removed? Does MoSAR still have an advantage over Fixed-S and Fixed-M under the same setting? If possible, please also report the computational cost, such as FLOPs.
  • Could the authors provide results for more model sizes and training lengths?
  • Could the authors improve the overall writing of the method section to make the paper more readable?
1 comment
AuthorsOpenReview ↗

Paper update

Dear reviewer,

thank you for appreciating the work and recognizing the potential of MoSAR. Here are brief answers to your questions:

  1. We've created a section in the results section and serveral appendices that answers your question.
  2. Unfortunately, it wasn't feasible to conduct large-scale experiments.
  3. We've certainly made an effort to improve the presentation. In particular, we corrected the ambiguous notations and the order of the definitions. We also provide a better argument and proof of the lemma that you might apprechiate. Thank you for reading it, evn though you were under no obligation.

Best regards.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.