acceptodds
Under review as a conference paper at ICLR 2027

Ask Self, Ask Others: Relation Is All You Need

Abstract

Attention dominates token mixing, but it collapses relation formation and flow allocation into a single score-to-flow step. We introduce Relation, which separates them by first organizing pairwise evidence into explicit Self and Exchange relations and deriving information flow afterward. Relation first decides whether a token should rely on itself or draw from its history, and if it draws from history, where to look. This relational organization gives rise to Full Relation, FlashRelation, Linear Relation, and Hybrid Relation. Across matched decoder-only models, Full Relation achieves lower mean final-validation NLL than MHA and reaches the paired MHA final training loss with 4.5–7.3% fewer tokens. Structural diagnostics further show that Relation learns a distinct depth organization: the first layer acts as a current-token anchor and a high-rank router, while later layers shift strongly toward history. In a fixed-context reference benchmark, FlashRelation is 4.17–5.28× faster than the materialized Full Relation implementation. Across scale-matched production workloads, it reaches 89.7–92.9% of PyTorch FlashAttention throughput while executing the exact Full Relation operator. Hybrid Relation demonstrates that Full and Linear Relation layers can be composed within a single decoder. These results support a relation-first view of token mixing: ask Self, ask Others, then let Flow follow Relation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Relation, Self-Exchange Relation, Token Mixing, Language Modeling, Efficient Sequence Modeling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Yvm8OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces Self‑Exchange Relation (SER) as an alternative token‑mixing primitive for decoder‑only language models. It separates relational construction and information flow, yielding four practical implementations. Experiments on 10M–100M models show better token efficiency than MHA. FlashRelation achieves high throughput close to FlashAttention, and Hybrid Relation validates layer‑wise composition.

Critical strengths
  1. Novel foundational paradigm for token‑mixing. The paper proposes the "Relation‑first, Flow‑follows‑Relation" design philosophy, cleanly separating relational construction and information flow allocation. It is not a minor tweak of multi‑head attention but delivers an alternative foundational viewpoint for sequence‑modeling research.
  2. Systematic practical operator family together with rigorous empirical analysis. Starting from the core SER primitive, the authors systematically derive Full, Flash, Linear and Hybrid Relation, covering exact computation, GPU‑fused execution, recurrent compression and layer‑hybrid composition. Beyond standard perplexity metrics, comprehensive structural diagnostics, multi‑seed repeated runs and hardware throughput benchmarks strengthen the empirical reliability.
Critical weaknesses
  1. The robustness of the proposed method under different hyperparameter choices has not been fully explored. Several key hyperparameters are fixed throughout the main experiments, including the self‑temperature τS=2\tau_S=2, activation selections (sigmoid for Self branch, SiLU for Exchange branch), and initialization of the layer‑wise count‑calibration coefficient λℓ\lambda_\ell. Without systematic ablation studies over these choices, it remains difficult to tell whether the reported performance gains generalize broadly or are tied closely to these particular configurations.
  2. The empirical evaluation scope is relatively limited, and some relevant baseline comparisons are missing. All main experiments are conducted on decoder‑only models up to ~100M parameters, while evaluations on larger models, encoder‑centric tasks, multimodal and instruction‑tuning settings are not provided. Moreover, the Linear Relation variant has not been compared against representative recurrent and linear‑time token‑mixing methods such as Retentive Network, KDA, and Mamba‑family models under matched computational budgets. Only one fixed (LLLF)3(LLLF)^3 interleaving layout is tested for Hybrid Relation, leaving alternative layer composition strategies unexplored.
Questions for authors
  1. Could you please provide ablation results on the key fixed design choices, including the self‑temperature τS\tau_S, alternative activation functions for the Self / Exchange branches, and the count‑calibration coefficient λℓ\lambda_\ell? This would help clarify whether the observed improvements are sensitive to these hyperparameter selections.
  2. For the Linear and Hybrid Relation variants, can you add controlled comparisons against competitive recurrent and linear‑time token‑mixing baselines under matched experimental setups? For Hybrid Relation, it would also be helpful to see an analysis of modeling quality and system overhead trade‑offs for layer interleaving patterns beyond the (LLLF)3(LLLF)^3 layout adopted in the paper.
Reviewer_bEJrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Relation, a token-mixing operator based on explicit Self and Exchange relations. Instead of directly converting pairwise interactions into information flow as in attention, the method first constructs a relation matrix consisting of Self and Exchange components and then derives normalized flow from this relation structure. The framework includes Full Relation, FlashRelation, Linear Relation, and Hybrid Relation. Experiments on decoder-only language models up to 100M parameters report improved validation NLL compared to matched multi-head attention baselines, and FlashRelation provides an efficient implementation of the Full Relation operator.

Critical strengths
  • Attempts an alternative design for token mixing beyond standard attention The paper explores an alternative token-mixing design based on explicitly separating Self and Exchange interactions and develops several variants within this framework, including Full Relation, FlashRelation, Linear Relation, and Hybrid Relation. While I am not fully convinced by the paper's broader claims, I appreciate the effort to investigate a different architectural design space beyond standard attention formulations.
Critical weaknesses
  • The empirical validation is not sufficient to support the paper's claim of a new token-mixing primitive

My main concern is the mismatch between the ambition of the paper's claims and the scope of its experimental validation. The paper presents Relation as a new token-mixing primitive and as an alternative organizing principle to attention. However, all language-modeling experiments are conducted on relatively small models (10M-100M parameters), with evaluation limited to this narrow scale regime. The paper therefore provides evidence that the approach can work in small-scale settings, but not that it remains effective, stable, or practically relevant at scales where modern language-model architectures are typically evaluated.

This concern is amplified by the fact that important components of the method appear to require explicit mechanisms such as count calibration to maintain desirable behavior. Yet the paper provides little evidence regarding how these design choices behave under substantially larger models or much longer contexts. Given the scope of the claims, I do not believe the current experimental evidence is sufficient to establish Relation as a compelling alternative token-mixing primitive.

  • The paper does not convincingly connect its “relation” interpretation to the observed improvements

The central narrative of the paper is that explicitly modeling relations before information flow leads to better behavior than attention-style token mixing. However, the evidence presented does not clearly support this conclusion.

From an operational perspective, the proposed method still relies on pairwise dot-product interactions, softmax normalization, and weighted aggregation. The most visible differences are the separate nonlinear transformations applied to Self and Exchange entries together with the count-calibration mechanism. As a result, it remains unclear whether the reported gains originate from a meaningful relational modeling advantage or simply from a particular score transformation and normalization design.

More importantly, the analysis primarily demonstrates that Relation behaves differently from MHA, not that the proposed relational structure is responsible for the performance improvements. The paper therefore does not convincingly establish a causal connection between its central conceptual claim ("relation before flow") and the empirical gains. Consequently, I am not yet convinced that the proposed method captures substantially more meaningful relational structure than a modified attention mechanism with a different inductive bias.

Questions for authors
  • The paper fixes the Self mapping to a sigmoid function with τS=2\tau_S=2 and the Exchange mapping to SiLU throughout all experiments. Did the authors explore alternative choices for either mapping function, and if so, how sensitive are the results to these design decisions? Understanding whether these choices are important or merely convenient implementation choices would help clarify the robustness of the framework.

  • The paper reports that count calibration is important for performance. Could the authors provide additional information about the learned values of λℓ\lambda_\ell​ across layers and scales? In particular, do consistent patterns emerge across model sizes, or do the learned calibration values vary substantially between runs?

  • The paper introduces Givens-based adjacent-head mixing as part of the Multi-Head Relation design. Since the reported ablations suggest a relatively small impact from this component, do the authors view it as an essential part of the framework or primarily as an architectural refinement? Additional discussion would help clarify its role within the overall design.

  • The evaluation primarily compares Relation against standard MHA, even though the paper's main motivation is explicit relation modeling. While I am not an expert on this particular literature, my understanding is that there exists prior work incorporating relation-aware attention, relative representations, and other mechanisms for modeling token relations. Could the authors comment on why such approaches were not included as baselines? Comparisons against more closely related methods would help clarify whether the reported gains stem from the proposed relational formulation itself rather than a particular score transformation or inductive bias.

Reviewer_zo2NOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents a novel attention mechanism designed to decouple relation formation from flow allocation. The authors point out that standard multi-head attention uses a single query-key inner product to represent relationships between tokens and determine information flow. To address this limitation, they propose a new attention mechanism that distinguishes between self relation (a token's relationship with itself) and exchange relation (relationships with other tokens). Experimental results show improved performance over conventional attention in language modeling tasks.

Critical strengths
  • This study goes beyond simply comparing language modeling performance by analyzing how the proposed method differs from conventional attention in terms of learned relations and information flow. The observation that self-flow dominates in the initial layers, while the focus shifts toward exchange in subsequent layers, provides useful insights into how the proposed method operates.

  • The paper also proposes FlashRelation, Linear Relation, and Hybrid Relation to address computational efficiency.

Critical weaknesses
  • The motivation regarding the limitations of standard attention is not sufficiently convincing. The authors argue that existing attention mechanisms use the same scores for relation formation and flow allocation. However, this concern could also be interpreted as a limitation of the softmax function, which has already been discussed in previous work.

  • The technical novelty is unclear. In [1], the attention score is computed as Uij=qi⊤(kj+rij)dU_{ij} = \frac{q_i^\top(k_j+r_{ij})}{\sqrt d} rather than using only the query-key inner product. By adjusting rijr_{ij}, it may be possible to achieve effects similar to those described in this paper.

  • The rationale behind the design choices is insufficient. For instance, it is unclear why sigmoid and SiLU were chosen for self relation and exchange relation, respectively, or why a bounded activation function was chosen for self relation.

  • The tuning procedure for τS\tau_S is not described.

  • The presentation requires significant improvement. The notation does not appear to follow ICLR format. Furthermore, Tables 3 and 4 are difficult to understand and would benefit from clearer formatting.

[1] Shaw et al., Self-Attention with Relative Position Representations, NAACL 2018

Questions for authors

Please see the weaknesses listed above.

2 comments
Reviewer_zo2NOpenReview ↗

Official Comment by Reviewer zo2N

Thank you for the detailed response and additional experiments. However, I remain concerned about the technical novelty of the proposed Relation mechanism, particularly given that a simpler attention modification achieves better performance than Full SER. In addition, the justification for the activation choices and the hyperparameter tuning procedure remains insufficient. Therefore, I am inclined to maintain my original Weak Reject rating for the following reasons.

1. Distinction from conventional attention and technical novelty

It remains unclear how the proposed method fundamentally differs from conventional Softmax-based Attention. The authors argue that conventional Attention couples relation formation and flow allocation, and that separating these processes is a key contribution. However, SER still uses Softmax to determine the final information flow, and the response does not sufficiently clarify which specific limitation of conventional Attention this design addresses.

In particular, in the response to Reviewer bEJr (Part 1), the authors explicitly acknowledge that A01 (Identity/Identity + Count Calibration) is mathematically equivalent to conventional causal Attention with a position-dependent diagonal logit bias. Nevertheless, the additional experiments show that A01 (NLL 1.626762) achieves a lower NLL than the proposed Full SER (NLL 1.644117).

These results directly relate to my original concern about technical novelty. While the additional experiments demonstrate the effectiveness of Count Calibration, they also show that a relatively simple modification to conventional Attention can achieve better performance than the proposed Full SER. Therefore, if a method mathematically equivalent to conventional Attention with a diagonal logit bias outperforms Full SER, it remains unclear what the distinct technical contribution of the proposed Relation mechanism is.

2. Justification for the Activation Function Choices

The additional ablation results show that A01 (Identity/Identity + Count Calibration, NLL 1.626762) outperforms the proposed Full SER using Sigmoid/SiLU (NLL 1.644117).

I acknowledge that the authors state in their response to Reviewer zo2N (Part 1) that the Sigmoid/SiLU activations are not essential for the observed performance improvements. However, these findings raise further questions about whether the proposed role specific nonlinear mappings provide any additional technical benefit over conventional Attention with Count Calibration. In particular, it remains unclear why Sigmoid/SiLU was selected as the default design when the simpler Identity/Identity configuration achieves better performance.

3. Hyperparameter Tuning Procedure

My original concern regarding the hyperparameter tuning procedure for τS\tau_S also remains unresolved. The authors provide a sensitivity analysis over several temperature values using seed 42. However, this analysis only shows how performance varies across different settings and does not explain how the submitted value of τS=2\tau_S=2 was originally selected.

I would appreciate clarification on which candidate values and evaluation criteria were used to select this hyperparameter, and why the same value was adopted across different model scales.

AuthorsOpenReview ↗

Clarification on the Technical Novelty of Relation

The reviewer's argument against technical novelty rests on a fundamental reversal of the proposed construction.

The reviewer takes the final Identity formulation, rewrites it as Attention with a diagonal bias, and concludes that Relation contributes little beyond that bias. But this algebraic rewriting does not explain where the bias comes from.

Without first distinguishing Self from Exchange, what principled reasoning leads to this particular count-dependent correction?

In causal token mixing, there is one Self candidate but an increasing number of historical Others. This asymmetry is the starting point of Relation. We explicitly construct Self and Exchange, identify their competition under normalization, and introduce Count Calibration to regulate the aggregate historical contribution.

This is why the correction depends on the number of historical candidates. This is why it acts on historical Exchange rather than indiscriminately on all scores. And this is why it changes the Self–Exchange balance without changing the relative distribution within history.

The derivation is:

Self–Exchange → Candidate-count asymmetry → Count Calibration → Flow.

The reviewer instead starts from the final Flow and works backward to an equivalent Attention expression. An algebraic rewriting is not a derivation of the design principle.

Consider ResNet. Its central operation is simply y=x+F(x)y=x+F(x). One could rewrite the entire expression as G(x)G(x). Would that make residual learning devoid of architectural novelty? The technical contribution lies in recognizing and explicitly organizing the residual path, not in inventing addition. Judging architecture solely by the simplicity of its final expression misses precisely what architectural research contributes.

The Identity experiment makes the same point from another direction. We introduced Identity to test the necessity of the original Sigmoid/SiLU mappings. It performs better, showing that those nonlinearities are not essential. It does not remove Self–Exchange or Count Calibration. Treating an ablation within Relation as evidence against the entire framework misinterprets what was ablated.

The evidence is concrete. Under path-matched Identity settings, introducing Count reduces validation NLL from 1.682799 to 1.626762. In the formal three-seed structural ablations, removing Self transport, independent Information projection, or multi-head organization worsens NLL by 0.0352, 0.0432, and 0.0530, respectively. All nine paired runs deteriorate. Across 10M, 30M, and 100M, both Original and Identity outperform their matched MHA baselines.

Regarding the original Self temperature, τS=2\tau_S=2 was selected through initial internal exploratory ablations and subsequently held fixed across scales. The later sensitivity experiments examine that established choice. More importantly, Identity removes this temperature altogether and still improves over MHA.

The novelty objection therefore needs to address the actual contribution: the explicit Self–Exchange formulation, the count asymmetry it exposes, the calibration derived from that asymmetry, and the experimental evidence supporting the resulting structure.

Reducing one instance to an equivalent expression does not answer any of those questions.

The reviewer has shown how Identity Relation can be rewritten. That is not the same as showing why Count Calibration should exist, or why the Self–Exchange construction lacks technical novelty.

This is the distinction we ask the Area Chair to evaluate.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.