- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work introduces Self‑Exchange Relation (SER) as an alternative token‑mixing primitive for decoder‑only language models. It separates relational construction and information flow, yielding four practical implementations. Experiments on 10M–100M models show better token efficiency than MHA. FlashRelation achieves high throughput close to FlashAttention, and Hybrid Relation validates layer‑wise composition.
Critical strengths
- Novel foundational paradigm for token‑mixing. The paper proposes the "Relation‑first, Flow‑follows‑Relation" design philosophy, cleanly separating relational construction and information flow allocation. It is not a minor tweak of multi‑head attention but delivers an alternative foundational viewpoint for sequence‑modeling research.
- Systematic practical operator family together with rigorous empirical analysis. Starting from the core SER primitive, the authors systematically derive Full, Flash, Linear and Hybrid Relation, covering exact computation, GPU‑fused execution, recurrent compression and layer‑hybrid composition. Beyond standard perplexity metrics, comprehensive structural diagnostics, multi‑seed repeated runs and hardware throughput benchmarks strengthen the empirical reliability.
Critical weaknesses
- The robustness of the proposed method under different hyperparameter choices has not been fully explored. Several key hyperparameters are fixed throughout the main experiments, including the self‑temperature , activation selections (sigmoid for Self branch, SiLU for Exchange branch), and initialization of the layer‑wise count‑calibration coefficient . Without systematic ablation studies over these choices, it remains difficult to tell whether the reported performance gains generalize broadly or are tied closely to these particular configurations.
- The empirical evaluation scope is relatively limited, and some relevant baseline comparisons are missing. All main experiments are conducted on decoder‑only models up to ~100M parameters, while evaluations on larger models, encoder‑centric tasks, multimodal and instruction‑tuning settings are not provided. Moreover, the Linear Relation variant has not been compared against representative recurrent and linear‑time token‑mixing methods such as Retentive Network, KDA, and Mamba‑family models under matched computational budgets. Only one fixed interleaving layout is tested for Hybrid Relation, leaving alternative layer composition strategies unexplored.
Questions for authors
- Could you please provide ablation results on the key fixed design choices, including the self‑temperature , alternative activation functions for the Self / Exchange branches, and the count‑calibration coefficient ? This would help clarify whether the observed improvements are sensitive to these hyperparameter selections.
- For the Linear and Hybrid Relation variants, can you add controlled comparisons against competitive recurrent and linear‑time token‑mixing baselines under matched experimental setups? For Hybrid Relation, it would also be helpful to see an analysis of modeling quality and system overhead trade‑offs for layer interleaving patterns beyond the layout adopted in the paper.
Official Comment by Reviewer zo2N
Thank you for the detailed response and additional experiments. However, I remain concerned about the technical novelty of the proposed Relation mechanism, particularly given that a simpler attention modification achieves better performance than Full SER. In addition, the justification for the activation choices and the hyperparameter tuning procedure remains insufficient. Therefore, I am inclined to maintain my original Weak Reject rating for the following reasons.
1. Distinction from conventional attention and technical novelty
It remains unclear how the proposed method fundamentally differs from conventional Softmax-based Attention. The authors argue that conventional Attention couples relation formation and flow allocation, and that separating these processes is a key contribution. However, SER still uses Softmax to determine the final information flow, and the response does not sufficiently clarify which specific limitation of conventional Attention this design addresses.
In particular, in the response to Reviewer bEJr (Part 1), the authors explicitly acknowledge that A01 (Identity/Identity + Count Calibration) is mathematically equivalent to conventional causal Attention with a position-dependent diagonal logit bias. Nevertheless, the additional experiments show that A01 (NLL 1.626762) achieves a lower NLL than the proposed Full SER (NLL 1.644117).
These results directly relate to my original concern about technical novelty. While the additional experiments demonstrate the effectiveness of Count Calibration, they also show that a relatively simple modification to conventional Attention can achieve better performance than the proposed Full SER. Therefore, if a method mathematically equivalent to conventional Attention with a diagonal logit bias outperforms Full SER, it remains unclear what the distinct technical contribution of the proposed Relation mechanism is.
2. Justification for the Activation Function Choices
The additional ablation results show that A01 (Identity/Identity + Count Calibration, NLL 1.626762) outperforms the proposed Full SER using Sigmoid/SiLU (NLL 1.644117).
I acknowledge that the authors state in their response to Reviewer zo2N (Part 1) that the Sigmoid/SiLU activations are not essential for the observed performance improvements. However, these findings raise further questions about whether the proposed role specific nonlinear mappings provide any additional technical benefit over conventional Attention with Count Calibration. In particular, it remains unclear why Sigmoid/SiLU was selected as the default design when the simpler Identity/Identity configuration achieves better performance.
3. Hyperparameter Tuning Procedure
My original concern regarding the hyperparameter tuning procedure for τS also remains unresolved. The authors provide a sensitivity analysis over several temperature values using seed 42. However, this analysis only shows how performance varies across different settings and does not explain how the submitted value of τS=2 was originally selected.
I would appreciate clarification on which candidate values and evaluation criteria were used to select this hyperparameter, and why the same value was adopted across different model scales.