- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MoSAR, an attention mechanism that uses separate query and key routers to select mixtures of predefined short-, medium-, and global-range decay regimes. The resulting content- and distance-dependent field is incorporated as an additive attention bias. Controlled pretraining experiments with approximately 500M-parameter models show competitive perplexity at the training context length and improved length extrapolation.
Critical strengths
- The method is clearly explained, and routing on both the query and key sides is an intuitive design.
- The experiments are well controlled and include useful fixed-decay and hard-window baselines.
Critical weaknesses
- Downstream evaluation is missing. Perplexity alone does not show whether the model preserves useful long-range dependencies. Benchmarks such as RULER would make the long-context claims more convincing.
- The comparison is too narrow. Comparisons with more related methods, such as Inkling, would help establish where MoSAR stands beyond positional-encoding and fixed-locality baselines.
- The benefit of adaptive routing needs stronger evidence. Fixed-M essentially matches MoSAR at 2k, and the advantage over ALiBi at 8k is small. Single-sided routing ablations and repeated runs would help.
Questions for authors
See weaknesses.
Paper update
Dear reviewer,
thank you for appreciating the intuitive idea of MoSAR and finding the experimental section useful.
We have rewritten the paper and reported new inference results, which you might find interesting.
Even though you didn't ask questions, we saw the weaknesses, and we will certainly take them into account for future work.
In particular, we believe that addressing these weaknesses requires resources we don't currently have.
But if the paper were accepted, it would certainly be easier for us to apply for hardware resources, and obtain them.
Best regards