- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
he paper proposes screening, an attention substitute in which unit-normalized query–key similarities in are mapped to relevance in by a thresholded quadratic, multiplied by a learned-window distance mask (eq. 9), and used to aggregate unit-normalized values without softmax normalization; the output is bounded by TanhNorm (eq. 12). A two-dimensional window-dependent rotation (eq. 6) replaces RoPE and becomes the identity for windows above . The full architecture, Multiscreen, stacks layers of parallel gated screening tiles (§3.3).
Claims and validation: (i) in a 353M-backbone comparison of 12 attention variants trained on SlimPajama tokens at length , only screening keeps low PG-19 perplexity and high retrieval accuracy up to without inference-time scaling (fig. 6); (ii) retrieval is measured on ABCDigits, a new synthetic letter–integer lookup task introduced by the authors (§4.3, App. M), and on a modified passkey task (App. N); (iii) Multiscreen reaches similar validation loss with ~30% fewer parameters (fig. 8a), wins 8/9 zero-shot tasks at 1.3B with +1.26 points mean (Table 1), has lower GPU-hour cost at 1.3B (fig. 8b) and 2.5–2.9× lower model-side TTFT (fig. 8c–d); (iv) Multiscreen trains stably at learning rate 1, and QKNorm + pre-logit normalization transfers this to Transformers (fig. 7, App. I).
Critical strengths
-
The controlled 12-variant comparison and nested ablations are well designed in structure. Holding the 353M backbone, dimensions and initialization fixed and swapping only the attention rule (§4.4, App. O, Tables 7–8), then peeling Softmask and MiPE off separately, is the right way to attribute behavior to a mechanism. The property table (Table 8) also states precisely which formal properties distinguish screening from Softmax$_1$, Sigmoid, Entmax and ReLU$^2$. This is consequential because it is the part of the paper most likely to be reused by others, independently of whether Multiscreen is adopted.
-
The learning-rate-1 stability result and its transfer to Transformers (fig. 7b, Table 6) is a falsifiable finding. Showing that QKNorm alone, PLNorm alone and SIO alone each diverge while QKNorm + PLNorm is stable, with PreNorm playing opposite roles in the two architectures, is a specific observation that the normalization literature can test. It supports the paper's claim of a transferable design pattern.
Critical weaknesses
-
The retrieval claims rest on a benchmark the authors invented for this paper, while standard long-context suites are absent. All retrieval evidence (figs. 5, 6, 18, 20, 22, 25) comes from ABCDigits, introduced here, and from a passkey variant whose scoring protocol was changed from the original (App. N: fixed depths instead of random insertion, exact position match instead of first-integer-in-100-tokens). ABCDigits is a single-needle exact-match lookup over 26 keys; it measures one capability, in a format tailored to the proposed mechanism's thresholded exact-rejection behavior, with no validation against any externally established benchmark. The community already has RULER (multi-key / multi-value NIAH, variable tracking, aggregation, long QA) and LongBench for exactly this purpose. Introducing a new task instead of reporting on these fragments the evaluation literature and makes the claim, "only screening maintains robust retrieval beyond the training context", uncomparable to any published number. The passkey result does not fix this: a 286M model at 100% on every cell (fig. 18) says the task is saturated. Saturation on one task does not show that retrieval is general. Given that the title and abstract make the general claim, this is the weakness that most determines my recommendation. RULER at – on the 1.3B models, and LongBench where the base-model setting permits, are necessary to support the claim as stated.
-
The Transformer baselines are the weakest available, so neither the "only screening" claim nor the efficiency claims are established, and the measured gains do not make the adoption case.
- The strongest published train-short-test-long Transformers are hybrids that interleave global NoPE layers with local sliding-window RoPE layers (SWAN, Puvvada et al., EMNLP 2025; RNoPE-SWA, Yang et al., 2025; Llama 4's iRoPE). These report scaling-free length extrapolation plus lower training cost and higher throughput, the same claims made here. The 12-way comparison (fig. 6) tests ALiBi, sliding window and NoPE separately shows each fails on retrieval or perplexity, and does not test their interleaving. This omission matters because App. K shows Multiscreen converges to exactly this structure: many local windowed units (MiPE is the identity for ) plus ~6% global units with and no positional encoding. Without a hybrid baseline it is unclear whether screening contributes anything beyond a learned local/global split.
- The RoPE baseline uses (App. A). Large-base RoPE (; Xiong et al., 2023; Liu et al., ICLR 2024) is standard practice and costs nothing to run. Extrapolation-designed encodings trained from scratch (FIRE, Li et al., ICLR 2024; xPos, Sun et al., ACL 2023) and data-dependent encodings (CoPE, DAPE), the comparison class for MiPE's window-adaptive rotation, are absent.
- YaRN is applied zero-shot at inference. Peng et al. (2024) fine-tune at the extended length, and the practitioner baseline is long-context continued pretraining at roughly 1% of the token budget (Fu et al., ICML 2024). None of this is reported, so the comparison is against a Transformer no one would deploy for long context.
- Every variant in the 12-way comparison is a single seed (App. O), and document masking is disabled for all of them while the headline Multiscreen runs use it (App. A, App. O). The 1.3B downstream, perplexity and TTFT results are also single-run (fig. 5 caption, §4.6).
- The Transformer baseline uses untuned Pythia learning rates (App. A: at 1.3B) with a warmup-then-constant schedule and no decay, while Multiscreen uses , ~300× larger. The "~30% fewer parameters for the same loss" claim (fig. 8a) therefore conflates architecture with learning-rate tuning; a baseline LR sweep at one scale is needed.
- TTFT compares custom Triton kernels for Multiscreen against stock varlen_attn/FlashAttention-2 (App. A, App. S). The 2.5–2.9× figure mixes mechanism with engineering effort, and FLOP counts are not reported.
- The "lower training cost at larger scales" claim rests on one point (1.3B); there is no 4B Transformer.
- "No inference-time scaling" is itself qualified by the default inference rule (§3.2), which post hoc sets for windows exceeding ; App. P shows the as-learned windows degrade retrieval (fig. 20). This is an inference-time adjustment of a different kind from YaRN scaling.
- Net effect at 1.3B: +1.26 mean accuracy, several per-task gaps (ARC-C +0.34, WinoGrande +1.02, PIQA +1.08) inside typical seed variance, and a 4-point loss on SciQ (Table 1). Combined with the custom-kernel dependency and the absence of a feed-forward block (whose downstream consequences at scale are unexplored), the paper does not yet give a practitioner a reason to switch architectures. This matters because the contribution is framed as a replacement for softmax attention.
Questions for authors
- Can you report RULER (and LongBench, if applicable to base models at this scale) for the 1.3B Transformer+YaRN and 1.3B Multiscreen at –? Without an externally defined benchmark I cannot weigh the retrieval claim.
- Does the 353M Softmax control in fig. 6 (document masking off) match the behavior of the 353M Transformer in fig. 17 (masking on) quantitatively? Please report both perplexity curves on the same axes.
- Was any learning-rate sweep run for the Transformer baseline at 353M or 1.3B? Fig. 7a covers only 45M. What is the Transformer validation loss at its best LR, and does the 30% parameter-efficiency gap survive?
- For TTFT: what are the forward FLOPs of the two 1.3B models at , and what fraction of the gap remains if the Transformer uses a comparably tuned kernel (e.g., FlashAttention-3 or a Triton implementation)?
- How many seeds would be needed, and are available, for the fig. 6 comparison? Even two seeds for Softmax, Sigmoid and Screening would show whether the ranking is stable.
- On the 353M backbone of App. O, can you add (a) a hybrid NoPE/SWA-RoPE Transformer (SWAN or RNoPE-SWA layout), (b) RoPE with , and (c) the Softmax baseline after ~1% of the budget as long-context continued pretraining at –? These are single extra runs each and would establish whether screening beats the strongest existing Transformer. At 1.3B, the best of the three would suffice.
- Under the as-learned window condition (App. P), 6% of 4B units exceed (App. K). Is the default rule's conversion the only reason global retrieval is preserved, and how does one choose the threshold without seeing the evaluation length?
References
- Bai, Y. et al. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024.
- Fu, Y. et al. Data engineering for scaling language models to 128K context. In International Conference on Machine Learning (ICML), 2024.
- Golovneva, O. et al. Contextual position encoding: Learning to count what's important. arXiv preprint arXiv:2405.18719, 2024.
- Hsieh, C.-P. et al. RULER: What's the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024.
- Li, S. et al. Functional interpolation for relative positions improves long context transformers. In International Conference on Learning Representations (ICLR), 2024.
- Liu, X. et al. Scaling laws of RoPE-based extrapolation. In International Conference on Learning Representations (ICLR), 2024.
- Meta AI. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. Meta AI Blog, April 2025. (iRoPE: interleaved attention layers without positional embeddings.)
- Puvvada, K. C. et al. SWAN: An efficient and scalable approach for long-context language modeling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.
- Shah, J. et al. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
- Sun, Y. et al. A length-extrapolatable transformer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
- Xiong, W. et al. Effective long-context scaling of foundation models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024.
- Yang, B. et al. Rope to Nope and back again: A new hybrid attention strategy. arXiv preprint arXiv:2501.18795, 2025.
- Zheng, C. et al. DAPE: Data-adaptive positional encoding for length extrapolation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
2 comments
W2 / Q5. Seeds and downstream performance.
Figure 6 now averages two seeds for Softmax and Screening; Sigmoid and other variants use one. We completed an additional independent training run for each 1.3B architecture, bringing both to two training seeds; our main models below 1.3B use three.
At 1.3B, the mean accuracy across the nine zero-shot downstream tasks increases from 52.32 ± 0.09% to 53.80 ± 0.13%, with eight of nine task means improved; SciQ is the only exception (Figure 7; Appendix O). Figure 7a now also includes results for smaller models averaged over three training seeds; the plotted curves favor Multiscreen at comparable parameter counts.
W2 / Q6. Stronger controls.
Figure 6 now includes all three requested controls on the 353M backbone: a SWAN-style global NoPE/local RoPE hybrid, RoPE trained from scratch with (with YaRN for ABCDigits), and Softmax continued at 32k for an additional of the pretraining tokens (Appendix N). We limited these additional controls to the matched 353M backbone, prioritizing controlled comparisons and independent replication of both 1.3B main models within the discussion period. Large-base RoPE and CPT do not match Screening's retrieval robustness; calibrated SWAN combines low perplexity with strong retrieval. We have accordingly revised the Abstract and Introduction to remove the claim that only Screening achieves both low long-context perplexity and robust retrieval beyond the training context.
Screening achieves this without SWAN's post-training calibration of score scaling on long-context data. It also provides bounded, pairwise-independent relevance, while SWAN retains normalization across competing keys (Table 7).
W2 / Q7. Window conversion.
We agree that window conversion is a post-training adjustment, and apologize for the ambiguous phrase "no inference-time scaling." We now use "No length-specific settings" to mean that the same attention and positional settings are used across input lengths (Table 7).
Only learned windows exceeding the 4,096-token training context are converted to infinity once after training and then held fixed for every input length (Section 3.2). The threshold uses the training context, not the evaluation input length or retrieval-score tuning. This rule requires no calibration on long-context data.
Appendix P shows that conversion preserves near-perfect retrieval with TanhNorm while barely changing perplexity; conversion without TanhNorm still leads to long-context collapse. Following your feedback, we have made the contribution of window conversion explicit in Section 4.3.
W2. Positional representations and scaling.
The YaRN paper also evaluates fine-tuning-free Dynamic-YaRN (Appendix B.7). At the 4k training context, the Transformer's YaRN factor is 1, yet Multiscreen improves the RULER mean from 43.09 to 64.73 (W1 / Q1). The downstream gains likewise occur with all inputs within 4,096 tokens (W2 / Q5). These improvements therefore cannot be explained solely by the use of zero-shot YaRN beyond the training context.
Section 2 now discusses FIRE, xPos, CoPE, and DAPE. MiPE reuses Softmask's learned window without additional trainable parameters and rotates only two coordinates of each query and key instead of all RoPE dimensions; even this rotation is omitted above a preset window threshold. Appendix Q isolates its contribution through RoPE/NoPE replacements.
W2. Screening as an attention replacement.
The Screening control in Figure 6 retains the Transformer FFN and changes only the attention mechanism and its positional representation. This demonstrates that screening can be used as an attention replacement without adopting the full Multiscreen architecture.
Thank you for your detailed feedback. We recognize that our original presentation could have made the central contribution clearer. We have clarified this contribution, replaced the ambiguous terminology, and added external benchmarks and stronger controls in the revised manuscript.
Central contribution: absolute query–key relevance.
Screening is designed for absolute query–key relevance: values lie on a fixed bounded scale, are independent of competing keys and context length, require no length-specific settings, and permit exact rejection and empty selection. Our experiments show that these properties coexist with low long-context perplexity and robust retrieval beyond the training context.
W1 / Q1. External evaluation.
ABCDigits deliberately isolates retrieval in base models while reducing instruction-following confounds; the original submission also evaluates passkey retrieval and nine standard zero-shot tasks. We evaluate RULER, LongBench, and HELMET-RAG using the same pretrained models without further training (Appendix O).
We added all 13 RULER tasks for the 1.3B models at all six requested context lengths, from 4k to 128k (Table 12; Appendix O). At 1.3B, Multiscreen improves the 13-task mean at every length, from 43.09 to 64.73 at 4k and from 4.05 to 13.96 at 128k. We also report length-wise means across model sizes, with three training seeds below 1.3B (Table 11). RULER evaluation of the second training seed for each 1.3B architecture will not finish before the discussion deadline, so we report one seed per architecture at this scale.
We added LongBench's 14 English and 2 code tasks as you suggested. At 1.3B, Multiscreen improves the six-category mean from 17.14 to 20.79, with gains in all six categories and 15 of 16 tasks (Figure 7).
We additionally report HELMET-RAG, in line with Reviewer 72z7's emphasis on contextual and semantic processing. HELMET-RAG tests question answering over natural-language evidence and retrieved distractors across four datasets (2-shot, 8k–128k). At 1.3B and 128k, Multiscreen improves the four-dataset mean from 2.75 to 14.08.
W2 / Q2. Document masking.
Figure 21 overlays the 353M Transformer curves from Figure 19 (with document masking) and the Softmax curves from Figure 6 (without), separately for each YaRN scaling factor. The perplexity curves and extrapolation breakdown points closely align across all seven factors.
W2 / Q3. Learning rates and parameter efficiency.
We did not run separate LR sweeps for the 353M or 1.3B Transformer, or their Multiscreen counterparts. For Transformer, we adopted Pythia's model dimensions and size-dependent learning rates. In the 45M sweep, the adopted learning rate gives validation loss close to the lowest observed value for this baseline (Figure 8a). Full-training sweeps at the larger scales would require multiple additional pretraining runs.
To reduce the confounding effect of different learning rates, we applied our normalization designs (QKNorm + PLNorm and QKNorm + SIO) to Transformer, enabling comparisons with Multiscreen at the same large LR. In the sweep, 2−4 gives the lowest validation loss for both 45M Transformer controls and the 28M Multiscreen (Figure 8a). Using this LR and a common 238-token budget, we compared 8M, 45M, and 353M normalized Transformer controls with their Multiscreen counterparts (one training seed per new control; Figure 12; Appendix E). The scaling trend still favors Multiscreen, although the gap narrows with these normalized Transformer controls. The original 30% estimate applies to the original Transformer baseline.
W2 / Q4. Efficiency measurements.
We added a FlashAttention-4 TTFT comparison (Figure 30; Appendix S), changing only the Transformer attention backend while keeping the trained models and measurement protocol unchanged. FA4 reduces Transformer TTFT, but Multiscreen remains faster across the paired model scales. At 1.3B, Transformer with FA4 has 2.03–2.53× the TTFT of Multiscreen across 2k–128k, including 2.03× at 128k. The 2.5–2.9× ratio refers specifically to the original FlashAttention-2 comparison (Figure 9d).
These TTFT results compare complete implementations, including both attention support patterns and kernel costs. Forward FLOPs remain unreported. Training-cost comparisons cover matched model scales through 1.3B (Figure 9b).