acceptodds
Under review as a conference paper at ICLR 2027

Screening Is Enough

Abstract

We call query–key relevance absolute when its values lie on a fixed bounded scale, depend on neither competing keys nor sequence length, require no sequence-length-dependent calibration, and can all be zero. To realize this notion, we introduce screening, whose explicit threshold transforms bounded query–key similarities into relevance values, enabling exact rejection, empty selection, and direct inspection on a common scale. In a controlled comparison of 12 attention mechanisms on a matched Transformer backbone, only screening maintains both low long-context perplexity and robust retrieval beyond the training context; notably, it does so without inference-time scaling. Building on screening, we introduce Multiscreen, a language-model architecture composed of parallel gated screening tiles. Multiscreen retains these long-context gains while achieving greater parameter efficiency, stronger general zero-shot downstream performance, lower training cost at larger scales, and lower model-side time to first token than Transformer baselines. We further develop a normalization design that keeps Multiscreen training stable even at a learning rate of 1 and show that an adapted version likewise stabilizes Transformer at the same learning rate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
language modeling, attention mechanisms, attention alternatives, long-context modeling, efficient sequence modeling, length generalization, token mixing, Multiscreen

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bhycOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

he paper proposes screening, an attention substitute in which unit-normalized query–key similarities in [−1,1][-1,1] are mapped to relevance in [0,1][0,1] by a thresholded quadratic, multiplied by a learned-window distance mask (eq. 9), and used to aggregate unit-normalized values without softmax normalization; the output is bounded by TanhNorm (eq. 12). A two-dimensional window-dependent rotation (eq. 6) replaces RoPE and becomes the identity for windows above wth=256w_{\text{th}}=256. The full architecture, Multiscreen, stacks layers of parallel gated screening tiles (§3.3).

Claims and validation: (i) in a 353M-backbone comparison of 12 attention variants trained on 2382^{38} SlimPajama tokens at length 2122^{12}, only screening keeps low PG-19 perplexity and high retrieval accuracy up to 2172^{17} without inference-time scaling (fig. 6); (ii) retrieval is measured on ABCDigits, a new synthetic letter–integer lookup task introduced by the authors (§4.3, App. M), and on a modified passkey task (App. N); (iii) Multiscreen reaches similar validation loss with ~30% fewer parameters (fig. 8a), wins 8/9 zero-shot tasks at 1.3B with +1.26 points mean (Table 1), has lower GPU-hour cost at 1.3B (fig. 8b) and 2.5–2.9× lower model-side TTFT (fig. 8c–d); (iv) Multiscreen trains stably at learning rate 1, and QKNorm + pre-logit normalization transfers this to Transformers (fig. 7, App. I).

Critical strengths
  1. The controlled 12-variant comparison and nested ablations are well designed in structure. Holding the 353M backbone, dimensions and initialization fixed and swapping only the attention rule (§4.4, App. O, Tables 7–8), then peeling Softmask and MiPE off separately, is the right way to attribute behavior to a mechanism. The property table (Table 8) also states precisely which formal properties distinguish screening from Softmax$_1$, Sigmoid, Entmax and ReLU$^2$. This is consequential because it is the part of the paper most likely to be reused by others, independently of whether Multiscreen is adopted.

  2. The learning-rate-1 stability result and its transfer to Transformers (fig. 7b, Table 6) is a falsifiable finding. Showing that QKNorm alone, PLNorm alone and SIO alone each diverge while QKNorm + PLNorm is stable, with PreNorm playing opposite roles in the two architectures, is a specific observation that the normalization literature can test. It supports the paper's claim of a transferable design pattern.

Critical weaknesses
  1. The retrieval claims rest on a benchmark the authors invented for this paper, while standard long-context suites are absent. All retrieval evidence (figs. 5, 6, 18, 20, 22, 25) comes from ABCDigits, introduced here, and from a passkey variant whose scoring protocol was changed from the original (App. N: fixed depths instead of random insertion, exact position match instead of first-integer-in-100-tokens). ABCDigits is a single-needle exact-match lookup over 26 keys; it measures one capability, in a format tailored to the proposed mechanism's thresholded exact-rejection behavior, with no validation against any externally established benchmark. The community already has RULER (multi-key / multi-value NIAH, variable tracking, aggregation, long QA) and LongBench for exactly this purpose. Introducing a new task instead of reporting on these fragments the evaluation literature and makes the claim, "only screening maintains robust retrieval beyond the training context", uncomparable to any published number. The passkey result does not fix this: a 286M model at 100% on every cell (fig. 18) says the task is saturated. Saturation on one task does not show that retrieval is general. Given that the title and abstract make the general claim, this is the weakness that most determines my recommendation. RULER at 2122^{12}–2172^{17} on the 1.3B models, and LongBench where the base-model setting permits, are necessary to support the claim as stated.

  2. The Transformer baselines are the weakest available, so neither the "only screening" claim nor the efficiency claims are established, and the measured gains do not make the adoption case.

    • The strongest published train-short-test-long Transformers are hybrids that interleave global NoPE layers with local sliding-window RoPE layers (SWAN, Puvvada et al., EMNLP 2025; RNoPE-SWA, Yang et al., 2025; Llama 4's iRoPE). These report scaling-free length extrapolation plus lower training cost and higher throughput, the same claims made here. The 12-way comparison (fig. 6) tests ALiBi, sliding window and NoPE separately shows each fails on retrieval or perplexity, and does not test their interleaving. This omission matters because App. K shows Multiscreen converges to exactly this structure: many local windowed units (MiPE is the identity for w≥256w \ge 256) plus ~6% global units with w=∞w=\infty and no positional encoding. Without a hybrid baseline it is unclear whether screening contributes anything beyond a learned local/global split.
    • The RoPE baseline uses θ=10,000\theta = 10{,}000 (App. A). Large-base RoPE (θ≈5×105\theta \approx 5\times10^5; Xiong et al., 2023; Liu et al., ICLR 2024) is standard practice and costs nothing to run. Extrapolation-designed encodings trained from scratch (FIRE, Li et al., ICLR 2024; xPos, Sun et al., ACL 2023) and data-dependent encodings (CoPE, DAPE), the comparison class for MiPE's window-adaptive rotation, are absent.
    • YaRN is applied zero-shot at inference. Peng et al. (2024) fine-tune at the extended length, and the practitioner baseline is long-context continued pretraining at roughly 1% of the token budget (Fu et al., ICML 2024). None of this is reported, so the comparison is against a Transformer no one would deploy for long context.
    • Every variant in the 12-way comparison is a single seed (App. O), and document masking is disabled for all of them while the headline Multiscreen runs use it (App. A, App. O). The 1.3B downstream, perplexity and TTFT results are also single-run (fig. 5 caption, §4.6).
    • The Transformer baseline uses untuned Pythia learning rates (App. A: 2×10−42\times10^{-4} at 1.3B) with a warmup-then-constant schedule and no decay, while Multiscreen uses 2−42^{-4}, ~300× larger. The "~30% fewer parameters for the same loss" claim (fig. 8a) therefore conflates architecture with learning-rate tuning; a baseline LR sweep at one scale is needed.
    • TTFT compares custom Triton kernels for Multiscreen against stock varlen_attn/FlashAttention-2 (App. A, App. S). The 2.5–2.9× figure mixes mechanism with engineering effort, and FLOP counts are not reported.
    • The "lower training cost at larger scales" claim rests on one point (1.3B); there is no 4B Transformer.
    • "No inference-time scaling" is itself qualified by the default inference rule (§3.2), which post hoc sets w=∞w=\infty for windows exceeding 2122^{12}; App. P shows the as-learned windows degrade retrieval (fig. 20). This is an inference-time adjustment of a different kind from YaRN scaling.
    • Net effect at 1.3B: +1.26 mean accuracy, several per-task gaps (ARC-C +0.34, WinoGrande +1.02, PIQA +1.08) inside typical seed variance, and a 4-point loss on SciQ (Table 1). Combined with the custom-kernel dependency and the absence of a feed-forward block (whose downstream consequences at scale are unexplored), the paper does not yet give a practitioner a reason to switch architectures. This matters because the contribution is framed as a replacement for softmax attention.
Questions for authors
  1. Can you report RULER (and LongBench, if applicable to base models at this scale) for the 1.3B Transformer+YaRN and 1.3B Multiscreen at 2122^{12}–2172^{17}? Without an externally defined benchmark I cannot weigh the retrieval claim.
  2. Does the 353M Softmax control in fig. 6 (document masking off) match the behavior of the 353M Transformer in fig. 17 (masking on) quantitatively? Please report both perplexity curves on the same axes.
  3. Was any learning-rate sweep run for the Transformer baseline at 353M or 1.3B? Fig. 7a covers only 45M. What is the Transformer validation loss at its best LR, and does the 30% parameter-efficiency gap survive?
  4. For TTFT: what are the forward FLOPs of the two 1.3B models at 2172^{17}, and what fraction of the gap remains if the Transformer uses a comparably tuned kernel (e.g., FlashAttention-3 or a Triton implementation)?
  5. How many seeds would be needed, and are available, for the fig. 6 comparison? Even two seeds for Softmax, Sigmoid and Screening would show whether the ranking is stable.
  6. On the 353M backbone of App. O, can you add (a) a hybrid NoPE/SWA-RoPE Transformer (SWAN or RNoPE-SWA layout), (b) RoPE with θ=500,000\theta = 500{,}000, and (c) the Softmax baseline after ~1% of the budget as long-context continued pretraining at 2152^{15}–2162^{16}? These are single extra runs each and would establish whether screening beats the strongest existing Transformer. At 1.3B, the best of the three would suffice.
  7. Under the as-learned window condition (App. P), 6% of 4B units exceed 2122^{12} (App. K). Is the default rule's w=∞w=\infty conversion the only reason global retrieval is preserved, and how does one choose the threshold without seeing the evaluation length?

References

  • Bai, Y. et al. LongBench: A bilingual, multitask benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), 2024.
  • Fu, Y. et al. Data engineering for scaling language models to 128K context. In International Conference on Machine Learning (ICML), 2024.
  • Golovneva, O. et al. Contextual position encoding: Learning to count what's important. arXiv preprint arXiv:2405.18719, 2024.
  • Hsieh, C.-P. et al. RULER: What's the real context size of your long-context language models? In Conference on Language Modeling (COLM), 2024.
  • Li, S. et al. Functional interpolation for relative positions improves long context transformers. In International Conference on Learning Representations (ICLR), 2024.
  • Liu, X. et al. Scaling laws of RoPE-based extrapolation. In International Conference on Learning Representations (ICLR), 2024.
  • Meta AI. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. Meta AI Blog, April 2025. (iRoPE: interleaved attention layers without positional embeddings.)
  • Puvvada, K. C. et al. SWAN: An efficient and scalable approach for long-context language modeling. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2025.
  • Shah, J. et al. FlashAttention-3: Fast and accurate attention with asynchrony and low-precision. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
  • Sun, Y. et al. A length-extrapolatable transformer. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), 2023.
  • Xiong, W. et al. Effective long-context scaling of foundation models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2024.
  • Yang, B. et al. Rope to Nope and back again: A new hybrid attention strategy. arXiv preprint arXiv:2501.18795, 2025.
  • Zheng, C. et al. DAPE: Data-adaptive positional encoding for length extrapolation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
2 comments

Thank you for your detailed feedback. We recognize that our original presentation could have made the central contribution clearer. We have clarified this contribution, replaced the ambiguous terminology, and added external benchmarks and stronger controls in the revised manuscript.

Central contribution: absolute query–key relevance.

Screening is designed for absolute query–key relevance: values lie on a fixed bounded scale, are independent of competing keys and context length, require no length-specific settings, and permit exact rejection and empty selection. Our experiments show that these properties coexist with low long-context perplexity and robust retrieval beyond the training context.

W1 / Q1. External evaluation.

ABCDigits deliberately isolates retrieval in base models while reducing instruction-following confounds; the original submission also evaluates passkey retrieval and nine standard zero-shot tasks. We evaluate RULER, LongBench, and HELMET-RAG using the same pretrained models without further training (Appendix O).

We added all 13 RULER tasks for the 1.3B models at all six requested context lengths, from 4k to 128k (Table 12; Appendix O). At 1.3B, Multiscreen improves the 13-task mean at every length, from 43.09 to 64.73 at 4k and from 4.05 to 13.96 at 128k. We also report length-wise means across model sizes, with three training seeds below 1.3B (Table 11). RULER evaluation of the second training seed for each 1.3B architecture will not finish before the discussion deadline, so we report one seed per architecture at this scale.

We added LongBench's 14 English and 2 code tasks as you suggested. At 1.3B, Multiscreen improves the six-category mean from 17.14 to 20.79, with gains in all six categories and 15 of 16 tasks (Figure 7).

We additionally report HELMET-RAG, in line with Reviewer 72z7's emphasis on contextual and semantic processing. HELMET-RAG tests question answering over natural-language evidence and retrieved distractors across four datasets (2-shot, 8k–128k). At 1.3B and 128k, Multiscreen improves the four-dataset mean from 2.75 to 14.08.

W2 / Q2. Document masking.

Figure 21 overlays the 353M Transformer curves from Figure 19 (with document masking) and the Softmax curves from Figure 6 (without), separately for each YaRN scaling factor. The perplexity curves and extrapolation breakdown points closely align across all seven factors.

W2 / Q3. Learning rates and parameter efficiency.

We did not run separate LR sweeps for the 353M or 1.3B Transformer, or their Multiscreen counterparts. For Transformer, we adopted Pythia's model dimensions and size-dependent learning rates. In the 45M sweep, the adopted learning rate gives validation loss close to the lowest observed value for this baseline (Figure 8a). Full-training sweeps at the larger scales would require multiple additional pretraining runs.

To reduce the confounding effect of different learning rates, we applied our normalization designs (QKNorm + PLNorm and QKNorm + SIO) to Transformer, enabling comparisons with Multiscreen at the same large LR. In the sweep, 2−42^{-4} gives the lowest validation loss for both 45M Transformer controls and the 28M Multiscreen (Figure 8a). Using this LR and a common 2382^{38}-token budget, we compared 8M, 45M, and 353M normalized Transformer controls with their Multiscreen counterparts (one training seed per new control; Figure 12; Appendix E). The scaling trend still favors Multiscreen, although the gap narrows with these normalized Transformer controls. The original 30% estimate applies to the original Transformer baseline.

W2 / Q4. Efficiency measurements.

We added a FlashAttention-4 TTFT comparison (Figure 30; Appendix S), changing only the Transformer attention backend while keeping the trained models and measurement protocol unchanged. FA4 reduces Transformer TTFT, but Multiscreen remains faster across the paired model scales. At 1.3B, Transformer with FA4 has 2.03–2.53× the TTFT of Multiscreen across 2k–128k, including 2.03× at 128k. The 2.5–2.9× ratio refers specifically to the original FlashAttention-2 comparison (Figure 9d).

These TTFT results compare complete implementations, including both attention support patterns and kernel costs. Forward FLOPs remain unreported. Training-cost comparisons cover matched model scales through 1.3B (Figure 9b).

W2 / Q5. Seeds and downstream performance.

Figure 6 now averages two seeds for Softmax and Screening; Sigmoid and other variants use one. We completed an additional independent training run for each 1.3B architecture, bringing both to two training seeds; our main models below 1.3B use three.

At 1.3B, the mean accuracy across the nine zero-shot downstream tasks increases from 52.32 ± 0.09% to 53.80 ± 0.13%, with eight of nine task means improved; SciQ is the only exception (Figure 7; Appendix O). Figure 7a now also includes results for smaller models averaged over three training seeds; the plotted curves favor Multiscreen at comparable parameter counts.

W2 / Q6. Stronger controls.

Figure 6 now includes all three requested controls on the 353M backbone: a SWAN-style global NoPE/local RoPE hybrid, RoPE trained from scratch with θ=500,000\theta=500{,}000 (with YaRN for ABCDigits), and Softmax continued at 32k for an additional 1/641/64 of the pretraining tokens (Appendix N). We limited these additional controls to the matched 353M backbone, prioritizing controlled comparisons and independent replication of both 1.3B main models within the discussion period. Large-base RoPE and CPT do not match Screening's retrieval robustness; calibrated SWAN combines low perplexity with strong retrieval. We have accordingly revised the Abstract and Introduction to remove the claim that only Screening achieves both low long-context perplexity and robust retrieval beyond the training context.

Screening achieves this without SWAN's post-training calibration of score scaling on long-context data. It also provides bounded, pairwise-independent relevance, while SWAN retains normalization across competing keys (Table 7).

W2 / Q7. Window conversion.

We agree that window conversion is a post-training adjustment, and apologize for the ambiguous phrase "no inference-time scaling." We now use "No length-specific settings" to mean that the same attention and positional settings are used across input lengths (Table 7).

Only learned windows exceeding the 4,096-token training context are converted to infinity once after training and then held fixed for every input length (Section 3.2). The threshold uses the training context, not the evaluation input length or retrieval-score tuning. This rule requires no calibration on long-context data.

Appendix P shows that conversion preserves near-perfect retrieval with TanhNorm while barely changing perplexity; conversion without TanhNorm still leads to long-context collapse. Following your feedback, we have made the contribution of window conversion explicit in Section 4.3.

W2. Positional representations and scaling.

The YaRN paper also evaluates fine-tuning-free Dynamic-YaRN (Appendix B.7). At the 4k training context, the Transformer's YaRN factor is 1, yet Multiscreen improves the RULER mean from 43.09 to 64.73 (W1 / Q1). The downstream gains likewise occur with all inputs within 4,096 tokens (W2 / Q5). These improvements therefore cannot be explained solely by the use of zero-shot YaRN beyond the training context.

Section 2 now discusses FIRE, xPos, CoPE, and DAPE. MiPE reuses Softmask's learned window without additional trainable parameters and rotates only two coordinates of each query and key instead of all RoPE dimensions; even this rotation is omitted above a preset window threshold. Appendix Q isolates its contribution through RoPE/NoPE replacements.

W2. Screening as an attention replacement.

The Screening control in Figure 6 retains the Transformer FFN and changes only the attention mechanism and its positional representation. This demonstrates that screening can be used as an attention replacement without adopting the full Multiscreen architecture.

Reviewer_4NYGOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes screening, an alternative to softmax attention based on normalized query-key similarities, explicit thresholding, and learned attention windows. The authors build Multiscreen using gated screening units and show strong long-context perplexity and retrieval performance up to 32x the training context length. They also report improvements in parameter efficiency, inference latency, and training stability compared to Transformer baselines.

Critical strengths
  1. The controlled comparison of 12 attention mechanisms (Fig. 6) is a strong part of the paper. The ablations provide useful evidence that the long-context improvements come from screening rather than the surrounding architecture.

  2. Evaluating perplexity and retrieval separately is a good choice, since the two can behave quite differently. The ABCDigits benchmark is a useful diagnostic, and the normalization ablations in Table 6 are also interesting.

Critical weaknesses
  1. Related work and baselines: Several components overlap with existing work, including nGPT (Loshchilov et al., 2025), Adaptive Attention Span (Sukhbaatar et al., 2019), and ReLA (Zhang et al., 2021). The normalization results are also related to Wortsman et al. (2024). These papers deserve discussion. More importantly, the entmax baseline uses fixed alpha=1.5, while Vasylenko et al. (2026) show stronger length generalization with ASEntmax. A NoPE Transformer and a hybrid local/global attention baseline would also strengthen the comparison.

  2. Fairness and robustness of comparisons: Fig. 6 and the 1.3B results use single seeds. The downstream improvement is relatively small (1.26 points on average), and the main Transformer baseline uses a much smaller learning rate without the normalization improvements identified in Table 6. It is unclear whether the claimed parameter efficiency and general performance improvements would hold against a better tuned Transformer. The reported inference speedups are also difficult to separate from the use of sparse windows and different kernels.

Questions for authors
  1. Could the authors compare against ASEntmax and a Transformer with mixed local/global attention? These seem like important baselines for the main long-context claim.

  2. Would the parameter efficiency results hold against a Transformer using QKNorm + PLNorm with a tuned learning rate?

  3. How much does the inference-time window conversion (L197-200) contribute to long-context retrieval? Appendix P suggests that keeping the learned windows unchanged hurts retrieval. It would be useful to clarify the role of this rule in the main results.

1 comment

Thank you for your constructive feedback and for recognizing the value of our controlled comparisons. The revised manuscript includes additional attention baselines, normalized Transformer controls, multi-seed results, and broader benchmark evaluations.

W1 / Q1. Related work and additional baselines.

We expanded the discussion of nGPT, Adaptive Attention Span, ReLA, and Wortsman et al. (Section 2; Appendices H and N). Screening combines pairwise-independent relevance in [0,1][0,1] with exact rejection and empty selection, without an explicit length term or length-specific settings (Table 7).

We added ReLA-g with RoPE on the same 353M backbone and training recipe; both its perplexity and retrieval degrade beyond the training context (Figure 6; Appendix N).

We report ASEntmax + NAPE through 64k, adapting the authors' kernel and truncating the same LCP inputs at 64k. Perplexity rises during extrapolation and retrieval becomes length- and depth-dependent; 128k evaluation is ongoing but too slow to finish within the discussion period (Figure 6; Appendix N).

We also added SWAN's mixed global NoPE/local RoPE hybrid. Calibrated SWAN combines low long-context perplexity with strong retrieval; Screening does so without post-training calibration on long-context data (Figure 6; Appendix N). SWAN's fitted coefficient is fixed across evaluation lengths, although its multiplier depends on position.

Once a span exceeds all observed training distances, Adaptive Attention Span's mask is one for every observed query–key pair, so the language-model loss provides no gradient to that span; the regularizer still favors shrinking it. Screening's Softmask can retain window-dependent gradients beyond this range. We therefore included ALiBi and NoPE-based Forgetting Attention to cover fixed and learned, input-dependent distance attenuation without a hard span limit (Appendix N).

W2 / Q2. Normalization, learning rates, and parameter efficiency.

Figure 6's Screening control retains the Transformer's backbone, FFN, learning rate, and training settings, with screening's internal Q/K/V normalization and TanhNorm. Its long-context performance does not require Multiscreen's larger learning rate or external SIO parameterization.

The revised 45M LR sweep includes Transformer + QKNorm + PLNorm and Transformer + QKNorm + SIO controls (Figure 8a). Both controls attain their lowest observed loss at 2−42^{-4}; SIO, introduced here, yields the lower minimum. At 8M, 45M, and 353M, we trained both normalized Transformer controls at LR 2−42^{-4} and 2382^{38} tokens, matching Multiscreen (Figure 12; Appendix E). The scaling trend still favors Multiscreen against these normalized controls at the tested scales; each new control uses one training seed. The approximately 30% parameter reduction in Figure 9a refers to the original Transformer baseline; the normalized controls narrow the gap.

W2. Training seeds and benchmark performance.

Figure 7 summarizes benchmark results for the original Transformer baseline and Multiscreen, using three training seeds below 1.3B and two at 1.3B. The curves favor Multiscreen in performance relative to parameter count, particularly on LongBench; its HELMET-RAG mean scores are higher at every tested context length for every model-size pair.

At 1.3B, Multiscreen improves the nine-task downstream mean from 52.32 ± 0.09% to 53.80 ± 0.13%, with gains on eight of nine tasks. LongBench's six-category mean increases from 17.14 ± 0.42 to 20.79 ± 0.77, with gains in all six categories and 15 of 16 tasks. At 128k, HELMET-RAG improves from 2.75 ± 0.46 to 14.08 ± 1.33. Both 1.3B Multiscreen runs exceed both Transformer runs on these three aggregates.

Figure 6 now also averages two seeds for the Softmax and Screening controls; other variants use one. Full protocols and task-level results are in Section 4.5 and Appendix O (Tables 8–10).

W2. Inference efficiency.

We added a FlashAttention-4 TTFT comparison on the same GH200, changing only the Transformer backend (Figure 30; Appendix S). At 1.3B, Transformer with FA4 has 2.03–2.53× the TTFT of Multiscreen across 2k–128k, including 2.03× at 128k. These measurements compare complete implementations, including both window sparsity and kernel costs.

Q3. Contribution of the window conversion.

We now explicitly discuss the conversion's contribution in Section 4.3, referring to the existing ablation in Appendix P (Figures 23–24). In that ablation, with TanhNorm, conversion maintains near-perfect ABCDigits accuracy across the tested lengths and depths, whereas as-learned windows show a gradual distance-dependent decline; perplexity is nearly unchanged. Only learned windows exceeding the 4,096-token training context are converted to infinity once after training and held fixed across evaluation lengths; the rule uses neither the target evaluation length nor retrieval-score tuning.

Reviewer_72z7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes replacing the transformer layer with multiscreen. This is motivated by an inability of standard softmax attention to evaluate query/key relevance without influence from other keys in the context. The work first introduces screening, a threshold-based mechanism mapping query/key similarities to relevances which drops keys with low relevance. It also introduces a benchmark for evaluating retrieval without semantic or instruction-following bias. Finally it introduces an end-to-end language model that integrates screening, replacing the transformer layer with a gated screening unit (sort of like an attention head) that importantly integrates screening, sliding windows, and a positional encoding that vanishes as context increases (to avoid context-rot like issues for long-range query/key computations). Multiscreen achieves a lower perplexity than a transformer at long context and has very stable training dynamics

Critical strengths
  1. Large body of work, with an entirely new language modeling layer designed around screening and a new synthetic benchmark, complemented by promising initial results
  2. Tackles an important problem of long context extension
  3. Easy to read and well-presented
Critical weaknesses

This work overindexes on being able to select relevant tokens for a query well and with high accuracy. This is reflected in the benchmark and in the nature of screening, which is emphatically strong at selecting individual tokens. From a natural language perspective, I have concerns with this motivation. I think it's more important to think about retrieving relevant keys for a query considering context.

Screening also isn't that novel: for example, Rectified Linear Attention (https://arxiv.org/pdf/2104.07012) similarly thresholds query/key scores.

Questions for authors
  1. how does this approach do on more context/semantically driven long context tasks like HELMET or (to a lesser extent) RULER?
  2. regular transformers observably have a way to place unnecessary attention, in the form of attention sinks. How does the sink phenomenon compare to this more direct approach of thresholding irrelevant keys out? Do multiscreen models show no sink behavior?
  3. What about chinchilla-style scaling laws for multiscreen models, varying flops and training data amounts (as a number of tokens)? i.e. we want to know the ideal number of tokens needed per parameter for multiscreen.
2 comments

Thank you for your constructive feedback and for recognizing the breadth of our work. We have added LongBench and HELMET-RAG evaluations to the revised manuscript.

W1 / Q1. Contextual retrieval and evaluation.

We agree that long-context tasks combining retrieval with contextual understanding strengthen the empirical evidence. Our original evaluation covers three complementary aspects:

  • Long-context perplexity: next-token prediction across context lengths.
  • ABCDigits: retrieval in pretrained base models with minimal semantic and instruction-following confounds.
  • Nine zero-shot downstream tasks: general language understanding and knowledge.

These additional evaluations use the same pretrained base models, without additional SFT or long-context continued pretraining (revised Section 4.5; Figure 7).

  • LongBench: At 1.3B, Multiscreen improves the six-category mean from 16.72 to 21.56 (0–100), with gains in all six categories and 14 of 16 tasks. Even the 286M Multiscreen exceeds the 1.3B Transformer on this aggregate.
  • HELMET-RAG: Multiscreen outperforms Transformer on the four-dataset mean at every evaluated context budget (8k–128k) for all model-size pairs. At 128k, the 1.3B substring exact-match score increases from 3.21% to 12.75%.

Full protocols and task-level results are provided in Appendix O (Tables 9–10).

W2. Relation to ReLA.

Our contribution is screening as a mechanism for absolute query–key relevance on a fixed bounded scale across context lengths. ReLA, like the ReLU2\mathrm{ReLU}^2 attention already discussed and evaluated in our paper (Related Work; Figure 6), permits exact rejection and empty selection but produces unbounded relevance weights.

Screening assigns relevance values in [0,1][0,1], with zero denoting exact rejection and one providing a fixed upper reference. Each value can therefore be interpreted against the same endpoints across context lengths. These values are computed independently for each query–key pair, without an explicit sequence-length term or inference-time rescaling.

To our knowledge, screening is the first mechanism to combine all seven properties required for absolute query–key relevance across context lengths (Table 7, Appendix N).

We will add a matched comparison with ReLA and implementation details to the revised manuscript before the discussion period ends.

Q2. Attention sinks.

Softmax attention can offload unused mass to a sink, whereas screening permits all relevance values to be zero, removing the structural need for one. Unit-normalized values also prevent a large relevance weight from being neutralized simply by shrinking its value norm.

Across the layers and tiles visualized in Appendix I (4M model, with the paper's abstract as input), we do not observe a pronounced persistent concentration on the initial tokens.

Q3. Chinchilla-style scaling.

Following the compute approximation used in the Chinchilla paper, we estimate total training FLOPs as C≈6NDC\approx6ND (NN: total parameters; DD: training tokens). Our preliminary visual comparison tracks the model size achieving the lowest training loss at each compute budget. The resulting log–log trends have similar slopes for both architectures. At a fixed model size, guides fitted by eye suggest that Multiscreen's compute-efficient token budget per parameter is roughly half that of the Transformer. This is an exploratory, recipe-dependent estimate from a sparse size grid that includes very small models; a precise characterization of compute-optimal allocation remains future work.

Please let us know if any of the main concerns raised in your initial review remain unresolved.

AuthorsOpenReview ↗

Follow-up: ReLA comparison and updated evaluations

W2. ReLA-g comparison.

We have now added ReLA-g with RoPE to the controlled comparison on the same 353M Transformer backbone and pretraining recipe (Section 4.4; Figure 6), with implementation details in Appendix N.

ReLA-g achieves low perplexity within the 4k training context, but its perplexity rises sharply during extrapolation under the tested scaling treatments, and its ABCDigits retrieval accuracy drops sharply beyond 4k. Screening maintains low perplexity and robust retrieval through 128k with the same settings across context lengths.

W1 / Q1. Additional training seeds.

We also completed a second independent training run for each 1.3B architecture and updated the main results accordingly. The LongBench and HELMET-RAG scores in our initial response were from one training seed. The updated results below average two training seeds per architecture:

  • Nine zero-shot downstream tasks: Mean accuracy increases from 52.32 ± 0.09% for Transformer to 53.80 ± 0.13% for Multiscreen, with gains on eight of nine tasks.
  • LongBench: The six-category mean increases from 17.14 ± 0.42 for Transformer to 20.79 ± 0.77 for Multiscreen, with gains in all six categories and 15 of 16 tasks.
  • HELMET-RAG at 128k: The four-dataset mean increases from 2.75 ± 0.46 to 14.08 ± 1.33.

Both Multiscreen runs exceed both Transformer runs on each aggregate. Figure 7 and Appendix O (Tables 8–10) report the updated means and standard deviations.

W1 / Q1. RULER evaluation.

We also added all 13 RULER tasks at six context lengths from 4k to 128k (Appendix O; Tables 11–12). At 1.3B, Multiscreen improves the 13-task mean at every length. At the 4k training context, Multiscreen already improves the mean from 43.09 to 64.73. At 128k, the mean increases from 4.05 to 13.96. The RULER results at 1.3B use one training seed per architecture.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of manuscript revisions

We thank the reviewers for their careful feedback. The revised manuscript includes the following updates:

  • External evaluation: Added LongBench's 14 English and 2 code tasks, HELMET-RAG, and RULER, with detailed evaluation protocols and task-level results (Section 4.5; Appendix O).
  • Stronger attention controls: Added ReLA-g, ASEntmax (evaluated through 64k), a SWAN-style local/global hybrid, large-base RoPE, and long-context continued pretraining (Figure 6; Appendix N).
  • Replication and fairness: Added a second training seed for both 1.3B architectures and for the Softmax/Screening controls on the 353M backbone; compared 8M, 45M, and 353M normalized Transformer controls with Multiscreen at a common learning rate and token budget, and added a document-masking comparison (Figures 6–8, 12, and 21).
  • Inference efficiency: Added a FlashAttention-4 backend comparison under the same TTFT measurement protocol (Figure 30; Appendix S).
  • Clarified contribution and scope: Refined the definition of absolute query–key relevance and the meaning of length-specific settings, clarified the contribution of window conversion, and expanded the related-work discussion (Sections 1–3 and 4.3; Appendices H, N, and P).

Detailed responses are provided in the individual review threads.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.