- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work evaluates 7 different masking families with a common masked-softmax on qwen2.5-0.5b and 1.5b. Using teacher forced loss on wikitext-2 and c4 with context lengths of up to 8k. They find good loss budget tradeoffs when using relative to max prob thresholding. Consistent sparsity patterns per head. Failed attempts at distributing thresholds based on sparsity patterns and keeping the argmax of each query can be beneficial. This paper just provides comparisons between these methods.
Critical strengths
-
Evaluating all masks through a shared attention implementation helps isolate selection rules from kernel-specific effects. The paired document-level analysis also provides useful evidence for the reported loss differences, including the 8K threshold-versus-top-k comparison.
-
The head-profile measurements and four-way window ablation are potentially informative diagnostics. Reporting unsuccessful allocation attempts is valuable, provided the conclusions remain limited to the tested configurations.
Critical weaknesses
-
The central budget interpretation is inconsistent, and the evidence does not establish the claimed matched-budget frontier. Equation 2 averages retained fractions across query rows; it is not the fraction of all retained causal attention entries. At n=1,024, top-k-64 has an average row fraction of approximately 0.235 but a total entry fraction of approximately 0.121. More directly, Section 3.2 compares a bare window-128 budget of 0.234 with window+argmax at 0.386 and attributes the increase to argmax preservation. Under Equation 2, the bare window already has budget approximately 0.3845, and adding at most one entry per row cannot increase it by more than approximately 0.0073. These numbers mix two budget definitions. Moreover, Table 2 imports budgets from the frontier protocol rather than measuring them in the paired runs, and Table 3 omits allocation budgets entirely. The selected operating-point comparisons therefore do not establish the general frontier or uniform-threshold near-optimality claims.
-
The additional contribution over prior work is unclear, and the proposed mechanisms are overstated. Twilight [1] already explains the limitations of fixed-k budgeting and evaluates adaptive mass-based selection. Top-Theta [2] studies calibrated thresholds and head-dependent calibration, while BLASST [3] uses a closely related relative-to-maximum logit-gap criterion. The paper does not compare against oracle top-p or calibrated thresholds under its controlled protocol. Its probability-versus-logit interpretation is also misleading: p ≥ τp_max is exactly s ≥ s_max + ln τ, so the experiment compares parameterizations rather than inherently different spaces. Finally, the window ablation does not establish that sink benefits are content-adaptive argmax benefits: fixed sinks alone reduce loss from 4.82 to 2.499, recovering most of the improvement achieved by the combined variant at approximately 2.352. Likewise, poor results from two uncalibrated proportional allocation schemes do not demonstrate that uniform thresholds are near-optimal.
[1] Chaofan Lin et al. Twilight: Adaptive Attention Sparsity with Hierarchical Top-p Pruning. 2025
[2] Konstantin Berestizshevsky, Renzo Andri, and Lukas Cavigelli. Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding. ICANN 2026
[3] Jiayi Yuan et al. BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding. MLSys 2026
Questions for authors
-
What budget was calculated for each run? Can you double check your window example in section 3.2 and calculate if your order still holds when calculating by total fraction of causal entries. With budgets being equal to each other (paired).
-
If we threshold relative to max do we still out perform a tighter top k search. Top p using mass and calibrated thresholds. At similar achieved budgets this would be helpful to know.
-
Were the different allocations in table 3 calibrated to match the uniform budget? If so what were those budgets. How do you know that section 7.3 shows content. It could be due to the benefits of keeping sinks and high scoring keys. Looks like there is a mistake in appendix A.5 for the -0.147.