- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes Matrix Approximation Sparse Attention (MASA), a scoring replacement for selecting structured sparse-attention units during prefill. Instead of ranking candidates by attention mass, MASA scores each candidate by its individual reduction in a squared matrix-product approximation error, using value-derived probes at the baseline's profiling resolution. It preserves the existing candidate families, sparse kernels, and budget rules. Experiments integrate MASA into MInference, SeerAttention, and FlexPrefill, reporting limited improvements across eight matched benchmark averages on Llama and Qwen family models, some additional latency, and lower executed attention-output error in four diagnostic comparisons.
Critical strengths
- MASA changes a selection score while reusing established sparse kernels and candidate structures. Integration into three different frameworks is meaningful evidence that the idea is implementable and not tied to one custom sparse pattern.
Critical weaknesses
-
The eight reported average improvements range from approximately 0.17 to 1.41 points, without corresponding downstream confidence intervals or multi-seed results. More importantly, Appendix F states that the main results use the best tested probe configuration on each benchmark. This introduces selection bias into the reported gains. Appendix E provides useful uncertainty estimates for output error, but explicitly describes selected-setting comparisons rather than an independent held-out evaluation. Standardized evaluation with frozen configurations and five fixed seeds for RULER data construction would help establish that the gains persist across synthetic inputs and are not dependent on evaluation implementation or benchmark-specific configuration selection.
-
Real attention can be highly concentrated, with many negligible entries; whether low-mass candidates have consequential value contributions under the actual execution precision is therefore an empirical question. The paper does not quantify this for the candidates whose selection changes. Moreover, the score models unnormalized retained contributions, whereas actual sparse kernels renormalize, and the implementation uses profiled and sometimes norm-controlled probes. Appendix E establishes an aggregate output-error improvement, but does not distinguish improved value-aware selection from increased retained attention mass, norm control, or numerical effects.
Questions for authors
-
Could you evaluate Original and MASA through lm-eval-harness using five predeclared seeds that control RULER synthetic-data construction, with the same generated instances for both methods at each seed? Please freeze probe parameters using separate calibration data, preserve the intended long-context task settings, and release the harness task/backend configuration, data-generation seeds, and per-example scores. Report each seed's paired improvement, the mean and standard deviation. Greedy decoding does not remove the variability in synthetic-data construction. This would address both reproducibility of the evaluation pipeline and stability of the small gains.
-
What are the dtypes and normalization/casting steps used for profiling, score computation, softmax intermediates, and the value multiplication? For blocks retained by MASA but omitted by the baseline, can you quantify their attention mass and effective value contribution under the actual kernel's numerical path, including how often relevant multiplication operands become zero? The important question is whether the proposed correction acts on meaningful contributions in trained models, rather than only on possibilities demonstrated by a constructed example.
-
Could you provide a controlled, block-by-block experiment tracing where contributions are lost along the actual FlashAttention online loop, comparing Original and MASA with identical Q, K, V, candidate units, actual retention budget, and kernel settings? For each block, record its selection status, QK-derived attention mass, running row maximum and row sum, P before and after low-precision conversion, PV contribution, and the resulting change in the output accumulator. Report both the fraction of nonzero P entries rounded to zero and the weight they carry. Track subsequent accumulator rescaling and final normalization so that block contributions are compared on the final-output scale, and relate these losses to the complete output error against dense attention.
Please use a common dense execution trace to characterize skipped blocks, together with each method's actual sparse execution trace to capture changes in online normalization. Focus on blocks retained only by Original or only by MASA: do MASA's replacement blocks preserve meaningful PV contributions after casting and accumulation, given the measured value-norm and variance distributions in the trained model? A small, budget-matched block-swap experiment on representative query tiles would help connect these selection differences to changes in final output error. This would distinguish the claimed value-aware preservation from alternatives such as retaining more effective attention mass or changing the normalization denominator, rather than relying solely on aggregate output-error improvements.