- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
MASaLA approximates attention with two branches computed from dense Q, K, and V. A block-sparse branch partitions the sequence into contiguous blocks, scores every key block for each query block, keeps the top-k blocks together with the query's own block, and computes exact softmax attention over that support. A low-rank branch uses RACE, a linear-attention form whose feature map is a vector of soft-LSH bucket probabilities. The two outputs are combined with sigmoid gates, and the sparse branch is additionally rescaled by its estimated share of the attention mass. The experiments train small models (1–8 layers, d = 128–512; Table 6) on classification tasks of 512 to 64K tokens. Appendix B describes a fused Triton kernel. On a single RTX PRO 6000 Blackwell (48 GB), the paper reports forward speedups over FlashAttention of about 3.5× at 32K and 13× at 128K and backward speedups of 3× and 11× for a single attention operator, and peak training memory 1.08× that of FlashAttention for a 4-layer model.
Critical strengths
S1. The fused-kernel design in Appendix B is careful and is the paper's main new technical content. Because RACE is associative, its key/value sums can be computed once and contracted per query inside the epilogue of the block-sparse kernel, so neither branch output is written to HBM. The backward pass recomputes both branches and keeps activation memory linear in N, with measured peak memory 1.08× that of FlashAttention, and the paper notes that its comparisons pad the head dimension of both kernels from 96 to 128. The explanation of why fp16 silently prevents the low-rank projections from training is useful to anyone fusing soft-hash features.
Critical weaknesses
W1. Results in three of the four main tables cannot be attributed to the described method, and the closest prior work is not cited. ELSAA (arXiv:2607.20214) proposes sparse-plus-low-rank attention with a RACE low-rank branch, the same mass-based scaling of the sparse branch, and the same gates, but uses a sortLSH sparse branch. More importantly, rows of Tables 1, 3, and 4 match that paper's reported results cell by cell under different method names (see Q1).
W2. Independently of W1, the evidence does not support the claims about long-context attention. There is no language-model evaluation: every task is classification or synthetic retrieval, and the reported models have 1–8 layers. On ArXiv (Table 2), all methods reach 99.9–100% training accuracy, MASaLA's test accuracy rises from 80.65% at 8K to 99.08% at 64K, and full attention (84.85% on average) trails RACE and every hybrid; the authors note that class-balanced packing may inflate long-length accuracy and that they did not run the per-document control. On vision, MASaLA's test accuracy is 22.51% on 37 classes and 28.33% on 101 classes, with train-test gaps of 69 and 48 points. The experiments therefore offer limited support for the paper's conclusions.
Questions for authors
Q1. Which sparse branch produced the rows labelled MASaLA, Block-Sparse, and BS+RACE? In Tables 3 and 4, these rows match, cell by cell, the rows labelled ELSAA, Sort LSH, and Sort Lsh RACE in the ELSAA paper (arXiv:2607.20214), whose sparse branch is sortLSH; in Table 1, the MASaLA and Block-Sparse rows match its ELSAA and Sparse LSH rows. Please also cite that paper and state the differences explicitly.
Q2. What top-k does the block-sparse branch use? It is not reported anywhere in the paper, including Table 7.
Q3. Do the reported speedups include block scoring, whose cost Appendix B.2 gives as quadratic in N? Please report them with and without it.
Q4. Could you add a language-modeling result, even a small one?
7 comments
Response to Reviewer P4wx (2/3)
Q2: What top-k does the block-sparse branch use?
Response to Q2 (revised Sec. 3.2, Table 7, App. F.5):
For the block-sparse vision experiments, Imagenette, and timing benchmarks, we use block size 64 and top- selected blocks, together with the query's own and first blocks, giving 384 keys per query.
For causal language modeling, we use top-8 past blocks plus the query's own block, without forcing the first block.
The sortLSH-based runs do not use block-sparse top- selection. The ArXiv configurations use block size 256 (512 at 64K), as specified in Table 7.
These details are now stated explicitly to distinguish the implementations.
Q3: Do the reported speedups include block scoring? Report results with and without scoring.
Response to Q3 (Fig. 2, revised Sec. 6, App. F.1):
Yes. The reported end-to-end speedups include block scoring and top- selection. We additionally measured a selection-free positional-block baseline to quantify this overhead.
| Sequence length | With scoring | Without scorer |
|---|---|---|
| 32K | 3.6× | 3.8× |
| 64K | 6.9× | 7.3× |
| 128K | 13.3× | 14.2× |
The selection-free configuration is a timing reference and was not trained.
At 128K, the forward takes 15.9 ms including selection versus 14.9 ms without it. Scoring and top- contribute approximately 7%, 6%, and 6% of forward time at 32K, 64K, and 128K, respectively. The backward pass has no selection overhead and achieves a 10.9× speedup at 128K.
We also corrected the scorer description. The original App. B.2 described a raw-key scorer that was never used in the experiments. Its forward speedups would be 2.8× / 4.7× / 7.2×, substantially below the implemented mean-pooled scorer.
The complete complexity is now stated as
which is quadratic in for fixed . We removed unsupported linear and subquadratic claims for the complete method.
Fig. 2 uses shared projections between the branches. With separate projections, the 128K speedups are 11.8× forward and 10.0× backward. The timing configuration (, , gate width 64) differs from that used for the trained vision models (, , gate width 128).
Thus, although block scoring introduces a quadratic term, the measured end-to-end efficiency advantage remains substantial with that cost included.
Response to Reviewer P4wx (3/3)
W2 and Q4: No language-model evaluation; small models; packed ArXiv inputs; low vision accuracy and large train–test gaps.
Response to W2 and Q4 (revised Secs. 5, 7–8, new App. F.4–F.6):
We agree that the submission insufficiently addressed these limitations. We added pretrained causal language modeling, a single-document ArXiv control, and a vision benchmark with multiple MASaLA seeds and smaller train–test gaps.
1. Pretrained causal language modeling (new App. F.5).
We replaced every attention layer of Qwen2.5-0.5B and adapted it with rank-8 LoRA for 300 steps at on WikiText-103, using strictly causal block selection and one seed per method.
Perplexity, lower is better:
| Method | 4K | 16K |
|---|---|---|
| Full attention | 10.95 | 10.38 |
| Block-Sparse | 11.94 | 11.74 |
| Scatterbrain | 12.34 | 12.04 |
| MASaLA (gated) | 12.12 | 12.09 |
| BS+RACE (ungated, ) | 13.85 | 13.16 |
MASaLA operates in a pretrained causal model at 16K, within approximately 1.2–1.7 perplexity points of full attention. Gating improves on ungated BS+RACE by 1.73 points at 4K and 1.07 at 16K. MASaLA is comparable to Scatterbrain, outperforming it at 4K but not 16K.
However, Block-Sparse alone outperforms both hybrids: the low-rank branch provides no predictive benefit in this short-adaptation setting. App. F.5 and Sec. 8 explicitly state this limitation rather than generalizing the vision results to causal language modeling.
This establishes causal compatibility and predictive performance, not causal speedup: runtime benchmarks remain non-causal.
2. Single-document ArXiv control (new App. F.4).
As the reviewer observed, the original long ArXiv inputs packed shorter same-class documents because few individual documents reach 34K tokens. The 32K–64K experiments therefore largely evaluated classification from multiple documents, not one long document.
We added one 4,096-token document per 32,768-token input, using the packed experiments' training budget.
Validation accuracy, one seed each:
| Method | Accuracy |
|---|---|
| MASaLA | 41.0% |
| Scatterbrain | 40.2% |
The 0.8-point difference does not establish superiority with one seed per method. Both results are far below packed-document accuracies (above 90%), supporting the concern that the original benchmark largely measures aggregation of same-class evidence.
Sec. 7 now states this limitation, and we give Imagenette greater weight in the accuracy comparison than packed ArXiv.
3. Vision performance and overfitting (new App. F.6).
Pet and Flowers-102 have approximately 3.7K and 1K training images. Low accuracy and overfitting from scratch at 16K tokens made those results difficult to interpret; even full attention achieved only 24.66% test accuracy on Pet.
We therefore added Imagenette: approximately 9.5K training images, 10 classes, and 16K-token inputs.
| Method | Train accuracy | Test accuracy |
|---|---|---|
| Full attention | 80.63 | 78.12 |
| MASaLA | 75.90 ± 0.04 | 74.42 ± 0.27 |
| Scatterbrain (RACE) | 72.41 | 73.86 |
| Block-Sparse | 71.23 | 71.85 |
| RACE | 68.93 | 71.92 |
MASaLA uses three seeds (mean ± standard deviation); other methods use one each. It exceeds the reported Scatterbrain run by 0.56 points and either individual branch by approximately 2.5 points. Its three test accuracies are 74.73%, 74.22%, and 74.32%. All exceed the reported baseline runs, but additional baseline seeds are needed to establish statistical significance.
Train–test gaps are much smaller than in the original small-data benchmarks: 1.48 points for MASaLA and 2.51 for full attention. Imagenette thus provides more interpretable evidence for the tested architecture. Models remain relatively small; we do not extrapolate beyond the evaluated sizes to larger pretrained systems.
We thank the reviewer for identifying these issues. The revision clarifies the relationship to ELSAA, corrects implementation and complexity descriptions, reports selection-inclusive efficiency, and adds causal language-modeling, single-document, and multi-seed vision experiments.
The fusion rule predates MASaLA, and some original rows still require fuller provenance documentation. We distinguish these limitations from the independently evaluated block-sparse implementation and fused Triton kernel, our principal technical contribution.
We hope the corrected attribution, clearer contribution boundaries, and new evidence address the concerns. We would be grateful if the reviewer would reconsider the evaluation and welcome further questions.
Response to Reviewer P4wx (1/3)
We thank Reviewer P4wx for the careful review, recognizing the fused kernel as "the paper's main new technical content", and finding the fp16 analysis useful. The ELSAA, provenance, and evaluation concerns prompted substantive corrections, marked in blue. Ethics-related authorship information is addressed separately to the AC and Program Chairs to preserve double-blind anonymity.
Response to W1 and Q1 (revised Secs. 2–3.2, Table 7, App. F):
We agree that the submission did not adequately distinguish MASaLA from ELSAA and inconsistently identified implementations. Some results using ELSAA's sortLSH branch appeared under Sec. 3.2's block-sparse description; ELSAA was also omitted from the references. Both were errors. The revision corrects attribution and identifies confirmed configurations.
1. Experimental provenance.
Table 7 lists the vision settings as block-sparse. As the reviewer notes, however, Table 3 and the IMDB columns of Table 4 coincide with ELSAA tables. We are therefore rechecking the run configurations for those rows in CIFAR-10 in Table 4 and for the Block-Sparse and Scatterbrain rows in Tables 1–2, and will post the row-by-row mapping in this thread. Until then, we do not count these rows as evidence for the block-sparse implementation.
2. What the results establish.
The sortLSH results characterize the previously developed fusion formulation with ELSAA's branch, not independent validation of MASaLA's block-sparse kernel. Direct evidence for the new implementation comes from kernel benchmarks, Imagenette, and causal language modeling, each with explicit block selection. We separate these from prior fusion results.
3. Does the sparse branch matter?
App. F.7 compares branches under the same fusion formulation:
Accuracy is comparable in these configurations: block-sparse has a modest Flowers-102 advantage and overlapping Pet performance. However, these are separate runs with incompletely logged seeds, not evidence of general interchangeability or unchanged method rankings across datasets. No directly corresponding block-sparse ArXiv run exists, so we do not extrapolate its sortLSH results.
4. What MASaLA adds beyond ELSAA.
The fusion rule
gSmOS+gROR
already appears in ELSAA and is not a MASaLA contribution. The revision cites ELSAA; introductory wording will be corrected wherever it still implies otherwise. MASaLA's distinct contributions are:
(a) A fused Triton implementation. One kernel computes both branches and fusion, with a recomputation-based backward pass. Peak training memory is 1.08× FlashAttention up to 262K tokens. Including selection, forward speedups are 3.6×, 6.9×, and 13.3× at 32K, 64K, and 128K. This is the implementation contribution highlighted by the reviewer.
(b) Block-sparse architecture and rank analysis. Unlike ELSAA's sortLSH branch, MASaLA selects contiguous key blocks and always includes the query's own and first blocks, enabling regular tiled computation and a fixed sparse budget. Lemma 1 establishes full rank for the ungated hybrid; new Proposition 1 extends it to the implemented gated operator for almost every parameter value. Full rank is not claimed as unique to MASaLA: the analysis establishes its preservation under low-rank fusion and gating.
(c) Fusion characterization. We measure complete selection-inclusive complexity, compare m with exact support mass, test gate-only fusion, analyze query/key-norm and temperature sensitivity, and compare fusion rules on matched sparse support. These correct two interpretations: m does not estimate exact sparse-support attention mass, and gate-only matches MASaLA's clean Imagenette accuracy.
These contributions distinguish MASaLA's architecture, implementation, and analysis without claiming the prior fusion rule as new.