- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ToSA makes online block skipping usable during training. Building on a FlashAttention-2 (FA2)-style Triton kernel, the forward pass computes the scores of each KV block, compares the block's row-wise maximum with the running row maximum, and skips the block when the largest gap over the query tile falls below . A skipped block bypasses value loading, exponentiation, and the PV product, and does not contribute to the softmax normalizer. The forward pass records the block decisions and the log-sum-exp ; the backward pass replays , recovers the attention weights as , and computes , , and only for retained blocks. The experiments fine-tune Qwen3.5-2B, a hybrid model, with a three-stage schedule, applying ToSA to its six full-attention layers with on layers 3/7/23 and on layers 11/15/19. ToSA reports accuracy comparable to the dense fine-tuned baseline (Dense-SFT). For efficiency, the ToSA kernel reaches a 1.77× speedup over FA2 at 128K tokens and 90% block sparsity on an A100-SXM4-80GB.
Critical strengths
S1. A correct, released sparse backward for online block skipping. BLASST's released kernels support only inference (prefill and decode). ToSA provides a differentiable Triton kernel whose backward pass replays the forward mask, so the skip rule used at inference can also be applied during fine-tuning. The released implementation matches Algorithm 1.
Critical weaknesses
W1. The contribution over BLASST is narrow, and the paper presents it as broader. BLASST's Algorithm 1 (arXiv:2512.12087, §3.1.2) performs the same forward computation. BLASST §3.4 further proposes training with this rule, noting that skipped blocks receive no gradient because they are not computed in the forward pass, and §5.5 reports the corresponding results. ToSA acknowledges that BLASST "also discusses sparsity-aware training", yet its abstract and contribution list present trainable online skipping as the main contribution.
W2. The speedup is not measured on the evaluated model, and the accuracy result comes from a single hybrid backbone. (1) Speed. The 1.77× is an attention-kernel measurement at 90% block sparsity on synthetic inputs, which the released benchmark (bench_forward.py) constructs to reach each target sparsity; the caption of Table 1 does not mention this. The fine-tuned models in Tables 2–4, however, run with , and the paper reports neither the sparsity nor the speed of that configuration. Because ToSA modifies only 6 of the 24 layers, the model-level gain will also be smaller than any kernel-level figure. (2) Accuracy. Qwen3.5-2B keeps 18 Gated DeltaNet layers that ToSA leaves untouched. These layers carry information across the whole context without any sparsification, a pathway that a model built only from attention layers lacks. Results on this one backbone therefore cannot show whether "sparse matches dense" is a property of ToSA or of the architecture.
Questions for authors
Q1. For the deployed configuration of the trained model, what block sparsity is realized at the evaluated sequence lengths, and what are the measured attention-kernel and end-to-end speedups? Please report forward and backward times separately, since only the backward pass skips .
Q2. Could you evaluate a non-hybrid model of similar size from a different family, such as a 2–3B Llama or Gemma dense model, with ToSA replacing every attention layer? This requires roughly one additional fine-tuning run at the reported scale, and it is the experiment that separates the method from the backbone.
Q3. Why do layers 3/7/23 use while layers 11/15/19 use ?
Minor clarifications (these do not affect my rating).
- §3.1 defines but uses throughout; the domain should be .
- The and updates described in §3.2 omit the scale that Algorithm 1 includes.
10 comments
Concern 2(weakness 2(1) and question 1)
(1) Speed. The 1.77× is an attention-kernel measurement at 90% block sparsity on synthetic inputs, which the released benchmark (bench_forward.py) constructs to reach each target sparsity; the caption of Table 1 does not mention this. The fine-tuned models in Tables 2–4, however, run with τ=0.85/0.95, and the paper reports neither the sparsity nor the speed of that configuration. Because ToSA modifies only 6 of the 24 layers, the model-level gain will also be smaller than any kernel-level figure.
Response to Weakness 2(1) and question 1. We time the forward pass, the backward pass, and the full training step of the fine-tuned model on NVIDIA A100 GPUs, with both arms taking identical inputs and differing only in whether blocks may be skipped. At 33,762 tokens, ToSA skips 74.4% of blocks and achieves a 1.23× full training-step speedup over the dense baseline. At 67,156 tokens, it skips 82.8% of blocks and achieves a 1.46× speedup.
| Seq. len | Phase | Dense (ms) | ToSA (ms) | Speedup |
|---|---|---|---|---|
| 33,762 | Forward | 2,103.70 | 1,749.76 | 1.202x |
| 33,762 | Backward | 7,327.55 | 5,888.83 | 1.244x |
| 33,762 | Full step | 9,685.41 | 7,895.52 | 1.227x |
| 67,156 | Forward | 5,215.46 | 3,811.19 | 1.368x |
| 67,156 | Backward | 18,781.07 | 12,432.40 | 1.511x |
| 67,156 | Full step | 24,506.29 | 16,751.11 | 1.463x |
The 1.77× speedup reported in Table 1 was measured for the attention kernel on synthetic inputs at 90% block sparsity.
Concern 3(weakness 2(2) and question 2)
(2) Accuracy. Qwen3.5-2B keeps 18 Gated DeltaNet layers that ToSA leaves untouched. These layers carry information across the whole context without any sparsification, a pathway that a model built only from attention layers lacks. Results on this one backbone therefore cannot show whether "sparse matches dense" is a property of ToSA or of the architecture.
Response to Weakness 2(2) and question 2. We agree that on a hybrid backbone the 18 untouched Gated DeltaNet layers provide an unsparsified path across the full context, so results there cannot by themselves separate ToSA from the architecture. We therefore also apply ToSA to a non-hybrid backbone, Qwen3-VL-2B, where every attention layer is sparsified, and train it under the same recipe. On Qwen3-VL-2B, ToSA scores 0.77 points higher than Dense-SFT on LongVideoBench, 0.46 points higher on Video-MME, and 0.63 points higher on LVBench. Its six-task LongBench average also rises from 35.01 to 35.41.
| Method | Params. | LongVideoBench | Video-MME | LVBench |
|---|---|---|---|---|
| General MLLMs | ||||
| InternVL3.5 | 2B | 57.4 | 58.4 | -- |
| Qwen3-VL | 2B | 55.6 | 61.9 | 47.4 |
| NEO-ov | 2B | 56.8 | 60.4 | 43.3 |
| Penguin-VL | 2B | 59.5 | 57.4 | -- |
| Token Compression Methods | ||||
| VideoChat-Flash | 2B | 58.3 | 57.0 | 44.9 |
| Sparse Attention Methods | ||||
| XAttention | 7B | 58.5 | 63.0 | -- |
| VecAttention | 7B | 59.4 | 64.8 | -- |
| VideoNSA | 7B | 60.0 | -- | -- |
| BLASST (Triton reimpl.) | 2B | 51.38 | 55.31 | 43.25 |
| NSA | 2B | 50.09 | 49.93 | 38.74 |
| Ours | ||||
| Dense-SFT (Qwen3-VL-2B) | 2B | 54.12 | 56.68 | 43.52 |
| ToSA (Qwen3-VL-2B) | 2B | 54.89 | 57.14 | 44.15 |
| Dense-SFT (Qwen3.5-2B) | 2B | 58.19 | 64.22 | 48.53 |
| ToSA (Qwen3.5-2B) | 2B | 60.02 | 63.90 | 49.12 |
Six LongBench tasks with average input length above 8K tokens:
| Method | DuReader | Hotpot | MuSiQue | NarrQA | QMSum | TriviaQA | Avg. (5) | Avg. (6) |
|---|---|---|---|---|---|---|---|---|
| AHN-DN (Qwen2.5-3B) | 25.12 | 42.83 | 19.78 | 19.11 | 22.35 | 86.17 | 38.05 | 35.89 |
| AHN-DN (Qwen2.5-7B) | 26.42 | 54.24 | 29.30 | 25.08 | 21.69 | 89.49 | 43.96 | 41.04 |
| NOSA (3B) | -- | 40.00 | 19.70 | 21.20 | 23.50 | 88.20 | 38.52 | -- |
| InfLLMv2 (3B) | -- | 30.10 | 14.70 | 19.70 | 23.20 | 87.20 | 34.98 | -- |
| FastKV (Llama-3.2-3B) | -- | 47.58 | 21.93 | 24.02 | 23.24 | 89.22 | 41.20 | -- |
| BLASST (Triton reimpl.) | 28.09 | 46.22 | 29.10 | 17.73 | 21.48 | 84.63 | 39.83 | 37.88 |
| NSA (Qwen3.5-2B) | 26.36 | 50.80 | 29.53 | 17.11 | 17.82 | 84.01 | 39.85 | 37.60 |
| Ours | ||||||||
| Dense-SFT (Qwen3-VL-2B) | 26.10 | 43.20 | 22.85 | 15.20 | 15.80 | 86.90 | 36.79 | 35.01 |
| ToSA (Qwen3-VL-2B) | 26.52 | 43.65 | 23.31 | 15.52 | 16.17 | 87.31 | 37.19 | 35.41 |
| Dense-SFT (Qwen3.5-2B) | 32.80 | 52.22 | 27.08 | 17.21 | 15.83 | 83.92 | 39.25 | 38.18 |
| ToSA (Qwen3.5-2B) | 33.94 | 50.18 | 31.31 | 17.75 | 19.07 | 85.46 | 40.75 | 39.62 |
Four representative tasks from the 128K subset of InfiniteBench (QA Avg. averages En.QA and Zh.QA; Avg. averages all four tasks):
| Method | En.QA | Zh.QA | En.MC | Code.Debug | QA Avg. | Avg. |
|---|---|---|---|---|---|---|
| AHN-DN (Qwen2.5-3B) | 10.61 | 16.41 | -- | -- | 13.51 | -- |
| AHN-GDN (Qwen2.5-7B) | 13.37 | 20.48 | -- | -- | 16.93 | -- |
| MiniCPM-2B-128K | 16.33 | 23.06 | 29.69 | 22.08 | 19.70 | 22.79 |
| MInference | 8.50 | 12.14 | 65.94 | 22.08 | 10.32 | 27.17 |
| FlexPrefill | 12.60 | 11.63 | 57.21 | 22.84 | 12.12 | 26.07 |
| BLASST (Qwen3.5-2B) | 16.49 | 16.98 | 60.43 | 40.89 | 16.74 | 33.70 |
| NSA (Qwen3.5-2B) | 17.93 | 20.89 | 65.86 | 47.81 | 19.41 | 38.12 |
| Ours | ||||||
| Dense-SFT (Qwen3-VL-2B) | 11.20 | 19.80 | 43.90 | 29.60 | 15.50 | 26.13 |
| ToSA (Qwen3-VL-2B) | 11.49 | 20.21 | 44.54 | 30.20 | 15.85 | 26.61 |
| Dense-SFT (Qwen3.5-2B) | 16.40 | 22.88 | 64.38 | 46.95 | 19.64 | 37.65 |
| ToSA (Qwen3.5-2B) | 15.83 | 28.24 | 71.18 | 48.73 | 22.04 | 41.00 |
Concern 1 (Weakness 1):
The contribution over BLASST is narrow, and the paper presents it as broader. BLASST's Algorithm 1 (arXiv:2512.12087, §3.1.2) performs the same forward computation. BLASST §3.4 further proposes training with this rule, noting that skipped blocks receive no gradient because they are not computed in the forward pass, and §5.5 reports the corresponding results. ToSA acknowledges that BLASST "also discusses sparsity-aware training", yet its abstract and contribution list present trainable online skipping as the main contribution.
Response to Weakness 1
We thank the reviewer for this insightful comment. We first acknowledge the significant contributions of BLASST, which introduces an effective online block-level sparsification that skips low-contribution blocks without requiring an additional selector. ToSA builds on this principle and focuses on making it efficiently trainable and portable to a broader range of GPU architectures.
Compared with BLASST, our main contributions lie in two aspects:
1. Reusing the Online Decision for Sparse Backward.
Building on the online decision of BLASST, ToSA introduces a block-level mask that is recorded during the forward pass, and reuses this same mask in the backward pass to compute gradients only over the retained blocks. Reusing this mask enables sparse backward computation in ToSA, making our implementation trainable without a second pass over KV blocks to repeat the selection. The forward kernel writes each mask entry at the point where it decides whether to skip the corresponding block.
2. Extension to Ampere GPUs with Trainable Sparsity.
The released BLASST implementation primarily targets Hopper and Blackwell GPUs. Building on its online block-skipping approach, we developed a Triton kernel that supports both sparse forward and backward computation on A100 GPUs. This gives researchers working with A100s the opportunity to explore this approach in training as well. We plan to release the kernel to support reproducibility and further research.