acceptodds
Under review as a conference paper at ICLR 2027

ToSA: Trainable Online Sparse Attention

Abstract

Processing long sequences is expensive because dense self-attention has quadratic complexity. Existing sparse attention methods, e.g., Native Sparse Attention, can skip score computation, but prediction and indexing introduce additional overhead. Online block skipping avoids a separate selection stage during inference, but high sparsity often reduces model accuracy. To address this issue, we propose ToSA, a trainable online sparse attention method for long-sequence modeling. ToSA reuses online-softmax statistics in a fused Triton kernel to identify low-contribution KV blocks after the attention scores are computed. For each skipped block, it bypasses value loading, softmax computation, and the subsequent P V accumulation. Its sparse backward kernel replays the block decisions recorded in the forward pass, enabling end-to-end training. We further propose a three-stage schedule that begins with dense training, gradually increases spar- sity, and ends with training at the desired sparsity. We evaluate ToSA with Qwen3.5-2B on long-context and long-video understanding. Compared with the dense fine-tuning baseline, ToSA improves the LongBench score from 38.18 to 39.62 and the LongVideoBench score from 58.19 to 60.02. In terms of efficiency, ToSA achieves up to a 1.77× attention-kernel speedup at a 128K sequence length. Together, these results demonstrate that ToSA substantially accelerates long-sequence modeling while preserving long-sequence understanding.

Primary area:
foundation or frontier models, including LLMs
Keywords:
sparse Attention, efficient attention, triton kernels, large language models, multimodal models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5hmkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ToSA makes online block skipping usable during training. Building on a FlashAttention-2 (FA2)-style Triton kernel, the forward pass computes the scores Sij=sQiKj⊤S_{ij}=sQ_iK_j^\top of each KV block, compares the block's row-wise maximum with the running row maximum, and skips the block when the largest gap over the query tile falls below log⁡τ\log\tau. A skipped block bypasses value loading, exponentiation, and the PV product, and does not contribute to the softmax normalizer. The forward pass records the block decisions MM and the log-sum-exp LL; the backward pass replays MM, recovers the attention weights as exp⁡(Sij−Li)\exp(S_{ij}-L_i), and computes dQdQ, dKdK, and dVdV only for retained blocks. The experiments fine-tune Qwen3.5-2B, a hybrid model, with a three-stage schedule, applying ToSA to its six full-attention layers with τ=0.95\tau=0.95 on layers 3/7/23 and τ=0.85\tau=0.85 on layers 11/15/19. ToSA reports accuracy comparable to the dense fine-tuned baseline (Dense-SFT). For efficiency, the ToSA kernel reaches a 1.77× speedup over FA2 at 128K tokens and 90% block sparsity on an A100-SXM4-80GB.

Critical strengths

S1. A correct, released sparse backward for online block skipping. BLASST's released kernels support only inference (prefill and decode). ToSA provides a differentiable Triton kernel whose backward pass replays the forward mask, so the skip rule used at inference can also be applied during fine-tuning. The released implementation matches Algorithm 1.

Critical weaknesses

W1. The contribution over BLASST is narrow, and the paper presents it as broader. BLASST's Algorithm 1 (arXiv:2512.12087, §3.1.2) performs the same forward computation. BLASST §3.4 further proposes training with this rule, noting that skipped blocks receive no gradient because they are not computed in the forward pass, and §5.5 reports the corresponding results. ToSA acknowledges that BLASST "also discusses sparsity-aware training", yet its abstract and contribution list present trainable online skipping as the main contribution.

W2. The speedup is not measured on the evaluated model, and the accuracy result comes from a single hybrid backbone. (1) Speed. The 1.77× is an attention-kernel measurement at 90% block sparsity on synthetic inputs, which the released benchmark (bench_forward.py) constructs to reach each target sparsity; the caption of Table 1 does not mention this. The fine-tuned models in Tables 2–4, however, run with τ=0.95/0.85\tau=0.95/0.85, and the paper reports neither the sparsity nor the speed of that configuration. Because ToSA modifies only 6 of the 24 layers, the model-level gain will also be smaller than any kernel-level figure. (2) Accuracy. Qwen3.5-2B keeps 18 Gated DeltaNet layers that ToSA leaves untouched. These layers carry information across the whole context without any sparsification, a pathway that a model built only from attention layers lacks. Results on this one backbone therefore cannot show whether "sparse matches dense" is a property of ToSA or of the architecture.

Questions for authors

Q1. For the deployed configuration of the trained model, what block sparsity is realized at the evaluated sequence lengths, and what are the measured attention-kernel and end-to-end speedups? Please report forward and backward times separately, since only the backward pass skips QK⊤QK^\top.

Q2. Could you evaluate a non-hybrid model of similar size from a different family, such as a 2–3B Llama or Gemma dense model, with ToSA replacing every attention layer? This requires roughly one additional fine-tuning run at the reported scale, and it is the experiment that separates the method from the backbone.

Q3. Why do layers 3/7/23 use τ=0.95\tau=0.95 while layers 11/15/19 use τ=0.85\tau=0.85?

Minor clarifications (these do not affect my rating).

  • §3.1 defines τ∈(0,1]\tau\in(0,1] but uses τ=0\tau=0 throughout; the domain should be [0,1][0,1].
  • The dQdQ and dKdK updates described in §3.2 omit the scale ss that Algorithm 1 includes.
10 comments

Concern 1 (Weakness 1):

The contribution over BLASST is narrow, and the paper presents it as broader. BLASST's Algorithm 1 (arXiv:2512.12087, §3.1.2) performs the same forward computation. BLASST §3.4 further proposes training with this rule, noting that skipped blocks receive no gradient because they are not computed in the forward pass, and §5.5 reports the corresponding results. ToSA acknowledges that BLASST "also discusses sparsity-aware training", yet its abstract and contribution list present trainable online skipping as the main contribution.

Response to Weakness 1

We thank the reviewer for this insightful comment. We first acknowledge the significant contributions of BLASST, which introduces an effective online block-level sparsification that skips low-contribution blocks without requiring an additional selector. ToSA builds on this principle and focuses on making it efficiently trainable and portable to a broader range of GPU architectures.

Compared with BLASST, our main contributions lie in two aspects:

1. Reusing the Online Decision for Sparse Backward.

Building on the online decision of BLASST, ToSA introduces a block-level mask that is recorded during the forward pass, and reuses this same mask in the backward pass to compute gradients only over the retained blocks. Reusing this mask enables sparse backward computation in ToSA, making our implementation trainable without a second pass over KV blocks to repeat the selection. The forward kernel writes each mask entry at the point where it decides whether to skip the corresponding block.

2. Extension to Ampere GPUs with Trainable Sparsity.

The released BLASST implementation primarily targets Hopper and Blackwell GPUs. Building on its online block-skipping approach, we developed a Triton kernel that supports both sparse forward and backward computation on A100 GPUs. This gives researchers working with A100s the opportunity to explore this approach in training as well. We plan to release the kernel to support reproducibility and further research.

Concern 2(weakness 2(1) and question 1)

(1) Speed. The 1.77× is an attention-kernel measurement at 90% block sparsity on synthetic inputs, which the released benchmark (bench_forward.py) constructs to reach each target sparsity; the caption of Table 1 does not mention this. The fine-tuned models in Tables 2–4, however, run with τ=0.85/0.95, and the paper reports neither the sparsity nor the speed of that configuration. Because ToSA modifies only 6 of the 24 layers, the model-level gain will also be smaller than any kernel-level figure.

Response to Weakness 2(1) and question 1. We time the forward pass, the backward pass, and the full training step of the fine-tuned model on NVIDIA A100 GPUs, with both arms taking identical inputs and differing only in whether blocks may be skipped. At 33,762 tokens, ToSA skips 74.4% of blocks and achieves a 1.23× full training-step speedup over the dense baseline. At 67,156 tokens, it skips 82.8% of blocks and achieves a 1.46× speedup.

Seq. lenPhaseDense (ms)ToSA (ms)Speedup
33,762Forward2,103.701,749.761.202x
33,762Backward7,327.555,888.831.244x
33,762Full step9,685.417,895.521.227x
67,156Forward5,215.463,811.191.368x
67,156Backward18,781.0712,432.401.511x
67,156Full step24,506.2916,751.111.463x

The 1.77× speedup reported in Table 1 was measured for the attention kernel on synthetic inputs at 90% block sparsity.

Concern 3(weakness 2(2) and question 2)

(2) Accuracy. Qwen3.5-2B keeps 18 Gated DeltaNet layers that ToSA leaves untouched. These layers carry information across the whole context without any sparsification, a pathway that a model built only from attention layers lacks. Results on this one backbone therefore cannot show whether "sparse matches dense" is a property of ToSA or of the architecture.

Response to Weakness 2(2) and question 2. We agree that on a hybrid backbone the 18 untouched Gated DeltaNet layers provide an unsparsified path across the full context, so results there cannot by themselves separate ToSA from the architecture. We therefore also apply ToSA to a non-hybrid backbone, Qwen3-VL-2B, where every attention layer is sparsified, and train it under the same recipe. On Qwen3-VL-2B, ToSA scores 0.77 points higher than Dense-SFT on LongVideoBench, 0.46 points higher on Video-MME, and 0.63 points higher on LVBench. Its six-task LongBench average also rises from 35.01 to 35.41.

MethodParams.LongVideoBenchVideo-MMELVBench
General MLLMs
InternVL3.52B57.458.4--
Qwen3-VL2B55.661.947.4
NEO-ov2B56.860.443.3
Penguin-VL2B59.557.4--
Token Compression Methods
VideoChat-Flash2B58.357.044.9
Sparse Attention Methods
XAttention7B58.563.0--
VecAttention7B59.464.8--
VideoNSA7B60.0----
BLASST (Triton reimpl.)2B51.3855.3143.25
NSA2B50.0949.9338.74
Ours
Dense-SFT (Qwen3-VL-2B)2B54.1256.6843.52
ToSA (Qwen3-VL-2B)2B54.8957.1444.15
Dense-SFT (Qwen3.5-2B)2B58.1964.2248.53
ToSA (Qwen3.5-2B)2B60.0263.9049.12

Six LongBench tasks with average input length above 8K tokens:

MethodDuReaderHotpotMuSiQueNarrQAQMSumTriviaQAAvg. (5)Avg. (6)
AHN-DN (Qwen2.5-3B)25.1242.8319.7819.1122.3586.1738.0535.89
AHN-DN (Qwen2.5-7B)26.4254.2429.3025.0821.6989.4943.9641.04
NOSA (3B)--40.0019.7021.2023.5088.2038.52--
InfLLMv2 (3B)--30.1014.7019.7023.2087.2034.98--
FastKV (Llama-3.2-3B)--47.5821.9324.0223.2489.2241.20--
BLASST (Triton reimpl.)28.0946.2229.1017.7321.4884.6339.8337.88
NSA (Qwen3.5-2B)26.3650.8029.5317.1117.8284.0139.8537.60
Ours
Dense-SFT (Qwen3-VL-2B)26.1043.2022.8515.2015.8086.9036.7935.01
ToSA (Qwen3-VL-2B)26.5243.6523.3115.5216.1787.3137.1935.41
Dense-SFT (Qwen3.5-2B)32.8052.2227.0817.2115.8383.9239.2538.18
ToSA (Qwen3.5-2B)33.9450.1831.3117.7519.0785.4640.7539.62

Four representative tasks from the 128K subset of InfiniteBench (QA Avg. averages En.QA and Zh.QA; Avg. averages all four tasks):

MethodEn.QAZh.QAEn.MCCode.DebugQA Avg.Avg.
AHN-DN (Qwen2.5-3B)10.6116.41----13.51--
AHN-GDN (Qwen2.5-7B)13.3720.48----16.93--
MiniCPM-2B-128K16.3323.0629.6922.0819.7022.79
MInference8.5012.1465.9422.0810.3227.17
FlexPrefill12.6011.6357.2122.8412.1226.07
BLASST (Qwen3.5-2B)16.4916.9860.4340.8916.7433.70
NSA (Qwen3.5-2B)17.9320.8965.8647.8119.4138.12
Ours
Dense-SFT (Qwen3-VL-2B)11.2019.8043.9029.6015.5026.13
ToSA (Qwen3-VL-2B)11.4920.2144.5430.2015.8526.61
Dense-SFT (Qwen3.5-2B)16.4022.8864.3846.9519.6437.65
ToSA (Qwen3.5-2B)15.8328.2471.1848.7322.0441.00
Reviewer_wsVrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ToSA makes online block-skipping attention trainable. After computing QK scores, it uses block-local and running maxima to skip softmax, value loading, and PV computation for selected blocks. A sparse backward kernel replays the forward mask, and fine-tuning uses dense warmup followed by progressively increased thresholds. Experiments on Qwen3.5-2B cover long-text and long-video tasks, generally matching or improving on a controlled dense-SFT baseline. Kernel benchmarks report up to 1.77× speedup over FA2 at 128K tokens and 90% block sparsity.

Critical strengths
  1. A concrete implementation of trainable online skipping. The fused forward path and mask-replaying backward avoid an auxiliary learned selector. The paper explicitly states that QK computation remains dense and that gradients are conditional on the recorded mask.

  2. Useful controlled training experiments. Dense-SFT uses the same backbone, data, and optimization settings. Tables 5–6 separate warmup, progressive sparsification, and sparse training/inference, supporting adaptation to the sparse computation path across text and video tasks.

Critical weaknesses
  1. The technical contribution beyond BLASST is unclear. The forward rule appears essentially unchanged, while BLASST already describes and evaluates sparsity-aware training. An unreleased backward kernel establishes an implementation gap, not an incompatibility with training. The authors do not clearly identify the nontrivial challenges solved by backward replay or the insight beyond the progressive recipe.

  2. The experiments do not adequately establish the quality and generality of trainable sparse attention. Dense-SFT is useful, but there is no matched trainable sparse-attention baseline. Evaluation uses one hybrid 2B backbone, unexplained layer-wise thresholds, and selected older benchmark tasks. Although a 1.77× kernel speedup is reported, end-to-end model forward, backward, and full training-step timings or speedups are missing, leaving the practical acceleration unclear.

Questions for authors
  1. What nontrivial challenges does ToSA solve in extending BLASST to backward computation, beyond BLASST's existing sparsity-aware training?
  2. How does ToSA compare with a trainable sparse baseline under matched data, training budget, and cost?
  3. How were the layer-specific thresholds chosen? Do the findings hold under threshold variation, larger models, and LongBench v2?
  4. What are the end-to-end model forward, backward, and full training-step timings and speedups relative to Dense-SFT under the main evaluation and training configurations?
4 comments

Concern 1 (Weakness 1 and question 1):

The technical contribution beyond BLASST is unclear. The forward rule appears essentially unchanged, while BLASST already describes and evaluates sparsity-aware training. An unreleased backward kernel establishes an implementation gap, not an incompatibility with training. The authors do not clearly identify the nontrivial challenges solved by backward replay or the insight beyond the progressive recipe.

Response to Weakness 1 and Question 1. We thank the reviewer for this insightful comment. We first acknowledge the significant contributions of BLASST, which introduces an effective online block-level sparsification that skips low-contribution blocks without requiring an additional selector. ToSA builds on this principle and focuses on making it efficiently trainable and portable to a broader range of GPU architectures.

Compared with BLASST, our main contributions lie in two aspects:

1. Reusing the Online Decision for Sparse Backward.

Building on the online decision of BLASST, ToSA introduces a block-level mask that is recorded during the forward pass, and reuses this same mask in the backward pass to compute gradients only over the retained blocks. Reusing this mask enables sparse backward computation in ToSA, making our implementation trainable without a second pass over KV blocks to repeat the selection. The forward kernel writes each mask entry at the point where it decides whether to skip the corresponding block.

2. Extension to Ampere GPUs with Trainable Sparsity.

The released BLASST implementation primarily targets Hopper and Blackwell GPUs. Building on its online block-skipping approach, we developed a Triton kernel that supports both sparse forward and backward computation on A100 GPUs. This gives researchers working with A100s the opportunity to explore this approach in training as well. We plan to release the kernel to support reproducibility and further research.

Concern 2 (Weakness 2(a) and question 2):

The experiments do not adequately establish the quality and generality of trainable sparse attention. Dense-SFT is useful, but there is no matched trainable sparse-attention baseline. Evaluation uses one hybrid 2B backbone, unexplained layer-wise thresholds, and selected older benchmark tasks. Although a 1.77× kernel speedup is reported, end-to-end model forward, backward, and full training-step timings or speedups are missing, leaving the practical acceleration unclear.

Part (a): missing matched trainable sparse-attention baseline and limited evaluation.

Response to Weakness 2(a) and Question 2. We agree that a matched trainable sparse-attention baseline is essential. To establish a matched trainable sparse-attention baseline, we integrated NSA into the same Qwen3.5-2B backbone and fine-tuned it using the same data and training budget as ToSA. Under the same SFT setup on Qwen3.5-2B, ToSA outperforms NSA on all three video benchmarks and achieves higher average scores on the six LongBench tasks (39.62 vs. 37.60) and the four InfiniteBench tasks (41.00 vs. 38.12).

MethodParams.LongVideoBenchVideo-MMELVBench
General MLLMs
InternVL3.52B57.458.4--
Qwen3-VL2B55.661.947.4
NEO-ov2B56.860.443.3
Penguin-VL2B59.557.4--
Token Compression Methods
VideoChat-Flash2B58.357.044.9
Sparse Attention Methods
XAttention7B58.563.0--
VecAttention7B59.464.8--
VideoNSA7B60.0----
BLASST (Triton reimpl.)2B51.3855.3143.25
NSA2B50.0949.9338.74
Ours
Dense-SFT (Qwen3-VL-2B)2B54.1256.6843.52
ToSA (Qwen3-VL-2B)2B54.8957.1444.15
Dense-SFT (Qwen3.5-2B)2B58.1964.2248.53
ToSA (Qwen3.5-2B)2B60.0263.9049.12

Six LongBench tasks with average input length above 8K tokens:

MethodDuReaderHotpotMuSiQueNarrQAQMSumTriviaQAAvg. (5)Avg. (6)
AHN-DN (Qwen2.5-3B)25.1242.8319.7819.1122.3586.1738.0535.89
AHN-DN (Qwen2.5-7B)26.4254.2429.3025.0821.6989.4943.9641.04
NOSA (3B)--40.0019.7021.2023.5088.2038.52--
InfLLMv2 (3B)--30.1014.7019.7023.2087.2034.98--
FastKV (Llama-3.2-3B)--47.5821.9324.0223.2489.2241.20--
BLASST (Triton reimpl.)28.0946.2229.1017.7321.4884.6339.8337.88
NSA (Qwen3.5-2B)26.3650.8029.5317.1117.8284.0139.8537.60
Ours
Dense-SFT (Qwen3-VL-2B)26.1043.2022.8515.2015.8086.9036.7935.01
ToSA (Qwen3-VL-2B)26.5243.6523.3115.5216.1787.3137.1935.41
Dense-SFT (Qwen3.5-2B)32.8052.2227.0817.2115.8383.9239.2538.18
ToSA (Qwen3.5-2B)33.9450.1831.3117.7519.0785.4640.7539.62

Four representative tasks from the 128K subset of InfiniteBench (QA Avg. averages En.QA and Zh.QA; Avg. averages all four tasks):

MethodEn.QAZh.QAEn.MCCode.DebugQA Avg.Avg.
AHN-DN (Qwen2.5-3B)10.6116.41----13.51--
AHN-GDN (Qwen2.5-7B)13.3720.48----16.93--
MiniCPM-2B-128K16.3323.0629.6922.0819.7022.79
MInference8.5012.1465.9422.0810.3227.17
FlexPrefill12.6011.6357.2122.8412.1226.07
BLASST (Qwen3.5-2B)16.4916.9860.4340.8916.7433.70
NSA (Qwen3.5-2B)17.9320.8965.8647.8119.4138.12
Ours
Dense-SFT (Qwen3-VL-2B)11.2019.8043.9029.6015.5026.13
ToSA (Qwen3-VL-2B)11.4920.2144.5430.2015.8526.61
Dense-SFT (Qwen3.5-2B)16.4022.8864.3846.9519.6437.65
ToSA (Qwen3.5-2B)15.8328.2471.1848.7322.0441.00

Concern 3 (Weakness 2(b) and question 3):

Evaluation uses one hybrid 2B backbone, unexplained layer-wise thresholds, and selected older benchmark tasks.

Response to Weakness 2(b) and question 3. We thank the reviewer for raising the question regarding threshold selection. The layer-wise configuration was originally explored in our early training-free analysis, where we studied how different layers respond to block-level sparsity. When moving to SFT, however, we found that different threshold configurations do not lead to noticeable differences in the final results, indicating that this configuration has limited influence on the trained model. We therefore report the table below, which compares several configurations and shows that performance is stable across them.

ConfigurationLongVideoBenchLVBenchVideoMME
0.95 @ L3/7/23, 0.85 @ L11/15/1960.0249.1263.90
0.9 @ L3/7/11/15/19/2360.0549.0363.12
0.95 @ L3/7/11/15/19/2360.0548.4863.45
0.85 @ L7/11, 0.95 @ L3/15/19/2360.0049.2263.33

Concern 4 (Weakness 2(c) and question 4):

Response to Weakness 2(c) and Question 4. We time the forward pass, the backward pass, and the full training step of the fine-tuned model on NVIDIA A100 GPUs, with both arms taking identical inputs and differing only in whether blocks may be skipped. ToSA achieves full training-step speedups of 1.23× at 33,762 tokens and 1.46× at 67,156 tokens relative to the dense baseline.

Seq. lenPhaseDense (ms)ToSA (ms)Speedup
33,762Forward2,103.701,749.761.202x
33,762Backward7,327.555,888.831.244x
33,762Full step9,685.417,895.521.227x
67,156Forward5,215.463,811.191.368x
67,156Backward18,781.0712,432.401.511x
67,156Full step24,506.2916,751.111.463x

We express our sincere gratitude to the reviewer for dedicating time to review our paper. We have provided comprehensive responses to all the concerns. As the discussion deadline looms within 2 days, we would like to inquire if our responses have adequately addressed your questions. We are more than willing to address any concerns and ensure a comprehensive resolution. Thank you for your time and consideration.

Reviewer_7ARmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

ToSA invents a way of gradually fine-tuning dense transformer layers into sparse ones. After computing a query-key block, it utilizes the attention scores to evict the less contributive KV cache tokens. During this process, the authors have also provided support for both forward and backward kernels to enable efficient learning of such sparse attention. In terms of the experiments, the authors have experimented with LongBench and long video understanding to show that this method can slightly improve the performance with increased efficiency.

Critical strengths
  1. I think the idea of the paper is conceptually clean, and its principle is applicable to wider domains.
  2. I like how the authors have proved the effectiveness of the method on both long context tasks and long video understanding
  3. The presented experiments support the claims in the paper.
Critical weaknesses
  1. The increment over the closest predecessor is not established. BLASST (MLSys 2026) already describes training with sparse forward attention and zero gradient on skipped blocks (Sec. 3.4), and reports a sparsity-aware training experiment (Sec. 5.5). More importantly, their principle is very similar to this paper in utilizing the attention scores as the major proxy for supporting sparsity. ToSA acknowledges the discussion but emphasizes that a backward kernel was unavailable in the released implementation. Providing that kernel can be a useful engineering contribution; it does not by itself establish a new trainable-attention principle. Therefore, I would really appreciate it if the authors can offer a more convincing explanation.

  2. The threshold configuration is difficult for another people to choose or transfer. The main model uses 0.95 in Layers 3/7/23 and 0.85 in Layers 11/15/19, while the uniform-threshold and training-free analyses use other configurations. Figures 5-6 explicitly use different inputs, and achieved sparsity varies even at the same threshold and length. The paper should explain how the layer choices were selected and characterize accuracy sensitivity near that operating point. This would show whether the recipe is robust without requiring another backbone.

Questions for authors

See above

This paper somehow reminds me of a series of papers doing learnable compression, e.g., beacon tokens for general long context tasks and "One Token per Highly Selective Frame" for long video understanding. The authors might consider cite these.

4 comments

Concern 1 (Weakness 1):

The increment over the closest predecessor is not established. BLASST (MLSys 2026) already describes training with sparse forward attention and zero gradient on skipped blocks (Sec. 3.4), and reports a sparsity-aware training experiment (Sec. 5.5).More importantly, their principle is very similar to this paper in utilizing the attention scores as the major proxy for supporting sparsity. ToSA acknowledges the discussion but emphasizes that a backward kernel was unavailable in the released implementation.Providing that kernel can be a useful engineering contribution; it does not by itself establish a new trainable-attention principle.I would really appreciate it if the authors can offer a more convincing explanation of how ToSA differs in principle (not just implementation) from BLASST.

Response to Weakness 1

We thank the reviewer for this insightful comment. We first acknowledge the significant contributions of BLASST, which introduces an effective online block-level sparsification that skips low-contribution blocks without requiring an additional selector. ToSA builds on this principle and focuses on making it efficiently trainable and portable to a broader range of GPU architectures.

Compared with BLASST, our main contributions lie in two aspects:

1. Reusing the Online Decision for Sparse Backward.

Building on the online decision of BLASST, ToSA introduces a block-level mask that is recorded during the forward pass, and reuses this same mask in the backward pass to compute gradients only over the retained blocks. Reusing this mask enables sparse backward computation in ToSA, making our implementation trainable without a second pass over KV blocks to repeat the selection. The forward kernel writes each mask entry at the point where it decides whether to skip the corresponding block.

2. Extension to Ampere GPUs with Trainable Sparsity.

The released BLASST implementation primarily targets Hopper and Blackwell GPUs. Building on its online block-skipping approach, we developed a Triton kernel that supports both sparse forward and backward computation on A100 GPUs. This gives researchers working with A100s the opportunity to explore this approach in training as well. We plan to release the kernel to support reproducibility and further research.

Concern 2 (Weakness 2):

The threshold configuration is difficult for another person to choose or transfer: The main model uses 0.95 in Layers 3/7/23 and 0.85 in Layers 11/15/19.The uniform-threshold and training-free analyses use other configurations.Figures 5–6 explicitly use different inputs, and achieved sparsity varies even at the same threshold and length. Explain how the layer-specific threshold choices were selected. Characterize accuracy sensitivity near that operating point. This would show whether the recipe is robust without requiring another backbone.

Response to Weakness 2

We thank the reviewer for raising the question regarding threshold selection. The layer-wise configuration was originally explored in our early training-free analysis, where we studied how different layers respond to block-level sparsity. When moving to SFT, however, we found that different threshold configurations do not lead to noticeable differences in the final results, indicating that this configuration has limited influence on the trained model. We therefore report the table below, which compares several configurations and shows that performance is stable across them.

ConfigurationLongVideoBenchLVBenchVideoMME
0.95 @ L3/7/23, 0.85 @ L11/15/1960.0249.1263.90
0.9 @ L3/7/11/15/19/2360.0549.0363.12
0.95 @ L3/7/11/15/19/2360.0548.4863.45
0.85 @ L7/11, 0.95 @ L3/15/19/2360.0049.2263.33

Concern 3 (question):

This paper somehow reminds me of a series of papers doing learnable compression, e.g., beacon tokens for general long context tasks and "One Token per Highly Selective Frame" for long video understanding. The authors might consider cite these.

Response to question

We sincerely thank the reviewer for highlighting the connection between our work and prior research on learnable compression. We appreciate the suggestion to consider both beacon-token approaches for long-context tasks and One Token per Highly Selective Frame for long-video understanding. We will cite both works in the revised manuscript.

We express our sincere gratitude to the reviewer for dedicating time to review our paper. We have provided comprehensive responses to all the concerns. As the discussion deadline looms within 2 days, we would like to inquire if our responses have adequately addressed your questions. We are more than willing to address any concerns and ensure a comprehensive resolution. Thank you for your time and consideration.

I appreciate the additional clarifications from the authors, and I think my concerns are addressed. Therefore, I keep my current positive rating of 3.

We are grateful for the reviewer's thoughtful consideration of our responses and are pleased that our clarifications have satisfactorily addressed the concerns raised. Your constructive feedback has been invaluable in helping us improve our work.

Comments on the paper

1 comment

Dear Area Chair,

Thank you and the reviewers for the careful reading and constructive feedback. We summarize the strengths highlighted during review, the evidence added during rebuttal, and the outcome of each discussion.

Strengths and additional evidence

  1. A conceptually clean method with an efficient sparse backward pass. Reviewer 7ARm described the idea as “conceptually clean.” Reviewer 5hmk highlighted ToSA’s released sparse backward kernel and recognized that recording block decisions during the forward pass allows the backward pass to skip QK⊤QK^\top recomputation for dropped blocks. The reviewer noted that “the backward timings bear this out.”

  2. Effective sparse fine-tuning for long text and long video. Reviewer 7ARm highlighted the evaluation on “both long context tasks and long video understanding.” Reviewer 5hmk summarized ToSA’s accuracy as “comparable to the dense fine-tuned baseline (Dense-SFT).”

  3. End-to-end training and inference speedups. During rebuttal, we measured forward, backward, full training-step, and inference prefill times on NVIDIA A100 GPUs. At a uniform τ=0.9\tau=0.9, ToSA achieved 1.23× and 1.46× full training-step speedups at 33,762 and 67,156 tokens, respectively (Table 2). On the full LVBench test set, it achieved a 1.42× prefill speedup at 80.9% mean block sparsity (Table 11).

Reviewer 7ARm (rating 3, confidence 3)

  • Concerns. The reviewer asked what ToSA contributes beyond BLASST and how the layer-wise thresholds were selected.
  • Our response. We explained how ToSA records the online block decisions in its fused forward kernel and reuses them for sparse gradient computation. We described our trainable Triton implementation on Ampere GPUs and traced the original layer choices to a training-free analysis. We compared four threshold configurations across three video benchmarks and evaluated all main benchmarks with a single inference threshold of τ=0.9\tau=0.9.
  • Reviewer’s feedback. The reviewer wrote that “my concerns are addressed” and maintained their positive rating of 3.

Reviewer wsVr (rating 2, confidence 4)

  • Concerns. The reviewer asked for a clearer account of the technical contribution, a matched trainable sparse-attention baseline, broader evaluation, and end-to-end training timings.
  • Our response. We explained how the backward kernel uses the forward mask. We compared four threshold configurations and included BLASST as an inference baseline. We also trained NSA on Qwen3.5-2B with the same SFT recipe as ToSA, evaluated a second backbone (Qwen3-VL-2B), and reported separate forward, backward, and full training-step timings.
  • Reviewer’s feedback. As of this summary, we have not received a follow-up from Reviewer wsVr on our rebuttal.

Reviewer 5hmk (rating raised to 3, confidence 3)

  • Concerns. The reviewer asked us to clarify ToSA’s contribution relative to BLASST, test a non-hybrid backbone, and measure realized block sparsity and model-level speedups.
  • Our response. ToSA records the mask as each block decision is made, avoiding a separate pass over the KV blocks. Its sparse backward kernel uses that mask to skip QK⊤QK^\top recomputation for dropped blocks. On the non-hybrid Qwen3-VL-2B, we applied ToSA to every attention layer and evaluated the original checkpoint under the same 256-frame protocol on the video benchmarks. The two training inputs yielded 74.4% and 82.8% block sparsity, and the full LVBench prefill run yielded 80.9% mean block sparsity at τ=0.9\tau=0.9.
  • Reviewer’s feedback. After reading the revision, the reviewer wrote, “I am raising my rating to 3.” In the same comment, they flagged several table entries. We updated the uniform-threshold InfiniteBench results (Table 5), labeled the original layer-wise ablation (Table 6), and re-evaluated the updated sparse-trained checkpoints (Table 7). We corrected the original Qwen3-VL-2B LongVideoBench score to 53.0 (Table 3) and specified the training threshold and mean prefill sparsity (Tables 2 and 11).

We appreciate your consideration of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.