- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Triage Attention, a training-free sparse attention method for accelerating long-video VLM prefill. It uses head-specific importance scores to define a protected core and a larger candidate set, then applies group-level filtering only to candidates outside the core. Experiments on three backbones across Video-MME and LongVideoBench report near-dense accuracy with reduced attention computation, while a configuration sweep shows a competitive quality–cost trade-off against the evaluated ProxyAttn and XAttention settings.
Critical strengths
-
The method uses a simple selection rule that preserves a head-specific core while filtering other candidates with shared scores. It requires no retraining, making it practical to apply to pretrained VLMs.
-
Experiments on three VLMs and two complete benchmarks report near-dense accuracy with lower S+K time than the evaluated baselines. The configuration sweep supports a competitive quality–cost trade-off, and the small TTFT study shows a response-latency improvement.
Critical weaknesses
-
The paper lacks clear motivation and technical insight. It does not clearly establish which limitation of existing methods it addresses or why the proposed pipeline should offer a better quality–cost trade-off. The combination of head-specific scoring, group-level estimation, budget selection, and protection rules lacks a clear design rationale. This concern applies to the method as a whole.
-
The experiments lack important baselines and targeted ablations. Comparisons omit VLM-specific methods such as VecAttention and MMInference, as well as SpargeAttention. The existing ablations do not sufficiently explain the contribution of the combined design. Pooling width, block size, and probe size are also not ablated. These gaps limit the evidence for the method’s advantage and the justification of its design choices.
Questions for authors
- Could the authors clarify the limitations of existing methods that motivate this work and the insight behind the overall design? Why should this combination of estimators and selection rules improve the quality–cost trade-off?
- Could the authors provide comparisons with VecAttention, MMInference, and SpargeAttention under the same conditions, or clarify any applicability constraints?
- Could the authors provide ablations that clarify the contributions of the main design choices, including the effects of pooling width, block size, and probe size on accuracy and cost?
- Beyond the reported aggregate timings (S+K time), could the authors provide each attention layer’s total latency and the time breakdown of its main steps? A breakdown of total end-to-end TTFT into its main components and their time shares would also help assess the practical benefit.
7 comments
Rebuttal to Reviewer tMGM (2/6)
[W2/Q2] VecAttention, MMInference and SpargeAttention
We added all three for which code exists, plus PSA, each from its official repository. For every baseline we checked the upstream evaluation script, CLI definitions, dispatch layer and kernel against our port, and adopted the upstream VLM entry point's settings where one exists, for instance PSA's xattn_stride=8 rather than the generic class default of 16.
SpargeAttention was run from the official repository with its own autotuner at its default error targets (L1 0.07, PV 0.08), tuned per backbone, in both the official quantized kernel and a BF16 execution of the same block map. At that bound roughly three quarters of blocks are kept and a third of heads run dense, which accounts for the limited saving.
| Qwen3-VL, 540-question Video-MME subset | Correct / 540 | Attention S+K |
|---|---|---|
| Triage, default configuration | 381 | 4.43 s |
| Triage γ=.85 | 373 | 2.91 s |
| MInference, official v1 config | 366 | 4.74 s |
| SpargeAttention, official quantized kernel, L1 0.07 | 374 | 15.42 s |
| SpargeAttention, BF16 execution of the same block map | 374 | 21.42 s |
| SpargeAttention, L1 0.10 | 382 | 14.62 s |
| Dense | 380 | 21.78 s |
SpargeAttention is barely faster than dense at its own error bound, and its best accuracy costs several times what our default spends for one more question.
MInference. The shipped configuration has no VL profile for any of our backbones, so we ran the official search_pattern offline search per backbone and used the resulting per-head configs, which is the official protocol. Its Qwen3-VL result is above; across backbones on the LongVideoBench subset, with the other added baselines:
| LongVideoBench subset, correct / 321 and attention S+K | Qwen3-VL | Qwen2.5-VL | MiniCPM-V |
|---|---|---|---|
| Triage | 203 / 2.76 s | 184 / 5.09 s | 208 / 0.183 s |
| MInference, official v1 config | 197 / 4.81 s | 184 / 3.98 s | 204 / 2.36 s |
| PSA default | 202 / 4.58 s | 183 / 6.37 s | 209 / 0.289 s |
| SpargeAttention, L1 0.07 | 196 / 9.66 s | 185 / 12.32 s | 201 / 1.08 s |
| Dense | 196 / 12.92 s | 187 / 15.06 s | 206 / 0.323 s |
MInference matches us on Qwen2.5-VL at lower cost and trails on the other two. On MiniCPM-V it is an order of magnitude slower than Triage and slower than dense, because its fixed absolute per-head budget does not shrink on short sequences. Over the three backbones it trails in count at well above our cost, and it requires a per-model offline search where our backbones share one frozen configuration.
VecAttention, from the official repository including its VLM-specific three-segment dispatch, in which the pre-vision prefix and the post-vision question rows run dense and only the video span is sparse. On Qwen3-VL over the 540-question subset, at the thresholds swept in its paper:
| Threshold | Correct / 540 | Attention S+K |
|---|---|---|
| 0.8 | 350 | 1.814 s |
| 0.6 | 357 | 2.122 s |
| 0.4 | 357 | 2.751 s |
| 0.1 | 372 | 6.813 s |
| 1e-5 | 381 | 24.200 s |
VecAttention spans the widest cost range we evaluated, and its two ends differ. At its sparsest thresholds it is the cheapest point on the frontier, faster than Triage, and where accuracy can be traded for a large per-question saving it is the better choice. At the dense end this reverses: matching our default's accuracy takes its densest threshold, which costs more than dense attention itself, and the setting below that trails Triage on both axes. The two look complementary rather than competing, with VecAttention owning the cheap end.
All four added baselines will appear in the revision, in the main comparison where the protocol matches ours and in an appendix otherwise, with the settings taken from each upstream repository recorded alongside them.
MMInference we could not compare against: no implementation has been released. The project page at aka.ms/MMInference directs to github.com/microsoft/MInference, where the two commits titled "Feature(MMInference)" (99d18c9, 3c645af) change only README.md and add a figure. Searching upstream 29ef197 for the modality-aware permutation kernels and the grid, slash and A-shape machinery described in the paper finds them only in README prose, and the open release request, issue #175, is unanswered. Recent long-video sparse-attention papers citing MMInference report no experimental comparison either. We record this as an applicability constraint rather than substitute a reimplementation of our own, particularly since this round showed how far a missing VLM dispatch path can move a baseline.
(continued in 3/6)
Rebuttal to Reviewer tMGM (3/6)
[W2/Q3] Ablations of pooling width, block size and probe size
Eight new cells on Qwen3-VL over the 540-question Video-MME subset, sharing one executor and environment with a default cell run in the same campaign under the paper's default configuration (γ=.95, c=.5, u=.95, floor 16):
| Ablation | Setting | Correct / 540 | Selector | Attention total |
|---|---|---|---|---|
| Cross-head grouping | G=1 | 378 | 0.709 s | 4.245 s |
| G=2 | 376 | 0.759 s | 4.295 s | |
| G=4 (default grouping) | 381 | 0.854 s | 4.426 s | |
| G=8 | 378 | 1.044 s | 4.667 s | |
| G=32 (no grouping) | 380 | 2.275 s | 5.887 s | |
| Pooling width | 8 | 380 | 1.029 s | 4.588 s |
| 16 (default) | 381 | 0.854 s | 4.426 s | |
| Probe length | 32 | 374 | 0.810 s | 3.813 s |
| 64 | 376 | 0.821 s | 4.024 s | |
| 128 (default) | 381 | 0.854 s | 4.426 s |
No setting in the sweep improves on the default, and all eight sit within a few questions of it. Two sweeps are nonetheless informative about where cost goes. Grouping is flat in accuracy from G=1 to G=32 while selector and total time rise, so it buys scoring cost rather than accuracy, and we now say so instead of implying a quality benefit. Probe length moves both axes together and in order: a shorter probe makes the budget estimate smaller and the mask sparser, which is the trade it exists to offer. Halving the pooling width spends selector time without a matching return.
Block size we cannot move here: the executor compiles for a fixed key-block width, so changing it means rebuilding selector and executor together. Pooling widths above the default and larger probes are not blocked; we have simply not run them, and will add both to the revision.
[Q4] Per-layer latency and the TTFT breakdown
We re-ran the timing study to answer both halves of this question directly, over the same frozen videos as the measurement in the submission, on one GPU per method with nothing else resident on the card.
Boundaries, so that every row below is unambiguous. The host pass is wall clock and carries no timers inside the model, which keeps the end-to-end number free of measurement cost. The device pass is CUDA events, nested so that each decoder layer opens a span for its attention call, that span opens one for selection and one for execution, and selection opens one per stage. Parent and child are recorded with their link, so each level is checked against the level above rather than assumed to add up. Every component below is either one measured span or the difference of two measured spans.
Time to first token, split into components
| Mean seconds, 24 videos | Triage | FlexPrefill | Prism | ProxyAttn | XAttention |
|---|---|---|---|---|---|
| Time to first token | 26.50 | 28.29 | 29.57 | 30.07 | 31.60 |
| Input construction | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 |
| Host-side model overhead | 9.99 | 9.82 | 10.05 | 9.92 | 9.91 |
| Vision encoder | 3.55 | 3.55 | 3.55 | 3.55 | 3.55 |
| Prefill outside attention | 8.04 | 8.05 | 8.02 | 8.05 | 8.05 |
| Attention inside prefill | 4.88 | 6.82 | 7.89 | 8.50 | 10.03 |
| Other device work | 0.05 | 0.05 | 0.06 | 0.05 | 0.05 |
The mean is used here rather than the median so that the components add to the total exactly.
The same split as a share of each method's own time to first token
| Share of TTFT | Triage | FlexPrefill | Prism | ProxyAttn | XAttention |
|---|---|---|---|---|---|
| Input construction | 0.0% | 0.0% | 0.0% | 0.0% | 0.0% |
| Host-side model overhead | 37.7% | 34.7% | 34.0% | 33.0% | 31.4% |
| Vision encoder | 13.4% | 12.6% | 12.0% | 11.8% | 11.2% |
| Prefill outside attention | 30.3% | 28.4% | 27.1% | 26.8% | 25.5% |
| Attention inside prefill | 18.4% | 24.1% | 26.7% | 28.3% | 31.7% |
| Other device work | 0.2% | 0.2% | 0.2% | 0.2% | 0.2% |
The four rows above the attention row are the same for every method to within the run-to-run spread, which is what we would expect since none of these methods touches input handling, the vision encoder, or the non-attention part of prefill. Attention is the only row that moves, and our method leaves it the smallest in both absolute time and share. We would rather state the ceiling this implies than leave it to be inferred: most of the time to first token lies outside attention, so the end-to-end ratio is necessarily smaller than the attention-only ratio, and that is the honest measure of the practical benefit the reviewer asks about.
(continued in 4/6)
Rebuttal to Reviewer tMGM (1/6)
We thank the reviewer for the time spent on this paper. The two weaknesses are motivation and baseline coverage, and we have tried to answer both with measurements rather than argument.
On what the paper is for, which bears directly on W1: we set out to build a sparse attention method that is simple and training-free, cheap enough that the selector does not eat the kernel savings, and usable on a pretrained long-video VLM without per-model search or tuning. Our three backbones run one frozen configuration, where for example the MInference comparison below required an offline search per model. Simplicity is the most prominent feature of our method.
Our implementation and the configurations for the paper's experiments are available in an anonymous repository, https://anonymous.4open.science/r/triage-attention-C45E/, so the selection rule described below can be inspected directly rather than taken on description.
[W1/Q1] Motivation and the reasoning behind the design
Our core observation: a top-p selector and a top-k selector reach almost the same accuracy. XAttention is top-p, keeping blocks by cumulative score until a mass target is met; ProxyAttn is top-k, giving each head a budget and keeping that many blocks by a subsampled shared score. In five of the six benchmark-by-backbone cells they differ by a few tenths of a point:
They do not keep the same blocks. Run on identical queries and keys, so that only the rule differs:
The overlap never reaches half, and the density ordering flips between layers, so the two do not even agree on how many blocks a layer needs. The selections are worth seeing: https://anonymous.4open.science/r/triage-attention-C45E/docs/why-two-estimators.md colours each block by whether both estimators keep it or only one, across layers and across the heads of one layer. XAttention's exclusive blocks fall in diagonal bands, which antidiagonal scoring rewards; ProxyAttn's lie near the diagonal and in the leading columns.
This suggested a hypothesis, not a conclusion: if two rules disagree this much and both work, neither is near the smallest support that preserves the model's answers. We tested it by keeping only the blocks both accept. The intersection is markedly sparser at a modest cost in accuracy, and the cost has a clear source, since either rule can veto a block the other considered essential.
Triage keeps the sparsity and removes that failure mode: the shared score has no authority over the head-specific core and filters only the candidates outside it. The core is small, and it comes from a score already being computed:
The shared, lossy evidence is therefore unnecessary over the small high-stakes set, and better spent on the large ambiguous remainder, where no estimator is clearly right and a cheap one suffices.
The budget is per head and per row rather than global, because the blocks needed to reach a given share of mass differ several-fold between the tenth and ninetieth percentile within one backbone. A global threshold would over-serve most rows and starve the rest.
Removing each part in turn, over all Video-MME questions in one campaign under one executor:
Each part moves the count as its role predicts, and the default leads every reduced rule while costing less than several. Section 3 has been rewritten around this account.
(continued in 2/6)