acceptodds
Under review as a conference paper at ICLR 2027

Triage Attention: Core-Protected Sparse Attention For Long-Video Vision-Language Models

Abstract

Attention accounts for a substantial share of prefill computation in long-video vision-language models. Sparse attention can reduce this cost, but inappropriate block selection may discard critical information. Existing selection strategies typically apply a uniform filtering rule without explicitly distinguishing the decision requirements of a high-confidence core from those of uncertain candidates. We introduce TRIAGE ATTENTION, a training-free sparse attention method that assigns distinct roles to two estimates of block importance. A head-specific score identifies a protected core and a larger candidate set, while a group-level score filters only candidates outside the core. This asymmetric rule preserves core support while restricting the auxiliary estimator’s authority to reject blocks, enabling a favorable quality cost trade-off. Across three backbones evaluated on the complete Video-MME and LongVideoBench benchmarks, Triage achieves accuracy close to dense attention. On Video-MME with Qwen3-VL-8B, it achieves 69.56% accuracy versus 69.44% for dense attention, while providing a 4.92×speedup in attention computation. A prespecified sweep of 15 configurations further demonstrates a competitive quality–cost trade-off against the evaluated ProxyAttn and XAttention configurations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Sparse Attention, VLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tMGMOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Triage Attention, a training-free sparse attention method for accelerating long-video VLM prefill. It uses head-specific importance scores to define a protected core and a larger candidate set, then applies group-level filtering only to candidates outside the core. Experiments on three backbones across Video-MME and LongVideoBench report near-dense accuracy with reduced attention computation, while a configuration sweep shows a competitive quality–cost trade-off against the evaluated ProxyAttn and XAttention settings.

Critical strengths
  1. The method uses a simple selection rule that preserves a head-specific core while filtering other candidates with shared scores. It requires no retraining, making it practical to apply to pretrained VLMs.

  2. Experiments on three VLMs and two complete benchmarks report near-dense accuracy with lower S+K time than the evaluated baselines. The configuration sweep supports a competitive quality–cost trade-off, and the small TTFT study shows a response-latency improvement.

Critical weaknesses
  1. The paper lacks clear motivation and technical insight. It does not clearly establish which limitation of existing methods it addresses or why the proposed pipeline should offer a better quality–cost trade-off. The combination of head-specific scoring, group-level estimation, budget selection, and protection rules lacks a clear design rationale. This concern applies to the method as a whole.

  2. The experiments lack important baselines and targeted ablations. Comparisons omit VLM-specific methods such as VecAttention and MMInference, as well as SpargeAttention. The existing ablations do not sufficiently explain the contribution of the combined design. Pooling width, block size, and probe size are also not ablated. These gaps limit the evidence for the method’s advantage and the justification of its design choices.

Questions for authors
  1. Could the authors clarify the limitations of existing methods that motivate this work and the insight behind the overall design? Why should this combination of estimators and selection rules improve the quality–cost trade-off?
  2. Could the authors provide comparisons with VecAttention, MMInference, and SpargeAttention under the same conditions, or clarify any applicability constraints?
  3. Could the authors provide ablations that clarify the contributions of the main design choices, including the effects of pooling width, block size, and probe size on accuracy and cost?
  4. Beyond the reported aggregate timings (S+K time), could the authors provide each attention layer’s total latency and the time breakdown of its main steps? A breakdown of total end-to-end TTFT into its main components and their time shares would also help assess the practical benefit.
7 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer tMGM (1/6)

We thank the reviewer for the time spent on this paper. The two weaknesses are motivation and baseline coverage, and we have tried to answer both with measurements rather than argument.

On what the paper is for, which bears directly on W1: we set out to build a sparse attention method that is simple and training-free, cheap enough that the selector does not eat the kernel savings, and usable on a pretrained long-video VLM without per-model search or tuning. Our three backbones run one frozen configuration, where for example the MInference comparison below required an offline search per model. Simplicity is the most prominent feature of our method.

Our implementation and the configurations for the paper's experiments are available in an anonymous repository, https://anonymous.4open.science/r/triage-attention-C45E/, so the selection rule described below can be inspected directly rather than taken on description.

[W1/Q1] Motivation and the reasoning behind the design

Our core observation: a top-p selector and a top-k selector reach almost the same accuracy. XAttention is top-p, keeping blocks by cumulative score until a mass target is met; ProxyAttn is top-k, giving each head a budget and keeping that many blocks by a subsampled shared score. In five of the six benchmark-by-backbone cells they differ by a few tenths of a point:

Accuracy, XAttention against ProxyAttnQwen3-VLQwen2.5-VLMiniCPM-V
Video-MME69.37 / 69.4161.52 / 63.0067.81 / 68.48
LongVideoBench63.72 / 63.6557.22 / 57.0460.81 / 60.88

They do not keep the same blocks. Run on identical queries and keys, so that only the rule differs:

One video, mean over the heads of the layerLayer 0Layer 6Layer 12Layer 18Layer 24Layer 30
Kept by XAttention42.5%43.9%47.9%39.0%44.7%52.2%
Kept by ProxyAttn40.6%28.1%39.5%69.0%54.6%29.4%
Kept by both24.3%19.6%27.3%34.7%31.0%21.1%
Overlap, Jaccard0.400.370.430.470.440.35

The overlap never reaches half, and the density ordering flips between layers, so the two do not even agree on how many blocks a layer needs. The selections are worth seeing: https://anonymous.4open.science/r/triage-attention-C45E/docs/why-two-estimators.md colours each block by whether both estimators keep it or only one, across layers and across the heads of one layer. XAttention's exclusive blocks fall in diagonal bands, which antidiagonal scoring rewards; ProxyAttn's lie near the diagonal and in the leading columns.

This suggested a hypothesis, not a conclusion: if two rules disagree this much and both work, neither is near the smallest support that preserves the model's answers. We tested it by keeping only the blocks both accept. The intersection is markedly sparser at a modest cost in accuracy, and the cost has a clear source, since either rule can veto a block the other considered essential.

Triage keeps the sparsity and removes that failure mode: the shared score has no authority over the head-specific core and filters only the candidates outside it. The core is small, and it comes from a score already being computed:

Measured over three backbones, 24 videos, five layersQwen3-VLQwen2.5-VLMiniCPM-V
Blocks carrying the first half of the mass, median8185
Share of visible blocks needed for almost all of it56%67%63%
Agreement of the cheap and the expensive estimator, Jaccard90.487.990.5
Attention output error from dropping the boundary, points7.411.47.6

The shared, lossy evidence is therefore unnecessary over the small high-stakes set, and better spent on the large ambiguous remainder, where no estimator is clearly right and a cheap one suffices.

The budget is per head and per row rather than global, because the blocks needed to reach a given share of mass differ several-fold between the tenth and ninetieth percentile within one backbone. A global threshold would over-serve most rows and starve the rest.

Removing each part in turn, over all Video-MME questions in one campaign under one executor:

Which part is removedCorrect / 2,700Attention S+K
Nothing, the default rule1,8784.31 s
The boundary, leaving the core alone1,8561.93 s
Core protection, leaving the plain intersection1,8684.07 s
The head-specific core, leaving the group score1,8695.08 s
The group filter, leaving head-specific selection1,8747.41 s

Each part moves the count as its role predicts, and the default leads every reduced rule while costing less than several. Section 3 has been rewritten around this account.

(continued in 2/6)

AuthorsOpenReview ↗

Rebuttal to Reviewer tMGM (2/6)

[W2/Q2] VecAttention, MMInference and SpargeAttention

We added all three for which code exists, plus PSA, each from its official repository. For every baseline we checked the upstream evaluation script, CLI definitions, dispatch layer and kernel against our port, and adopted the upstream VLM entry point's settings where one exists, for instance PSA's xattn_stride=8 rather than the generic class default of 16.

SpargeAttention was run from the official repository with its own autotuner at its default error targets (L1 0.07, PV 0.08), tuned per backbone, in both the official quantized kernel and a BF16 execution of the same block map. At that bound roughly three quarters of blocks are kept and a third of heads run dense, which accounts for the limited saving.

Qwen3-VL, 540-question Video-MME subsetCorrect / 540Attention S+K
Triage, default configuration3814.43 s
Triage γ=.853732.91 s
MInference, official v1 config3664.74 s
SpargeAttention, official quantized kernel, L1 0.0737415.42 s
SpargeAttention, BF16 execution of the same block map37421.42 s
SpargeAttention, L1 0.1038214.62 s
Dense38021.78 s

SpargeAttention is barely faster than dense at its own error bound, and its best accuracy costs several times what our default spends for one more question.

MInference. The shipped configuration has no VL profile for any of our backbones, so we ran the official search_pattern offline search per backbone and used the resulting per-head configs, which is the official protocol. Its Qwen3-VL result is above; across backbones on the LongVideoBench subset, with the other added baselines:

LongVideoBench subset, correct / 321 and attention S+KQwen3-VLQwen2.5-VLMiniCPM-V
Triage203 / 2.76 s184 / 5.09 s208 / 0.183 s
MInference, official v1 config197 / 4.81 s184 / 3.98 s204 / 2.36 s
PSA default202 / 4.58 s183 / 6.37 s209 / 0.289 s
SpargeAttention, L1 0.07196 / 9.66 s185 / 12.32 s201 / 1.08 s
Dense196 / 12.92 s187 / 15.06 s206 / 0.323 s

MInference matches us on Qwen2.5-VL at lower cost and trails on the other two. On MiniCPM-V it is an order of magnitude slower than Triage and slower than dense, because its fixed absolute per-head budget does not shrink on short sequences. Over the three backbones it trails in count at well above our cost, and it requires a per-model offline search where our backbones share one frozen configuration.

VecAttention, from the official repository including its VLM-specific three-segment dispatch, in which the pre-vision prefix and the post-vision question rows run dense and only the video span is sparse. On Qwen3-VL over the 540-question subset, at the thresholds swept in its paper:

ThresholdCorrect / 540Attention S+K
0.83501.814 s
0.63572.122 s
0.43572.751 s
0.13726.813 s
1e-538124.200 s

VecAttention spans the widest cost range we evaluated, and its two ends differ. At its sparsest thresholds it is the cheapest point on the frontier, faster than Triage, and where accuracy can be traded for a large per-question saving it is the better choice. At the dense end this reverses: matching our default's accuracy takes its densest threshold, which costs more than dense attention itself, and the setting below that trails Triage on both axes. The two look complementary rather than competing, with VecAttention owning the cheap end.

All four added baselines will appear in the revision, in the main comparison where the protocol matches ours and in an appendix otherwise, with the settings taken from each upstream repository recorded alongside them.

MMInference we could not compare against: no implementation has been released. The project page at aka.ms/MMInference directs to github.com/microsoft/MInference, where the two commits titled "Feature(MMInference)" (99d18c9, 3c645af) change only README.md and add a figure. Searching upstream 29ef197 for the modality-aware permutation kernels and the grid, slash and A-shape machinery described in the paper finds them only in README prose, and the open release request, issue #175, is unanswered. Recent long-video sparse-attention papers citing MMInference report no experimental comparison either. We record this as an applicability constraint rather than substitute a reimplementation of our own, particularly since this round showed how far a missing VLM dispatch path can move a baseline.

(continued in 3/6)

AuthorsOpenReview ↗

Rebuttal to Reviewer tMGM (3/6)

[W2/Q3] Ablations of pooling width, block size and probe size

Eight new cells on Qwen3-VL over the 540-question Video-MME subset, sharing one executor and environment with a default cell run in the same campaign under the paper's default configuration (γ=.95, c=.5, u=.95, floor 16):

AblationSettingCorrect / 540SelectorAttention total
Cross-head groupingG=13780.709 s4.245 s
G=23760.759 s4.295 s
G=4 (default grouping)3810.854 s4.426 s
G=83781.044 s4.667 s
G=32 (no grouping)3802.275 s5.887 s
Pooling width83801.029 s4.588 s
16 (default)3810.854 s4.426 s
Probe length323740.810 s3.813 s
643760.821 s4.024 s
128 (default)3810.854 s4.426 s

No setting in the sweep improves on the default, and all eight sit within a few questions of it. Two sweeps are nonetheless informative about where cost goes. Grouping is flat in accuracy from G=1 to G=32 while selector and total time rise, so it buys scoring cost rather than accuracy, and we now say so instead of implying a quality benefit. Probe length moves both axes together and in order: a shorter probe makes the budget estimate smaller and the mask sparser, which is the trade it exists to offer. Halving the pooling width spends selector time without a matching return.

Block size we cannot move here: the executor compiles for a fixed key-block width, so changing it means rebuilding selector and executor together. Pooling widths above the default and larger probes are not blocked; we have simply not run them, and will add both to the revision.

[Q4] Per-layer latency and the TTFT breakdown

We re-ran the timing study to answer both halves of this question directly, over the same frozen videos as the measurement in the submission, on one GPU per method with nothing else resident on the card.

Boundaries, so that every row below is unambiguous. The host pass is wall clock and carries no timers inside the model, which keeps the end-to-end number free of measurement cost. The device pass is CUDA events, nested so that each decoder layer opens a span for its attention call, that span opens one for selection and one for execution, and selection opens one per stage. Parent and child are recorded with their link, so each level is checked against the level above rather than assumed to add up. Every component below is either one measured span or the difference of two measured spans.

Time to first token, split into components

Mean seconds, 24 videosTriageFlexPrefillPrismProxyAttnXAttention
Time to first token26.5028.2929.5730.0731.60
Input construction0.000.000.000.000.00
Host-side model overhead9.999.8210.059.929.91
Vision encoder3.553.553.553.553.55
Prefill outside attention8.048.058.028.058.05
Attention inside prefill4.886.827.898.5010.03
Other device work0.050.050.060.050.05

The mean is used here rather than the median so that the components add to the total exactly.

The same split as a share of each method's own time to first token

Share of TTFTTriageFlexPrefillPrismProxyAttnXAttention
Input construction0.0%0.0%0.0%0.0%0.0%
Host-side model overhead37.7%34.7%34.0%33.0%31.4%
Vision encoder13.4%12.6%12.0%11.8%11.2%
Prefill outside attention30.3%28.4%27.1%26.8%25.5%
Attention inside prefill18.4%24.1%26.7%28.3%31.7%
Other device work0.2%0.2%0.2%0.2%0.2%

The four rows above the attention row are the same for every method to within the run-to-run spread, which is what we would expect since none of these methods touches input handling, the vision encoder, or the non-attention part of prefill. Attention is the only row that moves, and our method leaves it the smallest in both absolute time and share. We would rather state the ceiling this implies than leave it to be inferred: most of the time to first token lies outside attention, so the end-to-end ratio is necessarily smaller than the attention-only ratio, and that is the honest measure of the practical benefit the reviewer asks about.

(continued in 4/6)

Reviewer_nUa6OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper describes a sparse attention mechanism, focused on video mllms, using a block-based aggregation and ranking:

  1. Q,K pooled with stride 16, and compute softmax attention over pooled Q,K
  2. sum the stride-16 attention in windows of 8, to form stride-128 block scores
  3. rank blocks by score (ranking 128-blocks of K for each 128-block of Q)

Blocks are then partitioned into two sets:

"Core": blocks accumulating to first 50% of score "Boundary": next 45% (to go up to 95% of total score)

The "core" is always used, while the "boundary" is further subselected using "group consensus", maximum-based selection in blocks formed by cross-head aggregation into groups.

The resulting system outperforms recent sparse attention models on video mllm evaluations, evaluated over 3 models and 2 datasets/tasks. Ablation experiments profile the impact of core and boundary mass fractions as well as min block threshold on speed/accuracy.

Critical strengths

Overall this is a well tuned and performant attention mechanism. It is evaluated on multiple models and datasets, demonstrating sizeable performance gains (~2x speed gain in tables 2,3) over recent methods. It's also a relatively straightforward system, based on simple pooling and block aggregation. It's the tuning and breakdown of these components that yield good performance.

Critical weaknesses

Although the paper focuses on video mllms, there is little that actually appears specific about video, either in the method or in the analysis. How does the system perform on text-only benchmarks --- does it extend to these tasks as well? And if not, what is it about this method and video that makes it perform better here? Especially considering that most of the comparison systems were primarily developed for text, this is a key question that I think needs to be addressed for acceptance.

On the same lines, it would have been good to see analysis relating attention selection/sampling choices to video properties (e.g., text vs video tokens, and scene cuts within videos, for example).

The group consensus filter that intersects with the "boundary" should be compared to using the same budget of tokens taken just from the initial block scores that determine c (and u). That is, this step could use further justification/measurement compared to just a single ranking.

In addition, I didn't see measurements/ablations on the block sizes and strides, or the cross-head grouping: How do these affect speed/acc?

Questions for authors
  • scoring operations would benefit from equations to formulate. each score could have a simple math formula or pseudocode with the sum, max etc operations. this would complement the text which only describes them in prose --- the current descriptions are probably better at identifying how blocks are formed, but I get a little lost in how they are put together in terms of sum/max/softmax ops and order reading just the descs

    • detailed eqns are in App. B, but these are a little too detailed and far away from the descs --- I think a simplified formulas showing the main named ops in the main text would help
  • ablations: While sizes of core and bdy mass fractions are analyzed, I would have liked to see ablations on the stride pool size (currently 16), block size (currently 128), and especially cross-head grouping in the boundary subsample (currently 4). In particular, what happens if using no cross-head grouping?

  • the comparison methods may have had their sparsity configurations tuned using text (not video), and that if their ratios were tuned on video data, they might see similar gains. Fig 2 shows ProxyAttn and XAttention with different settings, but I didn't see similar sweeps for the other two comparison methods.

  • since this is running on video models/datasets, it would be interesting to see breakdowns of attention or selected blocks for video tokens vs text tokens. and within video tokens, perhaps an analysis relating selection to scene cuts or other semantic differences between frames (as well as possibly spatial location)

  • what is the distrib of how many blocks are included in initially ranked 50%, 95% mass --- some max thresholds in the text but would be good to see distribs of actual numbers of blocks. same question for group consensus ranking: how does this change the ranking and what is

  • table 4 line R_95 & P_95 : also looks like a good setting --- why not use this instead of core 50 ? What does this do for other datasets or are there any disadvantages? partially addressed in table 5, but analysis still a little sparse on this

  • also table 4: would be interesting to see core fractions below 50

6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer nUa6 (1/5)

We thank the reviewer for the time spent on this paper. W1 was our priority, and we ran both a text-only evaluation and a matched comparison between text and video; every other measurement asked for is below.

Since much of the review asks whether the individual design choices earn their place, we state what the paper aims at: a sparse attention method that is simple and training-free, usable on a pretrained long-video VLM without per-model search or tuning, with one frozen configuration across the three backbones. Our implementation and the configurations for the paper's experiments are available in an anonymous repository, https://anonymous.4open.science/r/triage-attention-C45E/.

[W1] What is specific to video, and how does the method do on text?

We measured this directly. We ran Triage on 24 Video-MME videos and on 24 RULER text passages from the same models, matched token-for-token in sequence length and in prompt layout, and computed where the exact dense attention mass actually sits.

Where the dense mass sitsQwen3-VL videoQwen3-VL textDifferenceQwen2.5-VL videoQwen2.5-VL textDifference
Same ~700-token slice24.5%38.3%−13.95.9%34.5%−28.6
1 slice back13.0%7.3%+5.710.4%10.0%+0.4
2–3 slices back7.3%3.4%+3.98.2%4.5%+3.7
4–7 slices back8.3%6.0%+2.311.3%5.4%+5.9
8–15 slices back10.1%8.8%+1.315.0%8.5%+6.5
16+ slices back36.8%36.2%+0.649.1%37.0%+12.2
First slice (sink)38.8%34.9%+3.942.1%38.7%+3.3

Text attention concentrates inside the current slice, holding 38.3% and 34.5% of the mass. Video moves that share out into the preceding one to fifteen frames, which hold 38.7% and 44.9% against 25.5% and 28.4% on matched text. Beyond sixteen slices the two modalities are almost identical (36.8% against 36.2% on Qwen3-VL), so the difference between them sits almost entirely in that mid-range band. With model, sequence length and prompt layout held fixed, it is attributable to modality rather than to length or formatting.

That band is what the second stage of the method exists to recover, and we can measure it on the same videos. The head-specific core alone retains 59.7% of the mass at four to seven frames back and 45.2% at eight to fifteen; adding the group-filtered boundary takes those to 81.9% and 73.0% on Qwen3-VL, and from 70.8% and 67.1% to 90.9% and 88.2% on Qwen2.5-VL, lifting total retained mass from 82.8% to 90.8%. The two stages therefore correspond to the two components of video attention: a local-and-sink part that the cheap head-specific score already identifies reliably, which is why the design does not let the auxiliary estimator overrule it, and a mid-range cross-frame tail, thicker in video than in text, which is what the auxiliary stage is there to pick up. How much of that tail a row needs varies widely, the mass-0.95 prefix running from 39 blocks at the tenth percentile to 270 at the ninetieth on Qwen3-VL (Q6), which is why the budget is set per head rather than globally. Triage's selection follows the same shape, taking frames two to fifteen back 6 to 31 percentage points more often on video than on matched text, with question rows keeping all of their visual keys on both backbones.

On text-only performance, the method does transfer. On RULER with Llama-3.1-8B-Instruct and Qwen3-8B, 13 tasks at six lengths from 4K to 128K with 120 samples per task:

MethodLlama, 6-length avgLlama 128KLlama 128K attention timeQwen3, 6-length avgQwen3 128KQwen3 128K attention time
Full (dense)89.2374.2150.8 s78.6965.1156.1 s
XAttention87.6568.8115.6 s77.2962.4220.5 s
ProxyAttn88.7672.9314.0 s76.9865.9419.6 s
FlexPrefill87.6472.96≥10.4 s77.4664.3313.7 s
FlashPrefill85.5864.2731.5 s76.0161.0125.6 s
Prism87.5869.0615.0 s76.1263.1814.9 s
Triage88.8372.967.9 s77.8565.8111.0 s

At 128K the attention time is the lowest of the seven on both models, about 1.3× below the fastest baseline and 6.4× and 5.1× below dense. On accuracy it is the closest of the sparse methods to dense on both models, 0.40 below on Llama and 0.84 below on Qwen3-8B, and at 128K alone it is above dense on Qwen3-8B, 65.81 against 65.11. Against Prism, the one other method we re-ran on byte-identical inputs in the current environment, Triage is ahead by 1.25 on Llama and 1.73 on Qwen3-8B, and by 3.90 and 2.63 at 128K. On comparability: the other baselines' accuracies come from an earlier software stack, so gaps under a point are not meaningful, and re-measuring ProxyAttn's 128K timing in the current environment gives 13.4 and 19.3 s against 14.0 and 19.6 s before.

(continued in 2/5)

AuthorsOpenReview ↗

Rebuttal to Reviewer nUa6 (2/5)

To answer directly: the method is not video-only, and on long text it is the fastest while staying competitive on accuracy. That is consistent with the account above rather than in tension with it, since on text the mid-range band is thinner and the boundary stage has less to recover; the part of the design that video exercises hardest is the part that matters most there.

[W2/Q5] Selection by token type, scene cuts, and spatial position

We analysed 24 videos per backbone, instrumenting the production selector at every layer. Frame geometry comes from video_grid_thw; scene changes from the Hellinger distance between adjacent sampled frames' RGB histograms, mapped onto slice boundaries.

On token type: visual payload is 97.0% and 99.4% of tokens, and at 128-token block granularity the only text blocks are the one or two question blocks at the end. Those question blocks select every visual key and keep all of their dense mass. The first slice, holding the system prompt and the first frame, behaves as a sink with 39% and 42% of visual queries' mass, of which Triage keeps 95.0% and 99.6%.

On frame distance, selection falls off monotonically (final support, then core only):

Frame distanceQwen3-VLQwen2.5-VL
Same frame98.3% / 95.5%96.8% / 91.7%
Adjacent89.5% / 80.4%90.6% / 73.7%
2–375.9% / 58.8%87.6% / 68.0%
4–758.5% / 35.4%81.4% / 58.4%
8–1543.4% / 19.4%70.9% / 45.3%
16+22.5% / 9.3%38.0% / 18.1%

The two columns also show the division of work, the measurement W1 above draws on: the core covers the same frame, the adjacent frame and the sink, and the boundary brings back mid-range context four to fifteen slices away. Overall retained mass goes from 82.8% and 78.1% with the core alone to 90.8% and 91.6%, at a visual block density of 33.3% and 50.5%.

Scene cuts and spatial position are not where the structure lies, for the model as much as for the method. Of 4,496 slice boundaries, 23% are scene changes at a cut threshold of 0.5, and selection of the previous slice across a cut is 2 to 3 points lower than within a scene on Qwen3-VL and indistinguishable on Qwen2.5-VL. The dense attention behaves the same way (17.2% of mass on the previous slice against 13.4%), so there is nothing at a cut for any selector to exploit. Frame similarity as a graded quantity does show a weak relationship, the selection rate for the previous slice tracking the cosine similarity of layer-0 value vectors at Spearman 0.2 on Qwen3-VL and around zero on Qwen2.5-VL. Key blocks in the same vertical band as the query are not preferred across frames (88.9% against 90.5% on Qwen3-VL, no difference on Qwen2.5-VL); a 128-token block spans three or four rows, so horizontal position is not resolvable here.

Across the four analyses, selection varies with temporal distance and with the sink, and not measurably with scene semantics or spatial correspondence. At a visual block density of 33.3% and 50.5%, the selected support holds 90.8% and 91.6% of the exact dense mass and all of the mass on question rows. We will add all of this to the revision as an appendix section, together with the text-versus-video control in W1, and Sections 1 and 5 will cite it.

[W3] Group consensus against the same budget taken from the initial ranking

Taken literally the comparison is one-sided, so we report a harder version of it. Filling the whole budget from the initial ranking, with no diagonal, sink or final-row keeps, leaves the renormalised attention output more than 100% away from dense, because those structural blocks carry most of the local mass. We therefore give every arm the core and the structural keeps and compare only how the remaining slots are filled.

On sampled interior query blocks across three backbones, 24 videos and five layers, we take the number of key blocks Triage itself kept for each head and query block and require every alternative to use that same number. Each alternative starts from the protected core together with the structural keeps (diagonal, sink, final rows) and fills the remaining slots by a single ranking: the initial head-specific score, the group score, or XAttention scoring. We then compute exact dense attention on sampled rows and report, for each support, the share of the dense attention probability it covers and the relative change in the attention output once attention is restricted to it and renormalised.

Support, at Triage's own per-row block countQwen3-VLQwen2.5-VLMiniCPM-V
Triage91.0 / 10.191.1 / 10.187.3 / 10.1
core + head-specific ranking fill91.3 / 9.591.4 / 9.987.2 / 10.3
core + group-score fill90.6 / 10.590.3 / 10.687.6 / 9.6
core + XAttention fill91.3 / 9.591.4 / 9.987.2 / 10.3
core and structural keeps only83.0 / 17.480.3 / 21.578.5 / 17.7

(continued in 3/5)

AuthorsOpenReview ↗

Rebuttal to Reviewer nUa6 (3/5)

Each cell is retained dense mass % / attention-output relative error %, so higher is better in the first number and lower in the second. The first four rows hold the same number of blocks per row and differ only in which blocks fill the slots outside the core. The last row is not budget-matched, since the core and structural keeps use fewer blocks than Triage; it is there to show what the boundary stage is worth.

Three things in the table. The three fills sit within half a point of each other on both measures: against the head-specific fill the gap runs against us on the two Qwen backbones, by 0.5 and 0.2 points of output error and consistently so across videos, and in our favour by 0.3 on MiniCPM-V, while against the group-score fill it runs the other way, 0.4 and 0.6 in our favour and 0.5 against. We would not order the three rankings by these numbers. The XAttention fill matches the head-specific fill to the decimal on all three backbones, so an estimator costing roughly 16× the arithmetic produces the same boundary ordering as the cheap pooled score. And the core on its own is 7 to 11 points worse on output error than any of the filled supports, which is also what the full benchmark shows: dropping the boundary takes Video-MME from 1,878 to 1,856.

What the fixed-budget setting cannot show is the part the stage is there for, because the budget it holds fixed is the quantity the group score produces. Removing the stage and letting the head-specific score set its own support gives the head-specific-only configurations, and on all 2,700 Video-MME questions those answer 1,873 at 4.55 s and 1,874 at 7.41 s against 1,878 at 4.31 s for the default. The stage earns its place by deciding how many boundary blocks each row keeps, accepting 45 to 57% of them, and by keeping the scoring cheap (see W4). The paper now claims budget allocation and cost rather than ranking quality, and the revision will carry this comparison in full.

[W4/Q3] Block size, stride, and cross-head grouping

On Qwen3-VL over the 540-question Video-MME subset, sharing one executor and environment with a default cell run in the same campaign under the paper's default configuration (γ=.95, c=.5, u=.95, floor 16):

AblationSettingCorrect / 540SelectorAttention total
Cross-head groupingG=13760.709 s4.245 s
G=23780.759 s4.295 s
G=4 (default grouping)3810.854 s4.426 s
G=83781.044 s4.667 s
G=32 (no cross-head grouping)3802.275 s5.887 s

To answer the specific question about using no cross-head grouping: accuracy comes out slightly below the campaign's default cell and both the selector and the total attention time rise, as the table shows. For scale, an independent rerun of the default configuration on this subset lands about as far from the cell measured here as the ends of the sweep do from each other, so the sweep from G=1 to G=32 sits inside the spread one configuration shows between runs. What the sweep separates is cost rather than accuracy: kernel time barely moves across it, so the grouping is acting on scoring cost. The grouping of 4 is inherited from the proxy mechanism the boundary stage adapts; we did not re-tune it, and the sweep gives no accuracy reason to. Block size is compiled by block-sparse executor for a fixed key-block width, we will provide this ablation as soon as possible!

[Q1/Q2] Equations in the main text

We agree, and this is a clear improvement. We are adding to Section 3 a short set of named equations for the three scores (the head-specific pooled score, the group score, and the cumulative-mass thresholds), together with a nine-line Algorithm 1 that makes the order of the sum, max and softmax operations explicit, keeping the detailed derivations in Appendix B for reference. The exact wording will come with the revised PDF.

[Q4] Configuration sweeps for the other two comparison methods

Five configurations each for FlexPrefill and Prism on the same 540 questions and backbone, plus ProxyAttn γ=.95 as a cross-environment anchor.

ConfigurationCorrect / 540Attention S+K
FlexPrefill 0.803642.493 s
FlexPrefill 0.903784.101 s
FlexPrefill 0.953746.224 s
FlexPrefill 0.9753768.708 s
FlexPrefill 0.9937811.854 s
Prism 0.853815.572 s
Prism 0.903826.639 s
Prism 0.933817.569 s
Prism 0.953798.436 s
Prism 0.9838010.460 s
ProxyAttn 0.95 (anchor)3757.697 s

(continued in 4/5)

Reviewer_sAQ8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Triage Attention is a training-free block-sparse prefill attention method for long-video VLMs. Head-specific pooled Q/K scores construct a protected core and a larger candidate set. Group-level scores and a per-head budget filter only candidates outside the core. Original Q/K/V are used to execute attention over the retained support. All token positions remain present, and decoding is unchanged. Evaluations on complete Video-MME and LongVideoBench sets across three backbones report near-dense accuracy with lower selector-plus-kernel time. A separate 24-video experiment reports a 1.78× median host-TTFT speedup over dense attention.

Critical strengths
  1. The asymmetric selection rule is simple and clearly motivated: a coarse shared estimator cannot remove support that a head-specific estimator identifies as core.

  2. Full evaluations across three backbones and two benchmarks, together with a quality–cost sweep, support the practical promise of the complete method. The paper distinguishes attention-module timing from end-to-end TTFT and reports limitations of natural-budget ablations, fixed-budget diagnostics, and measurement provenance.

Critical weaknesses
  1. The independent benefit and cross-model stability of explicit core protection remain uncertain. In Table 4, protection improves accuracy while increasing cost. The fixed-budget diagnostic does not show a clear advantage, and some transfer ablations slightly favor alternatives without explicit protection. Larger matched-budget or matched-cost comparisons with paired uncertainty are needed to distinguish the rule from simply retaining more support.

  2. End-to-end gains require more reliable measurement. The TTFT experiment covers 24 videos, measured once without warm-up, and does not establish repeated-run variability. Cold-start and steady-state latency, randomized method order, repeated paired measurements, and a common execution environment for key ablations would strengthen the claims. The different correctness counts in the historical default run and later rerun also require explanation. Quality evaluation is limited to multiple-choice QA. Because the method modifies target-model prefill attention, unchanged decoding does not guarantee lossless outputs. Detailed or dense captioning and open-ended cross-event reasoning should assess event coverage, factual accuracy, temporal ordering, and hallucinations. The core-agreement motivation should also be validated beyond a single video across multiple inputs and backbones.

Questions for authors
  1. Does core protection outperform head-specific ranking, group ranking, floor-only selection, and rank fusion under matched support or measured cost across backbones? How should the transfer ablations that favor alternatives be interpreted?

  2. Can the TTFT advantage over ProxyAttn be reproduced with repeated, randomized paired measurements, separately for cold-start and steady-state execution? What explains the 377/540 versus 381/540 default-run difference?

  3. How does sparse prefill affect detailed generation and distant-event reasoning? Can the authors extend core-agreement and omission-risk analyses to multiple videos, layers, and backbones?

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer sAQ8 (1/3)

We thank the reviewer for a careful review. The three questions pointed at exactly the experiments we were missing, and we were able to run all of them.

It may help to say what we were trying to build, since it shapes several of the answers below. We wanted a sparse attention method that is simple and training-free, and that can be dropped into an existing long-video VLM without per-model tuning: the three backbones in the paper run one frozen configuration. The contribution we are claiming is a selection rule that is cheap to run and safe to leave at its defaults, which is also why, where a difference turns out to be too small to attribute to the rule, we say so below rather than argue for it.

Our implementation and the configurations for the paper's experiments are available in an anonymous repository, https://anonymous.4open.science/r/triage-attention-C45E/, so the method itself can be inspected rather than taken on description.

[W1/Q1] Does core protection outperform the alternatives at matched support or cost?

Short answer: the default answers 1,878 of 2,700, more than every one of the nine alternatives, and it costs less than four of them.

Writing R_t for the blocks in the mass-t prefix of the head-specific score, and P for the blocks the group score accepts within its per-head budget:

  • Triage keeps R_.5 unconditionally (the protected core, with a 16-block floor) and adds the blocks of R_.95 minus R_.5 that P also accepts; γ sets the group-score budget.
  • Head-specific ranking only keeps R_t with no group filtering, so t=.5 is the core with no boundary stage.
  • Group score only keeps whatever P accepts, with no protection.
  • Direct intersection R_t ∩ P keeps only blocks both scores accept, so the group score may veto core blocks too — the configuration that isolates protection itself.
  • Floor-only protection protects just the 16-block floor instead of the mass-0.5 core.

All twelve ran as one campaign, one executor and one runtime version, over all 2,700 Video-MME questions.

PolicySettingCorrect / 2,700AccuracyAttention S+K
Triageγ=.901,87269.33%3.26 s
γ=.95 (default)1,87869.56%4.31 s
γ=.981,87369.37%6.01 s
Head-specific ranking onlyt=.5 (the core alone)1,85668.74%1.93 s
t=.851,87369.37%4.55 s
t=.951,87469.41%7.41 s
Group score onlyγ=.901,86569.07%2.97 s
γ=.951,86969.22%5.08 s
γ=.9751,87369.37%7.93 s
Direct intersection R∩Pt=.801,86068.89%2.86 s
t=.951,86869.19%4.07 s
Floor-only protection—1,87369.37%4.24 s

The default reproduces the main table (69.56% at 4.311 s against 4.302 s there), and its 1,878 is the highest of the twelve — nothing else here reaches it at any cost. Otherwise the sweep shows how little separates the configurations: eleven lie between 68.74% and 69.56%, and four quite different policies — Triage γ=.98, head-specific t=.85, group-only γ=.975 and floor-only — land on exactly 1,873.

Pairing each alternative with the most expensive Triage point that costs no more than it gives six matched comparisons, and in all six the two counts are within nine questions of each other on 2,700 — too close to tell us which rule is responsible. The other three configurations are cheaper than any Triage point, so those comparisons are not matched: at 3.26 s Triage γ=.90 answers 1,872, against 1,856 for the core alone at 1.93 s, 1,860 for R_.80∩P at 2.86 s and 1,865 for group-only γ=.90 at 2.97 s.

The one visible effect is removing the boundary, 1,878 down to 1,856. Since that configuration is also the cheapest at 1.93 s, it is a trade — 2.38 s per question saved for 22 questions given up — rather than evidence that protection wins at equal cost.

[W2/Q2] Measurement reliability, and the 377 / 381 difference

We re-measured TTFT under the protocol you described: three backbones, 24 videos, three repetitions, the model loaded once per GPU, method order randomized within each video, paired per video. We also fixed the cold-start labelling, which previously let a method's first device request be warmed by its own preceding host request; cold start is now assigned per request.

BackboneSteady host vs. denseSteady host vs. ProxyAttnSteady device vs. denseSteady device vs. ProxyAttn
Qwen3-VL1.48×, faster on 23 of 24 videos1.10×, faster on 24 of 241.74×1.16×
Qwen2.5-VL1.10×, faster on 24 of 241.04×, faster on 23 of 241.20×1.08×
MiniCPM-V1.02×1.01×1.09×1.08×

(Continued in Rebuttal to Reviewer sAQ8 (2/3))

AuthorsOpenReview ↗

Rebuttal to Reviewer sAQ8 (2/3)

The advantage over ProxyAttn reproduces. On Qwen3-VL, Triage is faster on all 24 videos in steady state, by 10% at the host level and 16% on the device; on Qwen2.5-VL it is faster on 23 of 24, by 4% and 8%. Repetitions vary by 1–3%, so the ordering is a property of the workload rather than of a single pass. On MiniCPM-V the host-level gain sits inside that variation: video decoding and preprocessing dominate time to first token on that backbone, for every method alike, and no attention change can move them. The device-side improvement there still holds, at 8%.

Cold start is governed by the same mechanics: a video's first request carries decoding and first-call kernel compilation, which are identical across methods and far larger than any attention saving, so no method separates from another there. We report it separately for that reason rather than folding it into the average.

On the single-pass figure in the submission, the re-measurement came out slightly higher rather than lower. The Qwen3-VL steady-state medians are 60.8 / 37.8 / 32.5 s for dense, ProxyAttn and Triage, a median ratio over dense of 1.87×, against 64.2 / 40.5 / 36.1 s and 1.78× originally. We also give the paired geometric mean, 1.48×, which is the lower of the two because short videos gain less.

On 377 versus 381, thank you for pressing on this. The appendix notes both values without resolving them, so we had the default configuration re-run independently: it gives 381/540, so the value in Table 4 and the sensitivity table is the correct one. We will say so in the revision rather than leave the two side by side.

The gap between those two runs is worth one more sentence, because it sets a scale. Four questions out of 540 is what the same configuration moves by between runs on this subset, which is the same size as most of the differences between methods we are asked about on it. We use that as the reason not to read much into one- to five-question gaps on 540-question comparisons, here and in our replies to the other reviewers.

[W2/Q3] Generation quality, hallucination, and core agreement beyond one video

You are right that leaving decoding unchanged does not make the output unchanged. We evaluated this on two public benchmarks.

MMBench-Video. The model answers in its own words and a judge scores each answer 0–3, so unlike multiple choice the benchmark is sensitive to what the generated text actually says. We use the official 64-frame configuration and the official judging pipeline with prompts and rubric unmodified, with Llama-3.1-8B-Instruct as the judge: it is on the toolkit's supported list and comes from a different family than the model under test. We also ran the same dense computation through a different attention kernel, as a reference for how much of any gap is just arithmetic.

Dimension groupDenseDense, other kernelTriageProxyAttnXAttention
Overall2.0362.0342.0412.0312.031
Perception2.0272.0322.0312.0332.034
Reasoning2.0562.0292.0432.0242.000
Coarse perception2.1062.1022.0912.1082.105
Fine-grained, single-frame2.0582.0682.0602.0592.068
Fine-grained, cross-frame1.8611.8671.8921.8781.890
Hallucination1.7871.7331.8811.7901.717
Logic reasoning2.1942.2022.1382.2072.110
Attribute reasoning2.1642.1152.1192.1192.077
Relation reasoning2.0872.0562.1502.0762.050
Common-sense reasoning2.0762.0642.0772.0262.086
Temporal reasoning1.8901.8641.8771.8481.843

Triage scores 2.041 overall, the highest of the five systems. What the result says is that sparse prefill does not degrade free-form output here: Triage sits 0.002 above dense, while the kernel swap alone moves the score 0.004 in the other direction, so the effect of sparsifying is smaller than the arithmetic. Across the eleven dimension groups Triage stays within 0.07 of dense with no pattern to it, and on the hallucination dimension, 62 questions, it scores above both dense and the kernel reference. One caveat: 4–5% of responses per system could not be parsed into a rating and are dropped by the official rule, at comparable rates across the five systems.

VideoHallucer: 976 paired basic and hallucinated yes–no items in five categories, scored by the official string-matching rule with no judge model involved.

MethodObject-relationTemporalSemantic detailExternal factualExternal non-factualOverallHallucinated itemsYes-rate
Dense65.059.769.540.574.561.876.152.9%
Triage65.059.769.540.573.061.576.352.4%
ProxyAttn65.059.769.541.073.561.776.152.8%
XAttention65.059.769.539.072.561.175.752.8%

(Continued in 2/3)

AuthorsOpenReview ↗

Rebuttal to Reviewer sAQ8 (3/3)

Every sparse method lands within 0.7 points of dense, Triage at 0.3 below. Accuracy on the hallucinated items (76.3 against 76.1) and the rate of answering yes (52.4% against 52.9%) are unchanged, so we see no sign that sparse prefill makes the model hallucinate more or shifts its yes/no habit. Restricting to the 534 pairs where sparsity actually engages — both prompts at 2,048 tokens or more — gives the same ordering: 63.4, 63.1, 63.3, 62.3. The benchmark has limited resolution between sparse methods, with even XAttention 0.7 below dense.

We also ran a finer in-house diagnostic (60 videos described at 512 tokens, plus 155 open-ended cross-event questions, judged by a reasoning-mode LLM and checked by hand on 24 clear cases, of which 22 agreed). It points the same way, with unsupported statements, event coverage and ordering all sitting at the level of the kernel reference. We report it as supplementary and let the two public benchmarks carry the conclusion.

On core agreement, you were right that one video is not enough. We extended it to three backbones, 24 videos each, five layers:

SourceVideosJaccardXAttention blocks covered by poolingPooling blocks covered by XAttention
Paper, single video189.292.995.6
Qwen3-VL2390.493.995.6
Qwen2.5-VL2487.993.193.3
MiniCPM-V1790.593.795.9

All 24 videos per backbone were run; the counts above are those long enough to contain internal rows under the Table 1 protocol, since one Qwen3-VL video and seven MiniCPM-V videos are shorter than 33 blocks. The numbers spread by about two and a half points across backbones, and the single video used in the paper sits at the low end of that, so if anything it understated the agreement. We also re-ran the original input in the current environment and reproduced Table 1 exactly (89.2 / 92.9 / 95.6), which also closes the provenance gap the appendix flagged.

AuthorsOpenReview ↗

Our response is posted, and we would welcome any remaining questions

Dear Reviewer,

We hope this finds you well, and we are sorry to add to your inbox during what we know is a busy period.

With less than a day left in the discussion period, we wanted to let you know that we have posted a detailed response to your review. We treated your points as a work plan: where an experiment could settle a question we ran it rather than argued it, and where a design choice needed justification we have tried to give the reasoning together with the measurement behind it. The implementation and the configuration of every experiment are available in an anonymous repository, so that each result can be reproduced directly rather than taken on our description. Its front page also carries a short screen recording of three shells racing to their first token on separate GPUs, which may be a quicker way to see the difference than reading the timing tables.

If any part of our response is unclear, or if there is something further that would help you reach a judgement, we would very much welcome the chance to address it while the discussion is still open.

Thank you again for the care you put into your review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.