- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies whether audio tokens can be ranked after audio encoder before the language model. It fits a linear predictor from encoder outputs to aggregated language-model attention and uses the predicted scores for token pruning. QA task additionally uses an early-layer refinement step, while ASR uses a single cut with temporal coverage. Experiments evaluate the quality–compression trade-off and the resulting efficiency and long-audio benefits.
Critical strengths
- The comparison of attention behavior across audio, images, and video provides useful motivation. I appreciate the progression from analyzing existing pruning strategies to designing the proposed method.
- The finding that a simple linear predictor can recover useful attention rankings across several model families is interesting. The controls for silence, energy, and position make this observation more convincing.
Critical weaknesses
- The practical latency gains are less substantial than the prefill speedups suggest. Including the encoder reduces the reported speedups to approximately 1.28–1.51×, and these measurements still exclude decoding, which is the biggest efficiency block?
- The quality–compression trade-off is uneven. Some WER increases are substantial in relative terms, and aggressive compression reduces Qwen 30B’s accuracy on natural long recordings from .779 to .652. The increased audio capacity therefore comes with a meaningful understanding-quality cost in some settings.
- The motivation is stronger than the analysis establishes. High early-layer attention and unstable rankings motivate better selection signals, but do not establish that audio tokens must be selected before the language model runs. (lines 134–152)
Questions for authors
- The question-independence argument assumes that the question follows the audio. Please clarify the scope of this assumption and the intended benefit of question-independent selection. (lines 162–170)
- The rationale for choosing .08 and .20 as the calibration budgets could be explained more clearly. (lines 321–338)
- The linear predictability result is intriguing. An intuitive discussion of what properties of the encoder representations might explain it would help. (172–210)
- HeadRouter uses uniform head weights on three models, while the Segmentwise Pruning implementation changes the attention aggregation and segmentation settings. Please clarify how these adaptations affect the interpretation of the comparisons with the original methods.
4 comments
Response to Reviewer 22JS (2/3): question order, the budgets, and linear predictability
Q1: the question-order assumption (lines 162–170). Every prompt in our evaluation places the question after the audio, and only Stage 2 sees the question (§1). The benefit of a question-free Stage 1 is reuse: one cut, made before any question arrives, serves every question about the same recording, as in multi-turn QA or a stream whose question comes later. If the question came first, g would depend on it, but Stage 1 would still run unchanged, since it reads only the encoder output. App. E.2 suggests that leaving the question out costs little even when the target depends on the question: on the attention the text rows pay the audio, the true question adds at most .043 in ρ over the mean question. Stage 2, which sees the question, then refines the cut.
Q2: why .08 and .20 (lines 321–338). App. F.1 now explains this. The two budgets mark two regimes, and we set their values empirically: .08 allows only a small drift and is the regime to deploy, while .20 is a stress test on which the selectors separate. Because the drop is measured against the model's own full-audio output, each value has a direct reading, needs no labels and can be the same for every model and task. On transcription, .08 lets about one word in twelve differ from the full-audio transcript, and .20 about one word in five. On multiple choice, .08 lets the probability of the full-audio answer fall by .08. The comparison does not rest on these two values alone: Fig. 5 compares the selectors on Qwen 3B at five compression ratios, without a budget.
Q3: why is attention linearly predictable (lines 172–210)? We tested the most natural explanation (App. E, Tab. E.4). Each first-layer key is a linear map of the RMS-normalised encoder output, so the first layer might read attention linearly. It explains only part of the signal. The first layer's linear readout predicts the all-layer target at ρ = .37 to .50, while the prior reaches .73 to .76, and the two weight vectors are nearly orthogonal. Three results point to what the prior reads instead:
- Content, not surface. The prior keeps 95 to 97% of its ρ when eleven waveform and lexical properties are controlled for at once, and it keeps the least silence and the fewest function words (App. E).
- Token norm. On Qwen 3B, tokens with a smaller RMS norm draw more attention (ρ = −.38 to −.54, App. E), although the sign of this signal differs across models (§3.2, Tab. 1).
- How actively the language model works on the audio. The prior works on the models whose language model moves the audio positions substantially in its first two layers, and it fails on the two Qwen-Audio models, where they barely move (§3.3).
Our reading is that the encoder output already places, along one direction, the tokens the language model will gather across its layers, and the first layer's keys capture this only in part.
Response to Reviewer 22JS (3/3): HeadRouter and Segmentwise pruning
Q4: HeadRouter's uniform head weights and Segmentwise pruning's settings. Where we could run both versions, neither adaptation weakens the baseline. HeadRouter's released head weights exist only for Qwen 3B, so the other three models weight its heads uniformly. On Qwen 3B, where both versions exist, uniform weights are .080 more accurate than the released ones on the complete multiple-choice sets at the aggressive budget. This suggests that uniform weighting does not handicap HeadRouter on the other three models. On Qwen 3B the tables follow HeadRouter's source and print the released weights, so there they show the weaker version. Triage still leads the uniform-weight version on MMSU and DREAM and ties on RACE (App. G.4).
Our Segmentwise baseline sums the encoder's self-attention over all layers and heads and keeps one token in each of K equal time segments, the configuration of the method's own illustration (its Fig. 1). The published method reads the last encoder layer and uses ten segments. We re-ran it with these settings on Qwen 3B's complete LibriSpeech test-clean and FLEURS sets (App. G.4). On test-clean, they have significantly higher WER than our version at both budgets (.263 against .168 at 2.00×). On FLEURS, where every selector keeps 85–90% of the tokens, they are at most .002 better than our version. On these two sets, the comparison is thus not biased against Segmentwise pruning.
We hope the revision addresses the reviewer's concerns, and we would be grateful if the reviewer would support our paper and reconsider the rating. We will be available until the discussion closes on November 18.
Thank you for the clarifications, which have addressed my concerns. I maintain my positive recommendation for this paper.
Response to Reviewer 22JS (1/3): latency, the quality cost, and the motivation
We thank the reviewer for a thoughtful review, for valuing the cross-modality motivation and the progression from analysis to method, and for questions that made the paper more precise. Each point led to a change in the revision (blue text).
W1: practical latency. We agree. We now time the deployed pipeline end to end, decoding included (App. I, Tab. I.1). At the eight deployed points, prefill falls 1.32–3.82×, and the time to first token, encoder included, 1.10–1.54×. A single batch-1 request with 64 output tokens changes only 0.98–1.11×, because, as the reviewer says, decoding dominates it. Triage's gain lies in memory instead. Its smaller KV cache lets one A100 serve 1.5–6.9× as many concurrent streams, and the context window holds about 62 instead of 21.8 minutes of audio. §4.4 now claims faster prefill, more concurrent streams and more audio per context window, and App. I reports that a single batch-1 request gains little.
W2: the quality–compression trade-off. We agree that the aggressive budget costs real quality in some settings. The conservative budget is the one to deploy (§4.2, App. F.1), and there Triage costs a median of .03 accuracy on multiple choice and .011 WER on transcription. In relative terms the WER cost is larger (§4.2): a median of 27%, and up to 4.2 times the full-audio WER. On Qwen 30B's natural long recordings (Tab. C.8), the aggressive point does cost .128 (.779 to .652). No token-pruning baseline that keeps as many tokens does better (the closest is .005 below), and the conservative point costs .042, less than any baseline. Where full audio does not fit the context window, as for Qwen 3B on these recordings, the added capacity raises quality: Triage at 2.86× is .046 above truncating each recording to fit and .026 above voting over one-minute windows (§4.4).
W3: the motivation (lines 134–152). We agree that the double bind alone does not establish that selection must happen before the language model. It shows that the attention at layer 2, where image pruners cut, is a poor guide for audio. The case for cutting before the language model is efficiency: only a cut there saves the prefill of every layer and extends the context window. We rewrote the motivation sentence of §1, and the first contribution now states this two-step argument explicitly. The prior then supplies a ranking before the language model that is also better than the model's own attention at the cut layer (§3.2, Fig. 2).