acceptodds
Under review as a conference paper at ICLR 2027

Audio Token Attention Is Predictable Before the Language Model Runs

Abstract

A large audio language model (LALM) turns a minute of speech into - tokens and prefills every one. Image-token pruning often cuts after the language model's first layers, where image tokens draw little attention. Audio tokens draw much more attention there, and their ranking is still far from final, so audio needs a ranking before the language model runs. Surprisingly, the attention an audio token will receive across the language model is already linearly predictable from its encoder output, before the language model runs. A linear map, fitted in closed form without labels, predicts this all-layer attention ranking at on eleven of thirteen LALMs. Our method, Triage, cuts audio tokens by this prediction and, on multiple choice, cuts again at layer , correcting the prediction with the attention observed there. Triage sets its compression without labels, under two budgets that limit how far its output may differ from the model's own full-audio output. At the conservative budget, its word error rate and accuracy stay within of full audio. At the aggressive budget, Triage beats every baseline in all twelve transcription cases. On multiple choice, at - compression, it outperforms DART, the strongest baseline on average, by in mean accuracy. Because it cuts before the language model, it raises the audio that fits in Qwen2.5-Omni-3B's context window from to about minutes. At its most compressive point, Triage lets one GPU serve as many concurrent -minute streams of that model. Project page: https://audio-triage.github.io

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
audio language models, multimodal large language models, inference efficiency, token pruning, prefill reduction, KV cache, long-context inference, attention analysis, linear probing, training-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_22JSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether audio tokens can be ranked after audio encoder before the language model. It fits a linear predictor from encoder outputs to aggregated language-model attention and uses the predicted scores for token pruning. QA task additionally uses an early-layer refinement step, while ASR uses a single cut with temporal coverage. Experiments evaluate the quality–compression trade-off and the resulting efficiency and long-audio benefits.

Critical strengths
  1. The comparison of attention behavior across audio, images, and video provides useful motivation. I appreciate the progression from analyzing existing pruning strategies to designing the proposed method.
  2. The finding that a simple linear predictor can recover useful attention rankings across several model families is interesting. The controls for silence, energy, and position make this observation more convincing.
Critical weaknesses
  1. The practical latency gains are less substantial than the prefill speedups suggest. Including the encoder reduces the reported speedups to approximately 1.28–1.51×, and these measurements still exclude decoding, which is the biggest efficiency block?
  2. The quality–compression trade-off is uneven. Some WER increases are substantial in relative terms, and aggressive compression reduces Qwen 30B’s accuracy on natural long recordings from .779 to .652. The increased audio capacity therefore comes with a meaningful understanding-quality cost in some settings.
  3. The motivation is stronger than the analysis establishes. High early-layer attention and unstable rankings motivate better selection signals, but do not establish that audio tokens must be selected before the language model runs. (lines 134–152)
Questions for authors
  1. The question-independence argument assumes that the question follows the audio. Please clarify the scope of this assumption and the intended benefit of question-independent selection. (lines 162–170)
  2. The rationale for choosing .08 and .20 as the calibration budgets could be explained more clearly. (lines 321–338)
  3. The linear predictability result is intriguing. An intuitive discussion of what properties of the encoder representations might explain it would help. (172–210)
  4. HeadRouter uses uniform head weights on three models, while the Segmentwise Pruning implementation changes the attention aggregation and segmentation settings. Please clarify how these adaptations affect the interpretation of the comparisons with the original methods.
4 comments

Response to Reviewer 22JS (1/3): latency, the quality cost, and the motivation

We thank the reviewer for a thoughtful review, for valuing the cross-modality motivation and the progression from analysis to method, and for questions that made the paper more precise. Each point led to a change in the revision (blue text).

W1: practical latency. We agree. We now time the deployed pipeline end to end, decoding included (App. I, Tab. I.1). At the eight deployed points, prefill falls 1.32–3.82×, and the time to first token, encoder included, 1.10–1.54×. A single batch-1 request with 64 output tokens changes only 0.98–1.11×, because, as the reviewer says, decoding dominates it. Triage's gain lies in memory instead. Its smaller KV cache lets one A100 serve 1.5–6.9× as many concurrent streams, and the context window holds about 62 instead of 21.8 minutes of audio. §4.4 now claims faster prefill, more concurrent streams and more audio per context window, and App. I reports that a single batch-1 request gains little.

W2: the quality–compression trade-off. We agree that the aggressive budget costs real quality in some settings. The conservative budget is the one to deploy (§4.2, App. F.1), and there Triage costs a median of .03 accuracy on multiple choice and .011 WER on transcription. In relative terms the WER cost is larger (§4.2): a median of 27%, and up to 4.2 times the full-audio WER. On Qwen 30B's natural long recordings (Tab. C.8), the aggressive point does cost .128 (.779 to .652). No token-pruning baseline that keeps as many tokens does better (the closest is .005 below), and the conservative point costs .042, less than any baseline. Where full audio does not fit the context window, as for Qwen 3B on these recordings, the added capacity raises quality: Triage at 2.86× is .046 above truncating each recording to fit and .026 above voting over one-minute windows (§4.4).

W3: the motivation (lines 134–152). We agree that the double bind alone does not establish that selection must happen before the language model. It shows that the attention at layer 2, where image pruners cut, is a poor guide for audio. The case for cutting before the language model is efficiency: only a cut there saves the prefill of every layer and extends the context window. We rewrote the motivation sentence of §1, and the first contribution now states this two-step argument explicitly. The prior then supplies a ranking before the language model that is also better than the model's own attention at the cut layer (§3.2, Fig. 2).

Response to Reviewer 22JS (2/3): question order, the budgets, and linear predictability

Q1: the question-order assumption (lines 162–170). Every prompt in our evaluation places the question after the audio, and only Stage 2 sees the question (§1). The benefit of a question-free Stage 1 is reuse: one cut, made before any question arrives, serves every question about the same recording, as in multi-turn QA or a stream whose question comes later. If the question came first, g would depend on it, but Stage 1 would still run unchanged, since it reads only the encoder output. App. E.2 suggests that leaving the question out costs little even when the target depends on the question: on the attention the text rows pay the audio, the true question adds at most .043 in ρ over the mean question. Stage 2, which sees the question, then refines the cut.

Q2: why .08 and .20 (lines 321–338). App. F.1 now explains this. The two budgets mark two regimes, and we set their values empirically: .08 allows only a small drift and is the regime to deploy, while .20 is a stress test on which the selectors separate. Because the drop is measured against the model's own full-audio output, each value has a direct reading, needs no labels and can be the same for every model and task. On transcription, .08 lets about one word in twelve differ from the full-audio transcript, and .20 about one word in five. On multiple choice, .08 lets the probability of the full-audio answer fall by .08. The comparison does not rest on these two values alone: Fig. 5 compares the selectors on Qwen 3B at five compression ratios, without a budget.

Q3: why is attention linearly predictable (lines 172–210)? We tested the most natural explanation (App. E, Tab. E.4). Each first-layer key is a linear map of the RMS-normalised encoder output, so the first layer might read attention linearly. It explains only part of the signal. The first layer's linear readout predicts the all-layer target at ρ = .37 to .50, while the prior reaches .73 to .76, and the two weight vectors are nearly orthogonal. Three results point to what the prior reads instead:

  1. Content, not surface. The prior keeps 95 to 97% of its ρ when eleven waveform and lexical properties are controlled for at once, and it keeps the least silence and the fewest function words (App. E).
  2. Token norm. On Qwen 3B, tokens with a smaller RMS norm draw more attention (ρ = −.38 to −.54, App. E), although the sign of this signal differs across models (§3.2, Tab. 1).
  3. How actively the language model works on the audio. The prior works on the models whose language model moves the audio positions substantially in its first two layers, and it fails on the two Qwen-Audio models, where they barely move (§3.3).

Our reading is that the encoder output already places, along one direction, the tokens the language model will gather across its layers, and the first layer's keys capture this only in part.

Response to Reviewer 22JS (3/3): HeadRouter and Segmentwise pruning

Q4: HeadRouter's uniform head weights and Segmentwise pruning's settings. Where we could run both versions, neither adaptation weakens the baseline. HeadRouter's released head weights exist only for Qwen 3B, so the other three models weight its heads uniformly. On Qwen 3B, where both versions exist, uniform weights are .080 more accurate than the released ones on the complete multiple-choice sets at the aggressive budget. This suggests that uniform weighting does not handicap HeadRouter on the other three models. On Qwen 3B the tables follow HeadRouter's source and print the released weights, so there they show the weaker version. Triage still leads the uniform-weight version on MMSU and DREAM and ties on RACE (App. G.4).

Our Segmentwise baseline sums the encoder's self-attention over all layers and heads and keeps one token in each of K equal time segments, the configuration of the method's own illustration (its Fig. 1). The published method reads the last encoder layer and uses ten segments. We re-ran it with these settings on Qwen 3B's complete LibriSpeech test-clean and FLEURS sets (App. G.4). On test-clean, they have significantly higher WER than our version at both budgets (.263 against .168 at 2.00×). On FLEURS, where every selector keeps 85–90% of the tokens, they are at most .002 better than our version. On these two sets, the comparison is thus not biased against Segmentwise pruning.

We hope the revision addresses the reviewer's concerns, and we would be grateful if the reviewer would support our paper and reconsider the rating. We will be available until the discussion closes on November 18.

Thank you for the clarifications, which have addressed my concerns. I maintain my positive recommendation for this paper.

Reviewer_nKy2OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Triage, an audio token-pruning framework that uses a lightweight linear regression model to predict language-model attention from audio encoder representations before the language model runs. By pruning tokens before prefill, the method reduces computational cost, with empirical measurements demonstrating improvements in latency across multiple model configurations.

Critical strengths

Strength 1: Novel approach for measuring token importance

Predicting language-model attention from audio encoder representations (the output of projector layer, which becomes input to LLM) before the language model runs is an interesting approach. The use of a simple linear regression model also makes the prediction computationally inexpensive. The authors also ablated other options (audio encoder's attention score, acoustic energy, vector norm and self-information) and showed their choice is the most correlated to the language model's attention.

Critical weaknesses

Issue 1:

The paper’s main performance comparisons evaluate the proposed method and competing baselines on subsets of established evaluation benchmarks. I do not consider these results sufficient to substantiate claims of superiority under the corresponding standard benchmark protocols. If the authors introduce a custom evaluation protocol, its results should also be accompanied by evaluation on the complete standard evaluation sets.

  • Main transcription benchmarks (Table 2a, page 8) | Dataset / split | Public benchmark size | Qwen 30B evaluation (coverage) | Other 3 ALMs evaluation (coverage) | | --- | --- | --- | --- | | LibriSpeech test-clean [a] | 2,620 | 100 (3.82%) | 300 (11.45%) | | Tedlium3-Longform [b] | 11 | 7 (63.64%) | 7 (63.64%) | | FLEURS en_us/test [c] | 647 | 100 (15.46%) | 300 (46.37%) |

  • Main multiple-choice benchmarks (Table 2b, page 8) | Dataset | Public benchmark size | Paper’s evaluation (Coverage) | | --- | --- | --- | | MMSU [d] | 5,000 | 400 (8.00%) | DREAM (AudioBench TTS) [e] | 1,913 | 400 (20.91%) | | AudioMarathon-RACE [f] | 820 | 400 (48.78%) |

    For AudioMarathon-RACE, Table H.1 additionally reports an evaluation pool containing only 483 items. The 400 evaluated questions therefore represent 82.82% of that stated pool. However, the manuscript does not explain how the pool was constructed or why it contains 483 items rather than the 820 question records in the public benchmark. Also, the dataset descriptions for MMAU-mini and MMAR appear inconsistent across Tables C.5, G.1, and I.3. Table C.5 groups both benchmarks under “general audio (sound, music, speech),” whereas Table G.1 describes MMAU-mini as “sound and music” and MMAR as “sound.” Table I.3 further describes both benchmarks as “sound”.

  • Additional benchmarks (Table G.1, page 33 and Table I.3, page 43) | Dataset | Public benchmark size | Paper’s evaluation (Coverage) | | --- | --- | --- | | MMAU-mini (sound, music) [g] | 333+334=667 | 200 (29.99%) | | MMAR (sound, music, mixed sound-music) [h] | 165+206+11=382 | 200 (52.36%) |

Departure from a standard evaluation protocol requires a careful and specific justification. Relevant circumstances might include:

  1. The absence of predefined training, validation, or evaluation splits, or an established community evaluation protocol, requiring researchers to construct their own splits.
  2. The evaluation of computational efficiency, such as inference latency, for which a smaller set of representative inputs may be sufficient when appropriate measurement procedures are followed.

The manuscript does not provide a clear justification for restricting its main performance comparisons to these subsets. Although evaluating competing methods on the same subsets provides a controlled comparison, it does not resolve the absence of results under the standard benchmark protocols. Consequently, I remain unconvinced that the reported improvements generalize across the complete evaluation sets.

Issue 2: Other issues in the evaluation protocol

Use of a deprecated MMAU-mini version

Appendix G.2 (page 32) states that the experiments use the historical AudioBench copy, AudioLLMs/MMAU-mini-do-not-use, at revision db5fcbf6 [i]. The underlying data correspond to the original MMAU-mini version, preceding the corrections introduced in the official MMAU-v05.15.25 release. The revised release updates approximately 25% of the questions and answers and refines approximately 5% of the audio files (https://github.com/Sakshi113/MMAU). The AudioBench maintainer added a warning directing users to the revised benchmark on August 19, 2025 [i], more than a year before the ICLR 2027 submission period. Furthermore, established audio language models and recent studies, including Audio Flamingo 3 [k], UALM [l], and ALARM [m], report results on MMAU-v05.15.25. The manuscript does not justify using the deprecated version or establish whether its conclusions hold on the corrected benchmark. Because the revisions affect both questions and answers and the underlying audio, this omission limits confidence in the reported MMAU results.

Missing LibriSpeech test-other evaluation

When evaluating ASR performance on LibriSpeech, it is common practice to report WER on both official test splits, test-clean and test-other [j, k]. However, the manuscript evaluates only test-clean and does not explain the omission of test-other. Evaluating the complete test-other split would help establish whether the proposed pruning method preserves its advantages under more challenging speech conditions. Without this evaluation, the scope of the LibriSpeech findings remains limited.

References

Questions for authors
  • Question 1: MMAU-Mini

    Could the authors clarify the rationale for using the historical MMAU-mini version rather than the official corrected release (MMAU-v05.15.25)? Could they also provide results on the corrected benchmark?

  • Question 2: Librispeech

    Could the authors report the performance of Triage and the competing baselines on the complete test-clean and test-other splits to demonstrate whether the improvements generalize to more challenging speech conditions?

5 comments

Response to Reviewer nKy2 (1/4): complete test sets

We thank the reviewer for a detailed and precise review of our evaluation protocol, and for finding our attention-prediction approach novel. We agree with the central point of the review: a claim about a standard benchmark should rest on its standard test set. Every main comparison (Table 2 and the full grids of App. H) now uses the complete standard test set. MMAU is the corrected MMAU-v05.15.25 release, and LibriSpeech test-other is added.

Issue 1 (subsets). The submission used fixed samples to bound the cost of its many runs. All four models and every selector of Table 2 now run on the complete sets, at the same number of kept tokens:

test-cleantest-otherFLEURS en_usTEDLIUM long formMMSUDREAMAudioMarathon-RACEMMAU-v05.15.25 test-miniMMAR
submission300 (100 on 30B)—300 (100 on 30B)7 talks400400400200 (old copy)200
revision2,6202,93964711 talks5,0001,9138201,0001,000

The transcription setup also follows standard practice: decoding allows 256 new tokens per clip, and both Qwen models use the transcription prompt from Qwen's own cookbook. With it, Qwen 3B's full audio reaches .022 WER on test-clean, close to the 2.2% on its model card (App. G).

What the complete sets show. At the aggressive budget, Triage leads DART in all 24 cases of Table 2, all 12 multiple-choice cases significantly after Holm correction. Part 2 gives the revised Table 2, with the best selector of each case in bold.

Response to Reviewer nKy2 (2/4): revised Table 2, complete sets

(a) Transcription, corpus WER (lower is better). LibriSpeech test-clean 2,620 clips, FLEURS 647, TEDLIUM 11 talks. At each budget, every selector in a column keeps the same number of tokens. Bold: best token-pruning selector at the aggressive budget.

3B clean3B FLEURS3B TED30B clean30B FLEURS30B TEDVoxtral cleanVoxtral FLEURSVoxtral TEDPhi-4 cleanPhi-4 FLEURSPhi-4 TED
full audio.022.054.048.014.031.025.022.042.080.022.048.075
conservative (×)1.541.111.251.541.541.671.541.541.111.541.541.11
ours.025.054.052.019.037.041.094.072.095.060.076.070
aggressive (×)2.001.181.672.002.002.002.002.501.672.502.502.00
pooling.195.093.122.180.154.143.137.327.199.273.273.115
VAD.167.056.092.188.163.158.417.491.180.375.338.281
random.382.069.233.438.407.392.506.613.348.534.494.441
DART.126.055.078.175.141.121.433.500.266.542.448.491
FastV_N.064.055.076.214.219.212.727.683.126.399.306.274
HeadRouter_N.038.054.052.262.157.172.819.708.080.618.435.251
ours.027.054.054.030.045.046.186.210.135.197.162.175

(b) Multiple choice, accuracy (higher is better). MMSU 5,000 items, DREAM 1,913, RACE 820. † = ahead of DART in a one-sided exact McNemar test, Holm-corrected over the 12 aggressive cases. Bold: best selector at the aggressive budget. Ours (prior only) is one bin-coverage cut with the prior, before the language model and without the question.

3B MMSU3B DREAM3B RACE30B MMSU30B DREAM30B RACEVox. MMSUVox. DREAMVox. RACEPhi-4 MMSUPhi-4 DREAMPhi-4 RACE
full audio.606.874.822.713.953.901.552.874.772.539.819.706
conservative (×)4.005.005.002.222.862.221.541.821.545.001.821.82
ours.581.841.820.684.891.862.537.819.740.459.789.682
aggressive (×)10.0010.0010.004.005.002.865.002.862.8610.002.864.00
pooling.406.435.560.536.541.750.410.737.680.398.718.599
VAD.477.514.600.598.648.767.436.662.641.428.658.582
random.436.474.598.544.554.734.397.589.620.401.600.581
DART.460.579.646.547.696.818.407.668.620.404.620.579
SpeechPrune.418.467.562.468.569.662.381.608.546.396.530.528
FastV_N.485.643.661.599.670.809.409.641.590.403.673.556
HeadRouter_N.496.614.638.587.624.821.425.719.634.402.663.559
ours (prior only).508.629.655.588.675.806.437.662.648.434.743.628
ours.537†.755†.751†.617†.795†.849†.438†.745†.650†.417†.708†.618†

Response to Reviewer nKy2 (3/4): the RACE pool, the descriptions, and MMAU

The RACE pool of 483. AudioMarathon-RACE has 820 questions on 237 articles. The first 100 articles (337 questions) were set aside for the prior's fit, and the submission drew its evaluation items from the remaining 483 questions. The revision explains this (App. G and the caption of Tab. H.1) and scores all 820 questions. The complete sets thus contain every item used for the prior's fit, for calibration and for method selection. Neither the fit nor the calibration reads a label. On the items that none of these used, Triage still leads DART significantly at the aggressive budget, by .081 in mean accuracy (App. F.1). RACE keeps only 40 such items, too few to read on its own.

Inconsistent MMAU/MMAR descriptions. Thank you for catching this. Tabs. G.1, H.7 and I.2 now use one description: MMAU test-mini covers sound, music and speech, and MMAR covers sound, music, speech and their mixtures. Tab. C.5 groups the two as general audio (sound, music, speech, mixtures).

Issue 2a and Q1 (MMAU version). The submission took MMAU test-mini from the AudioBench collection, the source of our DREAM data as well, and that copy predates the corrected release. We agree that the corrected release is the right benchmark. Every MMAU result in the revision, Tabs. C.5 and I.2 included, now uses the official MMAU-v05.15.25 test-mini. The table below (Tab. H.7) scores all 1,000 items of MMAU and all 1,000 of MMAR. Each model runs at its MMSU operating points (App. H.2). At the conservative budget Triage stays within .04 of full audio in all eight cases. At the aggressive budget it leads DART in all eight, three of them significantly. Where another token-pruning baseline is ahead of it, on Qwen 30B's MMAU and MMAR, Voxtral's MMAU and Phi-4's MMAR, the gap is at most .005. Our scores read the option-letter logits, which gives Qwen 30B's full audio .795 on MMAU. With free-text answers and the official evaluator, it scores 77.8%, against 77.5% on its model card.

accuracy3B MMAU3B MMAR30B MMAU30B MMARVoxtral MMAUVoxtral MMARPhi-4 MMAUPhi-4 MMAR
full audio.740.592.795.722.630.512.685.474
conservative (×)4.004.002.222.221.541.545.005.00
ours.731.585.789.692.607.484.648.442
aggressive (×)10.0010.004.004.005.005.0010.0010.00
pooling.592.441.705.589.535.386.581.392
VAD.651.491.732.611.545.400.600.423
random.614.482.708.604.521.386.577.404
DART.630.504.716.646.543.414.607.397
FastV_N.656.513.753.654.545.406.615.396
HeadRouter_N.661.513.743.646.557.421.627.394
ours.682.545.752.652.555.433.627.418
Reviewer_crTyOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the attention received by audio tokens inside a language model can be predicted before the language model runs. The authors find that a simple linear predictor on the audio encoder's output works well across a range of models. They use this observation to build Triage: for multiple-choice QA, an initial selection based on the predictor is followed by a second cut after two language-model layers, using the observed attention to refine the selection. For ASR, they use a single cut that maintains coverage of the timeline. Retention rates are calibrated against the model's own full-audio outputs, and the experiments measure task quality, prefill time, and KV cache costs.

Critical strengths
  1. The evidence for attention predictability is fairly thorough. The authors test multiple architectures and examine positional, acoustic, and lexical factors, alongside token-deletion experiments. Together, these make the finding more informative than a correlation result alone.

  2. The method is lightweight and has useful downstream validation. It keeps the encoder and language model frozen and adds a simple predictor. I also like that the evaluation includes both QA accuracy and transcription WER, with comparisons and ablations across models and compression levels. This helps connect the attention analysis to actual model behavior.

Critical weaknesses
  1. The ASR implementation raises a concern about the comparison with mid-prefill baselines. Apps. G.5 and H.1 describe FastV and HeadRouter as cutting at layer 2. In the supplied baselines/asr_full_matrix.py, their attention scores select token indices, but E(ix) gathers the original encoder embeddings and gen() starts a fresh language-model pass. This does not carry forward the hidden states computed with the full audio before the cut. It can change both quality and computation, and the two-stage ASR variants appear to follow the same route. This also matters for the evidence behind choosing a single cut for ASR. I would like to know which implementation produced the reported comparisons.

  2. I could not reconcile the ASR prior-fitting paths with the claim of one universal prior per model. App. G.3 states this protocol, and App. F.3 explicitly says the long-form results use the universal prior. However, asr_full_matrix.py uses a locally fitted w for s1_cov; the loaded universal weights are used in a separate s12_U branch. The default self-distillation path in baselines/asr_longform.py also fits a local prior, using attention from the model's own transcript tokens to audio. This changes both the fitting data and the target relative to the generic-prompt, audio-to-audio prior described in the paper. These may be experimental scripts rather than the final evaluation paths, but the correspondence needs to be made clear.

  3. The target and predictor ablations support a useful default, but I would be more cautious about the conclusion that Stage 1 can ignore the question at little cost. Table H.6 does test text-to-audio and RoPE-free targets, and App. E.1 includes a question-conditioned MLP. Still, an audio-only predictor has an information disadvantage when its target depends on the question. The aggregation also matters: per-clip standardization removes overall scale, but does not equalize the contributions of text positions that assign different amounts of attention to audio. A clearer account of these choices would help interpret the target comparison. A frozen small text encoder with a lightweight audio–question interaction head would also be a useful, relatively inexpensive comparison; it would not require training the language model. I see this as a way to test the scope of the conclusion, rather than a requirement to replace the simple method.

  4. I would like to see a comparison with speeding up the waveform before encoding, ideally while preserving pitch. I did not find this comparison in the paper or supplement. Uniform pooling and token decimation operate after the full audio has already been encoded, so they do not test whether simple input-level compression could achieve a similar quality–cost trade-off while also saving encoder computation.

Questions for authors
  1. Which code path produced the FastV, HeadRouter, and two-stage ASR results? Does generation continue from the retained hidden states at the cut layer, or restart from selected encoder embeddings? If the supplied restart path was used, how do the comparisons change with continuation from the cut layer?

  2. Could you identify the prior checkpoint, attention target, fitting data, and evaluation command used for each ASR table? In particular, how do the local fitting paths above relate to the universal-prior results in the paper?

  3. Could you provide the sample IDs used for fitting, method selection, calibration, and evaluation, including talk IDs for long-form ASR, and identify the evaluations independent of all these choices? Which MCQ calibration signal actually selected the main operating points: answer-probability drop or answer disagreement?

  4. Which implementation produced the efficiency results in App. I? The paper describes recomputing attention rows alongside SDPA, but early_cut_efficiency.py uses a dummy keep set and omits the predictor and attention extraction. Similarly, batch_throughput.py uses a zero-valued KV cache with a uniform per-layer length chosen to match the total cache bytes, and measures a single decode step. Which reported numbers use these approximations, and which include the actual selection overhead and cache produced by Triage?

4 comments

Response to Reviewer crTy (1/4): continuation from the cut layer

We thank the reviewer for a careful review, which included reading the released code. Every question below led to a change in the revision (blue text).

W1 and Q1: which code path, and how do the comparisons change with continuation? The reviewer is right. In the submission, FastV_N and HeadRouter_N scored the tokens at layer 2 and then restarted the language model on the selected encoder outputs (gen(E(ix)) on transcription, a fresh pass on multiple choice). The two-stage transcription variants followed the same route. In the revision, every complete-set run continues from the layer-2 hidden states: FastV_N, HeadRouter_N, and Triage's Stage 2 on both tasks. From the cut layer on, forward pre-hooks pass on only the kept positions, and the cache drops the removed tokens. With every token kept, the continued decode reproduces the full-audio decode exactly (App. G.4).

Continuation mostly helps both baselines on the Qwen models and mostly hurts them on Voxtral and Phi-4. The table repeats Tab. G.2: each entry is a mean at the aggressive budget over the cases where both versions ran, and Triage leads both versions of each baseline in every row.

Task, budgetBaselinecasesrestartedcontinuedTriage
MCQ, aggressive (acc.)FastV_N12.580.595.657
HeadRouter_N12.588.598.657
MMAU and MMAR, aggressive (acc.)FastV_N8.563.567.583
HeadRouter_N8.561.570.583
transcription, aggressive (WER)FastV_N13.360.320.135
HeadRouter_N13.319.359.135

Against the continued versions at the aggressive budget, Triage is significantly ahead of FastV_N in all 12 multiple-choice cases of Table 2b, and of HeadRouter_N in 11, tying in the other one. On transcription, the continued baselines have lower WER only on TEDLIUM: HeadRouter_N on Qwen 3B (by .002) and Voxtral, and FastV_N on Voxtral. HeadRouter_N also ties with Triage on Qwen 3B FLEURS.

The single cut for transcription. As the reviewer anticipated, continuation also changes the evidence behind the single cut. Continued, both stages together have significantly lower WER than bin coverage alone in 13 of the 16 aggressive transcription cases and tie in the other 3. At the conservative budget they trail only on Phi-4 test-clean and FLEURS, by at most .006. The revision therefore runs both stages on both tasks (App. F). We made this choice on the complete test sets, so Tabs. H.4, H.5 and H.8 also print bin coverage alone.

Response to Reviewer crTy (2/4): priors, sample IDs, calibration signal

W2 and Q2: prior checkpoint, target, fitting data and command for each table. In the revision every complete-set run loads its model's released prior file and fits nothing. These runs use the scripts in experiments/overlay/ and experiments/overlay_xarch/, where the local-fit branches the reviewer found, the locally fitted w of s1_cov and the self-distillation default of asr_longform.py, run only when no prior is given. No revised table uses these branches. The release README gives the command behind every revised table, each passing the released prior (--prior_npy, or --use_prior in the Voxtral and Phi-4 long-form scripts).

Each model's prior is in the supplement's priors/ folder, and App. G.3 gives its target and fit data. The target is g of §3.2: the attention each audio token receives from the audio positions, summed over all layers and heads under a generic prompt and standardised per clip. The fit data are unlabelled clips of MMSU, DREAM and RACE, and for the two Qwen priors also of LibriSpeech test-clean. The fit never sees a transcript or an answer. Refitting on these clips reproduces the Qwen 3B, Voxtral and Phi-4 priors byte for byte, and the Qwen 30B prior to a correlation of .9995. Every run behind Table 2 and the grids of App. H uses this one universal prior per model. So does every calibration grid except Qwen's transcription grids, which score candidates with a prior fitted on each benchmark's own unlabelled clips (App. F.1). App. G.3 lists these and the three other studies that fit their own prior by design.

Q3: sample IDs and independent evaluations. The supplement lists the IDs of every prior fit, development grid, calibration draw and complete test set, with the TEDLIUM talk IDs (experiments/sample_ids/README.md). The evaluations independent of all these choices are MMAU and MMAR, the natural long recordings of App. C.6, and, on the core benchmarks, the items that no fit, grid or draw used (App. F.1, G.3). The complete test sets informed one choice, running both stages on transcription (Part 1).

Q3: which MCQ calibration signal selected the operating points? The probability drop, the signal the paper defines. We have re-run those cases, so the label-free rule of App. F.1 now sets every operating point of the main results. pick_ops.py --check in the supplement checks any operating point against its budget.

Response to Reviewer crTy (3/4): efficiency with the real pipeline

Q4: which implementation produced the efficiency results? The reviewer is right on both counts. In the submission, early_cut_efficiency.py timed a loop over the language model's layers with a placeholder keep set, and batch_throughput.py timed one decoding step on a zero-valued KV cache. The revision replaces both (App. I, Tab. I.1). e2e_bench.py now times the deployed pipeline end to end on one A100, at eight deployed operating points of the two Qwen models: the encoder, the prior's scores and Stage-1 selection, the prefill with the layer-2 cut made inside the same forward pass, and decoding from the KV cache the cut leaves. Stage 2 computes only the attention rows it needs, inside the SDPA call, and --selfcheck compares them with eager attention. At the transcription points the timed shortlist is the top-K tokens instead of bin coverage, at the same sizes. batch_real.py serves batches from these real caches and checks its decoding against generate(). The two matched token for token at every point. Every number in Tab. I.1 therefore includes the selection overhead and the cache Triage produces. With these measurements, §4.4 claims faster prefill, more concurrent streams and more audio per context window, and App. I reports that a single batch-1 request gains little.

Within App. I, only the cascade timing of Tab. I.3 still uses the layer loop, as its caption states. Outside App. I, Tab. C.2 times a plain prefill over the kept tokens on synthetic audio to compare architectures, and Tab. C.10 times whole questions from the waveform to the answer letter.

Response to Reviewer crTy (4/4): question-conditioned predictor, speed-up baseline

W3: a question-conditioned Stage-1 predictor. We built the comparison the reviewer proposed (App. E.2). A frozen MiniLM embeds the question, and a rank-32 interaction head combines it with the standardised encoder output on top of the prior's ridge. It predicts the attention that the text rows (question, options and prompt tail) pay each audio token, and is tested on held-out items of Qwen 3B's complete multiple-choice sets (Tab. E.5):

Predictor (per-clip Spearman ρ)MMSUDREAMRACE
release prior (audio rows' target, as deployed).485.545.564
the prior's ridge refitted to this target.737.736.640
question-conditioned.869.812.714
with the mean question.826.807.689
with another item's question.806.798.706
attention observed at layer 2 (all audio tokens).642.524.474

It ranks best, but the question itself adds at most .043 in ρ over the mean question. Most of its lead over the released prior comes from the text rows' target and the ranking loss, and neither needs the question. Fitted to the text rows' attention, however, a prior cuts no more accurately than ours on DREAM (Tab. H.6). We agree with the reviewer on the aggregation (App. E.2): the target sums over rows, so a row that pays the audio more attention weighs more, and per-clip standardisation removes only the overall scale. As the reviewer suggested, we read this as a test of scope: leaving the question out of Stage 1 costs little in ranking, and Stage 2 brings it in.

W4: a waveform speed-up before encoding. Thank you for this suggestion (§4.3, App. D.4). We stretch each clip with WSOLA, which preserves pitch, to the factor that brings the re-encoded token count to the case's K, on the complete sets of all four models at both budgets. When compression is light, as at most conservative points, the speed-up is a useful alternative on speech. On DREAM at the conservative budget, it is significantly more accurate than Triage on Qwen 30B and Voxtral (by .026 and .022). Once the audio must play more than about 3× faster, it collapses on every model, and on Phi-4 already at 2.5×. At Qwen 3B's aggressive DREAM point, its accuracy falls from .874 with full audio to .450, while Triage keeps .755. At the aggressive budget, Triage is ahead in 17 of the 20 multiple-choice cases and behind only on Voxtral's RACE. On transcription it is ahead in 9 of the 16 cases and behind in 4 (Tab. D.5). The speed-up does shorten the encoder's input, but the stretch runs on the CPU. On transcription its time to first token is therefore 1.1 to 1.8 times as long as with full audio. It is shorter only at the more compressed multiple-choice points, where its accuracy collapses.

We hope these answers and the revision resolve the concerns behind the soundness score, and we would be grateful if the reviewer would support our paper and reconsider the rating. We will be available until the discussion closes on November 18.

Comments on the paper

4 comments

General response (2/2): where each change is

ChangeWhere in the revised PDFRaised by
Complete test sets. Table 2 and the grids of App. H on the complete standard sets, for all four models and every selector; LibriSpeech test-other added; the corrected MMAU and the complete MMAR; one consistent description of each set, including the submission's RACE pool; the transcription prompt and decoding capTab. 2; App. G.2 (Tab. G.1), G.3; App. H; Tabs. C.5, I.2nKy2
Baselines. FastV_N and HeadRouter_N continued from the cut layer, with the restarted versions kept for comparison; HeadRouter's released head weights; Segmentwise pruning's original settings§4.1, Tab. 2; App. G.4 (Tab. G.2)crTy, 22JS
Method and provenance. Both stages on transcription; each prior's target and fit data; where each operating point comes from, with the label-free budget check and the overlap with evaluation; why the budgets are .08 and .20; the sample IDs and the command behind each table§4; App. F, F.1; App. G.2, G.3; release READMEcrTy, 22JS
Efficiency. The real pipeline timed end to end, and batched serving on its own caches; claims rest on prefill, memory and serving instead of per-request latencyAbstract, §1, §4.4; App. I (Tab. I.1)crTy, 22JS
New comparisons. A pitch-preserving waveform speed-up, alone and followed by Triage; a question-conditioned Stage-1 predictor§4.3; App. D.4 (Tabs. D.5, D.6); App. E.2 (Tab. E.5)crTy
Motivation and intuition. Why the ranking must come before the language model; the scope of the question order and the benefit of a question-free Stage 1; a test of whether the first layer's key projection explains linear predictabilityAbstract, §1; App. E (Tab. E.4)22JS

General response (1/2): what the revision changes

We thank the three reviewers for careful and constructive reviews. All three reviews highlight the paper's central idea, that the language model's attention can be predicted from the encoder output before the language model runs. In the revised PDF every new or changed sentence is blue, and every rebuilt or new table carries a blue Revised or New tag in its caption.

What each reviewer asked, and what changed

  • nKy2: the complete standard test sets. Table 2 and the full grids of App. H now use the complete standard test sets for all four models and every selector. LibriSpeech test-other is added, MMAU is the corrected MMAU-v05.15.25 test-mini, and MMAR is complete.
  • crTy: the code paths, the provenance and two new comparisons. FastV_N and HeadRouter_N now continue from the layer-2 hidden states, as Triage's Stage 2 does. App. G and F.1 state each prior's provenance and where each operating point comes from. New: a pitch-preserving waveform speed-up and a question-conditioned Stage-1 predictor.
  • 22JS: practical efficiency, the quality cost and the motivation. App. I now times the real pipeline end to end, and the claims rest on prefill, memory and serving. §1 and App. F.1 now explain the motivation and the two budgets. New: a test of the first layer's key projection, and HeadRouter's and Segmentwise pruning's original settings.

What holds on the complete sets, and what changed

  • Conservative budget: accuracy falls by a median of .03 across the 12 multiple-choice cases, and by at most .080 (Phi-4 MMSU). WER stays within .04 in 11 of the 12 transcription cases, with a median cost of .011. The exception is LibriSpeech on Voxtral (.094 against .022).
  • Aggressive budget, multiple choice (2.9–10× compression): Triage leads DART by +.070 [+.063, +.076] in mean accuracy. It is ahead in all 12 cases, all of them significantly after Holm correction. No token-pruning baseline is more than .030 ahead of it in any of these cases, while each trails it by at least .12 in some case.
  • Aggressive budget, transcription: Triage's WER is below DART's in all 12 cases, by a median of .138. HeadRouter_N has lower WER on TEDLIUM for Voxtral and Qwen 3B, and so does FastV_N on Voxtral. Uniform pooling has lower WER on Voxtral LibriSpeech and Phi-4 TEDLIUM. Triage has the lowest WER in the other eight cases, shared with HeadRouter_N on Qwen 3B FLEURS.
  • Transcription design: with continuation, Triage's two stages have significantly lower WER than the submission's single bin-coverage cut in 13 of the 16 aggressive transcription cases and tie in the other 3. Triage therefore now runs both stages on both tasks. We made this choice on the complete test sets, test-other included, so Tabs. H.4, H.5 and H.8 also print the single cut.
  • Efficiency: prefill falls 1.32–3.82× at the deployed points. The gain lies in memory: one A100 serves 1.5–6.9× as many concurrent streams, and Qwen 3B's context window holds about 62 instead of 21.8 minutes of audio.

We will be available until the discussion closes on November 18 and welcome any further feedback.

General response, update (November 14): one label-free rule now sets every operating point

We have uploaded an updated revision. One label-free rule (App. F.1) now sets every operating point of the main results, for all four models. Our first revision still used gold-chosen points for Qwen multiple choice and mostly preset points for Voxtral and Phi-4 (Reviewer crTy, Q3), so we re-ran those cases. Voxtral's budgets now count only its real audio tokens, not the padding of its 30 s windows.

The rule compresses more than the earlier points did, which moves the results in both directions: Triage's lead over DART at the aggressive budget grows, and its cost against full audio at the conservative budget rises. We have edited our earlier comments so that every number matches the updated PDF, and the general response gives the new results.

We look forward to the reviewers' feedback.

Area_Chair_K4fXOpenReview ↗

Reminder: Discussion Phase Ends November 18 (11:59 PM AoE)

Hi reviewers, a quick reminder that the discussion phase ends on November 18 at 11:59 PM AoE. Please take some time to read the author responses, engage in further discussions if you have any remaining questions or concerns, and update your reviews and scores as appropriate. Thank you!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.