- Rating
- 2
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AudioTETRIS, a training-free audio token-pruning framework that combines an acoustic prior derived from audio encoder representations with depth-adaptive attention scores from the LLM decoder. The acoustic prior assigns higher importance scores to audio tokens near phonetic boundaries. Extensive experiments across 3 audio-language model backbones, 4 downstream tasks, and multiple compression ratios demonstrate competitive performance against existing training-free methods.
Critical strengths
S1. Speech-specific acoustic prior for token pruning
I think introducing an speech-specific prior derived from audio encoder representations is a promising idea for designing better token-pruning strategies in speech-oriented LLMs. In Section 4 and Appendices A.2.3-4, the authors provide empirical evidence that the encoder-state transition score is correlated with ground-truth phonetic boundaries. In particular, the agreement between the phoneme-boundary correlation and the peak-kurtosis analysis provides useful support for selecting the audio encoder's layer where to extract the acoustic prior.
S2. Extensive experimental evaluation and ablation studies
The authors provide extensive evaluations across three audio-language model backbones, four downstream tasks, and multiple compression ratios, demonstrating competitive performance against existing training-free token-compression methods. Sections 6.1-2 investigate the choice of distance metric and the contributions of individual scoring components, while Appendix A.3.3 examines alternative pruning schedules. These experiments provide useful evidence supporting several design choices of the proposed method.
Critical weaknesses
W1. The rationale behind the proposed pruning strategy needs clarification
Q1. Significance of the crossover layer
Why is the crossover layer , where cross-modal attention mass first exceeds intra-audio attention mass, specifically selected as the later token-pruning layer? I acknowledge the overall attention trend and the schedule experiments in Appendix A.3.3, but I think the functional significance of this particular crossing needs to be explained more clearly. Does it indicate that attention-based token-importance scores become sufficiently reliable for pruning at that depth? If so, what supports the connection between this attention crossover and the reliability of individual-token rankings?
Q2. Selection of attention scores for token ranking
The manuscript does not sufficiently justify the choice of cross-modal attention over intra-audio attention for ranking audio tokens at the crossover layer. Although the observed depth-wise shift in attention mass provides a qualitative motivation, it does not establish that cross-modal attention yields more reliable individual-token importance rankings at that depth. It remains unclear how the two attention sources differ in their resulting token rankings or score distributions, and why the cross-modal scores should be preferable for token pruning. Do the two attention sources produce substantially different token rankings or score distributions? or what makes the cross-modal scores preferable for token pruning?
W2. Completeness of ASR evaluation
In ASR research, reporting WER separately on both the test-clean and test-other splits of LibriSpeech is an established convention, particularly in a paper’s main benchmark comparison [a, b]. However, the manuscript doesn't report results on test-clean of LibriSpeech. I consider this a secondary concern, but reporting the missing results during the rebuttal would provide a more complete assessment of the method’s ASR performance.
References
- [a] Radford, A., et al. (2023). “Robust Speech Recognition via Large-Scale Weak Supervision.” Proceedings of the 40th International Conference on Machine Learning, PMLR 202:28492–28518.
- [b] Ghosh, S., et al. (2025). “Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models.” Advances in Neural Information Processing Systems, 38.
Questions for authors
- Q1-Q2: Please see W1. I think clarifying these points would substantially strengthen the paper.
- Q3: Could the authors provide the result of test-clean of LibriSpeech split?
- Q4: Could the authors clarify what they mean by “stable phonetic segments” in their hypothesis (in Section 3.2, L184-185)?
4 comments
Q3:
In the original submission we deliberately reported results on test-other as it is the more challenging split, and differences between compression methods would show up more clearly there, whereas test-clean WERs are usually lower. We agree, however, that reporting both splits is standard practice in ASR and makes the evaluation more complete. We have therefore added our test-clean results to the main comparison (Table 1) and to the Phi-4-Multimodal and Audio Flamingo 3 results (Tables 6, 7 in Appendix A.3.1). Section 5.1 now describes both splits.
On test-clean, the results match what we report on test-other. On Qwen2-Audio for example, AudioTETRIS reaches 1.78 WER at compression ratio 0.36, slightly better than the uncompressed model (1.90). It stays the best method at the highest compression ratio (0.75), with 3.04 WER versus 3.32 for the strongest baseline (LTBM [1]). On Phi-4-Multimodal and Audio Flamingo 3, AudioTETRIS stays within 0.15 WER of the uncompressed model at ratio 0.51.
Changes to the revision: Table 1 in the main text, Tables 6,7 in Appendix A.3.1 and Table 8 in A.3.2
[1] J. Luo, X. Liang, and H. Hu, “Locality matters for training-free audio token compression in audio-language models,” arXiv preprint arXiv:2605.25179, 2026.
Q4:
Thank you for this question. By ‘stable phonetic segments’ we mean the steady-state portions of speech, in the sense of the steady-state vs. spectral-transition distinction in speech perception research [1]. A stable segment is the steady part of a speech sound, such as the centre of a vowel, where consecutive frames barely change. Its opposite is the short transition from one sound into the next, such as the move from a consonant into the following vowel. Using truncated-syllable identification tests, [1] showed that a short interval around the maximum spectral transition carries the most important phonetic information, and that vowel nuclei are not necessary for vowel or syllable identification. Motivated by this, we expect consecutive frames within a stable segment to be largely redundant. They receive low and would be the first candidates for pruning, while transition frames receive high and are kept.
Changes to the revision. We have clarified this definition to the Motivation paragraph of Section 3.2 (highlighted in blue in the revised manuscript).
[1] S. Furui, “On the role of spectral transition for speech perception,” J. Acoust. Soc. Am., vol. 80, no. 4, pp. 1016–1025, Oct. 1986, doi: 10.1121/1.393842.
Thank you for your response and the additional analyses. I would appreciate clarification on the following points.
- In Appendix A.2.3(ii), L1011–1012 states that importance is measured using the occlusion-based quantity , normalized by its mean within each utterance. However, the caption describes Figure 9(a) as showing token density over attention-score percentiles, while Figure 9(b) shows the cumulative share of “score.” Could the authors clarify whether and how normalized enters either panel, and whether Figure 9(b) accumulates attention scores or occlusion importance? I think the purpose of the normalization remains unclear. If it is used only for within-utterance token rankings, dividing by a positive utterance-level mean does not change those rankings.
- Table 4 shows AudioLLMs/MMAU-mini as the evaluation source (huggingface). This repository redirects to AudioLLMs/MMAU-mini-do-not-use, and its maintainer posted a warning on Aug, 2025 directing users to MMAU-v05.15.25. The MMAU authors report revising approximately 25% of the benchmark’s questions and answers in that release. Could the authors specify the exact dataset revision used? Also, do the reported results cover the full test split, including speech, sound, and music, or only a subset? Please specify the number of evaluated parts.
- From Figures 8 and 10, I understand that the two attention sources differ in how informative they are across decoder depth. I wonder what motivates selecting one source exclusively, rather than aggregating attention from both audio and input-text query rows, which seems to be a simpler option.
Q1:
Thank you for the question and the opportunity to clarify this. The reviewer is right that in the previous version, the normalised score was not used in either panel of Figure 9. This was an error from our side, we have now added the completed figure (with Figure 9(c)) to the revised manuscript. To answer the specific questions:
Panels (a) and (b) use only attention scores. Panel (a) compares each token’s percentile under cross-modal attention with its percentile under intra-audio attention. Panel (b) adds up attention scores, not occlusion importance. For each source, it shows how much of the total attention is held by the highest-scoring tokens. Neither panel uses so we have revised the text and the caption to make this clear.
The normalisation is used only in the new panel (c). We agree that dividing by the utterance mean does not change the ranking of tokens within an utterance, and we do not use it for ranking. The new panel (c) looks at the tokens where the two selections disagree and compares how important they are. These tokens come from many different utterances, and raw occlusion scores can vary a lot from one utterance to another. Dividing by the utterance mean puts all utterances on the same scale, so that a value of 1 means ‘an average token in its utterance.’ Without this step, a few utterances with large raw scores could dominate the averages. On all three models, we find that tokens kept only by cross-modal attention are more important than those kept only by intra-audio attention. On Qwen2-Audio the average importance is 1.08 versus 0.73, a ratio of 1.5. The ratio is 1.3 on Phi-4-Multimodal and 1.8 on Audio Flamingo 3. In other words, when the two sources disagree, cross-modal attention tends to keep the tokens that matter more for the model’s output. We have updated the text accordingly and you can find more details in Appendix A.2.3.
We hope this clarifies the role of each panel and of the normalisation, and we thank the reviewer again for pointing out the inconsistency.
Q2:
We thank the reviewer for checking this carefully. All MMAU results reported in the paper use the updated release MMAU-v05.15.25, available at gamma-lab-umd/MMAU-test-mini, which includes the authors’ revisions to the questions and answers. The results in our paper come only from the updated release. In the new revision of our manuscript, we have corrected the typo on the source listed in Table 4.
We evaluate on the full test-mini split, which has 1,000 samples split roughly equally across the three domains: speech, sound and music. In the new revision, in Appendix A.3.3, we also break the results down by audio domain.
Q3:
We thank the reviewer for this suggestion. Combining both sources is indeed a natural/simpler-looking alternative, and we now test it directly in the latest revision (Table 14, Appendix A.3.5). We keep the pruning layers and the acoustic prior fixed and change only the attention rows used for scoring. We compare four options: intra-audio only, cross-modal only, both sources (attention averaged over the union of audio- and text-query rows), and our layer-adaptive selection. We find that combining both sources is the weakest variant at every compression rate and on every benchmark, and it falls below either single source. On test-other WER, using 'both sources' scores 9.32 vs. 5.77 for layer-adaptive at ratio 0.51, and 17.45 (both sources) vs. 8.50 (layer adaptive) at 0.75.
We think this follows from the depth dependence the reviewer notes in Figs. 8-10. At layer 1, cross-modal attention barely tracks occlusion importance (), and using it this early raises WER (Fig. 10, ), so including it could add noise to the more informative intra-audio signal. At the reverse holds. Intra-audio attention is spread almost evenly over tokens: the top 10% of tokens hold 22% of its mass, against 47% for cross-modal attention (Fig. 9b). Mixing it in could therefore flatten the sharper cross-modal ranking. Choosing one source per layer would keep the stronger signal at each depth.
Finally, we see selecting one source as the simpler of the two options in practice. A weighted aggregation (for example) could introduce a mixing weight to tune per backbone/depth. Our method needs only , which is obtained through a label-free calibration.
We thank the reviewer for their thoughtful review and for the time spent on our paper. All changes in the revised PDF are highlighted in blue.
Q1:
Thank you for this suggestion. You are right that a crossover in total attention mass does not, on its own, show that per-token rankings are reliable at that depth. We ran two new experiments to test this directly (new Appendix A.2.3, Figures 8,10). To validate that ‘attention’ based ranking is a reliable measure, we built a reference ranking of token importance that does not use attention, by occlusion. We take the ASR task, give the model the full audio, the prompt, and the correct transcript, and compute the teacher-forced negative log-likelihood (NLL) of that transcript. This gives one number, from a single forward pass, for how well the audio supports the correct transcript. The importance of the token is the increase in NLL caused by removing it. At every decoder layer, we compute the Spearman correlation ρ between this occlusion ranking and the ranking given by cross-modal attention. More details on the experiment setup can be found in Appendix A.2.3.
We find that cross-model attention becomes the better signal around the computed crossover ℓ† on every backbone (Figure 8). In early layers, attention rankings are close to uncorrelated with occlusion importance. By the crossover layer ℓ†, the agreement has risen roughly four- to five-fold (ρ≈0.29-0.37) and reaches most of its peak value. So ℓ† marks the depth where per-token attention rankings start to track how much the model actually relies on each token, not just where total attention mass shifts. This supports our pruning decision at ℓ†.
Next, we swept the crossover layer ℓ† (which is the second pruning layer) across the decoder and scored it with both cross-modal attention and intra-audio rows. We find that ℓ† is not always the ‘best’ layer for WER, there is a trade-off involved between performance and compute. The WER in some cases improve slowly after ℓ†, but it comes at the cost of more FLOPs. We discuss more on this in Appendix A.2.3 (Figure 10)
Changes to the revision. We revised the ‘Automated Crossover Discovery’ paragraph (Section 3.3) and added both experiments as Appendix A.2.3.
Q2:
We have now added a direct comparison of the two attention sources at ℓ† (Appendix A.2.3, Figure 9). It uses the 150 calibration utterances and the occlusion-based importance measure described in our response to the previous question (the increase in the model's loss on the correct transcript when a token and its two neighbours are removed).
From our new analysis, we find that the two sources rank tokens differently and have different score distributions. Firstly, at ℓ†, the Spearman correlation between the two rankings is only 0.38 on Qwen2-Audio. If each attention source is used to keep the top half of the tokens, the two selections differ on 36% of all tokens. Secondly, we also find that the cross-modal scores are much more concentrated: the top 10% of tokens hold 47% of the total cross-modal score, against 22% for intra-audio scores. Intra-audio attention is spread almost evenly over the audio, so it separates tokens less sharply. More details on the experiment setup & results can be found in Appendix A.2.3.
Changes to the revision: We added this comparison as Appendix A.2.3 with Figure 9, and one sentence in Section 3.3 pointing to it.