acceptodds
Under review as a conference paper at ICLR 2027

AudioTETRIS: Token Elimination via Transitional Importance Signals for Efficient Audio LLMs

Abstract

Large Audio Language Models (LALMs) achieve strong performance on speech-text tasks, but standard audio encoders produce hundreds of tokens for even short audio clips. Because self-attention scales quadratically in sequence length, this inflates inference-time compute substantially. Current training-free token compression methods prune or merge tokens based solely on coarse attention or feature similarity. However, treating audio tokens as generic, isotropic features smooths over spectral transitions, destroying fine-grained phonetic transitions essential for decoding speech. In contrast, speech signals exhibit highly structured temporal patterns where information density peaks at acoustic transitions. We propose AudioTETRIS, a training-free, two-stage framework that uses the audio encoder's representation geometry as an acoustic prior for token importance before applying attention-based compression. Instead of discarding encoder structure after the encoding stage, AudioTETRIS propagates this prior directly into the LLM decoder to compute a per-token fused score. This score drives progressive top- token selection across decoder layers without requiring additional training or auxiliary models. Evaluated on Qwen2-Audio-7B, Phi-4-Multimodal, and Audio Flamingo 3, AudioTETRIS retains near-uncompressed performance on several speech tasks using only 25% of audio tokens, matches or surpasses full-token accuracy at moderate compression rates, and consistently outperforms existing training-free baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Large Audio Language Models, Audio Token Compression, Efficient Inference, Training-free

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CPe4OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AudioTETRIS, a training-free audio token-pruning framework that combines an acoustic prior derived from audio encoder representations with depth-adaptive attention scores from the LLM decoder. The acoustic prior assigns higher importance scores to audio tokens near phonetic boundaries. Extensive experiments across 3 audio-language model backbones, 4 downstream tasks, and multiple compression ratios demonstrate competitive performance against existing training-free methods.

Critical strengths

S1. Speech-specific acoustic prior for token pruning

I think introducing an speech-specific prior derived from audio encoder representations is a promising idea for designing better token-pruning strategies in speech-oriented LLMs. In Section 4 and Appendices A.2.3-4, the authors provide empirical evidence that the encoder-state transition score is correlated with ground-truth phonetic boundaries. In particular, the agreement between the phoneme-boundary correlation and the peak-kurtosis analysis provides useful support for selecting the audio encoder's layer where to extract the acoustic prior.

S2. Extensive experimental evaluation and ablation studies

The authors provide extensive evaluations across three audio-language model backbones, four downstream tasks, and multiple compression ratios, demonstrating competitive performance against existing training-free token-compression methods. Sections 6.1-2 investigate the choice of distance metric and the contributions of individual scoring components, while Appendix A.3.3 examines alternative pruning schedules. These experiments provide useful evidence supporting several design choices of the proposed method.

Critical weaknesses

W1. The rationale behind the proposed pruning strategy needs clarification

Q1. Significance of the crossover layer

Why is the crossover layer ℓ†\ell^\dagger, where cross-modal attention mass first exceeds intra-audio attention mass, specifically selected as the later token-pruning layer? I acknowledge the overall attention trend and the schedule experiments in Appendix A.3.3, but I think the functional significance of this particular crossing needs to be explained more clearly. Does it indicate that attention-based token-importance scores become sufficiently reliable for pruning at that depth? If so, what supports the connection between this attention crossover and the reliability of individual-token rankings?

Q2. Selection of attention scores for token ranking

The manuscript does not sufficiently justify the choice of cross-modal attention over intra-audio attention for ranking audio tokens at the crossover layer. Although the observed depth-wise shift in attention mass provides a qualitative motivation, it does not establish that cross-modal attention yields more reliable individual-token importance rankings at that depth. It remains unclear how the two attention sources differ in their resulting token rankings or score distributions, and why the cross-modal scores should be preferable for token pruning. Do the two attention sources produce substantially different token rankings or score distributions? or what makes the cross-modal scores preferable for token pruning?

W2. Completeness of ASR evaluation

In ASR research, reporting WER separately on both the test-clean and test-other splits of LibriSpeech is an established convention, particularly in a paper’s main benchmark comparison [a, b]. However, the manuscript doesn't report results on test-clean of LibriSpeech. I consider this a secondary concern, but reporting the missing results during the rebuttal would provide a more complete assessment of the method’s ASR performance.

References

  • [a] Radford, A., et al. (2023). “Robust Speech Recognition via Large-Scale Weak Supervision.” Proceedings of the 40th International Conference on Machine Learning, PMLR 202:28492–28518.
  • [b] Ghosh, S., et al. (2025). “Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models.” Advances in Neural Information Processing Systems, 38.
Questions for authors
  • Q1-Q2: Please see W1. I think clarifying these points would substantially strengthen the paper.
  • Q3: Could the authors provide the result of test-clean of LibriSpeech split?
  • Q4: Could the authors clarify what they mean by “stable phonetic segments” in their hypothesis (in Section 3.2, L184-185)?
4 comments

We thank the reviewer for their thoughtful review and for the time spent on our paper. All changes in the revised PDF are highlighted in blue.

Q1:

Thank you for this suggestion. You are right that a crossover in total attention mass does not, on its own, show that per-token rankings are reliable at that depth. We ran two new experiments to test this directly (new Appendix A.2.3, Figures 8,10). To validate that ‘attention’ based ranking is a reliable measure, we built a reference ranking of token importance that does not use attention, by occlusion. We take the ASR task, give the model the full audio, the prompt, and the correct transcript, and compute the teacher-forced negative log-likelihood (NLL) of that transcript. This gives one number, from a single forward pass, for how well the audio supports the correct transcript. The importance of the token is the increase in NLL caused by removing it. At every decoder layer, we compute the Spearman correlation ρ\rho between this occlusion ranking and the ranking given by cross-modal attention. More details on the experiment setup can be found in Appendix A.2.3.

Backboneρ\rho of text rows: layer 1 →\rightarrow ℓ†\ell^\dagger →\rightarrow best
Qwen2-Audio0.08→0.29→0.450.08 \rightarrow 0.29 \rightarrow 0.45
Phi-4-Multimodal0.07→0.30→0.400.07 \rightarrow 0.30 \rightarrow 0.40
Audio Flamingo 30.07→0.37→0.440.07 \rightarrow 0.37 \rightarrow 0.44

We find that cross-model attention becomes the better signal around the computed crossover ℓ†\ell^\dagger on every backbone (Figure 8). In early layers, attention rankings are close to uncorrelated with occlusion importance. By the crossover layer ℓ†\ell^\dagger, the agreement has risen roughly four- to five-fold (ρ≈0.29\rho \approx 0.29-0.370.37) and reaches most of its peak value. So ℓ†\ell^\dagger marks the depth where per-token attention rankings start to track how much the model actually relies on each token, not just where total attention mass shifts. This supports our pruning decision at ℓ†\ell^\dagger.

Next, we swept the crossover layer ℓ†\ell^\dagger (which is the second pruning layer) across the decoder and scored it with both cross-modal attention and intra-audio rows. We find that ℓ†\ell^\dagger is not always the ‘best’ layer for WER, there is a trade-off involved between performance and compute. The WER in some cases improve slowly after ℓ†\ell^\dagger, but it comes at the cost of more FLOPs. We discuss more on this in Appendix A.2.3 (Figure 10)

Changes to the revision. We revised the ‘Automated Crossover Discovery’ paragraph (Section 3.3) and added both experiments as Appendix A.2.3.

Q2:

We have now added a direct comparison of the two attention sources at ℓ†\ell^\dagger (Appendix A.2.3, Figure 9). It uses the 150 calibration utterances and the occlusion-based importance measure described in our response to the previous question (the increase in the model's loss on the correct transcript when a token and its two neighbours are removed).

From our new analysis, we find that the two sources rank tokens differently and have different score distributions. Firstly, at ℓ†\ell^\dagger, the Spearman correlation between the two rankings is only 0.38 on Qwen2-Audio. If each attention source is used to keep the top half of the tokens, the two selections differ on 36% of all tokens. Secondly, we also find that the cross-modal scores are much more concentrated: the top 10% of tokens hold 47% of the total cross-modal score, against 22% for intra-audio scores. Intra-audio attention is spread almost evenly over the audio, so it separates tokens less sharply. More details on the experiment setup & results can be found in Appendix A.2.3.

Changes to the revision: We added this comparison as Appendix A.2.3 with Figure 9, and one sentence in Section 3.3 pointing to it.

Q3:

In the original submission we deliberately reported results on test-other as it is the more challenging split, and differences between compression methods would show up more clearly there, whereas test-clean WERs are usually lower. We agree, however, that reporting both splits is standard practice in ASR and makes the evaluation more complete. We have therefore added our test-clean results to the main comparison (Table 1) and to the Phi-4-Multimodal and Audio Flamingo 3 results (Tables 6, 7 in Appendix A.3.1). Section 5.1 now describes both splits.

On test-clean, the results match what we report on test-other. On Qwen2-Audio for example, AudioTETRIS reaches 1.78 WER at compression ratio 0.36, slightly better than the uncompressed model (1.90). It stays the best method at the highest compression ratio (0.75), with 3.04 WER versus 3.32 for the strongest baseline (LTBM [1]). On Phi-4-Multimodal and Audio Flamingo 3, AudioTETRIS stays within 0.15 WER of the uncompressed model at ratio 0.51.

Changes to the revision: Table 1 in the main text, Tables 6,7 in Appendix A.3.1 and Table 8 in A.3.2

[1] J. Luo, X. Liang, and H. Hu, “Locality matters for training-free audio token compression in audio-language models,” arXiv preprint arXiv:2605.25179, 2026.

Q4:

Thank you for this question. By ‘stable phonetic segments’ we mean the steady-state portions of speech, in the sense of the steady-state vs. spectral-transition distinction in speech perception research [1]. A stable segment is the steady part of a speech sound, such as the centre of a vowel, where consecutive frames barely change. Its opposite is the short transition from one sound into the next, such as the move from a consonant into the following vowel. Using truncated-syllable identification tests, [1] showed that a short interval around the maximum spectral transition carries the most important phonetic information, and that vowel nuclei are not necessary for vowel or syllable identification. Motivated by this, we expect consecutive frames within a stable segment to be largely redundant. They receive low wtw_t and would be the first candidates for pruning, while transition frames receive high wtw_t and are kept.

Changes to the revision. We have clarified this definition to the Motivation paragraph of Section 3.2 (highlighted in blue in the revised manuscript).

[1] S. Furui, “On the role of spectral transition for speech perception,” J. Acoust. Soc. Am., vol. 80, no. 4, pp. 1016–1025, Oct. 1986, doi: 10.1121/1.393842.

Thank you for your response and the additional analyses. I would appreciate clarification on the following points.

  • In Appendix A.2.3(ii), L1011–1012 states that importance is measured using the occlusion-based quantity gtg_t, normalized by its mean within each utterance. However, the caption describes Figure 9(a) as showing token density over attention-score percentiles, while Figure 9(b) shows the cumulative share of “score.” Could the authors clarify whether and how normalized gtg_t enters either panel, and whether Figure 9(b) accumulates attention scores or occlusion importance? I think the purpose of the normalization remains unclear. If it is used only for within-utterance token rankings, dividing by a positive utterance-level mean does not change those rankings.
  • Table 4 shows AudioLLMs/MMAU-mini as the evaluation source (huggingface). This repository redirects to AudioLLMs/MMAU-mini-do-not-use, and its maintainer posted a warning on Aug, 2025 directing users to MMAU-v05.15.25. The MMAU authors report revising approximately 25% of the benchmark’s questions and answers in that release. Could the authors specify the exact dataset revision used? Also, do the reported results cover the full test split, including speech, sound, and music, or only a subset? Please specify the number of evaluated parts.
  • From Figures 8 and 10, I understand that the two attention sources differ in how informative they are across decoder depth. I wonder what motivates selecting one source exclusively, rather than aggregating attention from both audio and input-text query rows, which seems to be a simpler option.

Q1:

Thank you for the question and the opportunity to clarify this. The reviewer is right that in the previous version, the normalised score gtg_t was not used in either panel of Figure 9. This was an error from our side, we have now added the completed figure (with Figure 9(c)) to the revised manuscript. To answer the specific questions:

Panels (a) and (b) use only attention scores. Panel (a) compares each token’s percentile under cross-modal attention with its percentile under intra-audio attention. Panel (b) adds up attention scores, not occlusion importance. For each source, it shows how much of the total attention is held by the highest-scoring tokens. Neither panel uses gtg_t so we have revised the text and the caption to make this clear.

The normalisation is used only in the new panel (c). We agree that dividing by the utterance mean does not change the ranking of tokens within an utterance, and we do not use it for ranking. The new panel (c) looks at the tokens where the two selections disagree and compares how important they are. These tokens come from many different utterances, and raw occlusion scores can vary a lot from one utterance to another. Dividing by the utterance mean puts all utterances on the same scale, so that a value of 1 means ‘an average token in its utterance.’ Without this step, a few utterances with large raw scores could dominate the averages. On all three models, we find that tokens kept only by cross-modal attention are more important than those kept only by intra-audio attention. On Qwen2-Audio the average importance is 1.08 versus 0.73, a ratio of 1.5. The ratio is 1.3 on Phi-4-Multimodal and 1.8 on Audio Flamingo 3. In other words, when the two sources disagree, cross-modal attention tends to keep the tokens that matter more for the model’s output. We have updated the text accordingly and you can find more details in Appendix A.2.3.

We hope this clarifies the role of each panel and of the normalisation, and we thank the reviewer again for pointing out the inconsistency.

Q2:

We thank the reviewer for checking this carefully. All MMAU results reported in the paper use the updated release MMAU-v05.15.25, available at gamma-lab-umd/MMAU-test-mini, which includes the authors’ revisions to the questions and answers. The results in our paper come only from the updated release. In the new revision of our manuscript, we have corrected the typo on the source listed in Table 4.

We evaluate on the full test-mini split, which has 1,000 samples split roughly equally across the three domains: speech, sound and music. In the new revision, in Appendix A.3.3, we also break the results down by audio domain.

Q3:

We thank the reviewer for this suggestion. Combining both sources is indeed a natural/simpler-looking alternative, and we now test it directly in the latest revision (Table 14, Appendix A.3.5). We keep the pruning layers {1,ℓ†}\{1, \ell^\dagger\} and the acoustic prior wtw_t fixed and change only the attention rows used for scoring. We compare four options: intra-audio only, cross-modal only, both sources (attention averaged over the union of audio- and text-query rows), and our layer-adaptive selection. We find that combining both sources is the weakest variant at every compression rate and on every benchmark, and it falls below either single source. On test-other WER, using 'both sources' scores 9.32 vs. 5.77 for layer-adaptive at ratio 0.51, and 17.45 (both sources) vs. 8.50 (layer adaptive) at 0.75.

We think this follows from the depth dependence the reviewer notes in Figs. 8-10. At layer 1, cross-modal attention barely tracks occlusion importance (ρ≤0.08\rho \leq 0.08), and using it this early raises WER (Fig. 10, ℓ2=3\ell_2 = 3), so including it could add noise to the more informative intra-audio signal. At ℓ†\ell^\dagger the reverse holds. Intra-audio attention is spread almost evenly over tokens: the top 10% of tokens hold 22% of its mass, against 47% for cross-modal attention (Fig. 9b). Mixing it in could therefore flatten the sharper cross-modal ranking. Choosing one source per layer would keep the stronger signal at each depth.

Finally, we see selecting one source as the simpler of the two options in practice. A weighted aggregation (for example) could introduce a mixing weight to tune per backbone/depth. Our method needs only ℓ†\ell^\dagger, which is obtained through a label-free calibration.

Reviewer_Xs28OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AudioTETRIS, a training-free method to reduce the computational cost of Audio LLMs. It combines acoustic transition scores from encoder representations with attention-based token importance to progressively prune audio tokens across LLM layers. The method is evaluated on three Audio LLMs and four downstream tasks, showing computational savings and generally better performance than the considered pruning baselines.

Critical strengths
  • Reducing the computational cost of Audio LLMs is an important problem, particularly given the large number of audio tokens typically processed by these models.
  • The method does not require retraining and combines speech-specific information with existing pruning techniques.
  • The experiments cover several models and tasks, showing meaningful computational savings and competitive performance compared with the considered baselines.
Critical weaknesses
  • I found the work somewhat incremental. Most components are already well established, although their speech-specific combination is useful. The paper should better position its contribution relative to existing token-pruning approaches. In particular, Segmentwise Pruning seems especially relevant but is neither cited nor compared against.

  • Incomplete comparison with alternative approaches: An important alternative is to directly train low-frame-rate or adaptive-frame-rate speech representations. These approaches can simplify inference by reducing the number of tokens entering the LLM, although they typically require training and downstream adaptation. The paper should discuss this trade-off more carefully. Moreover, the proposed method processes audio tokens through some LLM layers before pruning them. A comparison with simpler encoder-side pruning would help establish whether the additional complexity is justified.

  • There are some important missing details in the paper (see questions below).

Questions for authors
  • What are the actual audio-token rates before and after pruning for each model? This important missing detail should be reported in the tables. Recent speech tokenizers have shown strong results at 12.5 Hz and promising results around 6 Hz. Reporting absolute token rates would help contextualize the compression results, while accounting for the fact that pruning occurs progressively inside the LLM.

  • How does AudioTETRIS compare with pruning tokens directly before the LLM? For instance, could the authors evaluate a simple encoder-side pruning strategy using only the acoustic importance scores? This would help clarify the benefit of the additional attention-based pruning stages.

Minor presentation comments:

  • Figure 1 is somewhat misleading, as it seems to suggest that w_t is fed directly into the LLM. The figure should distinguish the audio embeddings entering the LLM from the auxiliary importance scores used for pruning.

  • Please export Figure 1 in a vector format with selectable text to improve readability and presentation quality.

5 comments

We thank the reviewer for their thoughtful review and for the time they spent on our paper. We have addressed each point below, and all changes in the revised PDF are highlighted in blue.

W1:

Thank you for this comment and for pointing us to Segmentwise Pruning [1]. We agree it is closely related. We have updated Section 2 to cite [1] and to clarify where our method sits in the related work.

We see the difference as follows. Segmentwise Top-K prunes once before the LLM (so on the encoder side, like LTBM [2] in our comparison tables). It ranks tokens by encoder attention and spreads the kept tokens evenly across fixed time segments. AudioTETRIS starts from a different assumption: in speech, information is not spread evenly over time but concentrated at phonetic transitions. This premise comes from speech perception research. Listeners rely heavily on spectral transitions (such as consonant–vowel boundaries) to identify phonemes, while steady-state portions such as sustained vowels and pauses are largely redundant [3]. We confirm this for LALM encoders in Section 4 (Figure 3), where the change between consecutive encoder states peaks at phoneme boundaries. So we keep tokens where the encoder's representation changes most, and then refine that choice inside the LLM using the task at hand.

On novelty more broadly, we agree that attention-based scoring and progressive pruning are established ideas, and we don't claim them as new. As stated in our contributions (Section 1), our contribution is to ground pruning in the acoustic structure of speech, which existing pruning methods do not use. We show that differences between consecutive encoder states track real phoneme transitions, and that the right encoder layer can be found without labels using measures like kurtosis (Section 4). We also show that this acoustic signal and LLM attention are complementary (Table 8, Appendix A.3.2).

[1] M. Gibier, R. Duroselle, P. Serrano, O. Boeffard, and J.-F. Bonastre, "Segmentwise Pruning in Audio-Language Models," in Proc. IEEE ICASSP, 2026, doi: 10.1109/ICASSP55912.2026.11463308.

[2] J. Luo, X. Liang, and H. Hu, "Locality matters for training-free audio token compression in audio-language models," arXiv:2605.25179, 2026.

[3] S. Furui, "On the role of spectral transition for speech perception," J. Acoust. Soc. Am., vol. 80, pp. 1016–1025, 1986, doi: 10.1121/1.393842.

W2:

We originally treated trained low-rate and adaptive-rate tokenizers as a separate setting from ours, because they require retraining the downstream model, while our comparisons focused on training-free methods that run on frozen checkpoints. We agree this distinction deserves explicit discussion, so we have added a dedicated paragraph to discuss the trade-off in the Related Work (Section 2) and added as a future work in Section 7. This line of work includes low-frame-rate neural codecs such as Mimi [1] and variable-rate or boundary-aligned tokenizers such as VARSTok [2], DFC [3] and DyCAST [4]. The two approaches differ mainly in where the cost lands:

  • Training cost. These tokenizers change the representation the downstream language model consumes, so that model must be trained on the new tokens. VARSTok [2] trains its TTS language model from scratch on its own tokens, and DFC [3] and DyCAST [4] each introduce a new codec. As the reviewer implied, for an already-deployed LALM, adopting such a tokenizer means re-aligning the encoder, the projector and usually the LLM. That needs multimodal alignment data and substantial compute. AudioTETRIS runs on frozen checkpoints with no gradient updates.
  • Rate control. Fixed low-rate codecs set their compression ratio at training time. Some adaptive tokenizers allow inference-time adjustment: DyCAST [4] has a boundary-predictor decoding setting that controls the effective frame rate, and VARSTok's [2] rate depends on its clustering threshold. That control acts only on acoustic granularity, and it stays tied to a model trained on that tokenizer. AudioTETRIS sets the compression rate as an inference-time parameter of an existing LALM, so one deployed model can serve different latency budgets.
  • Task-aware selection. Adaptive tokenizers allocate tokens from the speech signal alone. None of them conditions on the downstream task. Stage 2 of AudioTETRIS ranks audio tokens by text-query-to-audio attention at the calibrated crossover layer ℓ†\ell^\dagger.
  • Complementary. Finally, we see it as the 2 approaches not being mutually exclusive. A query-aware, in-LLM method such as AudioTETRIS could be applied on top of a low-rate or adaptive-rate tokenizer. We note this as future work (Section 7).

[1] A. Défossez, L. Mazaré, M. Orsini, A. Royer, P. Pérez, H. Jégou, E. Grave, and N. Zeghidour, "Moshi: a speech-text foundation model for real-time dialogue," arXiv:2410.00037, 2024.

[2] R.-C. Zheng, W. Liu, H.-P. Du, Q. Zhang, C. Deng, Q. Chen, W. Wang, Y. Ai, and Z.-H. Ling, "Say More with Less: Variable-Frame-Rate Speech Tokenization via Adaptive Clustering and Implicit Duration Coding," in Proc. AAAI, vol. 40, no. 37, pp. 35021–35029, 2026.

[3] X. Zhang, F. Fang, P. Gao, B. Qin, B. Ahmed, and J. Epps, "Distinctive Feature Codec: An Adaptive Efficient Speech Representation for Depression Detection," arXiv:2505.18516, 2025.

[4] L. Della Libera, C. Subakan, and M. Ravanelli, "Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenization," arXiv:2601.23174, 2026.

Q1:

We thank the reviewer for the suggestion and we have now added the absolute rates in Table 2 (Section 5.1). Qwen2-Audio and Audio Flamingo 3 produce 25 Hz audio tokens and Phi-4-Multimodal 12.5 Hz. For each compression ratio, the table gives the rate after each of the two pruning layers:

Compression ratioQwen2-Audio / AF3 (25 Hz)Phi-4-MM (12.5 Hz)
0.3620.0→16.020.0 \rightarrow 16.010.0→8.010.0 \rightarrow 8.0
0.5117.5→12.2517.5 \rightarrow 12.258.75→6.138.75 \rightarrow 6.13
0.6415.0→9.015.0 \rightarrow 9.07.5→4.57.5 \rightarrow 4.5
0.7512.5→6.2512.5 \rightarrow 6.256.25→3.136.25 \rightarrow 3.13

Because pruning is progressive, early decoder layers still process the higher rates. Therefore, we also include in Appendix A.1.5 a layer-averaged effective rate per backbone (eg: 15.3 Hz and 10.2 Hz on Qwen2-Audio at ratios 0.51 and 0.75). It also explains that pre-LLM compression baselines run every layer at the final rate, which is why their TFLOPs are slightly lower.

Q2:

Thank you for the comment. In Section 6.2, we have ablation experiments where we include this setting. The ‘Only Stage 1’ variant in our component ablation (Figure 6; and Table 8 in Appendix A.3.2) is the encoder-side pruning strategy. In this setting, audio tokens are ranked only by the acoustic transition score wtw_t and pruned once at the LLM input, before any decoder layer. No decoder attention is used for scoring. To make this link explicit, we made a revision in the text of Section 6.2 which now states where pruning happens for each ablation variant. On Qwen2-Audio, ‘Only Stage 1’ holds up well under light compression but falls behind the full method as compression increases, on LibriSpeech:

Compression ratio0.360.510.640.75
Only Stage 1, WER ↓\downarrow5.507.398.9213.78
AudioTETRIS (Stage 1 + Stage 2), WER ↓\downarrow4.525.776.628.50

Even on its own, the acoustic prior is a strong encoder-side pruner. At ratio 0.36 it already beats LTBM (5.90) and SpeechPrune (6.75) on WER in our setup. The non-ASR tasks follow the same trend but with smaller gaps. Deferring selection into the decoder has a modest compute cost: at ratio 0.51, the full method uses 6.13 TFLOPs, compared with 5.65 for pre-LLM pruning, while latency is essentially unchanged (1.00 s vs. 0.99 s). We think this is a favourable trade-off given the 1.6-point WER reduction at that ratio.

Presentation Comment:

We thank the reviewer for the suggestion. We have revised Figure 1 to separate the two information paths. The audio embeddings enter the LLM unchanged (solid arrows). The transition weights wtw_t and attention scores p~t\tilde{p}_t are now drawn as auxiliary scoring signals (dashed arrows) that bypass the LLM and feed only into the fused importance score used for Top-K pruning. We also made the figure vector format.

Summary of main changes in the revision (highlighted in blue):

  • Revised related work in Section 2 which includes Segmentwise Pruning and a discussion on low-rate tokenizers
  • Table 2 in Section 5.1 for the absolute token rates
  • Appendix A.1.5 to show the layer-averaged effective token rate per backbone
  • Section 6.2 now defines the ‘Only Stage 1’ and ‘Only Stage 2’ variants explicitly, including where pruning happens, and summarises the result above.
  • Appendix A.3.2 has a new ‘Setup’ paragraph. It describes how each ablation variant scores and prunes tokens
  • Table 8 now reports latency and TFLOPs for every variant, so the accuracy-compute trade-off between pre-LLM and in-decoder pruning can be read directly.
Reviewer_mJUtOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AudioTETRIS, a training-free token compression method for audio LLMs. It preserves information-rich tokens in spectral transition regions by jointly considering phoneme transition scores and attention scores when ranking token inportance.Experiments on multiple audio LLMs and audio understanding tasks show the effectiveness of the proposed method.

Critical strengths
  1. The paper highlights the effectiveness of preserving phoneme transitions in training-free token compression. The idea of considering phoneme transition scores for token compression is interesting, and the authors comfirmed the correlation between token-to-token distances and actual phonetic transitions via experiment.
  2. The evaluation is comprehensive, spanning three audio LLMs, four tasks, and multiple compression rates. On some models (i.e., Qwen2 Audio) and tasks (i.e. emotion recognition), performance even improved after pruning.
Critical weaknesses

See questions.

Questions for authors
  1. Stage 1 may be acting largely as a phoneme-boundary predictor. Have you considered deriving phoneme transition scores from pretrained phoneme prediction models which may better predict phoneme boundaries?
  2. The effectiveness of Stage 2 is not distinct. In Table 6, its improvement over Stage 1 concentrates on ASR and is comparably modest on IEMOCAP, MMAU-mini, and CoVoST2. More importantly, have you directly compared layer adaptive scoring with independent intra-audio or cross-modal filters?
  3. Does AudioTETRIS generalize to general audio beyond speech? If so, effectiveness of the proposed method on general audio tasks (especially those included in MMAU-mini) needs further explanation since there is no phoneme transitions in general audio.
3 comments

We thank the reviewer for the helpful questions. Some of the additional experiments we propose below are still running. Once they finish, we will post the results here and upload a revised manuscript with all the changes described in this response during the discussion period.

Q1:

We thank the reviewer for this observation. For speech inputs, Stage 1 does behave much like a phoneme-boundary detector, and this is intended. We use a signal from the LALM's own encoder rather than an external phoneme model for two reasons -

  • No extra model or cost - The encoder states have already been computed, so Stage 1 adds almost no overhead. An external phoneme detector would add a second model, plus extra latency and memory, which would work against our goal of efficient inference.
  • It also applies beyond speech - Phoneme models only work on speech and are often tied to particular languages and phoneme sets. Many LALM tasks involve non-speech audio such as sound events/ music/environmental audio, where phoneme boundaries don't exist. The encoder distance dtd_t could still respond to acoustic change in those cases, so Stage 1 works across all audio types without any change

Revision: We will clarify this in Section 1 and 2 (Introduction & Related Work) of the revised manuscript. We will state explicitly that Stage 1 acts as a boundary detector for speech, explain why we chose an encoder-internal signal over an external phoneme model, and discuss how it would behave on non-speech audio

Q2:

We agree that Stage 2's gains are largest on ASR and more modest on other tasks. We think this is expected as ASR is the task most sensitive to losing individual tokens whereas emotion recognition and audio QA rely more on overall properties of the clip, which the acoustic prior already preserves well.

On layer-adaptive scoring, our schedule ablation (Table 11) gives some evidence. Pruning at layer 15 with cross-modal scores clearly beats pruning at layer 1 with the intra-audio score (12.89 vs 18.78 WER at 0.75 compression rate). However, that comparison changes the layer and the filter together. To separate the 2, we are running the full ablation on Qwen2-Audio: pruning layers {1, 15}, Stage 1 and the compression rate are fixed, and we compare intra-audio-only, cross-modal-only and layer-adaptive scoring on all benchmarks and compression rates. Thank you for the suggestion, it would make the role of layer-adaptive scoring clearer.

Revision: We will add these results to Appendix A.3

Q3:

Thank you for raising this. You are right that 'phoneme transition' is a speech-specific term, and we clarify why the method still works on general audio

As noted in our response to Q1, Stage 1 measures how much the encoder representation changes between consecutive tokens (dtd_t). In speech, the largest changes happen at phoneme transitions, which is why we validated it against phonetic alignments. But the same principle could apply to other audio as information tends to be concentrated where the sound changes, such as note onsets/chord changes in music or the start/end of sound events. To check this directly, we split our MMAU-mini results by domain:

MMAU-mini accuracy by domain (Qwen2-Audio):

SettingOverallSoundMusicSpeech
No compression42.2047.8039.2039.60
AudioTETRIS (0.51)42.9147.5040.6040.63
AudioTETRIS (0.75)41.4647.3038.4038.68

AudioTETRIS stays within about 1 point of the uncompressed model on every domain at both compression rates. We find that 'sound' is the most stable (-0.3 and -0.5 points) and so performance on non-speech audio is therefore preserved even under aggressive compression.

Revision: We will clarify this point and report the domain-wise results on MMAU-mini across our other backbones in Appendix A.3.

We thank the reviewer again for the helpful comments. We have uploaded a revised manuscript, with all changes marked in blue. Here is how we addressed each point.

Q1:

The Introduction (Section 1) now says explicitly that for speech, Stage 1 acts as a phoneme-boundary detector. It also explains that the prior comes from the LALM’s own encoder states rather than from an external phoneme model, so it adds almost no cost. Because it measures acoustic change rather than phonemes, it can also apply to non-speech audio. We examine how the acoustic prior behaves on non-speech audio in Appendix A.3, using MMAU-mini results split by domain (see Q3).

Q2:

We finished the ablation and added it to Appendix A.3.5 (‘Effect of the Attention Score at Fixed Pruning Layers’). In this analysis, we fix the pruning layers {1, 15}, the Stage 1 prior and the per-stage retention. We compare intra-audio-only, cross-modal-only and layer-adaptive scoring on all benchmarks and compression rates. We also added a fourth variant called ‘both sources’, which averages attention over both audio- and text-query rows at both layers. We find:

  • Layer-adaptive scoring is best on every benchmark at every compression rate, and the gap grows with compression. At rate 0.75, test-other WER is 8.50, compared with 12.81 (intra-only), 15.55 (cross-only) and 17.45 (both sources).
  • The benefit is also not limited to ASR. At rate 0.51, layer-adaptive scoring beats the best fixed-source variants by 3.6 points on IEMOCAP, 2.4 points on MMAU-mini and 0.9 BLEU on CoVoST2.
  • Combining both sources at every layer is the weakest variant throughout. This matches our token-level analysis in Appendix A.2.3: at layer 1, cross-modal attention barely tracks token importance, while at ℓ† it is the sharper signal. Picking the right source for each layer keeps the strength of each one.

Q3:

Appendix A.3.3 now reports MMAU-mini accuracy by domain (sound, music, speech) for all three backbones, uncompressed and at compression rates 0.51 and 0.75. At rate 0.51, no domain drops by more than 1.2 points on any backbone. At rate 0.75, sound and music drop by at most 1.9 points (at most 0.8 on Qwen2-Audio and Audio Flamingo 3). Speech drops the most, by up to 3.2 points on Audio Flamingo 3. From this, we find that performance on non-speech audio can therefore hold up under aggressive compression.

We hope these revisions address the reviewer’s concerns, and we are happy to clarify anything further.

Thank you for your clarification and additional experiments, which fully addressed my concerns. It is now clearer that the proposed method generalizes to other audio modalities beyond speech, with the generalization ability properly explained. Therefore, I would like to maintain my overall assessment of weak acceptance.

Comments on the paper

1 comment

We thank all reviewers for their careful and constructive feedback. We have uploaded a revised manuscript, with all changes marked in blue.

What reviewers found valuable:

  • The acoustic prior: Reviewers found it interesting and promising to use an audio-specific prior from the LALM’s own encoder for token pruning (mJUt, CPe4). They also found the evidence that encoder-state transitions match phoneme transitions promising, with the label-free kurtosis analysis agreeing with forced-alignment correlation (mJUt, CPe4).
  • Evaluation: Reviewers found the evaluation very comprehensive: three backbones, four tasks, multiple compression rates and supporting ablations (mJUt, Xs28, CPe4). It showed meaningful compute savings, and in some settings performance was higher than the uncompressed model (mJUt).
  • Practical relevance: Reviewers agreed that reducing audio-token cost in LALMs is an important problem. They saw value in a method that is training-free and adds speech-specific information to existing pruning (Xs28).

Main changes in the revisions:

More important baselines (Xs28). We added Top-K [1], Segmentwise Top-K [2] and HeadRouter [3] to Table 1, each at all four compression rates and on all five evaluation sets. We find that AudioTETRIS still outperforms the new baselines in most tasks.

Why the layer-adaptive scoring works (mJUt, CPe4).

  • We include new experiments in Appendix A.2.3 which use an occlusion-based token importance to show that ℓ† is where cross-modal attention starts to rank tokens more reliably than intra-audio attention.
  • A new ablation (Appendix A.3.5) keeps the pruning layers fixed and changes only the attention source. Layer-adaptive scoring is best on every benchmark and at every rate, while averaging both sources (audio and text) is the weakest variant.

Reporting audio token rates and low-frame-rate tokenizers (Xs28).

  • We now report absolute token rates (Table 2): at rates 0.51 and 0.75 we reach 12.25 Hz and 6.25 Hz, and as low as 3.13 Hz on Phi-4 at rate 0.75. That puts AudioTETRIS on par with dedicated low-rate speech tokenizers (6.25 Hz - 12.5 Hz like [4]), and we get there training-free, without the need for new architectures and full retraining.
  • Our related work section has been expanded to also discuss the trade-off with low- and adaptive-frame-rate tokenizers.

Beyond speech (mJUt). MMAU-mini results are now broken down by domain (sound/music/speech) for all three backbones (Appendix A.3.3). We observe that sound and music accuracy drops by at most 1.9 points even at an aggressive compression rate 0.75, meaning that AudioTETRIS performs well for non-speech audio as well.

Completeness and clarity (CPe4, Xs28).

  • LibriSpeech test-clean results are added for all backbones.
  • We clarified the “stable phonetic segment” terminology and the role of Stage 1 as a phoneme-boundary detector.

We hope these revisions address the reviewers’ concerns, and we are happy to clarify anything further.

[1] Lee, Taehan, and Hyukjun Lee. “Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importance.” European Conference on Artificial Intelligence (ECAI), 2025.

[2] M. Gibier, P. Serrano, O. Boeffard, R. Duroselle and J. -F. Bonastre, "Segmentwise Pruning in Audio-Language Models," ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2026, pp. 15322-15326, doi: 10.1109/ICASSP55912.2026.11463308.

[3] He, Peize, Yaodi Luo, Xiaoqian Liu, Xuyang Liu, Jiahang Deng, Yaosong Du, Bangyu Li, Xiyan Gui, Yuxuan Chen, and Linfeng Zhang. “HeadRouter: Dynamic Head-Weight Routing for Task-Adaptive Audio Token Pruning in Large Audio Language Models.” Proceedings of the 34th ACM International Conference on Multimedia, 2026.

[4] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speechtext foundation model for real-time dialogue. arXivpreprint arXiv:2410.00037.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.