acceptodds
Withdrawn from ICLR 2027

X-AuT: Progressive Audio-Encoder Compression for Speech LLMs with Cross-Scale Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reducing audio-encoder depth lowers the inference cost of speech large language models, but removing complete blocks perturbs the embeddings consumed by the decoder and can cause deletion and premature end-of-sequence errors. We introduce X-AuT, a progressive framework that selects layer combinations through short behavioral probes and restores the pruned model through representation alignment, cross-scale distillation, scheduled student-policy supervision, and LoRA finetuning. The language-model backbone remains frozen, while attention LoRA adapters and the tied output embedding adapt during distillation. Training uses the highest-agreement tier from a transcript-consistency pipeline, followed by source reweighting during finetuning. On ten public Chinese–English benchmarks, compressing Qwen3-ASR-0.6B from 18 to 16 audio-encoder layers reduces macroaverage error from 5.61% to 5.27%. The 14-layer model reaches 5.75% with 20.7% fewer audio-tower parameters. Under the matched recipe, the 1.7B teacher yields 5.55% mean error, compared with 8.45% for self-distillation, and progressive 18→14 pruning outperforms direct pruning (5.75% vs. 6.73%). These singlerun results establish two practical operating points and show that the accuracy effects vary across benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech large language models, audio encoder compression

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ez4tOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes X-AuT, a framework for compressing the audio encoder of Qwen3-ASR through progressive layer pruning and cross-scale distillation. It combines behavioral layer selection, representation alignment, scheduled student-policy supervision, and fine-tuning. Evaluation covers ten Chinese and English benchmarks. The reported models remove up to 20.7% of audio-tower parameters while approximately preserving macro-average recognition error.

Critical strengths

The paper targets a practical problem and evaluates recognition accuracy across ten public benchmarks. The matched teacher-scale and progressive vs. direct pruning comparisons provide useful evidence supporting parts of the framework.

Critical weaknesses
  • W1: The empirical justification for the complete framework remains incomplete. There are no controlled ablations establishing the necessity of representation alignment or transcript-consistency filtering, even though the conclusion states that alignment is necessary. Moreover, Appendix A.6 reports that hybrid on-policy training performs worse than the matched off-policy control, with best TER of 6.41% versus 6.23%. An unpruned model adapted using the same data and teacher is also missing, making it difficult to separate the reported accuracy improvement from additional training and stronger supervision.
  • W2: The practical advantage over alternative compression approaches is insufficiently established. Appendix A.7 reports inference measurements, but the end-to-end latency reductions are only 2.6% on H800 and 4.7% on the in-vehicle accelerator. There is no comparison against competing compression methods or simpler pruning-and-recovery baselines at matched accuracy and training budgets. The cost of candidate screening and the complete recovery pipeline is also unreported, leaving the overall accuracy vs. efficiency tradeoff unclear.
Questions for authors
  • Q1: Can the authors provide controlled comparisons removing representation alignment and transcript filtering, alongside an unpruned model trained with the same data and teacher? These controls would clarify which components account for the results.
  • Q2: Given that matched off-policy training outperforms hybrid on-policy training in Appendix A.6, what evidence supports retaining student-policy supervision in the proposed framework?
  • Q3: Can the authors compare against a simpler compression baseline under matched recovery budgets and report both end-to-end inference efficiency and total compression cost?
Reviewer_iZJdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a multi-stage framework that can learn a smaller ASR model. The ASR model is speech LLM based. Before training, some layers in the student model are removed. For the first stage training, they attempt to align pruned student's representation with teacher's group representations. Then, teacher model is treated as larger scale, and is used to perform "cross-scale" distillation based on teacher forcing labels. Finally, LoRA finetuning is conducted to adjust the decoder's behavior. The ASR evaluations center around English and Mandarin datasets.

Critical strengths

None

Critical weaknesses

The experiments are conducted around ASR only

Although the paper and the title uses the term speech LLM, the experiments and the evaluations merely focus on ASR tasks. The generalizability to broader tasks is left untested.

The novelty is limited

The framework is packaged by fancy wordings, e.g., "progressive compression", "cross-scale distillation", "behavioral candidate screening", "scheduled student-policy training". However, the method is no more than a carefully engineered knowledge distillation framework for ASR, conducted on two languages.

The framework is complicated

The method is comprised of 5 main stages (as illustrated in Figure 2) and the training includes 3 stages additionally. This raises concerns about the practicality of the method. Is it extendable to other langauges? Are all the stages necessary? What's is the main intuition or main focus behind the complicated framework? How effective is it?

Questions for authors

Please directly address the weaknesses part.

Reviewer_AEwhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed X-AuT, which prunes in two hops, 18 to 16 to 14 layers, starting from the 18-layer audio encoder of Qwen3-ASR-0.6B. X-AuT is a practical and well-documented recipe for removing complete audio-encoder blocks from Qwen3-ASR-0.6B and recovering accuracy with a larger teacher, representation/logit alignment, LoRA, and progressive pruning. The 14-layer model removes 20.7% of audio-tower parameters and stays close to the 18-layer baseline on ten public benchmarks; the 16-layer model improves the observed macro error. The progressive-versus-direct and cross-scale-versus-self-distillation controls are useful.

Critical strengths
  1. Deployment-friendly structural compression. Removing complete blocks gives a regular shallower encoder without custom sparse kernels.

  2. Broad public evaluation. Ten Chinese and English benchmarks cover read, meeting, CommonVoice, TED, and multilingual FLEURS conditions.

  3. Useful progressive-pruning control. The 5.75% versus 6.73% comparison provides evidence that staged recovery is preferable to direct removal under the tested budget.

  4. Transparent reporting. Exact audio-tower parameter counts, per-benchmark regressions, single-seed limitations, selection protocol, and measured device latency are reported clearly.

Critical weaknesses
  1. The compression cost is confounded by unmatched post-training. Pruned students receive additional representation alignment, gold CE, cross-scale distillation, decoder LoRA, filtered data, and source reweighting, whereas the released 18-layer baseline does not. Without an unpruned 18-layer student trained with the identical recovery recipe, the 16-layer improvement cannot be attributed to compression and the true 14-layer retention gap is unknown.

  2. The student-policy contribution is contradicted by its matched ablation. Matched off-policy training achieves 6.23% best and 6.51% final TER, outperforming hybrid student-policy training at 6.41% and 6.99%. The reported control therefore does not support student-policy supervision as a beneficial component, and the realized on-policy fraction after rejection and fallback is not given.

  3. End-to-end efficiency gains are modest. Removing 20.7% of audio-tower parameters improves encoder latency by 21.4% on PPU and 11.4% on H800, but end-to-end latency improves by only 4.7% and 2.6% because decoding dominates. The 16-layer latency, total-model compression, peak memory, throughput, energy, and duration-dependent behavior are not reported.

  4. Generality and reproducibility are narrow. The study uses one Qwen3-ASR family, substantial proprietary data, and a filtered index based on agreement between two recognizers, which may favor easier speech. Stage-2 source reweighting adds another possible cause of benchmark-specific gains, and no public-only reproduction or second encoder family is provided.

  5. Missing relevant references:

  • H. Wang, S. Wang, Y. Gong, and W.-Q. Zhang, "DistilALHuBERT: A Distilled Parameter Sharing Audio Representation Model," in Proc. 6th International Conference on Signal Processing and Machine Learning (SPML), pp. 45-50, 2023, doi:10.1145/3614008.3614015.
  • B. Han, W. Huang, Z. Chen, A. Jiang, P. Fan, C. Lu, Z. Lv, J. Liu, W.-Q. Zhang, and Y. Qian, "Data-Efficient Low-Complexity Acoustic Scene Classification via Distilling and Progressive Pruning," in Proc. 2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5, 2025, doi:10.1109/ICASSP49660.2025.10890296.
  • R. Fan, Y. Wang, Y. Hu, B. Ren, Y. Xia, X. Wang, Y. Qian, S. Liu, and J. Li, "LLM can Read Spectrogram: Encoder-free Speech-Language Modeling," accepted to 2026 IEEE Spoken Language Technology Workshop (SLT), arXiv:2606.10231, 2026, doi:10.48550/arXiv.2606.10231.
  • K.-H. Lu, K. Deng, R. Fan, R. Zhao, and J. Li, "Compress the Cache, Not the Speech Embedding: KV Compression for Efficient Speech LLMs," arXiv:2607.06827, 2026, doi:10.48550/arXiv.2607.06827.
Questions for authors
  1. What does an unpruned 18-layer student achieve after the exact same stages, teacher, LoRA modules, data index, source weights, updates, and checkpoint-selection procedure?

  2. What fraction of scheduled rollout batches actually contributes on-policy KD after rejection and fallback, and does hybrid training outperform matched off-policy training in any repeated setting?

  3. Across repeated training seeds, paired bootstrap samples, and a larger or disjoint selection set, how stable are the 18L/16L/14L ordering, selected layer pair, and direct-versus-progressive comparison?

  4. What are the 16-layer latency and the total-model parameters, peak memory, first-token latency, throughput, real-time factor, and energy, stratified by utterance duration?

  5. Relative to CE-only recovery, what does representation alignment, the larger teacher, decoder LoRA, quality-tier filtering, and Stage-2 source reweighting each contribute?

  6. Can the complete recipe be reproduced using only public data or transferred to Whisper, another Speech-LLM family, or another encoder depth?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.