- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes UniAudio-Token, a single-codebook speech and general audio tokenizer. It's trained with Semantic-Acoustic Primitives, covering linguistic content, vocal attributes and auditory scenes. A Semantic-Acoustic Equilibrium module fuses the shallow acoustic and deep semantic features of the encoder. The tokenizer is initialized from a Whisper model and produces discrete tokens using a single 8,192-entry codebook at 25 Hz. Evaluation covers environmental-sound clustering, downstream audio understanding, speech reconstruction, and speaker-conditioned TTS.
Critical strengths
-
The proposed framework has consistent improvement on speech/audio understanding compared to its baselines. Using a common Qwen2.5-3B backbone, UniAudio-Token achieves the highest overall accuracy among the evaluated tokenizers on MMAU, MMAR, and MMSU . Improvements are particularly pronounced for sound and music, while speech performance remains competitive. These gains support the usefulness of extending semantic speech tokenizers to broader audio perception.
-
The tokenizer shows competitive speech generation performance. The single-codebook representation operates at 325 bps and achieves favorable speech reconstruction and TTS results. This allows the tokenizer to be deployed efficiently in SpeechLLM systems.
Critical weaknesses
-
Limited novelty and incomplete related-work positioning. Unified speech and general-audio understanding has already been addressed by SPEAR [1], while DashengTokenizer[2] supports understanding and generation by injecting acoustic information into semantic features. But neither of them were cited or discussed. DashengTokenizer’s acoustic-injection principle is particularly relevant to SAE’s motivation. Although UniAudio-Token’s compact discrete interface is distinct, the novelty of unifying speech and audio representations is not established. I would recommend the authors to position their contribution more precisely around SAP supervision and adaptive fusion within a single-codebook tokenizer.
-
Lack of controlled experiments for analyzing the contribution of SAP. For example, no comparison is made between SAP with ordinary natural language audio caption supervision, or adaptive SAE with fixed additive fusion. Without these experiments, it's difficult to establish that structured supervision and adaptive gating are responsible for the main improvements.
[1]: Yang, X., Yang, Y., Jin, Z., Cui, Z., Wu, W., Li, B., Zhang, C. and Woodland, P., 2025. "SPEAR: A unified SSL framework for learning speech and audio representations". in Proc. ICML 2026
[2]: Dinkel, H., Sun, X., Li, G., Mei, J., Niu, Y., Liu, J., Li, X., Liao, Y., Zhou, J., Zhang, J. and Luan, J., 2026. Dashengtokenizer: One layer is enough for unified audio understanding and generation. arXiv preprint arXiv:2602.23765.
Questions for authors
-
How much of the improved environmental-sound class separation comes from incorporating general audio training data and auditory-scene labels, given that several semantic tokenizer baselines are primarily trained for speech? Can the authors provide comparisons using matched audio corpora and training budgets, including ordinary caption supervision versus SAP and models with and without SAE?
-
What is the reconstruction performance on non-speech audio, such as environmental sounds and music? The current reconstruction and generation experiments only evaluate speech. I am interested in the non-speech reconstruction performance, to see if the tokens actually preserve general acoustic detail.
9 comments
Response to Reviewer ChK7 (2/2): Q2
Continuing from the previous comment, here we provide our responses to the rest questions.
Q2: Non-speech Audio Reconstruction
Thank you for raising this important point. Since UniAudio-Token is designed as a compact single-codebook interface optimized for universal audio understanding and high-fidelity speech interaction, reconstructing complex music or arbitrary environmental sounds is beyond its scope (which typically requires multi-codebook RVQ at much higher BPS). Nevertheless, our ESC-50 clustering (Table 2) and strong gains on MMAU/MMAR Sound/Music and MMSU Perception (Table 4) directly confirm that non-speech acoustic details are faithfully preserved in the token representations. We will explicitly clarify this limitation and design choice in the revised paper.
Thank you again for these valuable suggestions. We hope that these clarifications effectively address your concerns.
I thank the authors for the detailed rebuttal and the additional controlled experiments. I appreciate the discussion of SPEAR and DashengTokenizer, and the new ablation results in Tables R1 and R2 provide useful evidence supporting the effectiveness of SAP and SAE. I am largely satisfied with the response to my original concerns.
However, I still have a few follow-up questions and would appreciate further clarification.
Follow-up questions
-
Contribution of general audio training data. While the newly added ablations demonstrate the effectiveness of SAP and SAE under matched training configurations, the contribution of incorporating general audio data itself remains unclear. Given that most of the semantic tokenizer baselines are trained primarily on speech data, the improved environmental sound class separation in Table 2 is somewhat expected. It would be interesting to understand how much of this improvement comes from the inclusion of general audio training data rather than the proposed methodology. If additional controlled experiments are not feasible, I would encourage the authors to acknowledge this factor when discussing the results.
-
Supervision in Table R1. Could the authors provide an example of the unstructured caption targets and clarify whether they contain the same underlying information as SAP, particularly the ground-truth transcription and acoustic descriptions? This is important to establish whether the comparison isolates the effect of structured supervision rather than differences in the amount or quality of the supervision information. Additionally, since the ablation experiments use both LibriSpeech and the balanced training subset of AudioSet, how are ASR transcription targets obtained for AudioSet samples, particularly those containing no speech?
-
Speech reconstruction in Table 3. It is unclear whether the baseline tokenizers use their original reconstruction decoders or whether separate flow-matching decoders are trained for all tokenizers under matched configurations. Since reconstruction performance depends on both the tokenizer and the decoder, differences in decoder architectures, training data, or training budgets could potentially confound the comparison. Please clarify the reconstruction pipeline used for each tokenizer.
-
Speaker embedding conditioning in Section 5.2. The authors state that other tokenizers do not support speaker embedding conditioning. However, I would expect support for external speaker embeddings to depend primarily on the downstream generation architecture rather than the tokenizer itself. Could the authors clarify why the other tokenizers cannot be integrated with the same speaker-conditioned generation framework used for UniAudio-Token?
Regarding non-speech audio reconstruction, I understand the authors' design choice to focus on general audio understanding and speech generation. Nevertheless, I would encourage the authors to distinguish between retaining information useful for general audio understanding and faithfully preserving fine-grained acoustic details, as the latter is not directly established by the current evaluation.
I look forward to the response of the authors.
Response to Reviewer ChK7 (1/2): W1, W2 & Q1
We thank the reviewer for the constructive review. Below, we address your comments and questions in detail.
W1: Novelty and Related Work
Thank you for pointing out these relevant methods. Specifically,
UniAudio-Tokendiffers fromSPEARandDashengTokenizerin key architectural and methodological aspects:Comparison with SPEAR:
SPEARlearns a continuous speech/audio encoder: its discrete MVQ tokens derived by teacher model serve as pre-training targets, while downstream tasks use continuous encoder representations. It does not perform discrete quantization (VQ) or address the severe information compression bottleneck of a single discrete codebook, nor does it directly provide discrete tokens for autoregressive LLM audio generation.UniAudio-Tokenspecifically resolves how to compress both linguistic and general acoustic information into a compact single-codebook discrete interface, which requires evaluating which perceptual information remains useful under the discrete bottleneck and how the tokenizer should be trained to retain it.Comparison with DashengTokenizer:
DashengTokenizercombines acoustic reconstruction and semantic regularization targets with a static additive acoustic injection, and still provides a continuous representation. In contrast,UniAudio-Tokenintroduces SAP to learn both acoustic and semantic features. As shown in our new ablation (Table R1), such structured decomposition prevents reconstruction objectives from causing semantic entanglement and degrading speech representations, and yields superior downstream understanding. Moreover, our SAE employs a content-aware gating mechanism that dynamically adjusts shallow acoustic injection based on input characteristics. Our new ablation (Table R2) also confirms that adaptive gating substantially outperforms static additive fusion.We will discuss both works thoroughly in Section 2 (Related Work) of the revised paper, which will further highlight the importance of the problem addressed by UniAudio-Token.
W2 & Q1: Ablations
To directly answer whether our gains come mainly from SAP and SAE, while considering the massive scale of our full training corpus and limited computational resources and time, we conduct strictly controlled ablation studies on a balanced subset (LibriSpeech training set + AudioSet bal_train subset) and downstream Qwen2.5-0.5B LLM with identical training configuration (initialization, training data, codebook size, frame rate, training steps, etc.), to isolate the exact contributions of SAP and SAE.
SAP vs. ASR Transcription & Unstructured Captions
Table R1: SAP vs. ASR Transcription & Unstructured Captions
Analysis:
SAPoutperforms bothASR transcription(+2.90% MMAU Overall, +1.50% MMAR Overall, +1.60% MMSU Overall) andunstructured captions(+3.10% MMAU Overall, +3.40% MMAR Overall, +0.96% MMSU Overall).unstructured captionsdegrade speech understanding due to entangled, high-entropy text targets, whereasSAPimproves both speech and non-speech simultaneously.SAE vs. None & Additive Fusion
Table R2: SAE vs. None & Additive Fusion
Analysis:
SAEoutperformsNoneandAdditivefusion across every single benchmark, demonstrating the necessity of adaptive gating.Additivefusion improves overNoneon sound/music but slightly degrades speech performance, indicating the impact of injecting noise features indiscriminately.These controlled experiments confirm that the improvements in universal audio understanding are driven directly by SAP's structured disentanglement and SAE's adaptive gating. We will include these ablation results and discussion in the revised paper.
Due to character limitations, our response continues in the following comment.