acceptodds
Under review as a conference paper at ICLR 2027

UniAudio-Token: Empowering Semantic Speech Tokenizers with General Audio Perception

Abstract

Semantic speech tokenizers have become a widely used interface for Audio-LLMs, owing to their compact single-codebook design and strong linguistic alignment. However, their focus on linguistic abstraction induces acoustic blindness, limiting their applicability beyond speech-centric tasks. We propose UniAudio-Token, a framework that empowers semantic tokenizers with general audio perception without compromising speech ability. Instead of altering the semantic paradigm, UniAudio-Token mitigates its information loss through two key innovations: (1) Semantic-Acoustic Primitives (SAP) provide structured supervision by decomposing audio into linguistic content, vocal attributes, and auditory-scene primitives; and (2) Semantic-Acoustic Equilibrium (SAE) introduces a content-aware gating mechanism that adaptively restores fine-grained acoustic details from shallow layers. Extensive experiments show that UniAudio-Token learns comprehensive universal representations while preserving high-fidelity speech generation. When integrated with downstream LLMs, it outperforms all single-codebook baselines on both understanding and generation, effectively serving as a unified audio interface.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Tokenizer, Universal Audio, Speech, Audio Language Modeling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ChK7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UniAudio-Token, a single-codebook speech and general audio tokenizer. It's trained with Semantic-Acoustic Primitives, covering linguistic content, vocal attributes and auditory scenes. A Semantic-Acoustic Equilibrium module fuses the shallow acoustic and deep semantic features of the encoder. The tokenizer is initialized from a Whisper model and produces discrete tokens using a single 8,192-entry codebook at 25 Hz. Evaluation covers environmental-sound clustering, downstream audio understanding, speech reconstruction, and speaker-conditioned TTS.

Critical strengths
  • The proposed framework has consistent improvement on speech/audio understanding compared to its baselines. Using a common Qwen2.5-3B backbone, UniAudio-Token achieves the highest overall accuracy among the evaluated tokenizers on MMAU, MMAR, and MMSU . Improvements are particularly pronounced for sound and music, while speech performance remains competitive. These gains support the usefulness of extending semantic speech tokenizers to broader audio perception.

  • The tokenizer shows competitive speech generation performance. The single-codebook representation operates at 325 bps and achieves favorable speech reconstruction and TTS results. This allows the tokenizer to be deployed efficiently in SpeechLLM systems.

Critical weaknesses
  • Limited novelty and incomplete related-work positioning. Unified speech and general-audio understanding has already been addressed by SPEAR [1], while DashengTokenizer[2] supports understanding and generation by injecting acoustic information into semantic features. But neither of them were cited or discussed. DashengTokenizer’s acoustic-injection principle is particularly relevant to SAE’s motivation. Although UniAudio-Token’s compact discrete interface is distinct, the novelty of unifying speech and audio representations is not established. I would recommend the authors to position their contribution more precisely around SAP supervision and adaptive fusion within a single-codebook tokenizer.

  • Lack of controlled experiments for analyzing the contribution of SAP. For example, no comparison is made between SAP with ordinary natural language audio caption supervision, or adaptive SAE with fixed additive fusion. Without these experiments, it's difficult to establish that structured supervision and adaptive gating are responsible for the main improvements.

[1]: Yang, X., Yang, Y., Jin, Z., Cui, Z., Wu, W., Li, B., Zhang, C. and Woodland, P., 2025. "SPEAR: A unified SSL framework for learning speech and audio representations". in Proc. ICML 2026

[2]: Dinkel, H., Sun, X., Li, G., Mei, J., Niu, Y., Liu, J., Li, X., Liao, Y., Zhou, J., Zhang, J. and Luan, J., 2026. Dashengtokenizer: One layer is enough for unified audio understanding and generation. arXiv preprint arXiv:2602.23765.

Questions for authors
  1. How much of the improved environmental-sound class separation comes from incorporating general audio training data and auditory-scene labels, given that several semantic tokenizer baselines are primarily trained for speech? Can the authors provide comparisons using matched audio corpora and training budgets, including ordinary caption supervision versus SAP and models with and without SAE?

  2. What is the reconstruction performance on non-speech audio, such as environmental sounds and music? The current reconstruction and generation experiments only evaluate speech. I am interested in the non-speech reconstruction performance, to see if the tokens actually preserve general acoustic detail.

9 comments
AuthorsOpenReview ↗

Response to Reviewer ChK7 (1/2): W1, W2 & Q1

We thank the reviewer for the constructive review. Below, we address your comments and questions in detail.


W1: Novelty and Related Work

Thank you for pointing out these relevant methods. Specifically, UniAudio-Token differs from SPEAR and DashengTokenizer in key architectural and methodological aspects:

  1. Comparison with SPEAR: SPEAR learns a continuous speech/audio encoder: its discrete MVQ tokens derived by teacher model serve as pre-training targets, while downstream tasks use continuous encoder representations. It does not perform discrete quantization (VQ) or address the severe information compression bottleneck of a single discrete codebook, nor does it directly provide discrete tokens for autoregressive LLM audio generation. UniAudio-Token specifically resolves how to compress both linguistic and general acoustic information into a compact single-codebook discrete interface, which requires evaluating which perceptual information remains useful under the discrete bottleneck and how the tokenizer should be trained to retain it.

  2. Comparison with DashengTokenizer: DashengTokenizer combines acoustic reconstruction and semantic regularization targets with a static additive acoustic injection, and still provides a continuous representation. In contrast, UniAudio-Token introduces SAP to learn both acoustic and semantic features. As shown in our new ablation (Table R1), such structured decomposition prevents reconstruction objectives from causing semantic entanglement and degrading speech representations, and yields superior downstream understanding. Moreover, our SAE employs a content-aware gating mechanism that dynamically adjusts shallow acoustic injection based on input characteristics. Our new ablation (Table R2) also confirms that adaptive gating substantially outperforms static additive fusion.

We will discuss both works thoroughly in Section 2 (Related Work) of the revised paper, which will further highlight the importance of the problem addressed by UniAudio-Token.


W2 & Q1: Ablations

To directly answer whether our gains come mainly from SAP and SAE, while considering the massive scale of our full training corpus and limited computational resources and time, we conduct strictly controlled ablation studies on a balanced subset (LibriSpeech training set + AudioSet bal_train subset) and downstream Qwen2.5-0.5B LLM with identical training configuration (initialization, training data, codebook size, frame rate, training steps, etc.), to isolate the exact contributions of SAP and SAE.

SAP vs. ASR Transcription & Unstructured Captions

Table R1: SAP vs. ASR Transcription & Unstructured Captions

SupervisionMMAU SpeechMMAU SoundMMAU MusicMMAU OverallMMAR SpeechMMAR SoundMMAR MusicMMAR OverallMMSU PerceptionMMSU ReasoningMMSU Overall
ASR transcription42.3454.3548.2048.3038.7830.3033.5035.6029.3846.5737.70
Unstructured captions40.5454.6549.1048.1037.7629.7033.5033.7030.2746.9438.34
SAP (Ours)44.4458.5650.6051.2041.5032.1235.9237.1031.1647.9839.30

Analysis:

  • SAP outperforms both ASR transcription (+2.90% MMAU Overall, +1.50% MMAR Overall, +1.60% MMSU Overall) and unstructured captions (+3.10% MMAU Overall, +3.40% MMAR Overall, +0.96% MMSU Overall).
  • Notably, unstructured captions degrade speech understanding due to entangled, high-entropy text targets, whereas SAP improves both speech and non-speech simultaneously.

SAE vs. None & Additive Fusion

Table R2: SAE vs. None & Additive Fusion

Feature FusionMMAU SpeechMMAU SoundMMAU MusicMMAU OverallMMAR SpeechMMAR SoundMMAR MusicMMAR OverallMMSU PerceptionMMSU ReasoningMMSU Overall
None44.1451.3544.3146.6040.4830.9131.5534.4028.6845.2136.68
Additive43.8453.7549.4049.0039.8030.3034.9534.8028.9545.5436.98
SAE (Ours)44.4458.5650.6051.2041.5032.1235.9237.1031.1647.9839.30

Analysis:

  • SAE outperforms None and Additive fusion across every single benchmark, demonstrating the necessity of adaptive gating.
  • Notably, static Additive fusion improves over None on sound/music but slightly degrades speech performance, indicating the impact of injecting noise features indiscriminately.

These controlled experiments confirm that the improvements in universal audio understanding are driven directly by SAP's structured disentanglement and SAE's adaptive gating. We will include these ablation results and discussion in the revised paper.


Due to character limitations, our response continues in the following comment.

AuthorsOpenReview ↗

Response to Reviewer ChK7 (2/2): Q2

Continuing from the previous comment, here we provide our responses to the rest questions.


Q2: Non-speech Audio Reconstruction

Thank you for raising this important point. Since UniAudio-Token is designed as a compact single-codebook interface optimized for universal audio understanding and high-fidelity speech interaction, reconstructing complex music or arbitrary environmental sounds is beyond its scope (which typically requires multi-codebook RVQ at much higher BPS). Nevertheless, our ESC-50 clustering (Table 2) and strong gains on MMAU/MMAR Sound/Music and MMSU Perception (Table 4) directly confirm that non-speech acoustic details are faithfully preserved in the token representations. We will explicitly clarify this limitation and design choice in the revised paper.


Thank you again for these valuable suggestions. We hope that these clarifications effectively address your concerns.

I thank the authors for the detailed rebuttal and the additional controlled experiments. I appreciate the discussion of SPEAR and DashengTokenizer, and the new ablation results in Tables R1 and R2 provide useful evidence supporting the effectiveness of SAP and SAE. I am largely satisfied with the response to my original concerns.

However, I still have a few follow-up questions and would appreciate further clarification.

Follow-up questions

  1. Contribution of general audio training data. While the newly added ablations demonstrate the effectiveness of SAP and SAE under matched training configurations, the contribution of incorporating general audio data itself remains unclear. Given that most of the semantic tokenizer baselines are trained primarily on speech data, the improved environmental sound class separation in Table 2 is somewhat expected. It would be interesting to understand how much of this improvement comes from the inclusion of general audio training data rather than the proposed methodology. If additional controlled experiments are not feasible, I would encourage the authors to acknowledge this factor when discussing the results.

  2. Supervision in Table R1. Could the authors provide an example of the unstructured caption targets and clarify whether they contain the same underlying information as SAP, particularly the ground-truth transcription and acoustic descriptions? This is important to establish whether the comparison isolates the effect of structured supervision rather than differences in the amount or quality of the supervision information. Additionally, since the ablation experiments use both LibriSpeech and the balanced training subset of AudioSet, how are ASR transcription targets obtained for AudioSet samples, particularly those containing no speech?

  3. Speech reconstruction in Table 3. It is unclear whether the baseline tokenizers use their original reconstruction decoders or whether separate flow-matching decoders are trained for all tokenizers under matched configurations. Since reconstruction performance depends on both the tokenizer and the decoder, differences in decoder architectures, training data, or training budgets could potentially confound the comparison. Please clarify the reconstruction pipeline used for each tokenizer.

  4. Speaker embedding conditioning in Section 5.2. The authors state that other tokenizers do not support speaker embedding conditioning. However, I would expect support for external speaker embeddings to depend primarily on the downstream generation architecture rather than the tokenizer itself. Could the authors clarify why the other tokenizers cannot be integrated with the same speaker-conditioned generation framework used for UniAudio-Token?

Regarding non-speech audio reconstruction, I understand the authors' design choice to focus on general audio understanding and speech generation. Nevertheless, I would encourage the authors to distinguish between retaining information useful for general audio understanding and faithfully preserving fine-grained acoustic details, as the latter is not directly established by the current evaluation.

I look forward to the response of the authors.

Reviewer_E7RaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniAudio-Token is a 25 Hz, single-codebook tokenizer designed for general audio understanding and speech generation. Starting from Whisper-large-v3, it trains on structured targets containing transcripts, vocal attributes, and sound events. Its SAE module gates shallow encoder features into deeper features before quantization. The paper evaluates sound-category clustering, speech reconstruction, audio understanding with Qwen2.5, and TTS against CosyVoice2.

Critical strengths
  1. The tokenizer performs well on both perception and speech reconstruction. ESC-50 cluster purity reaches 0.390, compared with 0.216 for the strongest baseline in Table 2. Its average speech-reconstruction WER is 3.68, versus 4.47 for StableToken at the same reported 325 bps. These results support the goal of retaining acoustic information alongside speech content. The reported MOS values are MOSNet predictions, not human listening scores.

  2. The gains carry through to downstream understanding. With the Qwen2.5 setup reported in the paper, UniAudio-Token has the highest overall scores among the tested tokenizers on MMAU, MMAR, and MMSU. The improvements in sound and music are especially relevant to the proposed general-audio interface.

Critical weaknesses
  1. The tokenizer-level benefit of SAP is not isolated. The cited Unified Audio Schema work already describes a closely related three-part schema, acoustic-caption-to-structured-label pipeline, and derived QA tasks. Applying this supervision to a quantized tokenizer is a meaningful distinction, but the paper does not compare SAP with transcription-only or unstructured-caption targets under the same tokenizer initialization, data, and budget. Comparisons with released tokenizers show that the complete system works well; they cannot establish how much of the gain comes from SAP rather than the additional training data and other choices.

  2. The evidence does not establish the value of SAE’s adaptive gate over simpler fusion. Table 6 compares SAE with no SAE and changes which shallow layer is used. Figure 4 shows that gate activations vary with the input. Neither result shows that content-dependent gating improves outcomes beyond adding the shallow features with a fixed or learned static weight. The published tokenizer checkpoints also differ in their original training resources, while the submission gives limited detail about the downstream Qwen2.5 and speech-reconstruction training. This limits attribution of the system-level improvements to SAE.

Questions for authors
  1. How do SAP and SAP-Instruct differ from the previously published UAS and UAS-QA targets and data pipeline? Do you have a tokenizer-level comparison of transcription-only, unstructured-caption, and SAP supervision with the same Whisper initialization, audio data, codebook, and training budget? A controlled subset would be informative if full-scale retraining is impractical. Downstream understanding and speech-reconstruction results would both matter.

  2. With SAP supervision held fixed, how does the adaptive SAE gate compare with simple fixed or learned static shallow-feature fusion? Please also specify the data, update counts, and budgets used for the downstream Qwen2.5 systems and flow-matching speech reconstructors, including what was held constant across tokenizers.

2 comments
AuthorsOpenReview ↗

Response to Reviewer E7Ra

We thank the reviewer for the constructive review and positive feedback. Below, we address your concerns in detail.


W1 & Q1: SAP vs. ASR Transcription & Unstructured Captions

Following your suggestion, we conduct strictly controlled ablation studies on a balanced subset (LibriSpeech training set + AudioSet bal_train subset) and downstream Qwen2.5-0.5B LLM with identical training configuration (initialization, training data, codebook size, frame rate, training steps, etc.), to isolate the exact contributions of SAP and SAE.

Table R1: SAP vs. ASR Transcription & Unstructured Captions

SupervisionMMAU SpeechMMAU SoundMMAU MusicMMAU OverallMMAR SpeechMMAR SoundMMAR MusicMMAR OverallMMSU PerceptionMMSU ReasoningMMSU Overall
ASR transcription42.3454.3548.2048.3038.7830.3033.5035.6029.3846.5737.70
Unstructured captions40.5454.6549.1048.1037.7629.7033.5033.7030.2746.9438.34
SAP (Ours)44.4458.5650.6051.2041.5032.1235.9237.1031.1647.9839.30

Analysis:

  • SAP outperforms both ASR transcription (+2.90% MMAU Overall, +1.50% MMAR Overall, +1.60% MMSU Overall) and unstructured captions (+3.10% MMAU Overall, +3.40% MMAR Overall, +0.96% MMSU Overall).
  • Notably, unstructured captions degrade speech understanding due to entangled, high-entropy text targets, whereas SAP improves both speech and non-speech simultaneously.

Due to limited computing resources and time, we have not yet completed training the flow-matching models for these ablated models. We will report the speech reconstruction results of these ablated models in the revised version.

While SAP and SAP-Instruct demonstrate the same principle of structured supervision as UAS and UAS-QA, the main technical extension is to use this supervision to learn the discrete VQ tokenization. Compressing audio into a compact single codebook creates a severe bottleneck where high-entropy text targets cause codebook collapse or degrade speech, which is a representation-level problem that cannot be resolved by downstream training alone. SAP resolves this through low-entropy structured fields and explicitly separates temporal event types to align with a low BPS tokenizer.

We will include these ablation results and discussion in the revised paper.


W2 & Q2: SAE vs. None & Additive Fusion

With SAP supervision, training data, and budget held strictly constant, we compared three fusion strategies: None (no shallow connection), Additive (static addition of shallow features), and SAE (content-aware gated fusion). The results are shown in Table R2.

Table R2: SAE vs. None & Additive Fusion

Feature FusionMMAU SpeechMMAU SoundMMAU MusicMMAU OverallMMAR SpeechMMAR SoundMMAR MusicMMAR OverallMMSU PerceptionMMSU ReasoningMMSU Overall
None44.1451.3544.3146.6040.4830.9131.5534.4028.6845.2136.68
Additive43.8453.7549.4049.0039.8030.3034.9534.8028.9545.5436.98
SAE (Ours)44.4458.5650.6051.2041.5032.1235.9237.1031.1647.9839.30

Analysis:

  • SAE outperforms None and Additive fusion across every single benchmark, demonstrating the necessity of adaptive gating.
  • Notably, static Additive fusion improves over None on sound/music but slightly degrades speech performance, indicating the impact of injecting noise features indiscriminately.

Training configuration for Qwen2.5 system and flow-matching speech reconstruction:

  • Downstream Audio-LLMs for Understanding: trained on the SAP-Instruct dataset for 5 epochs, which is identical for all the tokenizers. The LLM backbone, batch size and learning rate schedule all remain identical. Only the tokenizer frontend (including the frame rate and codebook vocabulary size) of each tokenizer is different.
  • The flow-matching model and the downstream Audio-LLM for TTS: trained on all the speech datasets listed in Table 7 for 2 epochs.

We will include these ablation results and complete downstream training configurations in the revised manuscript.


Thank you again for these valuable suggestions. We hope that these clarifications effectively address your concerns.

AuthorsOpenReview ↗

Gentle reminder & update on our revised manuscript

Dear Reviewer E7Ra,

With the discussion deadline approaching, we wanted to briefly notify you that we have posted our detailed response to your review above and uploaded the revised version of the paper, with all changes highlighted in blue for your convenience.

We sincerely appreciate your time and effort in reviewing our work, and we would be grateful if you could take a look at our response and the updated manuscript. We are happy to answer any remaining questions.

Best regards,

The Authors

Reviewer_ngFAOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes UniAudio-Token, a single-codebook tokenizer that captures both linguistic and general acoustic information. It combines structured supervision for linguistic content, vocal attributes, and auditory scenes (SAP) with a gating mechanism that fuses low-level acoustic and high-level semantic features (SAE). Experiments show promising results on audio understanding, speech reconstruction, and TTS.

Critical strengths
  1. Relevant problem. Combining semantic and acoustic information in compact audio tokens is an important problem for Audio-LLMs.

  2. Promising results. The proposed method achieves strong performance on general audio understanding while maintaining good speech reconstruction and TTS quality. The single-codebook design is particularly appealing.

Critical weaknesses
  1. Limited methodological novelty. The proposed approach essentially combines standard supervised learning with structured annotations (SAP) and a relatively simple gating mechanism to fuse low-level and high-level features (SAE). Both are reasonable engineering choices, but I do not see a particularly novel learning principle or architectural contribution. The work appears mostly incremental methodologically.

  2. Incomplete literature on hybrid tokenization. The related work should be substantially expanded to cover recent hybrid semantic-acoustic tokenizers. Several existing approaches already combine semantic and acoustic information, including within a single codebook (e.g., FocalCodec, UniCodec, X-Codec-2.0). This is important because the proposed tokenizer essentially belongs to this family of hybrid approaches. Table 1 should also be expanded to better reflect the current literature.

  3. Insufficient discussion of disentanglement. Disentangling linguistic content, speaker characteristics, prosody, and acoustic information is a well-established research direction. The authors should better acknowledge this literature (e.g., FACodec and related work). .

  4. Limited baseline comparisons. The main experimental comparison focuses on relatively few tokenizers. The authors should consider more recent single-codebook hybrid approaches, particularly those with comparable frame rates and bitrates. I acknowledge the additional comparisons in Appendix F, including X-Codec-2.0, but the evaluation could be broader and better aligned with recent developments in the field.

  5. Insufficient ablation studies. The contributions of SAP and SAE are not convincingly isolated. For SAP, it would be useful to compare structured supervision with conventional audio-caption supervision. For SAE, simple feature addition or concatenation would be relevant alternatives. A more systematic ablation would help establish whether the proposed components provide meaningful benefits beyond existing techniques.

  6. Oversimplified discussion of semantic tokenizers. The paper focuses heavily on ASR-supervised semantic tokenizers. However, many tokenizers rely on self-supervised representations, such as HuBERT or WavLM, which can preserve information beyond linguistic content. This distinction matters given the paper's motivation. Comparing an ASR-based initialization with a strong SSL encoder would provide useful insights.

  7. Incomplete evaluation of general representation quality. Although the Audio-LLM understanding results are valuable, the evaluation should include more direct assessments of the information preserved in the tokens, such as speaker recognition, emotion recognition, and language identification. These tasks are particularly relevant given SAP's supervision objectives. Furthermore, the reconstruction and generation experiments focus exclusively on speech, limiting the evidence for a universal audio interface.

Questions for authors
  • What are the main technical differences between UniAudio-Token and existing hybrid single-codebook tokenizers such as FocalCodec, UniCodec, and X-Codec-2.0? There should be a better discussion of that.

  • Could the authors provide stronger ablations, particularly SAP versus conventional audio-caption supervision and SAE versus simpler feature-fusion approaches?

  • Have the authors considered using an SSL encoder such as WavLM instead of Whisper? Could this naturally preserve more acoustic information?

  • Can the authors provide evaluations on additional tasks, particularly speaker recognition and emotion recognition, to substantiate the claim that the tokens retain richer acoustic and vocal information?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ngFA (1/3): W1, W2 & Q1, W3

We thank the reviewer for the thorough and insightful review. We appreciate your recognition of the importance of compact single-codebook representations for Audio-LLMs and our strong results on audio understanding, speech reconstruction, and TTS. Below, we address each of your concerns in detail.


W1: Methodological Novelty

We appreciate the opportunity to clarify the core scientific insight and methodological novelty of UniAudio-Token.

The technical contribution lies in how general audio perception is learned and retained in a compact semantic tokenizer. With a compact single codebook, broad acoustic coverage is a representation learning problem: the training objective must preserve relevant attributes, and the training architecture must support learning from conflictual acoustic and semantic features. UniAudio-Token resolves the core semantic-acoustic conflict within a compact single codebook via dual rectification:

  1. SAP decomposes audio into low-entropy orthogonal primitives, avoiding the semantic entanglement of waveform reconstruction or unstructured captions, which is confirmed by our new ablation in Table R2, where unstructured captions drop MMAU Speech 44.44% -> 40.54%, and MMAR Speech 41.50% -> 37.76%.

  2. SAE uses content-aware gating to inject shallow acoustics only when needed (e.g., low SNR or background music; see Figure 4 in our paper), preventing noise from degrading speech, which is confirmed by our new ablation in Table R3, where static Additive fusion degrades MMAU Speech 44.44% -> 43.84%, and MMAR Speech 41.50% -> 39.80%.

These dual rectification together establishes a paradigm for building unified single-codebook tokenizers without relying on multi-codebook RVQ or waveform reconstruction losses.


W2 & Q1: Comparison with Recent Tokenizers (FocalCodec, UniCodec, X-Codec-2.0)

Thank you for pointing out these relevant methods, and we will expand our discussion and include them in the revised paper.

While these methods claim to unify semantic and acoustic cues within a single codebook, they still fall short of a truly unified audio tokenizer in key aspects:

  1. Speech-Centric Scope (FocalCodec, X-Codec-2.0): Both are evaluated exclusively on speech tasks without general sound or music benchmarks. At low bitrates, reconstruction objectives cause non-speech semantic entanglement (see Table 9 in our paper; UniAudio-Token outperforms X-Codec-2.0 on ESC-10 and ESC-50 with a much more compact tokenizer).

  2. Domain-Partitioned Codebook (UniCodec): UniCodec partitions its codebook into separate domain-specific subspaces and requires explicit domain specification during inference, making it inflexible for mixed audio containing overlapping speech, sound, and music.

  3. Our Unified Paradigm: Unlike these methods that all primarily rely on waveform reconstruction losses and speech-centric SSL distillation, UniAudio-Token uses SAP structured supervision across all modalities and SAE content-aware dynamic gating within a single, domain-agnostic codebook, handling arbitrary speech, sound, music, and their mixtures seamlessly.


W3: Disentanglement (FACodec)

Thank you for your insightful suggestion. We will cite and discuss FACodec and related work in Section 2.

FACodec achieves impressive disentanglement of content, prosody, timbre, and acoustic details by assigning them to separate codebooks in a multi-codebook architecture. However, in a single-codebook setting, physical separation across distinct codebooks is not possible. In contrast, UniAudio-Token resolves this constraint and achieves disentanglement within a single codebook by enforcing structural disentanglement at the supervision level (SAP).


Due to character limitations, our response continues in the following comment.

AuthorsOpenReview ↗

Response to Reviewer ngFA (2/3): W4, W5 & Q2

Continuing from the previous comment, here we provide our responses to the rest questions.


W4: Baselines

Thank you for your constructive feedback.

Our comparisons mainly cover single-codebook tokenizers, including compact semantic speech tokenizers and a general audio acoustic tokenizer. Appendix F extends the evaluation to some multi-codebook or hybrid designs to better position our work. These comparisons support the favorable balance between general audio perception, speech reconstruction, and compactness achieved by our method.

Following your suggestion, we also add comparison with FocalCodec and UniCodec in Table R1 and Table R4 (in our response to W7 & Q4).

Table R1: Clustering analysis on ESC-10 and ESC-50

ModelESC-10 Silhouette ↑\uparrowESC-10 Purity ↑\uparrowESC-50 Silhouette ↑\uparrowESC-50 Purity ↑\uparrow
FocalCodec0.0500.1850.0130.144
UniCodec0.0100.3030.0050.174
UniAudio-Token (Ours)0.0910.7300.0230.390

UniAudio-Token achieves consistently higher Silhouette Scores and Cluster Purity on both ESC-10 and ESC-50 compared to recent hybrid single-codebook tokenizers (FocalCodec and UniCodec). This confirms that our discrete tokens form valid and well-defined boundaries for environmental sounds, preventing the feature fragmentation and overlap seen in domain-partitioned or reconstruction-heavy baselines. We will include these comparisons and discussion with recent single-codebook approaches in the revised paper.


W5 & Q2: Ablations

Due to the massive scale of our full training corpus and limited computational resources and time, we conduct strictly controlled ablation studies on a balanced subset (LibriSpeech training set + AudioSet bal_train subset) and downstream Qwen2.5-0.5B LLM with identical training configuration (initialization, training data, codebook size, frame rate, training steps, etc.), to isolate the exact contributions of SAP and SAE.

SAP vs. ASR Transcription & Unstructured Captions

Table R2: SAP vs. ASR Transcription & Unstructured Captions

SupervisionMMAU SpeechMMAU SoundMMAU MusicMMAU OverallMMAR SpeechMMAR SoundMMAR MusicMMAR OverallMMSU PerceptionMMSU ReasoningMMSU Overall
ASR transcription42.3454.3548.2048.3038.7830.3033.5035.6029.3846.5737.70
Unstructured captions40.5454.6549.1048.1037.7629.7033.5033.7030.2746.9438.34
SAP (Ours)44.4458.5650.6051.2041.5032.1235.9237.1031.1647.9839.30

Analysis:

  • SAP outperforms both ASR transcription (+2.90% MMAU Overall, +1.50% MMAR Overall, +1.60% MMSU Overall) and unstructured captions (+3.10% MMAU Overall, +3.40% MMAR Overall, +0.96% MMSU Overall).
  • Notably, unstructured captions degrade speech understanding due to entangled, high-entropy text targets, whereas SAP improves both speech and non-speech simultaneously.

SAE vs. None & Additive Fusion

Table R3: SAE vs. None & Additive Fusion

Feature FusionMMAU SpeechMMAU SoundMMAU MusicMMAU OverallMMAR SpeechMMAR SoundMMAR MusicMMAR OverallMMSU PerceptionMMSU ReasoningMMSU Overall
None44.1451.3544.3146.6040.4830.9131.5534.4028.6845.2136.68
Additive43.8453.7549.4049.0039.8030.3034.9534.8028.9545.5436.98
SAE (Ours)44.4458.5650.6051.2041.5032.1235.9237.1031.1647.9839.30

Analysis:

  • SAE outperforms None and Additive fusion across every single benchmark, demonstrating the necessity of adaptive gating.
  • Notably, static Additive fusion improves over None on sound/music but slightly degrades speech performance, indicating the impact of injecting noise features indiscriminately.

These controlled experiments confirm that the improvements in universal audio understanding are driven directly by SAP's structured disentanglement and SAE's adaptive gating. We will include these ablation results and discussion in the revised paper.


Due to character limitations, our response continues in the following comment.

AuthorsOpenReview ↗

Response to Reviewer ngFA (3/3): W6 & Q3, W7 & Q4

Continuing from the previous comment, here we provide our responses to the rest questions.


W6 & Q3: Comparison of ASR model and SSL Encoders

We appreciate this thoughtful question. While SSL models like WavLM retain rich acoustic and paralinguistic cues in intermediate layers, we chose Whisper-large-v3 as our initialization for three reasons tailored to Audio-LLMs:

  1. Explicit Text-Audio Alignment: SSL encoders are trained solely via self-supervised masked prediction without text supervision, requiring substantial alignment effort for integration with LLMs, especially across multiple languages. Whisper is pretrained on large-scale multilingual ASR, providing a strong linguistic alignment prior.

  2. Universal Deep-Layer Acoustic Bottleneck: Previous work [1][2] on both Whisper and SSL encoders (e.g., WavLm/HuBERT) shows that deeper layers progressively abstract away low-level acoustic details in favor of semantic content, while shallow layers preserve fine-grained acoustics. Thus, the architectural bottleneck addressed by SAE applies broadly across deep speech encoders.

  3. Empirical Effectiveness: Combining Whisper initialization with SAP and SAE allows us to inherit Whisper's strong linguistic alignment while recovering the exact paralinguistic and acoustic cues needed for general audio perception and speaker/emotion preservation (as verified in our new experiments in Table R4)

We will expand Section 2 to discuss SSL-based tokenizers more thoroughly and clarify the distinction between ASR-oriented and SSL-based semantic tokenizers.


W7 & Q4: Direct Evaluation of Paralinguistic Tasks

Speaker Verification (SV) and Speech Emotion Recognition (SER): Following the DASB benchmark protocol [3], we evaluate the discrete tokens directly on VoxCeleb1 (SV) and IEMOCAP (SER) (Table R4). UniAudio-Token achieves 15.46% EER on SV and 52.66% accuracy on SER (both outperforming all the baselines). This provides direct evidence that SAP's vocal attribute supervision and SAE fusion preserve speaker timbre and emotional cues in the discrete tokens.

Table R4: Token-level Evaluation on Speaker Verification (SV) and Speech Emotion Recognition (SER)

ModelBPSSV EER (↓)(\downarrow)SER Acc (↑)(\uparrow)
WavTokenizer90019.38±0.3643.43±2.41
GLM-4-Voice-Tokenizer17526.70±0.3948.46±2.91
CosyVoice232518.54±0.4846.58±2.10
StableToken32523.32±0.5846.35±1.52
FocalCodec32516.72±0.1643.09±3.37
UniCodec1,05015.70±0.3845.79±1.87
UniAudio-Token (Ours)32515.46±0.1552.66±0.76

For non-speech reconstruction and generation, since UniAudio-Token is designed as a compact single-codebook interface optimized for universal audio understanding and high-fidelity speech interaction, reconstructing complex music or arbitrary environmental sounds is beyond its scope (which typically requires multi-codebook RVQ at much higher BPS). Nevertheless, our ESC-50 clustering (Table 2) and strong gains on MMAU/MMAR Sound/Music and MMSU Perception (Table 4) directly confirm that non-speech acoustic details are faithfully preserved in the token representations. We will explicitly clarify this limitation and design choice in the revised paper.


Thank you again for these valuable suggestions. We hope that these clarifications effectively address your concerns.

References:

[1] Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggers

[2] Comparative layer-wise analysis of self-supervised speech models

[3] DASB - Discrete Audio and Speech Benchmark

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.