- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Universal FocalCodec (UFC), a low-bitrate neural audio codec for speech, music, and environmental sounds. Building on FocalCodec, UFC uses intermediate features from a pretrained Qwen3-Omni Audio Transformer (AuT) and compresses them into a single stream of discrete tokens. The authors investigate different AuT layers, quantization methods, and token rates. Experiments show that UFC achieves competitive audio reconstruction quality across multiple audio domains and performs favorably compared with existing universal audio codecs.
Critical strengths
-
The paper extends low-bitrate neural audio compression to a broader range of audio domains, including speech, music, and environmental sounds.
-
The results show that UFC remains competitive with existing universal audio codecs across several evaluation settings.
-
The paper provides comprehensive ablation studies on key design choices, including AuT layers, quantization methods, token rates, and perceptual losses.
Critical weaknesses
-
The paper does not directly evaluate whether UFC improves downstream audio language modeling or generation.
-
Although UFC performs competitively across multiple audio domains, the reconstruction results suggest that its strengths and limitations are not uniform across speech, music, and environmental sounds.
-
While the paper includes extensive ablations of UFC's design choices, the contribution of individual modifications relative to the original FocalCodec could be clarified further.
Questions for authors
-
Does UFC's discrete representation benefit downstream audio language modeling beyond its demonstrated audio reconstruction performance?
-
What factors might explain the differences in reconstruction quality across speech, music, and environmental sounds, and how could these differences be further reduced?
-
Could the authors provide a more controlled comparison with the original FocalCodec to better isolate the contributions of the proposed modifications?
5 comments
Response to Q1, W1: Benefits for Downstream Audio Language Modeling
We acknowledge that the original paper did not directly evaluate UFC within an audio language model (ALM). To better assess its potential benefits, we conducted two additional sets of experiments:
-
Downstream probing experiments. To assess the information retained in discrete NAC tokens independently of waveform reconstruction quality, we follow the shallow-probing protocol of FocalCodec and DASB, training 2-layer BiLSTM probes on six tasks:
- Speech: Automatic Speech Recognition (ASR) and Speaker Identification (SI) on LibriSpeech.
- Music: Music Genre Classification (MGC) on FMA-Small, and Pitch Detection (NPD) and Instrument Classification (NIC) on NSynth.
- Sound: Sound Event Classification (SEC) on FSD50K.
We evaluate UFC variants alongside other universal NACs. Results are presented in new Section 5.3 and Figure 6, with the protocol detailed in Appendix H and complete results in Table 14 (Appendix I).
Overall, UFC-VQ performs best among our three quantization schemes on five of six tasks, with NPD at 195 bps being the exception. At 195 bps, UFC-VQ achieves the best MGC and SEC results among all evaluated NACs, including substantially higher-bitrate alternatives. At 390 bps, it outperforms all 480–550 bps baselines on MGC, NIC, and SEC. At 780 bps, UFC-VQ remains competitive across downstream tasks.
Although these experiments do not directly integrate UFC into an ALM, they demonstrate that UFC tokens retain semantic and acoustic information accessible to lightweight sequence models, supporting their potential use in Transformer-based ALMs.
-
Codec and autoregressive inference benchmarks. In Appendix G.1, we show that single-codebook NACs, including UFC, generally achieve lower encoding/decoding RTF than multi-codebook alternatives. In Appendix G.2, we demonstrate that single-codebook AR prediction is faster than flattened multi-codebook prediction and achieves comparable speed to delayed parallel prediction at similar frame rates. Unlike delayed parallel prediction, however, single-codebook NACs follow the conventional one-token-per-step paradigm, potentially simplifying LM training and integration into existing inference frameworks (e.g., vLLM) without specialized multi-codebook adaptations.
Combined, these findings suggest that single-codebook NACs may offer faster end-to-end inference due to lower codec runtime, even when AR generation speeds are comparable.
Overall, our new experiments provide evidence for both the downstream utility of UFC tokens and the computational advantages of single-codebook NACs. While direct ALM integration remains future work, UFC's competitive reconstruction and downstream performance, efficient codec architecture, and conventional token prediction scheme make it an attractive candidate for ALM applications.
Response to Q2, W2: Differences in Reconstruction Quality Across Domains
We thank the reviewer for raising this question. Since different audio domains are evaluated on different datasets and metrics, absolute reconstruction quality cannot be directly compared across domains. However, relative to other NACs evaluated under the same conditions, UFC achieves competitive or superior performance across speech, music, and environmental sound.
For environmental sound, we hypothesize that the lower SIM scores are partly due to the training data distribution, which contains substantially more speech and music than environmental sound. Increasing the diversity and amount of sound training data could potentially further improve semantic preservation.
For speech, our speech-only ablation (Appendix D) demonstrates that training exclusively on speech improves dWER for BSQ/FSQ, particularly at lower frame rates. This suggests interference between audio domains during universal training, consistent with the domain-disentanglement challenge reported in the literature. Importantly, UFC-52 performs similarly under universal and speech-only training, demonstrating UFC's robustness and that a global single-codebook representation can work effectively without explicit domain-aware partitioning. Our SharedVQ ablation (Appendix E) further investigates this issue using a domain-aware codebook design inspired by prior split and nested quantization approaches. While SharedVQ provides some additional benefits at 52 Hz, it offers no consistent improvements at lower frame rates, highlighting the need for alternative domain-disentanglement approaches at ultra-low token rates.
Additionally, following a data-overlap issue identified during the review, we replaced SongDescriber with the official FMA-Medium test split across all music evaluations. The updated results are reported in Figure 4, Table 7 (SharedVQ ablation), and Table 10 (full NAC comparison). This change does not alter our main findings or conclusions, with UFC remaining highly competitive relative to other NACs. We accordingly revised Section 5.2 and the SharedVQ discussion in Appendix E (lines 1103–1106).
General Thank You
We thank the reviewer for their constructive feedback and positive assessment of our work, particularly for recognizing UFC's competitive performance across speech, music, and environmental sound, its extension of low-bitrate audio compression to multiple domains, and the value of our comprehensive ablation studies. We appreciate the thoughtful questions, which motivated additional experiments and clarifications to strengthen our work. We address each concern in detail below.
For ease of review, changes in the revised PDF are highlighted in blue: individual modifications are highlighted directly, while entirely new sections are indicated by blue section headings.