acceptodds
Under review as a conference paper at ICLR 2027

UFC: Universal FocalCodec for Ultra-low Bitrate Audio Compression

Abstract

Neural audio codecs (NACs) provide compact discrete representations for audio language models (ALMs). However, many NACs rely on multiple codebooks, increasing token rates and often requiring a two-stage autoregressive and non-autoregressive decoding procedure, and are limited to speech. We propose Universal FocalCodec (UFC), a family of single-codebook NACs for 16 kHz speech, music, and sound. UFC builds on the Qwen3-Omni Audio Transformer (AuT) encoder and quantizes its intermediate representations into a single discrete token stream. We introduce variants operating at frame rates from 13 to 52 Hz, corresponding to bitrates of 0.195 to 0.780 kbps, offering a speed-quality trade-off. Across multiple domains and evaluation metrics, UFC is competitive with state-of-the-art universal codecs. UFC-13 achieves the strongest overall performance among universal codecs operating around 0.195 kbps, leading on most metrics and outperforming several higher-bitrate systems. Our ablations further reveal that early AuT layers provide a favorable balance of semantic and acoustic information for neural audio coding. UFC shows that ALMs can reuse their own audio encoders to generate audio instead of relying on a separate decoding methodology.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Neural Audio Codecs, Language Modelling, Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bu53OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Universal FocalCodec (UFC), a low-bitrate neural audio codec for speech, music, and environmental sounds. Building on FocalCodec, UFC uses intermediate features from a pretrained Qwen3-Omni Audio Transformer (AuT) and compresses them into a single stream of discrete tokens. The authors investigate different AuT layers, quantization methods, and token rates. Experiments show that UFC achieves competitive audio reconstruction quality across multiple audio domains and performs favorably compared with existing universal audio codecs.

Critical strengths
  1. The paper extends low-bitrate neural audio compression to a broader range of audio domains, including speech, music, and environmental sounds.

  2. The results show that UFC remains competitive with existing universal audio codecs across several evaluation settings.

  3. The paper provides comprehensive ablation studies on key design choices, including AuT layers, quantization methods, token rates, and perceptual losses.

Critical weaknesses
  1. The paper does not directly evaluate whether UFC improves downstream audio language modeling or generation.

  2. Although UFC performs competitively across multiple audio domains, the reconstruction results suggest that its strengths and limitations are not uniform across speech, music, and environmental sounds.

  3. While the paper includes extensive ablations of UFC's design choices, the contribution of individual modifications relative to the original FocalCodec could be clarified further.

Questions for authors
  1. Does UFC's discrete representation benefit downstream audio language modeling beyond its demonstrated audio reconstruction performance?

  2. What factors might explain the differences in reconstruction quality across speech, music, and environmental sounds, and how could these differences be further reduced?

  3. Could the authors provide a more controlled comparison with the original FocalCodec to better isolate the contributions of the proposed modifications?

5 comments
AuthorsOpenReview ↗

General Thank You

We thank the reviewer for their constructive feedback and positive assessment of our work, particularly for recognizing UFC's competitive performance across speech, music, and environmental sound, its extension of low-bitrate audio compression to multiple domains, and the value of our comprehensive ablation studies. We appreciate the thoughtful questions, which motivated additional experiments and clarifications to strengthen our work. We address each concern in detail below.

For ease of review, changes in the revised PDF are highlighted in blue: individual modifications are highlighted directly, while entirely new sections are indicated by blue section headings.

AuthorsOpenReview ↗

Response to Q1, W1: Benefits for Downstream Audio Language Modeling

We acknowledge that the original paper did not directly evaluate UFC within an audio language model (ALM). To better assess its potential benefits, we conducted two additional sets of experiments:

  1. Downstream probing experiments. To assess the information retained in discrete NAC tokens independently of waveform reconstruction quality, we follow the shallow-probing protocol of FocalCodec and DASB, training 2-layer BiLSTM probes on six tasks:

    • Speech: Automatic Speech Recognition (ASR) and Speaker Identification (SI) on LibriSpeech.
    • Music: Music Genre Classification (MGC) on FMA-Small, and Pitch Detection (NPD) and Instrument Classification (NIC) on NSynth.
    • Sound: Sound Event Classification (SEC) on FSD50K.

    We evaluate UFC variants alongside other universal NACs. Results are presented in new Section 5.3 and Figure 6, with the protocol detailed in Appendix H and complete results in Table 14 (Appendix I).

    Overall, UFC-VQ performs best among our three quantization schemes on five of six tasks, with NPD at 195 bps being the exception. At 195 bps, UFC-VQ achieves the best MGC and SEC results among all evaluated NACs, including substantially higher-bitrate alternatives. At 390 bps, it outperforms all 480–550 bps baselines on MGC, NIC, and SEC. At 780 bps, UFC-VQ remains competitive across downstream tasks.

    Although these experiments do not directly integrate UFC into an ALM, they demonstrate that UFC tokens retain semantic and acoustic information accessible to lightweight sequence models, supporting their potential use in Transformer-based ALMs.

  2. Codec and autoregressive inference benchmarks. In Appendix G.1, we show that single-codebook NACs, including UFC, generally achieve lower encoding/decoding RTF than multi-codebook alternatives. In Appendix G.2, we demonstrate that single-codebook AR prediction is faster than flattened multi-codebook prediction and achieves comparable speed to delayed parallel prediction at similar frame rates. Unlike delayed parallel prediction, however, single-codebook NACs follow the conventional one-token-per-step paradigm, potentially simplifying LM training and integration into existing inference frameworks (e.g., vLLM) without specialized multi-codebook adaptations.

    Combined, these findings suggest that single-codebook NACs may offer faster end-to-end inference due to lower codec runtime, even when AR generation speeds are comparable.

Overall, our new experiments provide evidence for both the downstream utility of UFC tokens and the computational advantages of single-codebook NACs. While direct ALM integration remains future work, UFC's competitive reconstruction and downstream performance, efficient codec architecture, and conventional token prediction scheme make it an attractive candidate for ALM applications.

AuthorsOpenReview ↗

Response to Q2, W2: Differences in Reconstruction Quality Across Domains

We thank the reviewer for raising this question. Since different audio domains are evaluated on different datasets and metrics, absolute reconstruction quality cannot be directly compared across domains. However, relative to other NACs evaluated under the same conditions, UFC achieves competitive or superior performance across speech, music, and environmental sound.

For environmental sound, we hypothesize that the lower SIM scores are partly due to the training data distribution, which contains substantially more speech and music than environmental sound. Increasing the diversity and amount of sound training data could potentially further improve semantic preservation.

For speech, our speech-only ablation (Appendix D) demonstrates that training exclusively on speech improves dWER for BSQ/FSQ, particularly at lower frame rates. This suggests interference between audio domains during universal training, consistent with the domain-disentanglement challenge reported in the literature. Importantly, UFC-52 performs similarly under universal and speech-only training, demonstrating UFC's robustness and that a global single-codebook representation can work effectively without explicit domain-aware partitioning. Our SharedVQ ablation (Appendix E) further investigates this issue using a domain-aware codebook design inspired by prior split and nested quantization approaches. While SharedVQ provides some additional benefits at 52 Hz, it offers no consistent improvements at lower frame rates, highlighting the need for alternative domain-disentanglement approaches at ultra-low token rates.

Additionally, following a data-overlap issue identified during the review, we replaced SongDescriber with the official FMA-Medium test split across all music evaluations. The updated results are reported in Figure 4, Table 7 (SharedVQ ablation), and Table 10 (full NAC comparison). This change does not alter our main findings or conclusions, with UFC remaining highly competitive relative to other NACs. We accordingly revised Section 5.2 and the SharedVQ discussion in Appendix E (lines 1103–1106).

Reviewer_3VtxOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Universal FocalCodec (UFC), a family of single-codebook neural audio codecs inspired by FocalCodec, achieving highly competitive reconstruction performance at extremely low frame rates and bitrates across speech, music, and environmental sounds. The proposed approach builds on the audio transformer of Qwen3-Omni, demonstrating that its learned representations can be effectively leveraged for audio compression and resynthesis. The paper presents an extensive experimental evaluation, including comparisons with existing neural audio codecs and comprehensive ablation studies investigating the choice of transformer layers, quantization methods, codec architectures, frame rates, bitrates, and multi-layer feature fusion.

Critical strengths
  • Excellent presentation. The paper is very well written, clearly structured, and easy to follow.

  • Well-designed system. The proposed architecture is carefully designed and achieves an excellent trade-off between reconstruction quality and bitrate.

  • Strong reconstruction performance. UFC achieves highly competitive results across speech, music, and environmental sounds, with most evaluation metrics lying on or near the Pareto frontier. It is primarily outperformed by substantially larger codecs, such as MOSS, which is understandable given the significant differences in model size and training data.

  • Comprehensive analysis of Qwen3-Omni representations. The analysis of representations across different layers of the Qwen3-Omni audio transformer is particularly interesting and provides useful insights into their suitability for audio compression and resynthesis.

  • Extensive ablation studies. The paper includes a thorough experimental investigation of several design choices, including quantization methods, codec architectures, frame rates, bitrates, and multi-layer feature fusion. These experiments provide valuable insights into the factors influencing codec performance.

Critical weaknesses
  • Limited methodological novelty. Most of the techniques employed in UFC have already been explored in the literature, particularly in FocalCodec and related neural audio codecs. Nevertheless, demonstrating that the FocalCodec design can be successfully adapted to Qwen3-Omni is an interesting contribution. The paper also provides novel insights into the properties of Qwen3-Omni audio representations and their suitability for compression and resynthesis.

  • Lack of downstream probing experiments. This is my main concern with the paper. One of its central claims is that UFC captures both semantic and acoustic information within a single stream of discrete tokens. However, the experimental evaluation does not sufficiently substantiate this claim. In particular, I believe it is important to evaluate how well the learned discrete representations work on downstream semantic tasks, such as automatic speech recognition (ASR), and acoustic tasks, such as speaker identification, using lightweight probing models. Similar evaluation protocols have been adopted in FocalCodec and DASB (https://arxiv.org/abs/2406.14294). Reconstruction metrics alone, including dWER, are generally insufficient to establish the semantic quality of the learned representations. Good intelligibility in reconstructed speech may simply result from accurate acoustic reconstruction, rather than from the representations capturing higher-level linguistic information. The extent to which UFC tokens preserve semantic information remains unclear. Ideally, if the tokens effectively capture both semantic and acoustic information, a shallow probing model should achieve competitive performance on both ASR and speaker identification tasks. I consider this an important missing piece of the evaluation, especially given the paper's emphasis on jointly capturing semantic and acoustic information.

Questions for authors
  • Evaluate downstream semantic and acoustic tasks. As discussed above, I strongly recommend including some lightweight probing experiments to assess the semantic and acoustic information retained by UFC tokens. In particular, semantic probing tasks such as ASR for speech or music genre classification for music would be especially valuable, as the current experiments already provide substantial evidence of strong acoustic reconstruction performance. Such experiments would help substantiate one of the paper's central claims of encoding semantic and acoustic in a single stream and provide a more complete understanding of the learned representations.

  • Subjective evaluation and audio samples. While the objective evaluation is already comprehensive, including a human listening test and providing audio samples on an anonymous project website would further strengthen the paper by allowing a more direct assessment of perceptual quality.

5 comments
AuthorsOpenReview ↗

General Thank You

We thank the reviewer for their detailed and constructive feedback, and for recognizing the clarity of our presentation, UFC's excellent trade-off between reconstruction quality and bitrate, its strong overall reconstruction performance, and the value of our extensive ablation studies and analysis of Qwen3-Omni representations. We particularly appreciate the suggestion to evaluate downstream tasks, which motivated a substantial extension of our experimental evaluation. We address each concern in detail below.

For ease of review, changes in the revised PDF are highlighted in blue: individual modifications are highlighted directly, while entirely new sections are indicated by blue section headings.

AuthorsOpenReview ↗

Response to Q1, W1: Downstream Probing Experiments

We thank the reviewer for this suggestion. Although our original evaluation included content-preservation metrics (e.g., dWER and speaker similarity for speech, semantic similarity for music and sound), these may be influenced by acoustic reconstruction quality.

To directly assess the semantic and acoustic information retained in discrete NAC tokens, we conducted six downstream tasks, following the shallow-probing protocol of FocalCodec and DASB, using 2-layer BiLSTM probes:

  • Speech: Automatic Speech Recognition (ASR) and Speaker Identification (SI) on LibriSpeech.
  • Music: Music Genre Classification (MGC) on FMA-Small, and Pitch Detection (NPD) and Instrument Classification (NIC) on NSynth.
  • Sound: Sound Event Classification (SEC) on FSD50K.

We evaluated UFC variants alongside other universal NACs. The results are presented in new Section 5.3 and Figure 6, with the evaluation protocol detailed in Appendix H and complete results reported in Table 14 (Appendix I).

Overall, UFC-VQ performs best among our three quantization schemes on five of six tasks, with NPD at 195 bps being the exception. At 195 bps, UFC-VQ achieves the best MGC and SEC results among all evaluated NACs, including substantially higher-bitrate alternatives. At 390 bps, it outperforms all 480–550 bps baselines on MGC, NIC, and SEC. At 780 bps, UFC-VQ remains competitive across downstream tasks.

These results provide additional evidence that UFC preserves both semantic and acoustic information in a single discrete token stream, beyond what can be inferred from waveform reconstruction metrics alone.

AuthorsOpenReview ↗

Response to Q2: Listening Tests and Audio Samples

We agree that human listening tests would further strengthen the evaluation. However, as discussed in Section 4.3, Mack et al. (ICASSP 2026) report strong correlations between MUSHRA scores and objective metrics, including UTMOS and reference-based SCOREQ (correlation of -0.86). They also identify Aesthetics CE among the strongly correlated metrics, although their evaluation focuses on speech. Accordingly, we employ UTMOS and reference-based SCOREQ for speech, and Aesthetics CE for music and environmental sound.

While these metrics cannot fully replace subjective evaluation, they provide useful indicators of perceptual quality. To facilitate direct listening comparisons, we have uploaded a supplementary archive containing two samples per audio domain, including the originals and reconstructions from all evaluated codecs (up to 1100 bps) at their respective bitrates. The accompanying README describes the archive structure. The 100 MB supplementary material limit and double-blind review constraints restricted the number of samples we could provide.

Finally, we are committed to releasing all model weights and training/inference code upon publication, enabling reproducible evaluation and generation of additional samples. We also plan to publish an extended online audio demo.

Reference:

  • Mack, W., Topaloglu, N., Lechler, L., Balić, I., Craciun, A., Yesilbursa, M., & Wojcicki, K. (2026, May). Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 22457-22461). IEEE.
Reviewer_Amd2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Universal FocalCodec (UFC), a single-codebook neural codec for 16 kHz speech, music, and environmental sound. UFC reconstructs audio from early layers of the frozen Qwen3-Omni audio encoder, then learns to quantize these representations. Its variants operate at 13-52 tokens/s and 195-780 bps. Contributions include extending FocalCodec to universal audio, comparing quantization methods, and analyzing encoder layers and domain interference. Evaluation on LibriSpeech, Multilingual LibriSpeech, SongDescriber, and AudioSet demonstrates promising compression-quality trade-offs using automatic metrics. The authors also commit to a fully open release of data, weights, and training/inference code.

Critical strengths
  1. Promising universal compression at very low token rates. UFC represents three audio domains with one codebook and achieves competitive automatic quality scores at 195 bps. This is a useful result for applications where representation length is a major constraint.
  2. Extensive, informative ablations. Comparisons across encoder layers, temporal resolutions, quantizers, perceptual losses, and speech-only versus universal training help identify where performance gains and information loss arise. These findings make the paper useful beyond the particular model.
Critical weaknesses
  1. Perceptual fidelity at the headline bitrate remains insufficiently validated. There are no listening tests, despite disagreement among automatic metrics and acknowledged metric biases. The disagreement is clearest between reference-free and reference-based metrics. Relative to the non-quantized UFC-NQ, UFC-13 keeps essentially the same UTMOS (3.77-3.95 vs. 3.87), while the reference-based PESQ drops from 3.73 to 1.56-1.59 and STOI from 0.97 to 0.82-0.83 (Table 8). Because UTMOS is reference-free, it rates how natural the output sounds rather than whether it matches the input, so it cannot by itself establish codec fidelity. For music and sound, the Aesthetics metrics are reference-free and measure quality rather than faithfulness to the input, so they cannot replace a reference-based listening test such as MUSHRA. This limits confidence that the 195 bps operating point preserves both perceptual quality and semantics.
  2. The claimed language-model benefits are untested, and the motivation overstates limitations of multiple codebooks. No downstream generation, understanding, or end-to-end efficiency experiment demonstrates the proposed advantage. Multiple codebooks also do not inherently require a separate non-autoregressive stage: MusicGen uses a single-stage model with delayed parallel codebook prediction. Consequently, token count alone does not establish the claimed computational benefit.
Questions for authors
  1. Were all SongDescriber tracks excluded from MTG-Jamendo training data in both training stages? SongDescriber samples recordings from MTG-Jamendo's split-0 test set, making explicit track-level separation important. Table 1 lists 49,978 MTG tracks (3,394 h), close to the 55,701 tracks in the MTG-Jamendo raw_30s release, and the paper does not mention excluding the test split.
  2. Is listening-test evidence available comparing UFC-13 with nearby-bitrate baselines, assessing both naturalness and faithfulness to the original content?
  3. Is there downstream or measured runtime evidence supporting the proposed language-model advantages, particularly against delayed parallel codebook prediction? Otherwise, can these advantages be framed as potential benefits?
  4. The layer analysis in Section 5.3 uses only LibriSpeech. Was layer 4 also validated on music and environmental sound, given that the abstract generalizes this finding to neural audio coding in general?
6 comments
AuthorsOpenReview ↗

General Thank You

We thank the reviewer for their detailed and constructive feedback, and for recognizing UFC's promising universal compression performance at ultra-low bitrates, as well as the value of our extensive ablation studies. We appreciate the thoughtful questions, which motivated several important experiments and clarifications. We address each concern in detail below.

For ease of review, changes in the revised PDF are highlighted in blue: individual modifications are highlighted directly, while entirely new sections are indicated by blue section headings.

AuthorsOpenReview ↗

Response to Q1: SongDescriber Data Contamination

We thank the reviewer for identifying this issue. Although UFC was trained exclusively on the training partitions, further investigation revealed that the MTG-Jamendo split provided on Hugging Face contains tracks overlapping with SongDescriber in its training partition.

To eliminate potential data leakage, we replaced SongDescriber with the official FMA-Medium test split for music evaluation across all experiments. The updated results are reported in Figure 4 and Table 7 (SharedVQ ablation) and Table 10 (full NAC comparison).

Importantly, this change does not alter our main findings or conclusions. UFC achieves even more competitive STFT/Mel reconstruction metrics on FMA-Medium compared to the original SongDescriber evaluation. We revised Section 5.2 and the SharedVQ discussion in the Appendix (lines 1103–1106) to reflect the updated results.

AuthorsOpenReview ↗

Response to Q2, W1: Listening Tests

As discussed in Section 4.3, Mack et al. (ICASSP 2026) demonstrate strong correlations between MUSHRA scores and objective metrics, including UTMOS and reference-based SCOREQ (with a reported correlation of -0.86). They also identify Aesthetics CE among the strongly correlated metrics, although their evaluation focuses on speech. Accordingly, we use UTMOS and reference-based SCOREQ for speech, and Aesthetics CE for music and general sound.

While we acknowledge that these metrics cannot fully replace subjective listening tests, they provide established perceptual quality indicators. To further facilitate qualitative assessment, we have uploaded a supplementary archive containing two audio samples per domain, reconstructed by every evaluated codec (up to 1100 bps) at its respective bitrate, alongside the originals. The included README explains the archive structure. The 100 MB supplementary material limit and double-blind review constraints prevented us from including a larger collection.

Finally, we are committed to releasing all model weights and training/inference code upon publication, enabling reproducible evaluation and generation of additional samples. We also plan to publish an extended online audio demo.

Reference:

  • Mack, W., Topaloglu, N., Lechler, L., Balić, I., Craciun, A., Yesilbursa, M., & Wojcicki, K. (2026, May). Assessing speech quality metrics for evaluation of neural audio codecs under clean speech conditions. In ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 22457-22461). IEEE.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.