- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes CleanCodec, a neural speech codec that improves tokenization efficiency by encouraging the preservation of perceptually important information. Its key idea is a denoising training objective that reconstructs clean speech from stochastically degraded inputs, reducing the encoding of irrelevant acoustic information. CleanCodec combines discrete local tokens with global speaker representations, supported by auxiliary semantic and speaker supervision. Experiments demonstrate strong performance in speech reconstruction, denoising, and downstream TTS and voice conversion tasks, particularly at a low token rate of 12.5 tokens/s.
Critical strengths
-
CleanCodec achieves strong speech reconstruction quality at only 12.5 tokens/s, substantially outperforming comparable low-rate codecs in both speaker similarity and intelligibility.
-
The paper provides extensive evaluations covering speech reconstruction, denoising, out-of-distribution robustness, representation disentanglement, TTS, and voice conversion.
-
CleanCodec demonstrates robust performance across different acoustic and linguistic domains, achieving substantially better intelligibility and speaker preservation than low-rate baselines on several OOD datasets.
-
The paper investigates several important design choices, including denoising, SSL supervision, speaker conditioning, and two-stage training, providing useful insights into their individual contributions.
Critical weaknesses
-
The novelty of the proposed denoising objective is somewhat limited, as reconstructing clean signals from degraded inputs is a well-established approach in denoising autoencoders and robust representation learning. The main contribution appears to be applying this existing idea to low-rate speech codecs, rather than introducing a fundamentally new learning principle. The paper could better clarify how its approach differs from existing denoising-based representation learning methods.
-
The paper argues that denoising encourages the codec to discard perceptually irrelevant information and allocate more capacity to important speech features. However, this hypothesis is not directly validated. The experiments mainly demonstrate improvements in reconstruction quality and downstream performance, without explicitly analyzing what information is retained or discarded by the discrete tokens. More direct evidence would strengthen the proposed selective information bottleneck interpretation.
Questions for authors
-
The paper claims that denoising encourages the codec to discard perceptually irrelevant information while preserving important speech features. Could the authors provide more direct evidence that the learned discrete tokens contain less noise-related information while retaining more linguistic and speaker-related information, beyond improvements in reconstruction and downstream metrics?
-
The ablation results show that removing the denoising objective leads to a relatively modest performance drop compared with removing SSL supervision. Could the authors further clarify how much of the overall improvement can be specifically attributed to the denoising objective, rather than auxiliary supervision or other architectural and training choices?
-
The proposed training framework applies several stochastic degradations, including background noise, reverberation, low-pass filtering, resampling, and MP3 compression. Could the authors provide an ablation study on the contribution of each degradation type and investigate how sensitive the model is to degradation severity? This would help clarify which degradations are most beneficial and whether the improvements depend on specific augmentation settings.
3 comments
Thank you again for your thoughtful feedback. We have added new ablations that validate the design choices used in CleanCodec, as well as further discussion in our paper clarifying the novelty of our denoising technique compared to existing robust representation learning literature. Are there remaining concerns that you have about our work? We would be happy to further clarify if any concerns remains unresolved. Thank you for your time!
Thank you for the detailed rebuttal and additional experiments, and apologies for my delayed response.
I appreciate the authors' efforts in addressing my concerns, particularly the new analysis of token stability under noisy conditions and the ablation studies on different degradation types. These additional results provide useful evidence supporting the effectiveness of the denoising objective and help clarify the contributions of the proposed approach.
The new experiments address several of my original questions. In particular, the token consistency analysis provides more direct evidence that the denoising objective encourages representations that are less sensitive to noise, while the additional ablations help clarify the contribution of different training components.
My remaining reservation mainly concerns methodological novelty, as the underlying denoising principle is well established. Nevertheless, I acknowledge that applying this principle to an ultra-low-bitrate discrete speech codec is a meaningful contribution, especially given the strong experimental results.
Overall, I maintain my positive assessment of the paper. I believe my initial rating already appropriately reflects its strengths and contributions. The rebuttal further supports my original evaluation, and I will therefore maintain my rating of 3 (Weak Acceptance).
We thank the reviewer for their detailed and thoughtful comments.
W1: Novelty. We agree that reconstructing clean signals from degraded inputs is well established, and have added a discussion of denoising-based representation learning to Sec. 2. We argue that the objective plays a different role in CleanCodec. In denoising autoencoders and robust SSL [1, 2, 3], corruption acts as a regularizer or invariance prior for continuous features used by downstream models, with little to no constraint on capacity. CleanCodec's representation is an ultra-low ~187 bps discrete bottleneck that must support full resynthesis, including speaker identity and prosody. Here the clean target determines what the bottleneck is asked to transmit, so the objective's role is rate allocation: capacity not spent on noise is available for perceptually important content. Our ablation (Table 9) supports this reading: removing the objective improves Mel L1 (0.44 → 0.41) but degrades WER, SIM, UTMOS and SI-SDR, a trade-off consistent with reallocating a fixed budget rather than a uniform regularization effect. CleanCodec differs in operating as a tokenizer for generative modeling at extremely low token rates, applying clean supervision to the acoustic, semantic and speaker streams, and showing that the gains transfer to TTS and voice conversion.
W2/Q1: Denoising effects. To demonstrate that our denoising objective causes CleanCodec to encode less noise, we encode the clean-noisy pairs from VoiceBank-DEMAND and compute what proportion of audio tokens change between the pairs. A small number of changed tokens indicate that the codec is ignoring the noise being added.
CleanCodec achieves the lowest proportion of changed tokens at both noise levels. We also test our ablation model without the denoising objective, and find it performs similarly to our baseline codecs. This indicates that our denoising objective reduces the amount of noise information being encoded. We also note that CleanCodec changes the fewest tokens despite having the largest codebook size and thus the most sensitive codebook, increasing the significance of these results.
Q2: Denoising contribution. We would like to clarify that SSL conditioning is a standard, commonly-used technique for significantly improving semantic reconstruction [4, 5], and we do not claim that our SSL conditioning technique is novel. We have updated our manuscript to make this more clear.
We note that our global conditioning technique does lead to improvements in speaker similarity. However, global conditioning causes minimal changes to other metrics. In contrast, our denoising objective improves 5 of 6 metrics, indicating that it is the primary driver of our performance improvements. Other architectural/training choices are covered in the ablation where we adjust our training recipe to match Kanade's model size, dataset, and bitrate. We find that CleanCodec still outperforms Kanade even after accounting for training recipe differences, indicating that the denoising objective truly accounts for the majority of improvements.
Q3: Denoising ablations. We train variants of CleanCodec where we enable exactly one of the five degradations in our paper. We then measure the reconstruction performance of each ablation:
From these results, we find that all degradations are somewhat helpful at improving reconstruction results, and no method in particular stands out as being particularly important or unimportant. We only see the best improvements after combining all the methods together.
We hope our extensive follow-up experiments have sufficiently resolved your concerns and comments.
[1] WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. https://arxiv.org/abs/2110.13900
[2] Towards Robust Speech Representation Learning for Thousands of Languages. https://arxiv.org/abs/2407.00837
[3] SoundStream: An End-to-End Neural Audio Codec. https://arxiv.org/abs/2107.03312
[4] Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model. https://arxiv.org/abs/2408.17175
[5] SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models. https://arxiv.org/abs/2308.16692