acceptodds
Under review as a conference paper at ICLR 2027

CleanCodec: Efficient and Robust Speech Tokenization via Perceptually Guided Encoding

Abstract

Neural audio codecs are a key component of speech processing pipelines, compressing audio into discrete tokens for downstream modeling. However, existing codecs struggle to balance reconstruction quality with token efficiency, often encoding perceptually irrelevant information such as background noise and recording artifacts at the expense of linguistically and acoustically meaningful content. We reframe audio tokenization as a selective information bottleneck problem and propose CleanCodec, a neural audio codec utilizing a denoising training objective, where the codec receives stochastically degraded speech as input but is trained to reconstruct the corresponding clean signal. CleanCodec learns to encode only perceptually important features and achieves state-of-the-art tokenization efficiency at just 12.5 t/s, improving speaker similarity by 0.21 and reducing word error rate by up to 16.3% compared to existing codecs. Evaluations on downstream text-to-speech and voice conversion tasks further demonstrate improved performance and up to 17x faster inference, highlighting significant efficiency gains due to reduced token rate.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio Codecs, Speech Tokenization, Text-to-speech

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5JCHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes CleanCodec, a neural speech codec that improves tokenization efficiency by encouraging the preservation of perceptually important information. Its key idea is a denoising training objective that reconstructs clean speech from stochastically degraded inputs, reducing the encoding of irrelevant acoustic information. CleanCodec combines discrete local tokens with global speaker representations, supported by auxiliary semantic and speaker supervision. Experiments demonstrate strong performance in speech reconstruction, denoising, and downstream TTS and voice conversion tasks, particularly at a low token rate of 12.5 tokens/s.

Critical strengths
  1. CleanCodec achieves strong speech reconstruction quality at only 12.5 tokens/s, substantially outperforming comparable low-rate codecs in both speaker similarity and intelligibility.

  2. The paper provides extensive evaluations covering speech reconstruction, denoising, out-of-distribution robustness, representation disentanglement, TTS, and voice conversion.

  3. CleanCodec demonstrates robust performance across different acoustic and linguistic domains, achieving substantially better intelligibility and speaker preservation than low-rate baselines on several OOD datasets.

  4. The paper investigates several important design choices, including denoising, SSL supervision, speaker conditioning, and two-stage training, providing useful insights into their individual contributions.

Critical weaknesses
  1. The novelty of the proposed denoising objective is somewhat limited, as reconstructing clean signals from degraded inputs is a well-established approach in denoising autoencoders and robust representation learning. The main contribution appears to be applying this existing idea to low-rate speech codecs, rather than introducing a fundamentally new learning principle. The paper could better clarify how its approach differs from existing denoising-based representation learning methods.

  2. The paper argues that denoising encourages the codec to discard perceptually irrelevant information and allocate more capacity to important speech features. However, this hypothesis is not directly validated. The experiments mainly demonstrate improvements in reconstruction quality and downstream performance, without explicitly analyzing what information is retained or discarded by the discrete tokens. More direct evidence would strengthen the proposed selective information bottleneck interpretation.

Questions for authors
  1. The paper claims that denoising encourages the codec to discard perceptually irrelevant information while preserving important speech features. Could the authors provide more direct evidence that the learned discrete tokens contain less noise-related information while retaining more linguistic and speaker-related information, beyond improvements in reconstruction and downstream metrics?

  2. The ablation results show that removing the denoising objective leads to a relatively modest performance drop compared with removing SSL supervision. Could the authors further clarify how much of the overall improvement can be specifically attributed to the denoising objective, rather than auxiliary supervision or other architectural and training choices?

  3. The proposed training framework applies several stochastic degradations, including background noise, reverberation, low-pass filtering, resampling, and MP3 compression. Could the authors provide an ablation study on the contribution of each degradation type and investigate how sensitive the model is to degradation severity? This would help clarify which degradations are most beneficial and whether the improvements depend on specific augmentation settings.

3 comments

We thank the reviewer for their detailed and thoughtful comments.

W1: Novelty. We agree that reconstructing clean signals from degraded inputs is well established, and have added a discussion of denoising-based representation learning to Sec. 2. We argue that the objective plays a different role in CleanCodec. In denoising autoencoders and robust SSL [1, 2, 3], corruption acts as a regularizer or invariance prior for continuous features used by downstream models, with little to no constraint on capacity. CleanCodec's representation is an ultra-low ~187 bps discrete bottleneck that must support full resynthesis, including speaker identity and prosody. Here the clean target determines what the bottleneck is asked to transmit, so the objective's role is rate allocation: capacity not spent on noise is available for perceptually important content. Our ablation (Table 9) supports this reading: removing the objective improves Mel L1 (0.44 → 0.41) but degrades WER, SIM, UTMOS and SI-SDR, a trade-off consistent with reallocating a fixed budget rather than a uniform regularization effect. CleanCodec differs in operating as a tokenizer for generative modeling at extremely low token rates, applying clean supervision to the acoustic, semantic and speaker streams, and showing that the gains transfer to TTS and voice conversion.

W2/Q1: Denoising effects. To demonstrate that our denoising objective causes CleanCodec to encode less noise, we encode the clean-noisy pairs from VoiceBank-DEMAND and compute what proportion of audio tokens change between the pairs. A small number of changed tokens indicate that the codec is ignoring the noise being added.

ModelNoisy (10 dB)Noisy (20 dB)
CleanCodec0.750.53
CleanCodec (no denoise)0.980.82
Kanade0.960.77
FocalCodec0.940.76

CleanCodec achieves the lowest proportion of changed tokens at both noise levels. We also test our ablation model without the denoising objective, and find it performs similarly to our baseline codecs. This indicates that our denoising objective reduces the amount of noise information being encoded. We also note that CleanCodec changes the fewest tokens despite having the largest codebook size and thus the most sensitive codebook, increasing the significance of these results.

Q2: Denoising contribution. We would like to clarify that SSL conditioning is a standard, commonly-used technique for significantly improving semantic reconstruction [4, 5], and we do not claim that our SSL conditioning technique is novel. We have updated our manuscript to make this more clear.

We note that our global conditioning technique does lead to improvements in speaker similarity. However, global conditioning causes minimal changes to other metrics. In contrast, our denoising objective improves 5 of 6 metrics, indicating that it is the primary driver of our performance improvements. Other architectural/training choices are covered in the ablation where we adjust our training recipe to match Kanade's model size, dataset, and bitrate. We find that CleanCodec still outperforms Kanade even after accounting for training recipe differences, indicating that the denoising objective truly accounts for the majority of improvements.

Q3: Denoising ablations. We train variants of CleanCodec where we enable exactly one of the five degradations in our paper. We then measure the reconstruction performance of each ablation:

DenoisingWERCERSIMUTMOSSI-SDRMel L1
All2.71.40.864.3221.960.44
None3.41.70.824.2420.550.41
Reverb3.21.50.824.2821.300.41
Noise3.21.60.814.2821.360.43
Low-pass3.01.50.844.2821.280.42
Resample3.01.60.824.2721.720.41
MP33.11.60.834.2420.380.44

From these results, we find that all degradations are somewhat helpful at improving reconstruction results, and no method in particular stands out as being particularly important or unimportant. We only see the best improvements after combining all the methods together.

We hope our extensive follow-up experiments have sufficiently resolved your concerns and comments.

[1] WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing. https://arxiv.org/abs/2110.13900

[2] Towards Robust Speech Representation Learning for Thousands of Languages. https://arxiv.org/abs/2407.00837

[3] SoundStream: An End-to-End Neural Audio Codec. https://arxiv.org/abs/2107.03312

[4] Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model. https://arxiv.org/abs/2408.17175

[5] SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models. https://arxiv.org/abs/2308.16692

Thank you again for your thoughtful feedback. We have added new ablations that validate the design choices used in CleanCodec, as well as further discussion in our paper clarifying the novelty of our denoising technique compared to existing robust representation learning literature. Are there remaining concerns that you have about our work? We would be happy to further clarify if any concerns remains unresolved. Thank you for your time!

Thank you for the detailed rebuttal and additional experiments, and apologies for my delayed response.

I appreciate the authors' efforts in addressing my concerns, particularly the new analysis of token stability under noisy conditions and the ablation studies on different degradation types. These additional results provide useful evidence supporting the effectiveness of the denoising objective and help clarify the contributions of the proposed approach.

The new experiments address several of my original questions. In particular, the token consistency analysis provides more direct evidence that the denoising objective encourages representations that are less sensitive to noise, while the additional ablations help clarify the contribution of different training components.

My remaining reservation mainly concerns methodological novelty, as the underlying denoising principle is well established. Nevertheless, I acknowledge that applying this principle to an ultra-low-bitrate discrete speech codec is a meaningful contribution, especially given the strong experimental results.

Overall, I maintain my positive assessment of the paper. I believe my initial rating already appropriately reflects its strengths and contributions. The rebuttal further supports my original evaluation, and I will therefore maintain my rating of 3 (Weak Acceptance).

Reviewer_9z5TOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CleanCodec that reframes audio tokenization as a selective information bottleneck problem by introducing a denoising training objective. It receives stochastically degraded speech signals as inputs but is trained to reconstruct the corresponding clean signal which is regarded as the perceptually important features. The experiments show that the codec achieves good performance in reconstruction, denoising, and downstream tasks at 12.5 tokens per second.

Critical strengths
  1. Elegant integration of enhancement and compression of speech signal. The paper reframes speech enhancement not as a preprocessing step but as an intrinsic training objective.
  2. Good performance at a low token rate was shown in the experiments.
  3. Strong out-of-distribution reconstruction results with expresso, AISHELL-3 and CML-TTS corpora besides LibriTTS test-clean corpus.
Critical weaknesses
  1. By forcing the codec to map noisy speech inputs to clean targets, there is a risk that the codec might remove critical paralinguistic information (such as emotional tone, speaker identity, and non-verbal cues) by treating them as noises, and thus might hurt the naturalness or emotional fidelity of speech. Suggest to conduct human listening tests. Furthermore, speech corpora with more diversified acoustic conditions can be used in the experiments to verify the performance.
  2. The denoising training objective has been claimed as the major contribution for achieving SOTA performance, but the experiments reveal that other factors contribute heavily. For example, in Figure 9, removing SSL conditioning increases WER from 2.7% to 8.1%, and the denoising objective alone increases WER from 2.7% to 3.4%.
Questions for authors
  1. Is the denoising objective effective in smaller model sizes? For example, if ConvNeXt backbones are scaled down to match the parameter count of Kanade or Mimi while keeping the denoising objective, how much of the performance gain can be retained?
2 comments

We are grateful to the reviewer for their thoughtful feedback.

W1: Denoising risks. We agree that human preference tests are the gold standard for evaluation. Therefore, we conducted a blind listening test on CleanCodec and 3 baselines: our two 12.5 t/s baselines FocalCodec, Kanade, plus BiCodec for a comparison at higher token rates. We evaluate using VoiceBank-DEMAND for noisy acoustic conditions, as well as Expresso for emotional speech. We measure several axes, including speech quality, speaker similarity, semantic similarity, and style similarity, using a 5-point comparison scale ranging from -2 (baseline is much better) to +2 (CleanCodec is much better). More details about the listening tests are at the bottom of our response. Our results are in the tables below, with statistically significant results in bold:

VoiceBank-DEMANDToken rate (t/s)QualityTimbreSemantic
BiCodec50+1.42 (p<0.001)+0.83 (p=0.011)+0.67 (p=0.039)
FocalCodec12.5+1.75 (p<0.001)+1.50 (p<0.001)+1.17 (p<0.001)
Kanade12.5+1.50 (p<0.001)+0.67 (p=0.039)+0.50 (p=0.053)
ExpressoToken rate (t/s)QualityStyleTimbreSemantic
BiCodec50-0.44 (p=0.028)-0.33 (p=0.026)+0.00 (p=1.00)-0.11 (p=0.43)
FocalCodec12.5+1.33 (p<0.001)+1.50 (p<0.001)+1.50 (p<0.001)+0.78 (p=0.003)
Kanade12.5+0.83 (p=0.002)+0.78 (p=0.002)+0.67 (p<0.001)+0.33 (p=0.083)

On VoiceBank-DEMAND, listeners strongly preferred CleanCodec over our baselines across nearly all metrics. Consistent with our paper, we find that CleanCodec successfully removes background noise from the input audio while other codecs do not, leading to higher perceived audio quality. We also argue that the prioritization of perceptually relevant information allows CleanCodec to better preserve speaker timbre and linguistic content, improving the other perceptual metrics.

On Expresso, CleanCodec significantly outperforms both 12.5 t/s baselines, and is competitive with BiCodec despite using 4x fewer tokens. This shows there is little evidence that our denoising pipeline harms the naturalness or emotional fidelity of speech.

W2: Denoising contribution. We would like to clarify that SSL conditioning is a standard, commonly-used technique for significantly improving semantic reconstruction [1, 2], and we do not claim that our SSL conditioning technique is novel. Our main contribution is showing that our novel denoising training objective achieves further gains over well-established techniques. We have updated our manuscript to make this more clear. We note that our global conditioning technique does lead to improvements in speaker similarity. However, global conditioning causes minimal changes to other metrics. In contrast, our denoising objective improves 5 of 6 metrics, indicating that it is the primary driver of our performance improvements. Other architectural/training choices are covered in the ablation where we adjust our training recipe to match Kanade's model size, dataset, and bitrate. We find that CleanCodec still outperforms Kanade even after accounting for training recipe differences, indicating that the denoising objective truly accounts for the majority of improvements.

Q1: Denoising at smaller model sizes. In our ablation study we tested CleanCodec's performance when shrinking the model and training setup to Kanade's size (see Table 9). We train another version of this smaller model while removing the denoising feature:

test-cleanWERCERSIMUTMOSSI-SDRMel L1
Kanade training setup3.21.70.784.2220.990.50
+ No denoise4.22.10.724.2120.360.48

According to our results, reducing the model size does not reduce the effectiveness of the denoising objective, with similar performance improvements as the larger model.

Listening test details For each trial, we provide the reference audio as ground truth, as well as 2 reconstructed utterances from CleanCodec and a baseline. We randomize the order they are presented to eliminate positional bias. For VoiceBank-DEMAND, we use the noisy reference audio as ground truth and as input to the codecs. Several of our baseline codecs (FocalCodec, BiCodec) output 16 kHz audio, while CleanCodec and others output at 24 kHz, which is an unfair advantage. To mitigate this, we resample to 16 kHz when comparing codecs of different sample rates. Each test contained 24 questions, with 2 comparing CleanCodec against the ground truth reference as an attention check. Participants who failed both checks were filtered out, but throughout testing no participants failed the checks. We compute p-values using a one-sample t-test.

[1] Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model. https://arxiv.org/abs/2408.17175

[2] SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models. https://arxiv.org/abs/2308.16692

Thank you again for your thoughtful feedback. We have added new ablations and discussions that validate the novel design choices used in CleanCodec, as well as a listening test that establishes perceptual improvements over existing baselines in addition to the metrics in our paper. Are there remaining concerns that you have about our work? We would be happy to further clarify if any concerns remains unresolved. Thank you for your time!

Reviewer_BgZJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CleanCodec is a speech codec that learns to reconstruct clean speech from degraded inputs. It combines a local FSQ token stream, operating at 12.5 tokens per second in the main configuration, with a continuous global embedding for the utterance. WavLM and TitaNet supervision encourage the local and global representations to retain linguistic content and speaker identity, respectively. The autoencoder and Vocos vocoder are trained separately before joint fine-tuning. The experiments cover reconstruction, denoising, representation disentanglement, voice conversion, and TTS, with the main goal of improving speech quality and downstream modeling at a low token rate.

Critical strengths
  1. The results at 12.5 tokens per second are promising, especially for linguistic content and speaker similarity. I like that the paper evaluates both reconstruction and downstream use. The TTS comparison includes several codecs under the same small language-model backbone and training recipe; CleanCodec reports better WER and SIM than the other 12.5-token-per-second models. This helps connect the representation quality to a practical use case.

  2. The ablations give useful evidence about the training recipe. Removing denoising, semantic supervision, or speaker supervision affects the expected aspects of the output. The experiment that matches Kanade's model size, information budget, and training data is also helpful: it retains improvements in WER and SIM, making it less likely that the main gains come entirely from scale differences.

Critical weaknesses
  1. The quality evaluation needs stronger validation, particularly where clean references are already available. Section 4.2 explicitly describes VoiceBank-DEMAND as paired clean and noisy speech, but Table 4 reports STOI, PESQ, and SI-SDR from SQUIM. These are learned estimates of the metrics. I do not understand why the available clean references were not used to compute the corresponding reference-based scores. The TorchAudio tutorial makes this distinction explicit and provides examples of both calculations. As presented, the table establishes an improvement in the predictor's scores, while the actual reference-based improvement remains unclear.

    This concern also extends to the multilingual evaluation. The released SQUIM objective model is trained on DNS 2020, whose speech data were English, as noted in Section 3 of the DNS 2021 paper. The paper evaluates Chinese and other languages without reporting calibration on those languages or on codec reconstruction artifacts. I would like to see whether the quality gains and method rankings survive reference-based evaluation and listening tests. This matters directly for the denoising and perceptual-quality claims.

  2. The global speaker representation seems to limit the scope of the codec, and this is not tested. Section 3.1 pools the entire utterance into one global embedding, and the training loss aligns it with a TitaNet speaker-verification embedding. This is a natural choice for a single-speaker utterance. I am less clear about what it should represent when speakers alternate within a recording or speak at the same time. These are common speech inputs, but I did not find an evaluation of either case.

    A single global vector does not by itself prove that multiple speakers cannot be represented; the local stream might retain some of that information. However, relying on this would need evidence, particularly given the intended separation of content and timbre. I would like the authors to show whether both content and speaker identity are preserved across speaker changes, or state the single-speaker scope more clearly. If the intended solution is to segment by speaker first, that dependency and its behavior on overlapping speech should be explained.

Questions for authors
  1. Why use SQUIM predictions in Table 4 when paired clean references are available? Could you report directly computed PESQ, STOI, and SI-SDR, including the resampling and alignment procedure? Which SQUIM checkpoint was used, and have its predictions or rankings been checked on the Chinese and multilingual evaluation sets? Listening results would also help establish whether the predicted improvements are audible.

  2. What happens when an input contains alternating or overlapping speakers? How is the global speaker target defined in that case? A small evaluation with speaker changes, checking content and speaker identity within each turn, would help clarify whether the limitation is mainly in the current evaluation or in the representation itself.

  3. Have you compared FSQ with a well-tuned VQ baseline at the same token rate and nominal bit budget, keeping the rest of the model and training recipe fixed? FSQ's training stability is attractive, but high codebook utilization alone does not establish that it offers the best reconstruction trade-off here. This comparison would help justify the quantizer choice.

  4. For the one-stage baseline in Table 9, have you tried training the full autoencoder-vocoder chain jointly with a reconstruction-only warm-up, then enabling the adversarial losses? The reported failure with adversarial training from the start does not distinguish the benefit of separate pretraining from the benefit of delaying GAN training. A comparison at a comparable training budget would make this conclusion more convincing.

  5. Could you break WER down into substitutions, deletions, and insertions, particularly for the low-token-rate models and the no-SSL ablation? The reconstruction protocol uses ASR transcripts of the original audio as references, so the zero WER of the original is true by construction. Reporting the recognizer's error against human transcripts on the evaluation set, along with a few checked error cases, would help interpret the changes. In particular, the rise from 2.7% to 8.1% WER with nearly unchanged UTMOS does not yet tell us whether the reconstructed speech actually says different words or mainly becomes harder for that recognizer to transcribe.

  6. Could you spell out the TTS evaluation protocol and provide an anonymous configuration or script with a few audio examples? The shared backbone and training recipe are a reasonable starting point; I would mainly like to know the evaluated Seed-TTS-eval split, prompt construction, source of the global embedding, sampling settings, and scoring procedure. The official benchmark uses Whisper-large-v3 for English, Paraformer-zh for Mandarin, and WavLM-large speaker verification for SIM. Does Table 8 follow that protocol or reuse the Parakeet/ReDimNet setup described for reconstruction? I understand the stated reason for preferring ReDimNet, but any changes should be explicit, and a check with the official evaluators would make the results easier to compare with other work.

4 comments

We sincerely appreciate the reviewer for their detailed and thorough review.

W1/Q1: Reference-based metrics and multilingual listening tests. We agree that human preference tests are the gold standard for evaluation. Therefore, we conducted a blind listening test on VoiceBank-DEMAND using CleanCodec and 4 baselines: our two 12.5 t/s baselines FocalCodec, Kanade, plus BiCodec and Qwen3 for comparisons at higher token rates. We use a 5-point comparison scale ranging from +2 (strongly prefer CleanCodec) to -2 (strongly prefer baseline). We measure several axes, including speech quality, speaker timbre, and semantic similarity. More details about the listening tests are at the bottom of our response. Our results are in the table below, with statistically significant results in bold:

ComparisonToken rate (t/s)QualityTimbreSemantic
Noisy GT–+1.75 (p<0.001)+1.00 (p<0.001)+1.33 (p<0.001)
Qwen3200+1.67 (p<0.001)+0.67 (p=0.025)+1.08 (p<0.001)
BiCodec50+1.42 (p<0.001)+0.83 (p=0.011)+0.67 (p=0.039)
FocalCodec12.5+1.75 (p<0.001)+1.50 (p<0.001)+1.17 (p<0.001)
Kanade12.5+1.50 (p<0.001)+0.67 (p=0.039)+0.50 (p=0.053)

According to our listening tests, CleanCodec's speech quality is strongly preferred over all codecs, as the reconstructed audio it produces is free of noise while the other codecs attempt to reconstruct the noise. We also note that CleanCodec achieves significant improvements to speaker and semantic preservation over existing baselines. This corroborates our claims that learning to ignore background noise allows other aspects of audio (e.g. speaker timbre and linguistic content) to be better preserved.

We use the objective SQUIM checkpoints built into torchaudio. Regarding our choice to use referenceless metrics, we include the reference-based STOI, PESQ, and SI-SDR below. We use the clean reference as the ground truth and the noisy reference as input for codec reconstruction.

ModelToken rate (t/s)STOIPESQSI-SDR
Noisy GT–0.921.988.53
Qwen32000.891.793.46
BiCodec500.851.67-12.11
FocalCodec12.50.741.15-33.51
Kanade12.50.801.31-27.53
CleanCodec12.50.851.74-23.98

CleanCodec has similar PESQ and STOI scores to BiCodec, and worse than Qwen3. In addition, CleanCodec has an extremely low SI-SDR score, while BiCodec and Qwen3 are much higher. However, this directly contradicts the subjective scores, where CleanCodec is very strongly preferred over both BiCodec and Qwen3. In addition, all codecs get worse objective scores than the unmodified noisy reference, despite the fact that the noisy reference performed among the worst in subjective listening. We argue that the reference-based metrics are poorly calibrated with subjective preferences because they are highly sensitive to waveform changes that are inaudible to human listeners. (For instance, SI-SDR heavily penalizes small phase shifts that sound identical to the human ear [1]) As a result, CleanCodec can achieve very strong subjective results despite not having SOTA reference-based PESQ, STOI, and SI-SDR scores.

Regarding multilingual calibration, we also conducted listening tests on AISHELL-3 to test Mandarin performance, with statistically significant results in bold:

ComparisonToken rate (t/s)QualityTimbreSemantic
BiCodec50-0.17 (p=0.438)-0.17 (p=0.504)-0.30 (p=0.193)
FocalCodec12.5+1.83 (p<0.001)+1.75 (p<0.001)+1.80 (p<0.001)
Kanade12.5+1.42 (p<0.001)+1.42 (p<0.001)+1.40 (p<0.001)

We find that our objective metrics in our paper closely follow our subjective listening tests, where CleanCodec vastly outperforms FocalCodec and Kanade and performs similarly to BiCodec despite using 4x fewer tokens. This indicates that the scores in our paper are well-calibrated with listening preferences.

W2/Q2: Multi-speaker encoding. We would like to clarify that all disentangled codecs generally assume that utterances are single-speaker only. We do not make any claims about the performance of CleanCodec in multi-speaker or overlapping speaker settings, as this is outside the scope of our design. We have updated our manuscript to reflect this assumption. We do think testing speaker changes would be a valuable experiment to perform though. We create multi-speaker utterances by combining LibriTTS utterances from different speakers and measure how much SIM and WER scores degrade when encoding the combined utterance vs. encoding the utterances separately.

ModelModeWERSIM
CleanCodecJoint1.80.39
CleanCodecSeparate2.90.60
KanadeJoint3.90.32
KanadeSeparate2.70.51
BiCodecJoint3.10.46
BiCodecSeparate3.10.57

We see that SIM scores drop across all disentangled codecs in multi-speaker utterances, as expected. We also find that CleanCodec's WER scores did not degrade from joint encoding, indicating that content is fully preserved across speaker changes. We also note that BiCodec achieves the best SIM scores when encoding jointly, but not when encoding separately. This is consistent with our downstream VC task (Table 7) which showed that BiCodec leaks speaker information into its audio tokens. Altogether, these results further demonstrate that CleanCodec achieves the best separation of speaker and content information into its global embedding and audio tokens, respectively. If encoding multi-speaker inputs we would recommend segmenting the audio first.

Q3: FSQ vs. VQ. We train another variant of CleanCodec where we replace FSQ with VQ. We use a codebook size of 32,768 to match FSQ, and we apply the following standard optimizations to mitigate codebook collapse: (1) small codebook dim D=8 with L2-normalized codes, (2) EMA codebook updates, (3) dead code restarts. We also add commitment loss to our loss function. We report the reconstruction metrics and codebook utilization on LibriTTS test-clean in the table below:

test-cleanNorm. entropyWERCERSIMUTMOSSI-SDRMel L1
FSQ0.9852.71.40.864.3221.960.44
VQ0.9932.71.50.874.3021.870.44
test-otherWERCERSIMUTMOSSI-SDRMel L1
FSQ5.73.20.844.1721.730.44
VQ6.03.40.854.1421.290.44

Surprisingly, we find that VQ is actually stable during training and achieves higher codebook utilization than FSQ, despite documented codebook collapse in prior research [2]. However, we find that this does not translate into improved reconstruction metrics, where both methods perform comparably. In addition, VQ requires additional training complexity and hyperparameter tuning, as we needed to adjust the EMA decay rate, dead code restart ratio, and commitment loss weight. Therefore, we argue that FSQ is still a superior choice over VQ due to its simplicity and comparable performance.

Q4: Joint training with delayed adversarial loss. We train another variant of CleanCodec where we jointly train the autoencoder and vocoder while delaying adversarial loss. We train for 2M steps, with adversarial loss added at step 1M. In practice, we found that the point where adversarial loss was added made little difference:

Training CurriculumWERCERSIMUTMOSSI-SDRMel L1
Two-stage2.71.40.864.3221.960.44
Delayed GAN2.61.40.823.7916.160.47

We see that joint training with delayed adversarial loss significantly degrades reconstruction, particularly in speech quality. We speculate that this is because only the highly compressive autoencoder is harmed by early GAN training, and the vocoder benefits from adversarial loss during pretraining to ensure high-quality mel-to-waveform reconstruction. This validates our two-stage training curriculum.

Q5: WER breakdown and error analysis. We would like to clarify that SSL conditioning is a standard, commonly-used technique for significantly improving semantic reconstruction [3, 4], and we do not claim that our SSL conditioning technique is novel. We have updated our manuscript to better reflect this. We analyze the WER breakdown on a sample of LibriTTS test-clean, showing the substitutions, insertions, and deletions. To evaluate the changes, we compute the errors on the original ASR transcript vs. human transcript as well as reconstructed ASR transcript vs. human transcript.

TypeWERSubstitutionDeletionInsertion
ASR vs. human1.711.160.290.27
Reconstructed ASR vs. human2.35 (+0.64)1.69 (+0.53)0.37 (+0.08)0.29 (+0.02)

Substitutions make up most of the new errors. Deletions and insertions barely change, indicating that the codec isn't muffling or dropping many words. After manually listening to audio clips that Parakeet mistranscribed, we find that the reconstructed audio does indeed contain different words than the original audio, generally vowel or consonant swaps in a single word. We provide some examples below:

  • lake → leek
  • have → had
  • plural → borough
  • come → calm
  • dovecote → dub coat
  • part → pot
  • but Hilda → behilda
  • the hit → behit

In most cases, the speech stays high-quality, just with different semantics. This explains why UTMOS scores can stay the same even as WER increases.

Q6: TTS evaluation protocol. We have added our TTS training code and examples in the supplementary material. Regarding evaluation, we use the standard seed-eval methodology using their official codebase to allow for easy comparison, and we do not use our Parakeet/ReDimNet setup from our reconstruction evals. Eval audio is generated from the en split using temperature=0.8 and top_p=0.95, with a 30 second generation limit. As stated in the paper, we use prompts of the format <prompt tokens><target text><BOS>, where <prompt tokens> refers to the audio tokens from encoding the reference wav. For disentangled codecs, we obtain the embedding from encoding the reference wav and prepend it to the sequence, so the prompt will have the format <global embedding><prompt tokens><target text><BOS>. The embedding is mapped to the TTS model's latent space via a learned linear projection. The details of our implementation can be found in the supplementary material.

Listening test details For each trial, we provide the reference audio as ground truth, as well as 2 reconstructed utterances from CleanCodec and a baseline. We randomize the order they are presented to eliminate positional bias. For VoiceBank-DEMAND, we use the noisy reference audio as ground truth and as input to the codecs. We note that several of our baseline codecs (FocalCodec, BiCodec) output 16 kHz audio, while CleanCodec and the remaining baselines output 24 kHz audio, which could give us an unfair advantage. To mitigate this, we resample all audio to 16 kHz when comparing codecs of different sample rates. Each test contained 24 questions, with two questions comparing CleanCodec against the ground truth reference as an attention check. Participants who failed both checks were filtered out, but throughout testing none of the participants failed the checks. We compute p-values using a one-sample t-test.

We hope our extensive follow-up experiments have sufficiently resolved your concerns and comments.

[1] On The Compensation Between Magnitude and Phase in Speech Separation. https://arxiv.org/abs/2108.05470

[2] Finite Scalar Quantization: VQ-VAE Made Simple. https://arxiv.org/abs/2309.15505

[3] Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model. https://arxiv.org/abs/2408.17175

[4] SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Models. https://arxiv.org/abs/2308.16692

Thank you again for your thoughtful feedback. We have conducted new ablations and experiments that validate the design choices used in CleanCodec, as well as a listening test that establishes perceptual improvements over existing baselines in addition to the metrics in our paper. Are there remaining concerns that you have about our work? We would be happy to further clarify if any concerns remains unresolved. Thank you for your time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank the reviewers for their constructive feedback. We are glad that reviewers all agreed on the strength of the low-token-rate results and the breadth of our evaluation process. Throughout the rebuttal process, we conducted new experiments and ablations that aimed to further validate CleanCodec's methodology and address reviewers' concerns. Here are the major additions:

  1. Human listening tests (BgZJ, 9z5T). We conducted blind listening tests on noisy, Mandarin, and expressive speech in order to confirm that our improved objective metrics translate to an audible difference in reconstruction quality. Our aggregate results are below and in Table 6 (lines 379-387), with full per-dataset results in Table 15 (1084-1103).
Modelt/sQualityStyleTimbreSemantic
BiCodec50+0.17 (p = 0.35)−0.33 (p = 0.026)+0.19 (p = 0.13)+0.07 (p = 0.57)
FocalCodec12.5+1.60 (p < 0.001)+1.50 (p < 0.001)+1.57 (p < 0.001)+1.19 (p < 0.001)
Kanade12.5+1.19 (p < 0.001)+0.78 (p = 0.002)+0.88 (p < 0.001)+0.65 (p < 0.001)

We find that CleanCodec is strongly preferred to both codecs at equal token rate. Compared against BiCodec, which uses 4x the token rate, CleanCodec is statistically similar in all metrics except emotional style, where it is slightly worse. It is also strongly preferred against all codecs on noisy speech, demonstrating the audible improvements caused by denoising. Overall, our listening results demonstrate that our gains hold for human listeners.

  1. New ablations (BgZJ, 9z5T, 5JCH). We add several ablations, including (a) VQ quantizer (b) Denoising at smaller scales (c) Per-degradation effects (d) single-stage training with delayed GAN loss. These ablations validate the design choices made in CleanCodec.

  2. New experiments (BgZJ, 5JCH). We conduct new experiments including (a) directly testing the effect of denoising on token representations (b) Breakdown and analysis of WER errors.

  3. Novelty (5JCH). We distinguish our contributions from existing denoising autoencoder and robust SSL work in Section 2.

These new results were added to our manuscript in blue. Please let us know if there any other concerns we can address, we would still appreciate any follow-up comments before the rebuttal period ends!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.