acceptodds
Withdrawn from ICLR 2027

CLEAN-SPEECH TRANSPARENCY IS NOT DEPLOYMENT TRANSPARENCY: PREDICTING AND REPAIRING THE WORDS NEURAL AUDIO CODECS LOSE ON DEGRADED SPEECH

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Neural audio codecs are benchmarked on clean speech, but deployed speech systems encode degraded audio. We find that clean-speech transparency understates the deployed loss several-fold yet predicts it, and that a codec-dependent share of it arises at the codeword lookup. Across ten operating points from six frozen codec families, the word error rate (WER) a codec adds for a recognizer reading its decoded audio is, at 5 dB SNR, 4.5–22× its clean value, and two recent recognizers replicate it. A power law of the clean loss, fitted on four families and then fixed, predicts held-out codecs, prospective measurements, and six codec families predicted blind (60 measurements: median error 1.32×, 90% within 2×, against 1.47× and 75% for a constant ratio). Tokenizers built on ASR-trained encoders fall below it. A quantizer-bypassed path through the same decoder attributes 6–96% of the loss, a lower bound, to the lookup, and re-fitting the codebook on noisy speech recovers little. We therefore repair the lookup’s input: an identity-initialized causal adapter between the frozen encoder and quantizer, trained by latent regression, halves Mimi’s added WER at 5 dB under noise of its training type (21.32 to 11.25 points) with clean WER and 99.8% of clean semantic codes unchanged. Mimi’s quantization penalty falls from 17.57 to 6.43 points while its bypassed path does not improve. The adapter removes less on two acoustic codecs and lets a small recognizer trained on clean tokens read degraded speech better than one retrained on degraded tokens. Placed after the lookup or in its place, the same network raises the loss on clean speech and under unseen babble.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
neural audio codec, speech tokenization, residual vector quantization, noise robustness, speech recognition, speech language models, evaluation metrics, discrete representations

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_AcHDOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that the clean-speech codec WER understates the loss on degraded speech. Across ten operating points from six codec families on LibriSpeech test-clean with WHAM! noise, babble, and reverberation, it reports 4.5–22× amplification at 5 dB, fits a power law �0��0�, attributes 6–96% of the loss to the codeword lookup via a quantizer-bypassed path, and proposes a pre-lookup latent-regression adapter that halves Mimi’s 5 dB WER while preserving vocabulary. The empirical coverage is broad, but the headline claims are not supported at the level asserted.

Critical strengths

The idea of quantitatively linking clean-speech and degraded-speech codec behavior is somewhat novel. It motivates deployment-aware evaluation and rate provisioning, which clean-only benchmarks cannot address.

Critical weaknesses
  1. The quantitative results of both the 'clean-to-deployment law' and the quantizer-bypassed path attributes seem to vary largely across different codec families, code rates, SNR levels, and ASR models. There is a lack of a systematic schema for drawing practical guidance from these findings.
  2. The adapter is indeed a post-hoc noise remover, which not only overfits to the training noise, but also removes background sound events that contain environmental information. I don't think this kind of setting still establishes a 'transparent' codec.
  3. Only WER is used as the evaluation tool, without alignment to human perception.
Questions for authors
  1. What is the precise definition of so-called "transparency"?
Reviewer_mduEOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the additional recognition errors introduced by neural audio codecs on degraded speech. It proposes a power-law relationship between clean and degraded codec-added WER, validated on held-out codecs, multiple recognizers, and real far-field recordings. It also uses quantizer bypass experiments to investigate the source of the loss and introduces a pre-quantization adapter that substantially reduces Mimi’s added WER under noise of the training type while keeping the codec frozen.

Critical strengths
  1. The predictive relationship is novel and supported by validation
  2. The repair results on Mimi are convincing. At 5 dB, codec-added WER drops from 21.32 to 11.25 with essentially unchanged clean WER. A frozen recognizer reading the tokens directly also improves. This supports the practical value of repairing a codec while preserving its vocabulary.
Critical weaknesses

Major issues

On the power-law analysis

  1. The power-law analysis and the adapter appear largely disconnected. The fitted relationship neither informs the repair method nor yields predictions tested by it. The observed degradation alone motivates the adapter, leaving the role of the predictive analysis in the overall contribution unclear.
  2. The choice of a power law is insufficiently justified. The relationship is empirically fitted, without a theoretical derivation or sufficient comparison against alternatives such as an affine or piecewise model. The constant-ratio baseline is only the special case where the exponent equals one. The results support an empirical predictor, but do not establish why the power-law form is particularly appropriate.
  3. The practical benefit of the predictor remains unclear. An unfamiliar deployment environment still requires representative recordings, reference transcripts, and measurements for calibration. Once these are available, directly evaluating a small set of candidate codecs may incur limited additional cost, while prediction errors can lead to incorrect WER-budget decisions.

On the adapter training

  1. The trade-off between recognition accuracy and audio fidelity is not adequately evaluated. A codec preserves more than lexical content. Speaker characteristics, prosody, and environmental sounds are also part of the transmitted information. Mapping noisy representations toward clean ones may reduce WER by discarding some of this information. Evaluation should include reconstruction fidelity relative to the original noisy input and speech quality relative to the clean target. Clean WER and code-retention rates do not establish that broader information preservation has improved.
  2. Subjective evaluation and accessible listening examples are missing. The paper reports no human intelligibility, transcription, or listening-quality study. The supplied manuscript also provides no accessible listening examples, only a commitment to release them later. Lower ASR WER does not by itself establish that listeners find the reconstructed speech more intelligible or perceptually better. Automatic measures such as UTMOS cannot substitute for this evidence.

On the presentation

  1. The presentation is poor. The paper is difficult to follow, and many formulations fall short of the precision and restraint expected in an academic paper. Core findings, supplementary checks, and exceptions are presented with insufficient hierarchy. Dense numerical descriptions and long tables dominate the exposition, while key comparisons lack intuitive visual summaries. The “Finding” boxes frequently repeat the preceding discussion. Specific examples include:
    • Redundant cross-references: Table 2 repeatedly labels baseline rows “Frozen codec (Table 1).”
    • Overlong captions: The Table 2 caption spans L379–385
    • Informal and overlong headings: Examples include “Clean-speech transparency understates deployment loss several-fold” (L220–221), “The quantizer’s share, and a rate-matched scalar quantizer loses more words” (L324–325), and “Which metrics see the loss” (L463).
    • Excessive contrastive rhetoric. L228, 250, L254, 345–347, 365, 454–455, 477, etc.

Minor issues

  1. The primary WER aggregation requires better justification. The main results use utterance-averaged WER rather than the conventional corpus-WER computed from total edit counts divided by total reference words. This changes the relative weighting of short and long utterances and complicates comparison with existing benchmarks.

  2. Lack of citations, like [1] [2].

[1] Wang, Haoran, et al. "Towards general discrete speech codec for complex acoustic environments: A study of reconstruction and downstream task consistency." 2025 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2025. [2] Wojcicki, Kamil, et al. "Low-resource audio codec (lrac): 2025 challenge description." ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2026.

Questions for authors
  1. What insight from the power-law analysis informs the adapter’s design or is validated by the repair experiments?

  2. What supports the choice of a power law beyond its fit to the measured points? Have you compared it with affine or other simple nonlinear models on held-out codec families, using the same fitting data and evaluation protocol?

  3. In a new deployment environment, what measurements and annotation effort are required to use the predictor? Once calibration data are available, what cost or decision-accuracy advantage does it offer over directly evaluating each candidate codec on a small target-domain sample?

  4. Is the adapter intended to preserve the original recording or to produce enhanced speech? What evidence establishes the trade-off between lower WER and preservation of speaker characteristics, prosody, and environmental sounds? Please report any available fidelity results against the original input and quality results against the clean target.

  5. Do you have human intelligibility or subjective-quality results showing that the adapter’s ASR improvements translate into perceptual benefits? Are paired listening examples available now?

Reviewer_RDyVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether neural audio codecs, which are benchmarked on clean speech, remain effective when the input speech is noisy. If not, which part of the codecs causes the degradation and how to fix the degradation. The authors discover that noise greatly increases the word errors a codec adds, but in a predictable way, as the noisy-speech error fits a power law of the clean-speech error. The paper argues this loss primarily comes from the quantizer, instead of the encoder or decoder. It then proposes a small adapter placed between the frozen encoder and the frozen quantizer, which halves Mimi's added WER at 5 dB without changing its token vocabulary. Finally, it audits which metrics (e.g., STOI, WavLM distance) track the word loss under noise.

Critical strengths
  1. The paper shows that the WER a codec adds on noisy speech can be predicted from its clean-speech value by a simple power law, validated by various SNRs, noise types and datasets. This is an interesting finding and suggest directions for noise-robust ASR.
  2. Pinpointing the degradation source by decoding with and without the lookup through the same frozen decoder is well motivated. Using this measurement, the paper finds that the lookup accounts for more than half of the noisy-speech loss at 6 of the 10 operating points (e.g., 82% for Mimi and up to 96% for EnCodec), although the share varies widely across codecs.
  3. The evaluation is broad and comprehensive, including six codecs, the full LibriSpeech test-clean set, several recognizers etc.
Critical weaknesses
  1. The paper is very hard to read and new terminologies are introduced frequently, but without proper explanations. For instance, many terms (e.g., clean loss, deployed loss, operating point, rung, rate ladder, transparency) are used in the abstract and introduction but only defined at the end of Section 3, and some are used inconsistently (e.g., "deployed loss" vs. "deployment transparency").

  2. There are several claims that have already been studied in other papers. For instance: Tseng & Harwath 2025 show that noise degrades codecs. Li, Yip, Fan & Chng "Speech Enhancement Using Continuous Embeddings of Neural Audio Codec" (2025) have already proposed to add an enhancement module between the quantizer and the encoder. These weakens the contribution of the paper.

Questions for authors
  1. In which situation would one predict the noisy-speech loss with the law instead of measuring it directly? Or in other words, why predicting the WER instead of measuring it?
  2. Under which conditions do you expect the law to fail, e.g., background music? Under such conditions will the speech enhancement module still work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.