CLEAN-SPEECH TRANSPARENCY IS NOT DEPLOYMENT TRANSPARENCY: PREDICTING AND REPAIRING THE WORDS NEURAL AUDIO CODECS LOSE ON DEGRADED SPEECH

Neural audio codecs are benchmarked on clean speech, but deployed speech systems encode degraded audio. We find that clean-speech transparency understates the deployed loss several-fold yet predicts it, and that a codec-dependent share of it arises at the codeword lookup. Across ten operating points from six frozen codec families, the word error rate (WER) a codec adds for a recognizer reading its decoded audio is, at 5 dB SNR, 4.5–22× its clean value, and two recent recognizers replicate it. A power law of the clean loss, fitted on four families and then fixed, predicts held-out codecs, prospective measurements, and six codec families predicted blind (60 measurements: median error 1.32×, 90% within 2×, against 1.47× and 75% for a constant ratio). Tokenizers built on ASR-trained encoders fall below it. A quantizer-bypassed path through the same decoder attributes 6–96% of the loss, a lower bound, to the lookup, and re-fitting the codebook on noisy speech recovers little. We therefore repair the lookup’s input: an identity-initialized causal adapter between the frozen encoder and quantizer, trained by latent regression, halves Mimi’s added WER at 5 dB under noise of its training type (21.32 to 11.25 points) with clean WER and 99.8% of clean semantic codes unchanged. Mimi’s quantization penalty falls from 17.57 to 6.43 points while its bypassed path does not improve. The adapter removes less on two acoustic codecs and lets a small recognizer trained on clean tokens read degraded speech better than one retrained on degraded tokens. Placed after the lookup or in its place, the same network raises the loss on clean speech and under unseen babble.