- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes a method for dynamically modifying the number of residual vector quantization tokens sent in a neural speech codec, without requiring changing the underlying codec in any way. The dynamic token allocation computation is performed identically by the encoder and decoder, which further means that no additional bits are needed in the stream itself to indicate the number of tokens to expect.
Critical strengths
The results of the paper are extremely well-tested. A wide variety of quality measures and ablations are applied, including a round of subjective testing. Regardless of any other concerns, this paper strikes me as a model of how to do a speech paper.
Critical weaknesses
Probably the biggest issue for the paper is motivation; this seems like a very niche application.
Specifically, it's not clear why introducing an extra model (which must match and be maintained on both sides of the communication) is any easier than swapping the codec for an explicitly variable-bitrate codec (like those discussed in the related works section). (my best guess is serving lots of audio that has already been encoded with a specific fixed-rate neural codec, where re-encoding the audio at-rest is too expensive?)
Questions for authors
-
I'm curious how the results here compare with the language model + entropy coding approach in the Encodec paper: https://arxiv.org/pdf/2210.13438 Their entropy coding can similarly be applied to any frozen token stream to reduce the overall bitrate (though no actual RVQ frames will be dropped).
-
I'm also curious (again, motivation) about the impact of the bitrate alloctation algorithm on stream latency. IIUC, both sides need to look at the top token from a set of frames and run an algorithm to determine which tokens to drop. This would seem to introduce a -frame delay to the entire system.
Thank you for raising the deployment question and for suggesting the entropy-coding comparison. The section and table numbers below refer to the revised manuscript.
Weakness; Why retrofit an existing codec? We target offline batch processing, such as archival of call-center recordings, where the codec weights and codebooks must stay fixed, but both endpoints can update bitstream handling. This avoids retraining or replacing the base codec, not updating the allocator and parser; an unchanged receiver is not compatible. The archival scenario illustrates the constraint, not a deployment we tested. Previously discarded indices cannot be recovered from a low-depth archive. Revision: Secs. 1 and 3.3.
Q1. Comparison with EnCodec entropy coding: Entropy coding reduces the bits needed to transmit the same indices without changing the reconstruction; our method changes which indices are retained. On 1,000 LRAC public-test inputs, we compared fixed-width, ungated Proposal with entropy-coded, uniform-depth CBR on the same frozen DAC. The CBR arm uses arithmetic coding with a per-stage causal bigram fitted on training data, not EnCodec’s language model. At CBR depth 2, the bigram lowers the measured rate from 1.521 to 1.426 kbps (6.3%). At 1.521 kbps, Proposal gains 0.099 PESQ and 0.065 ViSQOL over entropy-coded CBR interpolated to the same rate. Rates are computed from total serialized bits, including headers, divided by total audio duration. Each paired bootstrap draw recomputes the rates and interpolation without extrapolation. At CBR depths 4 to 8, this bigram increases bitrate relative to fixed-width packing, so the higher-rate results do not establish an advantage over stronger entropy models. (Secs. 2 and 4.3; App. D.6, Table 20.)
Q2. Lookahead and stream latency: Mean RVQ depth sets the index budget, not the number of frames to buffer. The receiver needs the full first-stage prefix of the utterance before it can recover the allocation and parse later indices. Selective also compares two complete reconstructions at the sender. The current method therefore requires whole-utterance buffering and is intended for offline use. In the DAC depth-4 control, median receiver coordination takes 422 ms for Proposal and 411 ms for Ref fallback, excluding buffering and waveform decoding. These are component timings, not end-to-end streaming latency. (Secs. 3.3 and 4.3; App. E.2, Table 27.)