Agreeing on Allocation: Shared-Prefix Selective Adaptation for Frozen Speech Codecs

Neural speech codecs typically allocate a uniform code budget across all frames. Adaptive frame-wise allocation can improve coding efficiency but often requires retraining or extra signaling. We introduce Agreeing on Allocation, a post-hoc method that distributes a fixed code budget in frozen codecs. Using the mandatory first-stage residual vector quantization (RVQ) codes as a shared prefix, the encoder and decoder independently recover the same budget-constrained stage assignment without frame-wise signaling. However, agreement alone does not guarantee better reconstruction. We therefore add selective deployment, in which the encoder compares proposed and reference reconstructions at the same budget using fidelity and perceptual criteria and signals only an utterance-level choice. For the fixed selection rule, held-out evaluation provides a finite-sample upper confidence bound on the probability of a quality drop beyond a predefined margin relative to the reference. Objective and subjective evaluations on frozen neural speech codecs show quality improvements at matched code budgets, while selective deployment reduces the risk of substantial quality drops.