VP-VAE: Rethinking Vector Quantization via Adaptive Vector Perturbation

Vector-quantized autoencoders (VQ-VAEs) are the standard way to turn continuous signals into discrete tokens, yet their training is unstable and prone to codebook collapse. We argue that a central cause is the `coupling' between continuous representation learning and discrete codebook optimization, which forces the encoder to chase a moving codebook through non-differentiable, surrogate-gradient updates. We propose VP-VAE, a perturbation-first training paradigm that decouples representation learning from discretization by removing the dependence on codebook optimization during training. Our key insight is that, from the decoder's viewpoint, quantization primarily manifests as a bounded, local perturbation in the latent space. Instead of quantizing during training, we therefore train the decoder to be robust to latent perturbations that emulate the effect of future quantization, requiring those perturbations to be scale-aligned with the target codebook capacity and distribution-consistent with the learned latent manifold. We realize this paradigm through two instantiations. As a general prototype, we introduce an adaptive vector perturbation mechanism via Metropolis–Hastings sampling, which validates the paradigm without imposing any structural prior on the latent space. As a lightweight and practical solution, we further derive FSP (Finite Scalar Perturbation) under an approximately uniform latent prior, which offers a principled theoretical interpretation of, and a consistent empirical improvement over, FSQ-style fixed quantizers. Extensive experiments on image and audio tokenization show that both VP-VAE instantiations consistently improve reconstruction fidelity and maintain stable and balanced codebook utilization over competing baselines, across both in-domain and out-of-distribution benchmarks. These results demonstrate that our perturbation-first paradigm is a stable and effective alternative to conventional quantize-during-training pipelines. Code is available at https://anonymous.4open.science/r/VP-VAE-4DD5/.