acceptodds
Under review as a conference paper at ICLR 2027

VP-VAE: Rethinking Vector Quantization via Adaptive Vector Perturbation

Abstract

Vector-quantized autoencoders (VQ-VAEs) are the standard way to turn continuous signals into discrete tokens, yet their training is unstable and prone to codebook collapse. We argue that a central cause is the `coupling' between continuous representation learning and discrete codebook optimization, which forces the encoder to chase a moving codebook through non-differentiable, surrogate-gradient updates. We propose VP-VAE, a perturbation-first training paradigm that decouples representation learning from discretization by removing the dependence on codebook optimization during training. Our key insight is that, from the decoder's viewpoint, quantization primarily manifests as a bounded, local perturbation in the latent space. Instead of quantizing during training, we therefore train the decoder to be robust to latent perturbations that emulate the effect of future quantization, requiring those perturbations to be scale-aligned with the target codebook capacity and distribution-consistent with the learned latent manifold. We realize this paradigm through two instantiations. As a general prototype, we introduce an adaptive vector perturbation mechanism via Metropolis–Hastings sampling, which validates the paradigm without imposing any structural prior on the latent space. As a lightweight and practical solution, we further derive FSP (Finite Scalar Perturbation) under an approximately uniform latent prior, which offers a principled theoretical interpretation of, and a consistent empirical improvement over, FSQ-style fixed quantizers. Extensive experiments on image and audio tokenization show that both VP-VAE instantiations consistently improve reconstruction fidelity and maintain stable and balanced codebook utilization over competing baselines, across both in-domain and out-of-distribution benchmarks. These results demonstrate that our perturbation-first paradigm is a stable and effective alternative to conventional quantize-during-training pipelines. Code is available at https://anonymous.4open.science/r/VP-VAE-4DD5/.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
VQ-VAE, FSQ, Tokenization, Vector Quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CqLCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper argues that VQ-VAE-style tokenizers are unstable to train and prone to codebook collapse because representation learning and codebook optimization are coupled. Based on this view, it proposes VP-VAE, a framework that replaces quantization during training with latent perturbations that emulate future quantization error. The perturbations are designed to have a magnitude that matches the target codebook size (scale alignment) and to stay in high-density regions of the latent distribution (distribution consistency). The method has two instantiations: a general MH-based version that imposes no prior on the latent distribution, and a lightweight version called FSP. The MH-based version sets the perturbation radius from nearest-neighbor distances among latent vectors stored in a queue, accepts or rejects each perturbation with a one-step Metropolis–Hastings update whose target distribution is a kNN density estimate, and builds the codebook with K-Means after training. FSP shapes the latent variables into an approximately uniform bounded distribution, trains with perturbations given by uniform noise of bin width, and decodes to the interval centroids (the Lloyd–Max optimal values). The paper evaluates both versions on images (COCO) and audio (LibriSpeech) at four codebook capacities, in both in-domain and OOD settings, and claims that they consistently achieve high performance in both modalities.

Critical strengths
  1. The design that sets the perturbation radius by the distance to the ⌈∣S∣/G⌉\lceil|S|/G\rceil-th nearest neighbor is clever. Without a codebook, it reflects the target capacity GG in the perturbation magnitude, and it adapts to the local density automatically. In addition, because the radius is equivariant to isotropic scaling of the latent space, the encoder cannot escape the noise by enlarging its outputs. This is an interesting design property.

  2. The paper points out an inefficiency in the level placement of FSQ and shows a simple fix: decoding to the centroids. This is useful in practice. Because FSQ also places levels at both ends of the range, the end levels own only half-width cells (Remark 3). As a result, under a uniform input, the end codes are used less often and the distortion becomes larger. The experiments also show that FSP outperforms the noise-injected FSQ on all reconstruction metrics under all conditions, and this result is consistent with the difference in level placement. If the level placement is confirmed to be the main source of the improvement, this would be a useful finding that can be applied directly to existing FSQ.

Critical weaknesses
  1. The “perturbation-first” framework is an extension of noise substitution in learned compression, and its positioning should be revised.

Replacing quantization with additive noise during training and quantizing only at test time has been a standard relaxation since Ballé et al. (2017). There are also several directly related prior works.

  • Agustsson & Theis (2020) remove the train–test mismatch of this approach by using universal quantization.
  • NSVQ (Vali & Bäckström, 2022) replaces the VQ error with a vector of the same norm in a random direction. Except that it requires a codebook, this is a scale-aligned vector perturbation.

None of these works is cited. I think it would be better to cite and discuss them. FSP has a further issue. With η=1\eta=1, Eq. (8) is exactly the bin-width uniform noise of Ballé et al. In addition, according to Appendix A.2, FSP is trained with a hybrid protocol that quantizes with the STE with probability 1/2. This seems inconsistent with the claim in Section 1 that the method avoids training-time quantization and surrogate gradients.

  1. The source of the improvement of FSP is not isolated, and the settings of some baselines are questionable.

On audio, the effective number of codes computed from CVU is only about 1–2 for SimVQ and VQ-VAE, which means that they have essentially collapsed. However, the original SimVQ paper reports that almost all codes are used on audio as well, and VQ-based audio codecs such as SoundStream are trained with EMA updates and re-initialization of unused codes. If this collapse is caused by insufficient tuning rather than by the coupling, the advantage on audio is in effect a comparison with FSQ only, and the claim of consistently high performance in both modalities becomes less supported. I suggest reporting a VQ baseline with standard stabilization techniques, or SimVQ with the audio settings of the original paper. In addition, Section 4.2 states that FSP and FSQ differ only in the training-time perturbation and the decoding rule. In practice, however, FSP uses the normalization loss in Eq. (7), which is neither a perturbation nor a decoding rule, and the paper does not explicitly state whether the FSQ baseline uses this regularization or how it mixes noise and quantization. Therefore, it is unclear where the improvement comes from. I suggest an ablation of FSP that changes the decoding rule, the normalization loss, and the training procedure one at a time.

[Minor comments]

  • Prop. B.1 assumes that the density is approximately constant on the scale of a single cell, and then states convergence in probability. However, if the number of samples grows with GG fixed, DMD_M converges to the radius of the ball with mass 1/G1/G. This ball does not shrink, so the error from the assumption does not vanish as the number of samples grows. Therefore, the agreement with rcellr_{\mathrm{cell}} should be stated as an approximation, not as a limit.

  • Section 4.2 attributes the OOD advantage of VP-VAE to the training objective. However, FSP shares this objective, and FSP does not show this effect in image PSNR.

[References]

  • J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end Optimized Image Compression. ICLR, 2017.
  • E. Agustsson and L. Theis. Universally Quantized Neural Compression. NeurIPS, 2020.
  • M. H. Vali and T. Bäckström. NSVQ: Noise Substitution in Vector Quantization for Machine Learning. IEEE Access, 2022.
Questions for authors
  1. What is the MH acceptance rate for each GG? Also, does Figure 1a include rejected proposals (zero perturbation)? Showing the effective perturbation magnitude including rejections would make the scale-alignment claim clearer.

  2. Could you show per-dimension histograms of the latent variables for the trained FSP and FSQ models? For images at G=15360G=15360, the CVU of FSP is lower than that of FSQ, and I found this hard to reconcile with Corollary B.4. Such information would help readers judge how well the uniform-prior premise holds.

3 comments

Thank you for the detailed feedback and concrete experimental suggestions. We ran all requested controls and diagnostics, organized below as W1 (Part 1), W2 and minor comments (Part 2), and Q1–Q2 (Part 3). As in Table 1, "VP-VAE" refers to our MH-based instantiation.

W1. Positioning Relative to Noise Substitution

(a) Relation to noise substitution. We agree and will revise our positioning relative to the noise-substitution literature, citing and discussing Ballé et al. (2017), Agustsson & Theis (2020), and NSVQ (Vali & Bäckström, 2022) in Sections 1–2. The key distinction is the direction of design. Prior work either relaxes a predefined quantizer with noise—an integer grid (Ballé et al., 2017) or an active codebook (NSVQ)—or exactly realizes the training-time noise-channel distribution at inference through shared-dither universal quantization (Agustsson & Theis, 2020). VP-VAE reverses this order by specifying no quantizer during training, setting perturbations solely from the target capacity GG and the empirical latent distribution, and fitting the discretizer only after convergence. We will bring this distinction forward from Section 3.1 to Section 1.

Training without a reference quantizer introduces two design requirements beyond standard noise substitution: a capacity-to-scale rule linking GG and latent samples to local noise scale (e.g., the MM-th nearest-neighbor radius in Eq. (1)), and distribution consistency keeping perturbed latents in high-density regions (e.g., the MH step). Viewing discretization through capacity-calibrated perturbation rather than smoothing a given quantizer provides a unified perspective across latent priors: beyond MH-based VP-VAE (no global prior) and FSP (bounded uniform prior), applying scale alignment under N(0,I)\mathcal{N}(0,I) yields GVP, which analytically recovers a constant KL rate and competitive discretization without training-time codebooks (see Reviewer VtTf, W2).

(b) Eq. (8) at η=1\eta=1. Yes, at η=1\eta=1, Eq. (8) coincides with the bin-width uniform noise of Ballé et al. Under an approximately uniform prior on [0,1]d[0,1]^d, scale alignment naturally simplifies to the bin width, recovering the classical uniform relaxation. Carrying this uniform prior through the rest of FSP further introduces pre-activation variance matching (Eq. (7)), boundary rejection in Eq. (9) for distribution consistency, and Lloyd–Max interval centroids in Eq. (10) in place of FSQ's endpoint grid. As our ablation in W2(b) confirms, centroid decoding drives most of FSP's gain over FSQ.

(c) The 1/21/2-STE schedule in Appendix A.2. We agree that the manuscript should have clearly separated the definition of FSP from the hybrid schedule used in our main experiments rather than leaving that detail to Appendix A.2. By definition, FSP (Section 3.3, Eqs. (7)–(9)) uses pure perturbation with neither training-time quantization nor STE. Because our FSQ baseline adopts the stronger symmetric hybrid variant of Parker et al. (2024; Appendix A.2), which builds on Brendel et al.'s (2024) uniform-noise relaxation and mixes 1/21/2 noise injection with 1/21/2 STE, we trained FSP under the same schedule for a controlled comparison. Trained with perturbation alone ("Pure FSP" in W2(b), 0% STE), FSP still outperforms this FSQ baseline on every metric across both datasets. We will make this distinction explicit in Sections 1 and 4.1 alongside the Pure FSP results.

W2. Baseline Settings and the Source of FSP's Gain

(a) Audio baselines. We tested both suggested controls on the same SQCodec backbone (LibriSpeech, G=4096G=4096), and codebook collapse persists in both cases:

  • VQ-VAE with EMA + code reset. Our VQ-VAE baseline already uses EMA. Adding SoundStream-style dead-code resets still leads to NaN within the first audio epoch, although the same module trains stably on images.
  • SimVQ with adapted audio settings. We ported selected published SimVQ audio settings to SQCodec and tested commitment weights β∈10,100,1000\beta\in\\{10,100,1000\\} for 20 epochs each. All four runs collapse to about one effective code with PESQ below 1.41.4. Setting β=1000\beta=1000 briefly activates 218 codes at epoch 2 (3.63.6 effective codes) before collapsing again by epoch 8, without improving PESQ or STOI.

On the same backbone, VP-VAE activates nearly all 4,096 codes from epoch 1, maintains 100–200 effective codes, and steadily improves PESQ and STOI:

Audio training trajectories at G=4096: four SimVQ configurations vs. VP-VAE

Frequent silent and low-energy speech frames concentrate latents, limiting CVU even without collapse (0.0290.029–0.0560.056 in Table 1, or roughly 130130–230230 effective codes at G=4096/4375G=4096/4375 for FSQ, FSP, and VP-VAE). With coupled codebook updates, this concentration likely amplifies early assignment bias into single-code collapse (Section 4.2). These controls show that standard stabilization and the adapted SimVQ settings tested here do not prevent collapse on SQCodec without further hyperparameter search, whereas VP-VAE and FSP train stably without modality-specific tuning and outperform the non-collapsed FSQ baseline across all four capacities (Tables 1–2). We will include these controls in the appendix.

(b) FSQ setup and step-by-step FSP ablation. Our FSQ baseline mixes quantization (with STE) and uniform grid-step noise with probability 1/21/2 each (Parker et al., 2024; Appendix A.2), and does not use the normalization loss in Eq. (7). From FSQ, we change decoding, perturbation, normalization loss, and training procedure one at a time (G=4375G=4375, COCO / ImageNet):

ConfigurationCVU ↑LPIPS ↓PSNR ↑SSIM ↑rFID ↓rKID (×10³) ↓
FSQ baseline0.797 / 0.7760.165 / 0.17524.45 / 24.420.727 / 0.7073.80 / 0.672.25 / 2.65
+ centroid decoding (Eq. (10))0.804 / 0.7810.148 / 0.16024.79 / 24.950.738 / 0.7253.48 / 0.580.88 / 0.94
+ FSP perturbation (Eqs. (8)–(9))0.812 / 0.8010.151 / 0.15924.83 / 24.930.748 / 0.7313.47 / 0.611.06 / 1.39
+ normalization loss (Eq. (7)), i.e., FSP0.834 / 0.8160.151 / 0.15824.89 / 25.000.747 / 0.7323.50 / 0.540.83 / 1.21
− STE (pure perturbation), i.e., Pure FSP0.827 / 0.8110.152 / 0.16124.61 / 24.750.745 / 0.7293.53 / 0.611.00 / 1.32
  • Decoding rule. Replacing FSQ's endpoint levels with interval centroids (Eq. (10)) captures +0.34+0.34 / +0.53+0.53 dB of the total +0.44+0.44 / +0.58+0.58 dB PSNR gain and cuts rKID from 2.252.25 / 2.652.65 to 0.880.88 / 0.940.94. This confirms your point in Strengths: centroid placement is the main source of reconstruction gain and can be applied directly to existing FSQ.
  • Perturbation, normalization loss, and training procedure. Adding FSP's training-time perturbation (Eqs. (8)–(9)) and the normalization loss in Eq. (7) mainly improves codebook utilization: CVU rises from 0.8040.804 / 0.7810.781 to 0.8340.834 / 0.8160.816, whereas PSNR changes by at most 0.070.07 dB per step. Removing STE ("Pure FSP") retains strong utilization and outperforms the FSQ baseline on all metrics.

We will detail the FSQ setup in Section 4.1, include this ablation, and update Section 4.2 to list all differences between FSP and FSQ and credit centroid decoding as the main driver of reconstruction quality.

Minor Comments

Minor 1 (Prop. B.1). Agreed. With Fz(r)=Pr⁡(∣Z−z∣≤r)F_z(r)=\Pr(\\|Z-z\\|\le r), fixing GG and taking M=⌈N/G⌉M=\lceil N/G\rceil gives the exact limit DM(z∣S)→prG(z):=Fz−1(1/G)D_M(z\mid\mathcal{S})\xrightarrow{p} r_G(z):=F_z^{-1}(1/G) as N→∞N\to\infty. Connecting rG(z)r_G(z) to [Gρ(z)Vd]−1/d[G\rho(z)V_d]^{-1/d} relies on the local constant-density approximation 1/G=∫B(z,rG)ρ(u),du≈ρ(z)Vd,rG(z)d1/G=\int_{B(z,r_G)}\rho(u)\\,du\approx\rho(z)V_d\\,r_G(z)^d, whose approximation error need not vanish as NN grows at fixed GG. We will separate the exact quantile limit from the local density approximation in Prop. B.1 and Section 3.2.

Minor 2 (OOD attribution in Section 4.2). Thank you for pointing this out. Since FSP uses the same overall training objective without showing the same OOD image-PSNR pattern, we will revise Section 4.2 to state this explanation as a hypothesis specific to MH-based VP-VAE.

Q1. MH Acceptance Rates and Rejected Proposals in Figure 1a

Acceptance rates. Across 50 COCO training epochs, mean MH acceptance rates stay in a stable 0.530.53–0.620.62 range and are non-monotone in GG:

Target codebook size GGOverall meanFirst epochLast 10% of training
2560.6240.5360.627
10240.5710.4760.576
40960.5290.4040.537
163840.5830.4470.592

Rejected proposals in Figure 1a. Figure 1a plots only accepted moves (∣z~−z∣2>10−6\\|\tilde{z}-z\\|_2>10^{-6}). Including rejected proposals adds a spike at zero, shown below on 57,344 validation tokens at G=4096G=4096 (45.9% rejected, matching the 53.7% late-training acceptance rate above):

Training perturbation magnitude including rejected proposals vs. inference quantization error (G=4096)

Accepted moves have a mean norm of 0.49080.4908, closely matching the mean inference quantization error of 0.52130.5213. Including rejections brings the overall mean displacement to 0.541×0.4908=0.26550.541\times0.4908=0.2655, roughly half the quantization error. Scale alignment thus holds conditioned on acceptance, so the decoder trains on clean latents for roughly half the tokens and scale-matched perturbations for the rest. We will clarify this in Figure 1a's caption and add both the acceptance table and the full distribution plot to the appendix.

Q2. Per-Dimension Histograms and FSP's CVU at G=15360G=15360

Checking the uniform-prior premise. Below are 100-bin per-dimension histograms of continuous latents across all 5,000 COCO validation images against the dashed U[0,1]\mathcal{U}[0,1] reference line (density =1=1), shown with a logarithmic density axis. On this log scale, FSP stays near 11 across the interior with dips to about 0.30.3 just inside 00 and 11, whereas FSQ reaches 88–15×15\times the reference density at the endpoints and remains below 11 (around 0.50.5–0.80.8) across a relatively flat interior. Overall, FSP matches U[0,1]\mathcal{U}[0,1] more closely in per-dimension continuous KL (table below), with deviations localized near the boundaries:

Per-dimension continuous latent marginals of FSP and FSQ (G=4375 and G=15360)

Why FSP has lower CVU at G=15360G=15360. While per-dimension continuous KL measures proximity to U[0,1]\mathcal{U}[0,1], CVU reflects the discrete probability mass inside each quantization cell. Specifically, −log⁡CVU-\log\mathrm{CVU} decomposes exactly into per-dimension marginal imbalance ∑i[log⁡Li−H(Ki)]\sum_i[\log L_i-H(K_i)] (zero iff all LiL_i levels in dimension ii have equal frequency) plus cross-dimensional total correlation ∑iH(Ki)−H(K)\sum_i H(K_i)-H(K) (measuring statistical dependence across dimensions). On a random subset of 1,000 validation images at G=15360G=15360, total correlation is nearly identical for FSP (0.0800.080) and FSQ (0.0780.078), and marginal imbalance accounts for 97.1% of the CVU difference:

GGMethodKL to U[0,1]\mathcal{U}[0,1] ↓Marginal imbalance ↓
4375FSQ0.4640.152
4375FSP0.0700.124
15360FSQ0.3170.147
15360FSP0.0710.212

The discrete level frequencies below show how cell geometry interacts with these continuous marginals. In the three 8-level dimensions at G=15360G=15360, FSP's outermost equal-width cells ([0,1/8)[0,1/8) and [7/8,1][7/8,1]) become narrow enough to fall largely inside its boundary dips (0.50.5–0.6×0.6\times uniform frequency), increasing marginal imbalance from 0.1240.124 (G=4375G=4375) to 0.2120.212. In FSQ, the two endpoint cells have half the width of interior cells (Remark 3), which offsets its higher endpoint density to yield boundary cell frequencies of 1.31.3–1.8×1.8\times and lower discrete marginal imbalance (0.1470.147) at this resolution:

Discrete level frequencies of FSP and FSQ (G=4375 and G=15360)

Corollary B.4 assumes an exact U[0,1]\mathcal{U}[0,1] input in each dimension; in practice, closer continuous fit to U[0,1]\mathcal{U}[0,1] across the interior does not automatically imply higher discrete CVU at fine grid resolutions when narrow outer bins intersect boundary dips. We will add both plots and the CVU decomposition to the appendix, and note this boundary discretization effect alongside Corollary B.4.

Reviewer_cc5ZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents an approach to optimise the VQ-VAE quantiser by using dithering instead of quantisation during training. To implement dithering with the same distribution as the quantiser, it uses Metropolis-Hastings sampling. By estimating the accuracy of the final quantiser, it avoids the need to train the VQ during main-pass training, thus reducing complexity.

Critical strengths
  • Including MH-sampling in dithering seems to be novel and valuable.
  • Simulating codebook accuracy removes the need to optimise the codebook during the main pass of training, reducing complexity and avoiding codebook collapse (in the sense that some elements would not be used).
  • Theoretical analysis of dithering is useful.
Critical weaknesses
  • The paper does not cite papers by Mohammad Vali (see below) that address the same issue with very similar approaches (though not identical). Because they are so similar, a comparison is essential to assess performance differences. Of these, NSVQ inserts noise like the proposed method but with a simpler scaling approach, and DiVeQ avoids dithering by modifying the gradients. The latter will, in theory, be more accurate than NSVQ and the method proposed in this paper. The key difference in the method proposed here is using MH-sampling to generate noise with the same distribution and scaling as the true quantisation noise. This is potentially more accurate than NSVQ in that the noise is closer to the true noise distribution. Because DiVeQ uses the actual quantisation to generate the noise but reparametrises the quantiser to generate gradients, it is likely the most accurate of the three.
    • Vali, M. H., & Bäckström, T. (2022). NSVQ: Noise substitution in vector quantisation for machine learning. IEEE Access, 10, 3598-13610.
    • Vali, M. H., Bäckström, T., & Solin, A. (2025). DiVeQ: Differentiable vector quantisation using the reparameterization trick. arXiv preprint arXiv:2509.26469.
  • Probably not a big issue, but worth mentioning: the final VQ is trained with an error criterion in the latent space, not in the output space. Since the mapping between latent and output spaces can be highly non-linear, this is risky and can provide a non-optimal codebook.
Questions for authors
1 comment

Thank you for pointing us to NSVQ and DiVeQ and for the thoughtful comment on clustering geometry. Below we address W1 and W2 ("VP-VAE" refers to our MH-based instantiation, as in Table 1).

W1. Comparison with NSVQ and DiVeQ

NSVQ and DiVeQ are closely related to our work. We will discuss both in Section 2 and include a comprehensive comparison with the more recent DiVeQ in the revised paper.

Conceptual difference. NSVQ and DiVeQ train with an explicit codebook alongside the encoder, using random-direction noise matched to the quantization error norm (NSVQ) or a reparameterized nearest-codeword step (DiVeQ) as differentiable surrogates for the current codebook's error. VP-VAE removes the codebook from training altogether: it sets perturbations directly from the target capacity GG and the current latent distribution (via scale alignment and distribution consistency), and fits the codebook only after convergence. Decoupling noise design from an active codebook shifts the view from approximating an explicit quantizer's gradient to a perturbation-first perspective: beyond non-parametric (MH-based VP-VAE) and bounded uniform (FSP) settings, applying scale alignment under a standard Gaussian prior yields GVP, which already reaches competitive quality without a training-time codebook (see our response to Reviewer VtTf, W2).

Empirical comparison. We trained NSVQ and DiVeQ under our exact setup (VQGAN backbone, 50 COCO epochs, G=1024G=1024, evaluated on COCO / ImageNet):

MethodCVU ↑LPIPS ↓PSNR ↑SSIM ↑rFID ↓rKID (×10³) ↓
NSVQ0.948 / 0.9380.172 / 0.18023.86 / 24.140.720 / 0.7083.82 / 0.662.39 / 2.06
DiVeQ0.894 / 0.8810.166 / 0.17323.96 / 24.130.721 / 0.7083.75 / 0.621.54 / 1.75
VP-VAE0.852 / 0.8380.171 / 0.17623.96 / 24.250.724 / 0.7183.88 / 0.681.44 / 1.74

On both datasets, VP-VAE outperforms NSVQ on LPIPS, PSNR, SSIM, and rKID, while NSVQ attains higher CVU and slightly lower rFID. Compared with DiVeQ, VP-VAE achieves better SSIM and rKID and equal or higher PSNR, while DiVeQ leads on LPIPS, rFID, and CVU. Overall, VP-VAE and DiVeQ reach a very similar performance level.

Per-step error fidelity vs. decoupled representation learning. These results reflect two distinct factors in discrete autoencoder training. First, they are consistent with your analysis of noise geometry: VP-VAE uses MH sampling to match local scale and account for the empirical latent density rather than perturbing along a random direction as in NSVQ, and achieves better LPIPS, PSNR, SSIM, and rKID on both datasets (e.g., cutting COCO rKID from 2.392.39 to 1.441.44). Second, comparing VP-VAE with DiVeQ suggests a complementary benefit of codebook-free training. DiVeQ provides exact quantization error relative to the current codebook, but its encoder must continuously track a moving, partially converged codebook throughout training. Without a training-time codebook, VP-VAE lets the continuous representation converge under stable, capacity-calibrated perturbations before fitting K-Means++ once at the end. With this decoupled training, VP-VAE matches DiVeQ overall and edges ahead on SSIM (0.724/0.7180.724 / 0.718 vs. 0.721/0.7080.721 / 0.708) and ImageNet PSNR (24.2524.25 vs. 24.1324.13 dB), while keeping the simpler codebook-free main pass noted in your review.

W2. Clustering Criterion: Latent Space vs. Output Space

Thank you for raising this geometric point. On an unregularized continuous autoencoder, Euclidean distance in zz can indeed misalign with output-space distortion: in our ablation without training perturbations (no perturbations in Table 3), post-training K-Means++ achieves balanced cluster assignments (CVU=0.877\mathrm{CVU}=0.877) yet reconstruction drops sharply to 22.2222.22 dB PSNR.

VP-VAE addresses this during representation learning rather than during clustering. Whereas coupled methods such as DiVeQ adapt codewords to the output loss Lrec(x,x^)\mathcal{L}_{\mathrm{rec}}(x,\hat{x}), VP-VAE perturbs zz within a local Euclidean radius R(z)R(z) and optimizes both the encoder and nonlinear decoder end-to-end under Lrec(x,x^)\mathcal{L}_{\mathrm{rec}}(x,\hat{x}). Training penalizes sensitivity to local Euclidean displacements in zz, encouraging Euclidean distance to serve as a useful proxy for output distortion. Post-training K-Means++ in zz then clusters under the same local metric that the decoder was trained to tolerate (Figure 1a): quantized PSNR rises from 22.2222.22 to 24.6424.64 dB (Table 3), comparable to output-optimized codebooks such as DiVeQ in W1. We will add this discussion to Section 3.4. Extending our perturbation-first framework to perceptual or non-Euclidean latent metrics is also an interesting direction for future work.

Reviewer_VtTfOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Summary

  • This paper propose a general approach to avoid discrete training in discrete tokenizers. It propose to train a continous auto-encoder with perturbed latent simulating quantization error.
  • Based on this approach, this paper instantiates two discrete image / audio tokenizers, one is VP-VAE based on VQ-VAE, another one is FSQ based on FSQ.
Critical strengths

Strength

  • This paper achieves discrete tokenization without continous training, avoid STE and any other loss required by discrete training, as Tokenbridge and GaussainQuant.
  • Better than image tokenization paper such as FSQ and LFQ, this paper also evaluate their result on audio tokenization.
Critical weaknesses

Weakness

  • It would be a lot more convincing if the authors could provide auto-regressive generation results such as gFID, IS and generation related VQ-VAE metrics such as codebook utilization. Among the baselines, TokenBridge and FSQ are clearly designed as image tokenizer, and their primary objective is to serve as tokenizer for generation rather for reconstruction. Please consider providing generation results, toy setting with ImageNet dataset is convincing enough.
  • One particularly relevant work, GaussianQuant [ICML 26 Training-Free Vector Quantization via Gaussian VAEs] is missing from discussion and comparsion. Similar to this paper, GaussianQuant trains a continous VAE with fixed KL divergence term per dimension and construct construct VQ-VAE from continously trained VAE. It might be better to discuss and compare to this work. Besides, other relevant recent tokenizers such as LFQ and BSQ might also worth to compare to.
  • Although the MH perturbation looks insightful, its empirical performance is not way better than vanilla Gaussian noise according to Table 3.
  • In Table 1, please write PSNR as 23.17 rather than 23.1662 dB. We rarely report PSNR, rFID and rKID with so many digits.
  • In Table 1 and Figure 2, please confirm that the TokenBridge implementation is correct. TokenBridge will lose some PSNR from continous Gaussian VAE, but it should not produce broken images with PSNR 11 dB.
  • Table 3 seems to be a pretty important ablation study, could you please move it back to main text? Besides, can the authors provide un-discretized / perturbed latent reconstruction performance in Table 3?
Questions for authors

Questions

  • See weakness.

Justification

  • Overall I recommend to accept this paper. This paper propose an interesting approach to obtain a discrete tokenizer with continous training. Its performance is validated based on both VQ-VAE and FSQ, with both image and audio modality.
3 comments

Thank you for the positive evaluation and helpful suggestions. Below we address W1–W6 in order ("VP-VAE" refers to our MH-based instantiation, as in Table 1).

W1. Autoregressive Generation and Codebook Utilization

We agree that downstream generation is an essential test for discrete tokenizers. Appendix C.6 (Table 6) already reports AR generation on COCO, though the short rebuttal window did not allow an additional ImageNet run. Freezing each G=4096G=4096 tokenizer (43754375 codes for FSQ/FSP), we train the same 18-layer AR Transformer on its tokens and sample 5,000 images (temperature 1.01.0, top-k=16k=16):

TokenizerFID ↓KID ↓Precision ↑Recall ↑NLL ↓
VQ-VAE17.5125.210.9140.5708.94
SimVQ13.2424.630.8940.7489.07
FSQ14.5630.860.8850.7019.47
FSP13.7220.960.8720.7549.54
VP-VAE13.9020.640.8800.7409.37

VP-VAE achieves the best KID and FSP the best Recall; both come within 0.70.7 FID of SimVQ and outperform FSQ on FID, KID, and Recall. Consistent with Table 1's validation codebook utilization (CVU=0.240\mathrm{CVU}=0.240 vs. 0.7970.797–0.8600.860 for the others), VQ-VAE's concentrated usage eases next-token prediction (lowest NLL, highest Precision) at the cost of sample diversity and overall quality (lowest Recall, worst FID). Overall, continuous perturbation training yields balanced vocabularies that transfer directly to competitive AR generation without a jointly optimized codebook. We will move Table 6 (with corrected TokenBridge, W5) to the main text.

W2. GaussianQuant, LFQ, and BSQ

Thank you for pointing out GaussianQuant (GQ). We will expand Section 2 to discuss GQ, LFQ, and BSQ, and compare with GQ below.

Relation to GQ. Both methods discretize a continuously trained autoencoder after convergence, but via different mechanisms. GQ trains a Gaussian VAE with a learned variance head under a fixed per-dimension KL target calibrated to codebook size KK (Target Divergence Constraint, TDC), then draws a random Gaussian codebook. VP-VAE instead injects local perturbations matched to target capacity and the latent distribution without a variance head or KL target, emulating quantization error directly rather than inducing quantizability through a KL rate constraint.

GVP: what our perturbation-first perspective yields under GQ's Gaussian prior. To make this contrast concrete, we ask what our perturbation-first view produces under the same N(0,Im)\mathcal{N}(0,I_m) prior and random Gaussian codebook as GQ, which yields GVP (Gaussian Vector Perturbation). Following GQ's notation (codebook size KK, i.e., our GG, and dimension mm), for a latent mean μ∈Rm\mu\in\mathbb{R}^m, ϵ∼N(0,Im)\epsilon\sim\mathcal{N}(0,I_m), and KK random codewords Cj∼N(0,Im)C_j\sim\mathcal{N}(0,I_m), matching perturbation variance to the expected nearest-codeword error gives (via the noncentral χ2\chi^2 CDF Fm,∣μ∣2F_{m,\\|\mu\\|^2}):

vK,m(∣μ∣2)=1m,E[min⁡j≤K∣μ−Cj∣2]=1m∫0∞[1−Fm,∣μ∣2(s)]Kds,z=μ+vK,m,ϵ.v_{K,m}(\\|\mu\\|^2)=\frac{1}{m}\\,\mathbb{E}\Big[\min_{j\le K}\\|\mu-C_j\\|^2\Big]=\frac{1}{m}\int_0^\infty\big[1-F_{m,\\|\mu\\|^2}(s)\big]^K ds,\qquad z=\mu+\sqrt{v_{K,m}}\\,\epsilon .

Asymptotically, vK,m∼ame∣μ∣2/mK−2/mv_{K,m}\sim a_m e^{\\|\mu\\|^2/m}K^{-2/m}, which cancels the ∣μ∣2\\|\mu\\|^2 term in the Gaussian KL:

DKL(N(μ,vK,mIm),∥,N(0,Im))/log⁡2=log⁡2K−cm+o(1),D_{\mathrm{KL}}\big(\mathcal{N}(\mu,v_{K,m}I_m)\\,\Vert\\,\mathcal{N}(0,I_m)\big)/\log 2=\log_2 K-c_m+o(1),

where am,cma_m, c_m depend only on mm. Thus, the position-independent KL rate that GQ enforces via TDC emerges naturally from scale matching itself. GVP maps encoder features to μ\mu via a lightweight invertible layer, regularizes μ\mu toward N(0,Im)\mathcal{N}(0,I_m) by moment matching or a sliced-Wasserstein (SW) loss, and discretizes with GQ's random N(0,Im)\mathcal{N}(0,I_m) codebook at inference.

Results (VQGAN backbone, 50 COCO epochs, K=1024K=1024, evaluated on COCO / ImageNet):

MethodCVU ↑LPIPS ↓PSNR ↑SSIM ↑rFID ↓rKID (×10³) ↓
GaussianQuant0.885 / 0.8730.168 / 0.17624.41 / 24.600.723 / 0.7103.79 / 0.631.70 / 1.94
VP-VAE0.852 / 0.8380.171 / 0.17623.96 / 24.250.724 / 0.7183.88 / 0.681.44 / 1.74
GVP (moments)0.889 / 0.8800.168 / 0.17623.92 / 24.100.715 / 0.7033.90 / 0.651.06 / 1.68
GVP (SW)0.943 / 0.9380.168 / 0.17623.97 / 24.150.712 / 0.7003.88 / 0.661.14 / 1.71

VP-VAE and GQ perform comparably overall, with VP-VAE ahead on SSIM and rKID and GQ ahead on PSNR and rFID. Without TDC or a learned variance head, GVP reaches the same regime: relative to VP-VAE, it improves CVU and rKID and stays within 0.150.15 dB in PSNR, at a slight cost in SSIM. More broadly, GVP illustrates that our core contribution is a shift to a perturbation-first perspective rather than two isolated algorithms: starting from capacity-matched perturbations, effective tokenizers arise naturally across non-parametric (VP-VAE), bounded uniform (FSP), and Gaussian (GVP) geometries.

Distribution consistency in GVP. Because z=μ+vK,m(∥μ∥2) ϵz=\mu+\sqrt{v_{K,m}(\|\mu\|^2)}\,\epsilon slightly inflates the marginal variance of zz beyond N(0,Im)\mathcal{N}(0,I_m) (by an O(K−2/m)O(K^{-2/m}) radial term), GVP does not enforce distribution consistency as MH-based VP-VAE (MH rejection) or FSP ([0,1]d[0,1]^d boundary rejection) does. Under a smooth, unimodal Gaussian prior, however, small capacity-scaled perturbations still keep zz within the high-density support of N(0,Im)\mathcal{N}(0,I_m) rather than pushing latents into low-density voids, providing approximate distribution consistency.

LFQ and BSQ. Like FSQ, LFQ and BSQ quantize latents onto fixed binary hypercube or spherical grids during training via STE. Since FSQ already represents this fixed-grid family in our benchmarks, and GQ—which is closer to our continuous-training setup—reports stronger reconstruction than LFQ and BSQ on UNet and ViT backbones, we prioritized GQ in our rebuttal experiments and will discuss LFQ and BSQ in Section 2.

W3. MH Perturbation vs. Gaussian Noise (Table 3)

Thank you for raising this point. The label "fixed Gaussian perturbations" in Table 3 and its description in Appendix C.2 ("a tuned, globally constant variance") were inaccurate. Because latent scales evolve during training and a constant variance cannot be set fairly, that variant actually injects isotropic Gaussian noise whose standard deviation equals the batch mean of our adaptive radius R(z)R(z). It therefore retains VP-VAE's capacity-driven scale, which provides most of the PSNR gain over unperturbed training (24.5924.59 vs. 22.2222.22 dB, compared with 24.6424.64 dB for the full model).

Using our per-token uniform-ball proposal of radius R(z)R(z) without MH rejection (w/o MH) isolates MH itself: accepting all proposals drops CVU from 0.8490.849 to 0.8030.803 while leaving PSNR nearly unchanged (24.5924.59 dB). Scale alignment thus drives reconstruction quality, whereas MH enforces distribution consistency and improves codebook utilization (CVU and Figure 1b). We will rename the Gaussian row, fix Appendix C.2, and refine Section 4.3 to state this clearly.

W4. Decimal Precision in Table 1

Agreed. We will round all table entries to standard precision (e.g., 23.1723.17 dB).

W5. TokenBridge Implementation

Thank you for catching this. While reproducing GQ at your suggestion, we found an incorrect KL weight in our TokenBridge implementation. After fixing it, results at G=45=1024G=4^5=1024 on the full COCO / ImageNet validation sets are:

MethodCVU ↑LPIPS ↓PSNR ↑SSIM ↑rFID ↓rKID (×10³) ↓
TokenBridge (continuous)–0.141 / 0.15025.90 / 26.060.771 / 0.7583.14 / 0.491.00 / 1.16
TokenBridge (quantized)0.854 / 0.8420.320 / 0.33020.85 / 20.850.620 / 0.6097.48 / 2.6721.03 / 17.18
FSP (G=1000G=1000)0.863 / 0.8490.167 / 0.18024.21 / 24.220.723 / 0.7023.84 / 0.671.66 / 2.02
VP-VAE0.852 / 0.8380.171 / 0.17623.96 / 24.250.724 / 0.7183.88 / 0.681.44 / 1.74

As you expected, fixing the KL weight restores clean continuous reconstructions (25.9025.90 dB PSNR) and eliminates the broken images. Moreover, this corrected baseline makes the comparison in Section 4.2 much clearer: post-training discretization of the unperturbed Gaussian VAE drops COCO PSNR from 25.9025.90 to 20.8520.85 dB, well below FSP (24.2124.21 dB) and VP-VAE (23.9623.96 dB). This gap directly shows the benefit of preparing the decoder with scale-aligned perturbations during training. We will replace all TokenBridge results (Tables 1–2 and Figure 2) with corrected runs across every codebook size and on audio, add it to Table 6, and update Section 4.2.

W6. Moving Table 3 and Reporting Continuous / Perturbed Reconstruction

We will move Table 3 to the main text and include reconstruction metrics across all three decoder inputs on the same G=4096G=4096 COCO checkpoint:

Decoder inputLPIPS ↓PSNR ↑SSIM ↑rFID ↓rKID (×10³) ↓
Continuous zz0.12726.070.7752.950.94
Perturbed z~\tilde{z}0.13925.340.7613.230.83
Quantized q(z)q(z)0.15424.640.7483.610.91

PSNR decreases in two balanced steps, by 0.730.73 dB from zz to z~\tilde{z} and 0.700.70 dB from z~\tilde{z} to q(z)q(z) (1.431.43 dB overall). Perturbed z~\tilde{z} sits midway between zz and q(z)q(z) in PSNR, consistent with roughly 46% of MH proposals being rejected (z~=z\tilde{z}=z), so the average displacement in z~\tilde{z} is about half the inference quantization error. Overall, training on z~\tilde{z} keeps the end-to-end z→q(z)z \to q(z) gap to 1.431.43 dB while raising quantized PSNR from 22.2222.22 dB (no perturbations in Table 3) to 24.6424.64 dB.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.