- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper argues that VQ-VAE-style tokenizers are unstable to train and prone to codebook collapse because representation learning and codebook optimization are coupled. Based on this view, it proposes VP-VAE, a framework that replaces quantization during training with latent perturbations that emulate future quantization error. The perturbations are designed to have a magnitude that matches the target codebook size (scale alignment) and to stay in high-density regions of the latent distribution (distribution consistency). The method has two instantiations: a general MH-based version that imposes no prior on the latent distribution, and a lightweight version called FSP. The MH-based version sets the perturbation radius from nearest-neighbor distances among latent vectors stored in a queue, accepts or rejects each perturbation with a one-step Metropolis–Hastings update whose target distribution is a kNN density estimate, and builds the codebook with K-Means after training. FSP shapes the latent variables into an approximately uniform bounded distribution, trains with perturbations given by uniform noise of bin width, and decodes to the interval centroids (the Lloyd–Max optimal values). The paper evaluates both versions on images (COCO) and audio (LibriSpeech) at four codebook capacities, in both in-domain and OOD settings, and claims that they consistently achieve high performance in both modalities.
Critical strengths
-
The design that sets the perturbation radius by the distance to the -th nearest neighbor is clever. Without a codebook, it reflects the target capacity in the perturbation magnitude, and it adapts to the local density automatically. In addition, because the radius is equivariant to isotropic scaling of the latent space, the encoder cannot escape the noise by enlarging its outputs. This is an interesting design property.
-
The paper points out an inefficiency in the level placement of FSQ and shows a simple fix: decoding to the centroids. This is useful in practice. Because FSQ also places levels at both ends of the range, the end levels own only half-width cells (Remark 3). As a result, under a uniform input, the end codes are used less often and the distortion becomes larger. The experiments also show that FSP outperforms the noise-injected FSQ on all reconstruction metrics under all conditions, and this result is consistent with the difference in level placement. If the level placement is confirmed to be the main source of the improvement, this would be a useful finding that can be applied directly to existing FSQ.
Critical weaknesses
- The “perturbation-first” framework is an extension of noise substitution in learned compression, and its positioning should be revised.
Replacing quantization with additive noise during training and quantizing only at test time has been a standard relaxation since Ballé et al. (2017). There are also several directly related prior works.
- Agustsson & Theis (2020) remove the train–test mismatch of this approach by using universal quantization.
- NSVQ (Vali & Bäckström, 2022) replaces the VQ error with a vector of the same norm in a random direction. Except that it requires a codebook, this is a scale-aligned vector perturbation.
None of these works is cited. I think it would be better to cite and discuss them. FSP has a further issue. With , Eq. (8) is exactly the bin-width uniform noise of Ballé et al. In addition, according to Appendix A.2, FSP is trained with a hybrid protocol that quantizes with the STE with probability 1/2. This seems inconsistent with the claim in Section 1 that the method avoids training-time quantization and surrogate gradients.
- The source of the improvement of FSP is not isolated, and the settings of some baselines are questionable.
On audio, the effective number of codes computed from CVU is only about 1–2 for SimVQ and VQ-VAE, which means that they have essentially collapsed. However, the original SimVQ paper reports that almost all codes are used on audio as well, and VQ-based audio codecs such as SoundStream are trained with EMA updates and re-initialization of unused codes. If this collapse is caused by insufficient tuning rather than by the coupling, the advantage on audio is in effect a comparison with FSQ only, and the claim of consistently high performance in both modalities becomes less supported. I suggest reporting a VQ baseline with standard stabilization techniques, or SimVQ with the audio settings of the original paper. In addition, Section 4.2 states that FSP and FSQ differ only in the training-time perturbation and the decoding rule. In practice, however, FSP uses the normalization loss in Eq. (7), which is neither a perturbation nor a decoding rule, and the paper does not explicitly state whether the FSQ baseline uses this regularization or how it mixes noise and quantization. Therefore, it is unclear where the improvement comes from. I suggest an ablation of FSP that changes the decoding rule, the normalization loss, and the training procedure one at a time.
[Minor comments]
-
Prop. B.1 assumes that the density is approximately constant on the scale of a single cell, and then states convergence in probability. However, if the number of samples grows with fixed, converges to the radius of the ball with mass . This ball does not shrink, so the error from the assumption does not vanish as the number of samples grows. Therefore, the agreement with should be stated as an approximation, not as a limit.
-
Section 4.2 attributes the OOD advantage of VP-VAE to the training objective. However, FSP shares this objective, and FSP does not show this effect in image PSNR.
[References]
- J. Ballé, V. Laparra, and E. P. Simoncelli. End-to-end Optimized Image Compression. ICLR, 2017.
- E. Agustsson and L. Theis. Universally Quantized Neural Compression. NeurIPS, 2020.
- M. H. Vali and T. Bäckström. NSVQ: Noise Substitution in Vector Quantization for Machine Learning. IEEE Access, 2022.
Questions for authors
-
What is the MH acceptance rate for each ? Also, does Figure 1a include rejected proposals (zero perturbation)? Showing the effective perturbation magnitude including rejections would make the scale-alignment claim clearer.
-
Could you show per-dimension histograms of the latent variables for the trained FSP and FSQ models? For images at , the CVU of FSP is lower than that of FSQ, and I found this hard to reconcile with Corollary B.4. Such information would help readers judge how well the uniform-prior premise holds.
3 comments
W2. Baseline Settings and the Source of FSP's Gain
(a) Audio baselines. We tested both suggested controls on the same SQCodec backbone (LibriSpeech, ), and codebook collapse persists in both cases:
- VQ-VAE with EMA + code reset. Our VQ-VAE baseline already uses EMA. Adding SoundStream-style dead-code resets still leads to
NaNwithin the first audio epoch, although the same module trains stably on images. - SimVQ with adapted audio settings. We ported selected published SimVQ audio settings to SQCodec and tested commitment weights for 20 epochs each. All four runs collapse to about one effective code with PESQ below . Setting briefly activates 218 codes at epoch 2 ( effective codes) before collapsing again by epoch 8, without improving PESQ or STOI.
On the same backbone, VP-VAE activates nearly all 4,096 codes from epoch 1, maintains 100–200 effective codes, and steadily improves PESQ and STOI:
Audio training trajectories at G=4096: four SimVQ configurations vs. VP-VAE
Frequent silent and low-energy speech frames concentrate latents, limiting CVU even without collapse (– in Table 1, or roughly – effective codes at for FSQ, FSP, and VP-VAE). With coupled codebook updates, this concentration likely amplifies early assignment bias into single-code collapse (Section 4.2). These controls show that standard stabilization and the adapted SimVQ settings tested here do not prevent collapse on SQCodec without further hyperparameter search, whereas VP-VAE and FSP train stably without modality-specific tuning and outperform the non-collapsed FSQ baseline across all four capacities (Tables 1–2). We will include these controls in the appendix.
(b) FSQ setup and step-by-step FSP ablation. Our FSQ baseline mixes quantization (with STE) and uniform grid-step noise with probability each (Parker et al., 2024; Appendix A.2), and does not use the normalization loss in Eq. (7). From FSQ, we change decoding, perturbation, normalization loss, and training procedure one at a time (, COCO / ImageNet):
| Configuration | CVU ↑ | LPIPS ↓ | PSNR ↑ | SSIM ↑ | rFID ↓ | rKID (×10³) ↓ |
|---|---|---|---|---|---|---|
| FSQ baseline | 0.797 / 0.776 | 0.165 / 0.175 | 24.45 / 24.42 | 0.727 / 0.707 | 3.80 / 0.67 | 2.25 / 2.65 |
| + centroid decoding (Eq. (10)) | 0.804 / 0.781 | 0.148 / 0.160 | 24.79 / 24.95 | 0.738 / 0.725 | 3.48 / 0.58 | 0.88 / 0.94 |
| + FSP perturbation (Eqs. (8)–(9)) | 0.812 / 0.801 | 0.151 / 0.159 | 24.83 / 24.93 | 0.748 / 0.731 | 3.47 / 0.61 | 1.06 / 1.39 |
| + normalization loss (Eq. (7)), i.e., FSP | 0.834 / 0.816 | 0.151 / 0.158 | 24.89 / 25.00 | 0.747 / 0.732 | 3.50 / 0.54 | 0.83 / 1.21 |
| − STE (pure perturbation), i.e., Pure FSP | 0.827 / 0.811 | 0.152 / 0.161 | 24.61 / 24.75 | 0.745 / 0.729 | 3.53 / 0.61 | 1.00 / 1.32 |
- Decoding rule. Replacing FSQ's endpoint levels with interval centroids (Eq. (10)) captures / dB of the total / dB PSNR gain and cuts rKID from / to / . This confirms your point in Strengths: centroid placement is the main source of reconstruction gain and can be applied directly to existing FSQ.
- Perturbation, normalization loss, and training procedure. Adding FSP's training-time perturbation (Eqs. (8)–(9)) and the normalization loss in Eq. (7) mainly improves codebook utilization: CVU rises from / to / , whereas PSNR changes by at most dB per step. Removing STE ("Pure FSP") retains strong utilization and outperforms the FSQ baseline on all metrics.
We will detail the FSQ setup in Section 4.1, include this ablation, and update Section 4.2 to list all differences between FSP and FSQ and credit centroid decoding as the main driver of reconstruction quality.
Minor Comments
Minor 1 (Prop. B.1). Agreed. With , fixing and taking gives the exact limit as . Connecting to relies on the local constant-density approximation , whose approximation error need not vanish as grows at fixed . We will separate the exact quantile limit from the local density approximation in Prop. B.1 and Section 3.2.
Minor 2 (OOD attribution in Section 4.2). Thank you for pointing this out. Since FSP uses the same overall training objective without showing the same OOD image-PSNR pattern, we will revise Section 4.2 to state this explanation as a hypothesis specific to MH-based VP-VAE.
Q1. MH Acceptance Rates and Rejected Proposals in Figure 1a
Acceptance rates. Across 50 COCO training epochs, mean MH acceptance rates stay in a stable – range and are non-monotone in :
| Target codebook size | Overall mean | First epoch | Last 10% of training |
|---|---|---|---|
| 256 | 0.624 | 0.536 | 0.627 |
| 1024 | 0.571 | 0.476 | 0.576 |
| 4096 | 0.529 | 0.404 | 0.537 |
| 16384 | 0.583 | 0.447 | 0.592 |
Rejected proposals in Figure 1a. Figure 1a plots only accepted moves (). Including rejected proposals adds a spike at zero, shown below on 57,344 validation tokens at (45.9% rejected, matching the 53.7% late-training acceptance rate above):
Accepted moves have a mean norm of , closely matching the mean inference quantization error of . Including rejections brings the overall mean displacement to , roughly half the quantization error. Scale alignment thus holds conditioned on acceptance, so the decoder trains on clean latents for roughly half the tokens and scale-matched perturbations for the rest. We will clarify this in Figure 1a's caption and add both the acceptance table and the full distribution plot to the appendix.
Q2. Per-Dimension Histograms and FSP's CVU at
Checking the uniform-prior premise. Below are 100-bin per-dimension histograms of continuous latents across all 5,000 COCO validation images against the dashed reference line (density ), shown with a logarithmic density axis. On this log scale, FSP stays near across the interior with dips to about just inside and , whereas FSQ reaches – the reference density at the endpoints and remains below (around –) across a relatively flat interior. Overall, FSP matches more closely in per-dimension continuous KL (table below), with deviations localized near the boundaries:
Per-dimension continuous latent marginals of FSP and FSQ (G=4375 and G=15360)
Why FSP has lower CVU at . While per-dimension continuous KL measures proximity to , CVU reflects the discrete probability mass inside each quantization cell. Specifically, decomposes exactly into per-dimension marginal imbalance (zero iff all levels in dimension have equal frequency) plus cross-dimensional total correlation (measuring statistical dependence across dimensions). On a random subset of 1,000 validation images at , total correlation is nearly identical for FSP () and FSQ (), and marginal imbalance accounts for 97.1% of the CVU difference:
| Method | KL to ↓ | Marginal imbalance ↓ | |
|---|---|---|---|
| 4375 | FSQ | 0.464 | 0.152 |
| 4375 | FSP | 0.070 | 0.124 |
| 15360 | FSQ | 0.317 | 0.147 |
| 15360 | FSP | 0.071 | 0.212 |
The discrete level frequencies below show how cell geometry interacts with these continuous marginals. In the three 8-level dimensions at , FSP's outermost equal-width cells ( and ) become narrow enough to fall largely inside its boundary dips (– uniform frequency), increasing marginal imbalance from () to . In FSQ, the two endpoint cells have half the width of interior cells (Remark 3), which offsets its higher endpoint density to yield boundary cell frequencies of – and lower discrete marginal imbalance () at this resolution:
Discrete level frequencies of FSP and FSQ (G=4375 and G=15360)
Corollary B.4 assumes an exact input in each dimension; in practice, closer continuous fit to across the interior does not automatically imply higher discrete CVU at fine grid resolutions when narrow outer bins intersect boundary dips. We will add both plots and the CVU decomposition to the appendix, and note this boundary discretization effect alongside Corollary B.4.
Thank you for the detailed feedback and concrete experimental suggestions. We ran all requested controls and diagnostics, organized below as W1 (Part 1), W2 and minor comments (Part 2), and Q1–Q2 (Part 3). As in Table 1, "VP-VAE" refers to our MH-based instantiation.
W1. Positioning Relative to Noise Substitution
(a) Relation to noise substitution. We agree and will revise our positioning relative to the noise-substitution literature, citing and discussing Ballé et al. (2017), Agustsson & Theis (2020), and NSVQ (Vali & Bäckström, 2022) in Sections 1–2. The key distinction is the direction of design. Prior work either relaxes a predefined quantizer with noise—an integer grid (Ballé et al., 2017) or an active codebook (NSVQ)—or exactly realizes the training-time noise-channel distribution at inference through shared-dither universal quantization (Agustsson & Theis, 2020). VP-VAE reverses this order by specifying no quantizer during training, setting perturbations solely from the target capacity G and the empirical latent distribution, and fitting the discretizer only after convergence. We will bring this distinction forward from Section 3.1 to Section 1.
Training without a reference quantizer introduces two design requirements beyond standard noise substitution: a capacity-to-scale rule linking G and latent samples to local noise scale (e.g., the M-th nearest-neighbor radius in Eq. (1)), and distribution consistency keeping perturbed latents in high-density regions (e.g., the MH step). Viewing discretization through capacity-calibrated perturbation rather than smoothing a given quantizer provides a unified perspective across latent priors: beyond MH-based VP-VAE (no global prior) and FSP (bounded uniform prior), applying scale alignment under N(0,I) yields GVP, which analytically recovers a constant KL rate and competitive discretization without training-time codebooks (see Reviewer VtTf, W2).
(b) Eq. (8) at η=1. Yes, at η=1, Eq. (8) coincides with the bin-width uniform noise of Ballé et al. Under an approximately uniform prior on [0,1]d, scale alignment naturally simplifies to the bin width, recovering the classical uniform relaxation. Carrying this uniform prior through the rest of FSP further introduces pre-activation variance matching (Eq. (7)), boundary rejection in Eq. (9) for distribution consistency, and Lloyd–Max interval centroids in Eq. (10) in place of FSQ's endpoint grid. As our ablation in W2(b) confirms, centroid decoding drives most of FSP's gain over FSQ.
(c) The 1/2-STE schedule in Appendix A.2. We agree that the manuscript should have clearly separated the definition of FSP from the hybrid schedule used in our main experiments rather than leaving that detail to Appendix A.2. By definition, FSP (Section 3.3, Eqs. (7)–(9)) uses pure perturbation with neither training-time quantization nor STE. Because our FSQ baseline adopts the stronger symmetric hybrid variant of Parker et al. (2024; Appendix A.2), which builds on Brendel et al.'s (2024) uniform-noise relaxation and mixes 1/2 noise injection with 1/2 STE, we trained FSP under the same schedule for a controlled comparison. Trained with perturbation alone ("Pure FSP" in W2(b), 0% STE), FSP still outperforms this FSQ baseline on every metric across both datasets. We will make this distinction explicit in Sections 1 and 4.1 alongside the Pure FSP results.