- Rating
- 2
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates dimensional collapse in VQ-VAEs, where the learned representations occupy only a small fraction of the available latent dimensions, even when codebook utilization is high. The authors propose RD-AE, a simplified theoretical model that replaces vector quantization with a rate-distortion-optimal stochastic channel. Based on this model, they argue that early quantization suppresses low-variance directions and prevents them from being learned. To address this issue, the authors propose AE Warm-up, which first trains the encoder and decoder without quantization before switching to standard VQ-VAE training. Experiments on VQGAN and WavTokenizer show that this simple strategy increases the effective codebook dimension and improves several reconstruction metrics. The authors also derive theoretical bounds connecting warm-up duration with the final effective dimension and reconstruction loss, and provide empirical results supporting the predicted trends.
Critical strengths
The paper studies an interesting and meaningful problem in VQ-VAE training. As the authors point out, high codebook utilization does not necessarily mean that the learned representations make effective use of the available latent dimensions. I find the attempt to explain this phenomenon through training dynamics and rate-distortion theory interesting. Although the theoretical model is simplified, it provides an intuitive explanation for why introducing quantization too early may prevent some latent directions from being learned. Another strength is the simplicity of the proposed solution. AE Warm-up does not require architectural modifications or additional loss terms, and the experiments show improvements on both image and audio models. These results suggest that the proposed training strategy can be useful for conventional VQ-VAE architectures.
Critical weaknesses
My first concern is whether the theoretical analysis sufficiently explains dimensional collapse in real VQ-VAEs, and whether increasing the effective dimension is necessarily beneficial. RD-AE relies on several strong assumptions, including linear encoder-decoder networks, Gaussian inputs, and a rate-distortion-optimal quantization channel. However, real VQ-VAEs involve finite codebooks, imperfect codebook updates, STE approximation errors, and other optimization issues that may also contribute to dimensional collapse. The paper shows that the simplified model exhibits behavior similar to real VQ-VAEs, but this does not establish that the proposed mechanism is the main cause of collapse in practical models. More importantly, I am not convinced that a higher effective codebook dimension should always be considered desirable. For example, Control-GIC (Li et al., 2025) deliberately uses a 4-dimensional codebook while achieving competitive image compression performance. Although this is not directly equivalent to dimensional collapse in a higher-dimensional codebook, it suggests that low-dimensional representations can still be effective. Even in this paper's Table 2, AE Warm-up increases the effective dimension from 4 to 11 at , but PESQ, UTMOS, and STOI all become worse. This suggests that increasing the effective dimension does not necessarily improve reconstruction quality. The paper would be more convincing if it could clarify under what conditions dimensional collapse is actually harmful, rather than treating low effective dimension as a general problem.
My second concern is that the experimental evaluation is not sufficiently strong to support the broader practical claims. The image experiments mainly rely on the original VQGAN architecture, while considerably stronger image tokenizers have been developed in recent years. Even after AE Warm-up, the reported ImageNet-1K rFID is 5.35, compared with the publicly reported 4.98 for the original VQGAN (Esser et al., 2021). These numbers may not be directly comparable due to differences in training or evaluation, but the paper does not clarify this gap. It is therefore unclear whether the proposed method remains effective on stronger and more modern backbones. There are also concerns about experimental fairness. For example, Appendix B.2 describes different codebook initialization and dead-code respawn settings in the WavTokenizer training protocols, making it difficult to isolate the contribution of AE Warm-up itself. Finally, I find the connection to actual rate-distortion performance insufficiently explored. Although rate-distortion theory is central to the theoretical analysis, the practical experiments mainly evaluate reconstruction quality at fixed codebook sizes, without reporting actual bitrate or token entropy. Increasing codebook utilization and effective dimension may also change the entropy of the discrete representations, so better reconstruction quality at the same codebook size does not necessarily imply better compression efficiency at the same bitrate. Recent works such as HyperVQ (Yi et al., 2026) and RDVQ (Jiang et al., 2026) have explicitly explored entropy modeling and rate-distortion optimization for VQ-based compression. I am not suggesting that the authors must directly compare against these methods, since their research goals are somewhat different. However, with growing interest in entropy-aware and rate-distortion-optimized VQ methods, I believe it is important to discuss how AE Warm-up relates to these developments and whether improvements in effective dimensionality translate into better compression efficiency. This would help clarify the practical significance of the proposed approach.
References
[1] Li, A., et al. (2025). Once-for-All: Controllable Generative Image Compression with Dynamic Granularity Adaptation. International Conference on Learning Representations (ICLR).
[2] Esser, P., Rombach, R., & Ommer, B. (2021). Taming Transformers for High-Resolution Image Synthesis. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[3] Yi, N., et al. (2026). HyperVQ: Enabling Hyperprior Entropy Modeling for VQ-Based Generative Image Compression. Advances in Neural Information Processing Systems (NeurIPS).
[4] Jiang, S., et al. (2026). Differentiable Vector Quantization for Rate-Distortion Optimization of Generative Image Compression. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
Questions for authors
-
Theoretical explanation of dimensional collapse: The RD-AE analysis is based on a rate-distortion-optimal stochastic channel and simplified linear-Gaussian assumptions. However, actual VQ-VAE training involves finite codebooks, codebook update dynamics, and STE approximation errors. Is there any direct evidence showing that the dimensional collapse observed in real VQ-VAEs is mainly caused by the mechanism identified in RD-AE? For example, have the authors examined whether low-variance directions in real models actually stop receiving useful gradients after quantization, as suggested by the theoretical analysis?
-
Is a higher effective dimension always beneficial? The paper generally treats increasing the effective codebook dimension as an improvement, but some existing methods, such as Control-GIC, deliberately use low-dimensional codebooks while achieving strong reconstruction performance. Could the authors clarify under what conditions a higher effective dimension is actually necessary? Also, how do the authors explain the results at in Table 2, where AE Warm-up increases the effective dimension but makes several perceptual and speech quality metrics worse?
-
Experimental strength and fairness: Does AE Warm-up still provide meaningful improvements on stronger and more modern image tokenizer backbones? The best reported VQGAN rFID is 5.35, while the publicly released original VQGAN checkpoint reports 4.98. Could the authors clarify the differences in training and evaluation settings and explain this gap? In addition, could the authors clarify whether the codebook initialization and respawn settings in the WavTokenizer experiments are fully controlled, and how much of the improvement can be attributed to AE Warm-up alone?
-
Actual rate-distortion performance: Since the theoretical analysis relies heavily on rate-distortion theory, have the authors examined how AE Warm-up affects the entropy or actual bitrate of the learned discrete tokens? Models with the same codebook size can have different token distributions and coding costs. Could the authors provide token entropy statistics from their existing experiments, or clarify whether the improvements would still hold under an explicit rate constraint? Relatedly, how do the authors view the connection between AE Warm-up and recent entropy-aware VQ methods such as HyperVQ and RDVQ?
3 comments
Q3. Experimental strength and fairness
VQGAN baseline In Section 4.1 “VQGAN on ImageNet”, in the “Setup” paragraph, we state that we are comparing training on ImageNet-1K at 500k training steps (~6 epochs), with adversarial loss enabled at 250k steps. In contrast, the released vqgan_imagenet_f16_16384 checkpoint is tagged as step 1,032,743 and epoch 12, and their longer training time primarily explains the difference. We believe that 500k steps is a long enough window to clearly observe that the favorable dynamics of AE Warmup training cause significant gains in loss and rFID over the Vanilla VQGAN and VQGAN w/ Respawn recipe.
Longer training does not remove collapse: the released VQGAN checkpoints use about 4 of their 256 dimensions (Zhang et al., 2025).
Modern tokenizers. For images we use the VQGAN backbone, with several quantizers on top. Warm-up also improves DCVQ (Table 7) and VQGAN-LC at codebook dimension 16 (Table 6). In addition our VQGAN + warmup beats FSQ (Table 5), where warm up does not apply, since the codebook dimension is fixed by design. For audio, WavTokenizer is a recent tokenizer (ICLR 2025). Warm-up acts before quantization and changes only the schedule, so it is easy to apply to other backbones. Because it is a modification supported by our theoretical study with predictions that transfer (ablations on warmup length, water-filling dimension), we believe that combined it is enough to showcase its practical merit.
WavTokenizer settings. Our cold-start baseline uses the WavTokenizer recipe as released, including its k-means codebook initialization and dead-code respawn. We additionally train WavTokenizer at codebook sizes larger than the 4k released by the WavTokenizer authors, to demonstrate that AE Warm-up allows scaling past the 4k codebook size. For the warm-up runs, we scale the dead-code respawn threshold with , and use random initialization for our largest codebook size for efficiency.
The obtained cold-start baseline behaves as published: the WavTokenizer authors report the same flat curve when they enlarge the codebook with their recipe (PESQ 2.37, 2.39 and 2.36 at = 4096, 8192 and 16384; Ji et al., 2025, Table 5). Our recipe resolves their scaling issue.
For the cleanest estimate of warm-up alone we refer to VQGAN experiments. There, warm-up improves rFID by 17–35%.
Q4. Codebook entropy
Thank you for this question. Our focus is the geometry of the codebook, which token entropy does not measure. As noted in the introduction, prior work mostly raises codebook utilization, which bounds the entropy. However, any tokenizer would benefit from optimal codebook entropy, and in our experiments the entropy is optimal, see table below. Warm-up is complementary: it changes what each bit encodes.
A codebook of size costs at most bits per token. Our warm-up runs are close to this ceiling: 13.85 of 14 bits at on ImageNet-1K.
Token entropy and bitrate on ImageNet-1K (, code-usage entropy over the full 50k validation set):
| Model | Step | Entropy (bits/token) | Perplexity () | Codes used | Bitrate (bpp) | rFID |
|---|---|---|---|---|---|---|
| Vanilla | 500k | 9.78 | 880 | 987 | 0.0382 | 11.37 |
| Vanilla (as in Table 3) | 850k | 10.23 | 1,199 | 1,316 | 0.0400 | 7.91 |
| w/ Respawn | 500k | 13.95 | 15,784 | 16,384 | 0.0545 | 7.07 |
| = 1k | 500k | 13.94 | 15,714 | 16,384 | 0.0545 | 7.22 |
| = 6k | 500k | 13.86 | 14,880 | 16,384 | 0.0541 | 5.97 |
| = 10k | 500k | 13.85 | 14,770 | 16,384 | 0.0541 | 5.88 |
| = 20k | 500k | 13.83 | 14,604 | 16,384 | 0.0540 | 5.35 |
| = 30k | 500k | 13.84 | 14,696 | 16,384 | 0.0541 | 5.37 |
HyperVQ and RDVQ model and optimize the bits spent on the indices, while warm-up lowers the distortion at a given rate. The two are complementary and potentially could be combined.
Thank you for providing the responses, the training-step clarification in Q3, and the token entropy statistics in Q4. Given that these responses were posted only a few hours before the end of the discussion period, there is unfortunately no time left for multi-round clarification. After carefully reviewing your replies across all threads, my main concerns remain largely unresolved, and several points in the rebuttal raise further questions:
-
Lack of direct mechanistic evidence and contradiction on codebook dynamics (Q1): Rather than providing direct diagnostics in real VQ-VAEs (e.g., verifying whether low-variance directions actually stop receiving effective gradients after quantization), the response reiterates the RD-AE assumptions and macro-level trends already in the paper. More importantly, in Q1 you state that imperfect codebook updates are not needed to explain collapse because the RD-optimal channel models a strong codebook. However, in your response to Reviewer yPFG (W2), you attribute the failure of to recover collapsed dimensions at and to finite codebook update speed—noting that only ~22% and ~6% of codewords receive updates per step, causing codeword staleness. This indicates that the assumption of an always-optimal quantizer breaks down precisely in the large- regime () where you argue AE Warm-up is most needed (Q2).
-
Unexplained degradation at smaller and narrowed scope (Q2): While I appreciate clarifying that higher effective dimension is mainly beneficial for large codebooks, the response still does not explain why increasing from 4 to 11 at degrades PESQ, UTMOS, and STOI in Table 2 (or why L1 worsens at longer warm-up durations for in Table 1). The paper’s framing should be revised to reflect this trade-off rather than treating low effective dimension as universally harmful.
-
Uncontrolled confounds in WavTokenizer and baseline inconsistencies (Q3):
-
WavTokenizer fairness: Scaling the dead-code respawn threshold inversely with (
1.0 / 0.5 / 0.25 / 0.06) for AE Warm-up while keeping it fixed at1.0for cold-start remains an uncontrolled confound. Furthermore, your claim that the cold-start baseline "behaves as published" (citing flat PESQ of2.37, 2.39, 2.36across in Ji et al., 2025) directly contradicts your own Table 2, where cold-start utilization collapses to16.9%and8.4%and test-clean PESQ drops from2.336to2.179and2.209. Referring us to VQGAN for "the cleanest estimate of warm-up alone" effectively concedes that the WavTokenizer experiments do not isolate the contribution of AE Warm-up. -
Image baselines: While the step-count difference (
500kvs.~1Msteps) explains the rFID gap to the official VQGAN checkpoint (5.35vs.4.98), the new table in Q4 reveals that the Vanilla VQGAN baseline reported in Table 3 (7.91rFID) was actually trained for850ksteps rather than the500k-step budget indicated in Figure 5 (11.37rFID at500ksteps). Evaluation on stronger, modern image tokenizer backbones also remains absent.
- Rate-distortion interpretation (Q4): The new table confirms that AE Warm-up improves rFID over the Respawn baseline at comparable token entropy (
13.83vs.13.95bits/token). However, achieving near-uniform marginal entropy (~13.85of14bits) reflects full fixed-rate utilization rather than R-D optimality under learned entropy models (e.g., HyperVQ, RDVQ), and incurs a ~35–42% higher bitrate than Vanilla VQGAN (0.0540vs.0.0382–0.0400bpp). A clearer discussion of actual rate-distortion trade-offs remains necessary.
Because the rebuttal not only leaves my core concerns unresolved but also reveals cross-thread theoretical contradictions and confirms uncontrolled confounds in the audio baselines, I am considering lowering my score, if the authors are unable to provide new evidence,
Thank you for the detailed review. We answer your four questions, which should cover the weaknesses you raise.
Q1. Does the RD-AE mechanism explain collapse in real VQ-VAEs?
We treat RD-AE as an idealized model under which the collapse still occurs. In principle, any modeling choice is limited, and we believe that the strength of our setup is to be able to model the observed practical effect rigorously and then translate that into useful predictions (warm up method and water-filling for dimension prediction). The effects you list are not needed to reproduce collapse. Note that we do not dismiss components of the complete system:
As a result of our model, we obtain the intuitive explanation that collapse occurs because early quantization combined with sequential learning kills off weak representations (through water-filling picture). To support usefullness of our theory, the predictions we obtain carry over to both practical models. Without warm-up, the codebook dimension stays at 2–5 for every K (Tables 1–3). Warm-up raises it monotonically as Twu grows (Table 1, Fig. 5). Water-filling on the AE latent spectrum at the switch is tracking the final codebook dimension at every warm-up checkpoint (App. C.1, Fig. 11).
Moreover, the theory leads us o design the fix (warm-up) in the first place, that works in practice. With the quantizer, K and β unchanged, warm-up raises the VQGAN codebook dimension from 3–5 to 17–19 and lowers rFID by 17–35% (Table 1).
Finally, it builds a general perspective for the field. There are other practical works that play around with rate constraint in the system. For example, the Mimi codec quantizes only half of its training sequences and passes unquantized embeddings to the decoder otherwise (Défossez et al., 2024). Our analysis suggests why this helps: a weaker rate constraint early lets more modes be learned before quantization freezes them.
Q2. When is a higher effective dimension beneficial?
When the codebook is large. Low dimension hurts with growing codebook size, because adding codes then stops lowering the loss. Without warm-up, VQGAN reconstruction improves only modestly as K grows 64-fold (App. C, Fig. 9). With warm-up the gain grows with K: rFID improves by 17%, 27% and 35% at K=210, 214 and 216 (Table 1).
Control-GIC is a good example of the setting where choosing higher dimension is not required (though it might be beneficial). It uses K=1024 codes of dimension 4 (their Sec. 4.1), which is on the lower end of codebook sizes we study. We will cite it as an example of a setting where a low-dimensional codebook is appropriate.
Table 2 shows the same trend for audio, including at K=4096. This is again a smaller codebook size case. At K=4096 both arms use the full codebook and are close. Warm-up has the lower Mel loss and is 0.5–2.6% lower on PESQ, UTMOS and STOI. But as K grows, cold start stops improving and warm-up keeps improving, up to PESQ 2.63 at K=65536, 12% above the best cold-start run.
So to answer this question, higher dimension is useful at larger codebook size. This is supported by our experiments on VQGAN and WavTokenizer. Most existing research does not address the geometry of latents directly, primarily focusing on growing the codebook size at full utilization, which we show to be suboptimal without resolving the dimensional collapse.
LlamaGen shows that this training "failure" can influence practical decisions. Its ablation settles on codebook dimension 8 and size 16384 because the alternatives fail (their Table 2):
Thus they pick dimension 8 at K=16384 as the best their training reaches. This is the setting our work targets: with warm-up, both K and the codebook dimension can be raised and keep paying off (Tables 1, 2, 6).