- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 3(+1)
Read review
Paper summary
The paper studies smoothed vector quantization, where a code-selection vector p lies in the probability simplex. It argues that a good smoothing should (1) keep p close to one-hot vectors and (2) use all codebook entries, and that the common perplexity-of-the-mean regularizer only addresses (2). The proposed regularizer minimizes, for every simplex vertex e_m, the distance to its K nearest smoothed code-selectors in the batch. Experiments on ImageNet discrete autoencoding show 100% codebook usage and low per-sample perplexity (Table 2) without Gumbel sampling.
Critical strengths
The geometric motivation is clear and well illustrated (Fig. 1): perplexity of the mean cannot distinguish vertex-concentrated, centered, and uniform distributions, while the KNN-to-vertex loss can. The method is simple, drop-in, and removes the need for Gumbel noise and temperature annealing. The authors are transparent about reproduction failures and about the batch-size limitation.
Critical weaknesses
The method requires at least K·M latent tokens per (multi-)GPU pass; this inherently constrains M unless K is very small or product/factorized schemes are used. Missing comparisons to several recent and highly relevant anti-collapse/tokenizer methods (e.g., VQGAN-LC, StableVQ, TransVQ, GRIT-VQ, FSQ/LFQ/RQ-VAE) that target utilization and stability at scale. Product-quantized STE/RE/SimVQ runs are newly implemented here and perform poorly; hard-Gumbel+perplexity is strong, but the absence of other state-of-the-art VQGAN tokenizers makes it hard to pinpoint the absolute standing.
Questions for authors
Can you report more recent baseline results (e.g., VQGAN-LC, StableVQ, GRIT-VQ, TransVQ) on your autoencoding and VQGAN setups? If compute is prohibitive, can you at least provide a careful discussion of expected behavior and limitations relative to these methods? How does the proposed loss compare with a softmax baseline using PPL plus per-sample entropy minimization (the MAGVIT-v2-style entropy penalty)?
Clarifying the limitation & its mitigation; Additional baselines
Thank you for the follow-up and for the transparency about the review process. We have experimented two additional models, and we hope
Codebook expansion issue
As acknowledged in Section 4.2, the requirement of at least KM latent pixels per batch is a major limitation, as it constrains the size M of a single codebook. We have shown that this limitation can be mitigated in two ways.
First, a single neighbor per codebook entry per GPU (K/#GPUs$= 1$) was sufficient when cross-entropy was used as the deviation metric (KNN-CE; Tables 1-2), which reduces the requirement to M latent pixels per GPU.
Second, product quantization combines G codebooks into an effective vocabulary of MG, while the requirement depends only on M. This allowed us to build a competitive model with a large effective vocabulary of 10248 (Table 3).
Experimental baselines