acceptodds
Under review as a conference paper at ICLR 2027

Pushing Toward the Simplex Vertices: A Simple Remedy for Code Collapse in Smoothed Vector Quantization

Abstract

Vector quantization, which discretizes a continuous vector space into a finite set of representative vectors (a *codebook*), has been widely adopted in modern machine learning. Despite its effectiveness, vector quantization poses a fundamental challenge: the non-differentiable quantization step blocks gradient backpropagation. *Smoothed* vector quantization addresses this issue by relaxing the hard assignment of a codebook vector into a weighted combination of codebook entries, represented as the matrix product of a simplex vector and the codebook. Effective smoothing requires two properties: 1. smoothed code-selection vectors should remain close to a onehot vector, ensuring tight approximation, and 2. all codebook entries should be utilized, preventing *code collapse*. Existing methods typically address these desiderata separately. By contrast, the present study introduces a simple and intuitive regularization that promotes both simultaneously by minimizing the distance between each simplex vertex and its -nearest smoothed code-selectors. Representative benchmarks on image encoding demonstrate that the proposed method achieves more effective codebook utilization and improves performance over prior approaches.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vector quantization, code collapse, discrete autoencoder, smoothing, VQGAN

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_bYTWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

The paper studies smoothed vector quantization, where a code-selection vector p lies in the probability simplex. It argues that a good smoothing should (1) keep p close to one-hot vectors and (2) use all codebook entries, and that the common perplexity-of-the-mean regularizer only addresses (2). The proposed regularizer minimizes, for every simplex vertex e_m, the distance to its K nearest smoothed code-selectors in the batch. Experiments on ImageNet discrete autoencoding show 100% codebook usage and low per-sample perplexity (Table 2) without Gumbel sampling.

Critical strengths

The geometric motivation is clear and well illustrated (Fig. 1): perplexity of the mean cannot distinguish vertex-concentrated, centered, and uniform distributions, while the KNN-to-vertex loss can. The method is simple, drop-in, and removes the need for Gumbel noise and temperature annealing. The authors are transparent about reproduction failures and about the batch-size limitation.

Critical weaknesses

The method requires at least K·M latent tokens per (multi-)GPU pass; this inherently constrains M unless K is very small or product/factorized schemes are used. Missing comparisons to several recent and highly relevant anti-collapse/tokenizer methods (e.g., VQGAN-LC, StableVQ, TransVQ, GRIT-VQ, FSQ/LFQ/RQ-VAE) that target utilization and stability at scale. Product-quantized STE/RE/SimVQ runs are newly implemented here and perform poorly; hard-Gumbel+perplexity is strong, but the absence of other state-of-the-art VQGAN tokenizers makes it hard to pinpoint the absolute standing.

Questions for authors

Can you report more recent baseline results (e.g., VQGAN-LC, StableVQ, GRIT-VQ, TransVQ) on your autoencoding and VQGAN setups? If compute is prohibitive, can you at least provide a careful discussion of expected behavior and limitations relative to these methods? How does the proposed loss compare with a softmax baseline using PPL plus per-sample entropy minimization (the MAGVIT-v2-style entropy penalty)?

1 comment
AuthorsOpenReview ↗

Clarifying the limitation & its mitigation; Additional baselines

Thank you for the follow-up and for the transparency about the review process. We have experimented two additional models, and we hope

Codebook expansion issue

As acknowledged in Section 4.2, the requirement of at least KMKM latent pixels per batch is a major limitation, as it constrains the size MM of a single codebook. We have shown that this limitation can be mitigated in two ways.

First, a single neighbor per codebook entry per GPU (K/K/#GPUs$= 1$) was sufficient when cross-entropy was used as the deviation metric (KNN-CE; Tables 1-2), which reduces the requirement to MM latent pixels per GPU.

Second, product quantization combines GG codebooks into an effective vocabulary of MGM^G, while the requirement depends only on MM. This allowed us to build a competitive model with a large effective vocabulary of 102481024^8 (Table 3).

Experimental baselines

  • MAGVIT-v2-style entropy penalty: Added as DualEnt (Eq. 13). DualEnt successfully prevented code collapse and also achieved tight smoothing. Regarding reconstruction quality, however, DualEnt was never superior to KNN-CE (K/4 = 1) on any reconstruction metric in any setting of Tables 1 and 3.
  • FSQ: Added with effective codebook sizes matched to the configurations for models with learnable codebooks (Table 3). KNN-CE had better rFID and rIS, FSQ had better SSIM, PSNR, and LPIPS. We have also added a new subsection (2.4) under "Related Studies", where the scalar quantization approach is reviewed.
Reviewer_8VZUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a smoothed vector quantization regularizer that draws smoothed code-selection vectors toward every simplex vertex, encouraging sharp assignments and broad codebook utilization. Experiemental results on image encoding show that the proposed method achieves effective codebook utilization.

Critical strengths
  1. The proposed method demonstrated strong results in the high-dimensional setting. And the proposed regularizer has a clear geometric motivation. For (646464) features, Softmax KNN-CE achieves FID 2.86 with 100% codebook utilization, compared with 14.77 for Hard-Gumbel PPL and 12.54 for SimVQ (Table 1). Table 2 additionally supports the improvement in assignment sharpness.

  2. They conducted many experiments about the proposed methods under different configurations, such as a matched product-quantization configuration. With the same (G=8) configuration, KNN-CE improves VQGAN FID from 2.17 to 1.57 over Hard-Gumbel PPL (Table 3). Besides, the Appendix C also reports runtime and memory usage, showing computational costs comparable to perplexity-regularized softmax.

Critical weaknesses
  1. Although the authors provided extensive experiments, there are limited experiments against well-performing hard-quantization baselines. In Table 3, the locally evaluated STE, rotation-trick, and SimVQ baselines suffer severe collapse, while single-codebook STE and rotation results are taken from their original publications. Although these difficulties are disclosed, the comparison does not establish superiority over well-performing hard-quantization baselines under matched training conditions.

  2. It seems that the cross-configuration comparisons change the representation budget. With (M=1024), the (G=1) and (G=8) configurations require 10 and 80 fixed-length index bits per spatial position, respectively. The proposed method also changes its total latent dimension from 32 to 256. Although §4.2 acknowledges the larger vocabulary, these comparisons cannot isolate the contribution of grouping from increased representation capacity.

  3. The contribution of KNN construction is hard to determine without a direct joint-entropy control. The perplexity baseline in Eq. (11) encourages diversity of the mean assignment without directly encouraging individual sharpness. A matched softmax baseline combining per-sample entropy minimization and batch-level entropy maximization, as in the objective of MAGVIT-v2 (Yu et al., 2024, Eq. (5)), would better isolate the benefit of the KNN construction.

Questions for authors
  1. What causes the severe collapse of the hard-quantization baselines in Table 3, and does the advantage persist against successfully converged baselines under matched conditions?

  2. How do the product-quantization results compare when both the total latent dimension and the index-bit budget are controlled?

  3. Does KNN-CE outperform joint sample-entropy and batch-entropy regularization with the same logits, learned temperature, and training budget?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8VZU

Thank you for the careful and constructive review.

Q1: collapse of the hard-quantization baselines

The ultimate cause is unknown. Reproduction failures with the official VQGAN repository have been widely reported (e.g., the issue cited in Footnote 6), and our runs are likely another instance. Code collapse in the original VQGAN has also been repeatedly reported in follow-up studies. For example, the SimVQ paper (Zhu et al., 2025) reports that VQGAN used only 1.4% of its codes (when using M=65536M = 65536 codes and 128×128128 \times 128 input images). Moreover, previously proposed workarounds, such as resetting unused codes to encoder outputs (e.g., Dhariwal et al., 2020), also indicate the training instability of hard quantization.

For SimVQ, we used the same peak learning rate and commitment-loss weight as in the official code, which was designed for 128×128128 \times 128 inputs (Footnote 7). A potentially critical difference is the batch size; the SimVQ paper used a batch size of 256, whereas ours was limited to 64 owing to the larger input images, the deeper network, and limited computational resources.

We therefore think that the main advantage of the proposed method over hard quantization lies less in benchmark scores than in training stability itself; it prevented code collapse in all tested settings.

Q2: representation budget

Increasing the effective vocabulary is equivalent to increasing the index bit depth (10 bits per position for a single codebook with M = 1024, and 80 bits for G = 8). This increase is the purpose of product quantization in this study, not a confound: the proposed regularization requires at least KMKM latent pixels per batch, so MM cannot simply be enlarged, and grouping is the practical way to reach a large vocabulary. We therefore do not claim that grouping is more effective than a larger single codebook, nor that the G=1G = 1 and G=8G = 8 results isolate the effect of grouping.

Regarding the channel dimensionality, increasing it to C=256C = 256 with G=1G = 1 resulted in loose smoothing (i.e., degraded reconstruction when the soft quantizer is replaced by hard quantization at inference), whereas C/G=32C/G = 32 was stable. Setting G=8G = 8 therefore keeps the total width at C=256C = 256, as in the baselines of previous studies (Esser et al., 2021), while each group stays at 32 dimensions.

Q3: joint sample- and batch-entropy control

Added as DualEnt (Eq. 13). DualEnt successfully prevented code collapse and also achieved tight smoothing. Regarding reconstruction quality, however, DualEnt was never superior to KNN-CE (K/4 = 1) on any reconstruction metric in any setting of Tables 1 and 3.

Reviewer_yWjcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a new loss function as a regularization method to make the soft clustering (softmax-based code sampling) to end up in a similar was as in one-hot code sampling. The proposed method is compared with some baselines STE, Rotation Trick, Gumbel-Softmax, and SimVQ under two different image encoding tasks over different latent dimension sizes.

Critical strengths
  1. Paying attention in the detailed idea of how much tight (one-hot like) is the learned codebook representation when using soft smoothing for training the codebook, which was not looked into before.
Critical weaknesses

MAJOR

  1. Lack of contribution by only defining a new loss function as regularization.

  2. Not a comprehensive literature review. The related studies section lacks many related papers in the literature that solved the non-differentiability and codebook collapse problems of VQ in other different ways which is skipped in this paper.

  3. Not a proper choice of evaluation metrics. The method is compared in image encoding (compression) application, and then compare the quality of reconstructions (as claimed in the paper) over different methods. However, they use IS and rMSE methods as metrics of comparison. IS is a metrics for image generation, but I guess authors mean reconstruction IS. Also, rMSE is not necessarily a common metric used in image reconstruction. Better to use SSIM, PSNR, and LPIPS. The authors also report FID which I guess they mean reconstruction FID (rFID), but they show it with FID. Apart from these, they mentioned SSIM in the caption of Table 1, but the SSIM is not reported.

  4. The proposed method is compared with STE, Rotation Trick which do not propose any solution like defining loss function to resolve the codebook collapse. As the paper claims to resolve the codebook collapse issue, it should be compared with these methods when they also use some similar losses to have a fair comparison.

  5. No qualitative results provided. There is not any qualitative comparison of the reconstructed images not in the main paper not in the appendix.

  6. Although the idea of the paper is quite simple and straightforward, but the paper introduce it in a really hard way to understand. It might be challenging for the readers who are not expert in this filed to understand the paper easily. So, I recommend rewriting the whole paper in a more smooth and much simpler way to understand the idea.

  7. Missing ablation on a wider range of K in K nearest neighbor and also ablation of a wider range of codebook sizes.

  8. It seems that the evaluations are provided over a single individual run. As they are prone to randomization over different experiments, it is recommended to provide the results computed over several different runs.

MINOR

  1. In fig.1, it is not clear on which experiment, and which data the plots of B and C are created?

  2. Remove the code-selectors term completely from the paper, as it is misleading and talk about the idea based on input samples and codebook vectors. As there are two similar sentences in the paper but it is misleading what you refer to when saying code-selectors. Sentence1: "At the same time, each vertex must have some samples in its neighborhood to avoid code collapse" Sentence 2: "Moreover, each vertex should have at least some smoothed code-selectors in its neighborhood; otherwise, the quantization suffers from code collapse."

  3. In table 1 you have some metrics under STE, and then SimVQ there, which is a completely different method in which you can use Euclidean or Cosine distance as a metric, but it is listed under EUC and COS distance metrics. Prepare a separate row for SimVQ.

  4. Define a new name like LKNNL_KNN for your proposed loss not the same name as other losses.

Questions for authors

Q1. What does Fig.2 try to say? Was completely vague for me to digest what authors try to convey.

Q2. Is is not mentioned anywhere in the main paper how perplexity and code usage are computed in detail?

2 comments
AuthorsOpenReview ↗

Response to Reviewer yWjc

Thank you for the thorough review. Responses to the major points:

Contribution

We have experimented an alternative regularization based on entropy ("DualEnt"), and demonstrated superiority of our proposed method.

Related work

Section 2.4 now covers scalar quantization (FSQ, LFQ, BSQ). FSQ was additionally evaluated experimentally (Section 4.2, Table 3).

Evaluation metrics

FID/IS were renamed rFID/rIS. SSIM, PSNR, and LPIPS (VGG backbone) were added to Tables 1 and 3.

Regularization for hard quantization

We agree that some variant of our proposed regularization may mitigate code collapse in the hard quantization family (STE, Rotation Trick, SimVQ) just as in the smoothing approach explroed here. We could not explore this possibility within the discussion period, so we have added a remark on it to the Discussion section as a direction for future work.

Ablations

The possible range of KK is dependent on the codebook size and the batch size per GPU. Consequently, KK cannot be increased unboundedly, and we therefore consider that the K/4∈{1,2,4,8}K/4 \in \{1, 2, 4, 8\} covers representative values, especially given that K/4=1K/4=1 is shown to be sufficient empirically.

The codebook size cannot be enlarged arbitrarily, which constitutes the major limitation of the proposed method as we acknowledge in as we acknowledge in Section 4.2. We therefore consider that the two codebook sizes (1024 and 8192, chosen following Fifty et al., 2025) are representative enough.

"code-selectors" terminology

Thank you for pointing out this ambiguity.

In the first quoted sentence (Fig. 1 caption), "samples" denotes points on the simplex; i.e., the smoothed code-selection vectors p\mathbf{p} computed from the dot-product of the input features z\mathbf{z} and code vectors q1,…,qM\mathbf{q}_1,\dots,\mathbf{q}_M, not the input samples z\mathbf{z} themselves.

We therefore decided to use the term "smoothed code-selector" consistently instead of "samples".

Minor points

  • Fig. 1: panels B and C are computed on the synthetic samples of panel A (toy distributions on the simplex, not experimental data); the caption has clarified it.
  • Table 1: SimVQ now has its own row, separated from the distance-metric rows.
  • Loss name: the proposed loss is now named L_KNN (Eq. 14), distinct from L_ppl and L_dual.

Q1 (Fig. 2).

Fig. 2 shows how many distinct codes each validation image uses (top) and in what fraction of images each code occurs (bottom). SimVQ uses every code somewhere in the validation set (100% usage in Table 1), but in the 16x16x32 and 64x64x64 settings its per-image utilization is spread toward low values and its code popularity varies widely across codes. KNN-CE concentrates near the maximum attainable per-image utilization (25% for 16x16, 50% for 64x64). The new baseline, DualEnt, behaves similarly to KNN-CE in this respect.

Q2 (formal definitions).

Appendix B.3 defines codebook usage (Eq. 17-18), per-image utilization (Eq. 19), and code popularity (Eq. 20). The individual perplexity is defined in Section 4.1 and in Table 2's caption.

Reviewer_yWjcOpenReview ↗

Final Evaluation

I thank authors for their response and the new edits/experiments they made. Some of my main concerns are still there. The idea sounds interesting and quite new that it was not studied before, but not a proper comprehensive and fair evaluation lead to rejection. I recommend authors to check the latest works on methods proposed in the literature (I listed some) and only compare against those which have used a technique (e.g., entropy loss, codebook reset, etc.) to solve the codebook collapse problem. I believe that a comprehensive experimentation on bitrate is crucial in proposing new techniques in vector quantization domain. Al in all, I will keep my score because of the following reasons:

  • Not comparing against the methods which are proposed to solve the codebook collapse directly (please check the citations)
  • Lack of contribution/novelty (I do not consider comparing to a new method "DualEnt" as a contribution of your own work)
  • Lack of ablation on different codebook sizes (at least 5 different sizes)
  • As nothing raised by the authors, I still think the reported results are not statistically significant as they are considering a single run.
  • Still no qualitative results are provided.

[1] Zhu L, Wei F, Lu Y, Chen D. Scaling the codebook size of vq-gan to 100,000 with a utilization rate of 99%. Advances in Neural Information Processing Systems, 2024.

[2] Chang Y, Qin J, Qiao L, Wang X, Zhu Z, Ma L, Wang X. Scalable training for vector-quantized networks with 100% codebook utilization. In International Conference on Learning Representations, 2026.

[3] Vali M H, Backstrom T, Solin A. DiVeQ: Differentiable Vector Quantization Using the Reparameterization Trick. In International Conference on Learning Representations, 2026.

[4] Zhang, J., Zhan, F., Theobalt, C. and Lu, S., 2023, June. Regularized vector quantization for tokenized image synthesis. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 18467-18476). IEEE.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Major Revisions

Thank you for the detailed and constructive reviews. Newly added or modified text is highlighted in blue. The main changes are summarized below; point-by-point responses follow in separate comments.

1. Dual-entropy control (8VZU, bYTW)

A softmax baseline combining per-sample entropy minimization and averaged-probability entropy maximization, in the style of the entropy penalty of MAGVIT-v2 (Yu et al., 2024), was added as "DualEnt" (Eq. 13). It uses the same logits computation, learned temperature, and training schedule as KNN-CE, and was evaluated in discrete autoencoding (Tables 1-2, Fig. 2) and VQGAN (Table 3).

DualEnt successfully prevented code collapse and also achieved tight smoothing. Regarding reconstruction quality, however, DualEnt was never superior to KNN-CE (K/4 = 1) on any reconstruction metric in any setting of Tables 1 and 3.

2. Scalar quantization (bYTW)

Finite scalar quantization (FSQ; Mentzer et al., 2024) was added to the VQGAN experiment, with levels (8,5,5,5) and the same levels repeated eight times, so that the effective codebook sizes (10001000 and 100081000^8) approximately match the single-codebook (1024) and product-quantized (1024^8) configurations.

In product-quantized VQGAN, KNN-CE obtained the best rFID and rIS, whereas FSQ obtained better SSIM, PSNR, and LPIPS. Neither approach dominates, and the revised text clarified this.

Additionally, a new subsection (Section 2.4) discusses scalar quantization (FSQ and related binary variants) and its relation to learned codebooks.

3. Evaluation metrics (yWjc)

FID/IS were renamed rFID/rIS, and SSIM, PSNR, and LPIPS (VGG backbone) were added to Tables 1 and 3. Appendix B.3 defines codebook usage, per-image utilization, and code popularity; Table 2's caption and Footnote 5 define the individual perplexity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.