acceptodds
Under review as a conference paper at ICLR 2027

Asymptotically Optimal Quantizer-Guided Sinkhorn Alignment for Mitigating Codebook Collapse

Abstract

Codebook collapse remains a fundamental limitation of vector quantization (VQ), causing poor codebook utilization and increased quantization error. While recent methods improve utilization through distribution alignment, directly matching the codebook to the encoder distribution can deviate from the asymptotically optimal codeword distribution, leading to excessive codeword concentration in high-density regions and insufficient coverage of lower-density regions. We introduce AOptSA, a theory-driven quantization framework motivated by asymptotically optimal quantizer theory. Its key insight is to align the codebook with a density-reweighted target distribution, rather than the raw encoder distribution, thereby better balancing probability mass and spatial coverage. AOptSA realizes this target online via importance sampling from encoder outputs and minimizes Sinkhorn divergence between the sampled target and codebook distributions. The resulting alignment adapts throughout training without altering nearest-neighbor quantization or latent dimensionality. Across image benchmarks, AOptSA achieves full codebook utilization, consistently improves reconstruction quality over strong baselines, and maintains its gains as codebook size increases.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vector quantization, codebook collapse, distribution alignment, Sinkhorn divergence

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fU7hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper builds on distribution-alignment remedies for codebook collapse. It observes that, by asymptotic quantization theory, the empirical distribution of an optimal codebook converges to a density proportional, not to the encoder density h_z itself. AOptSA builds an online target set by importance-sampling encoder outputs, using a kNN density estimate and Efraimidis–Spirakis weighted reservoir sampling. It then aligns the codebook to this target with a debiased Sinkhorn divergence that updates only the codebook. Against VQGAN-EMA, FSQ, Wasserstein-VQ, and SimVQ, the method reaches 100% utilization and the best rFID / LPIPS / PSNR / SSIM (Tables 1–2), with ablations on dynamic alignment, the target distribution, codebook size, and dimensionality.

Critical strengths

The core observation is correct and well grounded in classical theory: matching the codebook to the raw encoder density is not what an asymptotically optimal quantizer does. The method keeps the standard nearest-neighbor assignment and full latent dimensionality, and the gradient only touches the codebook. It is easy to add to existing VQ models.

Critical weaknesses

In the main setting (d = 128), the theory predicts the reweighting should matter least, and the implemented weight largely reduces to a kNN-distance heuristic. MMD-VQ is described in Related Work as the method that removes Wasserstein-VQ's Gaussian assumption and aligns to the raw encoder distribution. It is the most direct competitor, but it does not appear in Exp.

Questions for authors

What happens if the importance-weight exponent is varied at full training length on both datasets? Can you add MMD-VQ, and Wasserstein-VQ at d = 128, to Tables 1–2?

1 comment
AuthorsOpenReview ↗

Response to Reviewer fU7h

Dear Reviewer,

We sincerely thank the reviewer for the positive assessment of the theoretical motivation and practical applicability of AOptSA. We greatly appreciate your constructive suggestions. We have integrated valuable feedback from all reviewers, carried out new experiments, and refined the manuscript. We apologize for the delayed response caused by this extensive revision work. The updates have helped us further enhance experimental validation and improve the clarity and rigor of our paper.

[W1 / Q1] Effectiveness of density reweighting at d = 128 and sensitivity to the reweighting exponent

Although the theoretical deviation between the encoder-output distribution and the asymptotically optimal target decreases as dimensionality increases, density reweighting can still provide practical benefits at d = 128. As shown in Table 5, compared with direct distribution alignment, AOptSA reduces MSQE from 0.269 to 0.242 and improves rFID from 8.23 to 8.05 under the same 128-dimensional setting.

To further investigate the importance of density reweighting, we have added Appendix D.3, which evaluates different reweighting exponents λw∈0,0.5,1.0,1.5,2.0\lambda_w \in \\{0, 0.5, 1.0, 1.5, 2.0\\} on ImageNet-1K and FFHQ. As shown in Figure 10, the theoretically derived exponent λw=1\lambda_w = 1 achieves the lowest validation and late-stage training MSE on both datasets. Stronger reweighting does not necessarily improve performance, suggesting that the theoretical exponent provides a favorable balance between low-density coverage and quantization accuracy.

We evaluated five exponent settings in controlled codebook-optimization experiments on ImageNet-1K and FFHQ. Full-length end-to-end sweeps remain open. Together, they provide additional empirical evidence supporting the proposed density-reweighted target.

[W2 / Q2] Comparison with MMD-VQ and Wasserstein-VQ at matched dimensions

In response to the reviewer's suggestion, we have added MMD-VQ and a matched-dimensional Wasserstein-VQ comparison in Table 3. All three methods use the same VQGAN backbone, a 128-dimensional quantization space, and a 65,536-entry codebook. Results are reported over three independent random seeds (mean ± standard deviation).

The new matched-dimensional comparison is conducted on ImageNet-100. These additional experiments strengthen the comparison with the most closely related distribution-alignment approaches and demonstrate the effectiveness of AOptSA under matched dimensionality.

Table 3: Distribution-alignment methods comparison on ImageNet-100 with matched 128- dimensional latent spaces and three independent random seeds.

MethodDimu(%)↑Perplexity↑rFID↓LPIPS↓
Wasserstein-VQ12897.5 ± 0.7056048 ± 4438.43 ± 0.110.068 ± 0.012
MMD-VQ128100.0 ± 0.058135 ± 3078.36 ± 0.140.067 ± 0.011
AOptSA (ours)128100.0 ± 0.061896 ± 4047.21 ± 0.090.062 ± 0.007
Reviewer_rnVQOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AOptSA to mitigate codebook collapse in vector quantization. Based on asymptotically optimal quantizer theory, it aligns the codebook to a density-reweighted encoder-output distribution using online importance sampling with kNN density estimation and Sinkhorn divergence, while preserving nearest-neighbor quantization and latent dimensionality. Experiments on ImageNet-100/1K and FFHQ demonstrate full codebook utilization and improved reconstruction quality, supported by comprehensive ablations.

Critical strengths
  1. Theoretical Motivation: Provides a principled basis for density-reweighted distribution alignment by distinguishing between matching encoder distributions and optimizing codeword allocation via asymptotically optimal quantizer theory.
  2. Practical Framework: Dynamically aligns codebooks using online importance sampling and Sinkhorn divergence without altering nearest-neighbor assignments or requiring dimensionality reduction, ensuring flexibility across VQ architectures.
  3. Strong Results & Ablations: Achieves full codebook utilization and improves reconstruction metrics over baselines (e.g., SimVQ), with informative ablations and minimal training-time/memory overhead.
Critical weaknesses
  1. Theory-Practice Gap: The theory assumes infinite codebooks and fixed encoder distributions, whereas AOptSA uses finite codebooks and dynamic objectives. Additionally, in 128-D experiments, the reweighting exponent (128/130≈0.985128/130 \approx 0.985) is close to one, questioning the practical impact of density reweighting.
  2. Unreliable Density Estimation: kNN density estimation with only up to 1000 reference points in 128-D space may be inaccurate due to high-dimensional distance concentration. The stability and sensitivity of these importance weights are insufficiently analyzed.
  3. Incomplete Comparisons: Key baselines like MMD-VQ are missing. The isolated gains from density reweighting are modest (e.g., rFID 8.23 to 8.05). Furthermore, results lack evaluations across multiple random seeds or confidence intervals.
  4. Limited Generalization & Scalability: Evaluation focuses on 128×128 image reconstruction, lacking downstream generation tasks or comprehensive ImageNet-1K baseline comparisons. A systematic computational scaling analysis across varying codebook sizes is also missing.
Questions for authors
  1. Practical Target Relevance: Given d/(d+r)≈0.985d/(d+r) \approx 0.985 in 128-D experiments, how much does the density-reweighted target quantitatively differ from the original encoder distribution, and how does this relate to the observed improvements?
  2. Density Estimation Robustness: How sensitive is AOptSA to kNN parameters, buffer sizes, and sampling strategies? Are 1000 reference points reliable in 128-D spaces, and would alternative estimators yield similar results?
  3. Baseline Comparisons: Can MMD-VQ be included under matched settings? Would replacing Sinkhorn divergence with MMD (while keeping the density-reweighted target) yield comparable gains?
  4. Statistical Significance & Applicability: Are improvements consistent across multiple random seeds (please report variance)? Do benefits extend to higher-resolution images or downstream generation tasks?
  5. Computational Scalability: How do training time and memory scale with codebook sizes beyond 65536? Why is the overhead so small (Table 8), and does this efficiency persist at larger scales?
3 comments
AuthorsOpenReview ↗

Response to Reviewer rnVQ (Part 1)

Dear Reviewer,

We sincerely thank the reviewer for the positive assessment of AOptSA's theoretical motivation, practical framework, and experimental results. We highly appreciate your constructive comments and suggestions. Your professional advice has been tremendously helpful to us, and we have accordingly added additional experiments to further substantiate our claims and improve the overall rigor of the manuscript.

[W1 / Q1] Theory–practice gap and practical relevance of density reweighting

To address this concern, we have added Appendix B.3 and Appendix D.3 to further validate the theoretical motivation and practical effectiveness of density reweighting.

Appendix B.3 provides analytical and Monte Carlo verification of the dimensional dependence. As shown in Figure 6, the KL divergence between the original distribution and the asymptotically optimal target decreases from 0.09257 at d = 8 to 0.00194 at d = 512, confirming that the theoretical deviation becomes smaller as dimensionality increases in the controlled Gaussian setting. At d = 128, the analytical KL divergence is 0.00765 in the controlled Gaussian setting, indicating a small but nonzero distributional deviation.

Appendix D.3 evaluates different density-reweighting exponents on ImageNet-1K and FFHQ. As shown in Figure 10, the theoretically derived exponent λw=1\lambda_w=1 achieves the lowest validation and late-stage training MSE on both datasets, demonstrating that appropriate density reweighting remains beneficial in practical quantization settings.

Together, these results support both the theoretical dimensional dependence and the practical effectiveness of the proposed density-reweighted target.

[W2 / Q2] Robustness of kNN density estimation

To address concerns about high-dimensional density estimation, we have added Appendix C.3, which systematically investigates the reference pool size and kNN neighborhood size.

Rather than requiring an accurate absolute density estimate in 128-dimensional space, our method relies on importance weights that provide informative distribution-alignment gradients. As shown in Figures 7–8, using 1,000 reference points achieves Sinkhorn gradient cosine similarities of 0.9261 on FFHQ and 0.9031 on ImageNet-100, relative to independently sampled 65,536-point reference pools.

Moreover, Table 9 shows that increasing the reference pool from 1,000 to 10,000 reduces validation MSE only from 12.26 to 12.22 (0.33%), while requiring 10 times as many pairwise distance computations. These results support 1,000 reference points as a practical trade-off between computational efficiency and quantization accuracy. The selected neighborhood sizes (k = 25 for FFHQ and k = 50 for ImageNet-100) are further examined in Figure 8.

Table 9: Effect of the kNN Reference Pool Size on ImageNet-100 Quantization Performance. Relative kNN cost denotes the number of pairwise distance computations normalized to the 1,000-point setting.

Reference SizeValidation MSEMSE Gap vs. 10kRelative kNN cost
50012.27+0.41%0.5×
100012.26+0.33%1×
200012.24+0.16%2×
500012.23+0.08%5×
1000012.22-10×

[W3 / Q4] Competitive baselines and statistical reliability

In response to the reviewer's suggestions, we have added MMD-VQ and a matched-dimensional Wasserstein-VQ comparison in Table 3, using the same VQGAN backbone, 128-dimensional quantization space, and 65,536-entry codebook. To improve statistical reliability, results are reported over three independent random seeds (mean ± standard deviation).

Table 3: Distribution-alignment methods comparison on ImageNet-100 with matched 128-dimensional latent spaces and three independent random seeds (mean ± standard deviation). All images are 128 × 128 resolution, and codebook size is set to 65,536.

MethodDimu(%)↑Perplexity↑rFID↓LPIPS↓
Wasserstein-VQ12897.5 ± 0.7056048 ± 4438.43 ± 0.110.068 ± 0.012
MMD-VQ128100.0 ± 0.058135 ± 3078.36 ± 0.140.067 ± 0.011
AOptSA (ours)128100.0 ± 0.061896 ± 4047.21 ± 0.090.062 ± 0.007

Under these controlled conditions, AOptSA achieves the lowest rFID (7.21 ± 0.09), compared with Wasserstein-VQ (8.43 ± 0.11) and MMD-VQ (8.36 ± 0.14), while also achieving the highest codebook perplexity (61,896 ± 404). These results demonstrate that the improvements are consistent across multiple runs and strengthen the evidence for AOptSA under matched experimental settings.

We acknowledge that higher-resolution reconstruction and downstream generation tasks remain outside the scope of the current evaluation and are important directions for future work.

AuthorsOpenReview ↗

Response to Reviewer rnVQ (Part 2)

[Q3] Sinkhorn divergence versus MMD

MMD can theoretically replace Sinkhorn divergence for distribution alignment. However, Sinkhorn divergence explicitly incorporates geometric distances between codewords and target samples through optimal transport. With a quadratic Euclidean ground cost (p=2p=2), it is consistent with the squared-error distortion criterion underlying vector quantization.

To further investigate this choice, we have added a controlled comparison between Sinkhorn divergence and MMD in Appendix D.4 (Figure 11) of the revised manuscript, keeping the density-reweighted target distribution unchanged. Specifically, we pretrain a vanilla VQGAN on ImageNet-1K for five epochs, freeze the encoder and decoder, reinitialize the codebook with Gaussian random samples, and optimize only the codebook for six additional epochs.

As shown in Figure 11, Sinkhorn achieves nearly full codebook utilization after the first epoch, consistently yields substantially higher perplexity, and reduces the mean squared quantization error (MSQE) more rapidly than MMD. After six epochs, Sinkhorn achieves 100% codebook utilization compared with 97.16% for MMD, increases perplexity from 3511.41 to 7207.36, and reduces MSQE from 29.0513 to 23.9675, corresponding to a 17.50% relative reduction in quantization error.

These results suggest that, even under the same density-reweighted target distribution, the choice of alignment metric substantially affects codebook optimization. The transport-based geometric structure of Sinkhorn divergence may facilitate more effective codeword allocation, leading to faster codebook activation, more balanced codeword usage, and lower quantization distortion. These empirical findings further support our choice of Sinkhorn divergence as the distribution-alignment objective in AOptSA.

[W4]Limited Generalization and Scalability

To address this concern, we have extended our experimental evaluation across different datasets, latent dimensions, and codebook sizes. On ImageNet-1K, Table 6 further evaluates AOptSA across quantization dimensions of 8, 64, and 128, achieving 100% codebook utilization in all three settings.

We have added Appendix D.1, which compares AOptSA with Wasserstein-VQ and MMD-VQ across four codebook sizes (1,024, 4,096, 8,192, and 65,536). As shown in Table 11, AOptSA consistently achieves the highest perplexity and lowest rFID and LPIPS across all evaluated codebook sizes, demonstrating its effectiveness and scalability beyond a single codebook configuration.

Table 11: Comparison of distribution-alignment methods on ImageNet-100 at different codebook sizes. All methods use 128-dimensional codewords and are trained for 50 epochs.

MethodCodebook Sizeu(%)↑Perplexity↑rFID↓LPIPS↓
Wasserstein-VQ1024100.091612.030.104
MMD-VQ1024100.093411.950.103
AOptSA1024100.094711.330.097
Wasserstein-VQ4096100.0339410.290.093
MMD-VQ4096100.0347110.240.091
AOptSA4096100.036819.750.085
Wasserstein-VQ8192100.069729.920.086
MMD-VQ8192100.070389.830.085
AOptSA8192100.074159.270.079
Wasserstein-VQ6553696.7559329.010.073
MMD-VQ65536100.0580748.940.072
AOptSA65536100.0618908.050.067

Furthermore, we have extended Table 10 to evaluate computational scalability with a larger codebook of 98,304 entries, with detailed results discussed in our response to Q5.

These additional experiments strengthen the evidence for AOptSA's applicability across different quantization configurations. Higher-resolution reconstruction and downstream generation tasks remain important directions for future work.

AuthorsOpenReview ↗

Response to Reviewer rnVQ (Part 3)

[Q5] Computational scalability

To address this concern, we have extended Table 10 with an additional experiment using a larger codebook of 98,304 entries.

The low computational overhead of AOptSA is mainly attributed to two design choices. First, AOptSA removes the learnable linear transformation used in SimVQ, partially offsetting the additional cost of distribution alignment. Second, Sinkhorn alignment is performed only every 10 global steps, using a memory-efficient online implementation. The kNN density estimation also uses only 1,000 reference samples.

At K = 65,536, AOptSA introduces only 0.42% additional training time and 0.45% additional peak GPU memory compared with SimVQ (1.907 vs. 1.899 h/epoch; 35.025 vs. 34.868 GB).

When the codebook size increases by 50% to 98,304, training time increases by only 18.20% (1.907 to 2.254 h/epoch), while peak GPU memory increases by 5.82% (35.025 to 37.065 GB).

These results demonstrate that AOptSA maintains manageable computational and memory costs beyond 65,536 codewords, supporting its practical scalability within the evaluated range.

Table 10: Runtime and peak GPU memory comparison on ImageNet-100. Peak memory is measured under the same training configuration.

MethodCodebook SizeTime/Epoch (h)Peak Memory (GB)
SimVQ65,5361.89934.868
AOptSA65,5361.90735.025
AOptSA98,3042.25437.065
Reviewer_wyLuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed AOptSA, in which an additional loss function is added to the objective function of VQGAN model to prevent codebook collapse. This new loss aligns the empirical codebook distribution with a density reweighted target distribution nsampled from online encoder output features. AOptSA is compared with some baselines over three differnet datasets in image reconstruction task, and hows superior performance against the baselines.

Critical strengths
  1. The paper discussed a new aspect of codebook learning which were not noticed before in the literature.

  2. The proposed idea was quite clearly conveyed through the text along with the equations.

Critical weaknesses

MAJOR

  1. Lack of relevant baselines;

1.1. The proposed method is not comapred againt the recent SOTA methods [1,2,3,4] which claims to resolve the same codebook collapse using different techniques like codebook reset, factorization by looking up in a lower dimensional latent space and VQBridge. Wheras, AOptSA aims to solve the same codebook collapse problem. It is expected to comapre AOptSA with different methods which resolves codebook collapse. In addition, the Vanilla VQGAN and VQGAN-EMA methods do not use any additional tools liek codebook reset or auxiliary loss function to resolve the codebook collapse. So, it is good to have them in the comparison, but I feel it is not fair to take this comparison to show the advantage of the proposed method.

1.2. Why the dimension for Wassertein-VQ method is d=8d=8, whereas according to what reported in Wassertein-VQ paper, the dimension for VQGAN equals d=32d=32. Apart from that, since Wassertein-VQ is the most similar method to the proposed AOptSA, I think for a fair comparison the authors should keep the dimensionality of Wassertein-VQ simialr to AOptSA in the experiments.

  1. No empirical analysis on the theoretical calim;

2.1. As discussed in the paper, in lower dimensional quantization sapces like d=8d=8, the encoder output distribution deviation from the asymptotic target is higher. So, this can be proved empirically by experimenting the proposed method under a high dimensional latent like d=512d=512 versus an 8-dim latent space. I am interested to see how the theoretical deviation will be appeared in this experiment.

2.2. Continuing in this direction, the proposed method can be compared to other caselines over different latent dimension, like {8, 256, 512}, as it is expected that the performance gap with the baselines to be increased as dimensionality decreases.

  1. Comparison using a single codebook size; The proposed method is comapred with the baselines using only a single codebook size of K=65,536K=65,536. For a fair comparison to discover difference in the performance of different methods, it is expexted that the comparison to be done over a range of codebook sizes like K=2048,4096,8192,16k,...K={2048, 4096, 8192, 16k, ...}.

  2. Results are reported over a single run; As training a model over different runs might end up in different results, it is expected to see the results averaged over several runs not a single run.

Citations

[1] Zhu L, Wei F, Lu Y, Chen D. Scaling the codebook size of vq-gan to 100,000 with a utilization rate of 99%. Advances in Neural Information Processing Systems. 2024 Dec 16;37:12612-35.

[2] Chang Y, Qin J, Qiao L, Wang X, Zhu Z, Ma L, Wang X. Scalable training for vector-quantized networks with 100% codebook utilization. InInternational Conference on Learning Representations 2026 Apr 20 (Vol. 2026, pp. 92236-92257).

[3] Vali M H, Backstrom T, Solin A. DiVeQ: Differentiable Vector Quantization Using the Reparameterization Trick. In International Conference on Learning Representations 2026.

[4] Yu J, Li X, Koh JY, Zhang H, Pang R, Qin J, Ku A, Xu Y, Baldridge J, Wu Y. Vector-quantized image modeling with improved vqgan. arXiv preprint arXiv:2110.04627. 2021.

Questions for authors

Q1. Does the Sinkhorn loss includes all the codebook entries or only the ones that are selected as nearest neighbors? According to what I understand from the idea, I think it involves all codebook entries into the Sinkhorn loss and hence, all codebook entries receives gradients from this loss. So, according to Equation 9, why your proposed method needs the codebook loss and commitment loss incorporated in L−\mathcal{L}-VQVAE (showed in Eq. 2)? I am interested to see the ablation if the authors skip L−\mathcal{L}-VQVAE loss in order to see the isolated performance of the proposed Sinkhorn loss.

4 comments
AuthorsOpenReview ↗

Response to Reviewer wyLu (Part 1)

Dear Reviewer,

We sincerely thank the reviewer for highlighting the novelty of our codebook-allocation perspective and the clarity of the presentation. We also greatly appreciate the constructive comments and suggestions, which have helped us further improve the rigor and clarity of our manuscript. Below, we address the related concerns together and provide supporting evidence and clarifications.

[W1.1 / W1.2 / W4] Strong baselines, matched dimensions, and repeated runs

We thank the reviewer for highlighting the importance of competitive baselines, fair comparisons, and statistical reliability. We have strengthened our experimental comparisons in three aspects.

First, we have included two additional competitive methods, FVQ (Table 1) and MMD-VQ (Table 3), alongside FSQ, SimVQ, and Wasserstein-VQ, to provide a more comprehensive evaluation against recent approaches to mitigating codebook collapse.

Second, the d = 8 configuration for Wasserstein-VQ in Table 1 follows its original paper. To ensure a fair comparison, we have added Table 3, which evaluates Wasserstein-VQ, MMD-VQ, and AOptSA using the same backbone, 128-dimensional quantization space, and 65,536-entry codebook.

Third, to improve statistical reliability, the results in Table 3 are reported over three independent random seeds (mean ± standard deviation). Under these matched conditions, AOptSA achieves the lowest rFID (7.21 ± 0.09), outperforming Wasserstein-VQ (8.43 ± 0.11) and MMD-VQ (8.36 ± 0.14).

These additions provide stronger empirical evidence for the effectiveness of AOptSA under controlled and statistically reliable comparisons.

Table 1: Quantitative comparison of VQGAN-based models on ImageNet-100 in terms of codebook utilization and reconstruction performance. All images are 128 × 128 resolution, and codebook size is set to 65,536.

MethodDimu(%)↑Perplexity↑rFID↓LPIPS↓PSNR↑SSIM(%)↑
Vanilla VQGAN1281.065810.270.09922.8766.25
VQGAN-EMA12828.8171328.570.07124.4171.55
FSQ6100.0446438.820.07124.4972.09
Wasserstein-VQ8100.0633999.090.07023.8173.43
SimVQ128100.0508147.940.06724.8174.20
FVQ128100.0502128.060.07124.3771.78
AOptSA (ours)128100.0%619877.200.06325.2476.23

Table 3: Distribution-alignment methods comparison on ImageNet-100 with matched 128-dimensional latent spaces and three independent random seeds (mean ± standard deviation). All images are 128 × 128 resolution, and codebook size is set to 65,536.

MethodDimu(%)↑Perplexity↑rFID↓LPIPS↓
Wasserstein-VQ12897.5 ± 0.7056048 ± 4438.43 ± 0.110.068 ± 0.012
MMD-VQ128100.0 ± 0.058135 ± 3078.36 ± 0.140.067 ± 0.011
AOptSA (ours)128100.0 ± 0.061896 ± 4047.21 ± 0.090.062 ± 0.007

Table 5 of the Wasserstein-VQ paper lists 8-dimensional codewords for its FFHQ VQGAN configuration. Our Table 3 additionally compares all three methods at d = 128.

Table 5: Hyperparameters for the experiments in Table 1, 2, 3 and Table 6 and 7.

FrameworksVQ-VAEVQ-VAEVQ-VAEVQGAN
DatasetCIFAR-10/SVHNFFHQImageNetFFHQ
Input size32 × 32 × 3256 × 256 × 3256 × 256 × 3256 × 256 × 3
Latent size8 × 8 × 816 × 16 × 816 × 16 × 816 × 16 × 8
encoder/decoder channels646464128
encoder/decoder channel mult.[1, 1, 2][1, 1, 2, 2, 4][1, 1, 2, 2, 4][1, 1, 2, 2, 4]
Batch size128323264
Initial Learning rate lrlr5×10−55 \times 10^{-5}5×10−55 \times 10^{-5}5×10−55 \times 10^{-5}5×10−45 \times 10^{-4}
Codebook Loss Coefficient1.01.01.01.0
Perceptual loss Coefficient000.01.0
Adversarial loss Coefficient000.00.2
Codebook dimensions8888
Training Epochs50304200
GPU ResourcesV100 16GBA100 40GBA100 40GB2 A100 80GB
AuthorsOpenReview ↗

Response to Reviewer wyLu (Part 2)

[W2.1 / W2.2] Empirical dimensional dependence and scope of the theory

To address this concern, we have added Appendix B.3, which provides both theoretical analysis and empirical verification of the deviation between the original encoder-output distribution and the asymptotically optimal target distribution.

Specifically, we consider isotropic Gaussian distributions with dimensions d∈8,32,128,512d \in \\{8, 32, 128, 512\\}. We analytically derive the KL divergence between the two distributions and validate it using Monte Carlo experiments with 200,000 samples per dimension. As shown in Figure 6, the empirical estimates closely match the theoretical values, and the KL divergence decreases from 0.09257 at d = 8 to 0.00194 at d = 512, a 47.6-fold reduction. This quantitatively confirms the predicted dimensional dependence in the controlled Gaussian setting, supporting the importance of density reweighting in lower-dimensional spaces.

We adopt this controlled setting because end-to-end comparisons involve multiple optimization objectives, including quantization, reconstruction, perceptual, and adversarial losses, making it difficult to isolate the distributional deviation predicted by the theory. The Gaussian experiment therefore provides a more direct verification of the theoretical claim, while the reconstruction experiments in Table 6 complement this analysis by evaluating practical performance across quantization dimensions.

[W3] Comparisons across codebook sizes

In response to the reviewer's suggestion, we have added Appendix D.1, which extends the comparison with two closely related distribution-alignment methods, Wasserstein-VQ and MMD-VQ, across four codebook sizes: K∈1,024,4,096,8,192,65,536K \in \\{1,024, 4,096, 8,192, 65,536\\}.

All methods use the same VQGAN backbone, 128-dimensional codewords, and 50 training epochs to ensure a fair comparison. As shown in Table 11, AOptSA consistently achieves 100% codebook utilization, the highest perplexity, and the lowest rFID and LPIPS across all codebook sizes. For example, at K = 65,536, AOptSA achieves an rFID of 8.05, compared with 9.01 for Wasserstein-VQ and 8.94 for MMD-VQ.

These additional results demonstrate that AOptSA consistently outperforms the competing distribution-alignment methods across different codebook sizes, confirming its effectiveness and scalability beyond a single codebook configuration.

Table 11: Comparison of distribution-alignment methods on ImageNet-100 at different codebook sizes. All methods use 128-dimensional codewords and are trained for 50 epochs.

MethodCodebook Sizeu(%)↑Perplexity↑rFID↓LPIPS↓
Wasserstein-VQ1024100.091612.030.104
MMD-VQ1024100.093411.950.103
AOptSA1024100.094711.330.097
Wasserstein-VQ4096100.0339410.290.093
MMD-VQ4096100.0347110.240.091
AOptSA4096100.036819.750.085
Wasserstein-VQ8192100.069729.920.086
MMD-VQ8192100.070389.830.085
AOptSA8192100.074159.270.079
Wasserstein-VQ6553696.7559329.010.073
MMD-VQ65536100.0580748.940.072
AOptSA65536100.0618908.050.067

[Q1] Full-codebook Sinkhorn gradients and the role of VQ-VAE losses

As correctly noted by the reviewer, the Sinkhorn alignment loss involves all codebook entries, allowing even unselected codewords to receive gradients. To clarify why the conventional VQ-VAE losses remain necessary, we have added Appendix D.2, which includes an additional ablation study (Figure 9 and Table 12).

The two objectives play complementary roles: Sinkhorn alignment globally regularizes the codebook distribution to encourage effective codeword allocation, while the VQ-VAE quantization loss locally minimizes the distortion between encoder outputs and their assigned codewords. The commitment loss further encourages consistency between encoder outputs and their quantized representations during end-to-end training.

To empirically verify this complementarity, we compare Sinkhorn alignment alone with its combination with the quantization loss under a frozen encoder–decoder setting. As shown in Figure 9 and Table 12, adding the quantization loss reduces test-set MSE from 1.993 to 1.911 (4.11%) and increases codebook perplexity from 6,576 to 6,824 (3.77%).

These results demonstrate that global distribution alignment and local quantization optimization are complementary, jointly improving quantization accuracy and codebook utilization.

Table 12: Quantization MSE on test set of the ablation study on the quantization objective

ObjectiveMSEPerplexity
Sinkhorn Alignment Only1.9936576
Quantization + Sinkhorn Alignment1.9116824
Relative improvement4.11%3.77%
Reviewer_wyLuOpenReview ↗

Final Evaluation

I thank authors for their response and the new edits/experiments they made. Some of my main concerns are still there and as a result, I will keep my score because of the following reasons:

  • Still I think the paper lacks comparing with the strong SOTA methods that are proposed to solve the codebook collapse problem. I think:
  1. FSQ is not relevant that much for comparison as it s a calar quantization technique (which lose too much info for extremly skipping corerlations among different latent dimension).
  2. Vanilla VQ and VQGAN-EMA is not relevant comparisons as they do not use any techniques to resolve the codebook collapse.
  • The rsukts on table 1 are still reported on a signle run, and are not statistically significant.
  • I asked for comapring the proposed method with SOTA baselines over different codebook sizes. However, the proposed method is only compared against Wassertain-VQ and MMD-VQ.
  • For table 11 and table 3, the SSIm and PSNR metrics are not reported.

The paper actullay lacks a proper and thorough experimental comparison agaisnt relevant baseline.

Dear Reviewer,

Thank you very much for carefully considering our rebuttal and providing further constructive feedback.

We appreciate your concerns regarding the comprehensiveness and statistical reliability of our experimental comparisons. We acknowledge the remaining limitations and agree that broader comparisons with competitive baselines, more extensive multi-seed experiments across different codebook sizes, and additional reconstruction metrics would further strengthen our work.

We will take your valuable suggestions into serious consideration in our future work. We sincerely appreciate your time, careful evaluation, and constructive criticism, which have helped us improve our manuscript and identify important directions for further research.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Summary of Rebuttal and Manuscript Revisions

Dear Area Chair and Reviewers,

 As the discussion period draws to a close, we would like to sincerely thank all reviewers and the Area Chair for their time, thoughtful evaluation, and constructive feedback. The reviewers' valuable suggestions have helped us substantially strengthen the theoretical and empirical validation of AOptSA and improve the clarity and rigor of our manuscript.

 During the rebuttal period, we have addressed all the raised concerns and incorporated the following major additions into the revised manuscript:

 Stronger baselines and statistically reliable comparisons (Tables 1 and 3). We added FVQ and MMD-VQ, conducted matched-dimensional comparisons with Wasserstein-VQ and MMD-VQ, and reported results over three independent random seeds. AOptSA consistently achieves superior reconstruction performance.

 Further theoretical and empirical validation of density reweighting (Appendices B.3 and D.3). We analytically and empirically verified the dimensional dependence of the distributional deviation and demonstrated that the theoretically derived reweighting exponent achieves the lowest quantization error on both ImageNet-1K and FFHQ in the controlled experiments.

 Robustness of kNN density estimation (Appendix C.3). We systematically investigated the reference pool and neighborhood sizes, verifying that a small reference pool provides informative alignment gradients while maintaining computational efficiency.  Additional ablations of the optimization objectives (Appendices D.2 and D.4). We demonstrated the complementary roles of global distribution alignment and local quantization optimization, and further showed that Sinkhorn divergence achieves faster codebook activation, higher perplexity, and lower quantization error than MMD under controlled conditions.

 Extended evaluation of effectiveness and scalability (Appendix D.1 and Table 10). We demonstrated consistent improvements over competing distribution-alignment methods across codebook sizes ranging from 1,024 to 65,536 and further verified manageable computational and memory costs at 98,304 codewords.

 We are particularly grateful to Reviewer fU7h for carefully reconsidering our work, confirming that their main concerns have been adequately addressed, and raising their overall score following our rebuttal. We hope that our detailed responses and additional experimental evidence also help address the concerns raised by the other reviewers.

 We respectfully hope that the Area Chair and the reviewers will take these additional findings and manuscript improvements into consideration in their final assessment.

 Once again, we sincerely thank all reviewers and the Area Chair for their constructive feedback, time, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.