- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper builds on distribution-alignment remedies for codebook collapse. It observes that, by asymptotic quantization theory, the empirical distribution of an optimal codebook converges to a density proportional, not to the encoder density h_z itself. AOptSA builds an online target set by importance-sampling encoder outputs, using a kNN density estimate and Efraimidis–Spirakis weighted reservoir sampling. It then aligns the codebook to this target with a debiased Sinkhorn divergence that updates only the codebook. Against VQGAN-EMA, FSQ, Wasserstein-VQ, and SimVQ, the method reaches 100% utilization and the best rFID / LPIPS / PSNR / SSIM (Tables 1–2), with ablations on dynamic alignment, the target distribution, codebook size, and dimensionality.
Critical strengths
The core observation is correct and well grounded in classical theory: matching the codebook to the raw encoder density is not what an asymptotically optimal quantizer does. The method keeps the standard nearest-neighbor assignment and full latent dimensionality, and the gradient only touches the codebook. It is easy to add to existing VQ models.
Critical weaknesses
In the main setting (d = 128), the theory predicts the reweighting should matter least, and the implemented weight largely reduces to a kNN-distance heuristic. MMD-VQ is described in Related Work as the method that removes Wasserstein-VQ's Gaussian assumption and aligns to the raw encoder distribution. It is the most direct competitor, but it does not appear in Exp.
Questions for authors
What happens if the importance-weight exponent is varied at full training length on both datasets? Can you add MMD-VQ, and Wasserstein-VQ at d = 128, to Tables 1–2?
Response to Reviewer fU7h
Dear Reviewer,
We sincerely thank the reviewer for the positive assessment of the theoretical motivation and practical applicability of AOptSA. We greatly appreciate your constructive suggestions. We have integrated valuable feedback from all reviewers, carried out new experiments, and refined the manuscript. We apologize for the delayed response caused by this extensive revision work. The updates have helped us further enhance experimental validation and improve the clarity and rigor of our paper.
[W1 / Q1] Effectiveness of density reweighting at d = 128 and sensitivity to the reweighting exponent
Although the theoretical deviation between the encoder-output distribution and the asymptotically optimal target decreases as dimensionality increases, density reweighting can still provide practical benefits at d = 128. As shown in Table 5, compared with direct distribution alignment, AOptSA reduces MSQE from 0.269 to 0.242 and improves rFID from 8.23 to 8.05 under the same 128-dimensional setting.
To further investigate the importance of density reweighting, we have added Appendix D.3, which evaluates different reweighting exponents λw∈0,0.5,1.0,1.5,2.0 on ImageNet-1K and FFHQ. As shown in Figure 10, the theoretically derived exponent λw=1 achieves the lowest validation and late-stage training MSE on both datasets. Stronger reweighting does not necessarily improve performance, suggesting that the theoretical exponent provides a favorable balance between low-density coverage and quantization accuracy.
We evaluated five exponent settings in controlled codebook-optimization experiments on ImageNet-1K and FFHQ. Full-length end-to-end sweeps remain open. Together, they provide additional empirical evidence supporting the proposed density-reweighted target.
[W2 / Q2] Comparison with MMD-VQ and Wasserstein-VQ at matched dimensions
In response to the reviewer's suggestion, we have added MMD-VQ and a matched-dimensional Wasserstein-VQ comparison in Table 3. All three methods use the same VQGAN backbone, a 128-dimensional quantization space, and a 65,536-entry codebook. Results are reported over three independent random seeds (mean ± standard deviation).
The new matched-dimensional comparison is conducted on ImageNet-100. These additional experiments strengthen the comparison with the most closely related distribution-alignment approaches and demonstrate the effectiveness of AOptSA under matched dimensionality.
Table 3: Distribution-alignment methods comparison on ImageNet-100 with matched 128- dimensional latent spaces and three independent random seeds.