- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Attentive Vector Quantization (AVQ), an STE-free differentiable quantization framework that replaces hard nearest-neighbor assignment during training with temperature-controlled cosine-similarity attention over a learnable codebook. AVQ further combines a soft-to-hard training curriculum, local entropy and global code-usage regularization, and a hard-code decoder calibration stage to reduce the mismatch between differentiable training and discrete inference. The method is also extended to residual quantization as ARVQ. Experiments on COCO, MiniPlaces, and Tiny ImageNet show that AVQ achieves near-maximal codebook perplexity and full code utilization, while obtaining the best rFID among the evaluated single-stage baselines. ARVQ similarly improves stage-wise codebook utilization compared with RVQ-EMA, although its reconstruction and perceptual metrics are generally weaker.
Critical strengths
None
Critical weaknesses
The paper is generally rough; the design of the experimental validation and the scale of the datasets used are mediocre, and there is significant room for improvement in the writing. Below are some weaknesses and specific suggestions:
-
One of my primary concerns is that the method in the paper is overly similar to that of [1]; the core operation in both works is essentially a temperature-controlled soft assignment over a learnable codebook, followed by a weighted sum of the codewords. Additionally, there is limited comparison with recent methods such as [2] and [3]; I suggest adding references to them.
-
AVQ actually incorporates several key design elements: temperature annealing, soft-to-hard mixing, local entropy regularization, global balance regularization, and Phase-II hard-code decoder calibration. However, current ablation studies focus primarily on codebook size, embedding dimension, normalization, code scaling, and spatial rate, making it impossible to determine which specific component drives the performance gains. Notably, the paper's most prominent results regarding high utilization and perplexity are directly driven by the explicit balance loss.
-
Although AVQ achieves near-maximal perplexity and 100% code utilization, it underperforms VQ-EMA in terms of MSE across all three datasets, and its LPIPS performance is not consistently optimal. This discrepancy is even more pronounced in ARVQ: while ARVQ exhibits significantly higher stage-wise perplexity, it lags behind RVQ-EMA in MSE, LPIPS, and rFID across the three datasets. Consequently, the paper has not yet adequately demonstrated why "more balanced code usage" is, in itself, a sufficiently valuable objective.
-
If computational resources permit, it is recommended to include downstream tasks—such as generation experiments—to demonstrate that a superior codebook has been learned.
refs:
[1] Chen, H., Wang, Z., Li, X., Sun, X., Chen, F., Liu, J., ... & Barsoum, E. (2025, June). Softvq-vae: Efficient 1-dimensional continuous tokenizer. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 28358-28370). IEEE.
[2] Yang, S., Hu, X., Wu, Q., & Yang, D. (2026, March). Vaevq: Enhancing discrete visual tokenization through variational modeling. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 14, pp. 11703-11711).
[3] Zhang, L., Li, S., Wang, Z., Xie, Q., Tan, C., Wang, A., ... & Wang, H. (2026). MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging. arXiv preprint arXiv:2605.30904.
Questions for authors
The core formulation of AVQ appears closely related to SoftVQ-VAE [1], as both use temperature-controlled soft assignment over a learnable codebook followed by weighted codeword aggregation. Could the authors clarify the key conceptual and algorithmic differences between AVQ and SoftVQ-VAE, beyond the soft-to-hard curriculum and hard-code decoder calibration? This distinction is important for assessing the novelty of the proposed method.
Final evaluation
I thank the authors for their new experiments. That improves the paper, but the paper still have the deficits I raised, which are skipping comparison with NWT, higher resolution datasets, two stage training as a hassle. Also, the experiment with the new codebook size K=8192 is not sill inserted into the paper, although the authors mentioned this is the experiment they have done. Apart from these, I also think the paper results are not still that much impressing to beat some baselines, and recommend to authors to check why the perplexity is high but the quality is worse with higher perplexity. Also, still lack of novelty. All in all, I will keep my score.