acceptodds
Under review as a conference paper at ICLR 2027

Attentive Vector Quantization

Abstract

Discrete latent representations are widely used in generative and representation learning, but conventional vector quantization commonly relies on hard nearest-neighbor assignment with a straight-through estimator (STE), introducing a mismatch between forward computation and backward gradients. Learned vector quantizers can also exhibit highly non-uniform code usage or codebook collapse. We introduce Attentive Vector Quantization (AVQ), an STE-free differentiable quantization approach that replaces nearest-neighbor assignment during training with temperature-controlled cosine-similarity attention between latent tokens and a learned codebook. AVQ combines a soft-to-hard training curriculum, a compact codebook-usage regularizer that encourages confident local assignments and balanced global utilization, and hard-code decoder calibration to bridge differentiable training and discrete inference. We evaluate AVQ against conventional vector quantization (VQ), EMA-based vector quantization (VQ-EMA), and finite scalar quantization (FSQ) on COCO, Tiny ImageNet, and MiniPlaces. Across all three datasets, AVQ achieves complete codebook utilization with near-maximal hard-code perplexity and the lowest rFID among the evaluated single-stage quantizers. On COCO, across three model seeds, AVQ uses all 1024 codes with a perplexity of and achieves an rFID of , an 18.0% reduction relative to the best baseline mean. Similar improvements in rFID are observed on MiniPlaces and Tiny ImageNet, while AVQ maintains complete utilization and perplexity above 1000 on all three datasets. Across all single-stage AVQ ablations, complete codebook utilization persists under changes in codebook size, embedding dimension, normalization, learned code scaling, and spatial rate. We further extend AVQ to Attentive Residual Vector Quantization (ARVQ). Across all three datasets, ARVQ maintains near-maximal stage-wise perplexity above 1016 at every residual stage, whereas EMA-based residual vector quantization (RVQ-EMA) exhibits substantially lower and less consistent stage-wise perplexity. These results indicate that attentive quantization provides robust use of discrete capacity in both single-stage and residual quantization settings.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vector Quantization, Discrete Representation Learning, Differentiable Quantization, Image Tokenization, Residual Vector Quantization, Representation Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_gfHROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Attentive Vector Quantization (AVQ), an STE-free differentiable quantization framework that replaces hard nearest-neighbor assignment during training with temperature-controlled cosine-similarity attention over a learnable codebook. AVQ further combines a soft-to-hard training curriculum, local entropy and global code-usage regularization, and a hard-code decoder calibration stage to reduce the mismatch between differentiable training and discrete inference. The method is also extended to residual quantization as ARVQ. Experiments on COCO, MiniPlaces, and Tiny ImageNet show that AVQ achieves near-maximal codebook perplexity and full code utilization, while obtaining the best rFID among the evaluated single-stage baselines. ARVQ similarly improves stage-wise codebook utilization compared with RVQ-EMA, although its reconstruction and perceptual metrics are generally weaker.

Critical strengths

None

Critical weaknesses

The paper is generally rough; the design of the experimental validation and the scale of the datasets used are mediocre, and there is significant room for improvement in the writing. Below are some weaknesses and specific suggestions:

  1. One of my primary concerns is that the method in the paper is overly similar to that of [1]; the core operation in both works is essentially a temperature-controlled soft assignment over a learnable codebook, followed by a weighted sum of the codewords. Additionally, there is limited comparison with recent methods such as [2] and [3]; I suggest adding references to them.

  2. AVQ actually incorporates several key design elements: temperature annealing, soft-to-hard mixing, local entropy regularization, global balance regularization, and Phase-II hard-code decoder calibration. However, current ablation studies focus primarily on codebook size, embedding dimension, normalization, code scaling, and spatial rate, making it impossible to determine which specific component drives the performance gains. Notably, the paper's most prominent results regarding high utilization and perplexity are directly driven by the explicit balance loss.

  3. Although AVQ achieves near-maximal perplexity and 100% code utilization, it underperforms VQ-EMA in terms of MSE across all three datasets, and its LPIPS performance is not consistently optimal. This discrepancy is even more pronounced in ARVQ: while ARVQ exhibits significantly higher stage-wise perplexity, it lags behind RVQ-EMA in MSE, LPIPS, and rFID across the three datasets. Consequently, the paper has not yet adequately demonstrated why "more balanced code usage" is, in itself, a sufficiently valuable objective.

  4. If computational resources permit, it is recommended to include downstream tasks—such as generation experiments—to demonstrate that a superior codebook has been learned.

refs:

[1] Chen, H., Wang, Z., Li, X., Sun, X., Chen, F., Liu, J., ... & Barsoum, E. (2025, June). Softvq-vae: Efficient 1-dimensional continuous tokenizer. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 28358-28370). IEEE.

[2] Yang, S., Hu, X., Wu, Q., & Yang, D. (2026, March). Vaevq: Enhancing discrete visual tokenization through variational modeling. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 14, pp. 11703-11711).

[3] Zhang, L., Li, S., Wang, Z., Xie, Q., Tan, C., Wang, A., ... & Wang, H. (2026). MergeTok: Unified Continuous and Discrete Visual Tokenization via Token Merging. arXiv preprint arXiv:2605.30904.

Questions for authors

The core formulation of AVQ appears closely related to SoftVQ-VAE [1], as both use temperature-controlled soft assignment over a learnable codebook followed by weighted codeword aggregation. Could the authors clarify the key conceptual and algorithmic differences between AVQ and SoftVQ-VAE, beyond the soft-to-hard curriculum and hard-code decoder calibration? This distinction is important for assessing the novelty of the proposed method.

Reviewer_xDp9OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

AVQ trains a cosine-similarity soft quantizer using temperature annealing, soft/hard interpolation, and entropy regularization, followed by decoder calibration on hard codes. Experiments on COCO, Tiny ImageNet, and MiniPlaces show better single-stage rFID and hard-code utilization. Its residual extension, ARVQ, improves stage-wise perplexity but trails RVQ-EMA on reconstruction and perceptual metrics.

Critical strengths
  1. AVQ improves rFID alongside almost uniform hard-code usage across three datasets. Since evaluation uses hard codes, these results concern the representation actually used at inference. This combination gives the approach practical promise, provided the gains hold under the comparisons discussed below.

  2. The three seeds, separate selection and final evaluation sets, matched backbones, and calibration for every baseline make the results easier to trust. Recalculating means and sample standard deviations of MSE, LPIPS, and rFID from the 54 supplied main-experiment JSON records reproduces the reported main results at the stated rounding. The calibration ablation also provides evidence about how much this step contributes.

Critical weaknesses
  1. For unit-normalized vectors, squared Euclidean distance equals twice one minus cosine similarity, so AVQ's cosine-based assignment is equivalent to the distance-based softmax in Agustsson et al., 2017 after temperature rescaling. That work also anneals soft assignments toward hard quantization. The local/global entropy structure is related to the already cited LFQ. The full workflow includes additional choices, but their contribution remains unclear. The existing calibration and architectural ablations leave interpolation and sharpening insufficiently isolated; a matched soft-VQ comparison would help establish their value.

  2. The checkpoint-selection protocol allows baselines to be selected throughout training but restricts AVQ selection until annealing completes. On Tiny ImageNet, all six VQ-EMA/FSQ main runs select epoch 5, with zero adversarial loss in the saved histories and nonzero loss the following epoch. AVQ selects epochs 41–43. The selected models have therefore received different amounts of adversarial training, even though the full training runs finished. The concern is specific: VQ selects epochs 43–49 and RVQ-EMA epochs 44–49 on the same dataset. Calibration is already applied to all methods, and the rFID advantage might persist. However, without comparable-stage or perceptual-selection results, it remains difficult to attribute these gains primarily to AVQ. Together with the unresolved contribution, this makes rejection the stronger recommendation on the current evidence.

Questions for authors
  1. Which component provides the main advance beyond normalized soft VQ and the cited entropy objective, and which existing result best supports that claim?

  2. A matched normalized soft-VQ baseline, using the same backbone, regularization, and calibration, is needed to establish the added value of the method. The effects of interpolation and sharpening should also be isolated.

  3. How do Tiny ImageNet VQ-EMA/FSQ results change when selection is restricted to checkpoints with adversarial training? Reporting the MSE–rFID trade-off on the internal selection set would help assess this comparison, with final evaluation data kept separate from selection.

Reviewer_ZPRFOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper prposed Attentive Vector Quantization (AVQ) method that is differentiable and does not need tircks like Straight Through Estimator (STE) to backpropagate gradients through non-differentiable VQ module. AVQ is inspired by attention mecahnism by a dot product between latnets and codebook vectors. AVQ uses auxiliary loss LCB\mathcal{L}_{CB} to encourgae higher perplexity (codebook usage) and uses soft-to-hard temperature annealing to make the sampling close to one-hot at the end of training. They evaluated and compared their method with VQ (using STE), and VQ optimized by EMA, and FSQ technqiues in an image reconstruction task.

Critical strengths
  1. Well-structured paper; The paper was well-structured and easy to read, and was successful to convey the message of the proposed method.
Critical weaknesses

MAJOR

  1. Lack of contribution; A quite similar attention idea is proposed in NWT paper [1]. Also, soft-to-hard annealing is well-discussed in the literature as in [2,3,4,5]. Also, adding an entropy-based auxiliary loss to increase the codebook usage is also stduied in the literature as in [2,6]. What is the individual contribution that AVQ brings over the previous methods in the literature?

  2. Not a fair comparison;

2.1. AVQ is compared with thre methods of VQ (using STE), VQ using EMA to update the codebook and FSQ (scalar quantization). The paper is compared with these three baselines in which there is no usage of auxiliary loss, codebook reset, or any other tools to skip codebook collapse issue. Whereas AVQ has an auxiliary loss to encourage high codebook usage. A fair comparison would be to use the same auxiliary loss or if not applicable a codebook reset for the baselines.

2.2. Apart from that, according to the results, the authors claim that AVQ reach higher codeook usage than the baselines. There are many more enhanced baseline SOTA methods to compare AVQ with for both single codebook and multi codeook (Residual VQ) in which they proposed solutions to solve codeook collapse like [1,7,8,9,10,11]. So, it is expected the authors to compare the AVQ at least with some of these baselines.

  1. No superior results than baselines; According to the results in Tables 1 and 3, AVQ clearly has worse perfrormance than the basleines reagrding the quality-based evaluation metrics like MSE, LPIPS, and r-FID (only in Table 3). Whereas AVQ has higher perplexity and codebook utilization. This can show that the AVQ's training pipeline learns a codebook which is densely concentrated on a localized region of the latent space which has high utilization and perplexity but it not a useful codebook representation that serves for the downstream task. In this line, I suspect that AVQ codebokk losses has some contradictory effect toward each other and suggest an ablation stuty to the authors to keep one of them for training or play with the weighting coefficinets of loss to see if this thing I suspect as a probelm will be solved! Because, in quantization reasearch if a codebook is learned appropriately, the higher perplexity and utilization should usually (not necessarily and always) lead to a better reconstruction quality.

  2. Evaluate on more dense datasets and higher resolutions; All the datasets in the paper are quite diverse. By diverse I mean they include a diverse set of images from a diverse range of objects and scenes. However, it is good to compare AVQ with some datasets like FFHQ, CelebA-HQ, and LSUN which are also used in other SOTA methods for reconstruction taks. In addition, it is recommended to do the evaluation on higher resolution datsets (like at least 256x256) in which the perfomance difference can be better noticed.

  3. Not a well-studied and comprehensive related work section.

  4. Additional training hassle; I find the second training stage (i.e., decoder callibration) as additional training hassle introduced in this work, because it needs an additonal hyperparameter setting like learning rate, batch size, wrm-up, etc. In recommend authors to try to marge them and have all the training in one phase, as what is proposed in most/all existing SOTA methods for image coding.

  5. Not a complete ablation on codebook size; I appreciate the ablation done in the paper. However, it sounds incomplete since it only covers one additional codebook size that is 512, which is really low. Add more codebook sizes which are lower and higher than K=1024 like 256, 2048, 4096, and 8192. This is while AVQ claims to solve codebook collapse, while codebook collapse is mainly challenging in larger codebook sizes.

  6. No comparison with the baseliens in the Ablation Study; Please include the baselines in the ablation study to point out what is the superiority of the proposed AVQ under different scenarios of ablations.

MINOR

  1. Complete evaluation metrics by including SSIM and PSNR.
Questions for authors

Please refer to the Weaknesses section.

2 comments
Reviewer_ZPRFOpenReview ↗

Final evaluation

I thank the authors for their new experiments. That improves the paper, but the paper still have the deficits I raised, which are skipping comparison with NWT, higher resolution datasets, two stage training as a hassle. Also, the experiment with the new codebook size K=8192 is not sill inserted into the paper, although the authors mentioned this is the experiment they have done. Apart from these, I also think the paper results are not still that much impressing to beat some baselines, and recommend to authors to check why the perplexity is high but the quality is worse with higher perplexity. Also, still lack of novelty. All in all, I will keep my score.

AuthorsOpenReview ↗

Response to Final Evaluation and Clarifications

We thank the reviewer for the final evaluation and for acknowledging that the additional experiments improved the paper.

We would like to clarify two points. First, the K=8192K=8192 experiments are included in the revised manuscript. The main text now explicitly discusses scaling AVQ and ARVQ to K=8192K=8192, and the complete results are reported in Appendix B.7. On Tiny ImageNet, increasing KK from 1024 to 8192 improves AVQ rFID from 28.27 to 14.82 and ARVQ rFID from 5.65 to 3.97, while maintaining near-maximal perplexity.

Second, we agree that the two-phase training procedure adds complexity, and we now explicitly discuss this as a limitation. However, Phase II is not an additional optimization stage applied only to AVQ/ARVQ. The same decoder-only hard-code calibration protocol is applied to all evaluated methods, with the encoder and quantizer frozen, so it does not provide AVQ/ARVQ with a method-specific optimization advantage.

We also agree that high perplexity does not necessarily imply better reconstruction or generation quality. Perplexity measures how broadly and evenly the available discrete codes are used, whereas reconstruction and generative quality additionally depend on the geometry and information content of the learned representation, decoder quality, and the predictability of the resulting token distribution. We have strengthened this discussion in the revised manuscript and added regularizer ablations showing that the global balance term is primarily responsible for the high perplexity. The revision also explicitly notes that broad and balanced code usage alone is not sufficient to guarantee better reconstruction or perceptual quality. At the same time, the larger-codebook experiment shows that, within AVQ/ARVQ, increasing KK substantially improves reconstruction and perceptual quality while retaining near-maximal perplexity.

We acknowledge the remaining concerns regarding NWT and higher-resolution datasets. Completing controlled NWT comparisons and three-seed experiments on datasets such as FFHQ, CelebA-HQ, or LSUN was not computationally feasible within the rebuttal period, particularly because the additional three-seed experiments included in the revision required approximately 1.5 months of compute. We therefore treat these as limitations and plan to include NWT and higher-resolution evaluations in subsequent work.

Finally, regarding novelty, we have substantially expanded the Related Work and clarified the positioning of AVQ relative to prior soft-to-hard and attention-based quantization approaches. Our contribution is not attention or annealing individually, but their integration into an STE-free quantization framework with temperature-controlled cosine assignment, explicit soft-to-hard decoder exposure, hard categorical inference, and extension to residual quantization, together with systematic analysis of utilization, perplexity, scaling, and downstream generation.

We thank the reviewer again for the detailed feedback, which helped us substantially strengthen the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript and Summary of Changes

We thank the reviewers for their careful evaluation of our paper and for the constructive comments and suggestions. We have uploaded a revised manuscript that incorporates the feedback received during the review period. We would be grateful if the reviewers could consider the revised manuscript. The main changes are summarized below:

  1. Page 1: Updated the abstract to include additional quantitative results, including gFID and codebook-scaling results, as well as the limitation associated with the Phase-I checkpoint-eligibility protocol.

  2. Page 2: Revised the contribution statements to better clarify the novelty of the proposed approach, the baseline comparisons, and the additional experiments and analyses included in the revision.

  3. Pages 2–3: Expanded and updated the Related Work section, including discussion of the additional literature and references suggested by the reviewers.

  4. Pages 5–9: Substantially revised the experimental results and analysis. We updated several tables with results for additional baseline methods; added codebook-scaling experiments up to K=8192K=8192; added AVQ ablation results showing the role of the global balance term in achieving high codebook perplexity; clarified the Phase-I checkpoint-eligibility limitation; added training-time and quantizer-runtime measurements; expanded the analysis of class-conditional generation and residual representations; and added a gFID comparison of the baseline methods and AVQ/ARVQ.

  5. Pages 9–10: Revised the Discussion, Limitations, and Conclusion sections to incorporate the additional experimental findings and explicitly discuss the limitations identified during review.

  6. Appendix: Updated several tables and figures and added a section describing the class-conditional prior together with additional analysis.

Regarding the suggested evaluation on standard higher-resolution datasets such as FFHQ, CelebA-HQ, and LSUN, the three-seed experiments reported in the revised manuscript required approximately 1.5 months of compute. Consequently, it was not feasible to complete additional large-scale experiments within the short rebuttal period. For the same reason, we were unable to rerun all experiments with a unified Phase-I checkpoint-eligibility protocol during the rebuttal period; we have instead clearly documented this limitation throughout the revised manuscript. We plan to conduct these higher-resolution experiments following the review period and will also include NWT/Memcodes as an additional attention-based quantization baseline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.