acceptodds
Withdrawn from ICLR 2027

A Unified Rate–Distortion Perspective on Vector, Product, and Scalar Quantization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Discrete visual tokenization relies mainly on vector, scalar, and product quantization, but lacks a unified conceptual framework for understanding their tradeoffs. In this paper, we propose a unified rate-distortion perspective on modern discrete visual tokenization. By viewing quantization as lossy compression, we characterize the nominal fixed-length coding rate through token count and codebook size, and quantization error as the distortion. Within this framework, we resolve three central questions. First, our theoretical and empirical results identify minimizing distortion, rather than maximizing codebook utilization, as the primary intrinsic objective for reconstruction fidelity and link distortion directly to the STE-induced gradient discrepancy. Second, we identify two conditions for fair intrinsic comparison: controlling latent feature statistics and matching nominal coding rates. Third, under these conditions, we recover the VQ-PQ-SQ distortion hierarchy in modern visual tokenization; modern VQ methods attain the lowest distortion in our experiments. The rate-distortion perspective clarifies quantizer evaluation by isolating intrinsic quantization effectiveness under controlled fixed-rate constraints.

Primary area:
generative models
Keywords:
Discrete Visual Tokenization, Vector Quantization, Product Quantization, Scalar Quantization, Rate-Distortion Theory, Quantization Distortion

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QyYYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified rate–distortion framework to analyze and fairly compare three dominant families of discrete visual tokenizers—vector quantization (VQ), product quantization (PQ), and scalar quantization (SQ). It makes three claims. (1) Distortion, not codebook utilization, is the primary intrinsic objective. This is supported by Proposition 1, a local Taylor bound linking distortion to the STE gradient gap, and Spearman correlations between E and r-FID. (2) A fair comparison requires matched latent distributions and matched nominal rate. (3) Under these conditions, VQ ≤ PQ ≤ SQ in optimal distortion. Proposition 3 shows a strict scaling separation for low-dimensional sources. Experiments use a frozen pretrained VAR encoder/decoder within the VQ-Transplant framework on ImageNet, FFHQ, CelebA-HQ, plus a pixel-space CelebA-HQ setting. Modern VQ variants (Wasserstein/MMD VQ) achieve the lowest E and, after decoder adaptation, the best r-FID.

Critical strengths

It introduces a clean, useful unifying perspective that re-anchors discrete visual tokenization in fixed-rate rate–distortion analysis, separating intrinsic quantizer effectiveness from confounders. The motivation is timely. The paper argues clearly that 100% utilization is not the same as good quantization, and it backs this with data: several PQ/SQ methods reach 100% usage with higher E than VQ. The experimental protocol (shared frozen encoder, matched T and K) is a sensible step toward isolating the quantizer from other tokenizer design choices.

Critical weaknesses

Condition 1 (matched latent distribution) does not appear to hold where E is measured. The distribution actually entering each quantizer, which is where E is presumably computed, is method-specific, and its scale is learned. By the paper's own propositions, a method whose projector shrinks feature scale would show lower E without being a better quantizer. Primary results center on reconstruction (PSNR/SSIM/LPIPS/r-FID/r-IS). Given tokenizers’ central role in AR generation, an end-to-end generative evaluation under matched tokenizers would greatly strengthen the conclusions.

Questions for authors

In which space exactly is E computed for each method (before or after the method-specific projector; for BSQ, before or after normalization)? Can you report a scale-normalized E? Does the VQ advantage persist under different latent encoders (e.g., diffusion-pretrained latents, self-supervised backbones) or under learned latent rescaling/whitening that equalizes per-dimension variances?

Reviewer_EAQoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a unified rate-distortion framework for comparing VQ, PQ, and SQ. It argues that quantization distortion is more fundamental than codebook utilization, establishes fairness criteria based on matched latent distributions and coding rates, and analyses their optimal distortion hierarchy. Experiments support the theoretical ordering and show a strong correlation between distortion and reconstruction quality.

Critical strengths
  • The paper provides a clear and unified theoretical formulation of VQ, PQ, and SQ, including finite-rate distortion bounds.
  • The experimental results extensively cover multiple quantization variants, datasets, and coding rates, showing the relationship between quantization distortion and reconstruction quality.
Critical weaknesses
  • The distortion-centric claim is not sufficiently justified. The theoretical analysis focuses on global optima but overlooks how low utilization and inactive codewords affect practical VQ optimization. Thus, it does not sufficiently establish why distortion minimization should take precedence over utilization-oriented strategies.
  • There is still potential bias in the correlation analysis. The reported correlations pool VQ, PQ, and SQ, despite their different utilization characteristics. Within the VQ variants in Table 2, utilization exhibits a stronger monotonic correlation with rFID than distortion, weakening the central empirical argument.
  • The correspondence between distortion and reconstruction fidelity is limited. Appendix I shows that reducing latent distortion does not necessarily improve rFID before decoder adaptation. This suggests that distortion alone is insufficient to characterize reconstruction quality, which also depends on decoder adaptation.
  • Matching nominal coding rates is theoretically valid but does not ensure comparable codebook resources. VQ uses 256 times more explicit codebook parameters than PQ for only a modest distortion improvement, limiting conclusions about practical reconstruction efficiency.
  • The experiments primarily compare 16-dimensional VQ with two 8-dimensional PQ groups. Given the established sensitivity of quantization to dimensionality, grouping, and codebook scaling, the conclusions may not generalize to other configurations.
Questions for authors
  • How does the distortion-centric principle account for the role of codebook utilization in practical VQ optimization? How do the authors interpret the VQ-only correlations in Table 2?

  • How should the reported VQ advantage be interpreted given the substantial difference in codebook parameters between VQ and PQ? To what extent do the conclusions generalize beyond the selected dimensionality and grouping configuration?

Reviewer_h4rVOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a unified framework to compare three different versions of quantization which are VQ, PQ, and SQ. The authors experimented the different methods using the pretrained VAR model proposed by Tian et. al. ("next-scale-prediction" paper) over different datasets of ImageNet, Celeba, FFHQ using the nominal code rate of ~65k. The paper concluded that VQ outperforms PQ and SQ under same code rate budget.

Critical strengths
  1. The paper include quite comprehensive background study.
Critical weaknesses

MAJOR

  1. Lack of contribution; The paper re-established what is found in the well-known Vector Quantization book of Gray [1] where it says unconstrained quantization (VQ) perform superior than constrained quantization methods of PQ and SQ. The authoes stated this finding as the third contribution of the paper. In addition, in the second contribution, it is quite obvious that the comparison of different quantization methods must be done under the same code rate considering the rate-distortion curve. So, this is not still a contribution.

  2. Lack of empirical evidence for the claims; from the paper "Under the conditions of Proposition 1, every global distortion minimizer fully utilizes the codebook, whereas full utilization does not imply distortion optimality." -> This can be empirically validated on some toy-sized datasets like Gaussian, and presented in the paper. I agree with the claim, but most of the claims in the paper are based on some assumptions (e.g., the VQ method is optimized by STE) and mathematically discussed not empirically.

  3. No qualitative results; There is no qualitative comparison of different quantization methods for none of the datasets neither in the main paper nor in the appendix.

  4. Inconsitency in the results in Table 2 I had difficulty to understand the experimental setup. As I understood, in substitution phase the aurhtors take the pretrained model (Encoder and Decoder) and quantized the latents with different quantization mehtods by training the codebook. Then, in adaptation phase, they freezed the learned codebook and Encoder and fine-tune the Decoder. Assumign this experimental setup, I have these comments.

    4.1 Strangely, the distortion and utilization remain the same for both substitution and adaptation phases for all VQ methods, while the metrics of SSIM, PSNR, LPIPS, etc. are different.

    4.2 What is the reson that PSNR and SSIM get worse from substitution phase to adaptation phase while LPIPS and r-FID improves. They should be coorelated to each other but they are not.

    4.3 The †\dagger symbol shows the experiment is done under the controlled latent-sapce setting. Why only VQ methods are experimendted under this setting, while it is expected all quantization methods to be evaluated under same setting, as authors claimed.

    4.4 Continuing on the previous comment, it is expected taht when fine-tuning the Decoder, the model perfroms better than the pre-trained one. But, it deteriorates PSNR and SSIM values and strangely keep the utilization and distortion fixed.

  5. No ablation on the code rate; The claim of the paper is that they provide a unified framework to compare different quantization methods under rate-distortion. So, there is no ablation of the rate as all experiments in the main paper and in the appendix is done using one fixed code rate K=65,536.

  6. Results are not statistically significant It seems that the experiments are done only by a single run, while considering the randomization in training and initialization, same method can end up in different finla results.

  7. Vague notations makes understanding difficult; there are many notations in the papert threader cannot find what they represent and some that the reader should go through different parts to find what it refers to. For example, Iϕ\mathcal{I}_{\phi}, E(f)\mathcal{E}(f), U(f)U(f), f∗f^*, E∗\mathcal{E}^*, supE∗\mathrm{sup} \mathcal{E}^*, and ...

MINOR

  1. from the paper "A longstanding optimization challenge in VQ is codebook collapse, where only a subset of the available code vectors is frequently selected.". It is not the correct definition of codebook collapse, as it means a subset of codes are not used at all, rather than being used less frequently.
Citations

[1] Gersho A, Gray RM. Vector quantization and signal compression. Springer Science & Business Media; 2012.

Questions for authors

Refer to Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.