- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a unified rate–distortion framework to analyze and fairly compare three dominant families of discrete visual tokenizers—vector quantization (VQ), product quantization (PQ), and scalar quantization (SQ). It makes three claims. (1) Distortion, not codebook utilization, is the primary intrinsic objective. This is supported by Proposition 1, a local Taylor bound linking distortion to the STE gradient gap, and Spearman correlations between E and r-FID. (2) A fair comparison requires matched latent distributions and matched nominal rate. (3) Under these conditions, VQ ≤ PQ ≤ SQ in optimal distortion. Proposition 3 shows a strict scaling separation for low-dimensional sources. Experiments use a frozen pretrained VAR encoder/decoder within the VQ-Transplant framework on ImageNet, FFHQ, CelebA-HQ, plus a pixel-space CelebA-HQ setting. Modern VQ variants (Wasserstein/MMD VQ) achieve the lowest E and, after decoder adaptation, the best r-FID.
Critical strengths
It introduces a clean, useful unifying perspective that re-anchors discrete visual tokenization in fixed-rate rate–distortion analysis, separating intrinsic quantizer effectiveness from confounders. The motivation is timely. The paper argues clearly that 100% utilization is not the same as good quantization, and it backs this with data: several PQ/SQ methods reach 100% usage with higher E than VQ. The experimental protocol (shared frozen encoder, matched T and K) is a sensible step toward isolating the quantizer from other tokenizer design choices.
Critical weaknesses
Condition 1 (matched latent distribution) does not appear to hold where E is measured. The distribution actually entering each quantizer, which is where E is presumably computed, is method-specific, and its scale is learned. By the paper's own propositions, a method whose projector shrinks feature scale would show lower E without being a better quantizer. Primary results center on reconstruction (PSNR/SSIM/LPIPS/r-FID/r-IS). Given tokenizers’ central role in AR generation, an end-to-end generative evaluation under matched tokenizers would greatly strengthen the conclusions.
Questions for authors
In which space exactly is E computed for each method (before or after the method-specific projector; for BSQ, before or after normalization)? Can you report a scale-normalized E? Does the VQ advantage persist under different latent encoders (e.g., diffusion-pretrained latents, self-supervised backbones) or under learned latent rescaling/whitening that equalizes per-dimension variances?