acceptodds
Under review as a conference paper at ICLR 2027

Visual Tokenization with Latent Anchored Binary Quantization

Abstract

Discrete visual tokenizers are bottlenecked by both codebook scalability and quantization error. While lookup-free and finite scalar quantization improve scalability, they forgo the adaptive codebook updates used in vector quantization to reduce quantization error. This mismatch leads to degraded reconstruction and slower convergence in training. To address this, we revisit the two-latent variational formulation that interprets codebook learning as continuous-latent recovery within the ELBO. This interpretation motivates a learned recovery map for binary codes, however direct realization of this objective leads to collapsed representation when the map and its target latents are jointly optimized. We therefore introduce latent-anchored binary quantization (LABQ), a tokenizer that learns binary codes and a recovery map against the fixed representations of a pretrained VAE encoder while fine-tuning its decoder. LABQ trains in 30 epochs, using approximately 6.7x fewer tokenizer-training image exposures than BSQ based on the reported epoch budgets. Without adversarial supervision, some blurring and color-shift artifacts remain. We systematically compare PatchGAN, DINO, and pretrained VAE encoders as discriminators, finding that the SD3 VAE encoder, already attuned to the target latent geometry, yields the lowest rFID under the same ten-epoch adversarial fine-tuning schedule. With this design, LABQ achieves competitive results in discrete image reconstruction on ImageNet (rFID 0.29, LPIPS 0.038) and COCO (rFID 2.84, LPIPS 0.037), approaching its continuous VAE reference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Tokenizer, Image tokenization, Image Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dmZ3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce a new method to train a VAE with discrete latents, by anchoring their quantization on an existing VAE (SD3). They use the pre-trained frozen encoder followed by ViT blocks to adapt the continuous latent zz to a quantized latent zqz_q. A symetric set of ViT blocks recovers zz from zqz_q, which is then decoded to pixels by a fine-tuned decoded. Taken together, those components ensure a fixed latent space anchor (frozen encoder) preserving the shape of latent space, and a reconstruction+GAN fine-tuned decoder ensuring that decoded images preserve their quality following the drif from reconstructed latents. LABQ achieves state-of-the-art reconstruction results for quantized-VAEs, as well as generation for ImageNet 128x128.

Critical strengths

The paper provides clear justification and explanation for the method, with results surpassing existing works on reconstruction metrics, while benefiting from fast convergence by using a pre-trained VAE. Application on imagenet generation yields similar results compared to VQ and BSQ.

Critical weaknesses

The central weakness of the paper is the limited experiments on generation. While the paper shows clear improvements in reconstruction, literature shows that they do not always follow generative results. To consolidate the claims, the paper would require at least a comparison on ImagetNet 256x256 against recent works (including CODA and Λ24\Lambda_{24}-SQ, whose checkpoints are available).

Questions for authors
  • Could the author provide a comparison on ImagetNet 256x256 against recent discrete tokenizers?
  • Does the method generalize to other auto-encoders or representation auto-encoders (as in RAE)?
Reviewer_tVfROpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
1(−1)
Soundness
1(−1)
Read review
Paper summary

The paper proposes Latent-Anchored Binary Quantization (LABQ), a discrete visual tokenizer built on a two-latent variational framework. The core idea is to learn binary codes alongside a recovery map that reconstructs the latents of a frozen pretrained VAE, with only the decoder being fine-tuned. On ImageNet and COCO, LABQ outperforms existing methods in reconstruction quality at high bitrates.

Critical strengths
  1. The paper is clearly written and easy to follow.

  2. LABQ achieves strong high-bitrate reconstruction results on both ImageNet and COCO.

  3. The discriminator comparison provides a useful empirical finding: the pretrained SD3 VAE encoder achieves lower rFID than the tested PatchGAN and DINO alternatives under the same ten-epoch adversarial fine-tuning schedule.

Critical weaknesses
  1. The paper contribution is minor. The second claimed contribution, namely discrete tokenization anchored to a frozen pretrained VAE, has already been investigated in VQ-Transplant [1]. This component should therefore be acknowledged as prior work rather than listed as a standalone contribution.

  2. The rationale for using normalized quantization error is unclear. Unnormalized quantization error can have a direct link to STE gradient discrepancy [2]. However, the relationship with normalized quantization error is unclear, the reported 8.4×8.4\times improvement in Figure 1(b) is also unclear.

  3. The connection between latent recovery error and STE gradient bias remains unclear. The recovery error measures the discrepancy after the recovery network, whereas STE operates at the sign quantizer. Lower recovery error does not imply lower gradient bias, and faster convergence is insufficient evidence.

  4. The ablations do not isolate the proposed design. Comparisons with other tokenizers vary the backbone, training procedure, and discriminator simultaneously. Controlled comparisons with VQ, FSQ, and BSQ within the same frozen SD3 framework are needed to establish what LABQ contributes beyond compressing pretrained VAE latents.

  5. The variational analysis offers limited guidance for the method. It does not establish why freezing the target is necessary, and the ELBO-derived alignment loss brings no measurable benefit in Table 4: removing it leaves rFID unchanged and slightly improves LPIPS and PSNR. The authors describe this loss as a safeguard against latent drift, but provide no evidence of such drift when it is removed.

  6. The reconstruction gains are concentrated at high bitrates and do not consistently translate to generation. Increasing the code width worsens gFID despite better reconstruction.

[1] VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers [2] A Unified Rate-Distortion Perspective on Vector, Product, and Scalar Quantization

Questions for authors
  1. Why not report unnormalized quantization error in Figure 1(b), given its connection to STE gradient discrepancy [2]? Although this error depends on the encoder latent distribution, could all methods be evaluated using the same fixed encoder latents for a fair comparison?

  2. What are the technical differences between LABQ and VQ-Transplant [1], particularly regarding discrete tokenization over frozen pretrained VAE representations?

  3. Can the authors establish a relationship between the recovery error measured after the recovery network and the STE gradient bias at the sign quantizer?

  4. How do VQ, FSQ, and BSQ perform within the same frozen SD3 framework, using matched architectures, losses, and training budgets? Does LABQ retain its reconstruction and convergence advantages?

Reviewer_VxbEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LABQ builds a binary image tokenizer around the frozen latent representation of a pretrained SD3 VAE. Small Transformer modules map continuous latents to binary codes and recover the continuous representation, while the VAE decoder is fine-tuned. A two-latent variational formulation motivates latent recovery, and adversarial refinement uses a trainable copy of the SD3 encoder as discriminator. The strongest evidence concerns high-rate reconstruction: at approximately 0.5625 nominal bpp, the reported ImageNet rFID improves from BSQ-36's 0.41 to 0.29, with improvements also on COCO. Controlled non-adversarial experiments investigate convergence and target stability. A smaller generation study uses a MaskGIT-style model on ImageNet at 128 x 128.

Critical strengths
  1. The high-rate reconstruction contribution is concrete and useful. At the matched nominal rate in Table 1, LABQ improves ImageNet LPIPS/rFID from 0.043/0.41 to 0.038/0.29 and COCO LPIPS/rFID from 0.041/3.42 to 0.037/2.84 relative to BSQ-36. The gains are perceptual rather than uniform across all metrics, but their consistency across the two datasets supports the proposed operating point. Table 3 further shows that the SD3 discriminator improves rFID over PatchGAN under the reported two-phase protocol (0.29 versus 0.4122).

  2. Using a fixed, informative latent target is a coherent and experimentally supported way to stabilize quantizer training. The controlled convergence comparison and Appendix D go beyond the final reconstruction table and examine frozen versus jointly trained targets. Appendix B clearly defines the 25.6-fold step ratio to a common training-loss threshold and distinguishes it from the 6.7-fold cross-paper image-exposure ratio. I consider the reconstruction and optimization results sufficient to support a weak acceptance, without relying on a claim of equivalent end-to-end compute savings.

Critical weaknesses
  1. The mechanistic and component-attribution claims are stronger than the evidence. Appendix B explicitly says that STE gradient bias is not directly measured; the 8.4-fold reduction concerns normalized latent-interface error, measured in each model's own latent space. This and faster convergence do not by themselves establish the claimed reduction of gradient bias in Section 4.1. Likewise, Table 4 is cumulative: alignment is removed only after the SD3 discriminator is removed, and pretrained initialization is tested only after the encoder is unfrozen. These comparisons do not establish the alignment term's effect in the full model or justify Section 4.4's broad conclusion that pretrained initialization is not responsible for the gains. This primarily limits the explanatory claims; it does not invalidate the reconstruction measurements.

  2. The generation comparison is less reproducible and less controlled than the reconstruction study. Section 4.2 reports 18-bit gFID 4.98 versus the cited BSQ-18 result of 5.44, while the 36-bit variant gives 6.64. Appendix E explains the grouped prediction head but does not fully specify generator size, optimization/training budget, or the grouping used for the 18-bit model. Consequently, the improvement over the cited BSQ result cannot be confidently attributed to the tokenizer alone. I recognize that generation is secondary and that the paper already acknowledges the 36-bit tradeoff and low-rate limitations; this is a request to delimit and document the existing comparison, rather than to establish broad generation superiority.

Questions for authors
  1. Could you clarify what the current evidence establishes about the mechanism and pretrained components? In particular, can you report the effect of removing alignment from the full adversarial model, and either provide evidence for the claimed STE-bias reduction and pretrained-initialization attribution or narrow those statements to the measured latent distortion, convergence, and conditional ablation results? A narrower interpretation would address much of this concern.

  2. Please specify the generator architecture/size, optimization and training budget, and code grouping for the 18-bit and 36-bit models, and identify which settings match the cited BSQ-18 experiment. Is the reported 18-bit improvement supported by a matched generator comparison, or should it be presented as a comparison between published systems with potentially different training configurations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.