- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce a new method to train a VAE with discrete latents, by anchoring their quantization on an existing VAE (SD3). They use the pre-trained frozen encoder followed by ViT blocks to adapt the continuous latent to a quantized latent . A symetric set of ViT blocks recovers from , which is then decoded to pixels by a fine-tuned decoded. Taken together, those components ensure a fixed latent space anchor (frozen encoder) preserving the shape of latent space, and a reconstruction+GAN fine-tuned decoder ensuring that decoded images preserve their quality following the drif from reconstructed latents. LABQ achieves state-of-the-art reconstruction results for quantized-VAEs, as well as generation for ImageNet 128x128.
Critical strengths
The paper provides clear justification and explanation for the method, with results surpassing existing works on reconstruction metrics, while benefiting from fast convergence by using a pre-trained VAE. Application on imagenet generation yields similar results compared to VQ and BSQ.
Critical weaknesses
The central weakness of the paper is the limited experiments on generation. While the paper shows clear improvements in reconstruction, literature shows that they do not always follow generative results. To consolidate the claims, the paper would require at least a comparison on ImagetNet 256x256 against recent works (including CODA and -SQ, whose checkpoints are available).
Questions for authors
- Could the author provide a comparison on ImagetNet 256x256 against recent discrete tokenizers?
- Does the method generalize to other auto-encoders or representation auto-encoders (as in RAE)?