- Rating
- 3(+1)
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a tokenizer that concatenates frozen VFM features with features learned by a complementary encoder for image reconstruction. Then, the paper trains a diffusion model on the resulting heterogeneous features for image generation.
Critical strengths
The proposed method is simple, and achieves a better trade-off between image reconstruction and generation (80-epoch training).
Critical weaknesses
- The paper lacks a clear discussion of prior work (though some is cited), such as [1-5], particularly how the proposed method differs from existing work, making its contribution claims less convincing.
- The evaluation is not comprehensive or convincing: (1) it lacks direct comparisons with related methods, such as [3, 6]; (2) most methods in Table 1 use smaller generators than the proposed method does, making the comparison less controlled; and (3) the text-to-image comparison includes only two related methods.
[1] Improving Reconstruction of Representation Autoencoder, arXiv:2602.08620.
[2] Latent Diffusion Model without Variational Autoencoder, ICLR 2026.
[3] Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing, ICML 2026.
[4] Boosting Generative Image Modeling via Joint Image-Feature Synthesis, NeurIPS 2025
[5] V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising, ECCV 2026.
[6] Improved Baselines with Representation Autoencoders, arXiv:2605.18324.
Questions for authors
-
As mentioned in Weakness 1, some related methods use VFMs to enhance tokenizers, while others employ joint diffusion training. Could the paper clarify the key differences and advantages of the proposed method relative to these approaches [1-5]?
-
The latent dimension of the tokenizer is important. Does the proposed tokenizer use larger latents than the compared methods? The number of parameters of the tokenizer is also important, so it would be better to report the parameter counts of different tokenizers in Table 1 if they are not the same. How can we ensure that the better trade-off comes from the claimed latent organization and diffusion training strategy, rather than larger tokenizers (if applicable) and generators?
5 comments
[W2/Q2] Effects of latent dimension and model capacity
-
Latent dimension. CoAE uses an 800-channel latent, its latent is larger than the 4 or 32 dimensions commonly used by VAE-based methods[1,2]. However, VFM-based methods generally start from pre-trained visual representations that are rich in semantic information. Due to their inherently high dimension, the latent dimensions of the associated tokenizers are correspondingly large. So, CoAE shares a comparable latent dimension with VFM-based tokenizers such as[3,4].
-
Tokenizer parameters. Our tokenizer has a 86M frozen DINOv2-B encoder, a 43M trainable detail encoder, and a 415M ViT-XL decoder. Tokenizer parameters are not commonly reported in the main comparison tables of related work. Following the reviewer’s helpful suggestion, we collected the encoder and decoder parameters of several representative autoencoders to provide additional context.
Tokenizer Latent dimension Encoder Params Decoder Params SD-VAE [1] 4 34.16M 49.49M VA-VAE [2] 32 28.41M 41.42M RAE [3] 768 86M 415M DRoRAE [4] 768 115M 415M CoAE 800 129M 415M VAE-based methods generally use relatively smaller tokenizers, whereas VFM-based methods typically contain larger pretrained visual encoders and higher-capacity decoders. Since RAE is our most directly related baseline, CoAE adopts the same frozen DINOv2-B encoder and ViT-XL decoder as RAE, with the only additional component being the 43M trainable detail encoder.
Table 4 verifies that the performance gain comes from the proposed latent organization and its corresponding diffusion-training strategy rather than increased model capacity.We additionally project the three homogeneous-fusion variants to the same 800 channels as Concat to control the latent dimension.
Tok. Params Organization Dimension PSNR rFID Gen. Params gFID IS 544M Additive 768 27.16 0.21 839M 5.15 158.18 544M Additive 800 27.20 0.21 839M 6.14 138.48 544M Projection 768 27.35 0.21 839M 5.78 145.50 544M Projection 800 27.21 0.22 839M 5.77 139.78 544M Modulation 768 27.27 0.21 839M 5.42 148.49 544M Modulation 800 27.31 0.20 839M 6.46 132.17 544M Concat 800 26.79 0.21 841M 2.16 207.08 Under matched tokenizer capacity and latent dimension, Concat still substantially outperforms the homogeneous-fusion variants.These results isolate the contributions of the distinct-subspace organization and its corresponding diffusion-training strategy.
-
Generator parameters. CoAE follows RAE and uses DiT$^{\mathrm{DH}}^{\mathrm{DH}}$ encoder blocks from 28 to 21, producing a 674.04M-parameter generator.
Generator Layers Params gFID IS CoAE full 28 841M 2.16 207.08 CoAE reduced 21 674M 2.40 200.90 With a comparable generator size, CoAE still demonstrates a substantial advantage in generation performance in Table1.
[1] High-resolution image synthesis with latent diffusion models.
[2] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models.
[3] Diffusion Transformers with Representation Autoencoders
[4] Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization.
Thank you again for your valuable comments. We have also provided a more detailed comparison with RAEv2 in (https://openreview.net/forum?id=lLKnXDciFk¬eId=4ASwExI34i). We would appreciate knowing whether our responses and revisions have addressed your concerns. We sincerely look forward to your feedback.
We sincerely thank you for the valuable insights and constructive feedback.
[W1/Q1] Relation to prior work
We have expanded the related work discussion in paper. They can be broadly divided into enhancing tokenizer [1–3,6] and joint-denoising [4,5].
Enhancing tokenizers These methods share the same general motivation as CoAE: supplementing the visual details missing from VFM representations. However, among these methods, CoAE is the only one that explicitly emphasizes preserving semantic and detail features in distinct latent subspaces.
LV-RAE [1] concatenates VFM and detail tokens before encoding, but subsequently combines the semantic and detail features through element-wise addition. Its final latent is therefore still a homogeneous fusion, similar to our Additive variant.
SVG [2] adopts a latent organization particularly similar to CoAE. However, SVG additionally requires distribution alignment, as reported in its Table 4, indicating that it still attempts to make the two feature distributions closer for diffusion modeling. In contrast, CoAE explicitly investigates alignment and demonstrates the advantage of keeping the two subspaces distinct.
PS-VAE [3] fine-tunes the VFM encoder for reconstruction while using semantic alignment to preserve the original semantics, thereby combining both types of information within a homogeneous latent.
Joint-denoising ReDi [4] and V-Co [5] instead focus on co-denoising during diffusion training. Their motivation is to introduce VFM features in the second stage and provide semantic priors for generation through joint denoising.
ReDi[4] applies PCA to reduce the VFM features to a channel dimension comparable to the VAE latent. This addresses an optimization imbalance similar to that targeted by our group-wise MSE, but the dimensionality reduction inevitably discards part of the original semantic information. CoAE instead preserves the complete VFM representation and directly balances the heterogeneous subspaces during training.
V-Co[5] addresses co-denoising through a specialized dual-stream DiT architecture that strengthens cross-stream interaction, whereas CoAE requires no complex backbone modification and can be applied more broadly.
In contrast to these joint-denoising methods, CoAE starts from a tokenizer-design question: how to introduce the missing visual details into a VFM latent without compromising its original semantic information. By explicitly preserving both semantic and detail subspaces in the tokenizer, CoAE naturally obtains the benefits of co-denoising in Stage 2, while achieving them through a simpler and more direct formulation without specialized backbone modifications.
As discussed in our paper, co-denoising is an important reason for CoAE’s strong generation performance, demonstrating the advantage of an independent semantic subspace during generation. Meanwhile, preserving this semantic subspace in the tokenizer is also important.
[W2] Comprehensive evaluation
We have added RAEv2 [6] to the ImageNet class-conditional comparison in Table 1. PS-VAE [3] primarily evaluates text-to-image generation and editing and does not report comparable ImageNet class-conditional results. Our text-to-image experiment is designed primarily as a controlled study of whether the proposed latent organization and diffusion adaptation remains effective when transferred to T2I. We therefore compare four representative latent organizations:
This controlled setup is designed to isolate the effect of latent organization rather than compare with state-of-the-art T2I models.
[1] Improving Reconstruction of Representation Autoencoder, arXiv:2602.08620.
[2] Latent Diffusion Model without Variational Autoencoder, ICLR 2026.
[3] Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing, ICML 2026.
[4] Boosting Generative Image Modeling via Joint Image-Feature Synthesis, NeurIPS 2025
[5] V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising, ECCV 2026.
[6] Improved Baselines with Representation Autoencoders, arXiv:2605.18324.