acceptodds
Under review as a conference paper at ICLR 2027

CoAE: A Visual Tokenizer with Complementary Latent Subspaces for Image Generation

Abstract

Vision foundation models (VFMs) have emerged as effective encoders for image generation due to their rich semantic information. However, VFM features often lack low-level details, leading to inferior reconstruction quality. Existing methods attempt to address this issue by fusing semantic and detail information into a homogeneous latent space. This forces the two types of information to compete within the same representation, leading to a reconstruction–generation tradeoff. In this work, we propose Complementary AutoEncoder (CoAE), which organizes the original VFM features and detail features as distinct subspaces within a heterogeneous latent space, allowing them to preserve their own structures and serve complementary roles. We further introduce a diffusion training strategy tailored to this latent organization. Ablation studies show that homogeneous latent designs rely on alignment to improve generation quality at the cost of reconstruction, while CoAE performs best without alignment. On class-conditional ImageNet , CoAE achieves strong performance in both reconstruction and generation, with an rFID of 0.21 and a gFID of 2.16 without guidance after only 80 epochs. CoAE also transfers effectively to text-to-image generation and outperforms other latent spaces.

Primary area:
generative models
Keywords:
Diffusion, Image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_6BXCOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a tokenizer that concatenates frozen VFM features with features learned by a complementary encoder for image reconstruction. Then, the paper trains a diffusion model on the resulting heterogeneous features for image generation.

Critical strengths

The proposed method is simple, and achieves a better trade-off between image reconstruction and generation (80-epoch training).

Critical weaknesses
  • The paper lacks a clear discussion of prior work (though some is cited), such as [1-5], particularly how the proposed method differs from existing work, making its contribution claims less convincing.
  • The evaluation is not comprehensive or convincing: (1) it lacks direct comparisons with related methods, such as [3, 6]; (2) most methods in Table 1 use smaller generators than the proposed method does, making the comparison less controlled; and (3) the text-to-image comparison includes only two related methods.

[1] Improving Reconstruction of Representation Autoencoder, arXiv:2602.08620.

[2] Latent Diffusion Model without Variational Autoencoder, ICLR 2026.

[3] Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing, ICML 2026.

[4] Boosting Generative Image Modeling via Joint Image-Feature Synthesis, NeurIPS 2025

[5] V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising, ECCV 2026.

[6] Improved Baselines with Representation Autoencoders, arXiv:2605.18324.

Questions for authors
  • As mentioned in Weakness 1, some related methods use VFMs to enhance tokenizers, while others employ joint diffusion training. Could the paper clarify the key differences and advantages of the proposed method relative to these approaches [1-5]?

  • The latent dimension of the tokenizer is important. Does the proposed tokenizer use larger latents than the compared methods? The number of parameters of the tokenizer is also important, so it would be better to report the parameter counts of different tokenizers in Table 1 if they are not the same. How can we ensure that the better trade-off comes from the claimed latent organization and diffusion training strategy, rather than larger tokenizers (if applicable) and generators?

5 comments

We sincerely thank you for the valuable insights and constructive feedback.

[W1/Q1] Relation to prior work

We have expanded the related work discussion in paper. They can be broadly divided into enhancing tokenizer [1–3,6] and joint-denoising [4,5].

Enhancing tokenizers These methods share the same general motivation as CoAE: supplementing the visual details missing from VFM representations. However, among these methods, CoAE is the only one that explicitly emphasizes preserving semantic and detail features in distinct latent subspaces.

  • LV-RAE [1] concatenates VFM and detail tokens before encoding, but subsequently combines the semantic and detail features through element-wise addition. Its final latent is therefore still a homogeneous fusion, similar to our Additive variant.

  • SVG [2] adopts a latent organization particularly similar to CoAE. However, SVG additionally requires distribution alignment, as reported in its Table 4, indicating that it still attempts to make the two feature distributions closer for diffusion modeling. In contrast, CoAE explicitly investigates alignment and demonstrates the advantage of keeping the two subspaces distinct.

  • PS-VAE [3] fine-tunes the VFM encoder for reconstruction while using semantic alignment to preserve the original semantics, thereby combining both types of information within a homogeneous latent.

Joint-denoising ReDi [4] and V-Co [5] instead focus on co-denoising during diffusion training. Their motivation is to introduce VFM features in the second stage and provide semantic priors for generation through joint denoising.

  • ReDi[4] applies PCA to reduce the VFM features to a channel dimension comparable to the VAE latent. This addresses an optimization imbalance similar to that targeted by our group-wise MSE, but the dimensionality reduction inevitably discards part of the original semantic information. CoAE instead preserves the complete VFM representation and directly balances the heterogeneous subspaces during training.

  • V-Co[5] addresses co-denoising through a specialized dual-stream DiT architecture that strengthens cross-stream interaction, whereas CoAE requires no complex backbone modification and can be applied more broadly.

In contrast to these joint-denoising methods, CoAE starts from a tokenizer-design question: how to introduce the missing visual details into a VFM latent without compromising its original semantic information. By explicitly preserving both semantic and detail subspaces in the tokenizer, CoAE naturally obtains the benefits of co-denoising in Stage 2, while achieving them through a simpler and more direct formulation without specialized backbone modifications.

As discussed in our paper, co-denoising is an important reason for CoAE’s strong generation performance, demonstrating the advantage of an independent semantic subspace during generation. Meanwhile, preserving this semantic subspace in the tokenizer is also important.

  • Table 4 shows that introducing semantic features in Stage 1 tokenizer get a better rFID and gFID(from 2.54 to 2.16) meanwhile.
  • Our new analysis in Appendix Figure 8 further visualizes the contribution of each subspace to image decoding: semantic features primarily maintain the global structure and spatial coherence, whereas detail features complement the missing textures. This complementary division leads to a better reconstruction–generation performance.
  • Retaining the complete VFM representation provides the latent space with stronger semantic capabilities and greater potential for unified understanding and generation.

[W2] Comprehensive evaluation

We have added RAEv2 [6] to the ImageNet class-conditional comparison in Table 1. PS-VAE [3] primarily evaluates text-to-image generation and editing and does not report comparable ImageNet class-conditional results. Our text-to-image experiment is designed primarily as a controlled study of whether the proposed latent organization and diffusion adaptation remains effective when transferred to T2I. We therefore compare four representative latent organizations:

  • FLUX VAE represents a compact VAE latent space;
  • RAE represents a semantic VFM latent space;
  • Additive represents homogeneous fusion;
  • CoAE represents distinct semantic and detail subspaces.

This controlled setup is designed to isolate the effect of latent organization rather than compare with state-of-the-art T2I models.

[1] Improving Reconstruction of Representation Autoencoder, arXiv:2602.08620.

[2] Latent Diffusion Model without Variational Autoencoder, ICLR 2026.

[3] Both Semantics and Reconstruction Matter: Making Representation Encoders Ready for Text-to-Image Generation and Editing, ICML 2026.

[4] Boosting Generative Image Modeling via Joint Image-Feature Synthesis, NeurIPS 2025

[5] V-Co: A Closer Look at Visual Representation Alignment via Co-Denoising, ECCV 2026.

[6] Improved Baselines with Representation Autoencoders, arXiv:2605.18324.

[W2/Q2] Effects of latent dimension and model capacity

  • Latent dimension. CoAE uses an 800-channel latent, its latent is larger than the 4 or 32 dimensions commonly used by VAE-based methods[1,2]. However, VFM-based methods generally start from pre-trained visual representations that are rich in semantic information. Due to their inherently high dimension, the latent dimensions of the associated tokenizers are correspondingly large. So, CoAE shares a comparable latent dimension with VFM-based tokenizers such as[3,4].

  • Tokenizer parameters. Our tokenizer has a 86M frozen DINOv2-B encoder, a 43M trainable detail encoder, and a 415M ViT-XL decoder. Tokenizer parameters are not commonly reported in the main comparison tables of related work. Following the reviewer’s helpful suggestion, we collected the encoder and decoder parameters of several representative autoencoders to provide additional context.

    TokenizerLatent dimensionEncoder ParamsDecoder Params
    SD-VAE [1]434.16M49.49M
    VA-VAE [2]3228.41M41.42M
    RAE [3]76886M415M
    DRoRAE [4]768115M415M
    CoAE800129M415M

    VAE-based methods generally use relatively smaller tokenizers, whereas VFM-based methods typically contain larger pretrained visual encoders and higher-capacity decoders. Since RAE is our most directly related baseline, CoAE adopts the same frozen DINOv2-B encoder and ViT-XL decoder as RAE, with the only additional component being the 43M trainable detail encoder.

    Table 4 verifies that the performance gain comes from the proposed latent organization and its corresponding diffusion-training strategy rather than increased model capacity.We additionally project the three homogeneous-fusion variants to the same 800 channels as Concat to control the latent dimension.

    Tok. ParamsOrganizationDimensionPSNRrFIDGen. ParamsgFIDIS
    544MAdditive76827.160.21839M5.15158.18
    544MAdditive80027.200.21839M6.14138.48
    544MProjection76827.350.21839M5.78145.50
    544MProjection80027.210.22839M5.77139.78
    544MModulation76827.270.21839M5.42148.49
    544MModulation80027.310.20839M6.46132.17
    544MConcat80026.790.21841M2.16207.08

    Under matched tokenizer capacity and latent dimension, Concat still substantially outperforms the homogeneous-fusion variants.These results isolate the contributions of the distinct-subspace organization and its corresponding diffusion-training strategy.

  • Generator parameters. CoAE follows RAE and uses DiT$^{\mathrm{DH}}−XL,resultinginan841M−parametergenerator,whereasmanyVAE−basedmethodsinTable1usea675M−parameterDiT.Foraparameter−matchedcomparison,wereducethenumberofsharedDiT-XL, resulting in an 841M-parameter generator, whereas many VAE-based methods in Table 1 use a 675M-parameter DiT. For a parameter-matched comparison, we reduce the number of shared DiT^{\mathrm{DH}}$ encoder blocks from 28 to 21, producing a 674.04M-parameter generator.

    GeneratorLayersParamsgFIDIS
    CoAE full28841M2.16207.08
    CoAE reduced21674M2.40200.90

    With a comparable generator size, CoAE still demonstrates a substantial advantage in generation performance in Table1.

[1] High-resolution image synthesis with latent diffusion models.

[2] Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models.

[3] Diffusion Transformers with Representation Autoencoders

[4] Beyond the Last Layer: Multi-Layer Representation Fusion for Visual Tokenization.

Thank you again for your valuable comments. We have also provided a more detailed comparison with RAEv2 in (https://openreview.net/forum?id=lLKnXDciFk&noteId=4ASwExI34i). We would appreciate knowing whether our responses and revisions have addressed your concerns. We sincerely look forward to your feedback.

Reviewer_mq9FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoAE, a visual tokenizer that concatenates frozen VFM features with learned detail features, and trains a diffusion model on the joint latent using a group-wise loss and separate prediction heads. The motivation is reasonable: improving reconstruction can alter a semantic representation that is already effective for generation. On ImageNet-256, CoAE substantially improves reconstruction over the RAE baseline while retaining comparable generation quality.

Critical strengths
  • The reconstruction–generation trade-off is an important problem, and preserving the original VFM features is a simple, plausible design.

  • The paper evaluates several fusion alternatives and includes ablations of the diffusion objective and detail dimension. The reconstruction improvement over RAE is substantial.

Critical weaknesses
  • The motivation and conceptual contribution need clearer positioning. The idea that semantic and pixel-level information may benefit from distinct representations is related to prior work such as TokenFlow[1]. The paper should discuss this connection and clarify the specific insight contributed by CoAE in the continuous VFM and diffusion setting.

  • The method presentation is harder to follow than necessary. Section 3.1 presents Homogeneous Fusion and Distinct Subspaces together without clearly separating the alternatives studied from the final CoAE design. The paper would benefit from a more explicit account of why concatenation was selected.

  • The latent-organization comparison does not isolate the effect of concatenation. In Table 3, the fusion variants use a 768-channel latent and standard diffusion training, whereas Concat uses an 800-channel latent and the proposed training adaptation. Thus, their gFID differences cannot be attributed solely to the organization of the features.

  • The proposed explanation of complementarity lacks direct evidence. At 80 epochs, CoAE improves rFID from 0.57 to 0.21 over RAE, but both obtain a gFID of 2.16. This establishes a better reconstruction–generation trade-off, but does not show that interactions between the two feature groups improve generation. More strikingly, Table 5 reports that standard MSE on the concatenated latent gives a gFID of 16.75, versus 2.51 with group-wise MSE. The paper does not show which feature group fails under standard training or how its prediction error affects decoded images. Table 6 varies the detail dimension, but does not explain why 32 channels is preferred or why gFID worsens at 64.

[1] TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation

Questions for authors
  1. Can the fusion and concatenation variants be compared with better matched latent capacity and diffusion training, to isolate the effect of latent organization?

  2. Under standard versus group-wise MSE, what are the per-group normalized prediction errors across diffusion timesteps? Can the authors show how errors in each group affect the decoded images?

  3. Can the authors provide quantitative and visual analyses of what the semantic and detail features each contribute? How does detail dimensionality interact with the group-wise loss, and what evidence supports complementarity beyond improved reconstruction at unchanged gFID?

6 comments

We sincerely thank you for the valuable insights and constructive feedback.

[W1] Relation to TokenFlow

We thank the reviewer for pointing out the connection to TokenFlow[1] and the need for clearer positioning. Both TokenFlow and CoAE recognize that semantic features and pixel-level details are useful for different tasks. TokenFlow regards pixel-level details as essential for image synthesis, including reconstruction and generation, while semantic features mainly benefit multimodal understanding. It therefore starts from discrete VQ tokens, which are effective for synthesis but insufficient for multimodal understanding, and introduces CLIP semantic features to build a unified tokenizer for understanding and synthesis. In contrast, CoAE focuses on the reconstruction–generation trade-off: detail features primarily improve reconstruction, whereas semantic VFM features facilitate generation. We have added TokenFlow to the Related Work section and clarified this distinction.

MethodMain goalSemantic featuresDetail featuresLatent organization
TokenFlowUnify understanding and synthesisEnable multimodal understandingPreserve image synthesis qualityDiscrete codebooks
CoAEBalance reconstruction and generationImprove generationImprove reconstructionDistinct continuous subspaces

[W2] Method presentation

In addition, we revised Sec. 3.1 to clearly distinguish the homogeneous-fusion alternatives from the final CoAE design. We further explain that homogeneous fusion may compromise the original information and cause a clear trade-off, whereas distinct subspaces preserve the complete information.

[W3/Q1] Matched comparison

In the ablation study of Table 3, the homogeneous-fusion variants use 768-channel latents, whereas Concat uses an 800-channel latent. We agree that this introduces latent capacity as an additional variable beyond latent organization. We therefore add a linear projection to map each of the three fused latents to the same 800 channels and conduct additional experiments.

OrganizationChannelsPSNRrFIDgFIDIS
Additive76827.160.215.15158.18
Additive80027.200.216.14138.48
Projection76827.350.215.78145.50
Projection80027.210.225.77139.78
Modulation76827.270.215.42148.49
Modulation80027.310.206.46132.17
Concat80026.790.212.16207.08

With the same latent dimensionality, Concat still shows a clear advantage over the three fusion strategies. Increasing the fused latents from 768 to 800 channels generally results in slightly worse generation performance, possibly because the additional projection further degrades the semantic information. Nevertheless, the difference between 768 and 800 channels is relatively small, since the latent input is immediately projected to the same hidden size in the DiT architecture.

For diffusion training, adapted diffusion strategy is necessary for Concat to realize the benefit of its distinct-subspace organization. Even a simple group-wise MSE is sufficient to address the optimization imbalance between the two subspaces, as demonstrated by the ablation study in Table 5. Therefore, our results support the advantage of latent organization when it is paired with the diffusion adaptation.

[1] TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation

[W4/Q2–Q3] Optimization balance and detail dimensionality

We first analyze the per-subspace prediction errors across diffusion timesteps using the default detail dimension Cd=32C_d=32. We evaluate the Standard-MSE and Group-wise-MSE checkpoints on the same 5,000 ImageNet validation images using identical noise and timesteps. For each subspace, we calculate

\frac{\sum\lVert\hat{\mathbf v}_g(t)-\mathbf v_g(t)\rVert_2^2} {\sum\lVert\mathbf v_g(t)\rVert_2^2}, \quad g\in\{\mathrm{semantic},\mathrm{detail}\}.$$ | Timestep | Standard: Semantic | Standard: Detail | Group-wise: Semantic | Group-wise: Detail | |:---:|:---:|:---:|:---:|:---:| | 0.05 | 0.5225 | 0.5775 | 0.5336 | 0.5097 | | 0.10 | 0.4458 | 0.6015 | 0.4612 | 0.5241 | | 0.20 | 0.2878 | 0.6567 | 0.3090 | 0.5558 | | 0.30 | 0.1956 | 0.6961 | 0.2141 | 0.5672 | | 0.40 | 0.1468 | 0.7107 | 0.1615 | 0.5666 | | 0.50 | 0.1200 | 0.6958 | 0.1316 | 0.5588 | | 0.60 | 0.1060 | 0.6548 | 0.1153 | 0.5430 | | 0.70 | 0.1023 | 0.5979 | 0.1096 | 0.5196 | | 0.80 | 0.1130 | 0.5394 | 0.1184 | 0.4942 | | 0.90 | 0.1638 | 0.4997 | 0.1671 | 0.4836 | | 0.95 | 0.2340 | 0.4969 | 0.2365 | 0.4921 | | Average | 0.2216 | 0.6115 | 0.2325 | 0.5286 | Under Standard MSE, the detail subspace consistently exhibits a substantially higher NMSE than the semantic subspace. Group-wise MSE reduces the average detail NMSE from 0.6115 to 0.5286, while the semantic NMSE changes only from 0.2216 to 0.2325. This confirms that the detail subspace is severely under-optimized under Standard MSE. We further examine how this imbalance changes with the detail dimension. The following table reports the NMSE averaged across all evaluated timesteps. | Detail dim. | Loss | Semantic NMSE | Detail NMSE | |:---:|:---|:---:|:---:| | 16 | Standard | 0.2223 | 0.6428 | | 16 | Group-wise | 0.2307 | 0.5344 | | 32 | Standard | 0.2216 | 0.6115 | | 32 | Group-wise | 0.2325 | 0.5286 | | 64 | Standard | 0.2216 | 0.4872 | | | 64 | Group-wise | 0.2302 | 0.4112 | | Across all detail dimensions, Group-wise MSE changes the semantic NMSE only slightly while consistently reducing the detail NMSE. Moreover, the absolute reduction becomes larger as the dimensional gap between the 768-channel semantic subspace and the detail subspace increases: 0.0760, 0.0829, and 0.1083 for 64, 32, and 16 detail channels. This confirms that a lower-dimensional detail subspace is more severely under-optimized by Standard MSE and therefore benefits more from group-wise balancing. We have added the complete timestep-wise results for all three detail dimensions to Figure 8 and the corresponding analysis to Appendix C.3 of the revised PDF. Regarding why we select $C_d=32$ and why gFID worsens at $C_d=64$, this is an expected result of the reconstruction–generation trade-off. This observation is consistent with VA-VAE[1].During Stage 1, a higher-dimensional detail latent preserves more local and high-frequency information, improving reconstruction. During Stage 2, however, the diffusion model must learn a higher-dimensional latent containing more complex details, which increases its modeling burden. **[W5/Q2–Q3] Subspace contributions and complementarity** To answer how prediction errors in each group affect the decoded images, we conduct a controlled subspace-perturbation experiment. Starting from normalized tokenizer latents, we independently add Gaussian noise $\sigma\boldsymbol{\epsilon}$ to either the semantic or detail subspace while keeping the other subspace unchanged, and then decode the perturbed latent using the same tokenizer decoder. We evaluate $\sigma\in\{0.5,1,2,4\}$ across different noise levels to show the progressive effect of each perturbation. As shown in Figure 8, perturbing the detail subspace primarily degrades textures , while the global structure remains largely coherent under strong perturbations. Perturbing the semantic subspace produces clear patch artifacts and structural inconsistencies. This indicates that the semantic subspace mainly controls global structure and patch consistency, whereas the detail subspace primarily preserves local details. The different degradation patterns provide visual evidence that the two subspaces encode distinct and complementary information. The detailed experimental setup has been added to Appendix C.4 of the revised pdf. We explain the complementarity from three aspects. - **Information level:** The spatial PCA and t-SNE analyses in Figure 9 of the revised pdf show that they contain different information. - **Functional level:** The perturbation experiment shows that, during image decoding, the semantic subspace controls global structure and spatial coherence, the detail subspace controls local textures. - **Task level:** We achieve strong rFID and gFID. These results demonstrate that the two subspaces serve complementary roles. [1]Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models
Reviewer_mq9FOpenReview ↗

Response of Author‘s rebuttal

Thank you for the rebuttal and the additional analyses. The per-subspace prediction errors and perturbation experiments clarify the role of group-wise MSE and provide useful evidence that the two feature groups serve different functions. However, I remain unconvinced that CoAE offers a sufficiently clear advantage over the simpler RAEv2 approach. At 80 epochs, its reported reconstruction–generation results fall between the two RAEv2 configurations in Table 1, while CoAE requires an additional detail encoder and a specialized diffusion objective. The necessity of preserving all 768 semantic channels also remains unexamined. I therefore maintain my score of 2.

Reviewer_pnToOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoAE, a visual tokenizer that concatenates semantic features from a frozen vision foundation model with low-dimensional detail features. It employs a group-wise MSE loss and separate prediction heads for joint denoising. The method aims to improve the trade-off between reconstruction fidelity and generation quality and is evaluated on ImageNet class-conditional generation and text-to-image generation.

Critical strengths
  1. The method is intuitive and demonstrates clear empirical gains. After 80 training epochs on ImageNet, CoAE reduces reconstruction rFID from 0.57 to 0.21 compared with RAE while maintaining the same unguided generation FID (gFID = 2.16), demonstrating a better balance between reconstruction and generation.

  2. The evaluation is reasonably comprehensive. The authors compare different feature fusion strategies, detail dimensions, vision foundation models, and detail sources. Ablation studies examine the contributions of the group-wise MSE loss and separate prediction heads, while the T2I experiments provide additional evidence of the method’s applicability.

Critical weaknesses
  1. The implementation details of the prediction heads remain unclear. The paper states that the two prediction heads share a backbone and are split only at the output layer. However, their relationship to the wide DDT head in DiTᴰᴴ, their internal structures, and the source of the approximately 2M additional parameters are not sufficiently explained.

  2. The suitability of the architecture for the T2I baselines is not sufficiently validated. DiTᴰᴴ-XL is used as the diffusion backbone for all latent spaces, but does not evaluate whether the wide DDT head is suitable for low-dimensional FLUX VAE latents. The original RAE paper reports that DiTᴰᴴ can perform worse than standard DiT on SD-VAE latents. A corresponding architectural comparison for FLUX VAE would therefore help assess the robustness of the baseline results.

Questions for authors
  1. Does CoAE retain the original wide DDT head from RAE and attach two independent output mappings to it? Please clarify the structures of the two prediction heads and identify which components account for the approximately 2M additional parameters.

  2. Does the FLUX VAE baseline in the T2I experiments also use a wide DDT head? Please compare standard DiT and DiTᴰᴴ under comparable computational budgets to assess whether the results are robust to the choice of architecture.

2 comments

We thank the reviewer for recognizing our clear structure and rigorous experiments.We address the concerns below.

[W1/Q1]Implementation details of the prediction heads

The standard DiT$^{\mathrm{DH}}$-XL used by RAE and our homogeneous-fusion baselines contains 28 encoder blocks of width 1152, followed by two wider decoder blocks of width 2048. A DDTFinalLayer, consisting of RMSNorm, AdaLN modulation, and an output projection, predicts the CC-channel latent. For CoAE, the 800-channel latent consists of 768 semantic and 32 detail channels. After the shared 28 encoder blocks, an additional DDTFinalLayer(1152,1,32) predicts the detail features. The semantic features are processed by the two wider decoder blocks and theb predicted by another DDTFinalLayer(2048,1,768). The extra detail prediction head introduces 2.695M parameters, primarily from its AdaLN modulation layer. This design is motivated by our empirical findings: for the detail space, which contains richer low-level information and has a lower dimensions, a wide-head decoder may unnecessary .Allowing the wide decoder to focus on the high-dimensional semantic features is better. We also evaluated a simpler variant that places both heads after the wide decoder:

Detail-prediction-head locationgFIDIS
After wide decoder2.31208.00
After shared encoder(ours)2.16207.08

The simpler design is also effective, while our branching design achieves better gFID. We have added these details to Appendix A.2.

[W2/Q2] Architecture of the FLUX VAE baseline.

We thank the reviewer for pointing out this issue in our T2I evaluation. The FLUX VAE baseline in our original experiments indeed uses the wide DDT head. Our primary goal was to isolate the effect of latent organization; therefore, we kept the generator architecture identical across all latent spaces, following the controlled evaluation setting of RAEv2.

Following the reviewer’s suggestion, we additionally compared standard LightningDiT and DiT$^{\mathrm{DH}}$-XL on FLUX VAE. We increased LightningDiT to 38 blocks, resulting in 608.9M608.9\mathrm{M} parameters, which closely matches the 611.3M611.3\mathrm{M} parameters of DiT$^{\mathrm{DH}}$-XL.

TokenizerGeneratorParamsGenEval ↑DPG-Bench ↑
FLUX VAELightningDiT608.9M81.7584.11
FLUX VAEDiT-DH-XL611.3M82.8283.43

The two architectures achieve comparable overall performance: DiT$^{\mathrm{DH}}$-XL performs slightly better on GenEval, whereas LightningDiT performs slightly better on DPG-Bench. The result indicates our conclusions are robust to the generator architecture.

Thank you for the detailed clarifications and additional experiments. The explanation of the prediction-head architecture and parameter overhead, along with the additional architectural comparison for the FLUX VAE baseline, adequately addresses my concerns. I maintain my rating of 3.

Comments on the paper

1 comment

Summary for Reviewers and AC

We sincerely thank all reviewers for their constructive feedback. Based on the feedback, we have provided analyses and revised the paper. All revisions are highlighted in blue.

1. Clarified positioning and implementation (pnTo, mq9F, 6BXC)

  • Relation to prior work: We expanded the Related Work section to clarify the differences between other works. CoAE focuses on preserving semantic and detail information in distinct continuous subspaces rather than merging them into a homogeneous latent.And we added Appendix A provides detailed distinctions from prior work.
  • Method introduction: We revised Section 3.1 to clearly separate the homogeneous fusion alternatives from the final distinct-subspace design and explain why fusion may compromise the original information.
  • Prediction heads: We added detailed architectural descriptions to Appendix B.2.

2. Controlled comparisons of latent and model capacity (mq9F, 6BXC)

  • Matched latent dimension: We project Additive, Projection, and Modulation to the same 800 channels as Concat. Concat still substantially outperforms all homogeneous-fusion variants.
  • Tokenizer capacity: We report latent dimensions and encoder/decoder parameter counts for representative VAE- and VFM-based tokenizers. CoAE uses the same frozen DINOv2-B encoder and ViT-XL decoder as RAE, with an additional detail branch in its default implementation.
  • Matched generator capacity: We reduce the shared DiT encoder from 28 to 21 blocks, producing a 674M-parameter generator comparable to the 675M baselines. The reduced model still achieves a competitive gFID of 2.40 and IS of 200.90.
  • T2I architecture: We compare LightningDiT and DiT$^{\mathrm{DH}}$-XL on FLUX VAE under closely matched parameter counts (608.9M vs. 611.3M). Their comparable GenEval and DPG-Bench results show that our T2I conclusion is robust to the generator architecture.

3. Optimization balance and complementarity (mq9F,6BXC)

  • Per-subspace prediction errors: We added Appendix D.3 and Figure 7 to report the semantic and detail NMSE across diffusion timesteps.
  • Interaction with detail dimension: We further extended Appendix D.3 and Figure 7 to cover 16, 32, and 64 channel detail subspaces. The results show that the benefit of group-wise balancing increases as the dimension gap between the two subspaces becomes larger.
  • Subspace perturbation: We added Appendix D.4 and Figure 8 to visualize how errors in each subspace affect decoded images. Perturbing semantic features disrupts global structure and spatial coherence, whereas perturbing detail features primarily degrades local textures.
  • Latent visualization: We added Appendix E.1 and Figure 9 with spatial PCA and t-SNE analyses, providing additional evidence that the semantic and detail subspaces encode distinct and complementary information.

4. Robustness to detail source and training recipe (mq9F, 6BXC)

  • Detail source: CoAE does not must require an independent detail encoder. Using a shallow feature from the same frozen VFM as the detail source achieves an rFID of 0.23, a gFID of 1.97. This confirms that the key factor is the distinct organization rather than additional encoder capacity.
  • REPA compatibility: Our method with REPA experiment improves gFID from 2.16 to 1.96 and IS from 207.08 to 222.80.
  • RAEv2 comparison: We added RAEv2 to Table 1. Although CoAE falls between its K=7K=7 and K=23K=23 configurations, it provides a competitive balance: compared with K=23K=23, CoAE has only a 0.24 lower PSNR while improving gFID from 3.01 to 1.96; compared with K=7K=7, it improves PSNR by 4.22 with only a 0.31 gFID difference.

We have discussed the paper with the reviewers and addressed their main concerns.We are delighted to provide further clarifications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.