acceptodds
Under review as a conference paper at ICLR 2027

CoVA: Composing VAE Representations for Generative Learning

Abstract

Representation alignment with pretrained visual encoders improves diffusion transformer training but requires auxiliary encoders and extra training overhead. Recent approaches instead exploit representations from the generative model's own variational autoencoder (VAE). However, reconstruction-oriented VAE features are not optimized for the semantic discrimination and spatial structure needed for generative learning. We introduce CoVA (Composed VAE Representations), a lightweight framework that composes VAE features online into stage-wise guidance for flow-matching training. A small latent composer is jointly trained with class supervision, masked latent reconstruction, and horizontal-flip consistency, yielding semantically discriminative, spatially structured dense features and sample-dependent class posteriors. CoVA aligns intermediate SiT features with the composed representations early in training, then removes the increasingly detrimental alignment and reuses detached composer posteriors for sample-dependent soft label conditioning. We further uncover a hidden confound in soft label conditioning: averaging class embeddings attenuates their magnitude, coupling semantic interpolation with conditioning strength. This attenuation alone can substantially improve FID while degrading Recall, revealing a non-semantic source of the apparent gain. CoVA corrects this confound by restoring the hard-label magnitude, isolating posterior-dependent directional semantics. CoVA adds only 8.5M training-only auxiliary parameters (1.3% of SiT-XL/2) and introduces no inference-time overhead. On ImageNet 256x256, CoVA reduces SiT-XL/2 FID from 17.2 to 7.4 (57%) in 400K steps. With classifier-free guidance, it achieves an FID of 2.03 in only 400 epochs, versus 2.06 after 1,400 epochs for vanilla SiT-XL/2.

Primary area:
generative models
Keywords:
VAE Representations, Representation Alignment, Semantic Composition, Soft Label Conditioning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VWWYOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CoVA, a lightweight framework that improves flow-matching training by composing representations from a pretrained VAE without relying on additional pretrained visual encoders. The proposed method introduces an online latent composer trained with classification, masked latent reconstruction, and horizontal-flip consistency objectives to produce semantically informative and spatially structured features.

CoVA adopts a stage-adaptive training strategy: the composed features provide intermediate representation alignment during early training, while the composer's detached class posteriors are reused for sample-dependent soft label conditioning during later stages. The authors further investigate a magnitude confound in soft label conditioning, showing that changes in class-embedding magnitude can substantially affect generation quality and diversity independently of semantic interpolation.

Experiments on ImageNet 256×256 demonstrate improvements across three SiT model scales. In particular, CoVA reduces the unguided FID of SiT-XL/2 from 17.2 to 7.4 at 400K iterations while introducing only 8.5M training-only auxiliary parameters and no inference-time overhead. Extensive ablations investigate the composer objectives, training schedules, and direction–magnitude decomposition of soft conditioning.

Critical strengths

Strength 1: Efficient representation guidance with consistent improvements in unguided generation. CoVA provides an effective alternative to representation alignment methods that require large pretrained visual encoders. The proposed approach consistently improves unguided generation quality across SiT-B/2, SiT-L/2, and SiT-XL/2 while introducing relatively modest auxiliary training cost and no inference-time overhead. This makes the framework practically relevant for efficient generative-model training.

Strength 2: Insightful analysis of soft label conditioning. The direction–magnitude decomposition is a particularly valuable aspect of the paper. The experiments demonstrate that much of the apparent improvement from naive soft conditioning can be attributed to changes in conditioning magnitude, accompanied by a fidelity–coverage trade-off. The additional null-centered controls strengthen this observation. This analysis provides a useful methodological insight beyond the specific CoVA framework.

Critical weaknesses

Weakness 1: Unclear generalization beyond discrete class conditioning. The composer uses class supervision, while the late-stage SLC mechanism relies on an ImageNet-style class posterior and a learned class-embedding table. It is not clear how this component would extend to text-to-image generation, where the condition is typically a sequence of continuous text embeddings encoding compositional semantics. The current paper evaluates only ImageNet class-conditional generation, leaving the claimed broader applicability insufficiently established. This does not mean such an extension is impossible, but a concrete formulation or evidence is needed to assess its feasibility and significance.

Weakness 2: Insufficient comparison and positioning against related approaches. Self-Flow (Chefer et al., ICML 2026) also learns useful generative representations within a flow-matching framework without an external pretrained representation encoder and reports ImageNet 256×256 results with a SiT-XL/2 backbone. CoVA neither discusses nor benchmarks this closely related direction.

Because architectural details, training budgets, and evaluation protocols may differ, headline FID values alone are not an adequate comparison; a careful protocol-aware comparison is needed to establish CoVA's relative strengths, costs, and scope.

In addition, a matched early-stopped raw-VAE alignment (SRA2-style) control would clarify how much of CoVA's gain is due to the learned composer rather than previously established stage-dependent alignment.

Questions for authors

Q1. Generalization to text-to-image (central concern). How would the classification-supervised composer and the posterior-based SLC mechanism be adapted to continuous, token-level text conditioning? In particular, what would replace the class posterior qq and the interpolation over a class-embedding table, and how would the magnitude correction be defined? Please distinguish which components can transfer directly and which require redesign. A concrete technical formulation, or preliminary evidence if available, would materially affect my assessment.

Q2. Relation to Self-Flow (central concern). Could the authors discuss and, where feasible, compare CoVA with Self-Flow (Chefer et al., ICML 2026) under comparable backbone, training compute/steps, sampling protocol, and unguided FID/Precision/Recall settings? If matched experiments are not feasible, a clearly qualified comparison using reported results and a discussion of relative computational overhead and supported conditioning modalities would be useful.

Q3. Incremental contribution over VAE alignment. Does CoVA retain its advantage over an SRA2-style raw-VAE alignment baseline when both use the same early-stopping schedule? A controlled ablation of the composer versus the stopping strategy would strengthen the attribution of the reported gains. Potential score revision.

I currently lean toward weak rejection, but I would be willing to increase my rating if the rebuttal convincingly addresses the text-to-image extension and establishes a clear, fair comparison with Self-Flow and related baselines.

1 comment

Q1. Generalization to text-to-image

We added a text-to-image experiment to our General response Part2 and the revised manuscript. In text-to-image generation, the text condition is already a soft condition provided by a pretrained model, so intuitively there is no need to apply SLC on top of it during training. Our SLC analysis also shows that conditioning magnitude can change the degree of conditioning extrapolation during sampling. This principle can be applied at sampling time in any conditioning setting.

Q2. Relation to Self-Flow

Both methods learn alignment targets without an external pretrained visual encoder. Self-Flow aligns a student processing tokens at two noise levels with deeper features from an EMA teacher that sees the cleaner noise level throughout. CoVA learns its targets from VAE latents using a lightweight online composer. On ImageNet, the same composer also supplies the posteriors used for late-stage soft label conditioning.

SiT-XL ImageNet 256x256 comparison.

Most experiments in Self Flow use different settings from ours. CoVA generally follows the experimental setup of REPA. The table below compares the results from both papers under the most closely matched evaluation settings.

MethodLatent spaceTraining stepsFID ↓sFID ↓IS ↑Precision ↑Recall ↑
Self-Flow + RAERAE1M2.955.50222.340.840.56
Self-FlowSD-VAE4M5.704.97151.400.720.67
CoVASD-VAE1M4.414.62180.740.740.62
CoVASD-VAE2M3.594.68205.120.740.64

Training cost.

CoVA adds 8.5M training-only parameters and increases SiT-XL/2 compute by 7.2%, with throughput changing from 8.1 to 7.0 iterations/s (Table 4). Self-Flow reuses the model’s EMA weights and incurs an additional teacher forward pass.

Q3. Incremental contribution over VAE alignment.

We added a unified ablation study to disentangle the contributions of the alignment target, early stopping, and soft label conditioning. Part 1 of our general response summarizes the findings. Table 1 and the accompanying discussion in the revised manuscript provide the full results (lines 324–347 and 366–373).

Reviewer_k4zVOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoVA (Composed VAE Representations), a lightweight framework that leverages representations from the existing VAE to improve diffusion transformer training without requiring an external pretrained visual encoder. CoVA introduces an online latent composer trained with classification, masked latent reconstruction, and horizontal-flip consistency objectives to produce semantically discriminative and spatially structured features. These features are used for intermediate representation alignment during early training, while the composer's class posteriors are subsequently reused for sample-dependent soft label conditioning. The authors further identify a magnitude confound in naive soft label conditioning, showing that changes in class embedding magnitude can substantially improve FID at the expense of sample diversity. To address this, CoVA introduces magnitude normalization to preserve the conditioning strength of hard-label embeddings. Experiments on ImageNet 256×256 demonstrate consistent improvements across SiT model scales, achieving an FID reduction from 17.2 to 7.4 on SiT-XL/2 after 400K training steps, with modest training overhead and no additional inference-time cost.

Critical strengths
  1. Strong empirical performance with low computational overhead.

The proposed method demonstrates substantial and consistent improvements in generation quality across multiple SiT model scales. In particular, CoVA reduces the FID of SiT-XL/2 from 17.2 to 7.4 under the same 400K-step training budget, while also outperforming existing representation alignment methods such as REPA and SRA² in this setting. Importantly, these improvements are achieved without relying on an additional pretrained visual encoder, with relatively modest training overhead and no additional inference-time cost. These results establish the practical effectiveness of the proposed framework and provide strong empirical support for its acceptance.

  1. An insightful analysis of the magnitude confound in soft label conditioning.

A particularly valuable contribution is the identification and systematic analysis of the magnitude confound in soft label conditioning. Through controlled experiments that disentangle embedding direction and magnitude, the authors demonstrate that much of the FID improvement from naive soft conditioning can be attributed to changes in conditioning magnitude rather than semantic interpolation, accompanied by a reduction in sample diversity. The additional inference-time scaling experiments and null-centered controls further strengthen this finding. This analysis provides a useful conceptual insight into the relationship between conditioning strength and the fidelity–diversity trade-off, with implications extending beyond the proposed CoVA framework.

Critical weaknesses
  1. Insufficient isolation of the contribution of composed VAE representations.

While CoVA demonstrates strong empirical improvements, the current experiments do not fully disentangle the benefits of representation composition from those of stage-adaptive alignment and soft label conditioning. In particular, early stopping of representation alignment has already been explored in prior work, yet the paper does not provide a raw-VAE alignment baseline with the same early-stopping schedule and subsequent conditioning strategy [1][2]. Consequently, it remains unclear how much of CoVA's improvement can be attributed specifically to the proposed representation composer. A matched comparison with SRA² under the same training schedule and conditioning strategy would be important for establishing the independent contribution of the central methodological component.

[1] Wang, Z., et al. REPA Works Until It Doesn't: Early-Stopped, Holistic Alignment Supercharges Diffusion Training. Advances in Neural Information Processing Systems (NeurIPS), 2025.

[2] Wang, M., et al. SRA²: Variational Autoencoder Self-Representation Alignment for Efficient Diffusion Training. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026.

Questions for authors

Q1. Clarification on the shuffled-posterior experiment in Table 5.

In Table 5, the authors report that shuffling the soft conditioning directions substantially degrades FID (from 22.73 to 34.13), interpreting this result as evidence that the posterior directions contain useful sample-specific semantic information. Could the authors clarify whether the shuffling was performed across all samples or only among samples belonging to the same ground-truth class?

This distinction is important because cross-class shuffling may introduce class-condition mismatches. For example, if an image of a dog is assigned the posterior originally predicted for an airplane image, the resulting soft condition may become inconsistent with the ground-truth class label. In this case, the performance degradation could be explained by incorrect class conditioning, rather than the loss of sample-specific semantic information.

A more informative control would be to shuffle posteriors only among samples sharing the same ground-truth class. This would largely preserve class-level semantic information while disrupting the correspondence between each image and its individual posterior. If performance still degrades substantially under within-class shuffling, it would provide stronger evidence that the improvement from SLC depends on meaningful instance-specific information rather than class-level conditioning alone.

Q2. The role of MAE-style masked latent reconstruction in learning prediction-oriented representations.

I think one interpretation of CoVA is that MAE-style masked latent reconstruction encourages the composer to learn contextual and prediction-oriented representations from reconstruction-oriented VAE latents. Such representations may be better suited for downstream generative modeling, which could partly explain the effectiveness of CoVA.

However, according to Table 4, using MIM alone provides only a modest FID improvement (33.02 to 31.92). Moreover, Table 8 shows that MIM-only representations achieve a Spatial LDS of 0.06, even lower than the 0.28 of raw VAE features. Substantial improvements in spatial structure appear only when MIM is combined with classification supervision and horizontal-flip consistency.

Could the authors further explain what role MAE-style masked reconstruction plays in constructing generation-useful feature structures? In particular, why does MIM alone fail to improve Spatial LDS, and to what extent do the improvements in generative modeling depend on the interaction between masked prediction, semantic supervision, and spatial consistency?

2 comments

[W1] Insufficient isolation of the contribution of composed VAE representations.

We added a unified ablation study to disentangle the contributions of the alignment target, early stopping, and soft label conditioning. Part 1 of our general response summarizes the findings. Table 1 and the accompanying discussion in the revised manuscript provide the full results (lines 324–347 and 366–373).

[Q1] Clarification on the shuffled-posterior experiment in Table 5.

Thank you for pointing out the ambiguity in the original Table 5. The reported FID of 34.13 corresponds to within-class shuffling: we randomly exchanged soft conditioning among images with the same ground-truth label. This increased FID from 22.73 to 34.13, showing that matching each image to its own posterior matters even when class identity is preserved. We have clarified this in the revised Table 7 and added a global shuffle control, in which directions are permuted across all samples without regard to class. FID increases further to 63.06, consistent with the additional class-condition mismatches introduced by global shuffling.

[Q2] The role of MAE-style masked latent reconstruction in learning prediction-oriented representations.

Appendix B (Spatial LDS Protocol, Table 10 new version and the discussion immediately below it analyzes why MIM alone yields low Spatial LDS and how its effect changes with classification and flip consistency.

We agree that masked prediction encourages the composer to use context. With MIM alone, nearby and distant patches both have high cosine similarity (0.92 and 0.86), yielding a Spatial LDS of only 0.06, compared with 0.28 for raw VAE features. One interpretation is that shared context helps predict missing latents while also making distant patches more similar. The reconstruction objective itself places no explicit constraint on the near–far similarity gap measured by LDS.

The contribution of MIM is clearest when the other objectives are held fixed. Adding MIM to classification improves LDS from 0.29 to 0.36 and FID from 30.85 to 28.11. Adding MIM to classification plus flip consistency improves LDS from 0.09 to 0.41 and FID from 28.06 to 26.83. For comparison, MIM alone gives a smaller FID gain, from 33.02 to 31.92. These comparisons support a complementary role for masked prediction alongside semantic and spatial supervision.

Thank you for the response and clarifications. I will take these clarifications into account in my final assessment.

Reviewer_wAVGOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to improve the generation performance and training convergence of latent-space diffusion transformers. Specifically, the SRA2-style VAE-based representation alignment is extended with an online composer trained using classification, masked reconstruction, and flip-consistency losses. Alignment is stopped at 200K steps, and the composer's class posteriors are used for soft label conditioning during the final 50K steps. Experiments on ImageNet show improved generation FID and advantages over REPA and SRA2 across model scales.

Critical strengths

The improvement in generation quality, particularly unguided FID, is strong and seems to outperform REPA without relying on external pre-trained visual encoders.

Critical weaknesses
  1. Coupled mechanisms and training schedules make the source of improvement difficult to identify.
  • The method combines several interacting mechanisms, many related to existing approaches:
    • (1) VAE-based representation alignment, following SRA2;
    • (2) alignment early stopping at 200K, following HASTE;
    • (3) auxiliary CLS and MIM losses. Although applied to the composer rather than directly to DiT, the resulting representations subsequently supervise DiT through alignment. This might be related to other methods such as LSEP, SD-DiT, and Self-Flow.
    • (4) the proposed Soft Label Conditioning, but used only in the final 50K steps.
  • However, the current ablations do not clearly separate these contributions. For example, one important missing baseline is SRA2 with the same 200K early stopping schedule, since Table 1 shows that the improvement over SRA2 is relatively modest at 200K (22.5-20.4=2.1) but increases at 400K (14.3-9.3=5.0). How much of this additional gain comes from the composed alignment targets, early stopping, or the final 50K of SLC?
  • Table 4 studies the composer objectives without early stopping or SLC, but does not establish their contributions under the final recipe. Since the composer keeps evolving between the early stopping point and the warmup point (e.g., 200K-350K), its objectives shape both the early alignment targets and the late conditioning. Ablations under the complete recipe would be more convincing.
  • The delayed activation of SLC is motivated by allowing the composer to warmup. However, why is SLC still restricted to the final 50K in 1M/2M runs, rather than activated earlier or used longer, if it is really valuable?

Training curves like Fig. 3(c), with controlled comparisons and explicit annotations of when each mechanism is active, would help clarify these questions.

  1. The effects and applicability of the Soft Label Conditioning need further clarification.
  • Table 1 shows that at 1M (200 epochs) /2M iters (400 epochs), unguided FID improves from SRA2's 8.2/7.0 to 4.4/3.6. However, Table 2 shows that guided FID becomes worse, from 1.98/1.84 to 2.19/2.03. Why does CoVA benefit less from CFG?
  • SLC modifies the conditioning embeddings during training, but inference still uses hard labels. Could this mismatch contribute to the weaker CFG gains?
  • Both the CLS loss and the embedding mixture in SLC depend on discrete class labels. It is unclear how the method would extend to other conditioning settings, particularly text-to-image. More fundamentally, why inject sample-dependent semantic information by replacing the original class condition, rather than preserving it and adding the information as a complementary signal (e.g., as in Conditioning Residuals, arXiv:2505.10999)? Discussing this design choice would help clarify whether the dependence on class labels is necessary.
Questions for authors

Please refer to the weaknesses.

3 comments

[W1] Coupled mechanisms and training schedules make the source of improvement difficult to identify.

We added a unified ablation study to disentangle the contributions of the alignment target, early stopping, and soft label conditioning. Part 1 of our general response summarizes the findings. Table 1 and the accompanying discussion in the revised manuscript provide the full results (lines 324–347 and 366–373).

[W2] The effects and applicability of the Soft Label Conditioning need further clarification.

1. Why does CoVA benefit less from CFG?

The relationship between SLC and CFG is an interesting research question. We observed a connection between them in our experiments. In Appendix D, Understanding the Magnitude Confound in Soft Label Conditioning, we therefore include a subsection (lines 1046–1068) that derives this connection and explains the differences. In particular, SLC has an implicit class-conditioning strength extrapolation effect similar to that of CFG. We also found that when a model is trained with SLC and sampled with CFG, the optimal CFG scale generally needs to be lower than the scale used in conventional settings. As shown in the appendix subsection CFG Sensitivity after SLC Training (lines 994–1014), adjusting the CFG scale brings the model’s FID below 2.0. CoVA also achieves higher recall than SRA2 (0.62 vs. 0.61).

2. It is unclear how the method would extend to other conditioning settings, particularly text-to-image.

We added a text-to-image experiment to our general response and the revised manuscript. In text-to-image generation, the text condition is already a soft condition provided by a pretrained model, so intuitively there is no need to apply SLC on top of it. Furthermore, our SLC analysis also shows that conditioning magnitude can change the degree of conditioning extrapolation during sampling. This principle can be applied at sampling time in any conditioning setting.

3. Could this mismatch contribute to the weaker CFG gains? Discussing this design choice would help clarify whether the dependence on class labels is necessary.

For training tasks on datasets such as ImageNet, it is difficult to obtain accurate soft labels without introducing an additional model. We therefore use hard labels during sampling. To avoid a mismatch between training and inference, during training, we randomly replace hard labels with soft labels rather than combining them in a fixed weighted average.

Thank you for the additional experiments. The new ablations indeed show that early stopping has a relatively modest effect, while Composer and SLC improve more. The MS-COCO experiment also demonstrates that the composer alignment can transfer to T2I settings.

However, I find that adding SLC can already improve Raw VAE from 30.41 to 24.45, and switching the alignment target to Composer (I assume the new representation alignment is the main contribution) gives a smaller additional gain, from 24.45 to 22.62. Furthermore, another important comparison, i.e., "SRA2+SLC, Raw VAE, 200K Align. stop, 350K SLC start" vs 21.45, seems to be missing. I would also appreciate a Raw VAE (SRA2) baseline under the same protocol to compare with CoVA's 4.93 on MS-COCO.

Thank you for these follow-up questions. We have added the two requested comparisons to the updated General Response.

[1] Contribution of composer alignment.

We agree that SLC provides a larger absolute FID improvement in the comparison you highlighted. The improvement from 24.45 to 22.62 measures the additional benefit of composer alignment when SLC is already enabled. With SLC disabled and alignment active throughout training, replacing raw-VAE targets with composer targets improves FID from 30.41 to 26.83, a reduction of 3.58. For reference, the vanilla SiT-B/2 baseline achieves 33.02. These matched comparisons demonstrate the benefit of composer alignment both with and without SLC, with the size of the gain depending on the conditioning configuration.

[2] Matched early-stopping baseline.

We have added the requested configuration with raw-VAE alignment stopped at 200K and SLC activated at 350K. It achieves 24.64, compared with 24.45 when raw-VAE alignment remains active throughout training and 21.45 for CoVA under the same early-stopping and SLC schedule. This result shows a small benefit from early stopping for SRA2 + SLC, while composer alignment retains an advantage under the matched schedule. The full results are included in the updated General Response.

[3] MS-COCO baseline.

Under the same training and evaluation protocol, the added SRA2 baseline achieves 5.03, compared with 4.93 for CoVA and 6.05 for vanilla MMDiT. We have included this comparison in the updated General Response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Reviewers,

  • Firstly, We thank the reviewers for recognizing CoVA’s strong performance without an external pretrained visual encoder, and its modest training overhead and zero additional inference cost (wAVG, k4zV, VWWY). We also appreciate the positive assessment of our insightful analysis of the magnitude confound in soft label conditioning. (k4zV, VWWY).
  • We added two sets of experiments to address questions raised across the reviews.

1. Separating the contributions of composed VAE, early stopping, and SLC.

We added a unified ablation to the revision (Table 1, lines 324–347 and 366–373) and reproduce it below.

Table 1. Component and schedule ablations on SiT-B/2. All runs are evaluated at 400K training steps on ImageNet 256×256 without CFG. “Full” denotes alignment throughout training. In the SRA2 + SLC rows, raw VAE features provide all intermediate supervision, and the online composer supplies only the late conditioning posteriors.

MethodAlignment targetAlign. stopSLC startFID ↓sFID ↓Prec. ↑Rec. ↑IS ↑
SiT-B/2None––33.026.460.530.6343.71
SRA2Raw VAEFull–30.416.150.560.6350.47
SRA2Raw VAE100K–30.536.170.560.6450.09
SRA2Raw VAE200K–30.066.160.560.6350.80
SRA2Raw VAE300K–29.226.030.560.6452.33
SRA2+SLCRaw VAEFull200K25.505.530.570.6361.76
SRA2+SLCRaw VAEFull300K24.905.580.570.6363.44
SRA2+SLCRaw VAEFull350K24.455.590.570.6365.27
SRA2+SLCRaw VAE100K350K24.875.510.570.6365.05
SRA2+SLCRaw VAE200K350K24.645.520.570.6364.77
SRA2+SLCRaw VAE300K350K24.515.510.570.6365.02
CoVAComposerFull–26.836.320.570.6457.69
CoVAComposer100K–27.566.400.570.6356.14
CoVAComposer200K–24.875.960.580.6461.19
CoVAComposer300K–27.226.240.570.6457.60
CoVANone–200K30.395.870.540.6353.07
CoVANone–300K29.695.790.540.6354.89
CoVANone–350K29.325.730.540.6455.77
CoVAComposerFull200K23.845.700.580.6267.05
CoVAComposerFull300K23.055.640.580.6369.62
CoVAComposerFull350K22.625.640.580.6370.86
CoVAComposer200K200K23.555.690.590.6267.70
CoVAComposer200K300K22.585.660.590.6370.30
CoVAComposer200K350K21.455.690.590.6374.57
SRA2 + CoVARaw VAE + Composer200K350K20.525.530.600.6376.39

These additional experiments support the following conclusions.

  1. Early stopping yields a modest gain for SRA2. On SiT-B/2 at 400K, stopping raw-VAE alignment at 200K or 300K improves FID from 30.41 to 30.06 or 29.22, respectively.

  2. Composed targets and SLC each contribute under matched schedules. With alignment stopped at 200K and SLC disabled, replacing raw-VAE targets with composed targets improves FID from 30.06 to 24.87. With raw-VAE alignment active throughout, enabling SLC for the final 50K steps improves FID from 30.41 to 24.45. Under CoVA’s default schedule, combining raw and composed targets further improves FID from 21.45 to 20.52, suggesting that the two targets provide complementary supervision.

2. Transfer to text-to-image generation.

To test whether CoVA’s representation alignment transfers beyond discrete class labels, we added an MS-COCO experiment with MMDiT. We replace the composer’s classification loss with a CLIP-style image–text contrastive loss and retain masked latent reconstruction and horizontal-flip consistency. MMDiT retains its original text-token conditioning throughout training and inference. MMDiT already conditions on continuous text embeddings, and the same caption provides semantic supervision for the composer through the image–text contrastive loss. We therefore retain MMDiT’s original text conditioning and disable class-posterior SLC to isolate the contribution of representation alignment.

Following the REPA setup, we train for 150K iterations with a batch size of 256. Given this shorter training budget, we keep alignment active throughout. We evaluate using an ODE sampler with 50 function evaluations and a CFG scale of 2.0.

MethodMS-COCO FID ↓
MMDiT6.05
MMDiT + REPA4.73
MMDiT + SRA4.85
MMDiT + SRA25.03
MMDiT + CoVA alignment4.93

CoVA reduces FID from 6.05 to 4.93, a 1.12-point improvement over MMDiT. This result shows that image–text supervision can replace class supervision in the composer and produce useful alignment targets for text-conditioned generation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.