acceptodds
Under review as a conference paper at ICLR 2027

From Pixels, Without Pre-training: Joint Generative and Self-Supervised Representation Learning in One Model

Abstract

Strong image generation models today are conditioned on class labels, aligned to frozen pretrained encoders, or built on separately trained autoencoders or tokenizers. While effective, the cost is that generation becomes contingent on supervision and pretraining: labels must be annotated, and encoders, autoencoders, and tokenizers must be pretrained for the target domain. This paper studies **joint generative and self-supervised representation learning in one model**, such that generation is self-conditioned without any labels or pretrained models. This is challenging since the two objectives are mismatched: contrastive learning consumes clean augmented views and keeps coarse, invariant semantics, while flow matching consumes noisy images and must preserve the fine detail and spatial layout that contrastive learning discards. We propose a framework Self-conditioned Generation on Self-supervised representation (SCION), whose core is a single pixel-space encoder conditioned on both the diffusion timestep and a conditioning embedding. The encoder serves two roles that differ in what that embedding is. For representation learning, the conditioning embedding is one learned global vector shared by all images, and the encoder’s [CLS] token yields the semantic embedding trained by the contrastive loss. For generative training, the conditioning embedding is the [CLS] embedding of the image itself, and the encoder’s patch tokens pass through a decoder to predict the image. To sample without a reference image at inference, we also learn a prior over the embedding in the same run. With gradient-norm balancing and stop- gradient tricks, all losses are jointly optimized in a single run. The resulting method is fully self-supervised, with no pretrained model and no labels. On ImageNet 256 × 256, with the same JiT-B/16 recipe and without representation guidance, SCION reaches 8.92 FID, surpassing the class-unconditional baselines iREPA, which aligns to a pretrained DINOv2 (46.44), and RCG, which conditions on it (14.27). Under the JiT-L/16 recipe, SCION achieves the best pixel-space class-unconditional FID of 5.89 and 3.47 without and with representation guidance, outperforming RCG with the ADM recipe (6.24).

Primary area:
generative models
Keywords:
Flow Matching, Contrastive Learning, Generative Modeling, Representation Learning, Self-Supervised Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_apE9OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies unifying image representation learning and generation within a single model. To achieve this, it proposes training an encoder with 3 losses (contrastive loss, flow matching loss, and conditioned-branch consistency loss). However, it also trains a separate token generator to produce semantic token, which is input to the encoder as additional condition during inference. Overall, it targets at perception and generation without any class labels in an end-to-end manner.

Critical strengths
  • Unifying image representation learning and generation is an interesting and challenging topic.
  • The figures are clear and easy to understand.
Critical weaknesses
  1. Despite introducing additional complexity, the empirical results are less persuasive as the performance falls short of the baseline's official implementations. For example, as shown in Table 10, the official implementation of RCG achieves 3.44 without guidance and 2.15 with CFG, surpassing SCION’s 3.47(w RG). This performance discrepancy makes the claims in Figure 1 somewhat misleading.
  2. There are actually two separate models to achieve unified representation learning and image generation, contradicting the “one model” claim of the paper.
  3. The key technical contritions of the paper are not clear, and crucial discussion of related work is missing from the main paper. The authors should discuss and compare with related methods in the main paper to provide a direct technical background and properly position the method within the literature. Besides, the chosen baseline RCG is somewhat outdated, omitting several recent and relevant works, such as EPG[1], SD-DiT[2], Self-Flow[3], USP[4].

[1] There is No Vae: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-training.
[2] SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer.
[3] Self-Supervised Flow Matching for Scalable Multi-modal Synthesis.
[4] USP: Unified Self-Supervised Pre-training for Image Generation and Understanding.

Questions for authors
  1. Compared to the two-stage training framework [1,4], where the first stage is also self-supervised pre-training without labels, what is the primary motivation or practical necessity for the proposed end-to-end paradigm given its added training complexity? Are there essential benefits in model performance or training efficiency? What’re the differences between SCION and SD-DiT[2] consider they both use a similar training method?
  2. What’s the necessity of combining four distinct losses (GCL, CFM, prior, CBC) given the marginal gains in performance (e.g. FID of RCG(pixel-space ADM) v.s. SCION = 6.24 v.s. 5,89). What’s the overall training cost, measured in GPU hours, of SCION compared to RCG?
  3. In the main experiments, could the authors explain the rationale of adding attention residuals to JiT rather than keeping JiT’s original designs?
  4. What’s the size of the network used to learn prior of the semantic embeddings?
  5. What’s the similarity metric used for computing CBC loss in line 302?
7 comments

We believe that our responses have addressed the reviewer's concerns, and we would be grateful if the reviewer would consider re-evaluating our paper. We are happy to answer any further questions.

Q1.

This performance discrepancy in Table 10 and Figure 1 for RCG makes the claims in Figure 1 somewhat misleading.

A: The results in Figure 1 and Table 10 for RCG are two different settings. Figure 1 presents a controlled comparison (apples to apples), corresponding to Tables 2 and 4, where all methods use the same JiT-B/16 backbone, data, training recipe, and epochs. RCG is a representation-conditioning framework that can be combined with different generators. Here, we combine it with JiT (same backbone as SCION) to remove backbone differences. Under this setting, SCION outperforms RCG both without guidance and with guidance, while also learning substantially stronger representations. The official RCG results in Table 10 cited by the reviewer instead use a different generator, MAGE-L, trained on the tokens of a pretrained VQGAN whose perceptual loss relies on a network trained with labels. We have clarified the controlled setting in Figure 1's caption.

Q2.

the empirical results are less persuasive as the performance falls short of the baseline's official implementations. For example, as shown in Table 10, the official implementation of RCG achieves 3.44 without guidance and 2.15 with CFG, surpassing SCION's 3.47(w RG).

A: Table 10 of the submission, provided in the appendix for reference, includes results from both pixel-space and latent-space generative models. In particular, the RCG (MAGE-L) results cited by the reviewer (3.44/2.15 FID) use the latent-space MAGE generator with a pretrained tokenizer, which differs from our pixel-space setting. The relevant official pixel-space comparison is RCG (ADM), reported in Table 3, which achieves 6.24 FID with ~617M parameters. In the submission, SCION (L/16) already improved upon this result with fewer parameters (559M), achieving 5.89 FID without guidance.

Furthermore, following the reviewer, we trained an improved SCION (L/16) with ~536M parameters and only 300 epochs. It achieves 3.32 FID without guidance and 2.53 with representation guidance. Remarkably, this even surpasses the unguided result of the latent-space RCG (MAGE-L) (3.44), while approaching its guided result (2.15), even though SCION operates entirely in pixel space, without pretrained components or labels.

Q3.

Regarding SCION introducing additional complexity

A: We respectfully disagree when considering the complete training pipeline. One of SCION's goals is precisely to simplify image generation by avoiding pretrained tokenizers/encoders and multi-stage training. Following the reviewer's example, RCG (MAGE-L) requires a pretrained VQGAN tokenizer (with adversarial and LPIPS networks and losses) and MoCo v3 encoder training, followed by separate prior and generator training, whereas SCION drops the tokenizer and trains jointly from scratch in a single stage.

Moreover, multi-stage training introduces additional tuning decisions, such as allocating training budgets and selecting intermediate checkpoints, whose quality may not predict downstream generation performance [1,2]. These dependencies can also complicate adaptation to new domains where pretrained components may be unavailable. Thus, while SCION's training step is more involved than RCG's generator step, its overall pipeline is much simpler.

Thus, SCION improves upon RCG under controlled conditions and achieves strong performance against official pixel-space baselines, while substantially simplifying the overall training pipeline.

[1] Yao et al., Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models, CVPR 2025. [2] Yue et al., What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion, 2026.

Q4.

There are actually two separate models to achieve unified representation learning and image generation, contradicting the "one model" claim of the paper.

A: We apologize for the confusion. The encoder and decoder of SCION are not two separate models, but two views of the layers of a single JiT model. The encoder is the first 8 of the 12 Transformer blocks of JiT-B/16, and the decoder is the remaining 4 blocks. The first blocks provide the representation (the [CLS] token after the encoder blocks), while the full model generates images. This is how a single model unifies representation learning and image generation. We have made this explicit in Section 3 and in Figures 2 and 3.

Q5.

The key technical contributions of the paper[...]

A: The central challenge we address is diffusion-based image generation that is simultaneously self-contained (no pretrained components), self-supervised (no labels), single-model, and single-stage, which prior approaches do not achieve together (Table 1). Our key contribution is a framework that achieves this through semantic self-conditioning: SCION's unified pixel-space architecture and training framework, which reconciles contrastive representation learning with flow matching in a shared backbone, using its own learned representations for generative conditioning.

Q6.

crucial discussion of related work is missing from the main paper.

A: We discuss related work in the introduction and Table 1 summarizes it, showing prior work does not achieve the "4s", with a more detailed discussion in Appendix A. Given your suggestion, we have incorporated a related work section into the main paper (Section 5) and included several recent works (including the ones mentioned by the reviewer).

Q7.

Besides, the chosen baseline RCG is somewhat outdated, omitting several recent and relevant works, such as EPG[1], SD-DiT[2], Self-Flow[3], USP[4].

A: RCG is the closest existing baseline to SCION, and shares the class-unconditional/representation-conditioned setting, while the four works mentioned are class-conditional. That is, RCG is particularly relevant because, like SCION, it replaces class labels with self-supervised representations for generative conditioning. RCG's conditioning comes from a pretrained model, while SCION learns its own representations. This makes it a natural controlled baseline for isolating SCION's contributions.

In contrast, the four works mentioned by the reviewer rely on class labels for conditioning, not being self-supervised. EPG and USP use self-supervised pretraining to initialize subsequent class-conditional generators, while SD-DiT and Self-Flow introduce self-supervised auxiliary objectives into class- or text-conditional generative models. Although these approaches demonstrate the complementarity of self-supervised learning and generation, they still rely on external conditioning.

Q8. Comparison with EPG, SD-DiT, Self-Flow, USP

A: Despite operating without labels, SCION compares favorably with these class-conditional methods at comparable scales. We compare their unguided generation performance reported in the tables below.

At B scale, SCION-B achieves 13.16 FID after only 100 epochs (125K steps), outperforming all four methods' B-scale results (22.10–28.62), despite using substantially fewer optimization steps.

At L/XL scale, the submitted SCION-L (5.89 FID, 400 epochs) outperforms EPG, SD-DiT, and USP, while approaching Self-Flow (5.70 FID), which uses approximately 8× more optimization steps. Furthermore, our new SCION-L variant achieves 3.32 FID and 171.6 IS after only 300 epochs (375K steps), surpassing all four methods' reported results without using class labels or pretrained components.

We have added these works to the related work (Section 5 and Appendix A) and these comparisons to Table 12 (Appendix C.5).

B scale

methodlabelsparamsbatchepochsstepsFIDIS
EPG-B/16✓229M1024n.r. + 240n.r. + 300K25.154.3
SD-DiT-B/2✓n.r.25680400K28.62n.r.
USP (SiT-B/2)✓130M4096 / 2561600 + 80500K + 400K22.1061.2
SCION-B✗231M1024100125K13.1681.9
SCION-B✗231M1024400500K8.92118.4
SCION-B✗231M10248001M7.67127.6

L/XL scale

methodlabelsparamsbatchepochsstepsFIDIS
EPG-XL/16✓583M1024480 + 800600K + 1M6.6141.1
SD-DiT-XL/2✓741M2564802.4M7.21144.7
SD-DiT-XL/2✓741M2567003.5M6.74n.r.
Self-Flow (SiT-XL/2)✓675M2568004M5.70151.4
USP (SiT-XL/2)✓675M4096 / 2561600 + 400500K + 2M6.96141.5
SCION-L (submission)✗559M1024400500K5.89151.6
SCION-L (revision, see Q2)✗536M1024300375K3.32171.6

Q9.

Compared to the two-stage training framework [1,4] [...] what is the primary motivation or practical necessity for the proposed end-to-end paradigm given its added training complexity? Are there essential benefits in model performance or training efficiency?

A:

1) SCION vs [1,4]. As discussed in Q8, the main benefit of SCION is it outperforms both methods while being end-to-end self-supervised and label-free generation, as opposed to [1,4] which require labels for training and inference.

That is, SCION fundamentally differs from [1,4] by learning its own representations for generative conditioning, enabling label-free generation. Although the cited approaches use self-supervised pretraining, their generators still rely on class labels, and thus are not self-supervised end-to-end.

2) SCION vs multi-stage. The multi-stage alternative to SCION is exactly RCG, which separately trains a representation encoder, prior, and representation-conditioned generator. As shown in Tables 2, 3, and 10, SCION outperforms this multi-stage variant. As discussed in Q3, SCION also simplifies the overall training pipeline by avoiding separate stages and their associated tuning decisions.

Moreover, sharing the encoder part of JiT for CL and FM benefits generation. We conducted an ablation using a separate encoder for CL not sharing with any parts of the JiT generator network. We trained the models together with the prior model jointly. With 400 epochs, this baseline has a FID score of 10.50, which is worse than SCION with a single model to unify encoder and generator (8.92 FID; Appendix C.3, Table 9).

Q10.

What's the overall training cost, measured in GPU hours, of SCION compared to RCG?

A: For comparing training resources of SCION with RCG (JiT), we need to account for the encoder pretraining of RCG (JiT). RCG (JiT) uses the released DINOv2-B, distilled from DINOv2-g, whose pretraining alone takes 22,016 A100 GPU-hours. On top of that encoder, the prior and generator of RCG (JiT) take 13.6 and 95.1 H200 GPU-hours (Appendix C.8, Table 15). In contrast, SCION, which needs no pretrained encoder, costs only 249.4 and 498.7 H200 GPU-hours at 200 and 400 epochs. At 200 epochs, SCION achieves better generative performance with fewer total FLOPs and GPU-hours than RCG after accounting for its encoder pretraining time (Appendix C.8, Table 15).

Appendix C.8 also estimates RCG with other ViT-B encoders trained on ImageNet.

Q11

What're the differences between SCION and SD-DiT[2] consider they both use a similar training method?

A: SCION differs substantially from SD-DiT. While both combine self-supervised discrimination with generative training, the 3 main differences are:

(1) Role of representations: self-conditioning vs class-conditioned. SD-DiT uses discrimination as an auxiliary objective for class-conditional generation, without using its learned representations for generative conditioning or evaluating them for downstream understanding. SCION instead leverages it by replacing class labels with its own learned representations for conditioning.

(2) Training dependencies: no dependencies vs label- and VAE-dependent. SD-DiT operates in the latent space of a pretrained VAE and relies on class labels, whereas SCION operates directly in pixel space, without pretrained components or labels.

(3) Self-supervised objective: contrastive learning vs self-distillation. SD-DiT applies teacher-student distillation to noised views, while SCION instead applies contrastive learning to clean, augmented views, without a teacher network or masking.

Reviewer_w297OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SCION jointly learns pixel-space image generation and self-supervised representations from scratch, without labels or pretrained components. A shared encoder extracts contrastive features from clean views and processes noisy images conditioned on detached clean-image representations. A representation prior is trained simultaneously to enable sampling without a reference image. The experiments evaluate generation and representation quality, include a construction ladder and capacity-matched contrastive-only controls, and diagnose the learned prior's coverage tradeoff.

Critical strengths
  1. Table 2 reports 8.92 FID and 72.35% ImageNet probe accuracy without pretrained components or training labels. At 200 epochs, SCION already achieves 10.10 FID versus RCG's 14.27 with a 200-epoch generator and an additional 200-epoch prior stage (Table 9). Extra generator epochs therefore cannot explain the entire advantage.
  2. Representation-conditioned generation and joint representation/generation learning have substantial precedent, including RCG and SODA. SCION combines them in a shared dual-mode pixel encoder, with a prior learned alongside the evolving representation. I find this a useful system contribution.
  3. The construction ladder attributes the largest generative improvement to adding self-conditioning and the prior together, from 54.79 to 11.66 FID, with CBC reaching 10.10. Capacity-matched contrastive-only controls show useful representations survive joint training, while Table 11 explicitly identifies a precision/recall tradeoff introduced by the learned prior.
Critical weaknesses
  1. Appendix E.2 (p.29, lines 1512–1513) reports the best test accuracy over ten regularization values. The reported scores therefore include test-set selection bias. This affects transfer-8 comparisons, including the claimed advantage over contrastive-only training (p.9, lines 464–469; Table 8). Applying the same procedure to all methods does not ensure equal bias; the magnitude and effect on rankings are unknown. Please select regularization within the training data or on validation data and report corrected per-dataset and aggregate comparisons. The ImageNet probe uses held-out-training selection; this issue does not extend to that probe or establish contamination of FID.
  2. Appendix D (p.24, lines 1288–1295) suggests that stop-gradient leaves representation content to the contrastive objective. However, CFM/CBC still update the shared encoder, as p.6, lines 319–321 correctly acknowledges. Stopping the direct conditioning path does not eliminate generative influence through shared weights. The fixed-conditioning FM/CFM decomposition does not itself establish semantic isolation during joint training. Please narrow the claim and make the prior input/target detachment explicit in Algorithm 1.
  3. The 400-epoch comparison assigns RCG 200 prior and 200 generator epochs, whereas SCION has 400 joint epochs and additional clean-view processing (p.7, lines 369–375). This does not establish equal computation. Table 9 partially addresses this concern, as noted above. Likewise, Table 8 matches encoder capacity but uses independently tuned contrastive-only runs with a 600-epoch cosine schedule (p.20, lines 1057–1061), while SCION uses a constant learning rate. These comparisons support the reported capabilities, but do not isolate optimization effects. Please report component updates and resources and qualify the comparison accordingly.
  4. Appendix A (p.18, line 966) says SODA requires a source image for synthesis. However, its public preprint, Section 3.3 and Appendix G.3 describes masking latent/pose information, including generating novel objects at a requested pose. Please distinguish the source-conditioned settings evaluated there from all supported sampling modes, and clarify the scope of Tables 1/6. SCION's shared encoder and learned-prior evaluation on unconditional ImageNet remain distinct.
Questions for authors
  1. Was transfer regularization selected on the test set as written? Please provide validation-selected per-dataset and aggregate results, or verifiable code and split details showing that the text misstates the protocol. Which split supplies the feature-standardization statistics?

  2. Can Appendix D distinguish the blocked conditioning path from generative influence through shared parameters? Does the implementation detach both the prior's noisy input and its target? Please also give the initialization and decay used for the representation-standardization statistics.

  3. What are the component update counts, view exposures, and measured training resources for SCION and the RCG reimplementation? Please label the probe results as generator-internal features, since RCG's low score does not evaluate its external DINOv2 encoder. Do repeat runs support the smaller balancing/CBC gains?

  4. Can the SODA comparison specify the evaluated sampling setting and publication version, accounting for its latent/pose masking modes?

  5. The prior is trained alongside a changing encoder. Beyond the final-checkpoint diagnostic, do existing training curves or observations show how well it tracks representation drift, and whether any lag affects generation? If this has not been measured, please discuss what remains unknown about the simultaneous-training recipe.

  6. Appendix C.4 reports the unguided prior-versus-real-conditioning diagnostic. How does the precision/recall tradeoff change at the guidance settings used for the guided results? Please share measurements if available, or distinguish the observed unguided behavior from expectations about guided sampling.

8 comments

We thank the reviewer for his/her questions which have helped improve our work. We believe that our responses have addressed the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions.

Q1: transfer-8 numbers

Appendix E.2 (p.29, lines 1512–1513) [...] Please select regularization within the training data or on validation data and report corrected per-dataset and aggregate comparisons.

Was transfer regularization selected on the test set as written? [...]

A: We sincerely thank the reviewer for catching this. We have corrected it and re-run the evaluation by selecting the regularization hyperparameter based on validation data. We re-evaluated every transfer-8 number in the paper accordingly. We note that scores change by only −0.15 points on average, and no ranking or conclusion changes. In the revision, we have added Table 14 (Appendix C.7) with the per-dataset scores.

Q2:

Which split supplies the feature-standardization statistics?

A: The training split only.

Q3. Strong claim about using stop-gradient and standardization statistics.

Appendix D (p.24, lines 1288–1295) suggests that stop-gradient leaves representation content to the contrastive objective. However, CFM/CBC still update the shared encoder, as p.6, lines 319–321 correctly acknowledges. Stopping the direct conditioning path does not eliminate generative influence through shared weights. The fixed-conditioning FM/CFM decomposition does not itself establish semantic isolation during joint training. Please narrow the claim [...]

Can Appendix D distinguish the blocked conditioning path from generative influence through shared parameters?

A: We thank the reviewer for this precise point, and we agree that our claim in Appendix D is too strong. Appendix D shows that without the stop-gradient, CFM gains a gradient term from R(θ)R(\theta), which pushes ee toward a lossless code of xx. The stop-gradient removes this term. However, since the weights are shared, LCFM\mathcal{L}_{CFM} and LCBC\mathcal{L}_{CBC} still update fθf_\theta, which also computes ee, so generation can still affect the representation implicitly.

Appendix D covers only the first effect and does not establish semantic isolation. We have weakened every statement about the stop-gradient in the revision.

Q4:

[...] and make the prior input/target detachment explicit in Algorithm 1.

Does the implementation detach both the prior's noisy input and its target?

A: Yes, we detach them. We thank the reviewer for noting that Algorithm 1 does not show it. We have updated Algorithm 1 in the revision.

Q5:

Please also give the initialization and decay used for the representation-standardization statistics.

A: We believe the reviewer is referring to μ\mu and σ2\sigma^2, which are running statistics of the representation for the original view.

  • Initialization: μ=0\mu = 0 and σ2=1\sigma^2 = 1. The initial values fall below 1% of the statistics within 460 steps, well within the 5-epoch warmup.
  • Update: at every step, μ←(1−m) μ+m μbatch\mu \leftarrow (1-m)\,\mu + m\,\mu_{\text{batch}} and σ2←(1−m) σ2+m σbatch2\sigma^2 \leftarrow (1-m)\,\sigma^2 + m\,\sigma^2_{\text{batch}}, with momentum m=0.01m = 0.01 and statistics over the global batch. We use σ=σ2+10−6\sigma = \sqrt{\sigma^2 + 10^{-6}}.

We have added these details to Appendix E.1 and Algorithm 1 in the revision.

Q6: Compute, schedules, probes, and repeat runs.

What are the component update counts, view exposures, and measured training resources for SCION and the RCG reimplementation?

A: Following your comment, we have replaced "matched training budget" with "matched number of epochs" in Sections 1 and 4.

1) Component update counts. Each optimizer step updates the whole model, so the JiT (encoder and decoder) and the prior both receive 250,200 updates at 200 epochs (500,400 at 400). Our reimplemented RCG (JiT) trains its prior and then its generator, with 250,200 updates for each over 200 epochs. So the number of updates is the same for SCION and RCG (JiT).

2) View exposures. In each SCION step, every image contributes its original view and two augmented views.

3) Training resources (Appendix C.8, Table 15). For comparing training resources of SCION with RCG (JiT), we need to account for the encoder pretraining of RCG (JiT).

RCG (JiT) uses the released DINOv2-B, distilled from DINOv2-g, whose pretraining alone takes 22,016 A100 GPU-hours. On top of that encoder, the prior and generator of RCG (JiT) take 13.6 and 95.1 H200 GPU-hours (Appendix C.8, Table 15). In contrast, SCION, which needs no pretrained encoder, costs only 249.4 and 498.7 H200 GPU-hours in total at 200 and 400 epochs.

Appendix C.8 also estimates RCG with other self-supervised ViT-B encoders trained on ImageNet.

Q7:

Likewise, Table 8 matches encoder capacity but uses independently tuned contrastive-only runs with a 600-epoch cosine schedule, while SCION uses a constant learning rate.

A: SCION uses a constant learning rate following JiT, without schedule tuning. In contrast, the contrastive-only controls use a cosine schedule following prior work (e.g., SimCLR and SogCLR), as it improves their Acc.

To isolate the effect of the learning-rate schedule, we retrained the CL-only baseline with a constant learning rate under two hyperparameter settings: one using its independently tuned learning rate and weight decay, and the other using SCION’s values.

ModelBase LRWeight decayAcc. (200 / 400 ep.)Transfer-8 (200 / 400 ep.)
CL (tuned hp, Table 8)Cosine, 3e-40.171.76 / 74.1577.54 / 78.55
CL (tuned hp)Constant, 3e-40.171.11 / 71.8477.59 / 78.56
CL (SCION's hp)Constant, 1e-4071.31 / 71.9074.13 / 73.44
SCIONConstant, 1e-4071.32 / 72.3579.75 / 79.56

Under the same constant schedule and optimizer settings, SCION outperforms the contrastive-only control. SCION also achieves the highest transfer-8 across all configurations, including the independently tuned cosine controls. We have added these controls to Table 8 (Appendix C.2).

Q8:

Please label the probe results as generator-internal features, since RCG's low score does not evaluate its external DINOv2 encoder.

A: Section 4 (Evaluation) and Appendix E.2 state which features are probed, the mean-pooled patch tokens for the baselines, and the [CLS] token for SCION. We agree that this should be clearer, and we have rewritten Section 4 (Evaluation) to state this explicitly.

Q9:

Do repeat runs support the smaller balancing/CBC gains?

A: Yes, they do. Our main contribution, and the strongest driver of generative performance, is representation self-conditioning. This is consistent with prior work demonstrating the benefits of conditioning for generation. We agree that the gains from gradient balancing and CBC are comparatively smaller. CBC is an auxiliary consistency regularizer that further improves self-conditioning and is only applicable in conjunction with it, rather than being a standalone contribution.

Gradient balancing is a practical heuristic designed to avoid manually tuning the contrastive loss weight, whose appropriate value can vary throughout training and across settings. In our experiments (Table 7), it outperforms the fixed weights we evaluated without requiring additional tuning. However, more sophisticated weight schedules might perform differently. Exploring optimal strategies for balancing these objectives is an interesting direction for future work, but is beyond the scope of this paper.

Q10:

Appendix A (p.18, line 966) says SODA requires a source image for synthesis. [...] Please distinguish the source-conditioned settings evaluated there from all supported sampling modes, and clarify the scope of Tables 1/6.

A: We thank the reviewer for the correction and agree. In Appendix A, we have replaced line 966 with: "SODA moves to cross-view generation. While it can also generate without a source image by masking its representation conditioning, it reports quantitative results only for source-conditioned generation." We have also made clear that the Synth. column of Tables 1 and 6 reports "synthesis evaluated without a source image".

Q11. Prior tracking.

The prior is trained alongside a changing encoder. Beyond the final-checkpoint diagnostic, do existing training curves or observations show how well it tracks representation drift, and whether any lag affects generation? [...]

A: Thank you for this question. We have run some additional analysis to measure this.

1) Lag during training. At every saved checkpoint, we froze the model and fine-tuned only its prior on the current encoder's embeddings (100k steps, about 1 GPU-hour). We then compared unguided FID with that of the jointly trained prior at every checkpoint (Appendix C.6, Table 13b):

epochs100200300400500600700800
jointly trained prior (SCION)13.1610.109.838.928.567.977.807.67
prior fine-tuned on the frozen encoder12.789.839.158.308.137.547.417.06
difference-0.38-0.27-0.68-0.62-0.43-0.43-0.39-0.61

The jointly trained prior lags slightly, letting it catch up improves FID by 0.3 to 0.7 at every checkpoint. This gap does not grow during training, although the representation keeps changing. The diagnostic of Table 13 also gives the same picture at 200 and 400 epochs as at the end.

2) The recipe mitigating the lag. We have run an updated L-scale experiment with a smaller prior (77M parameters) which is updated three times per training step. This makes the prior keep up better, improving both unguided FID (from 6.41 to 3.85) and guided FID (from 5.29 to 3.52) at 300 epochs. Note each extra update of the prior costs about 0.1% of a training step. This is the prior of SCION (L/16) in the revision (Appendix E.1).

Q12. Precision/recall under guidance.

Appendix C.4 reports the unguided prior-versus-real-conditioning diagnostic. How does the precision/recall tradeoff change at the guidance settings used for the guided results? [...]

A: With the same protocol of Table 13, we measured precision and recall at 400 epochs, at different guidance scales (s in Equation 6), including our guided results (s=4.1) and at a low scale (s=1.2). We have added them to Table 13a. The results are shown below for the two conditioning modes: using the prior model to sample representation for conditioning and using sampled data's representation for conditioning.

conditioningguidance scalePrec. (cond. / disjoint)Rec. (cond. / disjoint)
priornone.718 / .722.524 / .510
prior1.2.740 / .744.516 / .499
prior4.1.794 / .793.490 / .469
real representationsnone.645 / .626.828 / .664
real representations1.2.675 / .646.849 / .669
real representations4.1.743 / .706.901 / .668

From the results, we observe that: 1) The unguided trade-off holds under guidance, prior samples give higher precision and real representations give higher recall.

2) Guidance raises precision with both conditionings, but it reduces recall only with the prior.

3) Guidance therefore widens the recall gap (.154 to .199), which suggests that the prior's recall might limit guided generation. Inspired by this, a simple post-hoc correction that matches the mean and covariance of prior samples to those of real representations improves recall and FID in our L-scale experiments. With guidance, it improves recall from .536 to .562, and improves FID from 3.52 to 2.53, and without guidance, it improves recall from .550 to .573, and improves FID from 3.85 to 3.32. We apply this correction to SCION (L/16) in the revision (Appendix E.2, Equation 22).

Reviewer_PGndOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the image generation and (contrastive) representation learning in one model and one framework. This paper argues that such model should be self-contained, self-supervised, single-model and single-stage. Then, the authors propose SCION, which combines JiT and SogCLR, as one model perform flow matching and contrastive learning. Also, the paper presents representation self-conditioning, where the diffused embedding is also used to guide the dense image flow matching generation. Under the same setting, this paper outperforms vanilla JiT and JiT variants with plain contrastive learning methods.

Critical strengths

This paper is the first present a method to combine the flow matching and contrastive learning on pixels within single model, reaching competitive representation capability. Under the controlled experiment on ImageNet 256x256, SCION outperforms the baselines in both generation FID and ImageNet probing accuracy. Also, this paper has done rigor ablation studies on every component, especially in the appendix, which should be appreciated.

Critical weaknesses
  1. The burdensome pipeline: it seems that the representation self-conditioning is necessary for good performance, otherwise CL only brings minor positive impact as shown in Table 5. But the representation embedding is also diffusion generated. Therefore, the whole pipeline seems burdensome and the end-to-end latency is not well documented. Also, in my view, self-conditioning is another form of naive CFG, can authors compare it with naive CFG with labels?

  2. Delving deeper, why only CL brings minor positive impact? Also, why authors want to integrate contrastive learning with flow matching generation with awareness of potential conflicts? There are many effective dense self-supervised learning methods for diffusion/flow matching, like SRA, LayerSync, and SelfFlow. Could authors please justify why contrastive learning is needed here rather than dense SSL methods?

Overall, this is a sound paper with extensive experiments, but I have concerns about its motivation on combining contrastive learning and flow matching in one model, and eventually complicates the model and pipeline.

Questions for authors

See weaknesses.

3 comments

We thank the reviewer for the thoughtful questions. We believe our responses have addressed the reviewer's concerns, and we would be grateful if the reviewer would consider raising the score. We are happy to answer any further questions. We answer each point below.

Q1.

The burdensome pipeline: it seems that the representation self-conditioning is necessary for good performance, otherwise CL only brings minor positive impact as shown in Table 5.

A: We thank the reviewer for raising this concern. We respectfully disagree with the characterization of our pipeline as burdensome. One of our goals is precisely to simplify image generator training pipelines by avoiding pretrained tokenizers/encoders and multi-stage training. These dependencies introduce complexity and can complicate adaptation to new domains, such as medical imaging. Moreover, multi-stage training is not straightforward, as better representation encoders and improved VAE reconstruction do not necessarily translate into better end-generation quality [1,2,3].

We agree that representation self-conditioning is important for achieving strong performance, consistent with prior work such as RCG. However, CL and self-conditioning serve complementary roles: CL learns useful semantic representations, while self-conditioning enables the generator to use those representations. The limited gains from CL alone in Table 5 are therefore consistent with our motivation: improving representations does not automatically translate into better generation without a mechanism that allows the generator to exploit them. Our contribution lies in coupling CL with representation self-conditioning so that the learned semantic representations can effectively guide image generation.

[1] Yao et al., Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models, CVPR 2025. [2] Yue et al., What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion, 2026. [3] Chefer et al., Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis, 2026.

Q2.

But the representation embedding is also diffusion generated.

A: Partially correct. We would like to clarify the distinction between representation used in training and during inference.

The conditioning representations used during training are not diffusion- or flow-generated. They are extracted directly from input images through the encoder’s [CLS] token and supervised by CL (Figure 3a). The learned prior therefore provides a way to sample conditioning representations for unconditional inference; it does not produce the representations used to train the image generator.

During inference when no reference image is provided, SCION uses a prior trained with flow matching to generate the conditioning representation (Figure 3b). When a reference image is provided, SCION uses its encoder representation directly for conditioning (Appendix G.1, Table 13).

Q3.

the end-to-end latency is not well documented.

A: We agree that end-to-end latency should be documented and thank the reviewer for pointing this out. We interpret this as referring to inference-time sampling latency. We benchmarked all methods on one H200 GPU (batch size 128, 50-step Heun sampler). Time and FLOPs are reported per image: conditioning includes the complete prior or encoder forward pass, while generation includes the complete sampling process.

MethodConditioning sampling (time, GFLOPs)Generator sampling (time, TFLOPs)
RCG (JiT)Prior: 0.40 ms, 12.727.4 ms, 5.00
SCION (prior sampling)Prior: 1.13 ms, 19.728.1 ms, 5.02
SCION (reference image)Encoder: 0.17 ms, 33.028.1 ms, 5.02

For SCION with prior sampling, the prior sampling accounts for only 3.9% of SCION's total sampling time and 0.4% of its FLOPs. Its latency is mainly due to limited GPU utilization, as it is an MLP operating on a single 768-d vector per image. We note that SCION's prior sampling time is larger than RCG's since they used different prior models. If we use SCION's prior model in RCG, it degrades FID, as stated in Section 4 (Baselines).

The generator sampling latency of SCION is slightly higher than RCG's, adding 2.7% latency, as it introduces one extra token, the [CLS] token, in the first eight blocks (257 vs. 256 patch tokens).

We have added this table and discussion to Appendix C.8 (Table 16). For training costs, please refer to our response to Reviewer apE9.

Q4.

in my view, self-conditioning is another form of naive CFG

A: We respectfully disagree with this. Self-conditioning is used as a substitute of the label conditioning, thus being complementary to CFG.

Self-conditioning is a factorization of the data distribution. For sampling, SCION samples p(x)=∫p(x∣e) p(e) dep(x) = \int p(x \mid e)\, p(e)\, de by first drawing a representation e∼p(e)e \sim p(e) from the prior, and then an image x∼p(x∣e)x \sim p(x \mid e). Here ee is the model's own representation, which enters the generator exactly as the class label does in class-conditional JiT.

CFG is complementary. At every step, CFG extrapolates from the unconditional toward the conditional velocity. Since p(x∣e)p(x \mid e) is a conditional model, CFG can be applied on top of it with a learned null condition. This is used in our representation guidance (Equation 6).

Q5.

can authors compare it with naive CFG with labels?

A: We did the comparison in the paper. Please refer to Table 4. With representation guidance, SCION reaches 4.20 FID, which is close to two methods using CFG with labels, i.e., CFG on JiT (3.94) and JiT + REPA (3.57). Please note that these two methods use ground-truth labels at sampling time, while SCION does not. SCION is better than RCG (JiT) guided on DINOv2 representations (5.01) without using labels.

Q6.

why authors want to integrate contrastive learning with flow matching generation with awareness of potential conflicts?

A: Our goal is to learn useful representations and leverage them for generation without relying on pretrained encoders, tokenizers, or labels. Contrastive learning (CL) learns the representations, while flow matching exploits them through representation self-conditioning. Potential conflicts are addressed in SCION, and the representation survives joint training: at 200 epochs, SCION achieves 71.32% linear-probe accuracy, close to the capacity-matched CL-only control (71.76%), while achieving better transfer-8 (79.75% vs. 77.54%; Table 5).

Moreover, sharing the encoder part of JiT for CL and FM benefits generation. We conducted an ablation using a separate encoder for CL not sharing with any parts of the JiT generator network. We trained the models together with the prior model jointly. With 400 epochs, this baseline has a FID score of 10.50, which is worse than SCION with a single model to unify encoder and generator (8.92 FID; Appendix C.3, Table 9).

Q7.

Overall, this is a sound paper with extensive experiments, but I have concerns about its motivation on combining contrastive learning and flow matching in one model, and eventually complicates the model and pipeline.

A: We thank the reviewer for the positive assessment. The motivation is addressed in Q6.

We clarify that SCION uses the standard JiT backbone (a pixel-space DiT [1]), with the only architectural addition being a concatenated [CLS] token: it supports CL, while patch tokens support flow matching. Both losses are computed as in their standalone counterparts. The only change is the use of the representation as conditioning, implemented analogously to class conditioning in JiT/DiT using the same AdaLN mechanism, but replacing the class-embedding table with the learned embedding vector.

SCION is trained in a single training stage, without pretrained encoders, tokenizers, or additional networks. At inference, SCION follows the same sampling pipeline as RCG (same JiT inference, first with the prior to obtain the representation and then to the generator). Hence, we consider the training pipeline of SCION to be simpler than multi-stage training methods and the sampling pipeline to be similar to RCG.

Q8.

There are many effective dense self-supervised learning methods for diffusion/flow matching, like SRA, LayerSync, and SelfFlow. Could authors please justify why contrastive learning is needed here rather than dense SSL methods?

A: Thank you for pointing out these works. We agree that contrastive learning is not the only option: DINO-style self-distillation or the dense objectives of SRA, LayerSync and Self-Flow could also learn the representation inside the generator. What differentiates SCION is not the self-supervised objective, but the use of the learned representation as the condition of generation, which matters much more than alignment as demonstrated in our experiments.

These prior works still rely on labels. SRA, LayerSync and Self-Flow need no external encoder for alignment, similar to SCION, but their reported results are class-conditional ImageNet generation in SD-VAE latent space. They therefore keep the dependence on external conditioning that SCION removes.

Why we used contrastive learning. It directly yields a compact global embedding, which has similar shape to the label-conditioning, whereas dense objectives are per-token. Combining dense self-supervision with SCION's generative pass is a promising direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.