- Rating
- 2(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies unifying image representation learning and generation within a single model. To achieve this, it proposes training an encoder with 3 losses (contrastive loss, flow matching loss, and conditioned-branch consistency loss). However, it also trains a separate token generator to produce semantic token, which is input to the encoder as additional condition during inference. Overall, it targets at perception and generation without any class labels in an end-to-end manner.
Critical strengths
- Unifying image representation learning and generation is an interesting and challenging topic.
- The figures are clear and easy to understand.
Critical weaknesses
- Despite introducing additional complexity, the empirical results are less persuasive as the performance falls short of the baseline's official implementations. For example, as shown in Table 10, the official implementation of RCG achieves 3.44 without guidance and 2.15 with CFG, surpassing SCION’s 3.47(w RG). This performance discrepancy makes the claims in Figure 1 somewhat misleading.
- There are actually two separate models to achieve unified representation learning and image generation, contradicting the “one model” claim of the paper.
- The key technical contritions of the paper are not clear, and crucial discussion of related work is missing from the main paper. The authors should discuss and compare with related methods in the main paper to provide a direct technical background and properly position the method within the literature. Besides, the chosen baseline RCG is somewhat outdated, omitting several recent and relevant works, such as EPG[1], SD-DiT[2], Self-Flow[3], USP[4].
[1] There is No Vae: End-to-End Pixel-Space Generative Modeling via Self-Supervised Pre-training.
[2] SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer.
[3] Self-Supervised Flow Matching for Scalable Multi-modal Synthesis.
[4] USP: Unified Self-Supervised Pre-training for Image Generation and Understanding.
Questions for authors
- Compared to the two-stage training framework [1,4], where the first stage is also self-supervised pre-training without labels, what is the primary motivation or practical necessity for the proposed end-to-end paradigm given its added training complexity? Are there essential benefits in model performance or training efficiency? What’re the differences between SCION and SD-DiT[2] consider they both use a similar training method?
- What’s the necessity of combining four distinct losses (GCL, CFM, prior, CBC) given the marginal gains in performance (e.g. FID of RCG(pixel-space ADM) v.s. SCION = 6.24 v.s. 5,89). What’s the overall training cost, measured in GPU hours, of SCION compared to RCG?
- In the main experiments, could the authors explain the rationale of adding attention residuals to JiT rather than keeping JiT’s original designs?
- What’s the size of the network used to learn prior of the semantic embeddings?
- What’s the similarity metric used for computing CBC loss in line 302?
7 comments
Q5.
The key technical contributions of the paper[...]
A: The central challenge we address is diffusion-based image generation that is simultaneously self-contained (no pretrained components), self-supervised (no labels), single-model, and single-stage, which prior approaches do not achieve together (Table 1). Our key contribution is a framework that achieves this through semantic self-conditioning: SCION's unified pixel-space architecture and training framework, which reconciles contrastive representation learning with flow matching in a shared backbone, using its own learned representations for generative conditioning.
Q6.
crucial discussion of related work is missing from the main paper.
A: We discuss related work in the introduction and Table 1 summarizes it, showing prior work does not achieve the "4s", with a more detailed discussion in Appendix A. Given your suggestion, we have incorporated a related work section into the main paper (Section 5) and included several recent works (including the ones mentioned by the reviewer).
Q7.
Besides, the chosen baseline RCG is somewhat outdated, omitting several recent and relevant works, such as EPG[1], SD-DiT[2], Self-Flow[3], USP[4].
A: RCG is the closest existing baseline to SCION, and shares the class-unconditional/representation-conditioned setting, while the four works mentioned are class-conditional. That is, RCG is particularly relevant because, like SCION, it replaces class labels with self-supervised representations for generative conditioning. RCG's conditioning comes from a pretrained model, while SCION learns its own representations. This makes it a natural controlled baseline for isolating SCION's contributions.
In contrast, the four works mentioned by the reviewer rely on class labels for conditioning, not being self-supervised. EPG and USP use self-supervised pretraining to initialize subsequent class-conditional generators, while SD-DiT and Self-Flow introduce self-supervised auxiliary objectives into class- or text-conditional generative models. Although these approaches demonstrate the complementarity of self-supervised learning and generation, they still rely on external conditioning.
Q8. Comparison with EPG, SD-DiT, Self-Flow, USP
A: Despite operating without labels, SCION compares favorably with these class-conditional methods at comparable scales. We compare their unguided generation performance reported in the tables below.
At B scale, SCION-B achieves 13.16 FID after only 100 epochs (125K steps), outperforming all four methods' B-scale results (22.10–28.62), despite using substantially fewer optimization steps.
At L/XL scale, the submitted SCION-L (5.89 FID, 400 epochs) outperforms EPG, SD-DiT, and USP, while approaching Self-Flow (5.70 FID), which uses approximately 8× more optimization steps. Furthermore, our new SCION-L variant achieves 3.32 FID and 171.6 IS after only 300 epochs (375K steps), surpassing all four methods' reported results without using class labels or pretrained components.
We have added these works to the related work (Section 5 and Appendix A) and these comparisons to Table 12 (Appendix C.5).
B scale
| method | labels | params | batch | epochs | steps | FID | IS |
|---|---|---|---|---|---|---|---|
| EPG-B/16 | ✓ | 229M | 1024 | n.r. + 240 | n.r. + 300K | 25.1 | 54.3 |
| SD-DiT-B/2 | ✓ | n.r. | 256 | 80 | 400K | 28.62 | n.r. |
| USP (SiT-B/2) | ✓ | 130M | 4096 / 256 | 1600 + 80 | 500K + 400K | 22.10 | 61.2 |
| SCION-B | ✗ | 231M | 1024 | 100 | 125K | 13.16 | 81.9 |
| SCION-B | ✗ | 231M | 1024 | 400 | 500K | 8.92 | 118.4 |
| SCION-B | ✗ | 231M | 1024 | 800 | 1M | 7.67 | 127.6 |
L/XL scale
| method | labels | params | batch | epochs | steps | FID | IS |
|---|---|---|---|---|---|---|---|
| EPG-XL/16 | ✓ | 583M | 1024 | 480 + 800 | 600K + 1M | 6.6 | 141.1 |
| SD-DiT-XL/2 | ✓ | 741M | 256 | 480 | 2.4M | 7.21 | 144.7 |
| SD-DiT-XL/2 | ✓ | 741M | 256 | 700 | 3.5M | 6.74 | n.r. |
| Self-Flow (SiT-XL/2) | ✓ | 675M | 256 | 800 | 4M | 5.70 | 151.4 |
| USP (SiT-XL/2) | ✓ | 675M | 4096 / 256 | 1600 + 400 | 500K + 2M | 6.96 | 141.5 |
| SCION-L (submission) | ✗ | 559M | 1024 | 400 | 500K | 5.89 | 151.6 |
| SCION-L (revision, see Q2) | ✗ | 536M | 1024 | 300 | 375K | 3.32 | 171.6 |
Q9.
Compared to the two-stage training framework [1,4] [...] what is the primary motivation or practical necessity for the proposed end-to-end paradigm given its added training complexity? Are there essential benefits in model performance or training efficiency?
A:
1) SCION vs [1,4]. As discussed in Q8, the main benefit of SCION is it outperforms both methods while being end-to-end self-supervised and label-free generation, as opposed to [1,4] which require labels for training and inference.
That is, SCION fundamentally differs from [1,4] by learning its own representations for generative conditioning, enabling label-free generation. Although the cited approaches use self-supervised pretraining, their generators still rely on class labels, and thus are not self-supervised end-to-end.
2) SCION vs multi-stage. The multi-stage alternative to SCION is exactly RCG, which separately trains a representation encoder, prior, and representation-conditioned generator. As shown in Tables 2, 3, and 10, SCION outperforms this multi-stage variant. As discussed in Q3, SCION also simplifies the overall training pipeline by avoiding separate stages and their associated tuning decisions.
Moreover, sharing the encoder part of JiT for CL and FM benefits generation. We conducted an ablation using a separate encoder for CL not sharing with any parts of the JiT generator network. We trained the models together with the prior model jointly. With 400 epochs, this baseline has a FID score of 10.50, which is worse than SCION with a single model to unify encoder and generator (8.92 FID; Appendix C.3, Table 9).
Q10.
What's the overall training cost, measured in GPU hours, of SCION compared to RCG?
A: For comparing training resources of SCION with RCG (JiT), we need to account for the encoder pretraining of RCG (JiT). RCG (JiT) uses the released DINOv2-B, distilled from DINOv2-g, whose pretraining alone takes 22,016 A100 GPU-hours. On top of that encoder, the prior and generator of RCG (JiT) take 13.6 and 95.1 H200 GPU-hours (Appendix C.8, Table 15). In contrast, SCION, which needs no pretrained encoder, costs only 249.4 and 498.7 H200 GPU-hours at 200 and 400 epochs. At 200 epochs, SCION achieves better generative performance with fewer total FLOPs and GPU-hours than RCG after accounting for its encoder pretraining time (Appendix C.8, Table 15).
Appendix C.8 also estimates RCG with other ViT-B encoders trained on ImageNet.
Q11
What're the differences between SCION and SD-DiT[2] consider they both use a similar training method?
A: SCION differs substantially from SD-DiT. While both combine self-supervised discrimination with generative training, the 3 main differences are:
(1) Role of representations: self-conditioning vs class-conditioned. SD-DiT uses discrimination as an auxiliary objective for class-conditional generation, without using its learned representations for generative conditioning or evaluating them for downstream understanding. SCION instead leverages it by replacing class labels with its own learned representations for conditioning.
(2) Training dependencies: no dependencies vs label- and VAE-dependent. SD-DiT operates in the latent space of a pretrained VAE and relies on class labels, whereas SCION operates directly in pixel space, without pretrained components or labels.
(3) Self-supervised objective: contrastive learning vs self-distillation. SD-DiT applies teacher-student distillation to noised views, while SCION instead applies contrastive learning to clean, augmented views, without a teacher network or masking.
We believe that our responses have addressed the reviewer's concerns, and we would be grateful if the reviewer would consider re-evaluating our paper. We are happy to answer any further questions.
Q1.
A: The results in Figure 1 and Table 10 for RCG are two different settings. Figure 1 presents a controlled comparison (apples to apples), corresponding to Tables 2 and 4, where all methods use the same JiT-B/16 backbone, data, training recipe, and epochs. RCG is a representation-conditioning framework that can be combined with different generators. Here, we combine it with JiT (same backbone as SCION) to remove backbone differences. Under this setting, SCION outperforms RCG both without guidance and with guidance, while also learning substantially stronger representations. The official RCG results in Table 10 cited by the reviewer instead use a different generator, MAGE-L, trained on the tokens of a pretrained VQGAN whose perceptual loss relies on a network trained with labels. We have clarified the controlled setting in Figure 1's caption.
Q2.
A: Table 10 of the submission, provided in the appendix for reference, includes results from both pixel-space and latent-space generative models. In particular, the RCG (MAGE-L) results cited by the reviewer (3.44/2.15 FID) use the latent-space MAGE generator with a pretrained tokenizer, which differs from our pixel-space setting. The relevant official pixel-space comparison is RCG (ADM), reported in Table 3, which achieves 6.24 FID with ~617M parameters. In the submission, SCION (L/16) already improved upon this result with fewer parameters (559M), achieving 5.89 FID without guidance.
Furthermore, following the reviewer, we trained an improved SCION (L/16) with ~536M parameters and only 300 epochs. It achieves 3.32 FID without guidance and 2.53 with representation guidance. Remarkably, this even surpasses the unguided result of the latent-space RCG (MAGE-L) (3.44), while approaching its guided result (2.15), even though SCION operates entirely in pixel space, without pretrained components or labels.
Q3.
A: We respectfully disagree when considering the complete training pipeline. One of SCION's goals is precisely to simplify image generation by avoiding pretrained tokenizers/encoders and multi-stage training. Following the reviewer's example, RCG (MAGE-L) requires a pretrained VQGAN tokenizer (with adversarial and LPIPS networks and losses) and MoCo v3 encoder training, followed by separate prior and generator training, whereas SCION drops the tokenizer and trains jointly from scratch in a single stage.
Moreover, multi-stage training introduces additional tuning decisions, such as allocating training budgets and selecting intermediate checkpoints, whose quality may not predict downstream generation performance [1,2]. These dependencies can also complicate adaptation to new domains where pretrained components may be unavailable. Thus, while SCION's training step is more involved than RCG's generator step, its overall pipeline is much simpler.
Thus, SCION improves upon RCG under controlled conditions and achieves strong performance against official pixel-space baselines, while substantially simplifying the overall training pipeline.
[1] Yao et al., Reconstruction vs. Generation: Taming Optimization Dilemma in Latent Diffusion Models, CVPR 2025. [2] Yue et al., What Matters for Diffusion-Friendly Latent Manifold? Prior-Aligned Autoencoders for Latent Diffusion, 2026.
Q4.
A: We apologize for the confusion. The encoder and decoder of SCION are not two separate models, but two views of the layers of a single JiT model. The encoder is the first 8 of the 12 Transformer blocks of JiT-B/16, and the decoder is the remaining 4 blocks. The first blocks provide the representation (the [CLS] token after the encoder blocks), while the full model generates images. This is how a single model unifies representation learning and image generation. We have made this explicit in Section 3 and in Figures 2 and 3.