acceptodds
Withdrawn from ICLR 2027

Generation-induced Representation Learning: Self-Guided Diffusion without Pretext Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative training alone can drive semantic representation learning. We demonstrate this via a new framework that jointly trains an image encoder and a conditional diffusion model end-to-end from scratch, using no labels, no contrastive loss, no masking, and no data augmentation beyond horizontal flips. The encoder outputs a latent vector that both controls conditional generation and serves as a suitable feature representation for downstream semantic tasks. An additional diffusion model trained over the encoder's output space allows for sampling of latent representations, and, chained together with the conditional model, yields a system for unconditional image generation. Key to our system's learning dynamics are independently sampled noise timesteps for the conditional model's image and latent inputs. On CIFAR-10, CIFAR-100, and ImageNet-100, our system substantially improves unconditional generation over an architecturally matched DDPM baseline with identical compute. Our central claim is one of existence and minimality: an encoder trained purely by the denoising diffusion loss, under a deliberately minimal flip-only augmentation budget, reaches kNN classification accuracy comparable to dedicated self-supervised methods such as SODA, despite using no pretext task whatsoever. Moreover, equipping our method with the same data augmentation recipe as SimCLR, so that the two differ only in their training objective, our purely generative objective outperforms SimCLR's contrastive objective on three of four probing protocols across CIFAR-10 and CIFAR-100. Together, these results establish that conditional generation, when coupled with appropriate training mechanisms, is itself a viable representation learning objective.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Representation Learning, Unconditional Generation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qpRaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GiRL jointly trains an image encoder, an image diffusion model conditioned on the encoded representation, and a latent diffusion prior. The image and latent representations are independently noised, while a stop-gradient prevents the latent-prior loss from updating the encoder. A non-affine normalization and optional early stopping of latent sampling are used to stabilize or improve the system. Experiments on CIFAR-10, CIFAR-100, and ImageNet-100 evaluate generation and learned representations, including a setting with only horizontal-flip augmentation and no separate discriminative pretext objective.

Critical strengths
  1. The method improves generation even without tuning the latent early-stop parameter. Table 2 reports FID improvements from 18.08 to 16.12 on CIFAR-10, 23.12 to 19.94 on CIFAR-100, and 77.17 to 66.39 on ImageNet-100 for s=0. The tuned variant further improves these to 14.46, 16.73, and 63.88. Reporting both settings makes it possible to distinguish the training contribution from the sampling search.
  2. The flip-only representation results and stabilization ablations are useful. Table 3 reports GiRL kNN-20 accuracy of 77.05% on CIFAR-10 and 48.91% on CIFAR-100, and kNN-200 accuracy of 44.88% on ImageNet-100 without an added discriminative objective. Table 7 demonstrates that allowing latent-prior gradients into the encoder can strongly degrade representation quality, and examines normalization choices. These results identify a concrete difficulty in simultaneous encoder/prior learning and a simple way to mitigate it.
Critical weaknesses
  1. The novelty claim and mechanism controls are insufficiently precise. SODA is labeled as using a contrastive-plus-diffusion objective, but its original method trains the encoder/decoder with a conditional denoising objective for novel-view synthesis; augmentation is not itself a contrastive loss. DiffAE also already learns an encoder through diffusion, albeit with a separately trained latent prior. GiRL's minimal augmentation and simultaneous prior are meaningful distinctions, but generation-induced representation learning itself is established. Table 6 changes both training staging and latent-noise exposure, so it does not isolate the effect of independently sampling the two noise levels.
  2. The comparisons do not establish a quality advantage at matched total compute. GiRL adds an encoder and a latent prior to the image denoiser; equal training iterations or data exposure do not equalize FLOPs. Likewise, matching the evaluated encoder architecture does not match the full representation-pretraining system. The heterogeneous GPU-time information in Appendix B does not resolve this. The limited low-resolution benchmarks and substantial gaps to stronger representation learners also restrict the scope: for example, GiRL with flip-only augmentation obtains 44.88% ImageNet-100 kNN-200 accuracy, while MoCo-v3 with strong augmentation obtains 74.70%.
  3. The paper only evaluates CIFAR-10, CIFAR-100, and ImageNet-100 at 64×64 and does not have text-to-image results.
Questions for authors
  1. Mechanism test: Under identical simultaneous training, what happens with clean latent conditioning and with synchronized image/latent noise levels, compared with independent noising? These controls would distinguish the benefit of the proposed noise design from the benefit of jointly fitting a latent prior.
  2. Comparison clarification: After correcting the SODA objective description, what evidence distinguishes GiRL from a diffusion-autoencoder or SODA-style encoder/denoiser trained with the same augmentation and budget? A matched minimal-augmentation control would be especially informative for the paper's central claim.
  3. The paper only evaluates CIFAR-10, CIFAR-100, and ImageNet-100 at 64×64. It does not test standard high-resolution ImageNet-1K at 256×256 or 512×512, where latent diffusion and representation-guided generation are most commonly evaluated.
  4. GiRL is unconditional and learns only an image-level latent prior. Text-to-image is the primary practical setting for diffusion models, and it is unclear whether the proposed stop-gradient, non-affine LayerNorm, and independent noise schedules remain necessary or effective when the generator is also conditioned on text.
1 comment
AuthorsOpenReview ↗

Response to Reviewer qpRa

We thank the reviewer for the precise and constructive feedback. We are glad that the reviewer found the flip-only representation results and the stabilization ablations useful.

SODA description and novelty. The reviewer is correct: SODA trains an encoder with a conditional denoising objective (originally for novel-view synthesis), and augmentation is not itself a contrastive loss. We thank the reviewer for catching this and will correct the SODA entry in Table~1 and the related text. We also agree that other works also explore generation-based representation learning (e.g., Diff-AE, SODA, InfoDiffusion). Our contribution is: \emph{simultaneous}, from-scratch training of the encoder, conditional denoiser, and latent prior under a single denoising loss, together with the stabilization components and the minimal-augmentation regime.

Mechanism controls for the noise design. We agree that Table 6 changes both training staging and latent-noise exposure, and so does not isolate the effect of independently sampling the two noise levels. We will say this explicitly. Clean-latent conditioning and synchronized noise levels under identical simultaneous training are the right controls, and we will list them as future work. Table~7 does show directly that allowing latent-prior gradients into the encoder strongly degrades representation quality, which supports the stop-gradient design.

Matched compute and comparison to Diff-AE/SODA-style models. The added components do not impose the overhead the concern suggests. (1) The encoder is not needed for generation. If the goal is representation learning, the output of GiRL is a standalone, efficient encoder. (2) The generator can be run without latent conditioning (s=1s=1), in which case generation cost is identical to the baseline; alternatively, it can be run with latent conditioning for improved quality. In the latter case, sampling the latent is cheap because it is a single vector, so running the auxiliary latent-prior diffusion model adds minimal overhead. We make no claim to state-of-the-art representation quality; our claim is that a purely generative objective learns useful representations, which the comparisons establish.

Resolution and text-to-image. We agree that ImageNet-1K at 256×256256\times256 or 512×512512\times512 and text-to-image generation are common diffusion settings. Due to limited computational resources, we are unable to run experiments at this scale, and our study is therefore restricted to CIFAR-10, CIFAR-100, and ImageNet-100 at resolutions up to 64×6464\times64. Within this setting, we deliberately keep the problem minimal (unconditional generation with an image-level latent) to isolate whether denoising alone can drive representation learning. The architecture and optimization strategies at the core of our method (the stop-gradient, non-affine LayerNorm, and independent noise schedules) are unrelated to image resolution.

Reviewer_rmv6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GiRL, a framework that trains an image encoder, a conditional diffusion model, and a small latent diffusion model jointly from scratch, using only the denoising diffusion loss without any labels, contrastive loss, masking, or pretrained backbone. The encoder’s latent conditions image generation, and the latent diffusion model learns to sample latents so the system can generate images unconditionally. Joint training is made stable by three choices: independent noise schedules for the image and the latent, a stop-gradient so the encoder learns only from the image loss, and a non-affine LayerNorm that fixes the latent’s scale. On CIFAR-10, CIFAR-100 and ImageNet-100, GiRL improves unconditional FID over a matched DDPM baseline. Its encoder learns useful features, but still trails stronger SSL methods like MoCo v3, BYOL and MAE. All experiments are small-scale. The core claim is that generation alone is a viable representation learning objective.

Critical strengths

The question that the paper is trying to solve is clean and the method is minimal- whether denoising alone can drive representation learning. The method answers it without adding anything that could confound the answer. There is no contrastive term, no masking, no pretrained encoder and no auxiliary regularizer. Choosing flip-only augmentation for the main results reinforces this. Whatever the encoder learns cannot be easily be credited to built-in invariances.

The training is genuinely joint and from scratch. Table 1 positions this well. RCG and REPA depend on frozen SSL encoders, Diff-AE trains its latent prior in a second stage, and InfoDiffusion adds a mutual-information regularizer. GiRL trains all three components together under one loss. If the result holds up, it removes the dependency on pretrained backbones, which matters for domains where none exist.

Critical weaknesses
  1. The experiments are small-scale only. Everything is at 64×64 or lower with small models, and there is no ImageNet-1k. SSL rankings at CIFAR scale often do not transfer to larger settings, so the paper’s broad framing of a “new paradigm” outruns the evidence.

  2. Generation gains are shown only against a small, short-trained DDPM, at absolute FIDs far from current practice, so it is unclear whether they persist at stronger baselines. The paper also does not compare against conditioning on frozen SSL features or labels in the same architecture, which would show how much of the achievable conditioning benefit joint training captures.

  3. The claim that GiRL is less augmentation-dependent than contrastive methods is never tested directly. No baseline is trained flip-only in the paper’s setting, and the comparison relies on a figure from SimCLR’s ResNet-50 ImageNet experiments.

Questions for authors

What hyperparameter search was done for SimCLR, MoCo v3, BYOL and SODA in the ViT-S/4, 32×32 setting? For example temperature, learning rate and batch size? In particular, SimCLR’s CIFAR-10 kNN accuracy (76.56%) is unusually low relative to its linear probe (84.54%), and the headline +7.2 kNN result depends on that gap. If the baselines were run at codebase defaults tuned for other architectures and resolutions, the representation comparisons would need to be read more cautiously.

1 comment
AuthorsOpenReview ↗

Response to Reviewer rmv6

We thank the reviewer for the precise questions. We are glad that the reviewer found the question clean and the joint, from-scratch training under a single loss a meaningful distinction from RCG, REPA, Diff-AE, and InfoDiffusion.

Small scale. We respectfully disagree that larger-scale experiments are needed to establish the contribution. Representation learning on CIFAR and ImageNet without data augmentation is extremely challenging: augmentation-based methods are known to depend critically on strong augmentations such as cropping and color distortion. Showing that a purely generative objective learns non-trivial features in this regime, with matched baselines in every comparison, is the core contribution, and our controlled experiments establish it. A demand for ever-larger experiments is not an appropriate standard for a new methodology to meet at introduction; much of deep learning was developed and validated at smaller scale (e.g., MNIST, CIFAR, ImageNet) before being scaled to foundation models. Our setting is chosen deliberately so that the effect of the method is isolated rather than confounded by scale.

Generation gains against a small DDPM. We respectfully disagree that these gains need to be confirmed against stronger baselines. The purpose of the generation comparison is to answer a controlled question: does conditioning on a jointly learned representation improve the generator? The matched setting (same backbone, data, and schedule) answers exactly this, and the gains are consistent across all three datasets (Table~2).

Conditioning on frozen SSL features or labels. Conditioning a diffusion model on frozen SSL features is precisely the setting studied by RCG, which we discuss in Table~1 as a closely related approach. RCG conditions generation on features from a pretrained SSL encoder and trains a separate representation generator to sample them, so it depends on a pretrained backbone that must itself be trained beforehand. GiRL addresses a different question: whether the conditioning representation can be learned from scratch, jointly with the generator, using only the denoising loss and no pretrained encoder. We therefore view the frozen-SSL setting as a reference point rather than a like-for-like baseline, since it uses substantially more supervision-free pretraining signal and an additional training stage.

Augmentation dependence. What our experiments do show directly is that GiRL learns non-trivial features with flip-only augmentation (Table~3).

Q1: Hyperparameters of the SSL baselines. Due to computational resource limitations, we did not run a dedicated hyperparameter search for SimCLR, MoCo v3, BYOL, and SODA in the ViT-S/4, 32×3232\times32 setting; we used their default hyperparameters. The representation comparisons, including the +7.2+7.2 kNN improvement over SimCLR are in this context.

Reviewer_TsK7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Generation-induced Representation Learning (GiRL), which learns image features by training an encoder through a diffusion-based generation objective. The method explores whether generative training can be a competitive alternative to dedicated self-supervised representation learning methods. It trains the encoder and diffusion models together to learn useful image features while also supporting image generation. Experiments on CIFAR-10, CIFAR-100, and ImageNet-100 show improved generation quality, measured by FID over the evaluated DDPM baseline, and demonstrate representation utility through classification using the encoder features learned from the image denoising objective.

Critical strengths
  1. Learning useful representations and generating images without labels or a pretrained encoder is a relevant self supervision task, and the paper provides an approach that trains these together.
  2. The paper evaluates both generation and representation quality. The ablations along with the matched-augmentation comparison with SimCLR, help analyze the proposed method.
Critical weaknesses

It would be helpful for readers if the following points were clarified or improved:


  1. The DDPM baseline is useful because it uses the same DiT backbone. However, it would be helpful to include a stronger diffusion or flow-based baseline, such as rectified flow, under comparable data, supervision, and computational budgets. It could help the reader confirm the efficacy of the proposed approach over these training approaches.


  2. The representation evaluation mainly considers classification on the datasets used for pre-training. A comparison with a DINOv2-style teacher-student approach under comparable data, backbone, and computational budgets would be helpful. Showing results on other task than classification like segmentation, depth estimation etc can help the claim of representation quality learned. Also showing these results on other real world image datasets would help the readers understand the efficacy of the approach.


  3. Table 3 shows a gap to MAE, including linear-probe accuracy on ImageNet-100. MAE also performs better on CIFAR when GiRL uses stronger augmentation. Although the proposed method supports both generation and representation learning, it would be helpful to discuss this and qualify the claim of closing the performance gap to dedicated representation learning methods.

Questions for authors

Please refer to Weakness section for questions.

1 comment
AuthorsOpenReview ↗

Response to Reviewer TsK7

We thank the reviewer for the constructive feedback. We are glad that the reviewer found the joint evaluation of generation and representation quality, the ablations, and the matched-augmentation comparison with SimCLR useful.

Stronger diffusion/flow baseline. The DDPM baseline shares the DiT backbone, data, and training setup with GiRL, so the comparison isolates the effect of the encoder and latent prior in this setting. We agree that a rectified-flow baseline would be a valuable complement, and we will note it as future work. GiRL's components (independent noise schedules, stop-gradient, non-affine LayerNorm) are not specific to the DDPM objective.

Teacher--student baseline, other tasks, other datasets. MoCo v3 and BYOL, which we compare against, are already momentum-based teacher--student methods. Linear probing and kNN classification on the pretraining data are the standard first test of whether a new self-supervised paradigm learns semantic features, and they directly answer the question this paper poses: can useful representations be learned purely through generation, with flip-only augmentation and no discriminative objective? Downstream and transfer tasks measure how far a mature method can be pushed; they are a natural next step once a paradigm is established.

Gap to MAE. We respectfully suggest that absolute best numbers are not the right standard for judging a methodology that differs qualitatively from prior work. MAE is a dedicated representation learner built around masking, a strong input corruption designed for feature learning, and it does not generate images. GiRL is trained with flip-only augmentation and a single denoising loss, and the same model is simultaneously an image generator (Tables 2 and 3). The significant finding is that this purely generative training produces useful features in a regime where existing self-supervised methods depend heavily on strong augmentation. Our paper introduces a new way to learn representations, not an incremental engineering effort to top benchmarks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.