- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GiRL jointly trains an image encoder, an image diffusion model conditioned on the encoded representation, and a latent diffusion prior. The image and latent representations are independently noised, while a stop-gradient prevents the latent-prior loss from updating the encoder. A non-affine normalization and optional early stopping of latent sampling are used to stabilize or improve the system. Experiments on CIFAR-10, CIFAR-100, and ImageNet-100 evaluate generation and learned representations, including a setting with only horizontal-flip augmentation and no separate discriminative pretext objective.
Critical strengths
- The method improves generation even without tuning the latent early-stop parameter. Table 2 reports FID improvements from 18.08 to 16.12 on CIFAR-10, 23.12 to 19.94 on CIFAR-100, and 77.17 to 66.39 on ImageNet-100 for s=0. The tuned variant further improves these to 14.46, 16.73, and 63.88. Reporting both settings makes it possible to distinguish the training contribution from the sampling search.
- The flip-only representation results and stabilization ablations are useful. Table 3 reports GiRL kNN-20 accuracy of 77.05% on CIFAR-10 and 48.91% on CIFAR-100, and kNN-200 accuracy of 44.88% on ImageNet-100 without an added discriminative objective. Table 7 demonstrates that allowing latent-prior gradients into the encoder can strongly degrade representation quality, and examines normalization choices. These results identify a concrete difficulty in simultaneous encoder/prior learning and a simple way to mitigate it.
Critical weaknesses
- The novelty claim and mechanism controls are insufficiently precise. SODA is labeled as using a contrastive-plus-diffusion objective, but its original method trains the encoder/decoder with a conditional denoising objective for novel-view synthesis; augmentation is not itself a contrastive loss. DiffAE also already learns an encoder through diffusion, albeit with a separately trained latent prior. GiRL's minimal augmentation and simultaneous prior are meaningful distinctions, but generation-induced representation learning itself is established. Table 6 changes both training staging and latent-noise exposure, so it does not isolate the effect of independently sampling the two noise levels.
- The comparisons do not establish a quality advantage at matched total compute. GiRL adds an encoder and a latent prior to the image denoiser; equal training iterations or data exposure do not equalize FLOPs. Likewise, matching the evaluated encoder architecture does not match the full representation-pretraining system. The heterogeneous GPU-time information in Appendix B does not resolve this. The limited low-resolution benchmarks and substantial gaps to stronger representation learners also restrict the scope: for example, GiRL with flip-only augmentation obtains 44.88% ImageNet-100 kNN-200 accuracy, while MoCo-v3 with strong augmentation obtains 74.70%.
- The paper only evaluates CIFAR-10, CIFAR-100, and ImageNet-100 at 64×64 and does not have text-to-image results.
Questions for authors
- Mechanism test: Under identical simultaneous training, what happens with clean latent conditioning and with synchronized image/latent noise levels, compared with independent noising? These controls would distinguish the benefit of the proposed noise design from the benefit of jointly fitting a latent prior.
- Comparison clarification: After correcting the SODA objective description, what evidence distinguishes GiRL from a diffusion-autoencoder or SODA-style encoder/denoiser trained with the same augmentation and budget? A matched minimal-augmentation control would be especially informative for the paper's central claim.
- The paper only evaluates CIFAR-10, CIFAR-100, and ImageNet-100 at 64×64. It does not test standard high-resolution ImageNet-1K at 256×256 or 512×512, where latent diffusion and representation-guided generation are most commonly evaluated.
- GiRL is unconditional and learns only an image-level latent prior. Text-to-image is the primary practical setting for diffusion models, and it is unclear whether the proposed stop-gradient, non-affine LayerNorm, and independent noise schedules remain necessary or effective when the generator is also conditioned on text.
Response to Reviewer qpRa
We thank the reviewer for the precise and constructive feedback. We are glad that the reviewer found the flip-only representation results and the stabilization ablations useful.
SODA description and novelty. The reviewer is correct: SODA trains an encoder with a conditional denoising objective (originally for novel-view synthesis), and augmentation is not itself a contrastive loss. We thank the reviewer for catching this and will correct the SODA entry in Table~1 and the related text. We also agree that other works also explore generation-based representation learning (e.g., Diff-AE, SODA, InfoDiffusion). Our contribution is: \emph{simultaneous}, from-scratch training of the encoder, conditional denoiser, and latent prior under a single denoising loss, together with the stabilization components and the minimal-augmentation regime.
Mechanism controls for the noise design. We agree that Table 6 changes both training staging and latent-noise exposure, and so does not isolate the effect of independently sampling the two noise levels. We will say this explicitly. Clean-latent conditioning and synchronized noise levels under identical simultaneous training are the right controls, and we will list them as future work. Table~7 does show directly that allowing latent-prior gradients into the encoder strongly degrades representation quality, which supports the stop-gradient design.
Matched compute and comparison to Diff-AE/SODA-style models. The added components do not impose the overhead the concern suggests. (1) The encoder is not needed for generation. If the goal is representation learning, the output of GiRL is a standalone, efficient encoder. (2) The generator can be run without latent conditioning (s=1), in which case generation cost is identical to the baseline; alternatively, it can be run with latent conditioning for improved quality. In the latter case, sampling the latent is cheap because it is a single vector, so running the auxiliary latent-prior diffusion model adds minimal overhead. We make no claim to state-of-the-art representation quality; our claim is that a purely generative objective learns useful representations, which the comparisons establish.
Resolution and text-to-image. We agree that ImageNet-1K at 256×256 or 512×512 and text-to-image generation are common diffusion settings. Due to limited computational resources, we are unable to run experiments at this scale, and our study is therefore restricted to CIFAR-10, CIFAR-100, and ImageNet-100 at resolutions up to 64×64. Within this setting, we deliberately keep the problem minimal (unconditional generation with an image-level latent) to isolate whether denoising alone can drive representation learning. The architecture and optimization strategies at the core of our method (the stop-gradient, non-affine LayerNorm, and independent noise schedules) are unrelated to image resolution.