acceptodds
Withdrawn from ICLR 2027

Self-Supervised Representation Learning as Conditional Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Self-supervised representation learning (SSL) typically learns from view consistency or masked modeling by matching, reconstructing, or predicting specified targets. However, the relationship between an observation and its target can inherently be one-to-many: multiple semantically valid representations or visual contents may be plausible given the same observation, yet existing SSL objectives do not explicitly model this conditional uncertainty. We introduce CondGen, a new formulation of SSL as conditional generation in representation space, which models the conditional distribution \(p_\theta(z_i\mid z_j)\) between representations of different views using diffusion models. By explicitly modeling this distribution, CondGen captures the one-to-many relationship across views rather than enforcing consistency with a specified target. To further incorporate global semantic structure, we introduce Decoupled Sinkhorn Clustering, which learns balanced semantic prototypes independently of encoder optimization and uses them to refine target representations for conditional generation. Extensive experiments show that CondGen learns robust, semantically structured, and spatially coherent representations across diverse tasks, including shape/structural robustness, image generation, video object segmentation, and 3D correspondence estimation. In particular, CondGen even significantly surpasses DINOv3 on the unsupervised object discovery task, despite its substantially larger-scale pretraining, demonstrating the potential of conditional representation modeling as a new paradigm for scalable self-supervised learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised Learning, Diffusion, Conditional Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_BfraOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CondGen, a self-supervised representation learning method that formulates learning across augmented views as conditional generation. It uses conditional diffusion to predict a target view’s representation from another view’s representation. A decoupled Sinkhorn clustering module supplies semantic prototypes to refine the diffusion target without backpropagating the clustering loss through the encoder. After pretraining on ImageNet, CondGen achieves competitive results on structural robustness, unsupervised object discovery, image generation, video object segmentation, and 3D correspondence tasks. Its ViT-B/16 model obtains CorLoc scores of 67.3%, 72.3%, and 57.0% on VOC07, VOC12, and COCO-20k, respectively. These results demonstrate that CondGen learns transferable visual representations, although they do not directly establish whether it captures the proposed one-to-many conditional distribution.

Critical strengths
  1. An interesting research perspective. The paper reframes cross-view representation learning as conditional distribution modeling and combines conditional diffusion with semantic prototypes. This offers a useful way to examine whether matching a single target is sufficient, although the proposed one-to-many mechanism still needs direct validation.

  2. Broad downstream evaluation. The paper assesses the learned representations across structural robustness, unsupervised object discovery, image generation, video object segmentation, and 3D correspondence. The results are particularly strong on structural robustness and object discovery, providing evidence that the representations transfer across different tasks.

Critical weaknesses
  1. The need for one-to-many modeling is not established. The paper does not demonstrate that a given view admits multiple meaningfully different target representations, or that predicting a single target is inadequate. Its central motivation therefore remains an assumption.

  2. The performance gains cannot be attributed to conditional distribution modeling. The downstream results show that CondGen learns useful representations, but the paper neither directly evaluates the learned conditional distribution nor provides a controlled comparison with deterministic prediction. Its core mechanism has therefore not been isolated experimentally.

Questions for authors

1.Evidence for the premise: What empirical evidence shows that a fixed input view has multiple meaningfully distinct, valid target representations? In particular, how do you distinguish this uncertainty from variation introduced solely by the augmentation procedure?

2.Attribution of the gains: Have you compared CondGen with a deterministic cross-view predictor while holding the encoder, augmentations, EMA target, prototypes, training data, and compute budget fixed? The outcome of this comparison would materially affect whether the reported gains can be attributed to conditional distribution modeling.

3.Validation of the learned distribution: Do you have evidence that multiple samples conditioned on the same representation cover the corresponding valid target representations? Downstream accuracy alone does not establish that the proposed conditional distribution has been learned.

4.Clarification of the reported method and results: Please clarify the product upper limit in Equation (5), provide the values of the prototype mixing weight (\gamma), clustering temperature, EMA coefficient, and noise schedule, and correct the negative CorLoc entry in Table 11. These details affect interpretation and reproducibility.

6 comments

Q1: The need for one-to-many modeling is not established.

We thank the reviewer. We specifically consider one-to-many uncertainty in cross-view representation learning. Contrastive and feature-matching/prediction objectives encourage agreement between views. However, a partial source view does not uniquely determine what another view reveals: cropping may remove informative object parts or surrounding context, leaving multiple plausible target representations.

The augmentation procedure induces this uncertainty through partial observation. Our motivation concerns the resulting variability in visible content, beyond sensitivity to photometric perturbations. CondGen explicitly explores modeling this conditional target distribution through diffusion.

During rebuttal, we additionally assess its practical value through complementary experiments: (i) comparison with matched deterministic feature prediction, (ii) analysis of representation gains across uncertainty groups, (iii) condition-perturbation tests examining whether the denoiser uses the source representation, and (iv) conditional target-distribution evaluation measuring distributional agreement and diversity. Together with the downstream evaluations, these analyses support the usefulness of conditional distribution modeling in this setting.

Q2: Comparisons with deterministic prediction.

(I) Effectiveness of conditional diffusion modeling. We only replace the diffusion head with an MLP prediction head while keeping all other settings—including EMA, augmentations, prototype refinement, and training budget—the same as CondGen.

MethodLinear accuracy (%)
CondGen79.0
Deterministic cross-view prediction77.1
Shuffled or zeroed conditioningaround 0.2

Table R1. Comparison with deterministic cross-view prediction and conditioning-removal controls.

(II) Benefits from uncertainty modeling.

To examine whether CondGen’s representation gains are associated with cross-view uncertainty, we stratify ImageNet validation images using a fixed, independently pretrained ResNet-50 encoder from PyTorch. For each image xx, we sample M=32M=32 augmented views using the pretraining augmentation pipeline and extract their L2L_2-normalized features umu_m. We define an augmentation-induced uncertainty proxy as

\qquad \bar{u} = \frac{1}{M} \sum^{M}_{m=1} u_m$$ We divide images into three equal-sized groups with low, medium, and high uncertainty. These assignments are fixed and shared across methods, avoiding dependence on the representations being compared. Within each group, we evaluate frozen-feature linear-probe accuracy for CondGen and the matched deterministic cross-view baseline, using identical evaluation preprocessing and linear-probe training protocols. We report the accuracy on different uncertainty group images. CondGen’s larger gains in higher-uncertainty groups provide empirical support for the benefits of explicitly modeling one-to-many cross-view uncertainty. | Uncertainty | Images | DINO-B | Deterministic | CondGen (Ours) | |:--|--:|--:|--:|--:| | Low | 16,667 | **82.8** | 82.2 | 79.3 | | Medium | 16,667 | 77.1 | 75.6 | **79.3** | | High | 16,666 | 74.3 | 73.3 | **78.6** | | Overall | 50,000 | 78.0 | 77.1 | **79.0** | *Table R17. Linear-probe accuracy (%) across uncertainty groups defined by cross-view feature dispersion from a fixed PyTorch-pretrained ResNet-50. All evaluated methods use ViT-B encoders.*

Q3: Conditional target-distribution evaluation.

We evaluate conditional generation in the frozen CondGen target space. For each held-out image, we fix one conditioning view, generate multiple target features from independent Gaussian noise, and compare them with prototype-refined EMA features of independently augmented target views. We report the energy score (ES; lower is better), which evaluates distributional agreement while accounting for generated diversity.

For a fixed conditioning view cc, let skk=1K\\{s_k\\}_{k=1}^{K} denote independently generated target features and ymm=1M\\{y_m\\}_{m=1}^{M} the prototype-refined features of independently augmented target views. We evaluate their distributional agreement using the energy score:

ES^(c)=1KM∑k=1K∑m=1M∥sk−ym∥2−12K(K−1)∑k≠ℓ∥sk−sℓ∥2.\widehat{\mathrm{ES}}(c)=\frac{1}{KM}\sum_{k=1}^{K}\sum_{m=1}^{M}\lVert s_k-y_m\rVert_2-\frac{1}{2K(K-1)}\sum_{k\neq\ell}\lVert s_k-s_\ell\rVert_2.

The first term measures distance to observed targets, while the second accounts for generated spread. Lower scores indicate better distributional prediction. We use unsquared Euclidean distances and average scores across held-out images. For the collapsed-mean control, the predictive distribution contains a single point, so the spread term is zero.

As shown in the table below, correctly conditioned samples achieve an ES of 0.3210, compared with 0.4210 when the generated ensemble is collapsed to its mean, a 23.8% reduction. This control indicates that retaining the predicted diversity improves distributional prediction over the corresponding point forecast. Generated feature variance also tracks augmentation-induced target variance across images, with Spearman correlation 0.7264.

Random and same-class condition shuffling increase ES to 3.7498 and 1.5939, respectively, despite similar average generated variances. Thus, diversity alone does not explain the favorable score: the samples must also agree with the targets associated with the conditioning image. Together, these results directly support condition-dependent feature prediction with meaningful target diversity, complementing the matched deterministic-prediction and representation-quality evaluations.

Conditioning / controlES ↓VgenV_{\mathrm{gen}}ρ\rho
Correct0.32100.15940.7264
Randomly shuffled3.74980.1586-0.0292
Same-class shuffled1.59390.15530.2566
Collapsed conditional mean0.42100.0000—

Table R18. Conditional target-distribution evaluation. VgenV_{\mathrm{gen}} denotes generated feature variance; ρ\rho is its Spearman correlation with real target variance across images. Mean real target variance is 0.1992. The collapsed-mean control averages the correct-condition samples.

Q4: Clarification of the reported method and results.

(1) Equation (5). The product should extend to the current diffusion step mm, rather than TT: αˉm=∏k=1m(1−βk).\bar \alpha_m = \prod^{m}_{k=1}(1-\beta_k).

(2) Hyperparameters. The prototype mixing weight is γ=0.5\gamma=0.5. Both the clustering prediction temperature and target-refinement temperature are 0.2. Following DINO, the momentum-teacher EMA coefficient starts at 0.996 and increases toward 1.0 following a cosine schedule. Diffusion uses T=1000 steps with a cosine noise schedule, offset 0.008, and maximum per-step variance 0.999. Training timesteps are sampled uniformly.

(3) Table 11. The negative CorLoc entry for CondGen ViT-B/16 on COCO20K at threshold 0.35 is erroneous. We will replace −50.29-50.29 with 50.29.

We will incorporate these corrections and explicit settings into the revised manuscript.

Thank you for the detailed response and additional experiments. The deterministic prediction baseline, uncertainty-stratified analysis, and conditional distribution evaluation substantially address my original concerns. However, several concerns remain:

  1. The attribution of representation gains to conditional uncertainty modeling remains incomplete. Replacing the diffusion head with an MLP changes both the prediction mechanism and the training objective. The response does not sufficiently specify the deterministic baseline’s architecture, loss, normalization, tuning procedure, or how the training budget was matched. Moreover, the uncertainty proxy measures feature dispersion across augmentations of an image, rather than target uncertainty conditioned on a fixed source view. The stratified results are therefore suggestive, but do not isolate the proposed explanation.
  2. Eq.(13) and Eq.(14) assert that the projected gradients are opposite multiples of the same vector, without deriving this relationship from the actual objectives. The qualitative descriptions of preserving versus suppressing instance-level information do not establish these gradient directions. Consequently, the negative alignment in Eq.(15) follows from an asserted relationship rather than a demonstrated property of the method. Moreover, fig.2 supports the practical benefit of decoupling under the tested configuration, but does not independently establish the claimed gradient-conflict mechanism.
  3. The training description assigns crop scales of (0.08, 0.4) to global views and (0.4, 1.0) to local views. Please clarify whether these ranges were reversed in the text or used in the implementation, since the augmentation configuration is central to both the motivation and experimental interpretation. I acknowledge the substantial improvements made during rebuttal. Nevertheless, the remaining attribution and theoretical concerns leave me insufficiently convinced to change my score.
Reviewer_UxZZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper formulates self-supervised visual representation learning as conditional generation in representation space. Given two augmented views, an online encoder supplies the conditioning representation and an EMA encoder supplies the target representation. The method additionally trains a Sinkhorn clustering head without propagating its gradients into the encoder; prototype reconstructions are mixed with EMA representations to form refined diffusion targets. ViT-S/B models are pretrained on ImageNet and evaluated through linear probing, distribution-shift and shape-robustness benchmarks, image generation, unsupervised object discovery, DAVIS video segmentation, and 3D correspondence.

Critical strengths
  1. Representation-to-representation conditional diffusion is a conceptually simple way to reconsider cross-view SSL beyond point prediction.

  2. The paper evaluates semantic, robustness, generative, object-level, temporal, and geometric properties rather than relying solely on ImageNet linear probing.

Critical weaknesses
  1. The diffusion architecture can represent a conditional distribution, but the experiments do not establish that the learned encoder captures conditional multimodality or that the denoiser uses its condition. This affects the paper’s principal claim that performance arises from explicitly modeling one-to-many uncertainty. The results currently show that the complete training recipe learns useful features, not that conditional distribution modeling is responsible.

  2. Table 6 appears to select test-set-specific TokenCut thresholds, while protocol equivalence with DINOv3 is unclear.

  3. Section 2 and Table 1 do not engage with the closest diffusion-based representation-learning and cross-view prediction literature. Section 2 and Table 1. Relevant related directions include Diffusion Autoencoders, diffusion-based representation learning, SODA’s diffusion-conditioned self-supervised representation learning, and SiameseIM’s cross-view representation prediction. These methods are not necessarily identical to CondGen, but they complicate the blanket assertion that previous approaches do not model one-to-many structure.

Questions for authors
  1. How do downstream results change when zjz_j is shuffled, zeroed, or removed, and when diffusion is replaced by a capacity-matched deterministic predictor?
  2. What mechanism prevents encoder collapse or condition-ignoring solutions? Can you report feature variance/effective rank and the exact gradient and stop-gradient graph?
  3. Can Proposition 1 be derived without assuming the antiparallel gradients in Eqs. 13–14, and do measured gradient cosine similarities support it?
  4. How was τ\tau selected for each object-discovery dataset, and was DINOv3 rerun using exactly the same TokenCut pipeline?
  5. Are the global/local crop ranges reversed in the text? Please provide the complete diffusion, clustering, augmentation, and optimization configuration.
7 comments

Q1: Comparisons with deterministic prediction.

(I) Effectiveness of conditional diffusion modeling. We only replace the diffusion head with an MLP prediction head while keeping all other settings—including EMA, augmentations, prototype refinement, and training budget—the same as CondGen.

MethodLinear accuracy (%)
CondGen79.0
Deterministic cross-view prediction77.1
Shuffled or zeroed conditioningaround 0.2

Table R10. Comparison with deterministic cross-view prediction and conditioning-removal controls.

(II) Benefits from uncertainty modeling.

To examine whether CondGen’s representation gains are associated with cross-view uncertainty, we stratify ImageNet validation images using a fixed, independently pretrained ResNet-50 encoder from PyTorch. For each image xx, we sample M=32M=32 augmented views using the pretraining augmentation pipeline and extract their L2L_2-normalized features umu_m. We define an augmentation-induced uncertainty proxy as

\qquad \bar{u} = \frac{1}{M} \sum^{M}_{m=1} u_m$$ We divide images into three equal-sized groups with low, medium, and high uncertainty. These assignments are fixed and shared across methods, avoiding dependence on the representations being compared. Within each group, we evaluate frozen-feature linear-probe accuracy for CondGen and the matched deterministic cross-view baseline, using identical evaluation preprocessing and linear-probe training protocols. We report the accuracy on different uncertainty group images. CondGen’s larger gains in higher-uncertainty groups provide empirical support for the benefits of explicitly modeling one-to-many cross-view uncertainty. | Uncertainty | Images | DINO-B | Deterministic | CondGen (Ours) | |:--|--:|--:|--:|--:| | Low | 16,667 | **82.8** | 82.2 | 79.3 | | Medium | 16,667 | 77.1 | 75.6 | **79.3** | | High | 16,666 | 74.3 | 73.3 | **78.6** | | Overall | 50,000 | 78.0 | 77.1 | **79.0** | *Table R11. Linear-probe accuracy (%) across uncertainty groups defined by cross-view feature dispersion from a fixed PyTorch-pretrained ResNet-50. All evaluated methods use ViT-B encoders.* *Q2: Reliance on conditions.* We directly test condition usage on held-out ImageNet validation images, with all model parameters frozen. For each image, we sample a source–target view pair using the pretraining augmentation pipeline and construct the prototype-refined target $\hat z_i$ following Eq.~(9). We evaluate noise-prediction MSE under three conditioning settings: *correct*, using a source view from the same image; *randomly shuffled*, using a source view from a different image; and *same-class shuffled*, using a source view from a different image of the same class. Class labels are used only to construct the same-class evaluation pairs. Across all settings, we hold the target representation, sampled noise, diffusion timestep, and chunk indices fixed, changing only the conditioning representation. As shown in the Table, correct conditioning achieves an MSE of $0.0124$, whereas random and same-class shuffling increase the error to $0.1560$ and $0.0682$, respectively---approximately $12.6$ and $5.5$ times the error under correct conditioning. The substantial degradation even when class identity is preserved provides direct evidence that the denoiser uses predictive information specific to the conditioning image beyond class-level semantics. These results establish useful condition dependence, complementing the separate evaluations of conditional target diversity and representation quality. | Conditioning setting | Noise MSE ↓ | |:--|--:| | Correct | **0.0124** | | Randomly shuffled | 0.1560 | | Same-class shuffled | 0.0682 | *Table R12. Noise-prediction MSE under different conditioning settings. Targets, sampled noise, timesteps, and chunk indices are identical across settings; only the conditioning representation changes.*

Q3: Evidence for the one-to-many interpretation.

We evaluate conditional generation in the frozen CondGen target space. For each held-out image, we fix one conditioning view, generate multiple target features from independent Gaussian noise, and compare them with prototype-refined EMA features of independently augmented target views. We report the energy score (ES; lower is better), which evaluates distributional agreement while accounting for generated diversity.

For a fixed conditioning view cc, let skk=1K\\{s_k\\}_{k=1}^{K} denote independently generated target features and ymm=1M\\{y_m\\}_{m=1}^{M} the prototype-refined features of independently augmented target views. We evaluate their distributional agreement using the energy score:

ES^(c)=1KM∑k=1K∑m=1M∥sk−ym∥2−12K(K−1)∑k≠ℓ∥sk−sℓ∥2.\widehat{\mathrm{ES}}(c)=\frac{1}{KM}\sum_{k=1}^{K}\sum_{m=1}^{M}\lVert s_k-y_m\rVert_2-\frac{1}{2K(K-1)}\sum_{k\neq\ell}\lVert s_k-s_\ell\rVert_2.

The first term measures distance to observed targets, while the second accounts for generated spread. Lower scores indicate better distributional prediction. We use unsquared Euclidean distances and average scores across held-out images. For the collapsed-mean control, the predictive distribution contains a single point, so the spread term is zero.

As shown in the table below, correctly conditioned samples achieve an ES of 0.3210, compared with 0.4210 when the generated ensemble is collapsed to its mean, a 23.8% reduction. This control indicates that retaining the predicted diversity improves distributional prediction over the corresponding point forecast. Generated feature variance also tracks augmentation-induced target variance across images, with Spearman correlation 0.7264.

Random and same-class condition shuffling increase ES to 3.7498 and 1.5939, respectively, despite similar average generated variances. Thus, diversity alone does not explain the favorable score: the samples must also agree with the targets associated with the conditioning image. Together, these results directly support condition-dependent feature prediction with meaningful target diversity, complementing the matched deterministic-prediction and representation-quality evaluations.

Conditioning / controlES ↓VgenV_{\mathrm{gen}}ρ\rho
Correct0.32100.15940.7264
Randomly shuffled3.74980.1586-0.0292
Same-class shuffled1.59390.15530.2566
Collapsed conditional mean0.42100.0000—

Table R13. Conditional target-distribution evaluation. VgenV_{\mathrm{gen}} denotes generated feature variance; ρ\rho is its Spearman correlation with real target variance across images. Mean real target variance is 0.1992. The collapsed-mean control averages the correct-condition samples.

Q4: Table 6 appears to select test-set-specific TokenCut thresholds, while protocol equivalence with DINOv3 is unclear.

We follow the implementation of DINOv3 for selecting the best τ\tau. Here, we provide the results for DINOv3 with various τ\tau.

MethodViTτ\tauVOC07VOC12COCO20K
DINOv3 (Reported)7B/16—66.169.555.1
DINOv3 (Our evaluation)7B/160.2054.1058.5839.12
DINOv3 (Our evaluation)7B/160.2562.2666.3347.89
DINOv3 (Our evaluation)7B/160.3065.5468.9952.94
DINOv3 (Our evaluation)7B/160.3566.2369.5654.89
DINOv3 (Our evaluation)7B/160.4066.3169.2255.01
DINOv3 (Our evaluation)7B/160.4564.1867.6953.50

Table R14. Unsupervised object discovery results. CorLoc (%, ↑) on VOC07, VOC12, and COCO20K.

Q5: About related works.

We thank the reviewer for highlighting these relevant works. DiffAE, DRL, and SODA learn image representations that condition pixel-space diffusion models. DiffAE emphasizes reconstruction and manipulation; DRL evaluates denoising-based representations on downstream tasks, including semi-supervised classification; and SODA supports visual understanding and novel-view synthesis.

CondGen models the conditional distribution of target representations across augmented views directly in feature space, using prototype refinement and decoupled Sinkhorn optimization. We acknowledge these precedents and will clarify that our contribution concerns the modeling space, cross-view feature objective, and optimization mechanism. We will expand Related Work to discuss these methods and SiameseIM’s cross-view representation prediction, revise Table 1 to remove the overly broad claim that prior approaches do not model one-to-many structure, and include comparisons with SODA and SiameseIM in Table 3.

MethodArchitectureLinear accuracy (%)
SODARN50×272.2
SiameseIMViT-B78.0
CondGenViT-B79.0

Table R15. Comparisons with SODA and SiameseIM. We will include it in the revised manuscript.

Q6: What mechanism prevents encoder collapse or condition-ignoring solutions? Can you report feature variance/effective rank and the exact gradient and stop-gradient graph?

We thank the reviewer for the question. Similar to MoCo, our collapse-avoidance design is only the momentum teacher, which provides slowly evolving, stop-gradient targets for the online encoder. The online encoder receives diffusion-loss gradients through the conditioning representation, whereas the teacher is updated only by EMA. The clustering loss updates the clustering head and prototypes without propagating gradients into either encoder. We will explicitly document these gradient paths.

With the learned CondGen model, we separately examine condition usage (Q2) by comparing denoising errors under correct and shuffled conditioning, and assess generated-feature diversity through its variance (Q3). These empirical results show that CondGen does not suffer from condition ignoring and stochastic-output collapse.

Q7: About Proposition 1 and gradient alignment analysis.

We thank the reviewer for this constructive suggestion. We will recast Proposition 1 as a mechanistic hypothesis. Additionally, we examine the gradient alignment for joint optimization.

Gradient analysis for joint optimization. We train ViT-S for 15 epochs under joint optimization and measure gradient alignment between the conditional-generation and clustering losses on the shared encoder parameters. Across 344 valid measurements, the mean gradient cosine is 0.0595, and 40.99% exhibit negative alignment.

These results indicate frequent, intermittent gradient conflict despite a slightly positive average alignment. They support optimization tension as a possible contributor to slower convergence and motivate our decoupled optimization design.

MethodMean cosineConflict (%)
Joint optimization0.059540.99

Table R16. Gradient alignment of the shared encoder parameters. Conflict rate is the percentage of valid measurements with negative cosine.

Q8: Are the global/local crop ranges reversed in the text? Please provide the complete diffusion, clustering, augmentation, and optimization configuration.

We thank the reviewer and will provide the complete configuration in the revision. Thanks for pointing out the issue. For global crops, the range is (0.4, 1.0), while for local crops, it is (0.08, 0.4).

(I) Diffusion head. We follow MAR for the diffusion-head architecture and training objective.

(II) Clustering. We use a three-layer MLP with dimensions 2048–2048–256, with LayerNorm and GELU after each hidden layer. We use 3000 prototypes, 3 iterations of Sinkhorn optimization with an entropy regularization parameter of 0.05.

(III) Regarding data augmentations of global and local crops, we exactly follow DINOv1.

(IV) Optimization. We use AdamW with a base learning rate of 2×10−32\times10^{-3} and weight decay of 0.1. The learning rate is linearly warmed up over 20 epochs and then cosine-decayed toward 10−610^{-6}.

Reviewer_SmubOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CondGen, a self-supervised learning framework that models the conditional distribution of one augmented view's representation given another using a diffusion objective. An EMA teacher provides target features, and a decoupled Sinkhorn clustering module refines these targets through learned semantic prototypes without directly propagating the clustering loss into the encoder. The paper evaluates ImageNet-pretrained models on linear probing, distribution and structural robustness, unsupervised object discovery, image generation, video object segmentation, and geometric correspondence. It reports particularly promising results on structural robustness and object discovery, with training costs comparable to DINO.

Critical strengths
  1. Promising representation quality, especially for structural robustness and object discovery. Table 3 reports 79.0% ImageNet linear-probe accuracy for ViT-B/16 after 100 epochs, together with substantial improvements over the listed DINO baseline on Edge and ImageNet-Sketch. Table 6 reports CorLoc scores of 67.3%, 72.3%, and 57.0% on VOC07, VOC12, and COCO, respectively. These results provide consequential evidence that the method learns useful visual structure beyond standard image classification.
  2. Broad transfer evaluation with practical training costs. The paper evaluates the representations in both latent and pixel-space image generation, as well as video segmentation and geometric correspondence. Appendix A.2 also reports training time and memory consumption close to DINO under the stated hardware and batch-size settings. Since the diffusion head is discarded after pretraining, the method does not require iterative diffusion sampling for downstream feature extraction.
Critical weaknesses
  1. The central explanation in terms of one-to-many uncertainty is not sufficiently isolated by the experiments. Sections 3.1-3.2 motivate the method through conditional distribution modeling, but Figure 2 and Table 2 principally examine clustering, target refinement, and decoupling. They do not compare the diffusion objective against a deterministic cross-view predictor using the same EMA teacher, prototype-refined targets, augmentations, and comparable training budget. Consequently, the reported improvements establish the effectiveness of the complete recipe but do not establish the benefit attributable to modeling a distribution of plausible targets. The downstream image-generation results evaluate the usefulness of the learned encoder rather than directly testing this cross-view uncertainty claim. This distinction matters because conditional uncertainty is the main justification for the proposed formulation.
  2. Proposition 1 is not established by the supplied proof. In Appendix A.1, Equations (13) and (14) assign the projected gradients the forms of negative and positive multiples of the same unit vector, respectively. Neither Assumption 1 nor the qualitative statements about preserving and suppressing instance-level information establish these specific gradient directions for the actual objectives. The negative inner product in Equation (15) therefore follows from an unproved directional assumption. Figure 2 supports the empirical stability of the decoupled configuration, but training collapse in the joint configuration does not by itself identify gradient conflict as its cause. The proof should be derived from explicit objective-level assumptions, or the claim should be recast as an empirical explanation supported by appropriate measurements.
Questions for authors
  1. Evidence for the central mechanism: Do you have a controlled comparison between the diffusion objective and deterministic feature prediction with the same prototype-refined targets and training recipe, using comparable predictor capacity and training budget? Such a comparison would materially affect the assessment of the conditional-generation contribution.
  2. Clarification of the theoretical argument: What assumptions or derivations establish Equations (13) and (14) from the actual losses? Are there measurements of gradient alignment that support the proposed explanation of Figure 2?
  3. Clarification of contribution relative to prior work: Section 4 cites GenDD for implementation, but GenDD already uses conditional diffusion to model teacher representations and introduces target contraction toward class centers. What is the precise conceptual advance beyond adapting this construction to cross-view learning with an EMA teacher and unlabeled prototypes? A clearer comparison would help assess the scope of the claimed new formulation. Reference: Generative Distribution Distillation, Section 3.2.
4 comments

Q1: Comparisons with deterministic prediction.

(I) Effectiveness of conditional diffusion modeling. We only replace the diffusion head with an MLP prediction head while keeping all other settings—including EMA, augmentations, prototype refinement, and training budget—the same as CondGen.

MethodLinear accuracy (%)
CondGen79.0
Deterministic cross-view prediction77.1
Shuffled or zeroed conditioningaround 0.2

Table R6. Comparison with deterministic cross-view prediction and conditioning-removal controls.

(II) Benefits from uncertainty modeling.

To examine whether CondGen’s representation gains are associated with cross-view uncertainty, we stratify ImageNet validation images using a fixed, independently pretrained ResNet-50 encoder from PyTorch. For each image xx, we sample M=32M=32 augmented views using the pretraining augmentation pipeline and extract their L2L_2-normalized features umu_m. We define an augmentation-induced uncertainty proxy as

\qquad \bar{u} = \frac{1}{M} \sum^{M}_{m=1} u_m$$ We divide images into three equal-sized groups with low, medium, and high uncertainty. These assignments are fixed and shared across methods, avoiding dependence on the representations being compared. Within each group, we evaluate frozen-feature linear-probe accuracy for CondGen and the matched deterministic cross-view baseline, using identical evaluation preprocessing and linear-probe training protocols. We report the accuracy on different uncertainty group images. CondGen’s larger gains in higher-uncertainty groups provide empirical support for the benefits of explicitly modeling one-to-many cross-view uncertainty. | Uncertainty | Images | DINO-B | Deterministic | CondGen (Ours) | |:--|--:|--:|--:|--:| | Low | 16,667 | **82.8** | 82.2 | 79.3 | | Medium | 16,667 | 77.1 | 75.6 | **79.3** | | High | 16,666 | 74.3 | 73.3 | **78.6** | | Overall | 50,000 | 78.0 | 77.1 | **79.0** | *Table R7. Linear-probe accuracy (%) across uncertainty groups defined by cross-view feature dispersion from a fixed PyTorch-pretrained ResNet-50. All evaluated methods use ViT-B encoders.* *Q2: Reliance on conditions.* We directly test condition usage on held-out ImageNet validation images, with all model parameters frozen. For each image, we sample a source–target view pair using the pretraining augmentation pipeline and construct the prototype-refined target $\hat z_i$ following Eq.~(9). We evaluate noise-prediction MSE under three conditioning settings: *correct*, using a source view from the same image; *randomly shuffled*, using a source view from a different image; and *same-class shuffled*, using a source view from a different image of the same class. Class labels are used only to construct the same-class evaluation pairs. Across all settings, we hold the target representation, sampled noise, diffusion timestep, and chunk indices fixed, changing only the conditioning representation. As shown in the Table, correct conditioning achieves an MSE of $0.0124$, whereas random and same-class shuffling increase the error to $0.1560$ and $0.0682$, respectively---approximately $12.6$ and $5.5$ times the error under correct conditioning. The substantial degradation even when class identity is preserved provides direct evidence that the denoiser uses predictive information specific to the conditioning image beyond class-level semantics. These results establish useful condition dependence, complementing the separate evaluations of conditional target diversity and representation quality. | Conditioning setting | Noise MSE ↓ | |:--|--:| | Correct | **0.0124** | | Randomly shuffled | 0.1560 | | Same-class shuffled | 0.0682 | *Table R8. Noise-prediction MSE under different conditioning settings. Targets, sampled noise, timesteps, and chunk indices are identical across settings; only the conditioning representation changes.*

Q3: About Proposition 1 and gradient analysis for Figure 2.

We thank the reviewer for this constructive suggestion. We will recast Proposition 1 as a mechanistic hypothesis. Additionally, we examine the gradient alignment for joint optimization.

Gradient analysis for joint optimization. We train ViT-S for 15 epochs under joint optimization and measure gradient alignment between the conditional-generation and clustering losses on the shared encoder parameters. Across 344 valid measurements, the mean gradient cosine is 0.0595, and 40.99% exhibit negative alignment.

These results indicate frequent, intermittent gradient conflict despite a slightly positive average alignment. They support optimization tension as a possible contributor to slower convergence and motivate our decoupled optimization design.

MethodMean cosineConflict (%)
Joint optimization0.059540.99

Table R9. Gradient alignment of the shared encoder parameters. Conflict rate is the percentage of valid measurements with negative cosine.

Q4: Differences from GenDD.

(1) Different learning problem. GenDD addresses knowledge distillation, transferring representations from a well-trained teacher. CondGen studies self-supervised representation learning from scratch, without labels or externally pretrained models.

(2) Optimization mechanism. We introduce Decoupled Sinkhorn optimization to incorporate learned global semantic structure into conditional generation. The clustering objective learns prototypes separately from encoder optimization, while prototype-refined targets provide semantic guidance to the encoder. Figure~2 shows that directly optimizing clustering and conditional generation through the shared encoder can cause training failure under our experimental setting, whereas decoupled optimization converges successfully.

(3) CondGen further demonstrates that formulating SSL as conditional generation yields representations with surprisingly improved robustness and particularly strong performance in unsupervised object discovery.

AuthorsOpenReview ↗

Thank you for your time and constructive suggestions

Dear Reviewer Smub,

Thank you for your time and constructive suggestions. As the rebuttal deadline approaches, we would appreciate your feedback on whether our responses have addressed your concerns. If any questions remain, we would be happy to discuss them and provide further clarification.

Thank you again for your consideration.

The Authors

The discussion deadline is approaching. Could you please respond to the authors’ rebuttal? Thanks!

Your AC

Reviewer_mxdXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CondGen learns an image encoder by conditionally denoising a representation from another augmented view. The conditioning representation comes from the query encoder, while the target comes from an EMA encoder. A separate Sinkhorn clustering head learns balanced prototypes without updating the backbone. Their soft reconstruction is blended into the detached diffusion target. The diffusion head operates on global CLS representations and is discarded after pretraining. The paper evaluates frozen features for robustness, object discovery, video segmentation, correspondence, and as priors for image generation.

Critical strengths
  1. The robustness results are substantial, particularly for ViT-B/16. Table 3 reports 79.0% ImageNet linear accuracy after 100 epochs, alongside Edge accuracy of 24.58 versus DINO's 14.45 and iBOT's 16.46. ImageNet-Sketch improves to 31.77 versus 25.47 and 27.43. The method does not need to lead standard classification to make these gains useful. Appendix A.2 also reports comparable iteration cost to DINO under the stated common hardware and batch size.
  2. The evaluation establishes benefits beyond classification. Table 6 reports object-discovery CorLoc of 67.3/72.3/57.0 on VOC07/VOC12/COCO, versus DINO's 60.4/64.7/50.5. Table 4 shows improved generation gFID with both RAEv2 and JiT; at 50 RAEv2 epochs it is 1.32 versus 1.46 for DINO features. The component evidence is also informative: Table 2 improves linear accuracy from 78.0 without prototype refinement to 79.0 with it, while Figure 2 shows that the tested joint-clustering configuration fails during training. These support the usefulness of the complete design and its decoupling choice.
Critical weaknesses
  1. The defining conditional-uncertainty contribution is not isolated. The abstract and Table 1 position one-to-many modeling as the advance over existing SSL objectives. Table 2 varies prototypes and Figure 2 varies clustering coupling, but neither compares diffusion with deterministic cross-view prediction under the same EMA, augmentations, prototype refinement, and budget. Conditional diversity and reliance on the conditioning view are not measured either. Improved downstream image generators use frozen features and do not directly test that representation distribution. These results establish a useful recipe, but leave open what the claimed new treatment of uncertainty adds. This affects the main contribution, rather than only an auxiliary explanation.
  2. Proposition 1 is stronger than its derivation supports. Assumption 1 gives an orthogonal semantic decomposition, whereas Appendix A.1, Eqs. 13-14, writes the two projected gradients as exactly opposite multiples of the same vector. Preserving predictive information and reducing within-prototype variation do not by themselves imply those gradient equalities. Eq. 15 follows algebraically once they are assumed. The further claim about conflict in encoder gradients also needs conditions on the projection Jacobian and other gradient components. Figure 2 remains useful empirical evidence for this particular optimization choice, but it does not establish the proposition generally. This argument should either state sufficient assumptions explicitly or be presented as a mechanistic hypothesis.
Questions for authors
  1. Do existing matched deterministic-prediction or conditioning-removal controls distinguish conditional distribution modeling from the other changes in the training recipe? What evidence directly supports the one-to-many interpretation?
  2. What assumptions justify Eqs. 13-14 and their transfer to encoder gradients? Can Proposition 1 be narrowed accordingly?
  3. The cited GenDD already learns teacher features through conditional diffusion. Please clarify the conceptual distinction beyond the EMA/cross-view setting and explain which CondGen-specific changes are essential.
4 comments

Q1: Comparisons with deterministic prediction.

(I) Effectiveness of conditional diffusion modeling. We only replace the diffusion head with an MLP prediction head while keeping all other settings—including EMA, augmentations, prototype refinement, and training budget—the same as CondGen.

MethodLinear accuracy (%)
CondGen79.0
Deterministic cross-view prediction77.1
Shuffled or zeroed conditioningaround 0.2

Table R1. Comparison with deterministic cross-view prediction and conditioning-removal controls.

(II) Benefits from uncertainty modeling.

To examine whether CondGen’s representation gains are associated with cross-view uncertainty, we stratify ImageNet validation images using a fixed, independently pretrained ResNet-50 encoder from PyTorch. For each image xx, we sample M=32M=32 augmented views using the pretraining augmentation pipeline and extract their L2L_2-normalized features umu_m. We define an augmentation-induced uncertainty proxy as

\qquad \bar{u} = \frac{1}{M} \sum^{M}_{m=1} u_m$$ We divide images into three equal-sized groups with low, medium, and high uncertainty. These assignments are fixed and shared across methods, avoiding dependence on the representations being compared. Within each group, we evaluate frozen-feature linear-probe accuracy for CondGen and the matched deterministic cross-view baseline, using identical evaluation preprocessing and linear-probe training protocols. We report the accuracy on different uncertainty group images. CondGen’s larger gains in higher-uncertainty groups provide empirical support for the benefits of explicitly modeling one-to-many cross-view uncertainty. | Uncertainty | Images | DINO-B | Deterministic | CondGen (Ours) | |:--|--:|--:|--:|--:| | Low | 16,667 | **82.8** | 82.2 | 79.3 | | Medium | 16,667 | 77.1 | 75.6 | **79.3** | | High | 16,666 | 74.3 | 73.3 | **78.6** | | Overall | 50,000 | 78.0 | 77.1 | **79.0** | *Table R2. Linear-probe accuracy (%) across uncertainty groups defined by cross-view feature dispersion from a fixed PyTorch-pretrained ResNet-50. All evaluated methods use ViT-B encoders.* *Q2: Reliance on conditions.* We directly test condition usage on held-out ImageNet validation images, with all model parameters frozen. For each image, we sample a source–target view pair using the pretraining augmentation pipeline and construct the prototype-refined target $\hat z_i$ following Eq.~(9). We evaluate noise-prediction MSE under three conditioning settings: *correct*, using a source view from the same image; *randomly shuffled*, using a source view from a different image; and *same-class shuffled*, using a source view from a different image of the same class. Class labels are used only to construct the same-class evaluation pairs. Across all settings, we hold the target representation, sampled noise, diffusion timestep, and chunk indices fixed, changing only the conditioning representation. As shown in the Table, correct conditioning achieves an MSE of $0.0124$, whereas random and same-class shuffling increase the error to $0.1560$ and $0.0682$, respectively---approximately $12.6$ and $5.5$ times the error under correct conditioning. The substantial degradation even when class identity is preserved provides direct evidence that the denoiser uses predictive information specific to the conditioning image beyond class-level semantics. These results establish useful condition dependence, complementing the separate evaluations of conditional target diversity and representation quality. | Conditioning setting | Noise MSE ↓ | |:--|--:| | Correct | **0.0124** | | Randomly shuffled | 0.1560 | | Same-class shuffled | 0.0682 | *Table R3. Noise-prediction MSE under different conditioning settings. Targets, sampled noise, timesteps, and chunk indices are identical across settings; only the conditioning representation changes.*

Q3: About Proposition 1 and gradient analysis for Figure 2.

We thank the reviewer for this constructive suggestion. We will recast Proposition 1 as a mechanistic hypothesis. Additionally, we examine the gradient alignment for joint optimization.

Gradient analysis for joint optimization. We train ViT-S for 15 epochs under joint optimization and measure gradient alignment between the conditional-generation and clustering losses on the shared encoder parameters. Across 344 valid measurements, the mean gradient cosine is 0.0595, and 40.99% exhibit negative alignment.

These results indicate frequent, intermittent gradient conflict despite a slightly positive average alignment. They support optimization tension as a possible contributor to slower convergence and motivate our decoupled optimization design.

MethodMean cosineConflict (%)
Joint optimization0.059540.99

Table R4. Gradient alignment of the shared encoder parameters. Conflict rate is the percentage of valid measurements with negative cosine.

Q4: Evidence for the one-to-many interpretation.

We evaluate conditional generation in the frozen CondGen target space. For each held-out image, we fix one conditioning view, generate multiple target features from independent Gaussian noise, and compare them with prototype-refined EMA features of independently augmented target views. We report the energy score (ES; lower is better), which evaluates distributional agreement while accounting for generated diversity.

For a fixed conditioning view cc, let skk=1K\\{s_k\\}_{k=1}^{K} denote independently generated target features and ymm=1M\\{y_m\\}_{m=1}^{M} the prototype-refined features of independently augmented target views. We evaluate their distributional agreement using the energy score:

ES^(c)=1KM∑k=1K∑m=1M∥sk−ym∥2−12K(K−1)∑k≠ℓ∥sk−sℓ∥2.\widehat{\mathrm{ES}}(c)=\frac{1}{KM}\sum_{k=1}^{K}\sum_{m=1}^{M}\lVert s_k-y_m\rVert_2-\frac{1}{2K(K-1)}\sum_{k\neq\ell}\lVert s_k-s_\ell\rVert_2.

The first term measures distance to observed targets, while the second accounts for generated spread. Lower scores indicate better distributional prediction. We use unsquared Euclidean distances and average scores across held-out images. For the collapsed-mean control, the predictive distribution contains a single point, so the spread term is zero.

As shown in the table below, correctly conditioned samples achieve an ES of 0.3210, compared with 0.4210 when the generated ensemble is collapsed to its mean, a 23.8% reduction. This control indicates that retaining the predicted diversity improves distributional prediction over the corresponding point forecast. Generated feature variance also tracks augmentation-induced target variance across images, with Spearman correlation 0.7264.

Random and same-class condition shuffling increase ES to 3.7498 and 1.5939, respectively, despite similar average generated variances. Thus, diversity alone does not explain the favorable score: the samples must also agree with the targets associated with the conditioning image. Together, these results directly support condition-dependent feature prediction with meaningful target diversity, complementing the matched deterministic-prediction and representation-quality evaluations.

Conditioning / controlES ↓VgenV_{\mathrm{gen}}ρ\rho
Correct0.32100.15940.7264
Randomly shuffled3.74980.1586-0.0292
Same-class shuffled1.59390.15530.2566
Collapsed conditional mean0.42100.0000—

Table R5. Conditional target-distribution evaluation. VgenV_{\mathrm{gen}} denotes generated feature variance; ρ\rho is its Spearman correlation with real target variance across images. Mean real target variance is 0.1992. The collapsed-mean control averages the correct-condition samples.

Q5: Differences from GenDD.

(1) Different learning problem. GenDD addresses knowledge distillation, transferring representations from a well-trained teacher. CondGen studies self-supervised representation learning from scratch, without labels or externally pretrained models.

(2) Optimization mechanism. We introduce Decoupled Sinkhorn optimization to incorporate learned global semantic structure into conditional generation. The clustering objective learns prototypes separately from encoder optimization, while prototype-refined targets provide semantic guidance to the encoder. Figure~2 shows that directly optimizing clustering and conditional generation through the shared encoder can cause training failure under our experimental setting, whereas decoupled optimization converges successfully.

(3) CondGen further demonstrates that formulating SSL as conditional generation yields representations with surprisingly improved robustness and particularly strong performance in unsupervised object discovery.

AuthorsOpenReview ↗

Thank you for your time and constructive suggestions

Dear Reviewer mxdX,

Thank you for your time and constructive suggestions. As the rebuttal deadline approaches, we would appreciate your feedback on whether our responses have addressed your concerns. If any questions remain, we would be happy to discuss them and provide further clarification.

Thank you again for your consideration.

The Authors

The discussion deadline is approaching. Could you please respond to the authors’ rebuttal? Thanks!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.