- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes CondGen, a self-supervised representation learning method that formulates learning across augmented views as conditional generation. It uses conditional diffusion to predict a target view’s representation from another view’s representation. A decoupled Sinkhorn clustering module supplies semantic prototypes to refine the diffusion target without backpropagating the clustering loss through the encoder. After pretraining on ImageNet, CondGen achieves competitive results on structural robustness, unsupervised object discovery, image generation, video object segmentation, and 3D correspondence tasks. Its ViT-B/16 model obtains CorLoc scores of 67.3%, 72.3%, and 57.0% on VOC07, VOC12, and COCO-20k, respectively. These results demonstrate that CondGen learns transferable visual representations, although they do not directly establish whether it captures the proposed one-to-many conditional distribution.
Critical strengths
-
An interesting research perspective. The paper reframes cross-view representation learning as conditional distribution modeling and combines conditional diffusion with semantic prototypes. This offers a useful way to examine whether matching a single target is sufficient, although the proposed one-to-many mechanism still needs direct validation.
-
Broad downstream evaluation. The paper assesses the learned representations across structural robustness, unsupervised object discovery, image generation, video object segmentation, and 3D correspondence. The results are particularly strong on structural robustness and object discovery, providing evidence that the representations transfer across different tasks.
Critical weaknesses
-
The need for one-to-many modeling is not established. The paper does not demonstrate that a given view admits multiple meaningfully different target representations, or that predicting a single target is inadequate. Its central motivation therefore remains an assumption.
-
The performance gains cannot be attributed to conditional distribution modeling. The downstream results show that CondGen learns useful representations, but the paper neither directly evaluates the learned conditional distribution nor provides a controlled comparison with deterministic prediction. Its core mechanism has therefore not been isolated experimentally.
Questions for authors
1.Evidence for the premise: What empirical evidence shows that a fixed input view has multiple meaningfully distinct, valid target representations? In particular, how do you distinguish this uncertainty from variation introduced solely by the augmentation procedure?
2.Attribution of the gains: Have you compared CondGen with a deterministic cross-view predictor while holding the encoder, augmentations, EMA target, prototypes, training data, and compute budget fixed? The outcome of this comparison would materially affect whether the reported gains can be attributed to conditional distribution modeling.
3.Validation of the learned distribution: Do you have evidence that multiple samples conditioned on the same representation cover the corresponding valid target representations? Downstream accuracy alone does not establish that the proposed conditional distribution has been learned.
4.Clarification of the reported method and results: Please clarify the product upper limit in Equation (5), provide the values of the prototype mixing weight (\gamma), clustering temperature, EMA coefficient, and noise schedule, and correct the negative CorLoc entry in Table 11. These details affect interpretation and reproducibility.
6 comments
Q3: Conditional target-distribution evaluation.
We evaluate conditional generation in the frozen CondGen target space. For each held-out image, we fix one conditioning view, generate multiple target features from independent Gaussian noise, and compare them with prototype-refined EMA features of independently augmented target views. We report the energy score (ES; lower is better), which evaluates distributional agreement while accounting for generated diversity.
For a fixed conditioning view , let denote independently generated target features and the prototype-refined features of independently augmented target views. We evaluate their distributional agreement using the energy score:
The first term measures distance to observed targets, while the second accounts for generated spread. Lower scores indicate better distributional prediction. We use unsquared Euclidean distances and average scores across held-out images. For the collapsed-mean control, the predictive distribution contains a single point, so the spread term is zero.
As shown in the table below, correctly conditioned samples achieve an ES of 0.3210, compared with 0.4210 when the generated ensemble is collapsed to its mean, a 23.8% reduction. This control indicates that retaining the predicted diversity improves distributional prediction over the corresponding point forecast. Generated feature variance also tracks augmentation-induced target variance across images, with Spearman correlation 0.7264.
Random and same-class condition shuffling increase ES to 3.7498 and 1.5939, respectively, despite similar average generated variances. Thus, diversity alone does not explain the favorable score: the samples must also agree with the targets associated with the conditioning image. Together, these results directly support condition-dependent feature prediction with meaningful target diversity, complementing the matched deterministic-prediction and representation-quality evaluations.
| Conditioning / control | ES ↓ | ||
|---|---|---|---|
| Correct | 0.3210 | 0.1594 | 0.7264 |
| Randomly shuffled | 3.7498 | 0.1586 | -0.0292 |
| Same-class shuffled | 1.5939 | 0.1553 | 0.2566 |
| Collapsed conditional mean | 0.4210 | 0.0000 | — |
Table R18. Conditional target-distribution evaluation. denotes generated feature variance; is its Spearman correlation with real target variance across images. Mean real target variance is 0.1992. The collapsed-mean control averages the correct-condition samples.
Q4: Clarification of the reported method and results.
(1) Equation (5). The product should extend to the current diffusion step , rather than :
(2) Hyperparameters. The prototype mixing weight is . Both the clustering prediction temperature and target-refinement temperature are 0.2. Following DINO, the momentum-teacher EMA coefficient starts at 0.996 and increases toward 1.0 following a cosine schedule. Diffusion uses T=1000 steps with a cosine noise schedule, offset 0.008, and maximum per-step variance 0.999. Training timesteps are sampled uniformly.
(3) Table 11. The negative CorLoc entry for CondGen ViT-B/16 on COCO20K at threshold 0.35 is erroneous. We will replace with 50.29.
We will incorporate these corrections and explicit settings into the revised manuscript.
Thank you for the detailed response and additional experiments. The deterministic prediction baseline, uncertainty-stratified analysis, and conditional distribution evaluation substantially address my original concerns. However, several concerns remain:
- The attribution of representation gains to conditional uncertainty modeling remains incomplete. Replacing the diffusion head with an MLP changes both the prediction mechanism and the training objective. The response does not sufficiently specify the deterministic baseline’s architecture, loss, normalization, tuning procedure, or how the training budget was matched. Moreover, the uncertainty proxy measures feature dispersion across augmentations of an image, rather than target uncertainty conditioned on a fixed source view. The stratified results are therefore suggestive, but do not isolate the proposed explanation.
- Eq.(13) and Eq.(14) assert that the projected gradients are opposite multiples of the same vector, without deriving this relationship from the actual objectives. The qualitative descriptions of preserving versus suppressing instance-level information do not establish these gradient directions. Consequently, the negative alignment in Eq.(15) follows from an asserted relationship rather than a demonstrated property of the method. Moreover, fig.2 supports the practical benefit of decoupling under the tested configuration, but does not independently establish the claimed gradient-conflict mechanism.
- The training description assigns crop scales of (0.08, 0.4) to global views and (0.4, 1.0) to local views. Please clarify whether these ranges were reversed in the text or used in the implementation, since the augmentation configuration is central to both the motivation and experimental interpretation. I acknowledge the substantial improvements made during rebuttal. Nevertheless, the remaining attribution and theoretical concerns leave me insufficiently convinced to change my score.
Q1: The need for one-to-many modeling is not established.
We thank the reviewer. We specifically consider one-to-many uncertainty in cross-view representation learning. Contrastive and feature-matching/prediction objectives encourage agreement between views. However, a partial source view does not uniquely determine what another view reveals: cropping may remove informative object parts or surrounding context, leaving multiple plausible target representations.
The augmentation procedure induces this uncertainty through partial observation. Our motivation concerns the resulting variability in visible content, beyond sensitivity to photometric perturbations. CondGen explicitly explores modeling this conditional target distribution through diffusion.
During rebuttal, we additionally assess its practical value through complementary experiments: (i) comparison with matched deterministic feature prediction, (ii) analysis of representation gains across uncertainty groups, (iii) condition-perturbation tests examining whether the denoiser uses the source representation, and (iv) conditional target-distribution evaluation measuring distributional agreement and diversity. Together with the downstream evaluations, these analyses support the usefulness of conditional distribution modeling in this setting.
Q2: Comparisons with deterministic prediction.
(I) Effectiveness of conditional diffusion modeling. We only replace the diffusion head with an MLP prediction head while keeping all other settings—including EMA, augmentations, prototype refinement, and training budget—the same as CondGen.
Table R1. Comparison with deterministic cross-view prediction and conditioning-removal controls.
(II) Benefits from uncertainty modeling.
To examine whether CondGen’s representation gains are associated with cross-view uncertainty, we stratify ImageNet validation images using a fixed, independently pretrained ResNet-50 encoder from PyTorch. For each image x, we sample M=32 augmented views using the pretraining augmentation pipeline and extract their L2-normalized features um. We define an augmentation-induced uncertainty proxy as
\qquad \bar{u} = \frac{1}{M} \sum^{M}_{m=1} u_m$$ We divide images into three equal-sized groups with low, medium, and high uncertainty. These assignments are fixed and shared across methods, avoiding dependence on the representations being compared. Within each group, we evaluate frozen-feature linear-probe accuracy for CondGen and the matched deterministic cross-view baseline, using identical evaluation preprocessing and linear-probe training protocols. We report the accuracy on different uncertainty group images. CondGen’s larger gains in higher-uncertainty groups provide empirical support for the benefits of explicitly modeling one-to-many cross-view uncertainty. | Uncertainty | Images | DINO-B | Deterministic | CondGen (Ours) | |:--|--:|--:|--:|--:| | Low | 16,667 | **82.8** | 82.2 | 79.3 | | Medium | 16,667 | 77.1 | 75.6 | **79.3** | | High | 16,666 | 74.3 | 73.3 | **78.6** | | Overall | 50,000 | 78.0 | 77.1 | **79.0** | *Table R17. Linear-probe accuracy (%) across uncertainty groups defined by cross-view feature dispersion from a fixed PyTorch-pretrained ResNet-50. All evaluated methods use ViT-B encoders.*