- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper tackles unsupervised cross-domain image retrieval by proposing a framework called DUDE. It uses a frozen Stable Diffusion model to help the image encoder learn semantic object representations, and then applies a progressive contrastive learning strategy for cross-domain alignment. Specifically, image features are injected into prompts with domain style descriptions to train the encoder via a denoising loss. The alignment is then gradually expanded from instance augmentation and intra-domain mutual nearest neighbors (MNN) to cross-domain MNN alignment. Experiments on PACS, Office-Home, and DomainNet show solid improvements over existing baselines.
Critical strengths
-
Exploring diffusion model priors for unsupervised cross-domain retrieval by decoupling object semantics from domain style is an interesting and novel direction.
-
The overall pipeline makes sense. Combining representation learning with progressive alignment works well in practice, as shown by the ablation studies. Also, only the image encoder is needed during inference, which keeps the retrieval phase efficient.
Critical weaknesses
-
Training with a denoising objective does not automatically guarantee that object semantics are cleanly separated from domain styles. There seems to be a gap between the theoretical claims and the actual optimization. While the final retrieval metrics are good, there is a lack of direct analysis or visualization to verify that the decoupling actually happens as intended.
-
Since the method introduces strong pre-trained priors from Stable Diffusion, it is hard to tell how much of the improvement comes from the proposed alignment design versus the extra pre-trained knowledge itself. Without isolating this factor, it is difficult to evaluate the true contribution of the proposed framework over traditional baselines.
-
The progressive alignment strategy needs more justification. Although the ablation study shows that adding each stage improves performance, it does not properly control for the training schedule and total training budget (e.g., epochs). In addition, there is not enough discussion on how reliable the MNN matching is during the early stages and how stable the performance gains are.
Questions for authors
1.In practice, how does the optimization process guarantee that the learned representations actually satisfy the "minimal sufficiency" condition discussed in the appendix?
2.With the same Stable Diffusion prior, how much performance difference is there between using a simple "neutral" prompt versus a prompt that explicitly specifies the domain? Also, if we apply this same diffusion-guided initialization to existing baseline alignment methods, would DUDE still outperform them?
3.Could the authors clarify whether the disentanglement loss is still kept during the later alignment stages? Also, how are the training epochs allocated across the different ablation variants?
6 comments
In response, (ii) we isolate the effects of domain-aware prompting and pre-trained diffusion knowledge.
2. The effects of the domain prompt and diffusion pre-trained knowledge.
2.1 Domain-aware prompt v.s. neutral prompt
- Explicitly providing the domain condition consistently outperforms a neutral prompt under the same Stable Diffusion prior.
To isolate the effect of domain conditioning, we compare DUDE's domain-aware prompt an {domain} of a [z], with the neutral prompt an image of a [z], while keeping the remaining training configuration unchanged. The table reports DomainNet retrieval performance obtained directly using the learned representation after 50 epochs of training with . No progressive alignment is applied in either setting.
| Prompt / P@50 | Average | Clipart→Sketch | Sketch→Clipart | Infograph→Real | Real→Infograph | Infograph→Sketch | Sketch→Infograph | Painting→Clipart | Clipart→Painting |
|---|---|---|---|---|---|---|---|---|---|
| Domain Prompt | 67.68 | 75.76 | 82.71 | 36.17 | 75.21 | 42.78 | 64.52 | 91.01 | 73.27 |
| Neutral Prompt | 63.32 | 66.42 | 76.70 | 34.87 | 71.36 | 38.19 | 57.48 | 90.65 | 70.91 |
The domain-aware prompt improves average P@50 by 4.36%. The gains are especially clear for transfers involving strongly different visual styles, such as Clipart and Sketch. This comparison supports our motivation: explicitly supplying domain style to the diffusion model encourages the learned image token to focus more strongly on object-related information.
2.2 Applying the same diffusion guidance to an existing alignment method
-
The raw Stable Diffusion prior alone is insufficient for cross-domain retrieval, even when combined with an existing alignment method.
-
Under the same representation, DUDE's progressive alignment still outperforms ProtoOT.
We use ProtoOT as a representative state-of-the-art alignment module and evaluate the following settings on DomainNet:
- Stable Diffusion: directly use the frozen Stable Diffusion representation for retrieval.
- Stable Diffusion + ProtoOT: apply ProtoOT to the same frozen Stable Diffusion representation.
- DUDE () + ProtoOT: use Stable Diffusion to guide the learning of our designed object-focused representation through , and then align this representation using ProtoOT.
- DUDE () + DUDE (): use the same representation as in (3), but replace ProtoOT with our progressive alignment strategy.
| Method / P@50 | Average | Clipart→Sketch | Sketch→Clipart | Infograph→Real | Real→Infograph | Infograph→Sketch | Sketch→Infograph | Painting→Clipart | Clipart→Painting |
|---|---|---|---|---|---|---|---|---|---|
| Stable Diffusion | 17.42 | 19.60 | 21.55 | 11.20 | 15.85 | 12.03 | 16.73 | 25.63 | 16.79 |
| Stable Diffusion + ProtoOT | 28.67 | 30.32 | 35.62 | 17.49 | 33.14 | 15.99 | 27.36 | 38.82 | 30.61 |
| DUDE () + ProtoOT | 81.38 | 95.60 | 96.73 | 50.28 | 75.67 | 68.55 | 84.16 | 93.11 | 86.92 |
| DUDE () + DUDE () | 82.62 | 93.07 | 95.21 | 55.48 | 78.49 | 70.28 | 88.20 | 93.23 | 87.01 |
The raw Stable Diffusion representation obtains only 17.42% P@50, and applying ProtoOT raises it to 28.67%. In contrast, learning an object-focused representation with and then applying ProtoOT reaches 81.38%. This shows that the improvement cannot be attributed merely to importing pretrained Stable Diffusion knowledge. Instead, the proposed diffusion-guided object-disentanglement objective is essential for converting the diffusion prior into a representation suitable for cross-domain retrieval. Notably, our object disentanglement strategy should not be regarded as merely a "Stable Diffusion initialization". As also noted by Reviewer xtdf, "The idea of disentangling object features from domain-specific styles has not been explored in prior work". It is an important design of our work.
More importantly, with the same object representation, DUDE improves average P@50 from 81.38 to 82.62%. The comparison demonstrates that our progressive alignment provides an additional complementary improvement.
In response, (iii) we clarify the training schedules of the alignment ablations and analyze MNN reliability throughout the alignment process.
3. Ablation in alignment stages and MNN reliability
3.1 Is retained during the alignment stages?
- No. As stated in Appendix C.2 (“Implementation Details”), is used only during the first 50 epochs and is disabled during the subsequent alignment stages.
The full DUDE training schedule is epochs of object disentanglement with , followed by epochs of first-stage alignment with and epochs of second-stage alignment with . Thus, the three objectives are applied sequentially rather than jointly. We apologize that this training schedule was not sufficiently prominent in the main text. We will clarify it in the revised manuscript.
3.2 Training budget in the ablation study
- All alignment-only variants use the same 50-epoch budget, and all variants initialized by use the same 100-epoch total budget.
The exact schedules are as follows:
| Ablation | Training Epochs | DomainNet P@50 |
|---|---|---|
| 50 | 52.78 | |
| 50 | 53.49 | |
| (30) and (20) | 30+20=50 | 57.89 |
| (50) and (50) | 50+50=100 | 76.56 |
| (50) and (50) | 50+50=100 | 79.16 |
| (50), (30) and (20) | 50+30+20=100 | 82.62 |
These fixed-budget comparisons show that the improvement is not caused simply by increasing the number of alignment epochs. The progressive then schedule improves over either single-stage variant. In particular, directly applying cross-domain alignment through without first stabilizing the representation using is substantially less effective.
3.3 Reliability of MNN matching during alignment
- Cross-domain MNNs are indeed noisy early in training. DUDE is designed to delay their use until after in-domain structure has been consolidated.
We quantify MNN reliability on the Office-Home Art–Product task using MNN precision, defined as the proportion of selected MNN pairs that share the same ground-truth category. Category labels are used only for this post-hoc evaluation and are never available during training. We use the checkpoint immediately after the 50-epoch stage as the starting point, i.e., the representation state at which progressive alignment begins.
| Epochs | Training objective | In-domain MNN precision ↑ | Cross-domain MNN precision ↑ |
|---|---|---|---|
| 50 (Start of alignment) | (50) | 49.00 | 30.88 |
| 50 + 10 | (50) and (10) | 60.86 | 47.53 |
| 50 + 30 | (50) and (30) | 66.41 | 46.89 |
| 50 + 30 + 20 | (50), (30) and (20) | 64.03 | 51.70 |
At the beginning of alignment, cross-domain MNN precision is only 30.88%. Directly using cross-domain correspondences at this point would introduce considerable noise. During , DUDE therefore uses only in-domain MNN pairs to consolidate the local semantic structure. After 30 epochs of , in-domain MNN precision increases from 49.00 to 66.41%. Cross-domain MNNs are activated only in the subsequent stage, after which their precision increases from 46.89 to 51.70%.
Therefore, DUDE does not assume that early cross-domain MNNs are fully reliable. Instead, it explicitly avoids using these noisier correspondences in the early stage and activates cross-domain alignment only after in-domain stabilization. We will add the MNN-reliability analysis to the revision.
We sincerely thank the reviewer for the constructive feedback. In response, (i) we clarify the scope of the minimal-sufficiency analysis and provide direct evidence of object–domain decoupling.
1. Minimal sufficiency and direct evidence of object–domain decoupling.
We appreciate the reviewer for pointing out the distinction between the theoretical characterization and the practical optimization. The theoretical interpretation in appendix does not intend to claim that minimizing LOD necessarily converges to a minimally sufficient representation. Instead, it characterizes two properties of a non-redundant solution under the condition on a domain token D and an object token Z.
Thus, minimum sufficiency represents an ideal limiting solution among denoising-optimal representations, rather than an optimization guarantee. In practice, finite data, limited model capacity, and gradient-based optimization leave residual style information in Z. We will revise the corresponding claims to make this distinction explicit.
To directly examine whether the intended decoupling occurs, we add linear-probing (LP) experiments. Specifically, we freeze the representations and train separate linear classifiers to predict the object category and domain. For the domain probe, we train a linear binary classifier. To avoid different category distributions becoming a shortcut for domain LP, we balance the two domains within every object category by sampling an equal number of images from each domain, preventing differences in category distributions from becoming a shortcut for domain prediction. We use the labels only for this post-hoc analysis and do not involve them in DUDE training.
Compared with the ResNet50 baseline, DUDE reduces domain-probe accuracy by 16.09, 17.39, and 17.83% on PACS, Office-Home, and DomainNet, respectively, while improving category-probe accuracy by 26.01, 21.39, and 40.80%. These results show that DUDE disentanglement substantially reduces the domain semantics retained in the features, while preserving rich category semantics.