acceptodds
Under review as a conference paper at ICLR 2027

DUDE: Diffusion-Based Unsupervised Cross-Domain Image Retrieval

Abstract

Unsupervised cross-domain image retrieval (UCIR) aims to retrieve images of the same category across diverse domains without relying on annotations. Existing UCIR methods, which align cross-domain features for the entire image, often struggle with the domain gap, as the object features critical for retrieval are frequently entangled with domain-specific styles. To address this challenge, we propose DUDE, a novel UCIR method building upon feature disentanglement. In brief, DUDE leverages a text-to-image generative model to disentangle object features from domain-specific styles, thus facilitating semantical image retrieval. To further achieve reliable alignment of the disentangled object features, DUDE aligns mutual neighbors from within domains to across domains in a progressive manner. Extensive experiments demonstrate that DUDE achieves superior performance across three benchmark datasets over 13 domains. The code will be released.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Image retrieval, unsupervised image retrieval, cross-domain image retrieval, unsupervised learning.

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_faeMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper tackles unsupervised cross-domain image retrieval by proposing a framework called DUDE. It uses a frozen Stable Diffusion model to help the image encoder learn semantic object representations, and then applies a progressive contrastive learning strategy for cross-domain alignment. Specifically, image features are injected into prompts with domain style descriptions to train the encoder via a denoising loss. The alignment is then gradually expanded from instance augmentation and intra-domain mutual nearest neighbors (MNN) to cross-domain MNN alignment. Experiments on PACS, Office-Home, and DomainNet show solid improvements over existing baselines.

Critical strengths
  1. Exploring diffusion model priors for unsupervised cross-domain retrieval by decoupling object semantics from domain style is an interesting and novel direction.

  2. The overall pipeline makes sense. Combining representation learning with progressive alignment works well in practice, as shown by the ablation studies. Also, only the image encoder is needed during inference, which keeps the retrieval phase efficient.

Critical weaknesses
  1. Training with a denoising objective does not automatically guarantee that object semantics are cleanly separated from domain styles. There seems to be a gap between the theoretical claims and the actual optimization. While the final retrieval metrics are good, there is a lack of direct analysis or visualization to verify that the decoupling actually happens as intended.

  2. Since the method introduces strong pre-trained priors from Stable Diffusion, it is hard to tell how much of the improvement comes from the proposed alignment design versus the extra pre-trained knowledge itself. Without isolating this factor, it is difficult to evaluate the true contribution of the proposed framework over traditional baselines.

  3. The progressive alignment strategy needs more justification. Although the ablation study shows that adding each stage improves performance, it does not properly control for the training schedule and total training budget (e.g., epochs). In addition, there is not enough discussion on how reliable the MNN matching is during the early stages and how stable the performance gains are.

Questions for authors

1.In practice, how does the optimization process guarantee that the learned representations actually satisfy the "minimal sufficiency" condition discussed in the appendix?

2.With the same Stable Diffusion prior, how much performance difference is there between using a simple "neutral" prompt versus a prompt that explicitly specifies the domain? Also, if we apply this same diffusion-guided initialization to existing baseline alignment methods, would DUDE still outperform them?

3.Could the authors clarify whether the disentanglement loss is still kept during the later alignment stages? Also, how are the training epochs allocated across the different ablation variants?

6 comments
AuthorsOpenReview ↗

We sincerely thank the reviewer for the constructive feedback. In response, (i) we clarify the scope of the minimal-sufficiency analysis and provide direct evidence of object–domain decoupling.

1. Minimal sufficiency and direct evidence of object–domain decoupling.

  • Minimal sufficiency is a theoretical solution, rather than a guarantee of the convergence optimization.

We appreciate the reviewer for pointing out the distinction between the theoretical characterization and the practical optimization. The theoretical interpretation in appendix does not intend to claim that minimizing LODL_{\mathrm{OD}} necessarily converges to a minimally sufficient representation. Instead, it characterizes two properties of a non-redundant solution under the condition on a domain token DD and an object token ZZ.

  1. H(O∣Z⋆,D)=0H(O\mid Z^\star,D)=0: Z⋆Z^\star cannot discard information needed to distinguish object categories.
  2. I(Z⋆;D∣O)=0I(Z^\star;D\mid O)=0: As the domain token already provides the style condition required for denoising, domain-style information in Z⋆Z^\star is redundant and could be removed without degrading the denoising objective.

Thus, minimum sufficiency represents an ideal limiting solution among denoising-optimal representations, rather than an optimization guarantee. In practice, finite data, limited model capacity, and gradient-based optimization leave residual style information in ZZ. We will revise the corresponding claims to make this distinction explicit.

  • Linear-probing results provide direct evidence that the decoupling actually happens as intended.

To directly examine whether the intended decoupling occurs, we add linear-probing (LP) experiments. Specifically, we freeze the representations and train separate linear classifiers to predict the object category and domain. For the domain probe, we train a linear binary classifier. To avoid different category distributions becoming a shortcut for domain LP, we balance the two domains within every object category by sampling an equal number of images from each domain, preventing differences in category distributions from becoming a shortcut for domain prediction. We use the labels only for this post-hoc analysis and do not involve them in DUDE training.

DatasetRepresentationCategory LP ↑Domain LP ↓
PACSResNet50 Baseline35.5683.62
DUDE disentanglement61.5767.53
Office-HomeResNet50 Baseline26.2389.08
DUDE disentanglement47.6271.69
DomainNetResNet50 Baseline40.9782.53
DUDE disentanglement81.7764.70

Compared with the ResNet50 baseline, DUDE reduces domain-probe accuracy by 16.09, 17.39, and 17.83% on PACS, Office-Home, and DomainNet, respectively, while improving category-probe accuracy by 26.01, 21.39, and 40.80%. These results show that DUDE disentanglement substantially reduces the domain semantics retained in the features, while preserving rich category semantics.

AuthorsOpenReview ↗

In response, (ii) we isolate the effects of domain-aware prompting and pre-trained diffusion knowledge.

2. The effects of the domain prompt and diffusion pre-trained knowledge.

2.1 Domain-aware prompt v.s. neutral prompt

  • Explicitly providing the domain condition consistently outperforms a neutral prompt under the same Stable Diffusion prior.

To isolate the effect of domain conditioning, we compare DUDE's domain-aware prompt an {domain} of a [z], with the neutral prompt an image of a [z], while keeping the remaining training configuration unchanged. The table reports DomainNet retrieval performance obtained directly using the learned representation ZZ after 50 epochs of training with LODL_{\mathrm{OD}}. No progressive alignment is applied in either setting.

Prompt / P@50AverageClipart→SketchSketch→ClipartInfograph→RealReal→InfographInfograph→SketchSketch→InfographPainting→ClipartClipart→Painting
Domain Prompt67.6875.7682.7136.1775.2142.7864.5291.0173.27
Neutral Prompt63.3266.4276.7034.8771.3638.1957.4890.6570.91

The domain-aware prompt improves average P@50 by 4.36%. The gains are especially clear for transfers involving strongly different visual styles, such as Clipart and Sketch. This comparison supports our motivation: explicitly supplying domain style to the diffusion model encourages the learned image token to focus more strongly on object-related information.

2.2 Applying the same diffusion guidance to an existing alignment method

  • The raw Stable Diffusion prior alone is insufficient for cross-domain retrieval, even when combined with an existing alignment method.

  • Under the same LODL_{\mathrm{OD}} representation, DUDE's progressive alignment still outperforms ProtoOT.

We use ProtoOT as a representative state-of-the-art alignment module and evaluate the following settings on DomainNet:

  1. Stable Diffusion: directly use the frozen Stable Diffusion representation for retrieval.
  2. Stable Diffusion + ProtoOT: apply ProtoOT to the same frozen Stable Diffusion representation.
  3. DUDE (LODL_{\mathrm{OD}}) + ProtoOT: use Stable Diffusion to guide the learning of our designed object-focused representation through LODL_{\mathrm{OD}}, and then align this representation using ProtoOT.
  4. DUDE (LODL_{\mathrm{OD}}) + DUDE (LPA1,LPA2L_{\mathrm{PA1}},L_{\mathrm{PA2}}): use the same LODL_{\mathrm{OD}} representation as in (3), but replace ProtoOT with our progressive alignment strategy.
Method / P@50AverageClipart→SketchSketch→ClipartInfograph→RealReal→InfographInfograph→SketchSketch→InfographPainting→ClipartClipart→Painting
Stable Diffusion17.4219.6021.5511.2015.8512.0316.7325.6316.79
Stable Diffusion + ProtoOT28.6730.3235.6217.4933.1415.9927.3638.8230.61
DUDE (LODL_{\mathrm{OD}}) + ProtoOT81.3895.6096.7350.2875.6768.5584.1693.1186.92
DUDE (LODL_{\mathrm{OD}}) + DUDE (LPA1,LPA2L_{\mathrm{PA1}},L_{\mathrm{PA2}})82.6293.0795.2155.4878.4970.2888.2093.2387.01

The raw Stable Diffusion representation obtains only 17.42% P@50, and applying ProtoOT raises it to 28.67%. In contrast, learning an object-focused representation with LODL_{\mathrm{OD}} and then applying ProtoOT reaches 81.38%. This shows that the improvement cannot be attributed merely to importing pretrained Stable Diffusion knowledge. Instead, the proposed diffusion-guided object-disentanglement objective is essential for converting the diffusion prior into a representation suitable for cross-domain retrieval. Notably, our object disentanglement strategy LODL_{\mathrm{OD}} should not be regarded as merely a "Stable Diffusion initialization". As also noted by Reviewer xtdf, "The idea of disentangling object features from domain-specific styles has not been explored in prior work". It is an important design of our work.

More importantly, with the same object representation, DUDE improves average P@50 from 81.38 to 82.62%. The comparison demonstrates that our progressive alignment provides an additional complementary improvement.

AuthorsOpenReview ↗

In response, (iii) we clarify the training schedules of the alignment ablations and analyze MNN reliability throughout the alignment process.

3. Ablation in alignment stages and MNN reliability

3.1 Is LODL_{\mathrm{OD}} retained during the alignment stages?

  • No. As stated in Appendix C.2 (“Implementation Details”), LODL_{\mathrm{OD}} is used only during the first 50 epochs and is disabled during the subsequent alignment stages.

The full DUDE training schedule is 5050 epochs of object disentanglement with LODL_{\mathrm{OD}}, followed by 3030 epochs of first-stage alignment with LPA1L_{\mathrm{PA1}} and 2020 epochs of second-stage alignment with LPA2L_{\mathrm{PA2}}. Thus, the three objectives are applied sequentially rather than jointly. We apologize that this training schedule was not sufficiently prominent in the main text. We will clarify it in the revised manuscript.

3.2 Training budget in the ablation study

  • All alignment-only variants use the same 50-epoch budget, and all variants initialized by LODL_{\mathrm{OD}} use the same 100-epoch total budget.

The exact schedules are as follows:

AblationTraining EpochsDomainNet P@50
LPA1L_{PA1}5052.78
LPA2L_{PA2}5053.49
LPA1L_{PA1} (30) and LPA2L_{PA2} (20)30+20=5057.89
LODL_{OD} (50) and LPA1L_{PA1} (50)50+50=10076.56
LODL_{OD} (50) and LPA2L_{PA2} (50)50+50=10079.16
LODL_{OD} (50), LPA1L_{PA1} (30) and LPA2L_{PA2} (20)50+30+20=10082.62

These fixed-budget comparisons show that the improvement is not caused simply by increasing the number of alignment epochs. The progressive LPA1L_{\mathrm{PA1}} then LPA2L_{\mathrm{PA2}} schedule improves over either single-stage variant. In particular, directly applying cross-domain alignment through LPA2L_{\mathrm{PA2}} without first stabilizing the representation using LPA1L_{\mathrm{PA1}} is substantially less effective.

3.3 Reliability of MNN matching during alignment

  • Cross-domain MNNs are indeed noisy early in training. DUDE is designed to delay their use until after in-domain structure has been consolidated.

We quantify MNN reliability on the Office-Home Art–Product task using MNN precision, defined as the proportion of selected MNN pairs that share the same ground-truth category. Category labels are used only for this post-hoc evaluation and are never available during training. We use the checkpoint immediately after the 50-epoch LODL_{\mathrm{OD}} stage as the starting point, i.e., the representation state at which progressive alignment begins.

EpochsTraining objectiveIn-domain MNN precision ↑Cross-domain MNN precision ↑
50 (Start of alignment)LODL_{\mathrm{OD}} (50)49.0030.88
50 + 10LODL_{\mathrm{OD}} (50) and LPA1L_{\mathrm{PA1}} (10)60.8647.53
50 + 30LODL_{\mathrm{OD}} (50) and LPA1L_{\mathrm{PA1}} (30)66.4146.89
50 + 30 + 20LODL_{\mathrm{OD}} (50), LPA1L_{\mathrm{PA1}} (30) and LPA2L_{\mathrm{PA2}} (20)64.0351.70

At the beginning of alignment, cross-domain MNN precision is only 30.88%. Directly using cross-domain correspondences at this point would introduce considerable noise. During LPA1L_{\mathrm{PA1}}, DUDE therefore uses only in-domain MNN pairs to consolidate the local semantic structure. After 30 epochs of LPA1L_{\mathrm{PA1}}, in-domain MNN precision increases from 49.00 to 66.41%. Cross-domain MNNs are activated only in the subsequent LPA2L_{\mathrm{PA2}} stage, after which their precision increases from 46.89 to 51.70%.

Therefore, DUDE does not assume that early cross-domain MNNs are fully reliable. Instead, it explicitly avoids using these noisier correspondences in the early stage and activates cross-domain alignment only after in-domain stabilization. We will add the MNN-reliability analysis to the revision.

Reviewer_xtdfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents DUDE for unsupervised cross-domain image retrieval (UCIR), whose key idea is to disentangle object features from domain-specific styles and thus facilitate cross-domain category-level retrieval. The new model adopts a disentangle-then-align strategy and consists of two modules, corresponding to its two technical contributions: (1) an Object Disentanglement Module that uses a frozen diffusion-based T2I model (i.e., Stable Diffusion) with style-specific prompts and a learnable object-semantic token to disentangle domain-invariant object features from domain-specific styles; and (2) a Progressive Alignment Module that progressively constructs positive contrastive pairs from instance augmentations, in-domain mutual neighbors, and cross-domain mutual neighbors. Experiments are conducted on the PACS, Office-Home, and DomainNet datasets, and the results show that the new model outperforms the compared baseline methods, such as ProtoOT and CoDA. The authors also conduct ablation studies on the effectiveness of the loss components, while t-SNE and prompt-swapping visualizations further support the disentanglement design.

Critical strengths

The idea of disentangling object features from domain-specific styles proposed in this paper is intuitive and makes good use of the capabilities of existingT2I models. The results are also supportive. However, since I am not an expert in this field, I am somewhat surprised that this idea has not been explored in prior work. One related work I can think of is DEADiff[1]. Although that work focuses on image generation, it proposes a similar idea of feature disentanglement. This work should be discussed in related work.

[1] DEADiff: An Efficient Stylization Diffusion Model with Disentangled Representation, CVPR 2024

Critical weaknesses

None

Questions for authors
  1. How is the object token z injected into the frozen text encoder? The authors should provide further details.
  2. It is unclear whether the disentanglement loss L_OD remains active during the subsequent alignment stages.
2 comments
AuthorsOpenReview ↗

We sincerely thank the reviewer for the positive assessment and constructive suggestions. In response, we: (i) discuss the relationship between DUDE and DEADiff; (ii) clarify how the image-dependent object token is injected into the frozen text encoder; and (iii) provide the complete training schedule.

1. Relationship to DEADiff.

  • It is inspiring that DEADiff also demonstrates a broader potential of diffusion models for feature disentanglement, with a different approach from our DUDE.

DEADiff is designed for controllable image stylization and explicitly learns separate content and style representations from paired training data. In contrast, DUDE addresses unsupervised cross-domain image retrieval and requires neither category labels nor paired images. Rather than learning two image-conditioned encoder branches, DUDE represents the domain style using a domain-specific text token and learns only an image-dependent object token. By providing the domain condition separately to the frozen diffusion model, the denoising objective encourages the image token to retain object-relevant information while reducing redundant domain-style information.

We will cite DEADiff and clarify both this conceptual connection and the methodological differences in the revised manuscript. We agree that these complementary findings demonstrate the broader value of diffusion priors for disentangled representation learning.

2. How is the object token z injected into the frozen text encoder?

The injection consists of the following three steps.

  1. Register and locate the placeholder token. We first register [z] as a special token in the tokenizer. Given a prompt such as a sketch of a [z], the tokenizer maps each token to its token ID. The frozen token-embedding layer would ordinarily map the placeholder ID to a 768-dimensional embedding:
\in\mathbb{R}^{768}.$$ In our implementation, the registered ID is used to locate the placeholder position in the tokenized sequence; its original lookup embedding is subsequently replaced. 2. **Extract a dimension-matched object representation from the image.** Following prior UCIR methods, the image encoder $f_\theta$ is initialized with an ImageNet-pretrained ResNet50. We replace its final classification layer with a linear projection from 2048 to 768 dimensions, matching the token-embedding dimension of the frozen text encoder: $$z_i=f_\theta(I_i)\in\mathbb{R}^{768}.$$ 3. **Replace the placeholder embedding.** We find the position whose token ID equals \$id_{[z]\}\$ and replace its embedding \$e_{[z]\}\$. Omitting the start, end, and padding tokens for clarity, the replacement is: $$[e_{\mathrm{a}},e_{\mathrm{sketch}},e_{\mathrm{of}}, e_{\mathrm{a}},e_{[z]}] \quad\longrightarrow\quad [e_{\mathrm{a}},e_{\mathrm{sketch}},e_{\mathrm{of}}, e_{\mathrm{a}},z_i].$$ Consequently, $z_i$ occupies exactly the same sequence position as the `[z]` placeholder. The resulting prompt-embedding sequence has shape $n\times768$. After the standard positional embeddings are added, the complete sequence is passed through the frozen text Transformer. Gradients are propagated through $z_i$ to update the image encoder $f_\theta$, while the tokenizer, token-embedding layer, and text Transformer remain frozen. We will add these implementation details and the corresponding equations to the revised manuscript. ### 3. Does $L_{\mathrm{OD}}$ remain active during alignment? - **As stated in Appendix C.2 (“Implementation Details”), $L_{\mathrm{OD}}$ is used only during the first 50 epochs and is disabled during both subsequent alignment stages.** DUDE follows a sequential disentangle-then-align schedule: 1. **Object disentanglement:** train $f_\theta$ with $L_{\mathrm{OD}}$ for the first 50 epochs. 2. **First-stage alignment:** disable $L_{\mathrm{OD}}$ and continue training $f_\theta$ with $L_{\mathrm{PA1}}$ for the next 30 epochs. 3. **Second-stage alignment:** continue training $f_\theta$ with $L_{\mathrm{PA2}}$ for the final 20 epochs. Thus, the full schedule is $$50\; (L_{\mathrm{OD}}) + 30\; (L_{\mathrm{PA1}}) + 20\; (L_{\mathrm{PA2}}) = 100\text{ epochs}.$$ The image encoder remains trainable throughout all three stages, but the diffusion model is invoked only during the initial $L_{\mathrm{OD}}$ stage. Disabling $L_{\mathrm{OD}}$ during alignment avoids the expensive diffusion forward pass and allows the later stages to focus exclusively on progressively aligning the learned object representation. We will state this schedule explicitly in the revised paper.
AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Thank you again for your valuable comments and positive assessment of our submission. As the discussion period is drawing to a close, we would greatly appreciate it if you could let us know whether our rebuttal has adequately addressed your concerns. If any questions or concerns remain, we would be happy to provide further clarification and continue the discussion. Thank you for your time and consideration.

Reviewer_8ZMwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DUDE for unsupervised cross-domain image retrieval. The method uses a frozen Stable Diffusion model to guide feature learning through a denoising objective. Each prompt contains a domain-style descriptor and an image-dependent object token produced by the feature extractor. The learned representations are then aligned progressively using augmented views, in-domain mutual nearest neighbors, and cross-domain mutual nearest neighbors.

Experiments on PACS, Office-Home, and DomainNet show improvements over the evaluated baselines. The paper also includes ablation studies, prompt comparisons, and feature visualizations.

Critical strengths
  1. The use of a frozen diffusion model to train retrieval features is an interesting design. In particular, the image-dependent token is learned through the diffusion denoising objective, while the domain prompt provides style information. This differs from directly using the pretrained model as a feature extractor. The final retrieval model also does not require running Stable Diffusion at inference time.

  2. The reported retrieval improvements are substantial. DUDE outperforms the evaluated baselines across all three datasets. For example, Table 1 reports an improvement from 50.26% to 71.09% in P@50 on PACS compared with ProtoOT. Table 2 further shows that both the diffusion-guided loss and the alignment losses contribute to the final results.

Critical weaknesses
  1. The evidence for object-style disentanglement is still limited.

The main claim of the paper is that the learned object token [z] captures object semantics while excluding domain-specific style information. However, the denoising objective in Eq. (4) does not explicitly prevent [z] from retaining style information. Appendix B introduces a minimum-sufficiency assumption, but the actual training objective does not enforce this constraint.

Figure 3 provides qualitative prompt-swapping examples, and Table 8 shows that domain-specific prompts perform better than a fixed prompt. These results are useful, but they do not directly establish that the learned representation contains less domain-specific information. Since disentanglement is central to the paper, I think more direct evidence is needed.

  1. The novelty over related work and the source of the performance gains need to be better established.

The motivation of separating domain-specific information from semantic information is not entirely new to UCIR. In particular, TPSNet (Yang et al., CVPR 2026) already addresses semantic degradation caused by domain-specific information and uses pretrained CLIP-based text priors together with phase features. DUDE uses a different mechanism, but the paper does not sufficiently clarify what is gained by diffusion-guided disentanglement over these existing approaches.

The progressive alignment component also builds on established techniques, including instance-level contrastive learning, mutual nearest neighbors, and cross-domain alignment. Its main contribution appears to be the training schedule rather than a fundamentally new alignment mechanism.

More importantly, DUDE introduces additional pretrained knowledge from Stable Diffusion, while the evaluated baselines do not use comparable pretrained generative or vision-language priors. Table 2 demonstrates the benefit of the diffusion-guided loss, but does not separate the effect of the proposed disentanglement mechanism from the benefit of additional pretraining. A comparison with TPSNet and a controlled pretrained-feature supervision baseline would make the claimed contribution much more convincing.

Questions for authors
  1. Could the authors provide a quantitative evaluation of how much domain-specific information remains in [z]? For example, a domain classification probe could be used to compare DUDE with the ResNet-50 baseline and relevant pretrained-feature baselines. This would help clarify whether the proposed training objective actually reduces style information in the learned representation.

  2. How does DUDE compare with TPSNet (Yang et al., CVPR 2026)? Also, under the same progressive alignment procedure, how would a ResNet-50 trained using CLIP image-feature distillation perform? I think these comparisons are important for distinguishing the contribution of diffusion-guided disentanglement from that of pretrained semantic knowledge.

  3. Could the authors clarify the training and evaluation settings? Appendix C.2 describes 50 epochs of diffusion-guided training, followed by 30 epochs of in-domain alignment and 20 epochs of cross-domain alignment. How do these correspond to the two 50-epoch stages in Table 3? Also, are the features evaluated in Table 8 obtained after diffusion-guided training or after the complete alignment procedure?

5 comments
AuthorsOpenReview ↗

We sincerely thank the reviewer for the careful assessment and constructive suggestions. In response, (i) we provide direct quantitative evidence of object–domain decoupling.

1. The quantitative evidence for object-style disentanglement.

  • Linear-probing results provide direct evidence that the disentangled representation contains less domain-specific information.

To quantify how much domain-specific information remains in ZZ, we conduct linear-probing (LP) experiments for both object-category and domain prediction. Specifically, we freeze the representations and train separate linear classifiers to predict the domain or object category. For the domain probe, we train a linear binary classifier. To avoid different category distributions becoming a shortcut for domain LP, we balance the two domains within every object category by sampling an equal number of images from each domain, preventing differences in category distributions from becoming a shortcut for domain prediction. We use the labels only for this post-hoc analysis and do not involve them in DUDE training.

DatasetRepresentationCategory LP ↑Domain LP ↓
PACSResNet50 Baseline35.5683.62
DUDE disentanglement61.5767.53
Office-HomeResNet50 Baseline26.2389.08
DUDE disentanglement47.6271.69
DomainNetResNet50 Baseline40.9782.53
DUDE disentanglement81.7764.70

Compared with the ResNet-50 baseline, DUDE reduces domain-probe accuracy by 16.09, 17.39, and 17.83% on PACS, Office-Home, and DomainNet, respectively. Meanwhile, category-probe accuracy increases by 26.01, 21.39, and 40.80%. These results show that the learned representation becomes substantially more informantion of object category and less of domain style. We will include this analysis in the revision.

AuthorsOpenReview ↗

In response, (ii) we clarify the methodological differences from TPSNet and provide controlled comparisons under CLIP image-encoder initialization.

2. The novelty over related work and the source of the performance gains.

2.1 Compare with TPSNet (Yang et al., CVPR 2026)

  • DUDE and TPSNet share the goal of preserving semantics across domains, but they use fundamentally different priors and learning mechanisms.

TPSNet uses CLIP-generated class-specific domain prompts as a text prior and incorporates domain-invariant phase features as an additional phase prior. In contrast, DUDE uses neither class-specific prompts nor phase features. Instead, it repurposes the text-conditioning interface of a frozen diffusion model for object–domain decoupling: the textual prompt explicitly supplies the domain-style condition, while an image-dependent token learned through the denoising objective represents the object semantics. This conversion of diffusion-based text conditioning into a mechanism for learning object-focused retrieval representations constitutes a key design and contribution of DUDE.

  • Our contribution in the alignment module is its reliability-aware progressive organization strategy.

We agree that instance contrastive learning and MNN matching are established techniques. Our contribution in the alignment module is their reliability-aware progressive organization: DUDE first consolidates local structure using instance augmentations and in-domain MNNs, and introduces noisier cross-domain MNNs only after this structure has been stabilized.

  • Under the same MoCo-v2–ResNet50 initialization, DUDE consistently outperforms TPSNet.

TPSNet's main results use a CLIP–ResNet-50 image encoder, whereas the original DUDE experiments use a MoCo-v2–ResNet-50 encoder; therefore, those main results are not directly comparable. Fortunately, TPSNet also reports results with MoCo-v2 initialization. We compare the two methods under this matched setting and compute the averages over the same five DomainNet domains and eight retrieval directions.

Method and image initializationP@50P@100P@200
DUDE (MoCo-v2-ResNet50)82.6280.0774.90
TPSNet (MoCo-v2-ResNet50)80.0177.9473.82
Improvement+2.61+2.13+1.08

Under the same MoCo-v2-ResNet50 image initialization and evaluation directions, DUDE consistently outperforms TPSNet by 2.61, 2.13, and 1.08% at P@50, P@100, and P@200, respectively. This comparison indicates that DUDE remains competitive.

2.2 Performance using CLIP ResNet.

  • DUDE also outperforms TPSNet on average when both methods use a CLIP–ResNet-50 image encoder.

We additionally initialize DUDE with the same CLIP–ResNet50 visual backbone used by TPSNet, while retaining DUDE's training procedure. We compare the two methods under this matched setting and compute the averages over the same five DomainNet domains and eight retrieval directions.

Method and image initializationP@50P@100P@200
DUDE (CLIP-ResNet50)88.4786.8181.79
TPSNet (CLIP-ResNet50)86.6885.3381.52
Improvement+1.79+1.48+0.27

Under the CLIP–ResNet50 setting, DUDE achieves 88.47 average P@50, compared with 86.68 for TPSNet, yielding an improvement of 1.79%. Together with the MoCo-v2 comparison, these results show that DUDE's advantage is consistent across two different image-encoder initializations.

AuthorsOpenReview ↗

In response, (iii) we disentangle the contributions of pretrained diffusion knowledge, object disentanglement, and progressive alignment.

2.3 Distinguishing the contribution from pretrained semantic knowledge.

  • The raw Stable Diffusion representation performs poorly for cross-domain retrieval, even after applying an existing alignment method.

  • When the diffusion-guided object representation is held fixed, DUDE's progressive alignment still outperforms ProtoOT.

To separate the benefit of pretrained diffusion knowledge from the contributions of our object-disentanglement and alignment designs, we evaluate four progressively controlled settings:

  1. Stable Diffusion: directly use the frozen Stable Diffusion representation for retrieval.
  2. Stable Diffusion + ProtoOT: apply an existing alignment method to the same Stable Diffusion representation.
  3. DUDE (LODL_{\mathrm{OD}}) + ProtoOT: learn an object-focused representation through our diffusion-guided LODL_{\mathrm{OD}}, and then align it using ProtoOT.
  4. Full DUDE: use the same LODL_{\mathrm{OD}} representation, but replace ProtoOT with our progressive alignment strategy, LPA1L_{\mathrm{PA1}} followed by LPA2L_{\mathrm{PA2}}.
Method / P@50AverageClipart→SketchSketch→ClipartInfograph→RealReal→InfographInfograph→SketchSketch→InfographPainting→ClipartClipart→Painting
Stable Diffusion17.4219.6021.5511.2015.8512.0316.7325.6316.79
Stable Diffusion + ProtoOT28.6730.3235.6217.4933.1415.9927.3638.8230.61
DUDE (LODL_{\mathrm{OD}}) + ProtoOT81.3895.6096.7350.2875.6768.5584.1693.1186.92
DUDE (LODL_{\mathrm{OD}}) + DUDE (LPA1,LPA2L_{\mathrm{PA1}},L_{\mathrm{PA2}})82.6293.0795.2155.4878.4970.2888.2093.2387.01

The frozen Stable Diffusion representation obtains only 17.42 average P@50, and ProtoOT improves it to only 28.67. In contrast, learning the object representation through LODL_{\mathrm{OD}} before applying the same ProtoOT alignment raises average P@50 to 81.38. Thus, the gain cannot be explained by merely importing pretrained Stable Diffusion features; the proposed diffusion-guided learning objective is critical for converting this prior into a representation suitable for cross-domain retrieval.

Moreover, with the LODL_{\mathrm{OD}} representation controlled, replacing ProtoOT with our progressive alignment increases average P@50 from 81.38 to 82.62. The last two rows therefore isolate an additional 1.24% contribution from the proposed alignment schedule.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summarization of Rebuttal

Dear Area Chair and reviewers,

Thank you for your time and constructive feedback. With the rebuttal period nearing its end, we would like to summarize the assessments and our rebuttal in the following points:

Strengths recognized by reviewers

  1. Novelty: The idea of disentangling object features from domain-specific styles proposed in this paper is intuitive and has not been explored in prior work (Reviewer xtdf). Exploring diffusion model priors for unsupervised cross-domain retrieval by decoupling object semantics from domain style is an interesting and novel direction (Reviewer faeM).
  2. Method: The use of a frozen diffusion model to train retrieval features is an interesting design (Reviewer 8ZMw). The overall pipeline makes sense (Reviewer faeM).
  3. Evaluation: The reported retrieval improvements are substantial. DUDE outperforms the evaluated baselines across all three datasets (Reviewer 8ZMw). Only the image encoder is needed during inference, which keeps the retrieval phase efficient (Reviewer faeM).

Key new evidence

  1. Disentanglement is empirically verified. We add linear-probing experiments that show that DUDE reduces domain-prediction accuracy, confirming the practical effectiveness of the proposed decoupling.

  2. The gains come from DUDE, not extra pretraining knowledge. We compare the raw Stable Diffusion features (17.42%), Stable Diffusion + ProtoOT (28.67%), and our diffusion-guided disentanglement with ProtoOT (81.38%). This shows that pretrained knowledge alone provides little retrieval semantics.

  3. DUDE outperforms the latest related method under matched settings. On DomainNet, we compare DUDE with TPSNet using the same five domains and the same eight cross-domain retrieval directions. Under matched MoCo-v2–ResNet-50 and CLIP–ResNet-50 backbones, DUDE improves average P@50 by 2.61% and 1.79%, respectively. This confirms that our method is consistently effective across different backbone architectures.

  4. Progressive alignment is fairly and empirically justified. Equal-budget ablations confirm that the gains do not result from additional epochs. Cross-domain MNN precision is initially only 30.88 and increases to 51.70 after progressive alignment, supporting our strategy of stabilizing in-domain structure before activating cross-domain alignment.

We are encouraged that Reviewer faeM found our clarifications satisfactory and updated their score. We also appreciate Reviewer xtdf’s recognition that our idea of disentangling object semantics from domain-specific styles surprised him and has not been explored in prior work. Although Reviewer 8ZMw did not reply, we believe our rebuttal addresses their concerns: two of their three questions overlap with those of Reviewer faeM, and the remaining one could be addressed by additional experimental evidence.

More broadly, we hope DUDE contributes a new perspective to cross-domain retrieval and starts a new way to exploit diffusion models beyond image generation. We believe that repurposing knowledge from seemingly different research areas can inspire further advances in retrieval, representation learning, domain adaptation, and other related tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.