acceptodds
Withdrawn from ICLR 2027

Fused Embedding Edit for Zero-Shot Composed Image Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Zero-Shot Composed Image Retrieval (ZS-CIR) retrieves target images using a reference image paired with a modification text, without training on costly real triplet annotations. Existing methods project images into text space or synthesize pseudo-visual features, ultimately performing cross-modal matching that suffers from an inherent *representation gap*, losing fine-grained visual details. We propose **FusionDiff**, a framework that addresses this representation gap through retrieval within a unified joint vision-language embedding space. Its conditional embedding diffusion model learns the distribution of target fusion embeddings conditioned on the composed query. For data efficiency, we design a lightweight Control-Adapter that adapts pre-trained diffusion backbones using only 200K synthetic triplets (*two orders of magnitude less* than prior methods). Experiments on three benchmarks demonstrate state-of-the-art performance: +2.34 mAP@50 on CIRCO, +1.28 R@10 on CIRR, and +1.58 R@50 on FashionIQ over previous best methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Composed Image Retrieval, Diffusion Models, Multimodal Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oRbxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FusionDiff for Zero-Shot Composed Image Retrieval (ZS-CIR). Unlike prior works that map images into text tokens (Pic2Word, SEARLE, LinCIR) or generate pixel-level target images (CIG, IP-CIR), the authors run diffusion directly inside the embedding space of a frozen MLLM (GME-Qwen2-VL-2B). They first pre-train a DiT backbone on 33M image-text pairs to model the joint embedding space, and then train a ControlNet-style adapter on 200K synthetic triplets from HQEdit to handle image modifications. At test time, the MLLM also generates a text description of the target image to guide the diffusion sampling. The method reports strong numbers on CIRCO, CIRR, and FashionIQ.

Critical strengths

Moving CIR diffusion into MLLM embedding space: Prior diffusion-based CIR methods either suffer from pixel generation artifacts (CIG) or are limited by CLIP's dual-encoder space (CompoDiff). Using a unified MLLM retriever space (GME-Qwen2-VL and E5-V) as the diffusion target space makes sense and clearly preserves visual details better than text-only bottlenecks, as shown in the main tables.

Preventing forgetting via Control-Adapter: Freezing the Stage-1 diffusion model and only training a zero-initialized parallel branch on the 200K editing triplets works well. Table 7 convincingly shows that full fine-tuning ruins the pre-trained representation, so this design choice is well-justified.

Critical weaknesses

Unfair backbone comparison and questionable "data efficiency" claims: Every baseline in Tables 1 and 2 (Pic2Word, LinCIR, CompoDiff, CIReVL, LDRE) builds on standard CLIP. FusionDiff, however, combines GME-Qwen2-VL-2B, Flan-T5-XXL (11B), and a ~1.2B DiT+Adapter. Looking at Table 6, simply doing nearest-neighbor search with the frozen GME-Qwen2-VL-2B model, without any diffusion or training at al, already gets 36.52 R@1 on CIRR, which beats almost all CLIP baselines. The proposed diffusion module only adds about 3.5 points over this direct baseline. Also, claiming 100x better data efficiency than CompoDiff (200K vs 18M) ignores the fact that Stage 1 still requires pre-training a 954M DiT on 33M pairs, and training on 200K supervised editing triplets from HQEdit is arguably not strictly "zero-shot" compared to training-free MLLM baselines like CIReVL or LDRE.

Train-test mismatches and odd design choices:

  1. Target mismatch: In Stages 1 and 2, the diffusion model is trained to output an embedding of a target image paired with its caption. But at test time, gallery images are encoded using the image alone. Why train the model to predict image+text embeddings when retrieving image-only gallery embeddings?

  2. Conditioning mismatch: During Stage 2 training, the text condition is the short edit instruction. At test time, it is swapped for a long target description generated by the MLLM. There is no ablation showing how much of the gain comes simply from this MLLM description generation step (like CIReVL/LDRE) versus the diffusion model itself.

Incomplete literature review and weak dual-encoder baselines: The paper claims existing ZS-CIR methods are strictly limited to textual projection or noisy pixel synthesis, overlooking recent embedding-space interpolation and alignment methods such as Slerp (ECCV 2024).

Questions for authors

Simple training-free baseline: Since you already generate a target description with the MLLM at test time (Eq. 12), what is the score if you simply average or ensemble the raw query embedding and the generated description's embedding directly using GME-Qwen2-VL without running diffusion?

Target formulation: Why did you include the target text caption when constructing the ground-truth target embedding in Stage 2 (Eq. 7), given that gallery items at test time only have images?

Reviewer_Q3rBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FusionDiff for zero-shot composed image retrieval. It employs a conditional diffusion model to generate target embeddings in a joint vision-language space and introduces a Control-Adapter for query-conditioned feature editing. The model is pretrained on large-scale image-text pairs. Experimental results prove the effectiveness of the method.

Critical strengths
  1. The paper explores conditional diffusion in a unified vision-language embedding space for composed image retrieval.
  2. The Control-Adapter enables task-specific adaptation while preserving the pretrained diffusion backbone.
  3. The experiments cover three benchmarks, with additional ablations on model components, embedding spaces, and training data scales.
Critical weaknesses
  1. The comparison with recent SOTA methods is incomplete. Several competitive approaches are missing, making the claimed state-of-the-art performance insufficiently supported.
  2. The claimed data efficiency is potentially misleading. Although Stage 2 uses only 200K triplets, Stage 1 requires 33M image-text pairs and substantial diffusion pretraining, making the comparison with methods trained on more triplets incomplete.
  3. The comparison between CLIP and joint vision-language spaces does not fully isolate the effect of the embedding space, since the underlying encoders differ substantially in architecture, capacity, and pretraining. The claimed reduction of the representation gap is therefore not conclusively established.
Questions for authors
  1. How does FusionDiff compare with more recent training-free and diffusion-based CIR methods under comparable retrieval backbones?
  2. Can the authors clarify the computational cost and total training data requirements relative to CompoDiff and other diffusion-based approaches?
Reviewer_baAuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces FusionDiff, which combines fused MLLM embeddings, conditional diffusion, and a Control-Adapter for ZS-CIR. It pretrains on image-text pairs, adapts on synthetic editing triplets, and ranks gallery images using sampled target embeddings. Evaluation covers CIRR, FashionIQ, and CIRCO.

Critical strengths
  1. The paper explores a useful direction for ZS-CIR by refining fused multimodal embeddings without synthesizing pseudo-target images at retrieval time.
  2. The E5-V results also suggest that the approach is not tied to one embedding backbone.
  3. Adapting the frozen diffusion prior is practical. Comparisons with direct retrieval, MLLM fine-tuning, and alternative CLIP fusion strategies provide useful evidence about the design.
Critical weaknesses
  1. The limitation of the contribution in the proposed diffusion formulation. For example, Table 4 shows stronger direct retrieval with fused embeddings and similar CIRR refinement gains in both spaces.Moreover, the MLLM SFT baselines in Table 7 do not replace a deterministic refiner on the same frozen embeddings with comparable pretraining and adaptation data. This raises concerns for me.
  2. The one-to-many explanation needs direct evidence. Appendix E attributes the weaker SFT results to deterministic mappings, but this comparison does not establish that the gains come from modeling multiple valid targets. The value of S is not reported, and the effects of multiple samples and max aggregation are not separated.
  3. The zero-shot protocol needs clarification. GME reports training on M-BEIR, which includes CIRR and FashionIQ training data, whereas Appendix D claims no benchmark exposure. Please identify the exact checkpoint and any exclusions from the backbone's training. The E5-V results do not resolve the provenance of the main GME results.
  4. Training targets combine images and captions, whereas gallery embeddings use images alone. The denoiser is trained with modification instructions but uses generated target descriptions at inference. Please justify these changes and clarify whether the alignment analysis in Table 10 uses the actual image-only gallery embeddings.
  5. The sensitivity results need reconciliation. For example, Figure 6(b) and Table 13 report substantially different FashionIQ R@50 at the same denoising step counts. Figure 6(a) is also labeled as a CIRR test-set sweep. Please clarify which split was used to select the hyperparameters and whether this plot is a post-hoc analysis.
Questions for authors
  1. How does FusionDiff compare with a deterministic refiner on the same frozen embeddings and comparable training data?
  2. Which GME checkpoint was used, and were CIRR and FashionIQ excluded from its training? How does this reconcile with Appendix D?
  3. Does Table 10 use the image-only gallery embeddings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.