- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces FusionDiff for Zero-Shot Composed Image Retrieval (ZS-CIR). Unlike prior works that map images into text tokens (Pic2Word, SEARLE, LinCIR) or generate pixel-level target images (CIG, IP-CIR), the authors run diffusion directly inside the embedding space of a frozen MLLM (GME-Qwen2-VL-2B). They first pre-train a DiT backbone on 33M image-text pairs to model the joint embedding space, and then train a ControlNet-style adapter on 200K synthetic triplets from HQEdit to handle image modifications. At test time, the MLLM also generates a text description of the target image to guide the diffusion sampling. The method reports strong numbers on CIRCO, CIRR, and FashionIQ.
Critical strengths
Moving CIR diffusion into MLLM embedding space: Prior diffusion-based CIR methods either suffer from pixel generation artifacts (CIG) or are limited by CLIP's dual-encoder space (CompoDiff). Using a unified MLLM retriever space (GME-Qwen2-VL and E5-V) as the diffusion target space makes sense and clearly preserves visual details better than text-only bottlenecks, as shown in the main tables.
Preventing forgetting via Control-Adapter: Freezing the Stage-1 diffusion model and only training a zero-initialized parallel branch on the 200K editing triplets works well. Table 7 convincingly shows that full fine-tuning ruins the pre-trained representation, so this design choice is well-justified.
Critical weaknesses
Unfair backbone comparison and questionable "data efficiency" claims: Every baseline in Tables 1 and 2 (Pic2Word, LinCIR, CompoDiff, CIReVL, LDRE) builds on standard CLIP. FusionDiff, however, combines GME-Qwen2-VL-2B, Flan-T5-XXL (11B), and a ~1.2B DiT+Adapter. Looking at Table 6, simply doing nearest-neighbor search with the frozen GME-Qwen2-VL-2B model, without any diffusion or training at al, already gets 36.52 R@1 on CIRR, which beats almost all CLIP baselines. The proposed diffusion module only adds about 3.5 points over this direct baseline. Also, claiming 100x better data efficiency than CompoDiff (200K vs 18M) ignores the fact that Stage 1 still requires pre-training a 954M DiT on 33M pairs, and training on 200K supervised editing triplets from HQEdit is arguably not strictly "zero-shot" compared to training-free MLLM baselines like CIReVL or LDRE.
Train-test mismatches and odd design choices:
-
Target mismatch: In Stages 1 and 2, the diffusion model is trained to output an embedding of a target image paired with its caption. But at test time, gallery images are encoded using the image alone. Why train the model to predict image+text embeddings when retrieving image-only gallery embeddings?
-
Conditioning mismatch: During Stage 2 training, the text condition is the short edit instruction. At test time, it is swapped for a long target description generated by the MLLM. There is no ablation showing how much of the gain comes simply from this MLLM description generation step (like CIReVL/LDRE) versus the diffusion model itself.
Incomplete literature review and weak dual-encoder baselines: The paper claims existing ZS-CIR methods are strictly limited to textual projection or noisy pixel synthesis, overlooking recent embedding-space interpolation and alignment methods such as Slerp (ECCV 2024).
Questions for authors
Simple training-free baseline: Since you already generate a target description with the MLLM at test time (Eq. 12), what is the score if you simply average or ensemble the raw query embedding and the generated description's embedding directly using GME-Qwen2-VL without running diffusion?
Target formulation: Why did you include the target text caption when constructing the ground-truth target embedding in Stage 2 (Eq. 7), given that gallery items at test time only have images?