acceptodds
Under review as a conference paper at ICLR 2027

Decoupling Endpoint and Semantic Transition Learning for Zero-Shot Composed Image Retrieval

Abstract

Zero-shot composed image retrieval (ZS-CIR) retrieves a target image from a reference image and a text modification without human-annotated CIR triplets. Projection-based ZS-CIR methods are attractive because they do not rely on LLMs at inference and remain lightweight, but they often underperform LLM-based approaches on complex semantic modifications. This gap reflects a semantic transition bottleneck in projection-based ZS-CIR: endpoint-level matching can let the edit text act as a target-side attribute cue rather than grounding it as a source-conditioned semantic transition. We further show that adding semantic transition supervision to the same text adapter creates an endpoint–transition conflict between endpoint alignment and semantic transition alignment. To address this conflict, DeCIR decouples endpoint and transition learning. It constructs paired forward/reverse edit tuples from image-caption pairs, trains separate low-rank text adapter branches for endpoint alignment and semantic transition alignment, and merges them with Low-Rank Directional Merge (LRDM) into one deployable adapter. Extensive experiments on CIRR, CIRCO, FashionIQ, and GeneCIS demonstrate that DeCIR consistently improves projection-based ZS-CIR without increasing inference complexity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
composed image retrieval, zero-shot learning, vision-language models, multimodal retrieval

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VbRgOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies semantic transition learning and objective conflicts in zero-shot composed image retrieval (ZS-CIR) and proposes the DeCIR framework. The authors use LLM-generated forward and backward edit instructions to separately learn target endpoint alignment and semantic transformation through a dual-branch design, and then merge the two branches into a single adapter via low-rank directions. Experiments on CIRR, CIRCO, FashionIQ, and GeneCIS demonstrate the effectiveness of the proposed method without requiring LLM calls during inference.

Critical strengths
  1. The motivation is clear and reasonable.
  2. The experimental evaluation is relatively comprehensive, covering four benchmarks, two backbones, and relevant ablation studies.
  3. The method is deployment-friendly, requiring only a single retrieval path during inference.
Critical weaknesses
  1. The proposed method aims to mitigate endpoint shortcuts, but the current experiments mainly rely on retrieval performance and qualitative examples. More direct evidence is needed to demonstrate that the improvements come from better preservation of source image information.
  2. The contribution of LRDM is not sufficiently isolated, as the comparison with Task Arithmetic involves differences in both parameter sharing during training and the merging strategy. Therefore, the effectiveness of the proposed merging mechanism is not fully validated.
  3. Although the paper reports results averaged over three runs, standard deviations or confidence intervals are missing from the main comparisons. Given the relatively small gains in some settings, further statistical analysis is needed to establish their significance.
Questions for authors
  1. Further evaluation with hard negatives that satisfy the modification text but violate the reference image constraints is suggested to verify whether the proposed method better preserves reference image information.
  2. Comparisons that separately isolate the effects of the shared basis, asymmetric updates, and merging strategy under identical training settings are suggested to better validate their individual contributions.
  3. Standard deviations or confidence intervals should be reported for the main results and ablation studies to better assess the statistical reliability of the reported improvements.
1 comment

Response to Reviewer VbRg

We thank the reviewer for the constructive feedback. We address the concerns below.

W1 / Q1: Direct evidence of source-constraint preservation with hard negatives

We directly evaluate the hard negatives suggested by the reviewer using the GeneCIS Change Attribute task. Each query contains 15 candidate images, including nine distractors that satisfy the requested attribute but violate the reference image's object-category constraint.

For each Top-K cutoff, we identify queries where DeCIR retrieves the correct target but endpoint-only fails. Among these queries, we count those where DeCIR reverses the ranking of at least one hard negative that was previously ranked above the correct target.

CutoffQueries recovered by DeCIRWith hard-negative reversalProportion
Top-1522038.5%
Top-5593762.7%
Top-10363083.3%

These results provide direct evidence of improved source-category preservation. Among the queries recovered by DeCIR, 62.7% at Top-5 and 83.3% at Top-10 include a corrected ranking against at least one attribute-matching, category-mismatched hard negative. We will add this label-based analysis and its evaluation protocol to the appendix.

W2 / Q2: Isolating the contributions of LRDM

We conduct a controlled ablation to examine basis sharing and asymmetric updates under identical training settings:

SettingCIRR Rs@1 ↑CIRCO mAP@50 ↑
Task Arithmetic65.2227.07
Shared B, both objectives update B64.2726.77
Shared B, endpoint-only B update (Ours)66.0827.26

Asymmetric updates are critical to effective shared-basis merging. Under the same shared basis and merging rule, restricting B updates to endpoint alignment improves CIRR Rs@1 from 64.27 to 66.08 and CIRCO mAP@50 from 26.77 to 27.26. These results demonstrate the effectiveness of LRDM's shared-basis parameterization and asymmetric optimization. We will incorporate this ablation and clarify the contributions of each component in Section 3.4.

W3 / Q3: Statistical reliability

We report mean ± standard deviation over three independent training runs on the CIRR and CIRCO validation sets. DeCIR achieves 66.08 ± 0.12 CIRR Rs@1 and 27.26 ± 0.23 CIRCO mAP@50, outperforming endpoint-only training (64.46 ± 0.18 / 24.92 ± 0.21) and joint training (61.09 ± 0.23 / 21.93 ± 0.34).

Reviewer_KNN4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose DeCIR for zero-shot composed image retrieval by decoupling endpoint alignment from semantic transition learning. Using LLM-generated forward and reverse edits, they train specialized low-rank adapter branches and merge them into a single retrieval model, improving benchmark performance without requiring LLMs or additional branches during inference.

Critical strengths
  • The motivation is clear and relevant. The paper addresses the tendency to satisfy target attributes while overlooking reference-image information that should be preserved.

  • The proposed approach is evaluated on a variety of tasks and demonstrates competitive performance against the baselines​

Critical weaknesses
  • The transition objective relies on geometric assumptions that are not sufficiently validated. Although treating caption differences as visual edit directions and aligning reverse instructions with their negatives is intuitive, the paper does not provide direct evidence that these assumptions hold across different edit types.

  • The gains are not sufficiently disentangled from the additional LLM-generated supervision. Although DeCIR avoids LLMs at inference, it relies on synthetic target captions and bidirectional instructions during training.

  • With a shared basis, the proposed merging coefficients is equivalent to linearly merging weight updates. Its advantage over standard linear merging needs clarification.

  • Section 3.3 does not explain how the two disjoint gradient groups are constructed or whether the results are stable across different splits. Without these details, it is difficult to assess whether the same-objective baseline provides a fair and reliable reference.

  • The evaluation lacks comparisons with recent state-of-the-art methods, such as [1-2].

[1]. Self-guided Semantic Inspection for Zero-Shot Composed Image Retrieval. CVPR 2026.

[2]. Modality and Task Adaptation for Enhanced Zero-shot Composed Image Retrieval. AAAI 2026.

Questions for authors

In Eq. 6, the instruction encoder does not receive the reference image or source caption. How can the same instruction embedding align with different source-dependent displacement targets?

2 comments

Response to Reviewer KNN4

We thank the reviewer for the constructive feedback. We address the concerns below.

W1 / Q1: Source-dependent transition geometry

Different sources can yield different displacements for the same edit, yet share a common semantic direction. Our cosine objective encourages the instruction embedding to capture it. We group identical instructions with at least three distinct source–target pairs and report two group-averaged metrics: (1) mean pairwise cosine between displacements (coherence) and (2) cosine between the instruction embedding and mean displacement (alignment), averaged by edit type.

Edit typeGroupsDisplacement coherence (CLIP → DeCIR)Instruction alignment (CLIP → DeCIR)
Color8620.723 → 0.8120.060 → 0.370
Object860.603 → 0.7070.002 → 0.238
Action390.567 → 0.713-0.012 → 0.218
Other7610.632 → 0.7620.021 → 0.283

Source-dependent displacements exhibit substantial directional coherence, and DeCIR learns instruction embeddings that align more closely with their shared directions.

W2: Separating the gains from additional LLM-generated supervision

DeCIR consistently benefits from transition learning across different amounts of LLM-generated supervision. We vary the proportion of synthetic training tuples from 100% to 25%, comparing Endpoint-only and DeCIR using the same data subsets.

DataCIRR Rs@1 (Endpoint / DeCIR)CIRCO mAP@50 (Endpoint / DeCIR)
100%64.46 / 66.0824.92 / 27.26
75%64.51 / 66.0424.20 / 27.06
50%64.41 / 65.7724.01 / 27.01
25%63.86 / 65.3723.86 / 26.74

The advantage persists across all data fractions. Notably, DeCIR trained with only 25% of the tuples still outperforms Endpoint-only trained with 100%. These results demonstrate the effectiveness of our transition learning and decoupled optimization across supervision scales. Further improving data efficiency with substantially fewer synthetic training tuples is an interesting direction for future work.

W3: Equivalence to linear merging

Correct. With a shared basis B, coefficient interpolation is algebraically equivalent to linear weight merging. LRDM's contribution lies in its integrated training and merging design: an endpoint-aligned shared basis, transition-specific coefficient learning, and single-adapter integration. The coefficient-space formulation explicitly reflects this shared parameterization and asymmetric optimization. Our controlled ablation confirms the importance of asymmetric updates:

SettingCIRR Rs@1 ↑CIRCO mAP@50 ↑
Task Arithmetic65.2227.07
Shared B, both objectives update B64.2726.77
Shared B, endpoint-only B update (Ours)66.0827.26

Asymmetric updates are critical to effective shared-basis merging. Restricting B updates to endpoint alignment improves CIRR Rs@1 from 64.27 to 66.08, surpassing independent-basis Task Arithmetic (65.22). We will incorporate these comparisons and clarify the complete definition of LRDM in Section 3.4.

W4: Gradient interference diagnostic

The same-objective baseline separates objective conflict from ordinary stochastic gradient variation. At a fixed step-1400 checkpoint, each 72-sample batch is randomly split into disjoint 36/36 groups for same-objective agreement, while cross-objective gradients use all 72 samples. Gradients are aggregated over 100 batches before computing layer-wise cosine similarities. To test split stability, we repeat the 36/36 random partition ten times on the same fixed checkpoint and 7,200 samples. The standard deviation of macro-averaged GI across these splits is 0.0134, confirming low sensitivity to the random partition. Figure 3 reports mean ± standard deviation over five seeds, revealing concentrated interference in middle and deeper text blocks. We will clarify the partitioning and aggregation protocol in Section 3.3.

W5: Comparison with recent methods

We add DiffComp and LinCIR+MoTa using their published CLIP ViT-L/14 results:

MethodCIRR R@1CIRCO mAP@10FashionIQ R@10 / R@50
DeCIR33.4022.5030.40 / 51.60
DiffComp (CVPR'26)32.3617.3232.62 / 53.85
LinCIR+MoTa (AAAI'26)28.0219.4628.76 / 49.53

DeCIR leads both methods on CIRR and CIRCO. DiffComp performs better on FashionIQ, where its explicit super-patch representations and adaptive local fusion are particularly well suited to the dataset's fine-grained, localized clothing modifications. In contrast, DeCIR focuses on learning semantic transitions within a standard projection-based retrieval architecture, without introducing an additional local composition module at inference. We will include these comparisons in Section 4.2.

Dear Reviewer KNN4,

Thank you for your valuable feedback. In our response, we have addressed your concerns regarding the geometric assumptions underlying semantic transition learning, the impact of additional LLM-generated supervision, the relationship between LRDM and linear merging, the stability of gradient interference analysis, and comparisons with recent methods.

As the discussion period is nearing its end, we would appreciate your review of our responses and welcome any remaining questions.

Best regards, The Authors

Reviewer_VPzaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposed DeCIR, which studies projection-based zero-shot composed image retrieval. It constructs forward and reverse edits and target captions from image-caption pairs, trains endpoint-alignment and transition-alignment branches with a shared low-rank basis, and merges them into one adapter for inference. The transition objective aligns edit embeddings with source-to-target embedding displacements. Experiments on four benchmarks (i.e., CIRR, CIRCO and FashionIQ in main paper, but GenCIS in the appendix) report improved retrieval and include ablations of the objectives, training strategy, and merging.

Critical strengths
  1. The training scheme improves lightweight ZS-CIR while retaining a single retrieval model without online LLM calls.
  2. The PCGrad and loss-weight studies are informative, and integration with SEARLE and LinCIR suggests applicability beyond Pic2Word.
  3. Improvements across several benchmarks, particularly CIRCO, support the practical value of the approach.
Critical weaknesses
  1. The advance over prior work needs sharper justification. For example, DistillCIR [1] already discusses interference between retrieval and reasoning supervision. Even in Sec. 2.1, the authors explain the different objective, but the experiments do not establish what paired displacement learning contributes beyond that formulation.
  2. LRDM's contribution is not isolated from branch training. With a shared basis, coefficient averaging equals averaging the weight updates. Shared-coordinate merging also has precedent in Compress then Merge [2], though its basis is constructed after training. Table 4 compares separately trained bases with a shared basis, leaving the effects of basis sharing and asymmetric updates entangled.
  3. Evidence for source-conditioned transition learning remains indirect. Eq. 6 encodes the instruction alone, with source information entering the displacement target. Table 3 shows gains over endpoint-only training, but the qualitative comparisons use Pic2Word.
  4. Table 8 shows that the full scheme transfers to SEARLE and LinCIR, but lacks endpoint-only controls for these models. In my view, this does not show the benefit of decoupling across initializations. The mixed GeneCIS change-task results also call for a more qualified transition-learning claim.

[1] Zhong W, Barton R, An W, et al. Zero-Shot Composed Image Retrieval via Dual-Stream Instruction-Aware Distillation[C]//2025 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2025: 22221-22231.

[2] He Z, Ding R, Huang Z, et al. Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter[J]. arXiv preprint arXiv:2606.03723, 2026.

Questions for authors
  1. What evidence isolates the benefit of paired displacement learning over DistillCIR's instruction-aware supervision?
  2. Can you isolate the roles of sharing the basis and updating it only through endpoint alignment, while keeping the remaining training settings fixed?
  3. Compared with endpoint-only training, does DeCIR better preserve source constraints when distractors satisfy the requested target attribute?
2 comments

Response to Reviewer VPza

We thank the reviewer for the constructive feedback. We address the concerns below.

W1 / Q1: Contribution beyond instruction-aware supervision

Paired displacement supervision provides greater benefits when trained with decoupled objectives. We compare the reasoning supervision in DistillCIR (B) with our paired displacement supervision (C) under the same endpoint objective (A):

SettingCIRR Rs@1 ↑CIRCO mAP@50 ↑
A+B joint63.8623.62
A+C joint61.0921.93
A+B merged64.8625.51
A+C merged (Ours)66.0827.26

C underperforms B under joint training, yet outperforms it after decoupling, improving CIRR Rs@1 by 1.22 and CIRCO mAP@50 by 1.75 points. We also measure the gradient cosine between endpoint and auxiliary objectives using the joint checkpoints and the same 16 batches (896 samples each, 144 text-LoRA parameter tensors), without parameter updates. Negative-cosine ratio measures the proportion of gradient comparisons with opposing directions.

Auxiliary supervisionMean gradient cosineNegative-cosine ratio
B: DistillCIR reasoning supervision+0.21815.9%
C: Paired displacement−0.26880.2%

The stronger gradient conflict between paired displacement supervision and endpoint alignment explains the degradation under joint training. Decoupled optimization allows both objectives to specialize, while LRDM integrates the learned transition information, enabling our paired displacement supervision to outperform DistillCIR's reasoning supervision after merging.

W2 / Q2: Isolating LRDM and comparison with Compress then Merge

With a shared basis B, coefficient interpolation is algebraically equivalent to linear weight merging. LRDM's contribution lies in its integrated training and merging design: an endpoint-aligned shared basis, transition-specific coefficient learning, and single-adapter integration. The coefficient-space formulation reflects this design. We conduct controlled comparisons to isolate the effects of basis sharing and asymmetric updates, and additionally evaluate Compress then Merge.

SettingCIRR Rs@1 ↑CIRCO mAP@50 ↑
Task Arithmetic65.2227.07
Compress then Merge65.3227.04
Shared B, both objectives update B64.2726.77
Shared B, endpoint-only B update (Ours)66.0827.26

Asymmetric updates are critical to effective shared-basis merging. Under the shared-basis setting, restricting B updates to endpoint alignment improves CIRR Rs@1 from 64.27 to 66.08 and CIRCO mAP@50 from 26.77 to 27.26. LRDM also outperforms Compress then Merge, demonstrating the benefit of constructing and optimizing the shared basis during training. We will incorporate these comparisons and clarify the contributions of each component in Section 3.4.

W3 / Q3: Source-constraint preservation under attribute-matched hard negatives

To test whether DeCIR better preserves source-image constraints when distractors satisfy the requested target attribute, we evaluate the GeneCIS Change Attribute task. Each query contains 15 candidate images, including nine distractors that satisfy the requested attribute but violate the reference image's object-category constraint.

For each Top-K cutoff, we identify queries where DeCIR retrieves the correct target but endpoint-only fails. Among these queries, we count those where DeCIR reverses the ranking of at least one hard negative that was previously ranked above the correct target.

CutoffQueries recovered by DeCIRWith hard-negative reversalProportion
Top-1522038.5%
Top-5593762.7%
Top-10363083.3%

These results provide direct evidence of improved source-category preservation. Among the queries recovered by DeCIR, 62.7% at Top-5 and 83.3% at Top-10 include a corrected ranking against at least one attribute-matching, category-mismatched hard negative. These percentages are measured among recovered queries, not across the entire test set. We will add this label-based analysis and its evaluation protocol to the appendix.

W4: Transferability across initializations

We added endpoint-only controls under the same SEARLE and LinCIR initializations:

InitializationCIRR R@1 (Endpoint → DeCIR)
SEARLE30.30 → 31.13 (+0.83)
LinCIR28.96 → 29.78 (+0.82)

Transition learning improves upon endpoint-only adaptation under both initializations, demonstrating that its benefit is not limited to Pic2Word. On GeneCIS, the benefits vary across conditions. DeCIR improves the overall average R@1 over DistillCIR (16.8 → 17.1), including gains on Change Attribute (16.1 → 16.4), while Change Object remains more challenging. We will clarify the task-dependent nature of these improvements in the revised manuscript.

Dear Reviewer VPza,

Thank you for your valuable feedback. In our response, we have addressed your concerns regarding the contribution of paired displacement supervision beyond prior instruction-aware methods, the roles of basis sharing and asymmetric updates in LRDM, the preservation of source-image constraints under hard negatives, and the effectiveness of decoupling across different initializations.

As the discussion period is nearing its end, we would appreciate your review of our responses and welcome any remaining questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.