acceptodds
Under review as a conference paper at ICLR 2027

Preserve-and-Compose Training for Composed Image Retrieval

Abstract

Composed image retrieval (CIR) aims to retrieve images that satisfy a user-specified modification while preserving relevant visual content from a reference image. Collecting target images for this purpose is costly, motivating zero-shot CIR methods that instead use target captions as supervision. However, target captions may omit source details that should be preserved. We therefore propose PACT, Preserve-and-Compose Training, which complements target-caption supervision with visual evidence from the source image. PACT learns from image–text–text (ITT) triplets without target images or gallery updates, aligning composed queries with target captions while preserving source evidence through visual supervision. We further introduce Chord scoring, which combines target similarity with source-relative directional agreement in the frozen image space. Results across four ZS-CIR benchmarks show that combining target-caption supervision with source-image evidence leads to strong retrieval performance across datasets, backbone scales, and external galleries.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Composed Image Retrieval

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_ePw9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Preserve-and-Compose Training (PACT) for zero-shot composed image retrieval using image–text–text triplets, without target-image supervision or updating gallery embeddings. PACT trains a pseudo-word mapper and text-encoder LoRA with three objectives: caption-guided composition, source-image alignment of pooled queries across modifications, and negative-caption discrimination. It also introduces Chord scoring, which combines query–candidate similarity with agreement between query and reference-to-candidate change directions.

Overall, the results demonstrate strong performance across CIRR, CIRCO, FashionIQ, and GeneCIS, with PACT achieving particularly competitive results on CIRCO. The method also maintains consistently good performance across different backbone sizes, training seeds, objectives, and scoring functions. Importantly, these results are achieved with low training costs, making the approach both effective and computationally efficient.

Critical strengths

I particularly appreciate work with a clear motivation and a simple method whose design directly addresses that motivation. Figure 2 is especially effective at illustrating how the supervision signals connect to the training objectives and retrieval procedure, making the rationale behind the method easy to understand.

Effective use of available visual supervision without target images. PACT addresses a concrete limitation of caption supervision: descriptions can omit reference-specific visual information. The visual objective substantially improves identity preservation across multiple datasets, including under cosine scoring, while requiring only query-side adaptation. This supports a practically useful contribution beyond the final benchmark scores.

Broad empirical validation with meaningful separation of training and scoring gains. The paper evaluates multiple datasets and backbone sizes, reports variation across five training runs, and compares cosine and Chord using matched checkpoints. PACT improves over composition-only training under both scoring rules, while Chord also benefits existing query models. These results provide substantive evidence that both components contribute to retrieval performance.

Critical weaknesses
  1. The interpretation of pooling and embedding differences is insufficiently validated. In Equation (5), averaging modified queries does not necessarily isolate shared visual information, especially when modifications induce correlated changes. Similarly, the differences in Equation (12) are not necessarily restricted to the requested modification semantics. The results demonstrate empirical usefulness, but do not sufficiently establish these interpretations.

  2. The interaction between the auxiliary losses remains unexplained. In Table 3, adding the negative-caption loss to composition-only training improves the three-benchmark mean by only 0.28 points, whereas adding it alongside visual supervision yields a 1.53-point improvement. This discrepancy is rather odd and requires further explanation.

  3. The scoring comparison may favor Chord. The checkpoint is selected based on Chord performance, then used to evaluate both Chord and cosine. For a fair comparison, each scoring method should also be evaluated using its own best validation-selected checkpoint.

Questions for authors
  1. What evidence supports interpreting the average in Equation (5) as shared visual information when modification directions are correlated? Does pooling outperform per-query source-image alignment under comparable training conditions?

  2. What evidence shows that the query and image differences in Equation (12) capture comparable modification semantics?

  3. How does the cosine–Chord comparison change when each scoring rule selects its own checkpoint under the same validation budget? A cross-evaluation of cosine-selected and Chord-selected checkpoints would help separate scoring gains from checkpoint-selection effects.

2 comments
AuthorsOpenReview ↗

Clarifications on Pooling and Scoring with Additional Experiments

Thank you for your thoughtful review and for recognizing the clear motivation, simple design, and practical value of PACT. We also appreciate your careful questions, which helped us clarify the paper and add further experiments.

Q1.

Eq. (5) uses the average query to learn source preservation across different modifications.

MovieNet provides evidence of this preservation effect: adding the visual loss increased identity preservation from 58.0% to 78.5% with cosine and from 62.0% to 81.5% with Chord. This supports source preservation, without proving that the average contains only shared visual information.

We also compared pooling with per-query source-image alignment using the same data, batches, loss weights, and training steps. Below are pooling − per-query alignment differences, averaged over three seeds trained from scratch.

ScoringCIRR R@1CIRCO mAP@5FashionIQ R@10
Cosine+1.515+1.385+1.017
Chord+0.319+0.219+0.344

Chord coefficients were selected separately for each model. Pooling performed better on all three benchmarks under both scoring rules.

We have clarified the role of pooling in Section 3.3 (p. 5) and added the matched comparison in Appendix I.1, Table 17 (p. 31). These additions are highlighted in red.

Q2.

We tested this by replacing the query difference with one from a different modification of the same reference image, while keeping query–candidate similarity, image differences, and the Chord coefficient unchanged.

This reduced CIRR R@1 by 8.204 points across 2,840 eligible queries and FashionIQ R@10 by 7.576 points across 2,888 eligible queries.

These drops support that the query difference carries information about the requested modification that is useful when compared with reference-to-candidate image changes.

We have added this control experiment to Appendix I.3 (p. 31), highlighted in red.

Q3.

The cosine–Chord comparison changed little when each scoring rule selected its own checkpoints. We selected the top three models separately from the same five final checkpoints and cross-evaluated them. Scores below are averaged across the selected models and three benchmark metrics.

Selection ruleCosineChord
Cosine23.80329.423
Chord23.74429.487

Chord-based model selection added only 0.064 points compared with cosine-based model selection.

We have clarified the selection procedure in Appendix C.3 (p. 15) and added the cross-evaluation results in Appendix I.3 (p. 31). These changes are highlighted in red.

W1.

Eq. (5) uses the average query to supervise source preservation across different modifications. We were not claiming that correlated modification directions cancel. On MovieNet, adding visual supervision increased identity preservation from 58.0% to 78.5% with cosine and from 62.0% to 81.5% with Chord. Pooling also outperformed per-query source-image alignment on all three benchmarks under both scoring rules in our matched comparison.

For Eq. (12), we replaced the query difference with one from a different modification of the same reference, keeping the other scoring terms unchanged. This reduced CIRR R@1 by 8.204 points and FashionIQ R@10 by 7.576 points. These drops support that the directional comparison uses information specific to the requested modification.

These results support source preservation and modification-dependent retrieval. They do not establish that averaging or subtraction cleanly separates visual content from modification semantics.

W2.

Our interpretation is that the two losses balance source preservation and modification fulfillment. The visual loss strengthens source preservation. When this favors retaining the original state, the negative loss provides a corrective signal by rejecting captions that preserve source details but fail to satisfy the modification.

This complementary role offers an explanation for the larger gain from negative supervision when the visual loss is present.

W3.

We address this concern in our response to Q3.

Thank you for the clarificaiton and additional experiements, which has resolved some of my previous concerns. I maintain by positive rating.

Reviewer_eyTGOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the composed image retrieval problem using an image-text-text triplet training approach named PACT. The image is inverted into a pseudo-word token pic2word style. This inversion is used as input alone (and have target the original image caption) or with a modification text (and have target a modified target text) to a CLIP text encoder adapted with LoRA. The two target text embeddings are taken from a frozen CLIP text encoder. They use a symmetric infoNCE loss between the query and the text embeddings. They also use a visual preservation loss that demands the average embedding of the same image with many modification texts to be contrastively mapped close to the image itself. Finally, they use negative captions that preserve the image information but fail to express the modification text as hard negative targets. PACT is competitive with the current state-of-the-art on a number of datasets.

Critical strengths
  1. The paper is well written and already polished. I got the information I needed from the first read and everything was explained in an optimal way
  2. The proposed method is quite intuitive and every module makes sense. The authors did a great job motivating every part of the compose, visual, and negative part of the loss.
  3. The method is demonstrated in enough datasets and in an ablation study
Critical weaknesses

The paper is not reporting the numbers of DiffComp in Table 1, although a) the authors are aware of the method, reporting it in Table 2 where they have better numbers, and b) according to ICLR Q&A, CVPR 2026 is not considered co-current work (CVPR 2026 was before the cut-off July 17). PACT loses to DiffComp on FashionIQ and CIRR.

Questions for authors
  1. I am asking for the addition of DiffComp to the Table 1 and add in the discussion as well as update the state-of-the-art claim from the paper. Other than that, I have no objection against this paper.
  2. mostly out of curiosity, did you try to add the hard negative texts in equation (4) instead of adding a whole new loss parameter? Is there a reason on why equation 8 works better (if it does)?
2 comments
AuthorsOpenReview ↗

Comparison with DiffComp and Hard-Negative Supervision

Thank you for your thoughtful review and kind comments on the paper’s clarity, the motivation behind each loss, and the experimental evaluation. We also appreciate your suggestions on the DiffComp comparison and hard-negative supervision, which helped us improve the revised paper.

Q1.

Thank you for pointing out this omission. We have added DiffComp to Table 1 and the comparison text (p. 6).

With the same CLIP ViT-L/14 backbone, PACT scores 7.40 pp higher on CIRCO test mAP@5, but 1.12 pp lower on CIRR test R@1 and 0.19 pp lower on FashionIQ val R@10. We have also revised the broad state-of-the-art claim to describe our results as competitive (p. 2).

These changes are highlighted in red.

Q2.

We tested this alternative by removing the separate negative loss and adding hard negatives directly to the query-to-text denominator in Eq. (4).

The table reports direct denominator insertion − original PACT, in percentage points. Both methods were trained from scratch with three seeds; results are averaged across seeds.

ScoringCIRR R@1 ΔCIRCO mAP@5 ΔFashionIQ R@10 Δ
Cosine−3.038−2.031−2.705
Chord−1.260+1.094−0.397

We selected the Chord coefficient separately for each model on validation. Direct insertion performed worse on all three benchmarks with cosine. With Chord, it performed worse on CIRR and FashionIQ but better on CIRCO.

We used a separate loss to control hard-negative discrimination independently of caption alignment. In the denominator, a hard negative competes alongside all other in-batch captions, which may weaken its individual training signal. Eq. (8) directly supervises discrimination between the correct caption and its hard negative. We see the choice of how to incorporate hard negatives as an interesting direction for future work.

We have added this comparison to Appendix I.2, Table 18 (p. 31), highlighted in red.

W1.

We address this concern in our response to Q1.

I thank the authors for resolving my issues. I raised my rating to clear acceptance

Reviewer_LQyKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose PACT for composed image retrieval without target-image supervision. The framework combines target-caption alignment, source-image preservation across multiple modifications, and negative-caption discrimination. It adapts a pseudo-word mapper and text-encoder LoRA while freezing image representations, and introduces Chord scoring to combine query similarity with reference-relative directional agreement.

Critical strengths
  • The paper addresses a relevant limitation of target-caption supervision, which may capture the requested modification while overlooking source-specific visual details that should be preserved.
  • The method is intuitive. It combines target-caption alignment with source-image preservation and negative-caption discrimination, providing complementary supervision for retaining relevant visual content while following the requested modification.
  • The proposed approach is evaluated on a variety of tasks and demonstrates competitive performance against the baselines.
Critical weaknesses
  • The effect of uneven preservation-group sizes is unclear. Different sources contribute different numbers of queries, and repeated sampling may produce groups without distinct modifications.

  • I am concerned that averaging modified queries does not guarantee that modification-specific information cancels out. The paper provides insufficient evidence that this operation extracts shared visual information rather than simply biasing the queries toward the source image.

  • The paper lacks ablation studies on the weights α\alpha and β\beta in Eq.9.

  • In Table 1, PACT achieves suboptimal R@10 performance on CIRR. The authors are encouraged to analyze the possible reasons for this limitation.

Questions for authors

How is Ki\mathcal{K}_i determined in Section 3.3, and why do different source images contribute different numbers of modified queries?

2 comments
AuthorsOpenReview ↗

Clarifications and Additional Experiments

Thank you for recognizing the motivation behind source preservation, the intuitive design of PACT, and its competitive performance across tasks. We address your concerns below.

Q1.

KiK_i is the number of queries for source image ii in the current batch. Each source has up to three retained modification records after filtering. We group its in-batch queries, rather than loading all its modifications together.

In each preservation batch of 512 queries, 128 queries are deliberately paired with queries sharing the same source. Additional source occurrences can arise from sampling with replacement. Thus, group sizes reflect record availability and sampling; we do not assign an optimized size to each source.

Only groups with Ki≥2K_i \geq 2 contribute preservation anchors:

zi=normalize⁡(1Ki∑j=1Kiqij).z_i = \operatorname{normalize}\left(\frac{1}{K_i}\sum_{j=1}^{K_i}q_{ij}\right).

The preservation loss is averaged across eligible sources, so larger groups do not receive proportionally more loss terms.

The sampling procedure and source-level loss averaging are described in Section 3.3 (p. 5) and Appendix B.2–B.3 (p. 14).

W1.

Across the three full-training streams, 99.89% of eligible groups had size two. Groups containing only duplicate token inputs occurred 129 times among 384,228 eligible groups (0.03357%).

We also retrained after removing duplicate token inputs and requiring at least two distinct inputs per group. Relative to the original sampling procedure, performance changed by +0.136 / +0.051 / −0.011 pp on CIRR / CIRCO / FashionIQ.

We have added the group-size statistics and duplicate-input control to Appendix I.1 (p. 31), highlighted in red.

W2.

Eq. (5) uses the average query to learn source preservation across different modifications. It does not assume that modification-specific information cancels out.

On MovieNet, adding visual supervision increased identity preservation by +20.5 points with cosine and +19.5 points with Chord (Table 4).

To examine the role of pooling, we compared it with aligning each modified query separately to its source image, using the same data, batches, loss weights, and training steps. Below are pooling − per-query alignment differences, averaged over three seeds trained from scratch.

ScoringCIRR R@1CIRCO mAP@5FashionIQ R@10
Cosine+1.515+1.385+1.017
Chord+0.319+0.219+0.344

Chord coefficients were selected separately for each model. Pooling performed better on all three benchmarks under both scoring rules. These results support its value for source preservation and retrieval, without establishing that the average isolates shared visual information.

We have clarified the role of pooling in Section 3.3 (p. 5) and added the matched pooling comparison in Appendix I.1, Table 17 (p. 31). Both additions are highlighted in red.

W3.

Beyond the existing loss-removal ablations, we varied each weight between 0.5 and 2 while holding the other at 1. The table shows changes relative to α=β=1\alpha=\beta=1, in percentage points.

Preservation α\alphaNegative β\betaCIRR R@1 ΔCIRCO mAP@5 ΔFashionIQ R@10 Δ
0.51−0.064+0.087−0.194
110.0000.0000.000
21−0.088−0.168−0.045
10.5−0.271+0.306−0.063
12−0.279−0.774−0.147

Changing α\alpha produced differences below 0.2 pp on all three benchmarks. Reducing β\beta improved CIRCO but slightly reduced CIRR and FashionIQ; doubling it reduced all three. The default provides a balanced setting, rather than a universal optimum.

All new results average three fresh full-training seeds at fixed final-step checkpoints. For each model, the Chord coefficient was selected to maximize the mean validation score across the three benchmarks. These are same-validation-set results and are reported separately from the submitted benchmark results.

We have added the loss-weight comparison to Appendix I.2, Table 18 (p. 31), highlighted in red.

W4.

With ViT-L/14, PACT achieves higher CIRR R@1 and R@5 than De-MINDS, but lower R@10 (71.66 vs. 74.50). Our interpretation is that differences in handling relational and compound constraints affect the ranking of visually similar candidates, contributing to this gap.

The ordering also depends on the backbone. With ViT-G/14 (Table 11, p. 18), PACT outperforms De-MINDS on all three CIRR metrics, including R@10 (82.68 vs. 79.80). This suggests that the R@10 gap is specific to the ViT-L/14 comparison rather than a consistent weakness across backbones.

Thanks for the authors' response. My concerns have been addressed, and I have decided to maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.