- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Preserve-and-Compose Training (PACT) for zero-shot composed image retrieval using image–text–text triplets, without target-image supervision or updating gallery embeddings. PACT trains a pseudo-word mapper and text-encoder LoRA with three objectives: caption-guided composition, source-image alignment of pooled queries across modifications, and negative-caption discrimination. It also introduces Chord scoring, which combines query–candidate similarity with agreement between query and reference-to-candidate change directions.
Overall, the results demonstrate strong performance across CIRR, CIRCO, FashionIQ, and GeneCIS, with PACT achieving particularly competitive results on CIRCO. The method also maintains consistently good performance across different backbone sizes, training seeds, objectives, and scoring functions. Importantly, these results are achieved with low training costs, making the approach both effective and computationally efficient.
Critical strengths
I particularly appreciate work with a clear motivation and a simple method whose design directly addresses that motivation. Figure 2 is especially effective at illustrating how the supervision signals connect to the training objectives and retrieval procedure, making the rationale behind the method easy to understand.
Effective use of available visual supervision without target images. PACT addresses a concrete limitation of caption supervision: descriptions can omit reference-specific visual information. The visual objective substantially improves identity preservation across multiple datasets, including under cosine scoring, while requiring only query-side adaptation. This supports a practically useful contribution beyond the final benchmark scores.
Broad empirical validation with meaningful separation of training and scoring gains. The paper evaluates multiple datasets and backbone sizes, reports variation across five training runs, and compares cosine and Chord using matched checkpoints. PACT improves over composition-only training under both scoring rules, while Chord also benefits existing query models. These results provide substantive evidence that both components contribute to retrieval performance.
Critical weaknesses
-
The interpretation of pooling and embedding differences is insufficiently validated. In Equation (5), averaging modified queries does not necessarily isolate shared visual information, especially when modifications induce correlated changes. Similarly, the differences in Equation (12) are not necessarily restricted to the requested modification semantics. The results demonstrate empirical usefulness, but do not sufficiently establish these interpretations.
-
The interaction between the auxiliary losses remains unexplained. In Table 3, adding the negative-caption loss to composition-only training improves the three-benchmark mean by only 0.28 points, whereas adding it alongside visual supervision yields a 1.53-point improvement. This discrepancy is rather odd and requires further explanation.
-
The scoring comparison may favor Chord. The checkpoint is selected based on Chord performance, then used to evaluate both Chord and cosine. For a fair comparison, each scoring method should also be evaluated using its own best validation-selected checkpoint.
Questions for authors
-
What evidence supports interpreting the average in Equation (5) as shared visual information when modification directions are correlated? Does pooling outperform per-query source-image alignment under comparable training conditions?
-
What evidence shows that the query and image differences in Equation (12) capture comparable modification semantics?
-
How does the cosine–Chord comparison change when each scoring rule selects its own checkpoint under the same validation budget? A cross-evaluation of cosine-selected and Chord-selected checkpoints would help separate scoring gains from checkpoint-selection effects.
2 comments
Thank you for the clarificaiton and additional experiements, which has resolved some of my previous concerns. I maintain by positive rating.
Clarifications on Pooling and Scoring with Additional Experiments
Thank you for your thoughtful review and for recognizing the clear motivation, simple design, and practical value of PACT. We also appreciate your careful questions, which helped us clarify the paper and add further experiments.
Q1.
Eq. (5) uses the average query to learn source preservation across different modifications.
MovieNet provides evidence of this preservation effect: adding the visual loss increased identity preservation from 58.0% to 78.5% with cosine and from 62.0% to 81.5% with Chord. This supports source preservation, without proving that the average contains only shared visual information.
We also compared pooling with per-query source-image alignment using the same data, batches, loss weights, and training steps. Below are pooling − per-query alignment differences, averaged over three seeds trained from scratch.
Chord coefficients were selected separately for each model. Pooling performed better on all three benchmarks under both scoring rules.
We have clarified the role of pooling in Section 3.3 (p. 5) and added the matched comparison in Appendix I.1, Table 17 (p. 31). These additions are highlighted in red.
Q2.
We tested this by replacing the query difference with one from a different modification of the same reference image, while keeping query–candidate similarity, image differences, and the Chord coefficient unchanged.
This reduced CIRR R@1 by 8.204 points across 2,840 eligible queries and FashionIQ R@10 by 7.576 points across 2,888 eligible queries.
These drops support that the query difference carries information about the requested modification that is useful when compared with reference-to-candidate image changes.
We have added this control experiment to Appendix I.3 (p. 31), highlighted in red.
Q3.
The cosine–Chord comparison changed little when each scoring rule selected its own checkpoints. We selected the top three models separately from the same five final checkpoints and cross-evaluated them. Scores below are averaged across the selected models and three benchmark metrics.
Chord-based model selection added only 0.064 points compared with cosine-based model selection.
We have clarified the selection procedure in Appendix C.3 (p. 15) and added the cross-evaluation results in Appendix I.3 (p. 31). These changes are highlighted in red.
W1.
Eq. (5) uses the average query to supervise source preservation across different modifications. We were not claiming that correlated modification directions cancel. On MovieNet, adding visual supervision increased identity preservation from 58.0% to 78.5% with cosine and from 62.0% to 81.5% with Chord. Pooling also outperformed per-query source-image alignment on all three benchmarks under both scoring rules in our matched comparison.
For Eq. (12), we replaced the query difference with one from a different modification of the same reference, keeping the other scoring terms unchanged. This reduced CIRR R@1 by 8.204 points and FashionIQ R@10 by 7.576 points. These drops support that the directional comparison uses information specific to the requested modification.
These results support source preservation and modification-dependent retrieval. They do not establish that averaging or subtraction cleanly separates visual content from modification semantics.
W2.
Our interpretation is that the two losses balance source preservation and modification fulfillment. The visual loss strengthens source preservation. When this favors retaining the original state, the negative loss provides a corrective signal by rejecting captions that preserve source details but fail to satisfy the modification.
This complementary role offers an explanation for the larger gain from negative supervision when the visual loss is present.
W3.
We address this concern in our response to Q3.