acceptodds
Under review as a conference paper at ICLR 2027

Sharpen: Bringing the Requested Change into Focus for Composed Image Retrieval

Abstract

Composed image retrieval searches for an image that preserves relevant content from a reference image while showing a change specified by a modification text. In the training-free describe-then-retrieve paradigm, a multimodal large language model (MLLM) writes a target description, and Contrastive Language-Image Pretraining (CLIP) ranks database images by similarity to it. The description can favor unchanged context, placing reference-like images that miss the change above the target. A description-based similarity score also cannot compare a candidate directly with the reference to check what it preserves. To bring the requested change into focus, we propose Sharpen, which adjusts scores across the database and within the selected candidate set. Across the database, the global adjustment strengthens the requested change in the query embedding and suppresses popular images that resemble many composed queries before selecting candidates. Within the candidate set, the local adjustment reuses this MLLM to judge the change without the reference, and the complete request with it. Both the change judgment and the reference-conditioned judgment contribute to the final order. With a 4B open-weight MLLM and no training, Sharpen achieves state-of-the-art training-free results on three widely used benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
information retrieval, compoased image retrieval, multimodal large language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_i6KJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates training-free zero-shot compositional image retrieval, a task that involves retrieving a target image based on a reference image and a modification text—ensuring the target retains the reference image's content while satisfying the specified modifications. The authors propose "Sharpen," a method that globally adjusts relevance scores across the database by modifying scaling and popularity suppression factors. It then employs a Multimodal Large Language Model (MLLM) to evaluate candidate images based on modification consistency and overall alignment with the reference image, subsequently re-ranking the candidates using these scores. This approach eliminates the need to train new retrieval models, instead leveraging the MLLM to assess candidates against the intended target. Experiments conducted on the CIRCO, CIRR, and FashionIQ datasets demonstrate significant performance gains over baselines utilizing the same MLLM and CLIP models; the paper also presents component ablation studies, model analysis, and an evaluation of inference costs.

Critical strengths
  1. The problem is clearly defined, and the method design aligns strongly with the problem. To address a specific failure mode in composed image retrieval—where results retain the content of the reference image without actually executing the requested modification—Sharpen introduces three steps: modified text augmentation, popular image suppression, and candidate image verification. The overall workflow is simple and modular, and it does not rely on additional training.
  2. Compared to the controlled baseline, the method demonstrates significant performance gains and includes component-level validation. Using identical MLLM and CLIP configurations, Sharpen substantially outperforms the baseline across key metrics on the CIRCO, CIRR, and FashionIQ datasets. Furthermore, through experiments involving component ablation, varying MLLMs, candidate set sizes, and weight sensitivity, the authors demonstrate that both global adjustment and local candidate evaluation contribute to the final results.
Critical weaknesses
  1. The transductive query bank used for evaluation in the paper consists of the target-description embeddings of all evaluated composed queries, excluding only the query currently being scored. Consequently, the model leverages information from other test queries during evaluation. This differs from standard query-independent scenarios or settings involving new, online queries. It is recommended that the authors report results obtained without the query bank, or construct the query bank using an independent set of training or validation queries, to assess the impact of this setup on performance.
  2. Table 5 does not explicitly state whether the reported inference time accounts for processes such as query bank construction and similarity computation. Furthermore, details regarding the validation split used for hyperparameter tuning are missing. The authors are encouraged to provide these missing details concerning efficiency experiments and tuning protocols, enabling readers to evaluate the actual deployment costs and the extent to which the results depend on hyperparameter tuning.
Questions for authors
  1. Please report the results obtained when constructing the query bank using only independent training/validation queries (excluding evaluation queries) to assess the impact of the transductive setting on performance.
  2. Does the figure of 3.19 seconds in Table 5 account for costs such as query bank construction, preprocessing, similarity computation, caching, and storage? If not, please report the offline construction costs and online query costs separately. Additionally, please provide details regarding the size of the validation split, the parameter search range, and the method used to isolate the test set.
4 comments

Thank you for your positive assessment of the method and its gains over the controlled baseline. We address both questions below. The manuscript revisions are marked in blue.

To address your concern about using evaluated queries to compute popularity scores, we replaced the transductive bank with a fixed disjoint bank of 10,000 human-written COCO captions, keeping all other settings unchanged. The bank contains no evaluated queries or target labels. Popularity scores can be computed once before queries arrive, and each query is processed independently. For comparison, Table 3 reports Sharpen without a query bank, with popularity suppression disabled.

Query bankCIRCO mAP@5CIRR R@1FashionIQ R@10
None (no popularity suppression)35.8853.9143.53
Transductive query bank38.4153.9645.34
Fixed disjoint query bank (COCO captions)37.4454.1543.95

With the COCO-caption bank, popularity suppression improves CIRCO / CIRR / FashionIQ by 1.56 / 0.24 / 0.42 points over no suppression. These gains are obtained without access to the evaluated queries. Using independent banks of training-query descriptions gives 53.41 CIRR R@1 and 45.37 FashionIQ average R@10. We have added the independent-bank comparison and construction details in Appendix D.6 (Table 12), with a reference in Sec. 4.1.

The 3.19 s in Table 5 is the total online time per composed query, averaged over CIRCO queries processed in batches on one RTX 5080, excluding model loading. It includes target-description generation, CLIP text encoding, database scoring and candidate selection, 100 MLLM scoring passes at N=50, and score fusion. Candidate-only assessment takes 1.33 s and reference-conditioned assessment takes 1.81 s. The remaining stages take about 0.06 s.

Database image embeddings are computed offline and cached. Encoding takes 25.6 ms per image. Computing popularity scores takes about one second or less on CPU and is repeated when the bank changes. CIRCO's cached embeddings require 482 MB in fp32, and its popularity scores require 0.47 MB.

Constructing a bank of generated descriptions requires description generation and CLIP text encoding. Description generation takes 0.045 s per bank query under batched processing. A bank of existing COCO captions requires only text encoding.

We selected the hyperparameters on a small sample of 5% of the validation split, searching γ ∈ [0.1, 1.0], γ_h ∈ [0.25, 1.0], k ∈ [1, 16], w_s ∈ [0.2, 0.5], and w_a, w_r ∈ [0, 0.7]. We have clarified the validation sampling in Sec. 4.1 and the timing boundary in Sec. 4.5, and added the stage breakdown, offline costs, and search ranges in Appendix D.12 (Table 17).

Thank you again for your review. We have responded to each of your comments and revised the paper accordingly (changes marked in blue). As the discussion period ends on November 18, we would appreciate knowing whether our response has addressed your concerns, and we would be glad to clarify anything that remains.

Reviewer_SKg4OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This is a composed image retrieval paper that improves the describe-then-retrieve paradigm (CIReVL style composed image retrieval). In this line of work, a vllm describes the image and the modification text with only text (text-composed query) then the task is a simple text-to-image retrieval with the image database. This paper proposes a method that 1) adjusts the similarity of the text-composed query with the database by also using the similarity of the modification text with the database, and also penalizes database images that are close to the textual manifold. 2) It also proposes a reranking process, where for a small number of highly ranked database images are used as input to the vllm (in addition to the modification text and the original image) for re-evaluation. The proposed method looks to be state-of-the-art in CIRCO and competitive in CIRR and FashionIQ.

Critical strengths
  1. The proposed method has very intuitive components. a) punishing the database images that have high similarity with the query because they are close to the textual manifold and b) reranking by adding the database images as inputs to the vllm
  2. The paper provides a convincing ablation study and a parameter sensitivity study
  3. Although there are datasets that the method is not clearly the state-of-the-art, I think the performance seems promising for an ICLR paper, mainly because the dataset that the proposed method is sota is CIRCO, the least flawed dataset of the three.
Critical weaknesses
  1. poor writing: I had a difficult time reading the abstract. I will give you my initial thoughts while I was reading it for feedback. From lines 11-12 we have a) "an image" (to search from a database), "a reference image" (that gives context and to modify), and "a modification text". In line 17 there is the term "candidate", which I thought it was a database image. Then in line 20 there is the term "candidate set", used additionally with the term "database". Now I am expecting something like a reranker, but without reading the paper I am still not sure because it was not stated explicitly.

  2. CIRR and FashionIQ are famously noisy. I understand that they became the standard in the CIR community but in my opinion the community should start moving away from them. See questions for authors for what I propose instead.

  3. Typos or questions about the text: In 3.1 j is an index (of the database) but it is also used as an image both in the score function and in the distribution over the database. lines 212-215, I am not sure why q̄ is mentioned here, did you mean b̄?

Questions for authors

If the following questions/requests are addressed in a satisfactory manner, I am very positive (and looking forward) to proposing an acceptance grade for this paper

  1. What was the description text produced for the example in Figure 1? It seems like the cats in a suitcase were retrieved because the text includes the word cat in it. When you applied the "target-description prompt" from the appendix, did Qwen3 answer with the word "cat" in the description text? if yes, why don't you prompt specifically to exclude the negatives? Is your method useful also in cases where negatives are not in the text? Same questions for the case study of the appendix.

  2. About suppressing popular images. When an image is close to the average of its kNN composed neighbors you punish it for "being popular". But your composed bank is just captions, so practically you are measuring distance from the textual manifold. It makes sense to punish images that are close to your composed textual query because they are closer to all texts. Have you tried with the original captions instead of the combined texts? Have you tried with just a dictionary of words? How much worse is it if you punish with the distance from the mean of the textual manifold (single fixed vector instead of the average of a kNN)? This would be dramatically faster given that you won't need to calculate the kNN. Also is the description bank a different set than your test set? how are you creating this bank?

  3. The method seems to be quite slow as the candidate size increases. It makes sense for perspective to see a trade-off scatter plot (x axis time, y axis performace eg CIRCO mAP@5) with the values of candidate size N as in Table 5 but also the competitors of Table 1/2

  4. Could you repeat your experiments to the refined versions of CIRR and FashionIQ as in the CoLLM paper (Huynh et al., CVPR 2025)? Just to reproduce of the table 3 with the refined version of the two datasets should be enough.

  5. The describe-then-retrieve approach sounds counter-intuitive for instance-level cases. I propose some experiments in an instance level composed image retrieval dataset like iCIR (Psomas et al.,NeurIPS 2025), or even on stronger personalization datasets like concon-chi (Rosasco et al., CVPR 2024). The latter would need some repurposing from the task of personalization to the task of CIR, but it can answer whether your method (or any other describe-then-retrieve method) can tackle CIR on objects that have no common name.

10 comments

Thank you for your detailed feedback and suggestions. We address each point below. The manuscript revisions are marked in blue.

We have revised the abstract to distinguish database images from the top-N candidates and state the reranking step explicitly. Sharpen first adjusts the scores of all database images, selects the top-N images as candidates, and then reranks them. We now use "database image" when referring to an image before candidate selection. The revisions are marked in blue.

Following your suggestion, we repeated the Table 3 component ablation on the refined CIRR and FashionIQ benchmarks released by CoLLM (Huynh et al., CVPR 2025), keeping all settings and fusion weights unchanged.

ConfigurationRefined CIRR R@1Refined FashionIQ R@10
Baseline39.8448.58
Global adjustment only48.2958.03
Local adjustment only63.4260.16
Sharpen65.8665.34

Sharpen performs best on both refined benchmarks, and removing any single component lowers both primary metrics, consistent with Table 3. We have added the complete component ablation and refined-benchmark evaluation details in Appendix D.9 (Table 15).

Reviewer_ik9COpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses training-free zero-shot composed image retrieval (ZS-CIR) in the describe-then-retrieve paradigm, where an MLLM writes a target description and CLIP ranks database images against it. The authors correctly identify the core failure mode: similarity to the description rewards retained context, so reference-like images that miss the requested change outrank the true target.With Qwen3-VL-4B and OpenCLIP ViT-g/14, SHARPEN reports state-of-the-art training-free results on CIRCO, CIRR, and FashionIQ, beating systems built on GPT-4o and a 72B MLLM.

Critical strengths
  1. The paper addresses a well-motivated problem with a clear intuition. Figure 1 effectively illustrates the failure mode, showing the target ranked 85th behind five reference-like cat images, while the emphasis on focusing on the requested change provides a coherent rationale for the method.

  2. The reported results are practically compelling: a single 4B open-weight MLLM achieves state-of-the-art training-free performance, outperforming the evaluated systems based on GPT-4o or 72B models.

  3. The empirical evaluation covers component contributions, target inclusion, parameter sensitivity, inference cost, scoring formulations, question wording, and sensitivity across five MLLMs (Tab. 3–8; Fig. 3). The scoring equations, prompts, algorithm, hardware details, and anonymous code link also provide substantial support for reproducibility.

Critical weaknesses
  1. The weak internal baseline and component results make the contribution difficult to isolate. On CIRCO, local MLLM assessment alone raises performance from 16.40 to 36.91, compared with 38.41 for the full method. Given the close precedents for reranking and global adjustment, a stronger same-model baseline and controlled comparisons would better establish the value of the proposed components.

  2. Popularity suppression uses embeddings from the evaluated queries, introducing transductive access not shared by the inductive baselines in Tables 1–2. Although no target labels are used, an inductive variant would improve comparability. Table 4 also shows no CIRCO target-inclusion benefit (84.09 to 84.09).

  3. FashionIQ average R@50 reaches only 59.38, below several compared methods. The identical top-50 target-inclusion rate indicates that approximately 40% of queries cannot be recovered through reranking. The paper should discuss this coverage limitation and the concentration of gains at R@10.

Questions for authors
  1. How do the gains change with a stronger description strategy from the same Qwen3-VL-4B? Please keep the inference budget comparable or report the additional cost.

  2. How sensitive are the popularity-suppression gains to using a disjoint query bank instead of the evaluated queries?

  3. What explains the lower FashionIQ candidate coverage, and how does the N=50 cap contribute to the reported R@10/R@50 trade-off?

  4. Have you checked sensitivity to image presentation order while preserving the reference/candidate role labels?

  5. Please clarify the validation and hyperparameter-selection protocol and provide a paired confidence interval for the SHARPEN–WISER difference.

11 comments

Thank you for your careful review and for recognizing the motivation, practical results, and breadth of our evaluation. We address your questions below. The manuscript revisions are marked in blue.

Sharpen's gains persist with stronger target descriptions. We replaced our description prompt with OSrCIR's reflective prompt, keeping the same Qwen3-VL-4B, CLIP encoder, and hyperparameters. The results are:

Description promptConfigurationCIRCO mAP@5CIRR R@1FashionIQ R@10
OursBaseline16.4028.4930.83
OursSharpen38.4153.9645.34
OSrCIRBaseline24.8336.3334.65
OSrCIR+ global adjustment30.0941.0441.49
OSrCIRSharpen39.7854.8047.62

With the OSrCIR prompt, Sharpen improves the baseline by 14.95 / 18.47 / 12.97 points on CIRCO / CIRR / FashionIQ. Global adjustment alone improves this stronger baseline by 5.26 / 4.71 / 6.84 points before candidate assessment.

Under our original prompt, Table 4 shows that global adjustment raises top-50 target inclusion by 6.36 / 5.26 / 8.42 points. Table 3 shows that adding it to candidate assessment improves the final metrics by 1.50 / 2.49 / 4.66 points.

Under the batched processing used in Table 5 on one RTX 5080, the OSrCIR prompt increases description time from 0.044 s to 0.297 s per query. Substituting this description cost gives a total Sharpen time of 3.44 s versus 3.19 s per query. We keep our prompt as the default. We have added the stronger-prompt comparison in Sec. 4.3 (Table 6) and its processing cost in Sec. 4.5, with detailed results and costs in Appendix D.5 (Tables 10 and 11).

We tested popularity suppression with a fixed disjoint bank of 10,000 human-written COCO captions, keeping all other settings unchanged. The bank contains no evaluated queries or target labels. Popularity scores can therefore be computed once before queries arrive, and each query is processed independently.

Query bankCIRCO mAP@5CIRR R@1FashionIQ R@10
None (no popularity suppression)35.8853.9143.53
Transductive query bank38.4153.9645.34
Fixed disjoint query bank (COCO captions)37.4454.1543.95

With the fixed COCO-caption bank, popularity suppression improves CIRCO / CIRR / FashionIQ by 1.56 / 0.24 / 0.42 points over no suppression. These gains are obtained without access to the evaluated queries. We have added the independent-bank comparison and construction details in Appendix D.6 (Table 12), with a reference in Sec. 4.1.

Popularity suppression alone leaves CIRCO target inclusion at 84.09% (Table 4), while improving mAP@5 from 16.40 to 18.76 (Table 3). This shows better ranking despite unchanged target inclusion. Combined with change amplification, it raises target inclusion from 88.64% to 90.45%. Removing it from full Sharpen lowers mAP@5 from 38.41 to 35.88.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.