- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PVR-RFT, a two-stage framework for composed image retrieval. A frozen coarse retriever first retrieves the top-k candidates, and a vision-language model scores candidate independently using the log-odds of Yes/No token probabilities. The core idea is to combine VLM judgments with the initial retrieval ranking through stage0-aware reinforcement fine-tuning and rank-log fusion at inference. The scorer is trained through a GRPO-style policy-gradient objective over rankings sampled from a Plackett–Luce distribution, sampled using Gumbel perturbations and rewarded by the target’s reciprocal rank. The authors instantiate the framework with DQU-CIR and MCoT-MVS, separately training adapters for each and evaluate the framework on FashionIQ, CIRR and Shoes. The reported results demonstrate that PVR-RFT achieves strong performance, outperforming many academic methods and the corresponding stage0 baselines.
Critical strengths
-
Modular architecture with efficient pointwise scoring. The modular framework scores candidates independently with only O(K) forward passes, avoiding exhaustive pairwise comparisons.
-
Promising empirical performance. The method achieves consistent improvements over the corresponding stage0 baselines across multiple composed image retrieval benchmarks.
-
Complementary retrieval signals. The ablation results highlight the complementarity between VLM scores and stage0 rankings, showing that their fusion can outperform either component alone.
-
Clear architectural illustration. Figure 1 provides an intuitive overview of the framework, clearly illustrating both the training and inference pipelines.
Critical weaknesses
-
The comparison between pointwise and pairwise reranking-forms changes both the input resolution and the training objective. Consequently, the performance difference cannot be attributed solely to the reranking formulation.
-
The current ablations do not sufficiently show the contributions of reinforcement fine-tuning and the training-time stage0 prior. Comparisons with an unfine-tuned VLM and a supervised fine-tuning baseline under the same evaluation protocol would help establish the benefit of the proposed RL objective. Separately, setting the training-time prior weight to zero while retaining the same inference-time fusion protocol would clarify the contribution of incorporating stage0 ranks during training.
-
Both VLM-based answer-token probability scoring and retrieve–rerank pipelines have precedents in multimodal retrieval, limiting the novelty of these components.
-
Section 4.1 mentions paired-bootstrap evaluation, but the sentence ends abruptly at “and report 95” (line 312).
-
Although the target coverage of 93.31% is reported on the Dress setting, stage0 Recall R@256 should be provided for each evaluated retriever–dataset configuration. These values would clarify the recall ceiling imposed by the candidate pools and help contextualize the reranking gains.
Questions for authors
-
During evaluation, how are queries handled when the ground-truth target is absent from the stage0 top-256 candidate pool? Are these queries retained in the evaluation denominator and counted as retrieval failures?
-
Providing some examples where reranking promotes the correct target from a lower stage0 position into the top results would help illustrate the improvements achieved by proposed method.
-
Section 2.2 describes FiRE as directly generating ranked lists through text generation. However, FiRE’s Section 4.3 describes separately encoding queries and candidate images and retrieving images based on their similarity. Could authors clarify this discrepancy?
2 comments
Thanks the authors to considering my concerns and the rebuttal. Some of my concerns have been addressed, but the novelty concern seems to be ignored which accounts for a significant ratio of my rating, so I keep my original rating.
Response to Reviewer KyAG
Thank you for the helpful questions.
Q1. How much do RFT and the initial ranks used in training contribute? On 2,017 Dress queries, all rerankers share the same 256 candidates per query, 256px image cap and fusion. The trained models share data and update budget.
At alpha=4, RFT gains 3.87 R@10 points over Yes/No SFT (95% paired bootstrap CI [2.41,5.32]). At alpha=2, RFT scores 61.03% and SFT scores 62.07% (difference -1.04 points, CI [-2.65,0.58]). We resample queries grouped by reference image, keeping models fixed; each CI covers one comparison. RFT improves on SFT at alpha=4, with gains depending on the fusion weight.
We removed the initial ranks when sampling rankings for RFT training, then tested both models at alpha=4. Keeping initial ranks during training improves R@1 from 27.37% to 30.09% and R@10 from 59.30% to 63.41% (+4.12 points, 95% CI [2.88,5.39]): 128 targets enter the top 10 and 45 leave. Without initial ranks in training, tuning test-time alpha to 8 raises R@10 to 61.77%, still 1.64 points below 63.41%. Thus, using retrieval ranks during training adds value beyond test-time fusion alone.
Q2. How do the scoring methods compare, and what is new? Table 4 of the submitted paper uses a shared query subset with configuration-specific losses and resolutions. Pointwise/pairwise R@10 is 60.88/57.21% before fusion and 62.07/60.09% after fusion: 16 candidate evaluations versus 120 pairs.
With matched Bradley-Terry loss, 256px inputs, training pairs, updates and top-16 pools on all 2,017 Dress queries, pointwise/pairwise R@10 is 58.80/59.40% without fusion. At alpha=2, 4 and 8, it is 60.39/58.21%, 60.34/57.36% and 59.94/56.77%. Pointwise uses Yes/No logit differences; pairwise averages soft wins in [0,1]. Equal alpha gives unequal weight to retrieval, explaining pairwise's decline toward the initial ranking. These fused results show complementarity under the tested score scales; pointwise also needs fewer evaluations.
Yes/No scoring and retrieve-then-rerank pipelines have precedents. We combine VLM scores with initial ranks to sample orders from a Plackett-Luce distribution and reward ranking the target higher. Our contribution is this retrieval-aware training objective, supported by the ablations in Q1.
Q3. How often is the target in the top 256, and how do we count misses? Queries with missing targets count as failures, including all 135 misses among 2,017 original DQU Dress queries. We add missing targets only during training. R@256 bounds reranking recall because the reranker can only reorder the retrieved candidates.
Public MCoT retrieves 4174/4181 CIRR validation targets (99.83%); original hidden-test coverage is unavailable. Reported recall includes all retrieval failures.
Q4. Can you show examples where the correct image moves up? In query 01866, the user asks for yellow lace and a more open neckline. The retriever ranks a higher-neck pleated dress first. RFT moves the lace target from #11 to #1. RFT also moves a longer-sleeved target from #5 to #1 and a sculptural green dress from #158 to #10. A casual tank-style target falls from #2 to #13. We selected these cases after inspecting rankings.
For 200 Dress queries, a VLM labeled requested changes without seeing rankings. RFT raises R@10 over retrieval from 53.37% to 62.92% on 178 queries with multiple changes and from 55.47% to 64.23% on 137 detail-related queries. The groups overlap; both RFT–SFT intervals include zero. These results show that RFT can correct ranking errors involving multiple requested attributes.
Q5. How does our reranker differ from FiRE? FiRE's fine-tuned model encodes the query and candidate images separately, then ranks by vector similarity (Section 4.3). We will correct the related work. Our reranker feeds Qwen3-VL the reference image, requested change and one candidate together, then combines its next-token "Yes"/"No" scores with the initial rank. The key distinction is direct VLM assessment of each candidate against the request, rather than similarity between separately encoded vectors.
Q6. What was missing at line 312? Line 312 should read "95% confidence intervals". In the supplementary experiments, we compare methods on the same resampled query groups: 10,000 resamples for full Dress and 5,000 for the subsets. Each interval covers one comparison.