acceptodds
Under review as a conference paper at ICLR 2027

Follow the Edit: Counterfactual Diagnostics and On-Policy Rank Distillation for Composed Image Retrieval

Abstract

Composed image retrieval (CIR) retrieves target images based on the reference image and an editing instruction, satisfying the requested changes while preserving relevant visual content. Conventional retrieval metrics measure whether targets appear among the top-ranked results but do not directly assess responses to changes in the user's editing instruction. We introduce CIR-QDiag, a dataset of 10,131 local instruction edits, together with two complementary metrics, Directional Response Rate (DRR) and Calibrated Response Rate (CRR), to assess instruction following in CIR through models’ responses to these edits. Our diagnostic reveals that high retrieval recall does not guarantee high scores on both metrics and that instruction sensitivity varies across models. To improve CIR performance by transferring instruction-sensitive relevance judgments from stronger models to smaller retrieval models, we propose On-Policy Rank Distillation (OPRD), which matches a frozen teacher's soft relevance distribution over student-mined candidates. The student minimizes reverse KL on the shared candidates, focusing supervision on its current rankings. This transfers soft ranking information beyond the annotated target, while retaining student-only inference. OPRD trains on original CIR pairs without diagnostic edits. With BGE-MLLM-S2 as the teacher, OPRD achieves higher CRR than the InfoNCE baseline across all four student checkpoints. For CLIP-L, it improves CRR from 2.00% to 6.22% and CIRR Recall@1 from 12.80% to 42.72% relative to the pretrained checkpoint, with additional retrieval gains on PinPoint and FashionIQ. Code: https://anonymous.4open.science/r/QDiag-OPRD-877B/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Composed Image Retrieval, Counterfactual Diagnostics, Rank Distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1j68OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CIR-QDiag, a diagnostic dataset and paraphrase-calibrated metrics for measuring how CIR target scores respond to local instruction edits. It also proposes On-Policy Rank Distillation, which transfers a frozen teacher's soft ranking distribution over candidates periodically mined by the student. The approach is evaluated across four student checkpoints and CIR benchmarks.

Critical strengths
  1. CIR-QDiag addresses a limitation of aggregate retrieval recall. The distinction between any score decrease and a decrease beyond model-specific paraphrase variation is useful and carefully formalized.
  2. OPRD is simple, requires only the student model at inference, and is supported by candidate-selection, supervision, teacher-choice, and cross-dataset experiments. The gains are not limited to one student backbone.
Critical weaknesses
  1. CIR-QDiag measures target-score changes rather than rank changes, and its limitations note that a target score can fall without the target ranking falling. Since CIR-QDiag is intended to diagnose instruction sensitivity in retrieval models, an evaluation showing whether CRR is associated with rank changes under edited queries would strengthen the interpretation of the metric.
  2. The same model family generates and visually validates the edits, followed by a binary human acceptance check. The appendix states that each proposal was judged by one reviewer and explicitly notes that the 95% acceptance rate is not inter-annotator agreement. An independently double-annotated subset would help establish the reliability of the edit labels, particularly across ADD, DELETE, and SUBSTITUTE operations.
Questions for authors
  1. Can the authors show that CRR correlates with actual retrieval-rank changes on edited queries, or explain why target-score sensitivity is an adequate proxy when competing gallery scores also change?
  2. Was each proposed edit independently labeled by more than one annotator? If not, can an independently double-annotated subset be provided, along with agreement and per-operation error rates?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 1j68 (Part 1/2)

We thank the reviewer for the positive assessment. In particular, we appreciate the reviewer's recognition that:

  • the distinction between any score decrease and a decrease beyond paraphrase variation is "useful and carefully formalized";
  • OPRD is simple and requires only the student at inference;
  • OPRD is supported by candidate-selection, supervision, teacher-choice, and cross-dataset experiments across several student backbones.

W1 & Q1: Association between CRR and retrieval-rank changes

W1. CIR-QDiag measures target-score changes rather than rank changes, and its limitations note that a target score can fall without the target ranking falling. Since CIR-QDiag is intended to diagnose instruction sensitivity in retrieval models, an evaluation showing whether CRR is associated with rank changes under edited queries would strengthen the interpretation of the metric.

Q1. Can the authors show that CRR correlates with actual retrieval-rank changes on edited queries, or explain why target-score sensitivity is an adequate proxy when competing gallery scores also change?

Yes, CRR is strongly associated with rank changes. For each target–edit pair, we computed the target's rank under the original and the edited query over the full gallery (109,598 images).

  • Scope. The analysis covers 5,735 ADD/SUBSTITUTE edits in 870 groups (47,391 target–edit pairs).
  • Exclusion of DELETE. After a constraint is removed, the target remains valid, so its rank change has no normative interpretation.
  • Uncertainty. 95% CIs are obtained by group-level cluster bootstrap with 2,000 resamples, holding b_ψ fixed.
ModelRank demotion (CRR=1)Rank demotion (CRR=0)Difference [95% CI]Top-10 exit (CRR=1)Top-10 exit (CRR=0)Weighted Spearman ρ [95% CI]
CLIP-L (original)99.4661.3738.09 [36.59, 39.51]70.5816.920.827 [0.815, 0.838]
CLIP-L (OPRD)93.8264.7429.07 [26.60, 31.51]77.8631.890.675 [0.655, 0.695]
Qwen3-VL-Emb-2B (original)97.3663.6033.75 [32.23, 35.25]74.4624.020.794 [0.777, 0.808]
BGE-MLLM-S299.0167.3931.63 [30.04, 33.10]87.5737.170.825 [0.811, 0.837]

Main findings.

  1. When CRR = 1, the target is demoted in 94% to 99% of cases, compared with 61% to 67% when CRR = 0. None of the confidence intervals for the difference includes zero.
  2. Among targets initially ranked in the top 10, a calibrated response removes the target from the top 10 2.4 to 4.2 times as often as a non-calibrated response.
  3. The target-score change is strongly correlated with the log ratio of the target's rank after versus before the edit (weighted Spearman ρ from 0.68 to 0.83).

Competing gallery scores. Ranks already incorporate the movement of every competing gallery image, so this analysis directly addresses the concern that competing scores also change.

Rank demotion without CRR. Rank demotion also occurs when CRR = 0, which is expected. CRR is deliberately conservative: it counts only decreases beyond paraphrase variation, whereas a small decrease can still cost a rank when competing scores are close. CRR is therefore a high-precision indicator of rank deterioration that complements the rank itself.

This analysis is now in Sec. 4.2 (p. 7) and Appendix B.5 (Table 12, pp. 21–22), and the Limitations are revised accordingly (p. 10).

W2 & Q2: Reliability of edit annotations

W2. The same model family generates and visually validates the edits, followed by a binary human acceptance check. The appendix states that each proposal was judged by one reviewer and explicitly notes that the 95% acceptance rate is not inter-annotator agreement. An independently double-annotated subset would help establish the reliability of the edit labels, particularly across ADD, DELETE, and SUBSTITUTE operations.

Q2. Was each proposed edit independently labeled by more than one annotator? If not, can an independently double-annotated subset be provided, along with agreement and per-operation error rates?

No, not in the original pipeline. Each edit that passed automatic validation was reviewed by a single annotator, and the 95% acceptance rate is not a measure of inter-annotator agreement. To address this, we conducted an independent study in which five human annotators re-annotated a stratified sample.

AuthorsOpenReview ↗

Response to Reviewer 1j68 (Part 2/2)

  • Sample. A stratified random sample of 360 edits from the released CIR-QDiag (10,131 edits, all of which had passed both GPT-5.6-terra validation and the original human review), with 40 edits from each of the nine operation × category cells (120 per operation).
  • Annotators. Five PhD students in computer vision and natural language processing, none of whom participated in the original review. Each annotated independently, without access to the model's decisions or the other annotators' labels.
  • Protocol. Given the original and edited instructions, the reference image, and one annotated positive, each annotator labeled every edit as accept, reject, or uncertain, following the criteria in Appendix B.2.1.

Inter-annotator agreement is high.

  • Average pairwise agreement: 97.17% (ten annotator pairs, ranging from 94.44% to 99.17%).
  • Fleiss' κ: 0.6724. Krippendorff's α: 0.6726.
  • All five annotators agree on 93.61% of edits.

κ and α are much lower than raw agreement because the label distribution is highly skewed: 95.5% of all labels are accept. This is the well-known prevalence effect, under which chance-corrected coefficients are depressed even when annotators rarely disagree. The prevalence-adjusted bias-adjusted kappa (PABAK) is 0.9575 overall. The effect is strongest for DELETE, where only 19 of 600 labels are reject, so κ falls to 0.43 despite 96.50% pairwise agreement.

Operation#EditsPairwise agreementFleiss' κKrippendorff's αPABAKRejection rate (range across annotators)Majority rejection (≥3 of 5)
ADD12098.50%0.69430.69480.97750.83–1.67%1/120 (0.83%)
SUBSTITUTE12096.50%0.75990.76030.94755.83–9.17%7/120 (5.83%)
DELETE12096.50%0.42930.43020.94750.83–7.50%2/120 (1.67%)
Overall36097.17%0.67240.67260.95752.78–5.56%10/360 (2.78%)

Uncertain labels are rare (13 of 1,800 labels, 0.72%) and are not counted as rejections.

Per-operation error rates. Since no ground truth exists beyond human judgment, a true error rate cannot be measured directly. We instead report how often independent experts reject released edits, both per annotator and by majority vote. Because a rejection may also reflect an annotator's own error, these figures should be interpreted as disagreement rates rather than error rates.

  • Overall. 10 of the 360 edits (2.78%) are rejected by a majority.
  • SUBSTITUTE has the highest rejection rate for every annotator (5.83% to 9.17%; tied with DELETE for one annotator) and the most majority rejections (7 of 120). Five of these are near-synonym substitutions that do not alter the constraint (e.g., "lavender" → "lilac"). In the other two, the substituted value remains compatible with the positive image.
  • ADD and DELETE are rarely rejected by a majority (1 and 2 of 120). The ADD case adds a condition that the positive already satisfies. The two DELETE cases remove a redundant phrase, so the query is not actually relaxed (e.g., "subject of the image as a painting" → "subject as a painting").
  • DELETE disagreement is concentrated in one annotator. Nearly half of the DELETE rejection labels (9 of 19) come from a single annotator, who judged deletions of context-recoverable words (e.g., "add a spoiler to this car" → "add a spoiler") as not relaxing the query. This is a genuine boundary case, which we now discuss in Appendix B.2.2 (p. 18).

The same-model concern. The annotators are independent of the model that both generated and validated the edits. Errors shared by the generator and the validator would therefore surface as human rejections. The low majority-rejection rate (2.78%) suggests that such correlated errors are limited, and the SUBSTITUTE results indicate where they are most likely to occur.

Revisions. The revised PDF adds:

  • the study protocol and results, with examples of rejected SUBSTITUTE and DELETE edits (Appendix B.2.2, Table 6, pp. 17–18), referenced from Sec. 3.2.2 and the Limitations (p. 10);
  • a discussion of near-synonym substitutions and redundant-phrase deletions as limitations of the current validation criteria (Appendix B.2.2).

The original 95% acceptance rate is kept clearly distinct from inter-annotator agreement (Appendix B.2.1, p. 17).

I would like to thank the authors to the rebuttal. All of my concerns have been addressed, so I keep my positive rating.

Reviewer_vR8fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

Focusing on composed image retrieval, this work reveals that standard evaluation metrics fail to verify models’ genuine comprehension of local edit instructions. Many observed score decreases are caused by irrelevant semantic noise instead of targeted responses to edits. They also build a counterfactual diagnostic dataset for local instruction modifications and present On-Policy Rank Distillation (OPRD). Counterfactual diagnostic results validate the analysis result and the efficacy of OPRD.

Critical strengths

(1) Many observed score drops for locally-edited retrieval instructions in composed image retrieval stem from irrelevant semantic fluctuations, rather than meaningful semantic responses to the edits. A counterfactual diagnostic dataset is constructed to validate this observation. (2) Propose the On-Policy Rank Distillation (OPRD) algorithm to improve the retrieval performance and instruction-following capability of small models.

Critical weaknesses

(1) The paper does not analyze the nature of irrelevant semantic fluctuations and lacks discussion on the underlying reasons why such fluctuations give rise to the observed issues. (2) The contributions appear incremental instead of being fundamental.

Questions for authors

why teacher selection leads to substantial performance differences over various edit types?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vR8f (Part 1/2)

We thank the reviewer for the careful reading and constructive feedback. We have reorganized the Introduction and Sections 4.2–4.3. CIR-QDiag first shows that even a retriever with high recall can respond weakly when one instruction condition changes. We then examine the student’s own retrievals. Some images miss a single requirement, while others satisfy the full request despite being unannotated. One-hot supervision labels both as negatives. This observation motivates OPRD, which uses a teacher to grade these candidates alongside the target. We return to held-out edits to test what the student learns from those rankings. We hope this structure makes the paper easier to follow and clarifies how CIR-QDiag informs both the design and the evaluation of OPRD.

W1. Nature of irrelevant semantic fluctuations

W1. The paper does not analyze the nature of irrelevant semantic fluctuations and lacks discussion on the underlying reasons why such fluctuations give rise to the observed issues.

We hold the reference and target images fixed and compare target scores across human-verified PinPoint paraphrases. For each checkpoint, bψb_\psi is the group-balanced 95th percentile of the absolute score differences between paraphrases.

Magnitude. For checkpoints with a non-trivial edit response, the all-edit mean decrease (original score minus edited score) is approximately 13% to 29% of bψb_\psi.

CheckpointAll-edit mean decreasebψb_\psi
Pretrained CLIP-L0.00550.0425
CLIP4CIR0.01400.0965
BGE-MLLM-S20.01780.0712
Qwen3-VL-Emb-8B0.03290.1129

On incompatible ADD and SUBSTITUTE edits (A/S), pretrained CLIP-L lowers the target score in 66.15% of cases, but only 2.02% clear its paraphrase threshold (Table 1, p. 8). A decrease below that threshold is compatible with ordinary rewording.

Dependence on training. All four students have lower thresholds after BGE-supervised OPRD than after standard InfoNCE.

Studentbψb_\psi after InfoNCEbψb_\psi after OPRD
CLIP-L0.09120.0723
CLIP-B0.09230.0717
Qwen3-VL-Emb-2B0.14880.0849
Qwen3-VL-2B-Instruct0.15710.0732

Part of this variation reflects the checkpoint’s score scale. We therefore calibrate each checkpoint separately. A plausible explanation for wording sensitivity is that sentence-level contrastive training aligns whole captions with images and provides limited supervision for individual conditions. Wording, length, or word order can then affect the query embedding (Kargi et al., 2026 and Yuksekgonul et al., 2023). Testing this mechanism would require a controlled intervention. Section 4.2 (p. 7) summarizes the findings. Appendix B.4 (pp. 20–21) discusses the explanation, and Table 11 (p. 21) gives both comparisons above.

W2. Significance of the contributions

W2. The contributions appear incremental instead of being fundamental.

We respectfully disagree. A retriever can return the right image yet barely respond when one requested condition changes. Our contribution is to measure this weakness against the variation caused by paraphrasing and to show that it can be reduced on held-out edits using ordinary CIR training pairs. The training signal comes from a teacher grading how well the student’s own retrieved alternatives satisfy the original instruction.

What the diagnostic reveals. Among four additional CIR methods, CLIP4CIR has the highest CIRR R@1 (45.37%) and the lowest all-edit CRR (2.34%). Across those methods and the two teachers, all-edit DRR spans 64.67%–70.08%, while all-edit CRR spans 2.34%–10.21%. On incompatible ADD and SUBSTITUTE edits, the range in CRR is wider still, from 2.50% to 13.21% (Tables 1 and 3, pp. 8–9, and Tables 8 and 27, pp. 19 and 35).

Why distill rankings on the student’s own candidates? CIR-QDiag varies one instruction condition with the images fixed. We asked whether the same distinctions occur in training when the instruction stays fixed and the candidate images vary. We audited fourth-epoch CLIP-L rankings for 25 CIRR training queries. A single condition distinguished the target from 212 of 775 student-retrieved alternatives (27.4%) and 14 of 775 random alternatives (1.8%). Of the 212, 149 violated one requirement. The other 63 satisfied the request and could be distinguished by adding a condition. One-hot supervision labels both groups as negatives (Sec. 4.3, p. 7 and Table 2(a), p. 8). OPRD lets a frozen teacher grade the target and the student’s current alternatives using cached embeddings. The student refreshes the candidates each epoch and matches the teacher’s distribution by image ID using reverse KL, allowing the encoders to differ (Secs. 2.2 and 3.3, pp. 3 and 5–6).

AuthorsOpenReview ↗

Response to Reviewer vR8f (Part 2/2)

What transfers to held-out edits? OPRD trains on original CIRR pairs. CIR-QDiag edits are held out from training and model selection. With teacher, loss, target insertion, and refresh frequency fixed, switching from random to student-mined candidates raises CLIP-L’s held-out A/S CRR from 5.87% to 10.30%. Under the same mining rule, replacing one-hot supervision with OPRD raises CIRR R@1 from 33.92% to 42.72% and held-out A/S CRR from 6.45% to 10.30% (Table 2(b), p. 8). Each run uses its own evolving student. On ADD alone, CLIP-L’s CRR is 0.13% before training, 1.59% with standard InfoNCE, and 12.13% with OPRD (Table 13, p. 23).

The form of teacher supervision also matters. With the same BGE teacher, matching the relevance distribution yields higher CIRR R@1 than centered score regression (CMSE) for all four students and higher A/S CRR for three of four. The Instruct student favors CMSE on A/S CRR (11.11% versus 8.95%). Section 4.5 (p. 9) reports this comparison, with retrieval in Tables 3 and 19 and A/S CRR in Tables 1 and 10.

Q1. Effect of teacher choice across edit types

Q1. Why teacher selection leads to substantial performance differences over various edit types?

Under BGE-MLLM-S2, all four OPRD students gain on ADD and lose on SUBSTITUTE relative to standard InfoNCE. Under Qwen3-VL-Emb-8B, all four match or exceed their InfoNCE SUBSTITUTE CRR. Each cell gives ADD / SUBSTITUTE CRR (%) (Sec. 4.3, p. 7 and Table 13, pp. 22–23).

StudentStandard InfoNCEOPRD with BGE-MLLM-S2OPRD with Qwen3-VL-Emb-8B
CLIP-L1.59 / 10.2612.13 / 7.387.64 / 10.26
CLIP-B2.13 / 9.4012.26 / 6.688.15 / 9.96
Qwen3-VL-Emb-2B7.48 / 13.8628.18 / 8.0613.76 / 19.00
Qwen3-VL-2B-Instruct5.51 / 10.007.51 / 9.2112.21 / 14.35

Interpretation. The same teacher-specific patterns recur under CMSE (Appendix B.6, p. 22). BGE itself has 5.38% ADD CRR and 11.85% SUBSTITUTE CRR, while its Qwen3-VL-Emb-2B student reaches 28.18% and 8.06%, respectively. One plausible explanation is that the teachers assign different relevance to candidates that violate a requirement and to unannotated alternatives that also satisfy the request. A direct test would score the same labeled candidate sets with both teachers and compare the probability mass assigned to each type.

DELETE. Removing a condition can leave the target valid, so a larger decrease has no preferred direction. The signed mean change is smaller under BGE than under Qwen for CLIP-L, CLIP-B, and Qwen3-VL-Emb-2B. The Instruct student has a positive mean under both teachers (Table 14, pp. 23–24).

OPRD studentDELETE mean Δ\Delta, BGE teacherDELETE mean Δ\Delta, Qwen teacher
CLIP-L0.00080.0049
CLIP-B−0.00070.0032
Qwen3-VL-Emb-2B−0.00660.0131
Qwen3-VL-2B-Instruct0.01060.0095

Presentation

Presentation (score 1). Difficult to follow despite careful reading.

  1. Appendix B.1 (pp. 16–17) works through the example in Figure 1 (p. 4). The six steps are on p. 17, with a pointer in Sec. 3.2.3 (p. 5).
  2. Table 5 (p. 16) collects the notation used in that example.
  3. Section 4.2 (p. 7) presents the diagnostic findings. Section 4.3, From Student-Mined Candidates to Edit Sensitivity (p. 7), follows the candidate audit, the training controls, and transfer to held-out edits. Table 2(a–b) (p. 8) brings the audit and the corresponding training comparisons together.
  4. Section 4.6 (p. 10) presents Qwen3-VL-2B-Instruct as an exploratory study.
  5. Table 1 (p. 8) foregrounds A/S DRR and CRR and reports DELETE CRR separately. Table 8 (pp. 19–20) gives the full all-edit results.
AuthorsOpenReview ↗

Follow-up on Rebuttal

Dear Reviewer,

Thank you again for your thoughtful comments and valuable feedback on our submission.

We have carefully prepared our rebuttal and provided detailed responses to your comments and concerns. We would greatly appreciate it if you could take a moment to review our responses when your schedule permits.

We sincerely hope that our clarifications and additional explanations address your concerns. If anything remains unclear or requires further discussion, we would be more than happy to provide additional clarification.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_6VuTOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces CIR-QDiag, a diagnostic benchmark that measures CIR models’ responses to local instruction edits while keeping the reference and target images fixed. DRR measures the frequency of target-score decreases, while CRR calibrates these responses against paraphrase variation. The paper also proposes On-Policy Rank Distillation (OPRD), which uses reverse KL to match a frozen teacher’s relevance distribution over periodically refreshed, student-mined candidates. Training uses only original CIRR pairs. Experiments on CIR-QDiag, CIRR, PinPoint, and FashionIQ report improvements in retrieval and aggregate CRR across several student configurations.

Critical strengths
  1. CIR-QDiag offers an interesting diagnostic perspective beyond aggregate retrieval recall. Controlled instruction edits isolate responses to individual conditions, while paraphrase-based calibration accounts for ordinary wording variation. This provides a useful basis for studying textual-detail sensitivity in CIR.
Critical weaknesses
  1. The design of CIR-QDiag is insufficiently justified. It applies the same score-decrease criterion to DELETE and target-incompatible ADD/SUBSTITUTE operations, although removing a constraint does not necessarily invalidate the original target.

  2. The proposed CIR-QDiag and OPRD are largely disconnected. OPRD relies on conventional relevance distillation without explicitly modeling counterfactual instruction responses, leaving the connection between the diagnostic findings and the proposed training objective insufficiently established.

  3. The methodological novelty of OPRD is limited. It mainly combines student-driven hard candidate mining with soft relevance distillation, both of which are established techniques in retrieval.

  4. The reliability of the teacher's relative similarity scores as fine-grained supervision is unclear. CIRR provides no graded relevance annotations, and the paper does not establish whether these scores reflect instruction compliance rather than general visual similarity.

  5. The experimental comparisons are insufficient. CIR-QDiag lacks systematic evaluation of recent CIR methods, while OPRD is not compared against representative state-of-the-art CIR approaches. This limits the demonstrated value of both contributions.

Questions for authors
  1. What justifies using the same score-decrease criterion for DELETE and ADD/SUBSTITUTE in CIR-QDiag, particularly given prior studies on similarity behavior under textual modifications [1]?

  2. How does the proposed CLIP composition head differ from CLIP4CIR [2], and why adopt this relatively dated feature-fusion design rather than more recent CIR architectures?

  3. What ensures that the teacher's relative similarity scores capture fine-grained instruction compliance? Why not incorporate counterfactual instruction edits into the CIRR training data to provide more direct supervision?

  4. Why are recent CIR methods not included in the CIR-QDiag evaluation or the retrieval comparisons?

  5. How are retrieval embeddings extracted from Qwen3-VL-2B-Instruct, and what motivates its inclusion alongside dedicated embedding models?

References

[1] Half-Truths Break Similarity-Based Retrieval.

[2] Composed Image Retrieval Using Contrastive Learning and Task-Oriented CLIP-Based Features.

7 comments
AuthorsOpenReview ↗

Response to Reviewer 6VuT (Part 1/5)

We thank the reviewer for the detailed and careful review, and for recognizing that CIR-QDiag "offers an interesting diagnostic perspective beyond aggregate retrieval recall". Its controlled edits isolate responses to individual conditions, and its paraphrase-based calibration accounts for ordinary wording variation. We address each concern below.

W1 & Q1: Justification for a unified score-decrease criterion

W1. The design of CIR-QDiag is insufficiently justified. It applies the same score-decrease criterion to DELETE and target-incompatible ADD/SUBSTITUTE operations, although removing a constraint does not necessarily invalidate the original target.

Q1. What justifies using the same score-decrease criterion for DELETE and ADD/SUBSTITUTE in CIR-QDiag, particularly given prior studies on similarity behavior under textual modifications [1]?

We agree that removing a constraint does not invalidate the original target, and we now state this explicitly in the paper (Sec. 3.2.3, "Interpreting a decrease", pp. 4–5). In short, what we unify is the measurement, not its interpretation. Using one measurement places all operations on a common, paraphrase-calibrated scale, and the prior findings in [1] are precisely what tell us how to interpret a decrease under each operation. We elaborate below.

ADD and SUBSTITUTE. The edited instruction is false for the target, so a score decrease is the expected behavior.

  • ADD provides a particularly clean test. Half-Truths [1] shows that CLIP-style encoders often increase image–text similarity when a plausible but incorrect detail is appended. A decrease after ADD therefore runs against this bias.
  • In this sense, CIR-QDiag ADD is a CIR counterpart of the half-truth test, with the reference image included in the query.

DELETE. The target still satisfies the edited instruction, so a score decrease admits two explanations. The model may credit the removed (correct) detail, or it may simply favor longer descriptions, as documented in [1]. DELETE therefore has no normative direction. Table 14 already notes that a higher DELETE rate does not imply better semantic understanding.

Our results reproduce the finding of [1] in CIR. For incompatible ADD edits, the target-score decrease rarely exceeds the paraphrase threshold:

ModelADD CRR
Pretrained CLIP-L0.13%
CLIP-B0.29%
TEMA0.32%
MagicLens-B0.81%
MagicLens-L0.88%

A shared measurement allows operations to be compared on a common scale.

CLIP-LADD CRRDELETE CRR
Pretrained0.13%1.99%
After InfoNCE1.59%3.48%
After OPRD12.13%0.81%

OPRD increases ADD CRR while decreasing DELETE CRR. The student thus learns to penalize incorrect additions without favoring longer queries, which is the opposite of the half-truth pattern. InfoNCE, by contrast, increases both rates.

Revisions (in blue). Operation-level results were already reported in the appendix (Tables 7, 10, 13, and 14). In the revised PDF, we have:

  1. adopted ADD/SUBSTITUTE CRR as the primary metric in Table 1 (p. 8);
  2. reported DELETE in a separate column of Table 1, without a "higher is better" indicator;
  3. moved all-edit CRR to the appendix (Table 8, pp. 19–20);
  4. discussed [1] in Sec. 3.2.3 (pp. 4–5), with the notation and a worked example in Appendix B.1 (pp. 16–17).

These changes do not affect our conclusions. OPRD outperforms InfoNCE on ADD/SUBSTITUTE CRR for all four students (10.30 vs. 6.45, 10.40 vs. 6.39, 18.34 vs. 11.03, and 8.95 vs. 7.95), and the ablations in Tables 30 and 32 are already reported on this subset.

W2: Connection between CIR-QDiag and OPRD

W2. The proposed CIR-QDiag and OPRD are largely disconnected. OPRD relies on conventional relevance distillation without explicitly modeling counterfactual instruction responses, leaving the connection between the diagnostic findings and the proposed training objective insufficiently established.

CIR-QDiag both motivates the design of OPRD and serves as its evaluation.

Diagnostic findings. The diagnostic yields two observations:

  • High retrieval recall does not imply sensitivity to instruction edits.
  • Large embedding models are considerably more edit-sensitive than CLIP. Their A/S CRR is 9.18% (BGE-MLLM-S2) and 13.21% (Qwen3-VL-Emb-8B), compared with 2.02% for CLIP-L (Table 1, p. 8).

Limitation of InfoNCE. InfoNCE only identifies the annotated target. It provides no information about which near-miss candidates partially satisfy the instruction, which is precisely what counterfactual edits probe. OPRD addresses this limitation in two ways:

  1. The student mines its own near-miss candidates.
  2. The teacher assigns graded relevance to these candidates, rather than treating all non-targets as equally irrelevant.
AuthorsOpenReview ↗

Response to Reviewer 6VuT (Part 2/5)

Each component affects the diagnostic directly.

VariantA/S CRR
OPRD10.30%
Random instead of student-mined candidates (Table 30)5.87%
One-hot labels instead of teacher scores, same mining (Table 32)6.45%

Characterizing the mined candidates. We analyzed the epoch-4 mining records of the main OPRD run (CLIP-L student with the residual head, BGE-MLLM-S2 teacher).

  • Sample. We examined 25 queries with 31 candidates each, i.e., 775 query–candidate pairs. As a control, we drew the same number of random gallery candidates.
  • Definition of an implicit counterfactual.
    • A conflict-type candidate violates the instruction in exactly one constraint.
    • An ADD-type candidate satisfies the instruction together with the target, and adding a single constraint would distinguish the two.
  • Procedure. Candidate edits were first proposed automatically and then verified by human annotators. The annotators were blind to the candidate source, scores, ranks, and training epoch.
Student-minedRandom
Conflict type149/775 (19.2%)14/775 (1.8%)
ADD type63/775 (8.1%)0/775 (0.0%)
Either type212/775 (27.4%)14/775 (1.8%)

Student mining surfaces such candidates roughly 15 times as often as random sampling. ADD-type candidates are particularly informative: they also satisfy the original instruction, yet InfoNCE treats them as negatives, whereas the teacher can assign them substantial relevance. This offers a concrete explanation for the benefit of graded supervision. Since the inspection covers only 25 queries, we present it as illustrative evidence.

Separation of training and diagnostic data. Training uses only CIRR pairs, whereas CIR-QDiag is constructed from PinPoint and used exclusively for evaluation. The CRR improvements therefore reflect transfer from standard supervision, rather than fitting to the diagnostic distribution.

This discussion is now in Sec. 4.3 ("From Student-Mined Candidates to Edit Sensitivity", p. 7, with Table 2(a–b) on p. 8), and the inspection protocol and results are in Appendix D.2 (Table 31, p. 36).

W3: Methodological novelty of OPRD

W3. The methodological novelty of OPRD is limited. It mainly combines student-driven hard candidate mining with soft relevance distillation, both of which are established techniques in retrieval.

We agree that student-driven mining and soft relevance distillation each have precedents (RankDistil, AR2, RocketQAv2, QuRe), and we do not claim novelty for either component in isolation. Our contribution is a specific formulation for CIR, and our experiments show that the gains arise from this formulation rather than from the individual components alone.

Formulation. OPRD is most closely related to on-policy distillation for language models (GKD; Agarwal et al., 2024), in which the student is trained on its own generated outputs. In OPRD, the candidates retrieved by the student serve this role.

  • Teacher. The teacher is frozen, and its embeddings are precomputed and cached. It is therefore never executed during student updates or at inference. Because distillation operates on relevance distributions over shared candidate IDs rather than on embeddings, the teacher and student may have different architectures.
  • Objective. The student minimizes the reverse KL divergence to the teacher's relevance distribution. The target is always included in the candidate set, and candidates are refreshed at every epoch.

Relation to prior work.

  • RankDistil preserves the teacher's top-k ordering.
  • QuRe mines hard negatives but relies on one-hot supervision.
  • AR2 and RocketQAv2 jointly train the retriever and the re-ranker, which requires teacher inference throughout training.

Empirical evidence for the design. We first compare against score distillation (CMSE) under the same teacher, candidates, and training budget:

StudentCIRR R@1 (OPRD vs. CMSE)A/S CRR (OPRD vs. CMSE)
CLIP-L42.72 vs. 35.9010.30 vs. 6.93
CLIP-B40.49 vs. 33.4110.40 vs. 6.62
Qwen3-VL-Emb-2B53.26 vs. 51.5418.34 vs. 10.62
Qwen3-VL-2B-Instruct (exploratory)52.09 vs. 48.368.95 vs. 11.11

OPRD achieves higher CIRR R@1 for all four students and higher A/S CRR for three of the four. The exception is the exploratory Instruct student.

We then ablate each component on CLIP-L:

ComponentVariantCIRR R@1A/S CRR
Candidate source (Table 30)Random → student-mined36.31 → 42.725.87 → 10.30
Supervision (Table 32)One-hot → teacher distribution, same mining33.92 → 42.726.45 → 10.30
Target insertion (Table 34)Without → with+2.46 at epoch 1, +0.65 at epoch 4

This discussion is now in the Related Work section (Sec. 2.2, p. 3), Sec. 4.3 (p. 7, with Table 2(b) on p. 8), and Sec. 4.5 (p. 9).

AuthorsOpenReview ↗

Response to Reviewer 6VuT (Part 3/5)

W4 & Q3: Reliability of teacher scores as supervision

W4. The reliability of the teacher's relative similarity scores as fine-grained supervision is unclear. CIRR provides no graded relevance annotations, and the paper does not establish whether these scores reflect instruction compliance rather than general visual similarity.

Q3. What ensures that the teacher's relative similarity scores capture fine-grained instruction compliance? Why not incorporate counterfactual instruction edits into the CIRR training data to provide more direct supervision?

OPRD does not require the teacher's scores to be calibrated graded-relevance labels. It only requires the teacher's relative preferences over candidates to be more consistent with the instruction than those of the student. We provide three lines of evidence for this premise.

1. Benchmarks that disentangle instruction compliance from visual similarity

BenchmarkWhat it measuresBGE-MLLM-S2 (teacher)Initial CLIP-L
PinPoint NegRecall@10 (↓)Retrieval of visually similar negatives that violate the instruction12.19%20.56%
CIR-QDiag A/S CRR (↑)Response to instruction edits, with both images fixed9.18%2.02%

Because both images are fixed in CIR-QDiag, the A/S CRR gap cannot be attributed to visual similarity.

2. Blind pairwise comparison

  • Setup. We sampled 25 CIRR training queries from the epoch-1 mining records of the main OPRD run (CLIP-L student with the residual head, BGE-MLLM-S2 teacher). For each query, the teacher and the initial CLIP-L student ranked the same set of 32 images (the target and 31 mined candidates).
  • Comparison. At ranks 1, 2, and 3, we compared the teacher's image with the student's image at the same rank.
  • Judgment. A human annotator was shown only the reference image, the modification text, and the two candidates in random, anonymized order, and selected the one that better satisfied the composed query. Comparisons in which both models returned the same image were excluded (8 of 75).
RankValid comparisonsTeacher winsTeacher win rate
Top-12424100.0%
Top-2221777.3%
Top-3211361.9%
All675480.6% (95% CI [69.6, 88.3])
  • Overall. The teacher is preferred in 54 of 67 comparisons (one-sided sign test, p < 0.001, treating comparisons as independent).
  • Beyond retrieving the target. The teacher's top-1 image is the annotated target in only 11 of the 24 cases. In the remaining 13, annotators still preferred the teacher's non-target top-1 image.
  • Near-miss candidates. At ranks 2 and 3, the teacher wins 30 of 43 comparisons (p < 0.01), and the margin narrows at lower ranks.
  • Scope. Given the small sample, we regard this as supporting rather than primary evidence. The protocol and results are in Appendix D.4 (Table 33, p. 37).

3. Teacher distributions outperform one-hot labels on identical candidates

SupervisionA/S CRRCIRR R@1
Teacher distribution (OPRD)10.30%42.72%
One-hot labels (Table 32)6.45%33.92%

This indicates that the teacher's relative scores carry useful information beyond the target label.

Why counterfactual edits are not used for training.

  • An edited instruction has no annotated target in the CIRR gallery, so it cannot form a standard (reference, text, target) triplet. It could only serve as a text-side negative, which constitutes a different form of supervision.
  • Excluding edits from training preserves CIR-QDiag as an independent test of transfer.
  • Edit-based augmentation, such as the component-level foils in [1], is complementary to OPRD. We note it as future work in the Limitations (p. 10).

W5 & Q4: Comparison with recent CIR methods

W5. The experimental comparisons are insufficient. CIR-QDiag lacks systematic evaluation of recent CIR methods, while OPRD is not compared against representative state-of-the-art CIR approaches. This limits the demonstrated value of both contributions.

Q4. Why are recent CIR methods not included in the CIR-QDiag evaluation or the retrieval comparisons?

We evaluated four representative CIR methods using their official checkpoints. Each model is calibrated with its own paraphrase threshold, and all models follow the same evaluation protocol as in the paper.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

We sincerely thank all three reviewers for their thoughtful and constructive feedback. We are encouraged that the reviewers found that:

  • CIR-QDiag provides a useful diagnostic perspective beyond aggregate retrieval recall (6VuT, vR8f, 1j68);
  • the distinction between arbitrary score decreases and decreases beyond paraphrase variation is carefully formalized (1j68);
  • OPRD is simple, improves compact students, and generalizes across student backbones (vR8f, 1j68).

All revisions are marked in blue in the revised PDF, and the main text remains within the 10-page limit for the discussion phase. We have updated the section, table, and page references in our responses to match the reorganized manuscript.

Additional experiments

  1. Evaluation of four further CIR methods (TEMA, CLIP4CIR, MagicLens-B/L) on CIR-QDiag, CIRR, PinPoint, and FashionIQ, together with a comparison to the reported CIRR results of SPRC (Tables 1 and 3, pp. 8–9; Appendix C.4, pp. 34–35).
  2. OPRD with three alternative composition heads (Appendix D.8, Table 37, p. 39).
  3. An analysis of the association between CRR and actual retrieval-rank changes (47,391 target–edit pairs; Appendix B.5, Table 12, pp. 21–22).
  4. An independent five-annotator study of 360 edits (Appendix B.2.2, Table 6, pp. 17–18).
  5. Two blind inspections. The first compares teacher and student rankings (Appendix D.4, Table 33, p. 37); the second characterizes student-mined candidates (Appendix D.2, Table 31, p. 36).

Main revisions

  • ADD/SUBSTITUTE CRR becomes the primary metric. DELETE is reported separately, without a "higher is better" interpretation (Sec. 3.2.3, p. 5; Table 1, p. 8).
  • We clarify how OPRD relates to prior distillation methods (Sec. 2.2, p. 3). We also reorganized Secs. 4.2–4.3 (p. 7) to make the connection between CIR-QDiag and OPRD easier to follow. Sec. 4.2 shows that high retrieval recall can coexist with low sensitivity to a change in a single instruction condition. Sec. 4.3 then shows that the student’s own top-ranked candidates are more likely than random candidates to differ from the target on a single condition. Table 2(a) reports this candidate analysis, and Table 2(b) shows how candidate selection and teacher supervision affect the student’s sensitivity to held-out CIR-QDiag edits (p. 8).
  • We revise the presentation, as detailed in our response to vR8f.

All main conclusions hold under the stricter metric.

  • OPRD outperforms InfoNCE on ADD/SUBSTITUTE CRR for all four students (10.30 vs. 6.45, 10.40 vs. 6.39, 18.34 vs. 11.03, and 8.95 vs. 7.95).
  • Every OPRD student achieves higher all-edit and A/S CRR than all four additional CIR methods.
AuthorsOpenReview ↗

General Response_2

We thank the reviewers for their careful reading and constructive feedback. We have revised the Introduction and Sections 4.2–4.3 to clarify how the findings from CIR-QDiag motivate the design of OPRD and how we use held-out instruction edits to evaluate what the student learns. Table 2 now brings together the candidate audit and the training comparisons, and Table 3 reports the main retrieval results.

In our previously posted rebuttal, the updates are limited to table references, section titles, and page numbers. We hope these changes make the paper easier to follow and help reviewers locate the evidence cited in each response.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.