- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CIR-QDiag, a diagnostic dataset and paraphrase-calibrated metrics for measuring how CIR target scores respond to local instruction edits. It also proposes On-Policy Rank Distillation, which transfers a frozen teacher's soft ranking distribution over candidates periodically mined by the student. The approach is evaluated across four student checkpoints and CIR benchmarks.
Critical strengths
- CIR-QDiag addresses a limitation of aggregate retrieval recall. The distinction between any score decrease and a decrease beyond model-specific paraphrase variation is useful and carefully formalized.
- OPRD is simple, requires only the student model at inference, and is supported by candidate-selection, supervision, teacher-choice, and cross-dataset experiments. The gains are not limited to one student backbone.
Critical weaknesses
- CIR-QDiag measures target-score changes rather than rank changes, and its limitations note that a target score can fall without the target ranking falling. Since CIR-QDiag is intended to diagnose instruction sensitivity in retrieval models, an evaluation showing whether CRR is associated with rank changes under edited queries would strengthen the interpretation of the metric.
- The same model family generates and visually validates the edits, followed by a binary human acceptance check. The appendix states that each proposal was judged by one reviewer and explicitly notes that the 95% acceptance rate is not inter-annotator agreement. An independently double-annotated subset would help establish the reliability of the edit labels, particularly across ADD, DELETE, and SUBSTITUTE operations.
Questions for authors
- Can the authors show that CRR correlates with actual retrieval-rank changes on edited queries, or explain why target-score sensitivity is an adequate proxy when competing gallery scores also change?
- Was each proposed edit independently labeled by more than one annotator? If not, can an independently double-annotated subset be provided, along with agreement and per-operation error rates?
3 comments
Response to Reviewer 1j68 (Part 2/2)
- Sample. A stratified random sample of 360 edits from the released CIR-QDiag (10,131 edits, all of which had passed both GPT-5.6-terra validation and the original human review), with 40 edits from each of the nine operation × category cells (120 per operation).
- Annotators. Five PhD students in computer vision and natural language processing, none of whom participated in the original review. Each annotated independently, without access to the model's decisions or the other annotators' labels.
- Protocol. Given the original and edited instructions, the reference image, and one annotated positive, each annotator labeled every edit as accept, reject, or uncertain, following the criteria in Appendix B.2.1.
Inter-annotator agreement is high.
- Average pairwise agreement: 97.17% (ten annotator pairs, ranging from 94.44% to 99.17%).
- Fleiss' κ: 0.6724. Krippendorff's α: 0.6726.
- All five annotators agree on 93.61% of edits.
κ and α are much lower than raw agreement because the label distribution is highly skewed: 95.5% of all labels are accept. This is the well-known prevalence effect, under which chance-corrected coefficients are depressed even when annotators rarely disagree. The prevalence-adjusted bias-adjusted kappa (PABAK) is 0.9575 overall. The effect is strongest for DELETE, where only 19 of 600 labels are reject, so κ falls to 0.43 despite 96.50% pairwise agreement.
| Operation | #Edits | Pairwise agreement | Fleiss' κ | Krippendorff's α | PABAK | Rejection rate (range across annotators) | Majority rejection (≥3 of 5) |
|---|---|---|---|---|---|---|---|
| ADD | 120 | 98.50% | 0.6943 | 0.6948 | 0.9775 | 0.83–1.67% | 1/120 (0.83%) |
| SUBSTITUTE | 120 | 96.50% | 0.7599 | 0.7603 | 0.9475 | 5.83–9.17% | 7/120 (5.83%) |
| DELETE | 120 | 96.50% | 0.4293 | 0.4302 | 0.9475 | 0.83–7.50% | 2/120 (1.67%) |
| Overall | 360 | 97.17% | 0.6724 | 0.6726 | 0.9575 | 2.78–5.56% | 10/360 (2.78%) |
Uncertain labels are rare (13 of 1,800 labels, 0.72%) and are not counted as rejections.
Per-operation error rates. Since no ground truth exists beyond human judgment, a true error rate cannot be measured directly. We instead report how often independent experts reject released edits, both per annotator and by majority vote. Because a rejection may also reflect an annotator's own error, these figures should be interpreted as disagreement rates rather than error rates.
- Overall. 10 of the 360 edits (2.78%) are rejected by a majority.
- SUBSTITUTE has the highest rejection rate for every annotator (5.83% to 9.17%; tied with DELETE for one annotator) and the most majority rejections (7 of 120). Five of these are near-synonym substitutions that do not alter the constraint (e.g., "lavender" → "lilac"). In the other two, the substituted value remains compatible with the positive image.
- ADD and DELETE are rarely rejected by a majority (1 and 2 of 120). The ADD case adds a condition that the positive already satisfies. The two DELETE cases remove a redundant phrase, so the query is not actually relaxed (e.g., "subject of the image as a painting" → "subject as a painting").
- DELETE disagreement is concentrated in one annotator. Nearly half of the DELETE rejection labels (9 of 19) come from a single annotator, who judged deletions of context-recoverable words (e.g., "add a spoiler to this car" → "add a spoiler") as not relaxing the query. This is a genuine boundary case, which we now discuss in Appendix B.2.2 (p. 18).
The same-model concern. The annotators are independent of the model that both generated and validated the edits. Errors shared by the generator and the validator would therefore surface as human rejections. The low majority-rejection rate (2.78%) suggests that such correlated errors are limited, and the SUBSTITUTE results indicate where they are most likely to occur.
Revisions. The revised PDF adds:
- the study protocol and results, with examples of rejected SUBSTITUTE and DELETE edits (Appendix B.2.2, Table 6, pp. 17–18), referenced from Sec. 3.2.2 and the Limitations (p. 10);
- a discussion of near-synonym substitutions and redundant-phrase deletions as limitations of the current validation criteria (Appendix B.2.2).
The original 95% acceptance rate is kept clearly distinct from inter-annotator agreement (Appendix B.2.1, p. 17).
I would like to thank the authors to the rebuttal. All of my concerns have been addressed, so I keep my positive rating.
Response to Reviewer 1j68 (Part 1/2)
We thank the reviewer for the positive assessment. In particular, we appreciate the reviewer's recognition that:
W1 & Q1: Association between CRR and retrieval-rank changes
Yes, CRR is strongly associated with rank changes. For each target–edit pair, we computed the target's rank under the original and the edited query over the full gallery (109,598 images).
Main findings.
Competing gallery scores. Ranks already incorporate the movement of every competing gallery image, so this analysis directly addresses the concern that competing scores also change.
Rank demotion without CRR. Rank demotion also occurs when CRR = 0, which is expected. CRR is deliberately conservative: it counts only decreases beyond paraphrase variation, whereas a small decrease can still cost a rank when competing scores are close. CRR is therefore a high-precision indicator of rank deterioration that complements the rank itself.
This analysis is now in Sec. 4.2 (p. 7) and Appendix B.5 (Table 12, pp. 21–22), and the Limitations are revised accordingly (p. 10).
W2 & Q2: Reliability of edit annotations
No, not in the original pipeline. Each edit that passed automatic validation was reviewed by a single annotator, and the 95% acceptance rate is not a measure of inter-annotator agreement. To address this, we conducted an independent study in which five human annotators re-annotated a stratified sample.