acceptodds
Under review as a conference paper at ICLR 2027

Partial Reranking: Action Structure and Calibration

Abstract

A reranker can improve average retrieval while damaging queries already served well by a baseline. Disjoint permutation cycles allow favorable and unfavorable changes to be decided separately, using the same predicted relevance and proposed ranking. This paper studies the value of that choice under uncertain relevance. On a locked Flickr30k-derived gallery, separately calibrated partial replacement increases signed reciprocal-rank gain by 0.000930 for BLIP and 0.002255 for ALBEF; both adjusted intervals are positive, and only ALBEF resolves the specified 0.001 practical margin. Same-price contrasts retain the benefit, as does a finite calibration-optimal whole-query comparator. A separate split-calibration selection-and-guard procedure meets practical criteria in four setting–model cells, while COCO BLIP remains sub-practical and VizWiz BLIP loses gain. Equal-cardinality interventions further show that which changes are bound together matters beyond action count. Variation in cycle switching prices, weighted by predicted loss, characterizes predicted binding cost across prices, but reducing that cost does not ensure better calibrated gain. At a common observed loss, label-aware optima over fixed score menus favor partial actions in all six cells; these hindsight gains do not establish deployable improvement. Adding predicted opportunity to a fixed predictor worsens error in all four model–price settings. The evidence separates structural decision value from outcome prediction and identifies calibration as a consequential limit on partial reranking.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
image-text retrieval, partial reranking, permutation cycles, action grouping, risk-sensitive retrieval, calibration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JbxVOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to apply a reranker proposal after a top-32 permutation has already been produced. QUERY accepts or rejects the full proposal. CYCLE decomposes the permutation into disjoint cycles and decides whether to apply each cycle separately. The experiments use single-target reciprocal-rank gain and loss. Validation includes a locked Flickr setting, same-price decomposition, whole-query controls, calibration sensitivity, external transports, shortlist stress tests, and serving-time measurements.

Critical strengths
  1. The paper provides a clean controlled analysis of action structure. CYCLE and QUERY use the same predicted relevance and proposed ranking. The same-price decomposition shows positive conditional action contrasts in the locked Flickr setting, so the result is not explained only by different calibrated prices.

  2. The experimental and statistical reporting is unusually careful. The paper includes same-proposal whole-query gates, finite calibration-optimal menus, calibration-role sensitivity, external reversals, shortlist failures, and timing results. It clearly states that its intervals condition on the fixed pipeline and that partial action does not provide model-inference speedup.

Critical weaknesses
  1. The locked primary calibration rule does not directly optimize gain under the risk constraint. It selects the most permissive feasible price. The paper later performs SELECT/Guard optimization for CYCLE and a finite WHOLE menu, which partially addresses this issue, but that evidence is an exposed secondary analysis. The WHOLE menu is also finite, and VizWiz BLIP shows a clear reversal.

  2. The uncertainty analysis and generality remain conditional. The main intervals do not resample fitted heads, calibration composition, or operating-point selection. The main theoretical and empirical conclusions also use single-target reciprocal-rank utility and a finite whole-query comparator family. These limits are clearly disclosed, but they restrict how broadly the result can be applied.

Questions for authors
  1. Uncertainty clarification: How do the gain and loss intervals change when the full pipeline is repeated, including head fitting, calibration composition, and operating-point selection?
  2. Comparator clarification: Can a higher-capacity whole-query selector close the gap while using exactly the same proposer, inputs, and out-of-sample selection protocol?
  3. Generality clarification: Does cycle-level action retain an advantage with multiple relevant items and metrics such as NDCG or AP?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JbxV

What changes when fitting and calibration are repeated? Ten repeated fits materially change the ALBEF conclusion. Against richer WHOLE, BLIP's mean gain difference is +0.000768 [−0.000562,0.001723], with run SD 0.001466; ALBEF's is −0.001062 [−0.003812,0.001808], with SD 0.003444. For the designated fixed run of this same CYCLE−richer-WHOLE comparison, the adjusted gain-difference bounds are [0.000513,0.000900] and [0.002510,0.002897]. Added-positive-loss bounds change from [−0.000072,0.000060] to [−0.000057,0.000076] for BLIP, and from [−0.000018,0.000089] to [−0.000162,0.000013] for ALBEF, from the fixed run to the ten-run mean. These bounds are adjusted across the study's gain and risk comparisons. Repetition changes both effects and uncertainty. Table 23 (p. 26) reports gain, run SD and both additional SigLIP conditions.

What is repeated, and what information does WHOLE receive? Each run repeats source-data folds, posterior fitting, proposal-specific WHOLE fitting, source-model selection, calibration-split allocation and operating-point selection. Pretrained checkpoints and test images remain fixed. Three nested heads are averaged within each run; crossed run/image intervals reflect this scope. The richer selector receives the same posterior, proposal and public features and includes learned nonlinear gates. This is separate from the original same-head QUERY comparison.

A separate registered auxiliary experiment keeps the original proposal fixed while both CYCLE and strong WHOLE receive the same semantic-augmented information. Across five source-data fits, three of four COCO conditions meet the joint criterion. The CYCLE−WHOLE gain difference for SigLIP2/ALBEF is +0.003940 [0.003091,0.004783]. None of the four conditions in the Flickr30k-derived cohort or the four VizWiz conditions meets it (Appendix I.1, Tables 29–31). The external bounds retain the 64-coordinate family, separate from the Flickr30k-derived cohort's 32-coordinate family. These cycle decisions on a fixed proposal are distinct from LOCAL's global menu below.

Does WHOLE have the same policy opportunities as LOCAL? Yes. Matched WHOLE contains every global policy available to LOCAL and uses the same selection objective, loss constraint, head averaging and risk calibration. Its menu adds QUERY, ridge, nonlinear MLP and margin gates to LOCAL's BASE and eleven proposals, with the same two-vertex mixtures. Numerical replay confirms that an identical selected policy gives identical outcomes under either implementation. WHOLE's ordering on the selection split therefore follows from containment, rather than demonstrating better generalization (Appendix H.2).

With the Qwen3-VL rerankers, LOCAL has 21/24 positive means and 7/24 joint practical successes, while original CYCLE has 1/24 joint success (Tables 24–25, pp. 28–29). At common retention, LOCAL has lower mean gain in all 24 conditions and lower harm in 23. Common retention is not common loss: Figure 5 (p. 31) gives the complete source-development curve across nine loss budgets. The retention identity in Appendix H.3 explains the accounting without assigning a causal overfitting or calibration penalty.

Does the result extend to graded retrieval? The completed Hotpot, NFCorpus and lexical/dense × Qwen/monoT5 comparisons use actual query-level harm. The last study has positive primary CYCLE−WHOLE means in 5/12 conditions and no practical-bound success. BM25+Qwen on MSMARCO is −0.022911 [−0.027414,−0.018231], negative in all three fits (Table 26, p. 32). Single-target additivity is not transferred to graded losses.

How are selection and evaluation separated? Model fitting uses source data; policy selection uses a separate split. The mixture freezes before risk calibration on an independent guard split and final test evaluation. The original feasible-price rule and later gain-maximizing selection remain distinct. Earlier benchmark summaries informed the follow-up questions. The completed comparisons preserve the controlled original effect while showing how stronger selectors, repeated fitting and graded retrieval limit its practical generality.

Dear Reviewer JbxV,

Following your questions about repeated fitting and comparator strength, we added a common ten-fit comparison in Appendix I.6 (pp. 50–52). The same C3 calibration procedure is applied to CYCLE and strong WHOLE with their predictors and proposals unchanged. Under this common analysis, ordinary calibration meets the joint gain-and-risk criterion in 3/12 conditions and C3 in 8/12.

Tables 48 and 50 provide the complete twelve-condition comparisons for C3, ordinary calibration and the six-policy mixture. Table 49 reports calibration-allocation sensitivity separately; those reallocations are dependent.

These results supplement the earlier richer-selector and graded-retrieval comparisons, whose limitations remain. Is there a remaining aspect of uncertainty accounting or comparator matching that you would like us to clarify?

The detailed follow-up explains the calibration procedure, uncertainty accounting, and remaining limitations.

Thank you for your time.

AuthorsOpenReview ↗

A follow-up on stronger whole-query policy selection

Dear Reviewer JbxV,

We have one additional result on your question about how much gain remains under stronger whole-query selection. Using the ten existing matched source fits, we kept the predictors and proposed rankings fixed and gave both arms matched 15-entry policy menus. Each arm selected a mixture under the same loss constraint and used separate guard images to set its execution probabilities. We evaluated the resulting policies on two previously studied COCO panels.

Across four retriever–reranker pairs per panel, six of eight adjusted gain lower bounds were positive. All eight settings met the added-loss and absolute-loss criteria. None had a gain lower bound above the prespecified 0.001 margin. The largest lower bound was for SigLIP2/ALBEF on the 9,999-image panel. There the estimated CYCLE−WHOLE gain was +0.001931 [0.000978, 0.002916], with an added-loss upper bound of +0.000024. The intervals use paired resampling of images and fits, with simultaneous adjustment for four quantities in each of eight settings.

This follow-up measures single-target reciprocal rank. In our separate graded-text evaluation, none of the 12 primary comparisons cleared its practical gain margin. Thank you for suggesting that we examine comparator strength and fitting uncertainty together.

Reviewer_hBHnOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a "Partial Reranking" method based on permutation cycles. Unlike traditional "full-query reranking"—where the entire reranked result is either accepted or rejected—this approach allows for the independent selection of sub-cycles, thereby preserving beneficial ranking adjustments while avoiding potential negative impacts. Evaluations of BLIP and ALBEF models on the Flickr30k, COCO, and VizWiz datasets reveal that, at equivalent levels of "harm cost," partial reranking generally outperforms full-query reranking. On the Flickr30k dataset, employing independent calibration yields improvements in "signed gains" of 0.000930 for BLIP and 0.002255 for ALBEF. The calibration process significantly affects final performance and, under certain configurations, can even lead to a reversal in performance outcomes.

Critical strengths
  1. Utilizing permutation cycles ensures valid partial reranking operations, yielding a natural, mathematically clear action space.

  2. This paper controls for price when comparing performance, isolating sources of performance gain.

  3. Uses bootstrap confidence intervals, multiple comparison corrections, varied grouping controls, and stronger whole-query comparators to prevent misleading baselines.

Critical weaknesses
  1. On Flickr30k, BLIP improves by only 0.000930, failing to statistically exceed the author-defined 0.001 practical margin. While ALBEF reaches this threshold, no evidence shows partial reranking significantly reduces positive loss. Crucially, the paper proves that given ground-truth labels, optimal gains for partial and whole-query actions are identical. Thus, practical value depends entirely on decision quality under uncertain relevance estimates. Current results do not sufficiently prove that this additional decision flexibility reliably translates into meaningful performance gains.

  2. On VizWiz BLIP calibrated gain drops by 0.000692. On Flickr ALBEF, MIN-AREA optimizes predicted binding cost but loses ~0.000369 in real gain compared to SIMILAR. Changing shortlist size from 32 to 8 reverses partial reranking advantages for ALBEF. These phenomena reveal a fundamental issue: despite a larger theoretical decision space, the method lacks reliable mechanisms to convert theoretical flexibility into empirical performance.

Questions for authors
  1. Given that label-aware oracles yield identical optimal gains for partial and whole-query reranking, does the advantage of partial reranking stem from the action space itself or from specific relevance predictor error structures? Can systematically altering posterior calibration quality demonstrate when partial reranking advantages appear and disappear compared to whole-query reranking?

  2. Why does a theoretically larger action space perform worse after independent calibration? Can the authors propose a stable calibration strategy ensuring partial reranking performs no worse than whole-query reranking under reasonable conditions, rather than relying on dataset-specific operating points?

  3. Since risk-sensitive retrieval is the main focus, yet existing calibration fails to guarantee population-level risk, what is the exact risk control advantage over traditional reranking? Can the gain–risk frontiers of both methods be systematically compared at identical empirical loss levels, rather than at single calibrated operating points?

6 comments
AuthorsOpenReview ↗

Response to Reviewer hBHn

Can partial action help with an exact posterior? Yes: an exact posterior and a revealed target define different decisions. Section 3 (p. 3) gives baseline (1,2,3,4), proposal (2,1,4,3), exact posterior (0.20,0.50,0.12,0.18), and price 1. WHOLE has expected gain/loss/utility 0.155/0.110/0.045; applying only the first cycle gives 0.150/0.100/0.050. Utility improves although signed gain falls. Both label-aware oracles attain expected optimal utility 0.265 under this posterior, not for every target.

What does the identity establish? For single-target reciprocal rank, let uc=Ep[gc−λℓc]u_c=\mathbb E_p[g_c-\lambda\ell_c], where p is the target posterior, gcg_c is the signed gain from applying cycle c, ℓc\ell_c is its decrease in reciprocal rank clipped at zero, and λ≥0 is the shared harm price. For cycles C\mathcal C of a fixed proposal, at a common posterior and price,

∑c∈C[uc]+−[∑c∈Cuc]+≥0.\sum_{c\in\mathcal C}[u_c]_+-\left[\sum_{c\in\mathcal C}u_c\right]_+\ge0.

Here [z]₊ denotes max(z,0). This opportunity exists before target revelation and need not arise from prediction error. Prediction and calibration determine its realized effect: the original VizWiz BLIP action term +0.001580 and price term −0.002272 sum to −0.000692. Neither original loss interval establishes lower loss; MIN-AREA and adverse shortlist results remain reported.

Does this opportunity yield a practical gain? A separate auxiliary experiment fixes the original proposal and gives CYCLE and strong WHOLE the same semantic-augmented information and selection protocol. For the mean across five source-data fits, three of four COCO pipelines meet the prespecified joint gain-and-risk criterion with multiplicity-adjusted gain and risk bounds. For SigLIP2/ALBEF, the CYCLE−WHOLE gain difference is +0.003940; its adjusted lower bound 0.003091 exceeds the 0.001 practical margin. The paired added-loss upper bound is 0.000072; CYCLE/WHOLE absolute-loss upper bounds are 0.001569/0.001608, below the 0.002 budget. All three supported pipelines meet the 0.0002 added-loss tolerance. The other COCO pipeline, all four VizWiz pipelines and all four in the separate Flickr30k-derived cohort do not meet the joint criterion (Appendix I.1, Tables 29–31).

Each supported COCO pipeline has five positive paired gain point estimates. SigLIP2/ALBEF has gain-difference run SD 0.000590 and CYCLE/WHOLE MRR@32 0.506158/0.502218; Appendix I.3 reports run and absolute summaries for all twelve conditions.

Does cycle identity matter beyond quantity? At the original QUERY prices, subtracting each rule's own exact quantity-matched random-subset utility leaves a positive adjusted SIMILAR−CROSS contrast in four of six cells; both VizWiz intervals include zero. Matching is within each rule, not across the two rules (Appendix I.2, Table 32). This tests conditional cycle identity, not calibrated-gain superiority.

Does better posterior likelihood increase CYCLE's advantage? Not necessarily. With the native proposal, price 1 and cycle mass fixed, within-cycle temperature 1→2 lowers SigLIP2/ALBEF NLL from 1.377632 to 1.365953. CYCLE intervention-minus-native changes are −0.004168 in gain and −0.009768 in loss, increasing priced utility by 0.005600. Matched QUERY also improves utility, while the CYCLE−QUERY utility gap narrows from 0.001706 to 0.001569 (Table 28, p. 33). This test averages three frozen heads from one designated fitting run. NLL conditions on shortlist-covered targets; gain/loss retain every query. Global temperatures selected on source-development data are all 1. Mass-only interventions change NLL while preserving CYCLE actions.

What do equal-loss frontiers show? The original six hindsight optima at attained loss 0.002 favor CYCLE by 0.000563–0.002562; only Flickr ALBEF's entire adjusted interval exceeds 0.001. Figure 5 (p. 31) separately shows semantic LOCAL's complete nine-budget source-development curve. Matched WHOLE attains higher gain at every evaluated run and budget. Containment guarantees non-strict ordering; strict gaps are empirical. Replaying selected LOCAL policies through WHOLE reproduces their outcomes in all 120 runs. Test differences thus concern selection and calibration within a restricted menu.

What risk protection is established? Appendix H.3 states marginal expected-risk control under conditional exchangeability and independent data roles; WHOLE uses the same rule. This does not guarantee gain noninferiority. The COCO claims use adjusted bounds and an added-loss tolerance; they do not establish harm reduction. Separately, LOCAL mean harm increases in six of seven joint-success conditions. No independent paired gain-and-risk certificate is claimed.

Compare partial application with strong whole-query selection under the same proposal, information and risk definitions. Use separate validation data and prespecified gain and added-loss criteria before freezing the policy for evaluation.

Reviewer_hBHnOpenReview ↗

Follow-up Review

Thank you for the detailed response and additional experiments. I appreciate the theoretical clarification and the new matched comparisons. However, my main concerns remain insufficiently addressed.

First, the practical improvements remain small and inconsistent. The original Flickr30k BLIP gain of 0.000930 does not exceed the prespecified practical margin, and neither model demonstrates a significant reduction in positive loss. While the additional COCO results are encouraging, only 3 of the 12 CYCLE–strong-WHOLE conditions satisfy the joint gain-and-risk criterion, with no successful conditions on Flickr30k or VizWiz.

Second, calibration sensitivity remains a fundamental limitation. The authors explain the observed failures but do not provide a reliable solution. The adverse shortlist and MIN-AREA results remain unresolved, and the additional experiments do not establish consistent advantages under varying calibration conditions.

Third, the distinctive risk-control benefit remains unclear. The marginal expected-risk guarantee applies equally to WHOLE and CYCLE. Furthermore, the equal-loss hindsight results do not demonstrate deployable superiority, while the semantic LOCAL comparisons favor matched WHOLE across all evaluated source-development budgets.

The presentation also requires improvement. Numerical precision and table-caption placement are inconsistent, spacing around some tables is cramped, and several figures have low information density, particularly Figure 3.

The response does not resolve my central concerns about practical significance, robustness, and the method's distinctive contribution. I believe substantial revisions are still necessary, and I therefore maintain my rating of clear rejection.

AuthorsOpenReview ↗

Follow-up on effect size, calibration robustness, and contribution

Thank you for reviewing the additional evidence and clarifying the remaining concerns. We have added the completed analysis to Appendix I.6 (pp. 50–52). Tables 48–50 provide all twelve conditions for C3 and its two reference procedures, with adjusted bounds and a separate allocation-sensitivity analysis.

1. Practical significance and effect size.

We agree that the original BLIP difference of 0.000930 does not exceed the 0.001 practical margin. It is the incremental gain over QUERY, which already applies useful reranking. In the original BLIP same-price diagnostic at QUERY’s price, the target rank changes on only 2.250% of query mass, and beneficial and adverse changes coexist. Frequent ranking differences therefore need not produce a large MRR contrast. This accounting does not change the original practical-margin conclusion.

In the same new ten-fit analysis, ordinary calibration (M0) meets the unchanged joint gain-and-risk criterion in 3/12 conditions, the six-policy mixture (M2) in 2/12, and C3 in 8/12: Flickr30k-derived 3/4, COCO 4/4, and VizWiz 1/4. The earlier results and their original inference families remain unchanged.

Across the eight supported conditions, CYCLE–WHOLE gain differences are 0.003866–0.012385, with every adjusted lower bound above 0.001. The largest added-loss and absolute-loss upper bounds, conservatively rounded upward, are 0.000160 and 0.001884, below the 0.0002 and 0.002 limits. For scale, COCO ALBEF/ALBEF expected MRR@32 is 0.551557 for WHOLE and 0.555423 for CYCLE.

The crossed fit/image bootstrap uses a nominal family-wise error level of 0.05 over 480 coordinates, including all nine procedures and the specified arm-gain contrasts; p. 52 gives the composition and tail allocation. All nine procedures were frozen before this round's target evaluation. These follow-up evaluations reuse previously evaluated cohorts and are not a new independent holdout confirmation.

2. Calibration robustness.

C3 divides 2,500 CAL images into five folds. Each component selects on 500 images at loss cap 0.001516625, averages the three head policies, and applies GUARD on the other 2,000 images. The five completed policies are mixed uniformly. Both arms retain their predictors and proposals and receive the same procedure; the final risk budget remains 0.002.

The SELECT cap accounts for GUARD's finite-sample correction. Cap, allocation and mixture change together, so this experiment does not isolate one factor as the cause.

Across twenty dependent calibration allocations, C3 reduces contrast SD by 29.3–72.0% relative to M0 in every condition. M2 remains less variable. The COCO ALBEF/ALBEF and ALBEF/BLIP-large comparisons have no negative allocation-mean gain contrasts across the twenty allocations, with 19/20 and 14/20 meeting the joint thresholds on point estimates, respectively. These counts are descriptive, not additional confidence-bound successes. COCO SigLIP2/BLIP-large still has six negative allocation-mean gain contrasts. The adverse shortlist and MIN-AREA settings have not yet been retested with C3.

3. Distinctive contribution and risk control.

The risk guarantee is shared by WHOLE and CYCLE. Each component's SELECT set is disjoint from its own GUARD set. Under the existing assumptions, fixed mixing preserves marginal expected-risk control by linearity of action-wise expected loss; independence between completed components is not required.

Our contribution concerns which ranking moves must be accepted together and whether this flexibility provides additional gain under matched predictive information and risk criteria. C3 evaluates executable policies. The earlier equal-loss hindsight results describe structural opportunity. None of the eight supported conditions establishes loss nonincrease, so we claim neither harm reduction nor equal-attained-risk superiority.

The semantic LOCAL extension uses a global menu contained in matched WHOLE's menu. Its unfavorable SOURCE-development comparison remains a limitation of that extension and is distinct from the fixed-proposal CYCLE comparison.

4. Presentation and next steps.

The completed C3 evidence is now documented in Appendix I.6. Main-text integration and Figure 3 revisions remain to be completed. We will improve the comparison summary, numerical presentation, caption placement and spacing, and report any completed additional checks separately, including negative outcomes.

We would appreciate your reassessment of practical significance and calibration robustness in light of these completed comparisons and their stated limits.

Reviewer_6GrHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a reranker's proposed ranking should be applied only in part. The permutation between the baseline top-32 shortlist and the reranked list decomposes into disjoint cycles, and any subset of cycles is still a valid ranking. CYCLE accepts each cycle whose predicted utility (expected reciprocal-rank gain minus a harm price times expected loss) is positive, while QUERY accepts or rejects the whole proposal per query. Both use the same fixed relevance head and proposal, and each calibrates its harm price on held-out images so that a corrected loss stays below 0.002. On a locked Flickr30k-derived gallery with frozen BLIP and ALBEF, separately calibrated CYCLE raises signed reciprocal-rank gain over QUERY by 0.000930 (BLIP) and 0.002255 (ALBEF), without a resolved change in added loss. Secondary analyses cover transport to VizWiz and COCO, regroupings of the same cycles, a binding-cost identity, and a prediction test.

Critical strengths

S1. An interesting idea: merge the baseline and reranked results partially, by cycle. Decomposing the reranker's change into disjoint cycles gives a precise action space between keeping the baseline and accepting the whole proposal, and every subset of cycles remains a valid ranking. Partial reranking by cycle seems to be a very interesting idea. This lets favorable and unfavorable changes be decided separately within one query.

S2. A sound theoretical and evaluation background. The paper states exactly when separate choices can help (the predicted-opportunity identity, Eq. 3), splits the CYCLE − QUERY difference into an action effect and a price effect (Eq. 5), and calibrates each policy with a finite-sample loss criterion (Eq. 4). The main comparison is locked: nothing is selected on the FINAL split, and adjusted bootstrap intervals are reported.

Critical weaknesses

W1. The experimental results are weak. (most concerning)

  • Mainly one dataset: the primary evidence is one Flickr30k-derived gallery.
  • Very small, inconsistent gains: MRR moves from 0.6260 (QUERY) to 0.6269 (CYCLE) for BLIP and from 0.5281 to 0.5303 for ALBEF (L255–257), and BLIP's gain does not clear the paper's own practical margin (Table 1).
  • Unclear generalizability to other tasks: on VizWiz, BLIP's gain is −0.692 (×10⁻³, interval includes zero) and all 36 BLIP rows exceed the 0.002 loss budget (L282–283); on COCO, BLIP's gain of 0.982 misses the 1.0 margin and its added loss is +0.126 [0.034, 0.215] (Table 2). Only ALBEF gains clearly in both.
  • One task: image–text retrieval with a single relevant image per query, scored by its reciprocal rank. CYCLE and QUERY rankings differ on 57.8% (BLIP) and 60.8% (ALBEF) of queries, but the target's rank differs on only 2.25% and 3.49% (L250, L1029–1033), so most changes are invisible to this metric. It would be good to test this on text retrieval where they have several (not just one) graded gold documents (e.g., TREC DL on MS MARCO, BEIR) and the effect on nDCG.
  • Sensitive to the shortlist size: at m = 8, ALBEF's CYCLE − QUERY gain becomes −0.460 (×10⁻³), and at m = 16 the gain rises to 3.206 but CYCLE's loss (2.076) exceeds the 2.0 budget (Table 19). Only ALBEF is tested, so the m = 32 result may not generalize.

W2. The decision model is essentially one more pointwise scorer on top of old models.

  • CYCLE decides each cycle with a fixed small head, a five-input, 16-unit network over ITC and ITM scores and rank fractions (App. B, L751–753). In effect this adds one more pointwise scorer, and the same head also produces the proposal it then judges (App. B). The models are BLIP and ALBEF (2021–2022), the reranker is a small head over their ITM scores, and no stronger cross-encoder or LLM-based listwise reranker is tested.

W3. A key reference point is missing, and replication is limited.

  • No full-proposal result: the paper reports QUERY, CYCLE, and the unchanged baseline, but no numbers for accepting the full proposal (it appears to be the zero-price point of both paths, L252, L1045, and a menu endpoint, L1247–1249). Readers cannot see how much the reranker gains and loses if accepted in full, or how much of that gain CYCLE keeps while avoiding loss.
  • Limited replication: the intervals resample images only, the three head seeds are averaged rather than treated as replications (L191), and the 20 calibration splits are dependent; across them BLIP's gain ranges from 0.79 to 2.00 (×10⁻³, 10th–90th percentile; L450–452). Since the differences are tiny, standard deviations and confidence intervals over independent runs would matter.

W4. The benefit likely depends on the first-stage retriever and the reranker, which are not varied.

  • I suspect partial acceptance helps mainly when the baseline carries signal that the reranker lacks, but here the baseline is the same model's contrastive score and the head already takes it as input (L434, L751), so the two signals are not independent. It remains open whether CYCLE helps with a different or stronger first stage, such as a separate image retriever (e.g., CLIP) here, or BM25 and a lexical-plus-semantic hybrid in text retrieval. Experiments over more diverse combinations of first-stage retrievers and rerankers are needed to show that the method is retriever- and reranker-agnostic.

W5. No open-source code, so the method will be very hard to replicate.

  • The paper releases no code; the reproducibility statement only points to the appendices (Appendices B, D, F). The pipeline depends on many fixed choices, such as the fitted heads, calibration roles, tie rules, and an exposure-filtered gallery (Sec. 4, App. B), which are hard to reproduce from text alone.
Questions for authors
  1. On average, what fraction of the reranker's changes does CYCLE leave at the baseline (preserved), and what fraction does it apply (moved)? L250 gives only how often the CYCLE and QUERY rankings differ. Please report, at the calibrated prices, the number of nontrivial cycles per query, the fraction of cycles CYCLE accepts, the fraction of queries QUERY accepts, and the number of moved candidates.
  2. Please report G, L, and MRR for the full proposal, so that readers can see how much of the reranker's gain CYCLE keeps and how much loss it avoids.
  3. Since the differences are tiny, please report the main comparison over several independent runs (e.g., head seeds) with standard deviations and confidence intervals.

(Suggestions to try:)

  1. Add a text retrieval benchmark with several relevant documents per query (e.g., TREC DL, BEIR) and report nDCG.
  2. Does CYCLE still help with a different or stronger first stage, or with a stronger reranker?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 6GrH

How useful is FULL, and how much does CYCLE retain? FULL, which applies the entire proposed reranking, makes substantial useful changes. For the original BLIP/ALBEF pipelines, BASE MRR is 0.595912/0.468504 and FULL MRR is 0.667478/0.580980. FULL's signed gain/positive loss is 0.071566/0.023508 and 0.112476/0.020089; CYCLE obtains 0.030968/0.001439 and 0.061826/0.001804. CYCLE retains 43.27%/54.97% of FULL's signed gain while avoiding 93.88%/91.02% of its positive loss. Table 2 (p. 6) reports these absolute quantities with all evaluation queries retained.

What fraction of the proposal is applied? QUERY accepts 30.67%/39.03% of image-weighted query mass, whereas CYCLE applies at least one cycle on 65.33%/76.17%. FULL proposes 3.513/3.392 nontrivial cycles per query. CYCLE accepts 31.20%/43.80% of these, measured as the ratio of head-averaged, image-weighted counts, rather than an average of per-query fractions. It moves 10.789/14.363 candidates per query versus FULL's 29.133/29.534, restoring 62.97%/51.37% of FULL-moved positions to baseline. Moved-position counts are distinct from target-rank changes and from the fraction of all 32 positions.

Is the comparison controlled and replicated against a strong selector? The original comparison fixes the relevance head and proposal. CYCLE−QUERY is +0.000930 [0.000662,0.001192] for BLIP and +0.002255 [0.001927,0.002571] for ALBEF; BLIP does not clear the 0.001 practical margin. The same-head gain also remains positive at each of the two shared prices (Table 10, p. 17), so it is not explained solely by different calibrated prices. Against richer, information-matched WHOLE, ten repeated fits and calibration procedures give CYCLE−WHOLE means +0.000768 [−0.000562,0.001723] and −0.001062 [−0.003812,0.001808], with run standard deviations 0.001466 and 0.003444 for BLIP and ALBEF, respectively. Table 23 (p. 26) includes both additional SigLIP conditions. Three nested heads are averaged within each run, rather than counted as independent repetitions.

How broad are the retriever, reranker and metric comparisons? We evaluated Qwen3-VL-Reranker (2B and 8B), two capacities of the same trained reranker family, with SigLIP2 and ALBEF first-stage retrieval on full Flickr30k-derived, COCO and VizWiz cohorts. Original CYCLE passes the joint practical criterion in 1/24 conditions; the distinct LOCAL extension passes in 7/24 (Tables 24–25). WHOLE contains LOCAL's global proposal menu, so these seven gains concern restricted-menu selection and calibration. VizWiz contributes no joint success, and original VizWiz BLIP remains −0.000692 with loss 0.002505.

A separate registered auxiliary comparison freezes the original proposal and gives CYCLE and strong WHOLE the same semantic-augmented information and selection protocol. CYCLE meets the joint criterion in three of four COCO pipelines. The CYCLE−WHOLE gain difference for SigLIP2/ALBEF is +0.003940 [0.003091,0.004783]. The fourth COCO pipeline, all four VizWiz pipelines and the separate four-condition Flickr30k-derived cohort do not meet it (Appendix I.1, Tables 29–31). This identifies a conditional gain beyond the original same-head comparison.

We also completed Hotpot, graded NFCorpus, and lexical/dense retrieval crossed with Qwen/monoT5 reranking on DL19, DL20 and an MSMARCO holdout. The latter uses full-qrels linear nDCG@10, AP@32 and whole-query harm. Its primary CYCLE−WHOLE mean is positive in 5/12 conditions, with 0/12 practical-gain-bound successes (Table 26, p. 32).

What about shorter lists and reproducibility? BLIP's m=8/16/32 gain differences are +0.001477/+0.001328/+0.000930 (Table 27, p. 33). Both models restrict and renormalize the fixed top-32 head; ALBEF's m=8 reversal and m=16 loss excess remain reported. Code is not yet available to reviewers, but we plan to release it to support reproducibility. Appendix H specifies the data roles, tie rules, fitting and calibration procedures, and repeated-run analysis used in the reported experiments.

The completed comparisons quantify the original controlled gains and delimit their practical transfer to richer selectors and graded retrieval.

Dear Reviewer 6GrH,

Following your concern about small and inconsistent gains, we added a matched ten-fit calibration comparison in Appendix I.6 (pp. 50–52). Both CYCLE and strong WHOLE retain their predictors and proposals and receive the same C3 procedure. Under this common analysis, ordinary calibration meets the joint gain-and-risk criterion in 3/12 conditions and C3 in 8/12: 3/4 Flickr30k-derived, 4/4 COCO, and 1/4 VizWiz.

Tables 48 and 50 provide the complete twelve-condition comparisons, including the unsupported conditions, and Table 49 reports the remaining allocation sensitivity. The earlier shortlist and graded-retrieval limitations remain.

Does this comparison address part of your concern about the breadth and consistency of the empirical evidence?

The detailed follow-up explains the calibration procedure, uncertainty accounting, and remaining limitations.

Thank you for your time.

AuthorsOpenReview ↗

A follow-up on shortlist size and the loss budget

Dear Reviewer 6GrH,

We followed up on the gain reversal you identified at eight candidates and the loss-budget excess at sixteen. We repeated the shortlist comparison under C3 for both ALBEF and BLIP.

At eight candidates on Flickr ALBEF, CYCLE’s gain over QUERY was +0.000675 before the guard step and −0.001146 after each arm applied its own retention probabilities. We kept the selected policies fixed and gave both arms the lower retention probability for each component. This yielded +0.000460 [0.000311, 0.000607], below the 0.001 practical margin. Against QUERY at its original probabilities, the modified CYCLE still trailed by 0.002820.

At sixteen candidates, the choice of comparison also mattered. With common retention probabilities, the ALBEF gain was +0.001239 [0.001019, 0.001462], and CYCLE’s absolute-loss upper bound was 0.001966, within the 0.002 budget. Against QUERY at its original probabilities, the same modified CYCLE had a gain difference of −0.002210 [−0.002465, −0.001954].

These bounds adjust for 36 quantities and resample 23,966 evaluation images with the three fitted heads and calibration choices held fixed. With common retention probabilities, the joint criterion held for ALBEF at 16 and 32 candidates and for BLIP at 16. Thank you for drawing attention to both the gain and the loss budget at shorter list sizes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We have uploaded a revised manuscript incorporating the additional analyses and clarifications described in our responses. References to newly added sections, tables, and appendices refer to this revision. We will consolidate further changes arising from the discussion into a subsequent update and clearly summarize those changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.