- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to apply a reranker proposal after a top-32 permutation has already been produced. QUERY accepts or rejects the full proposal. CYCLE decomposes the permutation into disjoint cycles and decides whether to apply each cycle separately. The experiments use single-target reciprocal-rank gain and loss. Validation includes a locked Flickr setting, same-price decomposition, whole-query controls, calibration sensitivity, external transports, shortlist stress tests, and serving-time measurements.
Critical strengths
-
The paper provides a clean controlled analysis of action structure. CYCLE and QUERY use the same predicted relevance and proposed ranking. The same-price decomposition shows positive conditional action contrasts in the locked Flickr setting, so the result is not explained only by different calibrated prices.
-
The experimental and statistical reporting is unusually careful. The paper includes same-proposal whole-query gates, finite calibration-optimal menus, calibration-role sensitivity, external reversals, shortlist failures, and timing results. It clearly states that its intervals condition on the fixed pipeline and that partial action does not provide model-inference speedup.
Critical weaknesses
-
The locked primary calibration rule does not directly optimize gain under the risk constraint. It selects the most permissive feasible price. The paper later performs SELECT/Guard optimization for CYCLE and a finite WHOLE menu, which partially addresses this issue, but that evidence is an exposed secondary analysis. The WHOLE menu is also finite, and VizWiz BLIP shows a clear reversal.
-
The uncertainty analysis and generality remain conditional. The main intervals do not resample fitted heads, calibration composition, or operating-point selection. The main theoretical and empirical conclusions also use single-target reciprocal-rank utility and a finite whole-query comparator family. These limits are clearly disclosed, but they restrict how broadly the result can be applied.
Questions for authors
- Uncertainty clarification: How do the gain and loss intervals change when the full pipeline is repeated, including head fitting, calibration composition, and operating-point selection?
- Comparator clarification: Can a higher-capacity whole-query selector close the gap while using exactly the same proposer, inputs, and out-of-sample selection protocol?
- Generality clarification: Does cycle-level action retain an advantage with multiple relevant items and metrics such as NDCG or AP?
3 comments
Dear Reviewer JbxV,
Following your questions about repeated fitting and comparator strength, we added a common ten-fit comparison in Appendix I.6 (pp. 50–52). The same C3 calibration procedure is applied to CYCLE and strong WHOLE with their predictors and proposals unchanged. Under this common analysis, ordinary calibration meets the joint gain-and-risk criterion in 3/12 conditions and C3 in 8/12.
Tables 48 and 50 provide the complete twelve-condition comparisons for C3, ordinary calibration and the six-policy mixture. Table 49 reports calibration-allocation sensitivity separately; those reallocations are dependent.
These results supplement the earlier richer-selector and graded-retrieval comparisons, whose limitations remain. Is there a remaining aspect of uncertainty accounting or comparator matching that you would like us to clarify?
The detailed follow-up explains the calibration procedure, uncertainty accounting, and remaining limitations.
Thank you for your time.
A follow-up on stronger whole-query policy selection
Dear Reviewer JbxV,
We have one additional result on your question about how much gain remains under stronger whole-query selection. Using the ten existing matched source fits, we kept the predictors and proposed rankings fixed and gave both arms matched 15-entry policy menus. Each arm selected a mixture under the same loss constraint and used separate guard images to set its execution probabilities. We evaluated the resulting policies on two previously studied COCO panels.
Across four retriever–reranker pairs per panel, six of eight adjusted gain lower bounds were positive. All eight settings met the added-loss and absolute-loss criteria. None had a gain lower bound above the prespecified 0.001 margin. The largest lower bound was for SigLIP2/ALBEF on the 9,999-image panel. There the estimated CYCLE−WHOLE gain was +0.001931 [0.000978, 0.002916], with an added-loss upper bound of +0.000024. The intervals use paired resampling of images and fits, with simultaneous adjustment for four quantities in each of eight settings.
This follow-up measures single-target reciprocal rank. In our separate graded-text evaluation, none of the 12 primary comparisons cleared its practical gain margin. Thank you for suggesting that we examine comparator strength and fitting uncertainty together.
Response to Reviewer JbxV
What changes when fitting and calibration are repeated? Ten repeated fits materially change the ALBEF conclusion. Against richer WHOLE, BLIP's mean gain difference is +0.000768 [−0.000562,0.001723], with run SD 0.001466; ALBEF's is −0.001062 [−0.003812,0.001808], with SD 0.003444. For the designated fixed run of this same CYCLE−richer-WHOLE comparison, the adjusted gain-difference bounds are [0.000513,0.000900] and [0.002510,0.002897]. Added-positive-loss bounds change from [−0.000072,0.000060] to [−0.000057,0.000076] for BLIP, and from [−0.000018,0.000089] to [−0.000162,0.000013] for ALBEF, from the fixed run to the ten-run mean. These bounds are adjusted across the study's gain and risk comparisons. Repetition changes both effects and uncertainty. Table 23 (p. 26) reports gain, run SD and both additional SigLIP conditions.
What is repeated, and what information does WHOLE receive? Each run repeats source-data folds, posterior fitting, proposal-specific WHOLE fitting, source-model selection, calibration-split allocation and operating-point selection. Pretrained checkpoints and test images remain fixed. Three nested heads are averaged within each run; crossed run/image intervals reflect this scope. The richer selector receives the same posterior, proposal and public features and includes learned nonlinear gates. This is separate from the original same-head QUERY comparison.
A separate registered auxiliary experiment keeps the original proposal fixed while both CYCLE and strong WHOLE receive the same semantic-augmented information. Across five source-data fits, three of four COCO conditions meet the joint criterion. The CYCLE−WHOLE gain difference for SigLIP2/ALBEF is +0.003940 [0.003091,0.004783]. None of the four conditions in the Flickr30k-derived cohort or the four VizWiz conditions meets it (Appendix I.1, Tables 29–31). The external bounds retain the 64-coordinate family, separate from the Flickr30k-derived cohort's 32-coordinate family. These cycle decisions on a fixed proposal are distinct from LOCAL's global menu below.
Does WHOLE have the same policy opportunities as LOCAL? Yes. Matched WHOLE contains every global policy available to LOCAL and uses the same selection objective, loss constraint, head averaging and risk calibration. Its menu adds QUERY, ridge, nonlinear MLP and margin gates to LOCAL's BASE and eleven proposals, with the same two-vertex mixtures. Numerical replay confirms that an identical selected policy gives identical outcomes under either implementation. WHOLE's ordering on the selection split therefore follows from containment, rather than demonstrating better generalization (Appendix H.2).
With the Qwen3-VL rerankers, LOCAL has 21/24 positive means and 7/24 joint practical successes, while original CYCLE has 1/24 joint success (Tables 24–25, pp. 28–29). At common retention, LOCAL has lower mean gain in all 24 conditions and lower harm in 23. Common retention is not common loss: Figure 5 (p. 31) gives the complete source-development curve across nine loss budgets. The retention identity in Appendix H.3 explains the accounting without assigning a causal overfitting or calibration penalty.
Does the result extend to graded retrieval? The completed Hotpot, NFCorpus and lexical/dense × Qwen/monoT5 comparisons use actual query-level harm. The last study has positive primary CYCLE−WHOLE means in 5/12 conditions and no practical-bound success. BM25+Qwen on MSMARCO is −0.022911 [−0.027414,−0.018231], negative in all three fits (Table 26, p. 32). Single-target additivity is not transferred to graded losses.
How are selection and evaluation separated? Model fitting uses source data; policy selection uses a separate split. The mixture freezes before risk calibration on an independent guard split and final test evaluation. The original feasible-price rule and later gain-maximizing selection remain distinct. Earlier benchmark summaries informed the follow-up questions. The completed comparisons preserve the controlled original effect while showing how stronger selectors, repeated fitting and graded retrieval limit its practical generality.