acceptodds
Under review as a conference paper at ICLR 2027

Better in Hindsight: Why Oracle Fusion Gains Are Hard to Learn and Where Evidence Helps

Abstract

Retrieval systems often fuse a fast retriever's score with a stronger verifier's score, and the fusion weight that ranks the correct item first varies from query to query. In hindsight, choosing the best of 21 weights for each CIRR query adds 10.2–12.2 R\@1 points over a tuned constant weight, whereas a ridge policy that predicts the weight from score statistics gains 0.32 points. We explain this gap with two exact identities. For R\@1, the successful weights of a query form an interval that can be computed from its scores, so headroom is exactly the share of queries whose interval excludes the constant. Every policy's net gain is its rescued failures minus its harmed successes. Across 21 vision–language verifier configurations on CIRR and FashionIQ and five BEIR text datasets, learned policies gain through caution: they rarely harm solved queries and seldom rescue queries whose successful weights lie far from the constant. The bottleneck is knowing which candidate is the target. A label-derived hint lets the same procedure recover a median 89% of headroom, competitor-aware features stop helping under an equal tuning budget, and only decoders trained to identify the target beat the tuned baseline. Acting on this diagnosis, the winner-path duel asks a second verifier about only the candidates that some weight can rank first, 1.6–2.0 calls per query. Under protocols fixed in advance, it gains 2.17 and 1.16 R\@1 points on new CIRR and FashionIQ queries, and 3.99 and 1.85 points (36% and 22% of headroom) when a larger verifier scores only those candidates. We recommend reporting reachable headroom, rescue and harm, and comparing adaptive policies under matched tuning budgets. 中文

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Composed Image Retrieval, Score Fusion, Reranking, Vision-Language Models, Oracle Headroom, Algorithm Selection, Cascaded Reranking, Evaluation Methodology

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_QW35OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why, in "retrieve-and-rerank" systems, it is difficult to achieve significant per-query performance gains through score fusion, even when employing learned fusion weight strategies. Specifically, we conduct a precise contribution analysis regarding the R@1 metric by leveraging the valid range of fusion weights and the "gain-minus-loss" identity. Furthermore, we evaluate various learning strategies across image and text retrieval tasks and propose a novel method called "Winner-Path Duel" (WPD); this approach assigns a score from the second-stage ranker to a candidate result only when that result emerges as the "winner" under a specific fusion weight configuration.

Critical strengths
  1. Highly efficient and stable two-stage verification (WPD): Going away from global computation, WPD verifies candidates secondarily (only a small proportion of candidates that are more likely to rank first) (only 1.6–2.0 calls per query), it still switches results only when the model is quite confident, avoiding the incorrect rejection of correct results.
  2. Breaking the bottleneck of static strategies: Unlike the conservative and fixed strategies, WPD well exploits target information, and unlocking 22% (CIRR) and 15% (FashionIQ) of the potential performance headroom on the development set. Superior gains in generalization and scalability: On unseen test data, R@1 generally improves 1.16–2.17 percentage points.
  3. By using the computational power of bigger models strategically, focusing it precisely where it is needed (evaluating only candidate paths), performance gains increase to 1.85–3.99 percentage points, and up to 36% of the theoretical potential performance is recovered.
Critical weaknesses
  1. The visual style of the Figures in the paper leans too heavily towards AI generation; many of the tables are misaligned, and there is a noticeable misalignment in Part 2 of Figure 6. This raises doubts about the authenticity of the experiments. Perhaps the authors could adjust the figures using the actual experimental data.
  2. This paper current experimental comparison concentrates only on the variants from author’s proposal, such as ridge regression, candidate posterior, binding, neural set decoder, WPD, etc. So the core proposition ‘only decoders trained to find the target beat the tuned baseline’ is valid for specific method families and unable to draw a universal conclusion about adaptive fusion. Maybe the author can compare typical realizations of uncertainty gate, query router, generic cascade, pair reranker under the same conditions for more reliable conclusion.
  3. The evidence on WPD is convincing; but the actual reach of applicability for claims in the paper about its efficiency and robustness is not as wide as explained in the paper. The number of “1.6–2.0 calls per query”, measured by the number of candidate evaluations by the second verifier, is indeed convincing, but it does not readily reduce to end-to-end latency, FLOPs, memory cost, batching performance, cost of VLMs for different image resolution and tokenization sizes. Authors may consider adding an experiment specifically on efficiency.
Questions for authors

The specific issues have been raised in the section Critical Weaknesses.

1 comment

Dear Reviewer QW35,

Thank you again for your careful review. Since our reply above, we have made several revisions to make the experimental results and comparisons easier to verify. Here is a one-glance map of the changes (all marked with pink bars in the revised PDF):

Figures and tables → every figure was regenerated directly from the saved experiment outputs using a unified plotting pipeline, with an independent script verifying the source values (Appendix Q, pp. 54–55); the reported values remain unchanged.

Comparisons beyond our variants → new Table 1 (p. 10) adds uncertainty gates, query routers, generic cascades, and pair rerankers under the same queries, folds, seeds, and constant (Appendices M and O).

Measured cost → we now report end-to-end latency, FLOPs, and memory across batch sizes, image sizes, and caption lengths (Section 7, p. 9; Appendix P): 29–46% of latency, 26–47% of FLOPs, and 2.7–5.2 GB additional memory.

Regarding your first point, we also strengthened the reproducibility checks behind the figures. All plotted values are read directly from saved experiment outputs, and each figure is associated with its source file and hash; an independent checker additionally verifies the reported values. We would also be happy to provide the corresponding code and data during the discussion period if that would be helpful.

If these changes address your concerns, we would be grateful if you would reconsider your score. If any point remains unclear or insufficiently addressed, we would greatly appreciate your guidance.

Thank you again.

Sincerely, The authors

Reviewer_QWEFOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper explains the oracle gap for a retrieval system using two quantities: oracle headroom and policy’s net gain. Headroom is defined as the exact share of queries whose interval of successful weights excludes the tuned constant, and every policy’s net gain is defined as its rescued failures minus its harmed successes. Their winner-path duel (WPD) gains 2.17 and 1.16 R@1 points on new CIRR and FashionIQ queries, and 3.99 and 1.85 points (36% and 22% of headroom) when a larger verifier scores only the candidates that some weight can rank first.

Critical strengths

This paper introduces winner-path duel (WPD), a fusion-weight policy that consults a second verifier about only the candidates that some weight can rank first. It recovers 22% of CIRR and 15% of FashionIQ headroom, outperforming any fixed-information policy. Moreover, this paper performs controlled comparisons across 21 verifier configurations on CIRR and FashionIQ and five text datasets and demonstrates that learned gains are cautious and that only target identification beats equal-budget tuning.

Critical weaknesses
  1. This paper would benefit from a clearer summary of its contribution and a sequential flow of presentation. The introduction has a strange structure where several terms are used before introducing them properly. Even in Section 2. Problem Setup, the systems in consideration are not clearly described. The paper introduces two useful identities: headroom and policy’s net gain; however, how those identities motivate the winner-path duel (WPD) policy needs further explanation.

  2. The exact problem statement or hypothesis of the paper is not clearly stated.

  3. The paper also describes WPD in a later section, though this is the method this paper ultimately adopts. Introducing WPD earlier will help to follow the paper easily.

  4. The discussion on related works and the significance/novelty of this paper over related works is not discussed in detail.

  5. Could you discuss why these three policies in particular? A comparison with other policies will further help understand WPD’s strengths and limitations.

  6. Lack of sufficient baselines: Is it possible to compare/contrast the results with some of the related works on this topic?

Questions for authors
  1. Could you discuss the exact problem statement/core hypothesis of the paper and how the experiments prove it?

  2. See other weaknesses.

3 comments

Thank you again for your careful review. To make it as easy as possible for you to check each of your points against the revised paper, here is a one-glance map (all changes are marked with green bars in the right margin of the revised PDF):

Clearer summary and sequential flow → new "Contributions and outline" paragraph (p. 2); the system is now three numbered stages (p. 2). Exact problem statement and hypothesis → new "Problem" and "Hypothesis" paragraphs (pp. 1–2). WPD introduced too late → WPD is now described in Section 1 (p. 2) and defined in Section 3 (p. 4). Related work and novelty → Section 8 rewritten, with a new "What is new" paragraph (p. 9). Why these three policies → Section 2 now states what each policy asks (p. 3). Baselines → new Table 1 compares 17 methods, including gates, routers, cascades and pair rerankers (p. 10). Q1. Problem, hypothesis, and how the experiments test it → Section 1 states both; Sections 5–7 test four consequences of the hypothesis (pp. 6–8). If these changes now address your concerns, we would be grateful if you would reconsider your score. If any point still reads unclear, please tell us which, and we will fix it during the discussion.

Thank you again.

Sincerely, The authors

Reviewer_QWEFOpenReview ↗

Updated my rating!

Thanks for the rebuttal.

AuthorsOpenReview ↗

Thank you!

Dear reviewer So many thanks for you kind review! Thank you!

Reviewer_a26fOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies why choosing a fusion weight separately for each query often gives much smaller gains than an oracle suggests. It fixes the retriever, candidate list, and verifier scores, and characterizes the weights that rank the target first as a feasible interval. It then separates a policy's gains into rescued failures and harmed successes. Experiments cover 21 vision-language verifier configurations on CIRR and FashionIQ, additional query populations, CIRCO, and five BEIR datasets. The authors compare score-based policies, candidate-level predictors, target-information probes, and competition-aware features under matched tuning budgets. They find that the tested policies recover little of the oracle headroom, while several apparent feature gains disappear after stronger baseline tuning. Based on this analysis, they introduce winner-path duel, which asks a second verifier to score the few candidates that can become the top result under some fusion weight. This gives larger gains with relatively few additional verifier calls, including on the separate confirmation populations.

Critical strengths

1、The paper takes a useful question and studies it with unusually careful controls. An oracle improvement can make adaptive fusion look promising without showing that the correct decisions can actually be learned. The feasible-interval analysis and rescue-versus-harm accounting make this distinction concrete. More importantly, the authors check calibration, tuning budgets, target-group splits, and different prediction objectives instead of attributing every improvement to a new feature. For example, the binding-feature gain on CIRR falls from more than one point with a fixed decoder to about 0.14 points under matched tuning. This is a useful finding for work on adaptive retrieval and routing, where a weakly tuned baseline can otherwise make a method look more effective than it is.

2、The evidence-acquisition experiments give the analysis a practical outcome. The same duel judge gains much less without a second opinion, while adding a stronger verifier on the winner path gives clear improvements on new query populations. The prospective path-only experiment reports gains of 3.99 R@1 points on CIRR and 1.85 on FashionIQ, with about 1.6 and 2.0 second-verifier calls per query. I also appreciate that the paper reports failed predictions, the weaker FashionIQ comparison against the candidate policy, and cases where a larger verifier alone is better. These qualifications make the empirical message more convincing: selective extra evidence helps in the tested settings, but the benefit depends on the second verifier.

Critical weaknesses

1、The conclusion that the remaining gap is an information gap is stronger than the experiments establish. Equation 4 explicitly separates missing information from the fitted policy's failure to use available information, and the paper notes that these terms cannot be measured separately. Giving the learner a label-derived successful weight, or mixing the true target into its posterior, shows that it can use a near-answer hint. It does not show whether the original features contain information that the tested policies fail to use because of limited training data, the loss, or the model class. In fact, target-prediction objectives improve performance using the same inputs. The experiments support a limitation of the tested score-based learning procedures, but they do not yet identify an irreducible information limit. This matters because explaining the oracle-to-learned gap is the paper's main scientific claim.

2、The method-specific advantage of winner-path duel is less clear than the broader benefit of obtaining another opinion. Section 7 reports that spending the same calls on the constant ranking's top candidates gives nearly the same result, so the winner path itself has not shown a clear accuracy advantage over a simpler shortlist cascade. The practical comparison also relies mainly on verifier calls and, in Appendix L, a calls-times-parameter-count proxy. That is informative, but it does not establish the actual latency or memory tradeoff after paying for the primary verifier, maintaining two models, and handling small variable batches. I would therefore give more weight to the diagnostic findings than to a practical efficiency advantage for the proposed selection rule.

Questions for authors

1、Equation 4 says the information gap and fitted-policy gap cannot be measured separately. What evidence supports the stronger statement in the conclusion that "the gap is an information gap"? Would you narrow this to the tested feature sets and learning procedures, or can training-size and capacity analyses separate these explanations more clearly?

2、The budget-matched top-m duel performs about as well as winner-path duel on the development data. Does that comparison also hold on the confirmation populations and in the prospective larger-verifier experiment? What benefit of the winner-path rule remains beyond setting a small candidate budget?

3、What are the measured end-to-end latency, throughput, and memory costs of a small primary plus winner-path duel compared with the larger verifier alone? The comparison should include the primary verifier's 20 evaluations and the additional stage.

4、How sensitive are the small target-posterior gains and the larger duel gains to complete retraining on different training samples? Appendix D already reports refit variability for selected original policies; comparable results for the final duel would clarify whether its gain is similarly stable.

Reviewer_gwz7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates the problem of score fusion in two-stage retrieval—specifically, how to dynamically adjust the fusion weights between a retriever and a verifier for individual queries. The authors observe that while an oracle can achieve significant performance gains by selecting optimal weights for each query, strategies learned in practice often yield only marginal improvements. To explain this, this paper analyzes the theoretical upper bound of query-specific fusion and examine the performance of various strategies using metrics such as Rescue and Harm. It shows that the core challenge lies in accurately distinguishing the target candidate from others. It then proposes WPD, which employs a second verifier to double-check the competing candidates. Experimental results show that WPD achieves performance gains.

Critical strengths
  1. I think the theoretical analysis of this paper is strong. Rather than merely proposing a new fusion method, it investigates why the gains achievable by an oracle are difficult to realize through practical learning. In particular, the analysis of "Headroom", "Rescue", and "Harm" is insightful and helps clarify the limitations of existing adaptive fusion approaches.

  2. The experimental evaluation is comprehensive. The authors test their method across multiple datasets and various verifier configurations. They also analyze the true source of performance gains by controlling the tuning budget. The proposed WPD method is directly grounded in the preceding analysis and yields strong results on new query sets.

Critical weaknesses
  1. The scope is limited to a fixed retriever, a top-20 candidate list, and linear score fusion. Although the experiments cover both image and text retrieval, it is not clear whether these findings generalize to more complex retrieval pipelines, for example, those involving non-linear fusion, dynamic candidate lists, or end-to-end optimized retrieval systems.

  2. WPD relies heavily on the second verifier. The experimental results suggest that the final performance is highly sensitive to whether this second verifier can provide genuinely new, useful information.

  3. The paper focuses primarily on the number of model calls. I am not sure whether it should also analyze actual inference latency and computational costs to clarify whether it is sufficiently efficient for real-world deployment.

Questions for authors
  1. Beyond the number of verifier calls, have the authors evaluated the actual inference latency and computational overhead of WPD? How does its end-to-end efficiency compare to simply using a more powerful verifier directly?

  2. Does WPD require retraining the decision-maker or adjusting thresholds when applied to different datasets? Can it be directly transferred to unseen retrieval tasks?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.