acceptodds
Under review as a conference paper at ICLR 2027

When Fusion Hurts: Diagnosing and Repairing Recoverable Failures in Heterogeneous Retrieval

Abstract

Heterogeneous retrieval systems combine channels such as lexical search, dense embeddings, visual retrieval, metadata search, and domain-specific indexes. Fusion is intended to preserve complementary evidence, but it can also suppress a relevant item that one specialist channel had already retrieved. We study this recoverable failure mode, which we call fusion hurt: at least one component ranks a relevant item within the target cutoff, but the fused ranking does not. Classical remedies such as tuned RRF, score calibration, quotas, and tree-based learning-to-rank repair some failures, but our analysis shows that they remain limited because they impose global or weakly query-adaptive aggregation policies over uncalibrated, source-specific evidence. We therefore use fusion hurt not only as an evaluation subset, but as supervision for a lightweight post-fusion listwise reranker. The reranker is trained over candidate-union rank, score, source, fusion, and disagreement features, with a query-level loss weight for fusion-hurt training cases; at inference time it uses no qrel-derived feature or fusion-hurt label. Across controlled and public heterogeneous retrieval settings, FH weighting improves the recoverable failure slice over the same plain reranker by +2.8 Recall@10 on VisRAG and +3.1 Recall@10 on MMDocIR, while BEIR text-hybrid validation shows a +3.1 Recall@10 and +5.7 nDCG@10 gain over raw hybrid overall. The experiments also expose where classical fusion remains insufficient and identify boundary cases where strong individual retrievers or tuned fusion already repair much of the failure.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
information retrieval, loss learning, learning to rank, fusion

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_HhJmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how fusion can miss relevant results already found by individual retrievers. It trains a lightweight reranker with higher weights on these failure cases to improve the final ranking.

Critical strengths

S1. The motivation and problem definition are clear.

S2. The evaluation covers both visual-document and text retrieval.

Critical weaknesses

W1. The method has limited technical novelty.

W2. Overall gains over the plain reranker are marginal.

W3. The benefits of fusion-hurt weighting are inconsistent across benchmarks.

W4. Comparisons with other hard-query weighting methods are missing.

W5. The public visual-document tests use very small fusion-hurt subsets.

Questions for authors

Q1. Why does FH weighting perform worse than plain reranking on ViDoRe?

Q2. How does FH weighting compare with other hard-query weighting methods?

Q3. Can you evaluate the method on larger public fusion-hurt test sets?

Reviewer_AL2MOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies when fusing retrieval results loses a relevant document that one retriever ranked in the top k basically when fusing different ranked list hurts. They train a small reranker using ranks, scores, and source information, giving more weight to queries where fusion failed. They evaluate it on text and visual-document retrieval benchmarks.

Critical strengths
  • The problem is practical and useful.

  • The comparison with the same unweighted reranker is a useful control because it separates the effect of the proposed weighting from the benefit of adding a supervised reranker.

  • I also appreciate that the paper reports regressions and settings where the weighting does not help. This gives a clearer picture of the method’s limits.

Critical weaknesses
  • I find the methodological contribution fairly limited. The method uses a standard feature-based reranker and changes the training weight for a particular subset of queries. This is a reasonable approach, but the paper does not yet show why this weighting rule provides an insight beyond existing failure-based reweighting methods.

  • The results do not yet demonstrate a clear practical benefit. On VisRAG, improvements on fusion-hurt queries come with losses on other queries. On MMDocIR and ViDoRe, the plain reranker achieves higher overall recall than the proposed weighted version. The visual-only retriever also achieves higher recall on the fusion-hurt subset in both datasets. Generally the methodological change and the empirical findings offer less new insight than I had hoped for.

  • The baseline comparison is uneven across datasets. The paper includes tuned RRF, score fusion, quota fusion, and learned ranking baselines, but the most complete comparison is on VisRAG.

  • The failure subset depends on the chosen baseline fusion policy. Most experiments define fusion hurt using equal-weight fusion, which can underweight a strong retriever. It is unclear whether the proposed weighting remains useful when the authors define failures using well-tuned fusion. This matters because some of the reported failures may reflect a weak choice of source weights.

  • The training and evaluation setup needs clarification. Outside VisRAG, the paper does not consistently report training and development sizes or clearly explain whether models train separately on each benchmark.

  • Most importantly the related work does not sufficiently connect the contribution to earlier studies of fusion risk, query-dependent fusion, and failure-based reweighting. see the following:

  • Risk-reward trade-offs in rank fusion Benham et al.

  • An analysis of fusion functions for hybrid retrieval Bruch et al.

  • Predicting query performance for fusion-based retrieval Markovits et al.

Questions for authors
  • Do you train a separate reranker for each benchmark or transfer a model trained on VisRAG? If you transfer the model, how do you handle different retrieval channels and score distributions?

  • Does the BEIR experiment use the proposed weighting? The text describes its model as a CE repair model labeled FH-CE, and the table does not include a matched unweighted comparison.

  • please provide statistical testing on which of the improvements are meaningful.

Reviewer_CEuhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on Fusion Hurt: a relevant document that one retriever ranks within the top-k is out of the top-k after fusion. The authors argue that this happens because conventional fusion methods cannot adapt to the strengths of different retrievers based on the query's information. They propose using Fusion Hurt cases as a supervision signal, upweighting the loss of these queries when training a lightweight reranker, so that the reranker learns to trust the more appropriate retriever for each query. The method is evaluated on the Fusion Hurt subset, where it improves Recall@10 by 2.8 and 3.1 on VisRAG and MMDocIR, respectively, while giving no improvement on ViDoRe.

Critical strengths
  1. Fusion Hurt is a problem worth paying attention to: different retrievers have different strengths on different queries, yet fusion can discard a correct result, and the paper defines and analyzes this clearly.
  2. The authors compare against several classical fusion methods, showing how previous methods work and when they fail.
Critical weaknesses
  1. My main concern is novelty. The method looks like loss reweighting on Fusion Hurt samples to learn the fusion, which then yields better recall. I think there is some space to go further, for example by analyzing how different retrievers affect different queries and cause the FH phenomenon instead of just directly learning from the features.
  2. The gains are limited, and they are concentrated on the FH subset.
  3. The validation is insufficient. The FH test subsets of MMDocIR and ViDoRe are small. And BEIR has no comparison against CE. Also, all experiments use in-distribution splits within each dataset, with no ood validation, so it is hard to judge how well the method generalizes.
Questions for authors
  1. I hope the authors can explore how different retrievers affect queries, which would make the paper more convincing.
  2. I hope the authors can add a CE comparison on BEIR, which would make the experiments complete and further demonstrate the method's effectiveness.
  3. Does the method generalize? For example, after training on one dataset, can the reranker transfer to another dataset, and does it still work when the retriever combination or the fusion policy changes? I hope the authors can provide out-of-distribution validation.
Reviewer_PFhwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies "fusion hurt," defined as a query for which at least one component retriever places a relevant item in the top-k but the fused ranking does not. It uses this label to upweight training queries for a lightweight listwise MLP reranker over the candidate union. The reranker uses retrieval traces such as source ranks, scores, presence indicators, fusion features, and disagreement, while the fusion-hurt label is used only during training. Experiments include a controlled VisRAG setting, MMDocIR, ViDoRe, BEIR text-hybrid retrieval, and a small AToMiC diagnostic. The main controlled comparison holds the architecture and features fixed and changes the query loss weight. FH weighting improves the VisRAG failure subset, shows a smaller positive direction on MMDocIR, and does not improve Recall@10 on ViDoRe. The BEIR experiment shows gains from a listwise repair model over raw hybrid retrieval but does not separately isolate the effect of FH weighting.

Critical strengths

1、The paper defines a useful failure case that aggregate retrieval scores can hide. If a component already retrieved a relevant item, the system has an aggregation problem rather than a first-stage recall problem. The source-rescue breakdown makes this actionable by showing which channels supplied evidence that fusion lost. For example, the VisRAG analysis separates BM25-rescued, dense-text-rescued, and dense-image-rescued cases, and shows that the current feature-based model does not improve the last group through FH weighting. This is a useful distinction when deciding whether to adjust fusion, train a repair model, or improve the retrievers themselves.

2、The plain-CE versus FH-weighted-CE comparison is well chosen for testing the proposed supervision. The two models share the candidate union, features, and architecture, so the VisRAG difference is not simply the benefit of adding a neural reranker. On the 426-query FH test subset, Recall@10 rises from 0.8394 to 0.8676 without changing the inference procedure. The paper also reports non-FH regressions and the negative ViDoRe result instead of presenting the weighting as uniformly beneficial. This makes the controlled result a useful starting point for targeted failure repair, even though the broader case needs more evidence.

Critical weaknesses

1、The experiments do not yet show that FH weighting provides a better overall tradeoff than simpler ways of emphasizing difficult queries. On VisRAG, the 2.82-point FH improvement comes with a decline on non-FH queries from 0.5910 to 0.5852, leaving overall Recall@10 almost unchanged and overall nDCG@10 slightly lower. MMDocIR shows a similar tradeoff, and ViDoRe does not show an FH Recall improvement. The much larger gains over tree-LTR are mostly already present in plain CE, so they do not establish the benefit of the new weighting signal. BEIR also lacks a matched plain-CE versus FH-weighted-CE comparison. Since Equation 3 is query-level reweighting, comparisons with loss-based hard-query weighting, failure oversampling, or weighting failures of a tuned fusion baseline are needed to show why this particular failure label is worth prioritizing. Without them, the result is mainly a redistribution of errors toward a selected subset.

2、The public validation and its statistical reporting are not strong enough to support the claimed extension of the weighting effect. MMDocIR has only 26 held-out FH queries, and the paper acknowledges that its query-level confidence interval crosses zero. It nevertheless emphasizes significance from seed-query tests, even though repeated seeds for the same query are dependent. There is also a numerical inconsistency: Figure 2 and the text report four rescues and zero regressions across the matched MMDocIR seed-query pairs. Those counts give a two-sided exact McNemar p-value of 0.125, not the 0.0391 reported in Section 6.4. The actual contingency table and testing unit need to be clarified. In addition, the quantities labeled nDCG@10 for MMDocIR and ViDoRe are best-relevant-rank proxies rather than standard multi-positive nDCG. These issues matter because the public results are the main evidence that the weighting benefit extends beyond the controlled VisRAG experiment.

Questions for authors

1、What are the exact discordant counts behind the MMDocIR McNemar test? How do the reported four rescues and zero regressions produce p = 0.0391? Please also give query-clustered intervals that account for repeated training seeds.

2、How does FH weighting compare with equally tuned hard-query weighting or oversampling, and with defining FH relative to the dev-tuned fusion baseline rather than equal WRRF? Is there a better FH/non-FH tradeoff at matched overall retrieval quality?

3、Is the BEIR row labeled FH-CE actually trained with the FH weight in Equation 3? Please provide the matched plain-CE and FH-weighted-CE results, per-dataset results, and the separation between the training, tuning, and reported development queries.

4、Can MMDocIR and ViDoRe results be recomputed using the full relevance lists and standard nDCG@10? If those artifacts are unavailable, please rename the reported measure and explain whether the conclusions change under standard Recall@10 for queries with multiple relevant items.

5、What are the exact component checkpoints, split construction, feature sets, and tuning budgets in the controlled VisRAG experiment? The large gap between the MLP and both tree-LTR families is important to the comparison, but the current description does not give enough detail to understand it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.