- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
WBFusion formulates rank fusion as a restricted-support Wasserstein barycenter problem, incorporating inter-document geometry into support selection and ranking. The paper establishes monotonicity and restricted weak submodularity, then learns a dot-product surrogate for ANN-assisted greedy selection, followed by supervised reranking. Experiments on five BEIR datasets report improved ranking quality, component ablations, and speedups over exact OT-based greedy selection.
Critical strengths
-
Casting fusion as support-constrained barycenter optimization provides an explicit way to incorporate document geometry. The weak-submodularity analysis gives a meaningful foundation for greedy support selection.
-
The method improves reported ranking performance across five datasets. Comparisons with no KL regularization and an RRF-based KL target help isolate the contribution of barycentric regularization within the same reranking architecture.
Critical weaknesses
- The approximation guarantee does not cover the implemented pipeline. Theorem 6 assumes uniformly accurate barycenter predictions and small cost oscillation, without establishing these conditions experimentally or bounding the required feature dimension. Theorem 7 further assumes exact maximum-inner-product retrieval and does not analyze the subsequent -to- filtering. Consequently, the stated end-to-end guarantee does not establish the accuracy of the practical ANN procedure
Questions for authors
- Which guarantee applies to Algorithm 4 with approximate ANN search and subsequent support filtering? How large are the surrogate and search errors under the experimental settings?
- What ranking quality and end-to-end latency does the complete ANN pipeline achieve on full corpora, including candidate generation and refinement, compared with standard fusion pipelines?
6 comments
Response from Authors Part 2
Theorem 6 assumes uniformly accurate barycenter predictions and small cost oscillation, without establishing these conditions experimentally or bounding the required feature dimension.
We thank the reviewer for asking us to clarify these assumptions.
Small cost oscillation. To get an empirical estimate of cost oscillation in the data, we look at the spread of the cost matrix entries around their mean in terms of standard deviation and what percentage of the pairwise cost entries lies within a certain spread from the mean. We also report the absolute cost oscillation as .
| Dataset | std/mean | mean 5% | mean 10% | mean 15% | |
|---|---|---|---|---|---|
| MS-Marco | 0.039 | 83.2 | 98.2 | 99.6 | 0.001 |
| NF-Corpus | 0.088 | 42.7 | 75.5 | 92.5 | 0.001 |
| Hotpot-QA | 0.048 | 74.0 | 96.3 | 99.2 | 0.001 |
| Fever | 0.048 | 73.9 | 96.2 | 99.2 | 0.001 |
| SciFact | 0.077 | 46.4 | 82.0 | 96.0 | 0.252 |
Since most of the cost matrix entries are concentrated around their mean in all the datasets, the assumption of small cost oscillation is empirically validated.
Barycenter-prediction accuracy. Theorem 6 assumes the Barycenter to be well approximated by a contextual softmax model having a specific parameterization. For each of the 5 datasets, we fit a contextual softmax model to the barycenter distributions. We report the per query mean KL divergence and TV distance of the true barycenter from the fitted softmax model.
| Metric | MS-Marco | NF-Corpus | Hotpot-QA | Fever | SciFact |
|---|---|---|---|---|---|
| KL divergence | 0.070 | 0.003 | 0.106 | 0.020 | 0.012 |
| TV | 0.144 | 0.027 | 0.168 | 0.074 | 0.051 |
The goodness of fit indicates the plausibility of our softmax approximation assumption on the Barycenter.
Bounding feature dimension. The theorem actually comes up with a sufficient bound on the feature dimension through its proof. Note that we have an assumption of the barycenter being approximated by a softmax distribution, which we just empirically validated. If the feature dimension in that softmax approximator is , then our current proof naturally comes up with a sufficient feature dimension of for the dot product surrogate to achieve approximation of . But it should be noted that this is simply a theoretical upper bound, in practice we use neural features of dimension .
Which guarantee applies to Algorithm 4 with approximate ANN search and subsequent support filtering?
In the above responses, we have discussed how the newly added theorem 8 (along with theorem 7) can give the guarantees for the filtering with surrogate, then exact greedy approach. For the ANN search, the only thing that changes is that, we retrieve the top 1 for each greedy step by probing the index (that we build using the trained surrogate) instead of exhaustive search. So if one assumes exact NN search, the guarantee applies verbatim.
Response from Authors Part 3
How large are the surrogate and search errors under the experimental settings?
Our final efficient search method admits two approximations: (1) Marginal gain to dot product approximation and (2) approximate nearest neighbor search from index built on the corpus vectors obtained in the dot product.
First, we isolate the effect of marginal gain to dot product approximation from ANN search, by performing exact NN search (compute exact minimum over distances) over the dot products. We obtain and compare the corresponding performance given by obtained from exact greedy. Given a metric M (M can be NDCG and MAP), we report (a) Err(M) as (lower is better), (b) spearman rank correlation between and (higher is better) and Kendall Tau between and (higher is better).
| Metric | MS-Marco | NF-Corpus | Hotpot-QA | Fever | SciFact |
|---|---|---|---|---|---|
| NDCG@10: relative error (%) | 5.96 | 1.41 | 2.12 | 0.96 | 0.77 |
| NDCG@10: Spearman rank correlation | 0.976 | 0.985 | 0.931 | 0.952 | 0.963 |
| NDCG@10: Kendall Tau | 0.879 | 0.930 | 0.898 | 0.935 | 0.954 |
| MAP@10: relative error (%) | 6.09 | 1.95 | 2.21 | 1.05 | 0.74 |
| MAP@10: Spearman rank correlation | 0.979 | 0.977 | 0.934 | 0.952 | 0.963 |
| MAP@10: Kendall Tau | 0.885 | 0.930 | 0.904 | 0.937 | 0.954 |
We observe that the relative error is very low and rank correlation metrics are very high. Therefore, items retrieved by maximizing exact marginal gains and exact NN search is close. Such high accuracy is possible because the dot product is only used to order the items. Hence, as long as the marginal gains and the dot products have similar order, our method performs well, even if dot products do not mimic the scores.
Next, we compare performance of (1) obtained by ANN search of the index and (2) corresponding performance given by obtained from exact NN. Following table shows the results, which show that ANN search well approximates exact NN search.
| Metric | MS-Marco | NF-Corpus | Hotpot-QA | Fever | SciFact |
|---|---|---|---|---|---|
| NDCG@10: relative error (%) | 1.28 | 3.74 | 1.69 | 3.46 | 3.06 |
| NDCG@10: Spearman rank correlation | 0.941 | 0.947 | 0.819 | 0.940 | 0.958 |
| NDCG@10: Kendall Tau | 0.884 | 0.915 | 0.818 | 0.935 | 0.957 |
| MAP@10: relative error (%) | 4.51 | 5.58 | 1.23 | 3.30 | 2.98 |
| MAP@10: Spearman rank correlation | 0.893 | 0.941 | 0.819 | 0.939 | 0.958 |
| MAP@10: Kendall Tau | 0.846 | 0.917 | 0.818 | 0.935 | 0.957 |
What ranking quality and end-to-end latency does the complete ANN pipeline achieve on full corpora, including candidate generation and refinement, compared with standard fusion pipelines?
We report the end-to-end latency of the Fusion Baselines and compare it with WBFusion ANN pipeline. Note that search in the entire corpus is takes prohibitively large time, which is why we had first shortlisted base rankers top-100 items. Following is the time taken (in milliseconds) for retrieval averaged per-query. They show our method is fastest.
| Method | MS-Marco | NF-Corpus | Hotpot-QA | Fever | SciFact |
|---|---|---|---|---|---|
| RRF | 6384 | 1923 | 5117 | 4847 | 1660 |
| CombSUM | 6101 | 1932 | 5088 | 4817 | 1634 |
| CombMNZ | 6026 | 1930 | 4845 | 4707 | 1664 |
| MoE | 6020 | 1940 | 4814 | 4664 | 1697 |
| Ensemble | 6167 | 1929 | 4997 | 5806 | 1659 |
| NeuralRRF | 6040 | 1956 | 5124 | 5685 | 1631 |
| Hybrid | 6089 | 1960 | 4930 | 5495 | 1675 |
| WBFusion ANN | 4888 | 1057 | 3013 | 3128 | 976 |
Suppose instead of initial shortlisting from base rankers, we use the entire corpus for retrieval. We now run WBFusion ANN on the index built on the entire corpus, which is quite fast. Then the latency (in seconds per query) numbers are given below.
| Method | MS-Marco | NF-Corpus | Hotpot-QA | Fever | SciFact |
|---|---|---|---|---|---|
| RRF | 8736.1 | 2.7 | 4064.4 | 4386.2 | 5.1 |
| CombSUM | 8631.9 | 2.5 | 4120.1 | 4315.4 | 5.4 |
| CombMNZ | 8712.6 | 2.6 | 4078.6 | 4371.6 | 5.5 |
| MoE | 8817.3 | 2.8 | 4166.5 | 4356.7 | 5.2 |
| Ensemble | 8661.4 | 2.4 | 4115.3 | 4409.9 | 5.3 |
| NeuralRRF | 8726.1 | 2.7 | 4053.3 | 4341.8 | 5.6 |
| Hybrid | 8653.3 | 2.5 | 4130.7 | 4395.5 | 5.2 |
| WBFusion ANN | 5.6 | 1.2 | 3.8 | 3.5 | 1.1 |
We observe that our method is faster than 1000x for large datasets like MS-Marco, Hotpot-QA and Fever.
We also show average ranking accuracy under the constraint that RSWB objective be within 50% of optimal (— = threshold not achieved). Following tables show the results, which show that we perform better.
| NDCG@10 | MS-Marco | NF-Corpus | Hotpot-QA | Fever | SciFact |
|---|---|---|---|---|---|
| MoE | 0.587 | 0.342 | — | 0.586 | — |
| Ensemble | 0.625 | 0.290 | — | 0.826 | 0.698 |
| NeuralRRF | 0.608 | 0.369 | — | 0.708 | 0.684 |
| Hybrid | 0.566 | 0.333 | — | 0.596 | 0.673 |
| WBFusion ANN | 0.674 | 0.379 | 0.718 | 0.814 | 0.731 |
| MAP@10 | MS-Marco | NF-Corpus | Hotpot-QA | Fever | SciFact |
|---|---|---|---|---|---|
| MoE | 0.146 | 0.126 | — | 0.515 | — |
| Ensemble | 0.151 | 0.104 | — | 0.786 | 0.653 |
| NeuralRRF | 0.151 | 0.127 | — | 0.643 | 0.635 |
| Hybrid | 0.141 | 0.116 | — | 0.526 | 0.627 |
| WBFusion ANN | 0.163 | 0.142 | 0.634 | 0.764 | 0.700 |
Response from Authors Part 1
We thank the reviewer for their suggestions, which helped us improve our paper. We added the suggestions in the paper.
We thank the reviewer for identifying the distinction between the sequence analyzed in Theorem 7 and the output after k′-to-k filtering in Algorithm 4. We retain the original theorem as a guarantee for using the dot-product surrogate instead of the true marginals and add a separate result (Theorem 8 in Section 3.5 of our updated draft) that analyzes the true filtered output. We also show how the two results compose into a guarantee for the complete selection procedure.
The original theorem analyzes a singleton followed by k−1 surrogate selections. The practical procedure instead constructs a larger pool Sk′ and restarts selection within this pool. Write T1,…,Tk for the supports produced during filtering.
Our new theorem for the filtered support allows the pool to be generated by arbitrary surrogate scores and approximate retrieval. For a fixed query q, let Fk⋆=maxA⊆D,∣A∣=kF(q,A) and γ be the weak submodularity ratio of F. Our newly added theorem 8 says that: Fk⋆−F(q,Tk)≤E1+E2+E3, where
The three terms separate the effects of greedy selection, pool restriction, and imperfect filtering. E1 is the remaining gap allowed by the greedy analysis after k−1 additions from the initial singleton. E2 accounts for the quality lost when the candidate pool excludes documents: it compares the best size-k support in the original universe with the best size-k support still available in the pool. E3 is the cost of inexact selection, accumulating the shortfall between the best available marginal gain inside the pool and the gain actually selected at each filtering step. Exact greedy filtering makes E3 zero, while retaining a globally optimal size-k support makes E2 zero.
This result does not require uniformly accurate barycenter predictions, small cost oscillation, or exact maximum-inner-product retrieval.
Theorems 7 and 8 can also be combined. Suppose k′≥k and the pool retains the surrogate's first k distinct selections. Suppose its scores approximate the true marginal gains within O(ϵ) at all steps. If each retrieval returns an unselected item whose score is within O(η) of the maximum, then combining Theorems 7 and 8 gives us:
Fk⋆−maxA⊆Sk′,∣A∣=kF(q,A)≤Fk⋆−F(q,Sk)≤E1+E2, where E1=(1−kγ)k−1[Fk⋆−F(q,S1)] and E2=O(ϵ+η)∑j=0k−2(1−kγ)j.
For pool construction, E1 is the contribution for the surrogate's first k selections, while E2 accumulates errors from scoring and ANN search. The factor O(ϵ+η) accounts for score-approximation and the additional retrieval error. Since the pool contains the surrogate's size-k subset, the best size-k subset of the pool cannot be worse than that prefix. This connects the original surrogate guarantee to the implemented filtering-based algorithm.