acceptodds
Under review as a conference paper at ICLR 2027

CHORUS OR ECHO? AN ASSOCIATION-AWARE RANKING LABEL MODEL FOR UNSUPERVISED RETRIEVAL FUSION

Abstract

Aggregating multiple retrieval sources is a natural strategy for scientific literature retrieval, where relevance judgments are expensive to obtain at scale. When several such sources agree, however, it is unclear whether their agreement should count as stronger evidence: sources may jointly promote relevant candidates or repeat the same mistakes. Recent evidence of correlated errors among large language models shows that stronger sources do not eliminate this ambiguity. We quantify the consequences of this ambiguity in a controlled experiment: increasing agreement on irrelevant candidates reduces equal-weight fusion's NDCG@10 from 0.39 to 0.17, despite unchanged individual-source ranking quality. We propose the Deliberative Noisy Label Model (DNLM), a label model for fusing retrieval sources without relevance judgments. DNLM combines Plackett–Luce-based learning of source reliability with pairwise associations estimated from score deviations relative to pair-excluded anchors to determine fusion weights. Across four scientific and biomedical benchmarks, DNLM achieves the highest NDCG@10 in four of five settings and the highest MRR in all five against fixed-rule fusion, individual sources, and weak-supervision baselines. Controlled analyses show that DNLM mitigates the loss from shared irrelevant agreement, and that disagreement matters as well: discarding the associations that record where sources deviate in opposite directions lowers NDCG@10 by up to 3.4 points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
unsupervised rank aggregation, information retrieval, weak supervision

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_BmNeOpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper develops a new ranking label model (DNLM) and empirically demonstrates its retrieval effectiveness.

Critical strengths

The authors have, in my opinion, done a wonderful job in motivating the problem. Sections 2 and 3, where the main approach is discussed, is clear.

Critical weaknesses
  1. Absence of any theoretical analyses: the paper is purely empirical and its claims are only supported by limited empirical evidence (4 benchmarks). For instance, the optimization problem in Equation 11 can be nonconvex. But there is no discussion on why the author's approach results in a good solution.

  2. No discussion on the computational complexity of the proposed method.

  3. Finally, the whole analyses makes an implicit assumption of independent sources. I believe this may not hold in practice.

Questions for authors
  1. Can you give a theoretical reasoning on why your approach is better? Otherwise, a purely empirical analyses based on limited experiments is not convincing. Even a simple theoretical setup will go a long way in strengthening the work.

  2. Equation 11 can be nonconvex, as you have mentioned. But why does your proposed approach of solving it leads to a good solution? Again, some analytical work is needed here.

  3. Please discuss the computational complexity of the proposed method.

  4. Your analyses makes an implicit assumption of independent sources. I believe this may not hold in practice. If your approach can handle dependent sources then this should be discussed. If not, then it is a limitation and should be acknowledged. Ablation studies in this context will be useful.

3 comments
AuthorsOpenReview ↗

Response (1/2): W1/Q2 and Q1

We thank Reviewer BmNe for the detailed comments and questions. Please find our point-by-point response below. For each point, we give the location of the relevant material in the submission.


W1 / Q2. No theoretical analysis: Eq. (11) can be nonconvex, and why the proposed solution is a good one.

The submission raises nonconvexity itself (L249–253), as a possibility to be covered, and proves that it is handled: in exact arithmetic, the weight step returns a global minimiser of Eq. (11) for given AA and ρ\rho, whether or not AA is positive semidefinite (App. C.1, L889–971). It enumerates which sources receive nonzero weight and solves a linear system for each such set. In all 24 fits behind the main results (G5 and H7, Tables 1 and 10), AA is positive definite (smallest eigenvalue .089 to .209), so Eq. (11) is strictly convex in all of them. Nonconvexity is a case the solver is built for, not one that arose in these runs.


Q1. A theoretical reasoning on why the approach is better, even in a simple setup.

The submission isolates the reason with a controlled experiment, and we add a simple Gaussian setup that derives it in closed form.

In the submission. With each source's ranking quality held fixed, equal-weight NDCG@10 falls from .3897 to .1663 as three of five sources agree more on irrelevant candidates (L047–051, L396–400).

Added. Let source jj score a document as zj=μjy+ηjz_{j} = \mu_{j}y + \eta_{j}, with relevance yy, loadings μ≠0\mu \neq 0 and noise η∼N(0,Σ)\eta\sim\mathcal N(0,\Sigma), and let a fusion w⊤zw^{\top}z have signal-to-noise ratio SNR(w)=(w⊤μ)2/(w⊤Σw)SNR(w) = (w^{\top}\mu)^{2}/(w^{\top}\Sigma w).

Proposition 1: SNR(w)SNR(w) is maximised exactly by w∝Σ−1μw \propto \Sigma^{- 1}\mu. If kk sources have pairwise noise correlation rr and the others are independent, all with unit loadings and variances, equal weights give SNR=m2/(m+k(k−1)r)SNR = m^{2}/(m + k(k - 1)r), decreasing in rr, and the optimum discounts each of the kk sources by 1/(1+(k−1)r)1/(1 + (k - 1)r) relative to the others.

Proposition 2: on the set SS of sources with nonzero weight, DNLM's weights are wS=AS,S−1(ρS−λ1)w_S=A_{S,S}^{-1}(\rho_S-\lambda\mathbf 1) for nonsingular AS,SA_{S,S}, with λ\lambda set so that they sum to one. This is the same form as Σ−1μ\Sigma^{- 1}\mu, and when AA has the group structure of Proposition 1 and the ρj\rho_{j} are equal, the discount is the same. The correspondence is one of form, not a guarantee: AA and ρ\rho are not estimates of Σ\Sigma and μ\mu.

We have added the two propositions as App. I (Propositions 1 and 2), numbered the C.1 result as Proposition C.1, and stated in App. C.1 that AA is positive definite in all 24 fits.

AuthorsOpenReview ↗

Response (2/2): W2/Q3, W3 and Q4

W2 / Q3. No discussion of the computational complexity.

The submission reports measured running times (Section 4.3, Table 11), and we add the asymptotic orders.

In the submission. On LitSearch with five sources, fitting DNLM on one CPU thread takes 8.19 s for Stage I and 8.02 s for Stage II, against 58.83 s to prepare the MiniLM source. At inference, fusion takes 0.62 ms per query.

Added.

ComponentCost
Plackett–Luce (PL) update (Eqs. 3, 7)O(m∣Qtr∣(Nˉ+K))O(m\lvert Q_{tr}\rvert(\bar{N} + K)) per update
Association estimation (Eqs. 8–10)O(m2Ntr)O(m^{2}N_{tr})
Stage II weight step (Eq. 11)O(2mm3+T2mm2)O(2^{m}m^{3} + T2^{m}m^{2})
Inference, per queryO(mNq+Nqlog⁡Nq)O(mN_{q} + N_{q}\log N_{q})

Here mm is the number of sources, ∣Qtr∣|Q_{tr}| the number of training queries, Nˉ\bar{N} and NtrN_{tr} the mean and total number of training candidates, NqN_{q} a query's candidate count, KK the prefix length and TT the number of outer iterations. The cost is linear in queries and candidates, apart from sorting. The 2m2^{m} factor reflects the 2m−12^{m} - 1 possible sets of weighted sources, 31 for five sources and 127 for seven.

We have added this table as Table 15 in App. F.3.


W3. An implicit assumption of independent sources.

DNLM does not assume independent sources. It does the opposite: modeling dependent sources is the central problem of this paper, stated in its title (L001–004), abstract (L014–017), Section 1 (L041–045) and Section 3 (L154–158). The fusion weights of Eq. (11) model dependence through the association matrix AA: two positively associated sources pay a cost when both receive weight (L242–246). Only the PL objective of Eq. (3), which fits the reliability indices, treats sources as conditionally independent given the consensus (L141–147).

Q4. Whether the approach handles dependent sources, with ablation studies.

The submission contains both kinds of ablation. It varies the dependence: over 25 conditions, the rankings of three of five sources range from independent to fully shared (App. D, Fig. 2). It also removes the dependence term: the control A=IA = I is the same method with no inter-source association (Fig. 2(b), Table 7). With agreement only on irrelevant candidates, NDCG@10 is .0871 with A=IA = I and .2005 with DNLM. In the main experiments, adding AA to PL-only raises NDCG@10 from .358 to .430 on LitSearch and from .285 to .349 on STaRK-MAG (Table 2; STaRK-MAG by five-fold cross-validation, App. A).

I thank the authors for their detailed response, which has helped me to get a better understanding of their work. The paper is much stronger now and I am happy to raise the score to 3.

Reviewer_jeWDOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DNLM, an unsupervised retrieval fusion framework that combines Plackett–Luce-based source reliability estimation with pairwise source associations. Experiments on four scientific and biomedical retrieval benchmarks demonstrate improvements over several fusion baselines.

Critical strengths
  1. The paper addresses an important problem: correlated errors can undermine the effectiveness of retrieval fusion.

  2. The chorus vs. echo analogy is intuitive and effective in motivating the need to distinguish useful agreement from redundant or misleading agreement.

  3. The paper includes multiple real-world benchmarks, controlled experiments, and component ablations.

Critical weaknesses

1. Robustness to low-quality retrieval sources

My first question was what would happen if a random ranking system were introduced into the retrieval pool. Although this may be uncommon in practical deployments, it is worth considering from an algorithmic perspective. More importantly, how would DNLM handle a source that produces some correct results but is systematically wrong on many others? I acknowledge that ground-truth relevance labels are unavailable in this setting, but I would appreciate at least some discussion of this issue and the conditions under which such sources can be identified.

2. Source associations do not necessarily indicate shared errors

Two systems may produce similar rankings because both identify relevant documents, or because they share biases and rank the same irrelevant documents highly. These cases can exhibit similar statistical associations but have very different implications for fusion. Since the association matrix measures deviations from constructed anchors rather than ground-truth errors, it remains unclear to what extent the estimated associations capture harmful shared errors rather than useful common signals. The controlled experiments are helpful, but the validity of this interpretation in real-world retrieval remains insufficiently established.

3. Reliability estimates may not reflect actual retrieval quality

The PL model estimates source reliability based on observed rankings and a constructed consensus. However, consistency with this consensus does not necessarily imply agreement with true relevance. When sources share systematic errors, the consensus itself may be biased. The paper would benefit from a clearer justification of the assumptions under which PL-based reliability estimates serve as meaningful proxies for actual retrieval quality.

4. The choice of Plackett–Luce is insufficiently justified

Although the paper includes component ablations, it does not compare PL against alternative ranking models within the same framework. Would replacing PL with a Mallows-based model or another reasonable alternative yield comparable or better results? It remains unclear whether the gains depend on PL specifically or primarily arise from association-aware fusion.

More broadly, alternative formulations, such as conditional mutual information I(Y; C | A, B), could directly characterize the incremental information a new source C provides about true relevance Y, given the outputs of existing sources A and B. If C merely duplicates existing information or is independent of true relevance, it should provide little or no additional information. If it helps identify relevant documents missed by existing sources, it may provide valuable additional information. Estimating this quantity without relevance labels would require additional assumptions, but it illustrates that the current modeling choice is not the only possible approach.

Questions for authors

Comparing alternative ranking models within the same framework would help establish whether the proposed approach generalizes beyond the specific PL formulation.

5 comments
AuthorsOpenReview ↗

Response (1/3): W1 and W2

We thank Reviewer jeWD for the valuable and insightful feedback. Please find our detailed response below:


W1

A random ranking system introduced into the retrieval pool.

A random ranker is outside our setting, which assumes retrieval systems with non-trivial signal. Within DNLM, the only component that can lower its weight is the reliability index ρ\rho: a random ranker is unrelated to every other source, so AA records no redundancy specific to it.

A source that is correct on some results but systematically wrong on many others, and the conditions under which it can be identified.

DNLM lowers the weight of such a source through the reliability index ρ\rho when its errors are unique to it, and through the association matrix AA when it shares errors with some other sources. Three scenarios depend on who shares the errors.

Errors unique to the source. They pull its ranking off the consensus, which ρ\rho can register. This holds when the consensus is informative and not dominated by that source.

Errors shared with some sources. The quadratic term w⊤Aww^{\top}Aw discounts their joint weight. In App. D, when three of five sources agree only on irrelevant candidates, DNLM reaches .2005 NDCG@10 against .1565 for PL-only (Table 7).

Errors shared by all sources. No source is left to disagree. Such errors are not identifiable from the sources' outputs alone without further assumptions.

The conditions under which such sources can be identified.

Without relevance labels, DNLM can identify such a source under two conditions: its errors are not shared by all sources, and the consensus is informative and not dominated by that source.

We have added this discussion as App. H.4.


W2

To what extent the estimated associations capture harmful shared errors rather than useful common signals.

AA is designed to capture harmful shared errors: a pair of sources with a high estimated association receives less joint weight (Eq. 11). The question is then whether it also captures useful common signals, which would down-weight sources that agree because both are right. The constructed anchor limits this. A useful common signal that most sources find is also in the anchor, so it largely cancels in the deviations from which AA is estimated (Eqs. 8–10). In the controlled experiments, where we set which of the two cases holds, the estimated association is about twice as large for harmful shared errors (.698) as for useful common signals (.357) (App. D, Table 8).

The validity of this interpretation in real-world retrieval.

We appreciate the reviewer for acknowledging the controlled experiments. Our real-world benchmarks support this interpretation: agreement shared with the pool is more likely to be a useful common signal, and it largely cancels in the anchor before AA is estimated, so it is unlikely to be penalized through AA. We examined this further on the G5 and H7 pools.

Setup. For a pair of sources, the anchor is formed from the other sources (Eq. 8). We therefore take the documents that both sources of a pair rank in their top 10 and split these documents into two groups. "Shared" documents are also in the top 10 of at least half of the other sources, so they are in the anchor. "Pair only" documents are in the top 10 of no other source. For each group, we compute the percentage of relevant documents from the relevance judgments.

G5 sharedG5 pair onlyH7 sharedH7 pair only
LitSearch14.82.124.31.3
STaRK-MAG14.84.325.43.4
STaRK-Prime9.73.111.42.5
TREC-COVID81.359.286.558.9

Shared documents are relevant more often than pair-only documents on every dataset and pool, and for every cutoff from top 5 to top 100.

We have added this analysis as App. H.3 (Table 18) and summarized it in Section 4.3.

AuthorsOpenReview ↗

Response (2/3): W3

W3

Assumptions under which PL-based reliability estimates are proxies for actual retrieval quality.

ρ\rho measures agreement with the consensus, and this agreement is a proxy for quality under the standard conditions of label models: sources that are better than random and conditionally independent given the true label ([1], [2], [3]). The corresponding results for rankings are in [4] and, for (mixtures of) Plackett–Luce models, in [5]. DNLM's reliability component relies on these conditions.

A consensus biased by shared systematic errors.

This is the violation of conditional independence that the association matrix AA is designed for: the consensus leans toward a strict subset of sources that share the error, ρ\rho follows them, and the quadratic term w⊤Aww^{\top}Aw discounts their joint weight. In App. D, when three of five sources agree only on irrelevant candidates, PL-only gives them a weight of .620 and reaches .1565 NDCG@10. DNLM lowers the weight to .528 and reaches .2005 (Tables 7–8). The cases by who shares the error are in our reply to W1.

We have stated these conditions after Eq. (7) in Section 3.1 and added this discussion as App. H.2.

[1] Maximum Likelihood Estimation of Observer Error-Rates Using the EM Algorithm. Applied Statistics, 1979.
[2] Learning From Crowds. JMLR, 2010.
[3] Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods. ICML, 2020.
[4] Rank Aggregation via Heterogeneous Thurstone Preference Models. AAAI, 2020.
[5] Learning Mixtures of Plackett-Luce Models. ICML, 2016.

AuthorsOpenReview ↗

Response (3/3): W4 & Q1

W4 & Q1

Would replacing PL with a Mallows-based model yield comparable or better results?

For this response we replaced PL with a Mallows model, first in Stage I alone and then in both stages. The first gives comparable, slightly lower NDCG@10, and the second lowers it by 2.6 to 7.5 points, so the Mallows variants we tested do not improve on PL.

LitSearchTREC-COVIDSTaRK-MAGPrime syntheticPrime human
DNLM (PL).4671.6684.3666.1858.2732
Mallows in Stage I.4417.6609.3635.1858.2732
Mallows in both stages.4085.6304.3307.1602.1977
Mallows-only.3141.5900.2362.1601.1637

Mallows-only fuses with the Mallows reliabilities directly, the counterpart of PL-only in Table 2.

PL suits DNLM for two reasons. It takes the graded consensus scores directly (Eqs. 6–7), where Mallows needs a central permutation. Also, its top-KK likelihood (Eq. (2), K=20K = 20) fits the head of the ranking, where retrieval is evaluated.

Do the gains depend on PL specifically or arise from association-aware fusion?

Both matter: replacing PL lowers the results above, and under either ranking model the association-aware steps improve over reliability-only fusion on most datasets. With PL, the submission's ablation shows it (Table 2, L350–357): PL-only reaches .358 NDCG@10 on LitSearch and .285 on STaRK-MAG, and the full method .467 and .368 (STaRK-MAG by five-fold CV, App. A). With Mallows in both stages, adding AA and the Stage II update to Mallows-only raises NDCG@10 by 4.0 to 9.5 points on LitSearch, TREC-COVID and STaRK-MAG. On STaRK-Prime the paired-bootstrap 95% intervals include zero.

This also answers Q1: the approach is not tied to the PL formulation, since the association-aware steps improve over reliability-only fusion under Mallows as well.

Conditional mutual information as an alternative formulation.

We thank the reviewer for this detailed suggestion, which gives a direct way to examine what AA captures. We tested one part of it, a source that merely duplicates existing information, and found that AA tends to be higher when the weaker source of a pair provides little additional information given the other, relative to the information it carries alone.

Setup. Because AA is defined for pairs, we condition on one source rather than two. For each pair, with BB the more informative source and CC the other, we measure how much of CC's information BB also carries as 1−I(Y;C∣B)/I(Y;C)1 - I(Y; C \mid B)/I(Y; C). We estimate each term with the relevance labels, which DNLM does not use: a simple model predicts the relevant candidates of a query from the sources' scores and ranks, and we measure its log-probability gain over a uniform guess on queries it was not fitted on.

Across pairs, the Spearman correlation between AA and 1−I(Y;C∣B)/I(Y;C)1 - I(Y; C \mid B)/I(Y; C) is 0.51 to 0.70 in all five H7 settings and 0.38 to 0.75 in four of the five G5 settings (−0.03 on TREC-COVID, where nearly every pair includes a graph source with too little information for a reliable estimate). In these nine settings, the cost that Eq. (11) adds for weighting both sources of a pair thus tends to be larger when the weaker source adds less information given the other. Estimating this incremental information for retrieval sources without labels is a direction we plan to pursue.

We have added both Mallows experiments as App. F.2, with their tables and bootstrap intervals (Tables 13 and 14), and the CMI analysis as App. H.5 (Table 19).

Reviewer_VhvQOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

To address the issue in unsupervised retrieval fusion where multiple retrieval sources may repeatedly reinforce the same errors, this paper proposes a Dependency-Aware Ranking Label Model (DNLM). The method first estimates the reliability of individual retrieval sources using the Plackett–Luce ranking probability model. It then calculates the positive or negative dependencies between these sources by comparing their outputs against reference scores derived from other sources. Finally, it integrates these reliability estimates and the dependency matrix to optimize fusion weights, thereby generating pseudo-labels for document relevance. Experimental results demonstrate that this approach outperforms traditional fusion strategies and weakly supervised baselines, particularly in scenarios where multiple retrieval sources collectively recommend irrelevant documents.

Critical strengths
  1. The paper points out that consistent results across multiple retrieval sources do not necessarily imply greater reliability; rather, they may stem from shared biases that lead to recurring errors. This is further validated through controlled experiments demonstrating how erroneous correlations negatively impact fusion performance.

  2. The proposed approach integrates the estimation of retrieval source reliability, correlation modeling based on reference scores, and constrained weight optimization. It accounts for both positive and negative correlations, moving beyond fusion strategies that rely solely on the independent performance of individual sources or the degree of overlap among their results.

  3. Evaluated across four benchmark datasets and five experimental settings, the DNLM model achieves the best NDCG@10 in four out of five primary multi-source configurations, and the best MRR in all five. The individual components of the method are validated through ablation studies, controlled experiments on erroneous correlations, and analyses of interventions regarding negative correlations.

Critical weaknesses
  1. Even if two retrieval sources are inherently independent, subtracting a common, volatile reference score from both can artificially induce a positive correlation. Furthermore, since the weights assigned to the reference score are learned using all available retrieval sources, the "Pair-excluded Anchor" approach cannot guarantee that the estimated correlations remain unaffected by the original retrieval sources. The authors lack a theoretical analysis of this statistical bias or controlled experiments to verify independence; consequently, it is difficult to demonstrate that the estimated correlation matrix accurately reflects the true dependencies between the retrieval sources.

  2. The paper employs the Plackett–Luce model to estimate the reliability of retrieval sources. This self-referential optimization process may cause the model to tend to reinforce consistency among most retrieval sources, without being able to determine whether the consistency stems from a common error. This self-referential optimization process may bias the model toward reinforcing consensus among the majority of sources, without determining whether such consensus stems from shared errors.

  3. The paper primarily compares its approach against methods such as Equal, RRF, CombMNZ, and UWS. Notably, while the "Related Work" section discusses retrieval fusion methods based on correlation weighting (Wu & McClean, 2005), these methods are omitted from the experiments. Moreover, the implementation of UWS lacks a dependency graph or a mechanism to select triplets of retrieval sources that satisfy conditional independence, failing to adequately represent its capacity for dependency modeling.

  4. The experimental setup is highly sensitive to the selection of retrieval sources, and the approach’s performance in real-world deployment remains insufficiently validated. In the primary experiment (G5), the authors utilize BM25, MiniLM, and three graph-based retrieval sources; however, these graph-based sources rely on the initial outputs of BM25 and MiniLM, and their individual retrieval performance is generally weak.

Questions for authors

Please see the weaknesses.

2 comments
AuthorsOpenReview ↗

Response (1/2): W1 and W2

We appreciate Reviewer VhvQ's constructive feedback. Below, we address each of the reviewer's points in detail:


W1

Subtracting a common reference score can artificially induce a positive correlation

The key question for DNLM is not whether such a positive correlation exists, but whether it changes the relative association structure that determines the fusion weights. Under the conditions of the Remark below, the shared reference adds the same constant 1/(m−1)1/(m - 1) to every source pair, so no pair appears more redundant than another. It only rescales the diagonal term 12∑jwj2\tfrac12\sum_j w_j^2 of Eq. (11). To check this directly, we subtracted the predicted offset from AA and re-ran Stage II: the weights shifted away from BM25, mostly toward graph sources, and NDCG@10 changed by −1.4- 1.4 to +0.0+ 0.0 pp across the ten G5 and H7 settings.

The pair-excluded anchor cannot guarantee that the estimated correlations are unaffected by the original sources

We had stated this at L204–205: pair exclusion removes direct score contributions "but does not imply independence", since the normalized reliability indices u\boldsymbol u of Eq. (6), which weight the anchor, are learned from the full source pool. Its purpose is to stop a source from entering the reference it is measured against. We also refit uu without the pair: AA moves by 5–19% (mean 11%) of ∥A−I∥F\|A-I\|_F, every sign is kept, and NDCG@10 is unchanged in four of five settings.

Theoretical analysis or controlled experiments to verify independence

We add a Remark and the results of the zero-control condition (a new row of the controlled experiment in App. D, Table 8). Both show that the shared anchor does not make independent sources look more redundant than other pairs: subtracting it adds about the same amount to the association of every pair, which changes no comparison between them.

The Remark: with a common signal of equal loading and mutually independent noise, the anchor noise enters both deviations in Eq. (9), so Aj,k=1/(m−1)A_{j,k} = 1/(m-1) under equal noise variances and anchor weights, .25 for G5 and .167 for H7.

The zero-control condition sets both agreement controls to zero (crel=cirr=0c_{\mathrm{rel}} = c_{\mathrm{irr}} = 0): within each relevance grade, candidate identities are permuted independently across the three target sources. Their association is .255, at the level of .246 between them and the other two sources, and their weight is the uniform .600 = 3/5. When the agreement controls are raised above zero, so that the target sources share part of their rankings, AA detects this dependence: their association increases from .255 to .357–.698, as reported in the submission's Table 8.

We have added the Remark's full derivation (Remark 1, App. G.1), the subtract-offset and leave-pair-out tables (Tables 16 and 17, App. G.2 and G.4), and the zero-control row of Table 8 (App. D.5), with the between-group value in App. G.3.


W2

Self-referential PL estimation may reinforce a majority consensus that stems from shared errors.

This is the case the association matrix AA is designed for: PL reliability alone follows the majority, and AA adds a cost in Eq. (11) when redundant sources both receive weight. What follows depends on who shares the error.

Errors unique to one source. They pull its ranking off the consensus, which the reliability index ρ\rho can register. This holds when the consensus is informative and not dominated by that source.

Errors shared by some sources, possibly a majority. The quadratic term w⊤Aww^{\top}Aw discounts their joint weight. In App. D, when three of five sources agree only on irrelevant candidates, PL-only gives this majority a weight of .620 and reaches .1565 NDCG@10. DNLM lowers the weight to .528 and reaches .2005 (Tables 7–8).

Agreement on relevant candidates within a group of sources. The same discount has a modest cost: 0.36 points against PL-only, less than a tenth of the 4.40-point gain in the previous case (Table 6).

Errors shared by all sources. No source is left to disagree. Such errors are not identifiable from the sources' outputs alone without further assumptions.

We have added this discussion as App. H.1.

AuthorsOpenReview ↗

Response (2/2): W3 and W4

W3

Correlation-weighted fusion is not compared, and UWS is implemented without a dependency graph.

We have now run a label-free adaptation of Wu & McClean's correlation-weighted fusion [1] (WM) and UWS with a dependency graph under the submission's protocol. Neither reaches DNLM's NDCG@10 in any of the ten G5 and H7 settings.

G5 DNLMG5 WMG5 UWSH7 DNLMH7 WMH7 UWS
LitSearch.4671−10.37−13.22.5084−7.94−9.77
TREC-COVID.6684−5.88−6.90.7211−8.49−8.95
STaRK-MAG.3666−8.10−10.91.3920−6.31−8.07
STaRK-Prime synthetic.1858−1.74−2.60.2086−2.12−2.76
STaRK-Prime human.2732−8.53−11.53.2922−7.16−8.60

DNLM columns are NDCG@10. WM and UWS columns are differences to DNLM in points.

WM. WM weights each source by its mean correlation with the other sources. The resulting weights are close to uniform (max/min ratio at most 1.39), so WM stays within −0.02- 0.02 to +1.62+ 1.62 points of Equal. On MRR, WM reaches .8800 on TREC-COVID G5, above DNLM's .8717.

UWS with a dependency graph. The dependency graph marks the three graph sources as mutually dependent, since they share one graph and one seed set (Section 4.1). This gives UWS a dependence structure, declared from how the sources are built, as prior knowledge, which DNLM has to estimate. UWS then excludes every triplet that contains a dependent pair, leaving 3 of 10 triplets in G5 and 22 of 35 in H7. This changes its NDCG@10 by −0.21- 0.21 to +1.35+ 1.35 points relative to UWS without the graph.

We have added both rows to Tables 1 and 10 (definitions in App. B.2 and B.3) and eight variants of the WM recipe to App. F.1 (Table 12). We have also revised the claim that DNLM has the highest MRR in all five settings to four of five (abstract, Sections 4.2 and 6), and its TREC-COVID gain over the strongest baseline from 6.55 to 5.88 points (Section 4.2).

[1] Data Fusion with Correlation Weights. ECIR, 2005.


W4

The experimental setup is highly sensitive to the selection of retrieval sources, and real-world performance remains insufficiently validated.

The submission had evaluated a second pool, H7, which adds MPNet and E5 to G5, and DNLM's results hold there: it remains the strongest fusion method in four of five settings, and its NDCG@10 rises over G5 in all five, by 1.90 to 5.27 points (Table 10). The evaluation covers four benchmarks, including the human-written queries of STaRK-Prime, and DNLM is fitted without relevance labels (L322).

The graph sources in G5 rely on the outputs of BM25 and MiniLM, and are individually weak.

This is the setting DNLM targets, sources that "capture different aspects of relevance, but may also share limitations" (L038), and DNLM is built to limit the loss these sources cause. DNLM exceeds BM25, the best single source, in all five settings. Equal, RRF and UWS, on the same five sources, trail DNLM by 6.55 to 13.71 points in four of the five settings (Table 1; RRF leads on STaRK-Prime synthetic). DNLM's weights follow held-out single-source quality (Spearman .671 to .894 in G5, .556 to .883 in H7; Table 3).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to the AC and all reviewers

We thank all reviewers for their time and constructive comments, which have helped us improve the paper. We have responded to each point raised by each reviewer.

When a response refers to material that was already in the submission, it gives the location in the submitted version, by line number (e.g., L204–205) or by section, equation, table or appendix number.

All new experiments, discussions and propositions mentioned in our responses are included in the revised PDF and highlighted in blue. New tables are numbered from Table 12 onward. The last paragraph of each response states where the added material appears.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.