- Rating
- 3(+2)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+2)
Read review
Paper summary
This paper develops a new ranking label model (DNLM) and empirically demonstrates its retrieval effectiveness.
Critical strengths
The authors have, in my opinion, done a wonderful job in motivating the problem. Sections 2 and 3, where the main approach is discussed, is clear.
Critical weaknesses
-
Absence of any theoretical analyses: the paper is purely empirical and its claims are only supported by limited empirical evidence (4 benchmarks). For instance, the optimization problem in Equation 11 can be nonconvex. But there is no discussion on why the author's approach results in a good solution.
-
No discussion on the computational complexity of the proposed method.
-
Finally, the whole analyses makes an implicit assumption of independent sources. I believe this may not hold in practice.
Questions for authors
-
Can you give a theoretical reasoning on why your approach is better? Otherwise, a purely empirical analyses based on limited experiments is not convincing. Even a simple theoretical setup will go a long way in strengthening the work.
-
Equation 11 can be nonconvex, as you have mentioned. But why does your proposed approach of solving it leads to a good solution? Again, some analytical work is needed here.
-
Please discuss the computational complexity of the proposed method.
-
Your analyses makes an implicit assumption of independent sources. I believe this may not hold in practice. If your approach can handle dependent sources then this should be discussed. If not, then it is a limitation and should be acknowledged. Ablation studies in this context will be useful.
3 comments
Response (2/2): W2/Q3, W3 and Q4
W2 / Q3. No discussion of the computational complexity.
The submission reports measured running times (Section 4.3, Table 11), and we add the asymptotic orders.
In the submission. On LitSearch with five sources, fitting DNLM on one CPU thread takes 8.19 s for Stage I and 8.02 s for Stage II, against 58.83 s to prepare the MiniLM source. At inference, fusion takes 0.62 ms per query.
Added.
| Component | Cost |
|---|---|
| Plackett–Luce (PL) update (Eqs. 3, 7) | per update |
| Association estimation (Eqs. 8–10) | |
| Stage II weight step (Eq. 11) | |
| Inference, per query |
Here is the number of sources, the number of training queries, and the mean and total number of training candidates, a query's candidate count, the prefix length and the number of outer iterations. The cost is linear in queries and candidates, apart from sorting. The factor reflects the possible sets of weighted sources, 31 for five sources and 127 for seven.
We have added this table as Table 15 in App. F.3.
W3. An implicit assumption of independent sources.
DNLM does not assume independent sources. It does the opposite: modeling dependent sources is the central problem of this paper, stated in its title (L001–004), abstract (L014–017), Section 1 (L041–045) and Section 3 (L154–158). The fusion weights of Eq. (11) model dependence through the association matrix : two positively associated sources pay a cost when both receive weight (L242–246). Only the PL objective of Eq. (3), which fits the reliability indices, treats sources as conditionally independent given the consensus (L141–147).
Q4. Whether the approach handles dependent sources, with ablation studies.
The submission contains both kinds of ablation. It varies the dependence: over 25 conditions, the rankings of three of five sources range from independent to fully shared (App. D, Fig. 2). It also removes the dependence term: the control is the same method with no inter-source association (Fig. 2(b), Table 7). With agreement only on irrelevant candidates, NDCG@10 is .0871 with and .2005 with DNLM. In the main experiments, adding to PL-only raises NDCG@10 from .358 to .430 on LitSearch and from .285 to .349 on STaRK-MAG (Table 2; STaRK-MAG by five-fold cross-validation, App. A).
I thank the authors for their detailed response, which has helped me to get a better understanding of their work. The paper is much stronger now and I am happy to raise the score to 3.
Response (1/2): W1/Q2 and Q1
We thank Reviewer BmNe for the detailed comments and questions. Please find our point-by-point response below. For each point, we give the location of the relevant material in the submission.
W1 / Q2. No theoretical analysis: Eq. (11) can be nonconvex, and why the proposed solution is a good one.
The submission raises nonconvexity itself (L249–253), as a possibility to be covered, and proves that it is handled: in exact arithmetic, the weight step returns a global minimiser of Eq. (11) for given A and ρ, whether or not A is positive semidefinite (App. C.1, L889–971). It enumerates which sources receive nonzero weight and solves a linear system for each such set. In all 24 fits behind the main results (G5 and H7, Tables 1 and 10), A is positive definite (smallest eigenvalue .089 to .209), so Eq. (11) is strictly convex in all of them. Nonconvexity is a case the solver is built for, not one that arose in these runs.
Q1. A theoretical reasoning on why the approach is better, even in a simple setup.
The submission isolates the reason with a controlled experiment, and we add a simple Gaussian setup that derives it in closed form.
In the submission. With each source's ranking quality held fixed, equal-weight NDCG@10 falls from .3897 to .1663 as three of five sources agree more on irrelevant candidates (L047–051, L396–400).
Added. Let source j score a document as zj=μjy+ηj, with relevance y, loadings μ=0 and noise η∼N(0,Σ), and let a fusion w⊤z have signal-to-noise ratio SNR(w)=(w⊤μ)2/(w⊤Σw).
Proposition 1: SNR(w) is maximised exactly by w∝Σ−1μ. If k sources have pairwise noise correlation r and the others are independent, all with unit loadings and variances, equal weights give SNR=m2/(m+k(k−1)r), decreasing in r, and the optimum discounts each of the k sources by 1/(1+(k−1)r) relative to the others.
Proposition 2: on the set S of sources with nonzero weight, DNLM's weights are wS=AS,S−1(ρS−λ1) for nonsingular AS,S, with λ set so that they sum to one. This is the same form as Σ−1μ, and when A has the group structure of Proposition 1 and the ρj are equal, the discount is the same. The correspondence is one of form, not a guarantee: A and ρ are not estimates of Σ and μ.
We have added the two propositions as App. I (Propositions 1 and 2), numbered the C.1 result as Proposition C.1, and stated in App. C.1 that A is positive definite in all 24 fits.