- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces KNIT, a training-free passage selector for multi-hop RAG. It combines existing retrieval scores with entity connections from a proposition graph and question coverage to select ten passages from thirty candidates. An LLM then retains five passages for answering. Experiments across five retrieval pipelines and five corpora report improvements in evidence-chain completion, with corresponding gains in answer accuracy on the Wikipedia datasets.
Critical strengths
- The paper addresses a practical limitation of top-k truncation, where a passage needed to complete a reasoning chain receives a low individual relevance score. Preserving retrieval scores while incorporating entity connections is well motivated.
- The evaluation covers lexical, dense, reranking and graph retrieval pipelines. Improvements on four corpora excluded from tuning provide useful evidence of transfer.
- The ablations and deletion tests offer informative evidence about the role of entity links. The analysis of their limitations on news also helps clarify when the method is effective.
Critical weaknesses
-
The MuSiQue test is not fully independent of configuration selection. Appendix E states that the search space was expanded using PropRAG results from all 1,000 questions, including the nominal held-out half, after an earlier configuration failed to improve it significantly. This weakens the held-out evidence for the central claim about improving the strongest pipeline. The disclosure and results on untouched transfer corpora are valuable, but independent validation on fresh MuSiQue questions is still needed.
-
The contribution of the specific graph objective remains insufficiently isolated. In Appendix G, the primary comparison with equal tuning budgets yields a 0.60-point gain over the simpler cohesion selector, with a confidence interval of [-0.64, 1.84]. Some broader searches favor the graph stage, but the conclusion depends on the tuning range. Comparing a cohesion shortlist followed by the same LLM stage against the complete KNIT system would clarify the additional value of the spanning-tree and coverage formulation.
-
The practical advantage over direct LLM selection remains unclear. The same LLM reading all thirty candidates significantly outperforms KNIT on average on three corpora. KNIT reduces prompt tokens, but requires corpus-wide graph extraction. Table 7 evaluates collections containing roughly 6,000 to 12,000 passages, and Appendix L.2 replaces QASC’s 17-million-sentence collection with 7,671 annotated facts because full graph construction was infeasible. Although indexing costs are disclosed, a comparison accounting for corpus size and query volume is needed to establish the overall efficiency benefit.
-
The confidence intervals for individual performance differences require correction. Appendix N rescales a Clopper-Pearson interval for the win rate among discordant pairs by the observed discordance rate. This treats the latter as fixed and omits its sampling uncertainty, so the resulting intervals are not exact 95% intervals for population performance differences. Valid intervals for paired proportions are needed. This issue does not invalidate the exact McNemar tests or the pooled bootstrap intervals.
Questions for authors
Please refer to the summary of weaknesses.
3 comments
Response to Reviewer BjuH [2/2]
W3. "... a comparison accounting for corpus size and query volume is needed to establish the overall efficiency benefit." Added (GR-Q2; Sec. 2.2, App. A). Against the 30-passage reranker, which completes more chains than KNIT at ten on three corpora, each question saves about 6 to 7 GPU-seconds (1.6 on QASC's one-sentence facts), so KNIT's total cost, graph included, falls below the reranker's once a corpus has served 0.3 to 0.7 questions per passage. At accuracy matched to the reranker's, with the shortlist of 15 or 20 that closes its lead (20 on 2WikiMultihopQA), each question still saves 2.9 to 4.8 GPU-seconds and the total cost falls below the reranker's after 0.5 to 1.5 questions per passage (exploratory). For PropRAG on Wikipedia there is no graph to build. The saving per question comes from reading ten passages instead of thirty. What the graph buys is accuracy: over the same call run on the Top-10, which also reads ten passages, KNIT gains 3.2 to 5.1 points on Wikipedia and nothing measurable on news and QASC (GR-Q1). At ten, KNIT never falls below a pipeline's own Top-5, while the reranker falls below PropRAG's on MuSiQue; the longer shortlists give back 3.4 and 5.0 points of PropRAG's gain there (GR-Q2). For QASC's full 17M sentences, KNIT's total cost would fall below the reranker's only after about 5 million questions, a limit on corpus size that the revision states (App. A).
W4. "Valid intervals for paired proportions are needed." Agreed: the rescaled Clopper-Pearson interval treats the discordance rate as fixed. We recomputed all 234 single-pair intervals behind the main text's claims with Tango's score interval (Sec. 3, App. N). No significance call changes, nor any count of significant pairs, in the main text or elsewhere in the paper. Significance is still decided by the exact McNemar test, and Tango's interval excludes zero whenever that test rejects: it excludes zero exactly when McNemar's statistic exceeds 1.96, which holds for every split of up to 5,000 discordant questions whose exact p is below 0.05. The converse can fail: on four of the 234 pairs (exact p 0.053 to 0.061), and on seven pairs in the appendix figures, the new interval excludes zero while the test does not reject, so bold and filled marks are now defined by the exact test, in Table 2 and the figure captions. Endpoints move by a median of 0.36 points; the largest shift is 2.88 points, on held-out BM25, where KNIT wins all 47 discordant questions.
We hope these new results address your concerns, and we are happy to discuss any of them further.
Kind reminder
Dear Reviewer BjuH,
Thank you again for your specific, checkable points. As the discussion period is about to close, we would like to kindly check whether our responses have addressed them. We ran each experiment you proposed: the frozen configuration on 1,417 fresh MuSiQue questions (W1), cohesion's ten followed by the same LLM call on all 25 pairs (W2), the cost by corpus size and query volume (W3), and Tango's score interval for all 234 single-pair intervals behind the main text's claims (W4). Our reply above gives each result, including those in the direction of your concern, and the revised PDF carries all of them. If they resolve your concerns, we would be grateful if you would consider updating your assessment. We are happy to answer any remaining question while the discussion is open.
Response to Reviewer BjuH [1/2]
We thank the reviewer for the specific, checkable points, and for noting the breadth of pipelines, the evidence of transfer, the deletion tests and the analysis of news. We ran each experiment you proposed.
W1. "... independent validation on fresh MuSiQue questions is still needed." Done (GR-Q5; Sec. 4.1, App. I). On all 1,417 MuSiQue dev questions the paper never used, the frozen configuration improves all five pipelines significantly, with a mean of +10.28 [+9.15, +11.42], and by +13.20 on the 303 that share no single-hop sub-question with the 1,000 (1,114 share one). For PropRAG this run is a weaker test. There it reads our 14B graph, as on news and QASC in the paper: of the paper's corpora, PropRAG ships extracted propositions only for the three Wikipedia ones, and its documentation supports a new extraction only through a paid API (App. I). Its Top-5 there is 40.7, against 53.9 on the paper's questions, a fall far larger than the other pipelines' 0.6 to 3.3 points. Its gain is +7.55 [+5.32, +9.83] on the 1,417 and +3.96 [−1.08, +9.09] on the 303, not significant (n = 303). We agree that the held-out half is not fully independent for PropRAG, as the paper states, and that the MuSiQue PropRAG cell is therefore not independently established (App. I). The claim about the strongest pipeline rests on 2WikiMultihopQA and HotpotQA, which took no part in the choice and where PropRAG reads its own graph: KNIT improves it significantly there (+9.0, +1.7), and on news (+5.9), where it reads our graph; on QASC its +1.5 is not significant (Table 2). Section 4.2 now says so.
W2. "Comparing a cohesion shortlist followed by the same LLM stage against the complete KNIT system would clarify the additional value of the spanning-tree and coverage formulation." Done, on all 25 pairs, with cohesion at a shared and at per-pipeline constants (GR-Q3). It is consistent with a role for the coverage term and shows no measured value of the spanning tree over cohesion's form. You are right that the stand-alone conclusion depends on the tuning range, and Section 5 now says cohesion comes closest when its constant is tuned per pipeline. The graph stage alone minus cohesion:
Followed by the same call (table with intervals in GR-Q3), KNIT is significantly ahead of cohesion's ten on HotpotQA and news, whether the constant is shared or per-pipeline, and level on MuSiQue and 2WikiMultihopQA; at the shared constant it also leads on QASC. Per pair at c = 0.5, cohesion is significantly ahead on three (MuSiQue's Hybrid, 2WikiMultihopQA's Hybrid and MonoT5). HotpotQA and news, where KNIT leads at both constants, are the corpora where removing coverage costs the graph stage most, which points to the coverage term (exploratory; GR-Q3 lists the other differences between the two). Inside the search both score a candidate by its strongest link to the chosen passages; the spanning tree enters only when the two routes' sets are compared, and removing the second route lowers no corpus mean significantly, so we claim no separate contribution for the tree (UPw1 Q4). Alone, cohesion comes close only with a constant tuned per pipeline, which KNIT does not need.