acceptodds
Under review as a conference paper at ICLR 2027

KNIT: Recovering the Passage That Top-k Truncation Cuts

38 views

Abstract

Retrieval-augmented generation (RAG) passes the top passages of a retrieval pipeline to the LLM that answers. For a multi-hop question, the evidence is complete only when the top holds every passage of the question's evidence chain. But top- selection scores each passage on its own, by how well it matches the question. The passage that completes a chain is often linked to the others only through a bridge entity the question never names, so it ranks low and falls outside the top . We introduce KNIT, a training-free layer that selects the top as a set. KNIT keeps the pipeline's relevance scores and adds links between passages that share an entity, read from a graph built once per corpus. A graph stage selects linked passages that together cover the question, and one LLM call keeps of them. We apply one configuration, fixed on MuSiQue, to five published retrieval pipelines on five corpora. We measure chain completion: the share of questions whose top 5 holds the whole chain. KNIT raises it on all 25 pipeline–corpus pairs, significantly on 23. On the three Wikipedia corpora, the average gain is 10.6 to 13.7 points. KNIT improves even PropRAG, the strongest pipeline on Wikipedia, which already reads such a graph during retrieval. KNIT's gains carry over to question answering on Wikipedia: a 72B reader answers significantly more questions from KNIT's passages on 13 of the 15 Wikipedia pairs. On MuSiQue, our analysis shows that the graph stage's gain comes from the entity links rather than from diversity.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
retrieval-augmented generation, multi-hop question answering, evidence selection, set selection, reranking, proposition graph, chain completion

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_BjuHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces KNIT, a training-free passage selector for multi-hop RAG. It combines existing retrieval scores with entity connections from a proposition graph and question coverage to select ten passages from thirty candidates. An LLM then retains five passages for answering. Experiments across five retrieval pipelines and five corpora report improvements in evidence-chain completion, with corresponding gains in answer accuracy on the Wikipedia datasets.

Critical strengths
  1. The paper addresses a practical limitation of top-k truncation, where a passage needed to complete a reasoning chain receives a low individual relevance score. Preserving retrieval scores while incorporating entity connections is well motivated.
  2. The evaluation covers lexical, dense, reranking and graph retrieval pipelines. Improvements on four corpora excluded from tuning provide useful evidence of transfer.
  3. The ablations and deletion tests offer informative evidence about the role of entity links. The analysis of their limitations on news also helps clarify when the method is effective.
Critical weaknesses
  1. The MuSiQue test is not fully independent of configuration selection. Appendix E states that the search space was expanded using PropRAG results from all 1,000 questions, including the nominal held-out half, after an earlier configuration failed to improve it significantly. This weakens the held-out evidence for the central claim about improving the strongest pipeline. The disclosure and results on untouched transfer corpora are valuable, but independent validation on fresh MuSiQue questions is still needed.

  2. The contribution of the specific graph objective remains insufficiently isolated. In Appendix G, the primary comparison with equal tuning budgets yields a 0.60-point gain over the simpler cohesion selector, with a confidence interval of [-0.64, 1.84]. Some broader searches favor the graph stage, but the conclusion depends on the tuning range. Comparing a cohesion shortlist followed by the same LLM stage against the complete KNIT system would clarify the additional value of the spanning-tree and coverage formulation.

  3. The practical advantage over direct LLM selection remains unclear. The same LLM reading all thirty candidates significantly outperforms KNIT on average on three corpora. KNIT reduces prompt tokens, but requires corpus-wide graph extraction. Table 7 evaluates collections containing roughly 6,000 to 12,000 passages, and Appendix L.2 replaces QASC’s 17-million-sentence collection with 7,671 annotated facts because full graph construction was infeasible. Although indexing costs are disclosed, a comparison accounting for corpus size and query volume is needed to establish the overall efficiency benefit.

  4. The confidence intervals for individual performance differences require correction. Appendix N rescales a Clopper-Pearson interval for the win rate among discordant pairs by the observed discordance rate. This treats the latter as fixed and omits its sampling uncertainty, so the resulting intervals are not exact 95% intervals for population performance differences. Valid intervals for paired proportions are needed. This issue does not invalidate the exact McNemar tests or the pooled bootstrap intervals.

Questions for authors

Please refer to the summary of weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer BjuH [1/2]

We thank the reviewer for the specific, checkable points, and for noting the breadth of pipelines, the evidence of transfer, the deletion tests and the analysis of news. We ran each experiment you proposed.

W1. "... independent validation on fresh MuSiQue questions is still needed." Done (GR-Q5; Sec. 4.1, App. I). On all 1,417 MuSiQue dev questions the paper never used, the frozen configuration improves all five pipelines significantly, with a mean of +10.28 [+9.15, +11.42], and by +13.20 on the 303 that share no single-hop sub-question with the 1,000 (1,114 share one). For PropRAG this run is a weaker test. There it reads our 14B graph, as on news and QASC in the paper: of the paper's corpora, PropRAG ships extracted propositions only for the three Wikipedia ones, and its documentation supports a new extraction only through a paid API (App. I). Its Top-5 there is 40.7, against 53.9 on the paper's questions, a fall far larger than the other pipelines' 0.6 to 3.3 points. Its gain is +7.55 [+5.32, +9.83] on the 1,417 and +3.96 [−1.08, +9.09] on the 303, not significant (n = 303). We agree that the held-out half is not fully independent for PropRAG, as the paper states, and that the MuSiQue PropRAG cell is therefore not independently established (App. I). The claim about the strongest pipeline rests on 2WikiMultihopQA and HotpotQA, which took no part in the choice and where PropRAG reads its own graph: KNIT improves it significantly there (+9.0, +1.7), and on news (+5.9), where it reads our graph; on QASC its +1.5 is not significant (Table 2). Section 4.2 now says so.

  • In the direction of your concern: when the graph stage is used alone, the second search route, chosen on MuSiQue, favours PropRAG there. Removing it lowers PropRAG's graph-stage gain on MuSiQue by 2.0 points, to +1.4, which is not significant; on the transfer corpora it raises no mean significantly and lowers 2WikiMultihopQA's and HotpotQA's. With the call, the route is not needed: KNIT without it and the near-duplicate rule improves PropRAG's MuSiQue Top-5 by +4.6 (p = 0.00034), against +4.2 with them (exploratory). The revision marks it optional (GR-Q4, GR-Q5; App. A).

W2. "Comparing a cohesion shortlist followed by the same LLM stage against the complete KNIT system would clarify the additional value of the spanning-tree and coverage formulation." Done, on all 25 pairs, with cohesion at a shared and at per-pipeline constants (GR-Q3). It is consistent with a role for the coverage term and shows no measured value of the spanning tree over cohesion's form. You are right that the stand-alone conclusion depends on the tuning range, and Section 5 now says cohesion comes closest when its constant is tuned per pipeline. The graph stage alone minus cohesion:

TuningGraph stage minus cohesion
Both over five points (primary)+0.60 [−0.64, +1.84]
Graph stage frozen, cohesion re-tuned per pipeline+0.12 [−1.12, +1.40]
Both over 49 points+1.20* [+0.08, +2.32]
Both over 49, cohesion's spread over [0, 1]+1.04 [−0.04, +2.16]
One shared constant (c = 0.5)+1.84* [+0.52, +3.20]

Followed by the same call (table with intervals in GR-Q3), KNIT is significantly ahead of cohesion's ten on HotpotQA and news, whether the constant is shared or per-pipeline, and level on MuSiQue and 2WikiMultihopQA; at the shared constant it also leads on QASC. Per pair at c = 0.5, cohesion is significantly ahead on three (MuSiQue's Hybrid, 2WikiMultihopQA's Hybrid and MonoT5). HotpotQA and news, where KNIT leads at both constants, are the corpora where removing coverage costs the graph stage most, which points to the coverage term (exploratory; GR-Q3 lists the other differences between the two). Inside the search both score a candidate by its strongest link to the chosen passages; the spanning tree enters only when the two routes' sets are compared, and removing the second route lowers no corpus mean significantly, so we claim no separate contribution for the tree (UPw1 Q4). Alone, cohesion comes close only with a constant tuned per pipeline, which KNIT does not need.

AuthorsOpenReview ↗

Response to Reviewer BjuH [2/2]

W3. "... a comparison accounting for corpus size and query volume is needed to establish the overall efficiency benefit." Added (GR-Q2; Sec. 2.2, App. A). Against the 30-passage reranker, which completes more chains than KNIT at ten on three corpora, each question saves about 6 to 7 GPU-seconds (1.6 on QASC's one-sentence facts), so KNIT's total cost, graph included, falls below the reranker's once a corpus has served 0.3 to 0.7 questions per passage. At accuracy matched to the reranker's, with the shortlist of 15 or 20 that closes its lead (20 on 2WikiMultihopQA), each question still saves 2.9 to 4.8 GPU-seconds and the total cost falls below the reranker's after 0.5 to 1.5 questions per passage (exploratory). For PropRAG on Wikipedia there is no graph to build. The saving per question comes from reading ten passages instead of thirty. What the graph buys is accuracy: over the same call run on the Top-10, which also reads ten passages, KNIT gains 3.2 to 5.1 points on Wikipedia and nothing measurable on news and QASC (GR-Q1). At ten, KNIT never falls below a pipeline's own Top-5, while the reranker falls below PropRAG's on MuSiQue; the longer shortlists give back 3.4 and 5.0 points of PropRAG's gain there (GR-Q2). For QASC's full 17M sentences, KNIT's total cost would fall below the reranker's only after about 5 million questions, a limit on corpus size that the revision states (App. A).

W4. "Valid intervals for paired proportions are needed." Agreed: the rescaled Clopper-Pearson interval treats the discordance rate as fixed. We recomputed all 234 single-pair intervals behind the main text's claims with Tango's score interval (Sec. 3, App. N). No significance call changes, nor any count of significant pairs, in the main text or elsewhere in the paper. Significance is still decided by the exact McNemar test, and Tango's interval excludes zero whenever that test rejects: it excludes zero exactly when McNemar's statistic exceeds 1.96, which holds for every split of up to 5,000 discordant questions whose exact p is below 0.05. The converse can fail: on four of the 234 pairs (exact p 0.053 to 0.061), and on seven pairs in the appendix figures, the new interval excludes zero while the test does not reject, so bold and filled marks are now defined by the exact test, in Table 2 and the figure captions. Endpoints move by a median of 0.36 points; the largest shift is 2.88 points, on held-out BM25, where KNIT wins all 47 discordant questions.

We hope these new results address your concerns, and we are happy to discuss any of them further.

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer BjuH,

Thank you again for your specific, checkable points. As the discussion period is about to close, we would like to kindly check whether our responses have addressed them. We ran each experiment you proposed: the frozen configuration on 1,417 fresh MuSiQue questions (W1), cohesion's ten followed by the same LLM call on all 25 pairs (W2), the cost by corpus size and query volume (W3), and Tango's score interval for all 234 single-pair intervals behind the main text's claims (W4). Our reply above gives each result, including those in the direction of your concern, and the revised PDF carries all of them. If they resolve your concerns, we would be grateful if you would consider updating your assessment. We are happy to answer any remaining question while the discussion is open.

Reviewer_EyDbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Most retrieval pipelines score each document based on its relevance to the original question. So on multi-hop questions, the passage that finishes the evidence chain can rank low and gets dropped, because it is linked to entities that you only learn after resolving some part of the question. To resolve this, KNIT picks the top passages as a group, adding links between passages that share an entity using a graph built once per corpus. It first picks candidates from the top passages that are relevant, connected, and together cover the question, and then uses one LLM call to narrow them down to a small final set. With one fixed setting, KNIT improves chain completion on all 25 pipeline-corpus pairs (significantly on 23), adds 10.6–13.7 points on average on the Wikipedia corpora, and even helps the strongest pipeline, PropRAG.

Critical strengths

S1. The paper targets a real-world challenge for top-k retrieval: for these questions, whether a passage is useful depends on understanding all the entities that the question cares about, but top-k scores each passage alone, just based on its relationship to the original question, and can miss the unnamed bridge entities. On MuSiQue, the passage linked through an unnamed bridge entity is the lowest-ranked correct passage on 64–87% of questions. S2. One fixed, training-free setting improves chain completion on all 25 pipeline–corpus pairs, with significant gains on 23. It also improves the strongest pipeline, PropRAG, on all five corpora, while common rerankers make it worse on MuSiQue (Table 3). On Wikipedia, a 72B reader answers significantly more questions on 13 of the 15 pairs. S3. The evaluation is overall rigorous and thorough.

Critical weaknesses

W1. Much of the gain comes from the LLM call alone. Just running the same LLM call on the pipeline's own top 10 already gets most of KNIT's improvement over the top 5. Comparing Table 2 with Fig. 2d, the graph step only adds +4.20, +5.08, and +3.16 points on the Wikipedia corpora on top of that call (about 23–43% of KNIT's total gain). It adds just +0.27 on news and −0.04 on QASC. Also, giving the same LLM the whole top 30 beats KNIT on the three weaker pipelines on MuSiQue (21.8 vs. 19.9, 40.4 vs. 36.1, 40.5 vs. 40.0; Table 3). KNIT only wins on the two strongest pipelines. The graph step alone does add +6–7 points on Wikipedia without any model call, which is useful. But on top of the LLM, it adds only a little, and almost nothing outside Wikipedia.

W2. The graph step is complex, and the paper doesn't show that all its parts are needed. Besides the three scoring terms, Appendix A also adds many hard-coded rules, which look really complex. But the experiments show that 108 tested settings give similar gains (+5.42 to +7.54; Appendix E). This confuses me, since most of these choices seem to matter little. What's more, a much simpler one-parameter method, like MMR with the sign of its second term flipped, gets within 0.6–1.8 points of the graph step. To me, the graph step pipeline feels very arbitrary, and I want to understand more about how the authors made these design choices.

Questions for authors

see weakness I’m willing to raise my score if the authors address the weaknesses well, especially if they can clarify that the pipeline isn’t unnecessarily complex or overfitting to the benchmarks.

3 comments
AuthorsOpenReview ↗

Response to Reviewer EyDb

We thank the reviewer for recognizing the real-world problem (S1) and the consistent gains of one fixed, training-free setting on all 25 pairs (S2), and for finding the evaluation rigorous and thorough (S3). The evidence for your two conditions is below and in GR-Q4 and GR-Q5.

W1. "Much of the gain comes from the LLM call alone." Yes: on Wikipedia the call on the Top-10 alone gains +6.40 to +10.52, 57 to 77% of KNIT's gain, and on news and QASC the graph stage adds nothing measurable once the call is present (GR-Q1). On Wikipedia the graph stage still adds +3.16 to +5.08 after the call, significantly on each corpus, and on MuSiQue, with no model call, it does as well as the call: the call handed the Top-10 only ties the graph stage alone there (Sec. 5). On MuSiQue the 30-passage reranker is significantly ahead on BM25 and Hybrid, at 2.7 times the tokens, and level on MonoT5; KNIT is ahead on the two strongest pipelines, and on PropRAG the reranker falls below the pipeline's own Top-5 (49.0 against 53.9) while KNIT raises it to 58.1. Handed 15 or 20 passages instead of ten, the call brings MuSiQue's BM25 and Hybrid level with the reranker but costs PropRAG 3.4 and 5.0 points (GR-Q2).

W2. "The graph step is complex, and the paper doesn't show that all its parts are needed." The new ablation shows which parts the graph stage needs (GR-Q4). The connection and coverage terms are needed (coverage on all five corpora, connection on the four other than news), and so is alias matching, on three corpora. The near-duplicate rule and the second search route are not needed for any corpus mean of the graph stage alone: without both (Table 9, row h; exploratory), it is never significantly below the full stage and is above it on three corpora; with the call, KNIT can drop both together and keep PropRAG's MuSiQue gain (GR-Q4; exploratory). What remains is one greedy fill from the pool's best-scoring pair that scores each candidate as cohesion does, by its relevance and its strongest entity link to the chosen passages, plus β times coverage (GR-Q4). The spanning tree of Eq. 1 enters only when the two routes' sets are compared, so it drops out with the second route; Section 2.1 now says so. The revision marks both rules optional (App. A), and every main result keeps the frozen configuration. The fixed lists and caps of Appendix A (stoplist, closed-class list, alias cap, title disambiguators, the two normalisations) were not ablated.

  • How the choices were made, and the 108 settings (Appendix E). The four main constants (pool depth, coverage rule, γ, β) were chosen on MuSiQue's tuning half by the lexicographic rule in the released selection file: first the number of pipelines a setting lifts significantly, then the mean. The 108 settings are the cells of the search's first two stages. Each keeps a mean gain of at least +5.42, so, as you say, these choices matter little to the mean. They matter for PropRAG, which falls to or below its Top-5 at 20 of them, and the rule favours it: on the tuning half the frozen setting is PropRAG's own best, while each other pipeline has a better setting of its own. After an earlier configuration failed to lift PropRAG significantly, we widened the grid using PropRAG's results on all 1,000 questions (BjuH W1). The spanning-tree shape, the two-route seeding, λ, the near-duplicate rule, the path budget and δ were then varied around that setting on MuSiQue; the second route, which helps only PropRAG before the call (GR-Q5), is the seeding chosen there.
  • "MMR with the sign of its second term flipped, gets within 0.6–1.8 points of the graph step." Alone, the graph stage is not significantly ahead of cohesion tuned for each pipeline (+0.60 [−0.64, +1.84], both sides tuned over five points); at one shared constant, as KNIT runs everywhere, it leads by +1.84 [+0.52, +3.20]. Followed by the same call, KNIT is significantly ahead of cohesion on HotpotQA and news in both settings and level on MuSiQue and 2WikiMultihopQA; with the shared constant it also leads on QASC. Its margins point to the coverage term, which cohesion lacks (GR-Q3; exploratory).

Q. "... or overfitting to the benchmarks." The gains hold on data the configuration never saw. The four other corpora took no part in choosing the configuration or the reading stage's form, and on the two other Wikipedia corpora KNIT gains as much as on MuSiQue, or more (Table 2). On 1,417 fresh MuSiQue questions the mean gain is +10.28, against +10.60 on the paper's 1,000, and +13.20 on the 303 that share no single-hop sub-question with them (GR-Q5). By the criterion we fixed beforehand the gain is "sensitive" to γ on HotpotQA and news, but it stays significant at every γ from 2.0 to 3.5 on every corpus but news. Where the choice favoured one pipeline, it was PropRAG, and the second route lowers two transfer corpora's means and raises none significantly.

We are happy to discuss further.

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer EyDb,

Thank you again for your questions on complexity and overfitting. As the discussion period is about to close, we would like to kindly check whether our responses have addressed them. On complexity, we removed the coverage and connection terms and three of the graph stage's rules one at a time, and set out what the graph stage adds beyond the LLM call (GR-Q4, GR-Q1). On overfitting, we ran the frozen configuration, unchanged, on 1,417 MuSiQue questions the paper never used, and varied γ on all 25 pairs (GR-Q5). Our reply above gives what each of these shows, including where the results limit our claims, and the revised PDF carries all of them. If they resolve your concerns, we would be grateful if you would consider updating your assessment. We are happy to answer any remaining question while the discussion is open.

Thank you for the rebuttal. It addressed some of my main concerns, I will keep my score.

Reviewer_UPw1OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This work investigates a failure of top-k retrieval for multi-hop questions: passages are scored one by one against the question, so the passage that completes the evidence chain, connected to the others only through a bridge entity the question never names, ranks low and is cut. To address it, the authors proposed KNIT, a training-free layer that selects the top k as a set, combining the pipeline's relevance scores with an entity-link connection term from a proposition graph built once per corpus and a coverage term; a greedy stage selects 2k passages and one LLM call reduces them to k. Different from existing work that focuses on per-passage scoring, KNIT optimizes the set jointly, recovering a weak passage when its links complete the chain. Experimental results verify its effectiveness and superiority: with one configuration on five pipelines and five corpora, chain completion rises on all 25 pairs (significantly on 23), by 10.6–13.7 points on the Wikipedia corpora, including on the strongest pipeline, PropRAG, and reader exact match improves significantly on 13 of 15 Wikipedia pairs.

Critical strengths
  1. Chain completion rises on all 25 pipeline–corpus pairs, significantly on 23, by 10.6–13.7 points on the Wikipedia corpora; comparisons are paired per question, so selection alone explains the gains.
  2. The extra-LLM-call confound is controlled: the same call on the pipeline's own top ten completes significantly fewer chains.
  3. Boundary conditions are clear: a link-strength diagnostic predicts where connection helps, and the failing news corpus is analyzed.
Critical weaknesses
  1. A plain zero-shot reranking call over the top-30 pool beats KNIT on three corpora and ties it on two — KNIT wins on cost, not accuracy (Table 20).
  2. Cost is understated: graph construction takes one 14B call per passage, about 40 GPU-hours across five corpora, and QASC's real 17M-sentence corpus was replaced by 7,671 closed facts (Table 7).
  3. Chain completion does not always pay off: the reader gains under a point on news, and on QASC it already scores 87.8% with no facts at all.
Questions for authors
  1. On the held-out half your selection rule returns a different cell (γ = 3.0) and the frozen cell falls to rank 22. How sensitive are the transfer-corpus results to that choice — i.e., is the frozen configuration a stable optimum or one point on a flat ridge that happened to protect PropRAG?
  2. On WikiMultihopQA the 30-passage reranker's edge is attributed to depth: chains present in the thirty but absent from the graph stage's ten. Why cap the shortlist at 2k, and what does a shortlist of 15 or 20 do to both accuracy and token cost?
  3. Every reader result uses one fixed passage order, and under the earlier configuration shuffling that order made the primary reader contrast non-significant. Why was the shuffle not rerun at the frozen configuration?
  4. Cohesion also selects sets jointly and, tuned per pipeline, matches the graph stage. Mechanistically, what separates KNIT's connection term from cohesion with its sign flipped — the bridge weights themselves, or the coverage term and search built around them?
3 comments
AuthorsOpenReview ↗

Response to Reviewer UPw1 [1/2]

We thank the reviewer for the precise questions, and for noting that comparisons are paired per question, that the extra-LLM-call confound is controlled and that the boundary conditions are clear. Q1 to Q3 are now new experiments, and the answer to Q4 uses a fourth (GR-Q3); the shared evidence is in the general response.

W1. "A plain zero-shot reranking call over the top-30 pool beats KNIT on three corpora and ties it on two — KNIT wins on cost, not accuracy." At KNIT's ten passages, yes on average: the reranker is ahead on 2WikiMultihopQA, HotpotQA and news by 1.38 to 2.86 points, reading 2.3 to 3.0 times KNIT's prompt tokens. Two things limit that lead. It is never significantly ahead on a corpus's strongest pipeline, and on MuSiQue it falls below PropRAG's own Top-5 (53.9 to 49.0), where KNIT raises it to 58.1; KNIT improves all 25 pairs. And this lead comes from reading deeper: handed 15 or 20 passages by the graph stage, the same call no longer trails the reranker significantly on any of the three corpora (on 2WikiMultihopQA only at 20), at 52 to 74% of its tokens (GR-Q2; Sec. 4.4, App. J). The longer lists cost the strongest pipeline on MuSiQue, though: they lower PropRAG by 3.4 and 5.0 points, and at 20 it ends 0.8 below its own Top-5 (not significant). On MuSiQue, then, the shortlist length trades tokens and the weaker pipelines' accuracy against the strongest pipeline's: ten is the cheapest setting and the one at which KNIT lifts PropRAG significantly. Where the reranker led, 20, and 15 except on 2WikiMultihopQA, match its average at lower cost.

W2. "Cost is understated: graph construction takes one 14B call per passage, about 40 GPU-hours across five corpora, and QASC's real 17M-sentence corpus was replaced by 7,671 closed facts." The revision now gives the accounting by corpus and query volume (GR-Q2; Sec. 2.2, App. A). The 40 GPU-hours over five corpora are 1.0 to 12.3 per corpus, paid once. Against the 30-passage reranker, each question saves about 6 to 7 GPU-seconds (1.6 on QASC's one-sentence facts), so KNIT's total cost, graph included, falls below the reranker's after 0.3 to 0.7 questions per passage, or after 0.5 to 1.5 at the shortlist length that matches the reranker's accuracy (exploratory). A pipeline that already builds a proposition graph adds no build cost: KNIT reads that graph, as it reads PropRAG's own on Wikipedia, where KNIT still improves PropRAG significantly (+9.0 on 2WikiMultihopQA, +1.7 on HotpotQA; Table 2). On news and QASC, though, the same call on the pipeline's own Top-10 is as accurate as KNIT without any graph (GR-Q1). QASC's 17M sentences remain a real limit: at the break-even rate we measure on QASC's facts, a corpus that size would need about 5 million questions before KNIT's total cost fell below the reranker's, which the revision states as a limit on corpus size (App. A). The graph also does not need our 14B extractor: on MuSiQue a 7B extraction gives nearly the same gains on the four pipelines without a graph of their own (+5.8 to +9.9, against +6.0 to +9.3; all significant; App. A).

W3. "Chain completion does not always pay off: the reader gains under a point on news, and on QASC it already scores 87.8% with no facts at all." On news and QASC the reader answers most questions without the full evidence, so its accuracy there cannot show what a complete chain is worth; where the answer depends on the evidence, a complete chain pays off. On the Wikipedia corpora a complete chain goes with 26 to 44 more exact-match points on 14 of 15 pairs, and KNIT's gain reaches the reader on 13 of the 15. On QASC a complete chain helps too: the reader answers 97.6 to 99.2% of the questions whose two gold facts are in the Top-5, against 86.5 to 88.9%, the no-facts floor, when they are not. But of the questions whose chain KNIT completes, the reader had already answered 88.1 to 92.3% under the Top-5, so completing them adds only +0.3 to +1.1 points (App. L). On news, 1,343 of the 2,255 questions are comparison or temporal questions answered "yes" or "no", and the reader answers 77 to 79% of the questions whose annotated chain is incomplete (App. K). The paper therefore states the reader claim for the Wikipedia corpora (Sec. 4.3) and keeps chain completion as the measure of what retrieval delivers.

AuthorsOpenReview ↗

Response to Reviewer UPw1 [2/2]

Q1. "How sensitive are the transfer-corpus results to that choice — i.e., is the frozen configuration a stable optimum or one point on a flat ridge that happened to protect PropRAG?" Little on 2WikiMultihopQA and QASC, where the gain is flat in γ; more on HotpotQA and news (Sec. 4.1, App. E). Had we frozen the held-out half's γ = 3.0, the graph stage alone would have gained +0.42 on 2WikiMultihopQA and lost 0.58 on HotpotQA and 0.46 on news (means over the five pipelines); it would have been significantly better only on 2WikiMultihopQA. From γ = 2.0 to 3.5 the gain stays significant on every corpus but news, though by the criterion we fixed beforehand it is "sensitive" on HotpotQA and news (GR-Q5). On MuSiQue the configuration does favour PropRAG: the selection rule landed on PropRAG's own best setting on the tuning half (App. E), and the second search route, chosen on MuSiQue, is what keeps its graph-stage gain significant there (+3.4 on all 1,000 with it, +1.4 without; on the held-out half it is +3.0, p = 0.053, even with it); that route also lowers the mean on 2WikiMultihopQA and HotpotQA by 1.3 and 1.5. With the call, the route is not needed: KNIT without it and the near-duplicate rule improves PropRAG's MuSiQue Top-5 by +4.6 (p = 0.00034), against +4.2 with them (GR-Q4; exploratory). Whether the PropRAG claim depends on this protection is also tested on the corpora that took no part in the choice, where PropRAG reads its own graph: KNIT improves it significantly, by +9.0 on 2WikiMultihopQA and +1.7 on HotpotQA (Table 2). Its graph-stage gain alone is significant at every γ on 2WikiMultihopQA (+3.1 to +3.4); on HotpotQA it is +0.0 to +0.6 and not significant at any γ, where PropRAG's Top-5 already completes 94.9 of 100 chains (App. E, I). The revision marks the route optional and states its effect on PropRAG (App. A).

Q2. "Why cap the shortlist at 2k, and what does a shortlist of 15 or 20 do to both accuracy and token cost?" The length itself was never tuned: 2k came with the reading stage's form, which we chose over an earlier form on MuSiQue's tuning half. Shortlists of 15 and 20 now run on all 25 pairs (GR-Q2). On the three corpora where the reranker led, both improve significantly on ten and close its lead at 52 to 74% of its tokens, except 15 on 2WikiMultihopQA (−0.96). On MuSiQue they bring BM25 and Hybrid level with the reranker but cost PropRAG 3.4 and 5.0 points, so the mean does not move. Ten is the cheapest point of this trade-off; 15 or 20 buy the reranker's accuracy at fewer tokens, at a cost to the strongest pipeline on MuSiQue.

Q3. "Why was the shuffle not rerun at the frozen configuration?" It now is. With each question's five passages in three seeded random orders, KNIT's exact-match gain over the Top-5 on MuSiQue (72B reader), averaged over the orders, is +4.63 [+3.64, +5.65] over the five pipelines, against +5.30 [+4.20, +6.42] in the selector's own order, and it is significant on all five pipelines and in each order alone. The earlier non-significant shuffle was run under the earlier configuration. As planned before the run, the fifth limitation (Sec. 8) now states this result; the other reader results still use one fixed order (App. K).

Q4. "Mechanistically, what separates KNIT's connection term from cohesion with its sign flipped — the bridge weights themselves, or the coverage term and search built around them?" Our evidence points to the coverage term, not the bridge weights, though no run isolates it; we could not separate the search's share. Both read the same entity links, and the links carry most of the gain: replaced by passage cosines, the graph stage loses 6.22 of its 6.92 points. Within the greedy search, a candidate's connection score is its strongest link to the passages already chosen, the same form cohesion uses. Cohesion has no coverage term, and removing coverage lowers the graph stage's mean on all five corpora, by 0.46 to 3.70. Followed by the same call, KNIT is significantly ahead of cohesion on HotpotQA and news and level on MuSiQue and 2WikiMultihopQA, whether cohesion's constant is shared or tuned per pipeline (GR-Q3); HotpotQA and news are the corpora where removing coverage costs most. The spanning tree over the whole set enters only when the search compares the sets its two routes build, and removing the second route lowers no corpus mean significantly. The search also starts from the best-scoring pair rather than the top passage, a choice we did not isolate. Section 2.1 now describes the search this way.

We hope these answers address your questions, and we are happy to discuss any of them further.

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer UPw1,

Thank you again for your precise questions. As the discussion period is about to close, we would like to kindly check whether our responses have addressed them. Your questions led to the shortlists of 15 and 20 passages, the cost accounting by corpus size and query volume, the γ sweep on all 25 pairs, the passage-order rerun at the frozen configuration and the comparison with cohesion followed by the same LLM call. Our reply above answers each of your points, and the revised PDF carries all of these. If they resolve your concerns, we would be grateful if you would consider updating your assessment. We are happy to answer any remaining question while the discussion is open.

Reviewer_irVZOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies a particular failure of retrieval-augmented generation: a retriever may place every required passage in its candidate pool, yet the final top-k cut still removes a passage needed to complete a multi-hop evidence chain. KNIT replaces this last step with a two-stage, training-free selector. Its graph stage combines the original retrieval scores with entity-based connections and question coverage to select ten passages from a pool of thirty. A 32B LLM then selects the final five. The evaluation covers five retrieval pipelines and five corpora, with chain completion as the main metric and downstream question answering as an additional check. KNIT improves Chain@5 in all 25 pipeline–corpus combinations, with statistically significant gains in 23, and improves reader accuracy on most Wikipedia settings.

Critical strengths

The paper identifies a meaningful problem in multi-hop RAG that is not fully addressed by improving retrieval relevance alone. The distinction between retrieving the necessary evidence and retaining it within a limited context budget is important. I also appreciate that KNIT operates on the same candidate pool and final passage budget as the original pipelines, which makes the improvements in chain completion easier to interpret. The gains on PropRAG are particularly interesting, since this pipeline already incorporates proposition-level structure during retrieval. Another strength is the depth of the experimental analysis. Beyond the main results, the authors examine the contribution of different objective terms, replace entity connections with embedding similarities, and investigate how removing an earlier evidence passage affects the selection of a dependent passage. I found the nested-pool analysis in Section 5 and Appendix C especially useful, as it provides a more concrete characterization of how the proposed selector differs from a fixed per-passage ranking. The paper also reports negative results on the news corpus and cases where the proposed method breaks previously complete evidence chains. These analyses make the conclusions more informative and credible.

Critical weaknesses

(1) The incremental benefit of the proposed graph-based selection over simpler set-selection alternatives is not entirely clear. While the comparison with conventional diversity methods is informative, the results in Appendix G suggest that a simpler cohesion-based selector can be competitive when appropriately tuned. In the primary matched-tuning comparison, the graph stage is ahead of the tuned cohesion baseline by only 0.60 points on the five-pipeline held-out mean, with a confidence interval of [−0.64, +1.84]. Appendix H also reports a comparison with per-pipeline retuned cohesion where the difference is only +0.12 [−1.12, +1.40]. Neither result establishes a significant advantage. These findings do not invalidate the improvements over standard top-k selection, but they make it harder to determine how much additional value comes from the specific objective and search procedure proposed in KNIT, beyond rewarding connections between passages. I think this distinction matters for assessing the methodological contribution. The authors discuss these comparisons in the appendices, but their implications deserve greater emphasis in the main paper. (2) Some experimental comparisons make it difficult to isolate the benefit of KNIT, and its practical advantages are less consistent outside Wikipedia. In Table 3, the off-the-shelf rerankers generally replace the original pipeline scores, whereas KNIT explicitly retains them. As a result, the comparison reflects both the benefit of preserving a strong initial ranking and that of selecting passages jointly. The score-augmented comparisons with Provence and Set-Encoder in Appendix D are helpful, but a more directly comparable score-preserving set-selection baseline would make the central claim stronger. There is also a noticeable difference between improvements in chain completion and improvements in downstream question answering. The Wikipedia results are encouraging, but the mean reader gain is only 0.84 points on news and −0.30 points on QASC. Section 6 further shows that entity connections alone can actually hurt chain completion on news. This suggests that the effectiveness of the proposed structural signal depends considerably on the relationship between shared entities and the underlying evidence chains. Finally, KNIT requires an offline proposition graph and an additional 32B LLM call for each question. The same-model 30-passage reranker achieves higher average chain completion on several corpora, although at a higher token cost. I therefore see KNIT as offering a potentially useful quality–cost trade-off, rather than a consistently stronger selection method. A clearer discussion of this trade-off would help establish its practical significance.

Questions for authors

(1) Could the authors discuss the cohesion-based comparisons in Appendix G more directly? In particular, the primary matched-tuning comparison shows a relatively small and statistically insignificant difference between the graph stage and tuned cohesion. What evidence most clearly establishes the additional benefit of KNIT's particular combination of connection, coverage, and search, compared with a simpler cohesion-based selector using the same entity information? (2) Could the authors provide a more controlled comparison against score-preserving alternatives? Since KNIT retains the original retrieval scores while several reranking baselines replace them, it is difficult to isolate the contribution of joint passage selection. I appreciate the modifier experiments in Appendix D, but it would be useful to understand whether the main conclusions remain unchanged when competing selectors are given the same relevance information and comparable tuning budgets. (3) I have two related questions about the generality of the results. First, Appendix E indicates that an earlier PropRAG result on the held-out data influenced the subsequent parameter search. How should this affect the interpretation of the reported PropRAG improvement on MuSiQue? Second, given the limited reader gains on news and QASC, and the additional graph construction and LLM inference costs, what would the authors consider the most appropriate deployment setting for KNIT? In particular, it would help to clarify whether the method is primarily intended for relatively stable, entity-rich corpora rather than more general RAG applications.

3 comments
AuthorsOpenReview ↗

Response to Reviewer irVZ [1/2]

We thank the reviewer for the careful review, and for valuing the same-pool, same-budget comparison, the gains on PropRAG, the depth of the analysis and the reported failures. The other reviewers also raised cohesion, the 30-passage reranker and PropRAG on MuSiQue; the general response gives the shared evidence, and the revised PDF (changes in blue) now reports this evidence in the main text.

W1 and Q1. "What evidence most clearly establishes the additional benefit of KNIT's particular combination of connection, coverage, and search, compared with a simpler cohesion-based selector using the same entity information?" Cohesion is a simple selector that rewards KNIT's entity links: it is our own variant of MMR, run on KNIT's graph. Beyond the links, our evidence points to the coverage term (exploratory). The search can be simpler than in the paper, and we claim no share for the parts we removed.

  • The links. On MuSiQue, with passage cosines in place of the links, the graph stage loses 6.22 of its 6.92-point mean gain over the five pipelines. Selectors that reward diversity instead, MMR, DPPs and facility location, gain at most 1.32 points pooled over the held-out halves of ten MuSiQue pipelines, even at their own per-pipeline best (Sec. 5).
  • Coverage. Removing it significantly lowers the graph stage's gain on all five corpora, and cohesion has no such term. Followed by the same LLM call, KNIT is significantly ahead of cohesion's ten on HotpotQA and news and level on MuSiQue and 2WikiMultihopQA, whether cohesion uses one constant for every pipeline, as KNIT does, or a constant per pipeline tuned for selecting five, not ten; at the shared constant it is also ahead on QASC. Per pair at that constant, cohesion is significantly ahead on three of 25. HotpotQA and news, where KNIT leads at both constants, are the corpora where removing coverage costs the graph stage most (GR-Q3). On news and QASC, though, the call on the Top-10 alone already matches KNIT (GR-Q1), so on the Wikipedia corpora, where the graph adds to the call, KNIT leads cohesion only on HotpotQA.
  • The graph stage alone. Here the comparisons you cite stand. In the primary comparison, with both sides tuned per pipeline over five settings, the graph stage leads by +0.60 [−0.64, +1.84]; with only cohesion re-tuned, by +0.12 [−1.12, +1.40]; neither is significant. With one constant for all pipelines, as KNIT runs, it leads by +1.84 [+0.52, +3.20], a secondary reading (App. G, H).
  • The search. It can be simpler than in the paper: with its call, KNIT without the second route and the near-duplicate rule is never significantly worse on any corpus, and is significantly better on QASC (GR-Q4). The spanning tree, used only to compare the two routes' sets, then drops out. We did not separately ablate starting the search from the best-scoring pair.

Section 5 now reports both cohesion comparisons in the main text, as you suggest; GR-Q3 gives them per corpus.

W2 and Q2. "Could the authors provide a more controlled comparison against score-preserving alternatives?" The paper has three kinds of selector that keep the pipeline's scores. On MuSiQue, keeping the pipeline's scores narrows the plug-ins' losses, but tuning the weight they give their own score would at best leave PropRAG as it was (first bullet). Of the selectors that keep the scores, only the one that rewards KNIT's entity links comes close to the graph stage.

  • The two plug-ins with the pipeline's score added to their own (App. D). On Dense and PropRAG they lose less than half as much as when they replace the score, but they still lower PropRAG significantly. Tuning the weight of their own score per pipeline would set it to zero on PropRAG, so they would return PropRAG's own Top-5 unchanged.
  • MMR, DPPs and facility location, which trade the pipeline's score against diversity, each at its per-pipeline best, a tuning freedom KNIT's single configuration does not have (App. G): at most +1.32 as a ten-pipeline mean, and each significantly below the graph stage (Sec. 5).
  • Cohesion, which keeps the score and rewards KNIT's links (W1).

The first two comparisons were run on MuSiQue. RankZephyr, which is handed the pipeline's ranked list, also falls below PropRAG's Top-5 there (Sec. 4.4).

AuthorsOpenReview ↗

Response to Reviewer irVZ [2/2]

W2. "... the effectiveness of the proposed structural signal depends considerably on the relationship between shared entities and the underlying evidence chains." We agree, and Section 6 draws this boundary. The two passages joined by a question's strongest entity link are a gold pair about 14 times as often as a random pair on MuSiQue and 26 times on QASC, but only a third as often on news. On news, connection alone lowers chain completion on all five pipelines, while coverage alone raises it. The reader changes little on news and QASC for another reason: it answers most of these questions without a complete chain. On QASC it answers 87.8% with no facts at all, yet a complete chain still helps: it answers 97.6 to 99.2% of the questions whose two gold facts are in the Top-5, against 86.5 to 88.9% otherwise. It had already answered most of the questions whose chain KNIT completes, so completing them adds +0.3 to +1.1 points (App. L). On news, 1,343 of the 2,255 questions are comparison or temporal questions whose answer is "yes" or "no", and the reader answers questions with an incomplete chain at least as often as those with a complete one (Sec. 4.3, App. K). On Wikipedia, a complete chain goes with 26 to 44 more exact-match points on 14 of the 15 pairs, and KNIT's gain reaches the reader on 13 (UPw1 W3).

W2. "... a potentially useful quality–cost trade-off, rather than a consistently stronger selection method." On average, yes: at ten passages the 30-passage reranker is ahead on three corpora, at 2.3 to 3.0 times KNIT's prompt tokens, and Section 4.4 now presents this as a trade-off. On each corpus's strongest pipeline, though, it never beats KNIT significantly, and on QASC's Dense KNIT is significantly ahead (Sec. 4.4). Handed 15 or 20 passages by the graph stage, KNIT's call no longer trails it significantly on average (on 2WikiMultihopQA only at 20), at 52 to 74% of its tokens; per pipeline the reranker stays significantly ahead on 2WikiMultihopQA's Hybrid and MonoT5 at both lengths and on HotpotQA's Hybrid at 15, and the longer lists cost MuSiQue's PropRAG 3.4 and 5.0 points (GR-Q2).

Q3. "How should this affect the interpretation of the reported PropRAG improvement on MuSiQue?" As not independently established, which Section 4.2 now states: PropRAG's held-out half also helped choose the configuration. PropRAG's reader results on MuSiQue call for the same caution; Section 4.3 also reports them on the held-out half alone. The claim about PropRAG rests on 2WikiMultihopQA and HotpotQA, which took no part in tuning: there KNIT improves its chain completion significantly (+9.0 and +1.7; Table 2), though neither gain reaches the reader (Sec. 4.3). Fresh MuSiQue questions are a further check, though PropRAG reads our graph there and its Top-5 is 13.2 points lower. On all 1,417 of them KNIT improves PropRAG by +7.55 [+5.32, +9.83]; on the 303 that share no single-hop sub-question with the paper's questions, by +3.96 [−1.08, +9.09], which is not significant (GR-Q5).

Q3. "... what would the authors consider the most appropriate deployment setting for KNIT?" KNIT is built for multi-hop questions, where a top-k cut can break an evidence chain, and it suits corpora like our three Wikipedia corpora, rather than RAG in general. There KNIT completes 3.2 to 5.1 points more chains than the same call on the pipeline's Top-10; on news and QASC, that call alone is as accurate as KNIT and needs no graph (GR-Q1). How many entities a corpus has is not the test. The links help only where shared entities mark the hops of a chain; on news they rarely do (Sec. 6). That is necessary but not sufficient: on QASC they do, yet the call alone already matches KNIT there. Checking a new corpus therefore needs some questions with known evidence chains. On cost, what matters is how often a corpus is queried. Counting the graph, KNIT costs less than the 30-passage reranker after 0.62 to 0.66 questions per passage on the Wikipedia corpora, or, at the shortlist that matches the reranker's accuracy, after at most 1.5 (App. A; exploratory). A pipeline that already builds a proposition graph, as PropRAG does on Wikipedia, pays nothing for it: KNIT reads that graph (UPw1 W2). On MuSiQue a 7B graph also serves the graph stage nearly as well as our 14B one (App. A). A large corpus that is rarely queried does not repay the graph: at QASC's full 17M sentences, about 5 million questions would be needed (App. A).

We hope these answers address your concerns, and we are happy to discuss any of them further.

AuthorsOpenReview ↗

Kind reminder

Dear Reviewer irVZ,

Thank you again for your careful review. As the discussion period is about to close, we would like to kindly check whether our responses have addressed your questions. Our reply above sets out what KNIT adds over a cohesion-based selector using the same entity information, which selectors in the paper keep the pipeline's scores, the quality-cost trade-off against the 30-passage reranker (now so described in Section 4.4), how the MuSiQue PropRAG result should be read (now stated in Section 4.2), and the deployment setting KNIT suits. If these resolve your concerns, we would be grateful if you would consider updating your assessment. We are happy to answer any remaining question while the discussion is open.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response (1/3): summary and new experiments

We thank the reviewers for careful, constructive reviews, and are glad they found that:

  • the top-k blind spot is a real, practical problem (EyDb, BjuH) that better relevance alone does not fully address (irVZ), and keeping the pipeline's scores while adding entity links is well motivated (BjuH);
  • one frozen, training-free configuration improves chain completion on all 25 pipeline-corpus pairs, significantly on 23 (UPw1, EyDb), on each pipeline's own pool and budget (irVZ), with useful evidence of transfer to the four corpora excluded from tuning (BjuH);
  • the evaluation is rigorous and thorough (EyDb), covers lexical, dense, reranking and graph pipelines (BjuH), and controls the extra-LLM-call confound (UPw1);
  • the deletion tests and the analysis of news show what the entity links do and where they help (UPw1, BjuH, irVZ); the nested-pool analysis is especially useful, and the reported failures add credibility (irVZ).

In brief. What the new evidence shows, including where it goes against us:

  • The gains replicate on 1,417 MuSiQue questions the paper never used (+10.28, against +10.60 on its 1,000). The claim about the strongest pipeline, PropRAG, rests on 2WikiMultihopQA and HotpotQA, which took no part in tuning (+9.0, +1.7). By our fixed criterion the graph stage's gain is "sensitive" to γ on HotpotQA and news, though it stays significant at every γ on four corpora.
  • Against KNIT's ten passages, the same LLM reading all thirty candidates is more accurate on three corpora, at 2.3 to 3.0 times KNIT's prompt tokens, but never significantly on a corpus's strongest pipeline. Handed 15 or 20 passages by the graph stage, KNIT's call closes that lead (on 2WikiMultihopQA only at 20) at 52 to 74% of the reranker's tokens, at a cost to MuSiQue's PropRAG.
  • After the LLM call, the graph stage adds 3.2 to 5.1 points on Wikipedia and nothing measurable on news and QASC.
  • For the graph stage alone, coverage is needed on all five corpora, connection on four (on news it hurts) and alias matching on three; two implementation rules are not needed and are now marked optional. In full KNIT, with the call, dropping both rules never significantly lowers a corpus mean, significantly raises it on QASC and keeps PropRAG's MuSiQue gain significant (+4.6), whereas the graph stage alone needs the second route for that (GR-Q4; exploratory). Against cohesion followed by the same call, KNIT is significantly ahead on HotpotQA and news and level on MuSiQue and 2WikiMultihopQA; its margins point to the coverage term (exploratory).

Terms. Chain@5: the share of questions whose five passages hold every gold passage. Top-5, Top-10: the pipeline's own top five and ten. Graph stage: KNIT's first stage, which calls no model; before the call it selects ten of the pipeline's top 30 (five when used alone). The call: KNIT's one LLM call, which keeps five of those ten. 30-passage reranker: the same LLM picking five of all thirty in one call. Cohesion: greedy selection that rewards a candidate's strongest entity link to the passages already chosen (MMR with the sign flipped). Means are over a corpus's five pipelines; * marks an interval that excludes zero.

New experiments. All are in the revised PDF, with changes in blue. The few analyses we devised only after seeing earlier results are labelled "exploratory" wherever they appear.

#New resultAsked byWhere
1Fresh MuSiQue: 1,417 dev questions the paper never usedBjuH, EyDbSec. 4.1, App. I
2Cohesion's ten + the same LLM call, at shared and per-pipeline constantsBjuH, UPw1, EyDb, irVZSec. 5, App. G
3Shortlists of 15 and 20 vs the 30-passage rerankerUPw1, EyDb, BjuH, irVZSec. 4.4, App. J
4Cost by corpus size and query volumeUPw1, BjuH, irVZSec. 2.2, App. A
5Removing each set term and three rulesEyDb, UPw1, irVZSec. 5, App. A
6γ from 2.0 to 3.5 on all 25 pairsUPw1, EyDbSec. 4.1, App. E
7Passage order at the frozen configurationUPw1Sec. 8, App. K
8Tango's score interval for every per-pair differenceBjuHSec. 3, App. N
92×2 decomposition of graph stage and call (exploratory)EyDbSec. 5, App. J
10KNIT, with its call, without the two optional rules (exploratory)EyDb, BjuH, UPw1, irVZApp. A

Text changes besides these. Section 2.1 now says what the search scores and where the spanning tree enters (EyDb, UPw1 Q4); App. I.1 clarifies that PropRAG ships extracted propositions for the three Wikipedia corpora without a built index, correcting our submission (BjuH W1); the two optional rules are marked in App. A (EyDb); the corpus-size limit at QASC's full 17M sentences is stated in App. A (UPw1 W2, BjuH W3); bold and filled marks follow the exact test (BjuH W4).

AuthorsOpenReview ↗

General response (2/3)

GR-Q1. What the graph stage adds beyond the LLM call (EyDb W1, UPw1 W1). On Wikipedia, 3.2 to 5.1 points; on news and QASC, nothing measurable. The 2×2 cells are already in the paper (Table 2, Figure 2d). On Wikipedia the graph stage, with no model call of its own, adds 6.0 to 7.1 points before the call and 3.2 to 5.1 after it, and KNIT is significantly above either stage alone on each Wikipedia corpus. On Wikipedia, the call applied to the Top-10 gains 6.4 to 10.5 points (57 to 77% of KNIT's gain), while on MuSiQue it only ties the standalone graph stage (Sec. 5). On Wikipedia the two overlap (interaction −0.88 to −3.98, each interval below zero; exploratory), so each adds less when it comes second. On QASC's PropRAG the call on the Top-10 alone is significantly better, by 3.1 points.

GR-Q2. The 30-passage reranker (UPw1 W1, EyDb W1, BjuH W3, irVZ W2). Against KNIT's ten, the reranker is more accurate on three corpora; a longer shortlist closes its lead at fewer tokens, at a cost to the strongest pipeline on MuSiQue (new, all 25 pairs). Means minus the reranker, Chain@5 points:

CorpusKNIT (10)15, same call20, same callTokens, % of the reranker's (15 / 20)
2WikiMultihopQA−2.86*−0.96*−0.4057 / 74
HotpotQA−1.38*−0.22+0.0052 / 68
news−1.48*+0.76*+1.09*53 / 69

By the rule we fixed beforehand, 15 or 20 closes the gap on all three (on 2WikiMultihopQA only 20). Per pipeline, the reranker stays significantly ahead on 2WikiMultihopQA's Hybrid and MonoT5 at both lengths, and on HotpotQA's Hybrid at 15. On MuSiQue and QASC, where it did not lead, the mean does not move significantly: on MuSiQue the longer lists bring BM25 and Hybrid level with the reranker but cost PropRAG 3.4 and 5.0 points, and at 20 the strongest pipelines of MuSiQue and QASC end 0.8 and 0.9 below their own Top-5 (neither significant). Against KNIT at ten, the reranker is never significantly ahead on a corpus's strongest pipeline, and on MuSiQue it falls below PropRAG's own Top-5 (53.9 to 49.0), where KNIT raises it to 58.1. Cost: the reranker reads 2.3 to 3.0 times KNIT's prompt tokens and, in the paper's own runs on two A40s, takes 6.9 s a question on MuSiQue against KNIT's 3.9 s (generation only). KNIT's total cost, graph included, falls below the reranker's after 0.3 to 0.7 questions per passage, 2,300 to 7,400 questions on our corpora, and after 0.5 to 1.5 at the shortlist that matches the reranker's accuracy (exploratory). PropRAG on Wikipedia needs no new graph. On news and QASC the same call on the pipeline's Top-10 is as accurate as KNIT and needs no graph at all (GR-Q1).

GR-Q3. The objective vs cohesion (BjuH W2, UPw1 Q4, EyDb W2, irVZ W1). Followed by the same call, KNIT is significantly ahead of cohesion on HotpotQA and news and level on MuSiQue and 2WikiMultihopQA, whether cohesion's constant is shared or tuned per pipeline; at the shared constant it is also ahead on QASC. Its margins point to the coverage term. New, on all 25 pairs: cohesion selects ten, the same LLM call keeps five. KNIT minus this arm:

CorpusCohesion at c = 0.5Cohesion tuned per pipeline
MuSiQue+0.64 [−0.20, +1.50]+0.12 [−0.70, +0.94]
2WikiMultihopQA+0.04 [−0.58, +0.68]+0.18 [−0.50, +0.84]
HotpotQA+2.66* [+1.78, +3.56]+2.92* [+2.04, +3.82]
news+7.24* [+6.47, +8.02]+6.56* [+5.84, +7.29]
QASC+1.27* [+0.54, +2.01]+0.13 [−0.58, +0.84]

c = 0.5 is the constant cohesion's tuning picks on all five pipelines' tuning halves pooled: one constant for all, as in KNIT. The per-pipeline constants are those tuned for selecting five, not re-tuned for this two-stage form. At c = 0.5 KNIT is significantly ahead on three of five corpora, which our fixed rule reads as the objective adding over cohesion; against cohesion tuned per pipeline, on two. Per pair at c = 0.5, cohesion is significantly ahead on three of 25 (MuSiQue's Hybrid, 2WikiMultihopQA's Hybrid and MonoT5). Inside the greedy search both score a candidate by its strongest entity link to the passages already chosen, and cohesion has no coverage term. HotpotQA and news, where KNIT leads at both constants, are the corpora where removing coverage costs the graph stage most (GR-Q4; Table 9, row g), which points to the coverage term (exploratory: the two selectors also differ in their seed, the near-duplicate rule, the second route and the weighting of the link term). The spanning tree enters only when the search compares the sets its two routes build, and we claim no separate share for it. The submission did not run cohesion on QASC; those cells come from the same code, which reproduces cohesion's earlier selections on the other 20 pairs exactly.

AuthorsOpenReview ↗

General response (3/3)

GR-Q4. Is the graph stage more complex than it needs to be? (EyDb W2, irVZ Q1) For the graph stage alone, all the parts we removed are needed except two implementation rules, which the revision marks optional; with its call, KNIT is never significantly worse without them (exploratory).

  • Each set term is needed: removing coverage lowers the graph stage's mean gain on all five corpora, by 0.46 to 3.70 points, and removing connection lowers it on the four other than news, by 1.77 to 6.00; on news, where shared entities rarely mark hops (Section 6), removing connection helps.
  • Of the three rules we removed one at a time, alias matching is needed on three corpora, by up to 3.5 points. The near-duplicate rule and the second search route never raise a corpus's mean significantly: the second route lowers it on 2WikiMultihopQA and HotpotQA, and the near-duplicate rule on QASC. Only PropRAG gains from the second route (GR-Q5). Without both rules (Table 9, row h; exploratory), the stage is never significantly below the full one on a corpus mean and is above it on those three corpora by 0.65 to 1.48.
  • What remains is one greedy fill from the pool's best-scoring pair: each step adds the candidate with the highest relevance, plus γ times its strongest entity link to the passages already chosen, plus β times the coverage of the set with it added. Outside these ablations and the run below, all results in the paper keep both rules, so that they all come from one frozen configuration (App. A).
  • These ablations measure the graph stage alone, so we also ran KNIT itself, with its call, without both rules on all 25 pairs (exploratory; App. A). It is never significantly below KNIT, on a corpus mean or on any pair, and is significantly above it on QASC and on 2WikiMultihopQA's MonoT5:
CorpusWithout both rules, minus KNIT
MuSiQue+0.10 [−0.40, +0.60]
2WikiMultihopQA+0.18 [+0.00, +0.38]
HotpotQA+0.14 [−0.02, +0.32]
news−0.02 [−0.15, +0.12]
QASC+0.41* [+0.04, +0.82]

Its gain over each pipeline's own Top-5 is significant on the same 23 of 25 pairs as KNIT's. On MuSiQue it still improves PropRAG's Top-5 significantly, by +4.6 against +4.2 with the rules: without both rules, PropRAG's MuSiQue gain loses significance only when the graph stage is used alone (+1.4 there).

GR-Q5. Overfitting and held-out independence (BjuH W1, UPw1 Q1, EyDb, irVZ Q3). The gains replicate on 1,417 fresh MuSiQue questions, and the graph stage's gain stays significant at every γ on four of five corpora; choices made on MuSiQue favour PropRAG, so the claim about PropRAG rests on the transfer corpora.

  • Fresh MuSiQue questions. We ran the frozen configuration, unchanged, on all 1,417 MuSiQue dev questions the paper never used, with a corpus built from their own paragraphs:
QuestionsMean gain, five pipelinesPropRAG (its Top-5)
Paper's 1,000+10.60+4.2 (53.9)
Fresh 1,417 (all five significant)+10.28 [+9.15, +11.42]+7.55 [+5.32, +9.83] (40.7)
Fresh 303 sharing no single-hop sub-question+13.20 [+10.69, +15.78]+3.96 [−1.08, +9.09]

On the fresh questions PropRAG reads our 14B graph, as in the paper's news and QASC columns; its Top-5 falls by 13.2 points, against 0.6 to 3.3 for the other pipelines. 1,114 of the 1,417 share a single-hop sub-question with the paper's 1,000; the 303 that share none are two- and three-hop only.

  • γ from 2.0 to 3.5 on all 25 pairs (graph stage alone). The gain stays significant at every γ on MuSiQue (+6.78 to +7.06), 2WikiMultihopQA, HotpotQA and QASC; only on news, where links rarely mark hops, is the small gain lost above γ = 2.5. By the criterion we fixed beforehand (every γ significant and within 1.0 point of 2.5), the reading is "sensitive", not "flat", because of two corpora: γ = 3.5 costs HotpotQA 1.4 points (its gain is still +5.78), and news loses significance. γ = 3.0, the held-out half's own choice, would have been significantly better only on 2WikiMultihopQA (+0.42).
  • The selection rule and the grid widened on PropRAG's results favour it (App. E; UPw1 Q1, EyDb W2), and so does the second search route, chosen on MuSiQue (App. A): removing it lowers PropRAG's graph-stage gain on MuSiQue by 2.0 points, to +1.4, which is not significant, though KNIT, with its call, keeps a significant PropRAG gain on MuSiQue without it and the near-duplicate rule (+4.6; GR-Q4; exploratory). For the graph stage alone, on no transfer corpus does the route raise the mean significantly, and it lowers 2WikiMultihopQA's and HotpotQA's by 1.3 and 1.5 points.

We thank the reviewers again and are happy to answer further questions; each reply below gives the detail for that reviewer.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.