- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the multi-hop evidence selection problem: given a pool of candidate passages, select a small set that jointly provide complete support for an answer within a limited context budget. The authors argue that relational indexes expose connections across passages but do not by themselves determine which evidence should be retained. They propose HKVM, which extracts provenance-preserving relation triples from candidate passages, groups them around shared bridge entities to form bridge-group hypergraph keys while keeping original passages as values, performs query-conditioned weighted propagation over the hypergraph to compute structural signals per passage, and trains a supervised passage-level selector that combines structural signals with text retrieval features to rank candidates under a fixed context budget.
Critical strengths
- Exemplary experimental methodology with matched comparison designs that separate learning effects from organizational effects, and diagnostic analyses that go far beyond standard ablation studies to reveal mechanism.
- The key-value separation means HKVM can work with any base retrieval source, and the supervised selector can integrate any available features.
Critical weaknesses
- Hypergraph construction and propagation add complexity, and the paper does not adequately explore whether simpler features fed into the same supervised selector could achieve similar results.
- Evaluation uses a fixed budget of at most five passages. In practice context budgets vary, and HKVM’s performance across different budget sizes or adaptive budget strategies is unclear.
- Modest absolute improvement magnitude, support-selection F1 gains, while consistent, are numerically small, and it is unclear how much this selection-level improvement translates into final answer quality gains.
Questions for authors
- Has HKVM been evaluated in an end-to-end RAG setting where the candidate pool is generated by a first-stage retriever rather than fixed? How much do selection-level improvements translate into final answer accuracy gains?
- How does HKVM perform compared to simply adding entity overlap or relation count features to a text passage ranker? Does hypergraph propagation provide unique signals that simple structural features cannot capture?
- The set-level comparison found no significant advantage of group-completion features over pairwise interactions. In what scenarios do you expect explicit set-level modeling to yield greater benefits?
2 comments
Thank you for the detailed response and additional experiments. The matched simple-feature ablations and downstream QA evaluation partially address my concerns. My remaining concern is the practical significance of the structural stage, that is, fine-tuned BGE remains stronger, the tested readout fusion provides no additional gain, and the bounded retrieved-pool evaluation does not establish an advantage for HKVM. The proposed scenarios for set-level modeling are plausible, but its benefit over pairwise interactions remains unverified.
Response to Reviewer 2VU1
We thank the reviewer for the focused questions about simpler features, context budgets and answer quality. They prompted new controlled comparisons and four-reader QA, and helped us reorganize the paper around the information that relational membership contributes to budgeted evidence selection.
1. Comparison with simple structural features (W1, Q2). New Table 2A compares eight conditions in the same 34-slot passage MLP, with identical training rows, initialization and feature positions. Simple8 adds tuple/entity counts, question–entity overlap and coverage, relation-token coverage and cross-passage shared-entity counts to the text features. It improves selection on every dataset. Adding unit-weight endpoint/group readouts further raises all-support retention by 1.68/0.59/3.18 pp and support F1 by 0.44/0.26/0.96 pp on 2Wiki/MuSiQue/HotpotQA. The joint readout exceeds either updated path alone on both metrics in all three datasets (§5.2; Appendix E.1).
These controls show that which passages participate in particular connections carries information beyond aggregate counts and overlap. Table 2B separately examines readout after edge learning: one update improves retention over direct seed readout by 7.41/2.47/3.08 pp. The stronger-text experiment adds an important distinction: unit readouts do not improve a fixed fine-tuned BGE selector (Appendix F.2). Relational information is useful in the common learner, while its additional utility depends on the text model and fusion.
2. Context budget (W2). New Table 4 and Appendix H.1 report precision, recall and all-support retention at budgets 1/3/5/8/10; Figure 9 compares all seven methods at intermediate budgets. Moving from five to eight passages raises native retention from 93.31% to 97.94% on 2Wiki, 34.71% to 44.79% on MuSiQue and 79.96% to 93.48% on HotpotQA, with lower precision. Five provides a compact operating point in this precision–completeness trade-off. This experiment evaluates prefixes of fixed rankings; adaptive stopping and budget-dependent answer quality remain open.
3. Final answers and retrieved candidates (W3, Q1). Table 3 now reports EM/token F1 for five selection policies and four readers on the same 2,048 questions per dataset. HKVM achieves the highest answer F1 with three of four readers on both 2Wiki and MuSiQue, and exceeds learned ColBERT in both metrics in all twelve dataset–reader comparisons. These gains show that the retained evidence improves downstream answers across distinct readers. BM25 leads on HotpotQA. Appendix G includes the full reader results and the stronger single-Qwen fine-tuned BGE reference.
Appendix L additionally evaluates first-stage BM25 top-10/20/50 retrieval, five-passage selection and QA on 512 questions over a bounded structured corpus whose membership derives from an earlier query-conditioned retrieval union. This directly evaluates selection after first-stage retrieval. BGE outperforms HKVM in this retrieved-pool setting, complementing the original-pool results with a stronger text-selector comparison.
4. When set-level modeling may help (Q3). The main selector learns passage scores informed by relational context. Appendix J.2 separately compares group-completion and pairwise heads under a shared joint utility, with matched parameter counts; their F1 intervals span zero. Explicit set modeling is most motivated when individually weak passages become useful together, the budget is tight and the connections contain few distractors. The group-composition analysis (§5.5; Appendix I.2) shows why selectivity matters: real groups contain support-related signal but can also connect irrelevant passages. These observations motivate learning selection-dependent complementarity; the current diagnostic leaves its advantage over pairwise interactions unresolved (§6).
The accompanying Source data package contains complete structural, budget, QA and set-diagnostic results.
manuscript_map.csvlinks each current table/figure to its CSV and relevant filters.