- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces EviPort, a retrieval framework for constructing complementary evidence sets under a fixed reader budget. It decomposes questions into dependency-aware evidence ports, binds downstream queries to answers extracted from retrieved passages, and assembles the final context through round-robin selection. The framework reuses a frozen dense index without task-specific training or corpus graph construction. Experiments on HotpotQA, 2WikiMultiHopQA, and MuSiQue report improvements in evidence retrieval and question answering over dense, iterative, and graph-based baselines.
Critical strengths
- The paper addresses a meaningful limitation of passage-level ranking, where individually relevant passages can occupy the context budget without covering all required evidence.
- The reported QA improvements are consistent across three benchmarks under a shared reader protocol. The frozen transfer evaluations on HotpotQA and 2WikiMultiHopQA strengthen the empirical results.
- The mechanism interventions, alternative selectors, and oracle diagnostics provide useful insight into the sources of improvement and the remaining failures.
Critical weaknesses
-
The methodological contribution beyond existing question decomposition approaches is not sufficiently established. Evidence ports are subquestions with explicit dependencies, and their execution follows familiar retrieval and intermediate-answer extraction steps. The matched iterative control is useful, but it does not replace a direct comparison with a dependency-aware decomposition method such as Q-DREAM under the same retrieval and reader settings.
-
The allocation component has limited support as a distinct contribution. With identical plans, bindings, and candidate lists, the simpler port-heads plus dense-fill strategy achieves higher EM and F1 than round-robin in Table 4. Round-robin improves FullEvidence@5 by only 1.10 points, without reported uncertainty estimates. These results support preserving coverage across subquestions, but the additional value of the proposed allocation rule remains unclear.
-
The fixed execution plan is unreliable on longer reasoning chains. For four-hop questions, 53.01% of plans contain fewer steps than the annotated hop count, and FullEvidence@5 reaches only 7.23%. Appendix C also specifies that unsupported answer strings are propagated into dependent queries without correction. Although EviPort improves over the baselines, the absence of plan revision or recovery leaves a substantial weakness in the dependency mechanism central to the method.
-
The evaluation leaves important generalization questions unresolved. All corpora contain approximately 6,000 to 12,000 passages, while the main selector and matched iterative comparisons are confined to MuSiQue, which served as the development set. The frozen transfer results are valuable, but repeating these comparisons on held-out data and larger corpora would better establish the robustness of the conclusions.
-
The accuracy gains involve a substantial increase in online computation. Despite matching search and LLM-call counts, EviPort requires 7.53 seconds per question versus 4.92 seconds for the iterative control in Table 13. A comparison under equal latency or token budgets would clarify whether explicit port planning remains advantageous when the iterative baseline has access to comparable computational resources.
Questions for authors
Please refer to the summary of weaknesses.