- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents ModeRAG, a Planner–Executor–Reporter framework for scientific question answering that decomposes questions into sub-queries and assigns each to dense, entity-centered, or relationship-centered retrieval. Queries are fixed before execution; dependencies govern scheduling, while an evidence chain records source provenance for answer synthesis. The paper also introduces an LHC literature corpus and LHCpubs-100QA, a benchmark of 100 cross-paper questions with annotated evidence over a 96-paper subset.
Critical strengths
-
The paper addresses an important, concrete problem in scientific QA. Questions that compare measurements or connect findings across papers require adequate coverage of complementary evidence. Assigning retrieval modes at the sub-query level is an intuitive approach to this problem.
-
The evaluation provides a meaningful positive signal and a useful diagnostic resource. The Naive Only baseline retains the same planning and reporting framework, making the comparison more informative than one against single-call RAG alone.
Critical weaknesses
-
The experiments do not establish whether retrieval-mode selection explains the improvement. All Modes changes the decomposition, adds dense retrieval to every graph task, and uses approximately 20% more subtasks and 30% more tokens than Naive Only on LHCpubs-100QA. Consequently, additional queries and context remain plausible explanations for the gain. The authors acknowledge this limitation, and the complete-system comparison is still informative. However, because the method’s principal novelty is the allocation of retrieval modes, a comparison using identical sub-queries and matched context budgets is necessary to establish its distinctive contribution.
-
The central claim about when the method helps is not directly supported. Requiring evidence from multiple papers does not establish that a single query cannot retrieve sufficient evidence. The retrievability gate checks only partial reachability, and the genotype specifications do not independently verify that different retrieval modes are necessary. Moreover, a significant gain on one benchmark and an unresolved gain on another do not establish a boundary between useful and unnecessary orchestration. This matters because the paper’s strongest potential contribution is its diagnostic insight; the current evidence supports a narrower conclusion that the configured system performs better on the constructed cross-paper benchmark.
Questions for authors
-
Decision-relevant control: attribution of the improvement. Can you provide a comparison that fixes the sub-queries generated by All Modes, executes them using dense retrieval alone, and matches the generator-visible context budget? This would help determine whether graph evidence contributes beyond additional queries and context. If such results already exist, do they support a benefit from selecting retrieval modes over using a fixed fusion strategy?
-
Clarification: evidence for the claimed applicability boundary. Have you measured whether a single retrieval call obtains all necessary gold evidence, and whether ModeRAG’s gains concentrate on questions where that evidence is incomplete? The current distinction between cross-paper and mostly single-paper benchmarks does not directly establish this relationship. Is the “Only Where” claim supported by within-benchmark evidence, or should it be interpreted as a hypothesis suggested by the results?
-
Clarification: consistency of the headline results. Table 6 implies a correctness improvement of on LHCpubs-100QA, whereas the abstract and Table 9 report . Do the aggregate scores, genotype-level differences, and bootstrap intervals use the same predictions for the same 100 questions? Please clarify which results are authoritative and whether correcting the discrepancy changes the confidence interval.