acceptodds
Withdrawn from ICLR 2027

Does Agentic RAG over Knowledge Graphs Help Scientific QA? Only Where No Single Query Reaches the Evidence

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Retrieval-augmented generation grounds a language model in a corpus, but one retrieval call answers one kind of question. Questions that bridge two papers, aggregate a quantity reported across several, or combine a numeric detail with an entity relationship need evidence a single dense query does not return. Graph-based retrieval already offers the operations such questions require: dense vector search, entity-centred graph traversal, and relationship-centred synthesis. A deployed system, however, fixes one of them per query. We present ModeRAG, in which a Planner assigns each sub-task the retrieval mode its part requires and declares the dependencies between sub-tasks, an Executor runs ready sub-tasks in parallel, and a Reporter answers from an evidence chain that records the chunks, graph paths and entities behind each step. We release LHCpubs-100QA, 100 multi-paper questions over LHC papers with annotated evidence, and evaluate alongside AI4EIC2023, 50 mostly single-paper questions. ModeRAG improves answer correctness by 7.7 points over the same pipeline restricted to vector search, with the largest gains on questions that bridge or aggregate across papers; the difference is unresolved on the single-paper benchmark. Dense retrieval remains the backbone, and decomposition costs 24,268 tokens per question.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic RAG, GraphRAG, Knowledge Graphs, Multi-hop Question Answering, Query Decomposition, Scientific Literature Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_kmkdOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents ModeRAG, a Planner–Executor–Reporter framework for scientific question answering that decomposes questions into sub-queries and assigns each to dense, entity-centered, or relationship-centered retrieval. Queries are fixed before execution; dependencies govern scheduling, while an evidence chain records source provenance for answer synthesis. The paper also introduces an LHC literature corpus and LHCpubs-100QA, a benchmark of 100 cross-paper questions with annotated evidence over a 96-paper subset.

Critical strengths
  1. The paper addresses an important, concrete problem in scientific QA. Questions that compare measurements or connect findings across papers require adequate coverage of complementary evidence. Assigning retrieval modes at the sub-query level is an intuitive approach to this problem.

  2. The evaluation provides a meaningful positive signal and a useful diagnostic resource. The Naive Only baseline retains the same planning and reporting framework, making the comparison more informative than one against single-call RAG alone.

Critical weaknesses
  1. The experiments do not establish whether retrieval-mode selection explains the improvement. All Modes changes the decomposition, adds dense retrieval to every graph task, and uses approximately 20% more subtasks and 30% more tokens than Naive Only on LHCpubs-100QA. Consequently, additional queries and context remain plausible explanations for the gain. The authors acknowledge this limitation, and the complete-system comparison is still informative. However, because the method’s principal novelty is the allocation of retrieval modes, a comparison using identical sub-queries and matched context budgets is necessary to establish its distinctive contribution.

  2. The central claim about when the method helps is not directly supported. Requiring evidence from multiple papers does not establish that a single query cannot retrieve sufficient evidence. The retrievability gate checks only partial reachability, and the genotype specifications do not independently verify that different retrieval modes are necessary. Moreover, a significant gain on one benchmark and an unresolved gain on another do not establish a boundary between useful and unnecessary orchestration. This matters because the paper’s strongest potential contribution is its diagnostic insight; the current evidence supports a narrower conclusion that the configured system performs better on the constructed cross-paper benchmark.

Questions for authors
  1. Decision-relevant control: attribution of the improvement. Can you provide a comparison that fixes the sub-queries generated by All Modes, executes them using dense retrieval alone, and matches the generator-visible context budget? This would help determine whether graph evidence contributes beyond additional queries and context. If such results already exist, do they support a benefit from selecting retrieval modes over using a fixed fusion strategy?

  2. Clarification: evidence for the claimed applicability boundary. Have you measured whether a single retrieval call obtains all necessary gold evidence, and whether ModeRAG’s gains concentrate on questions where that evidence is incomplete? The current distinction between cross-paper and mostly single-paper benchmarks does not directly establish this relationship. Is the “Only Where” claim supported by within-benchmark evidence, or should it be interpreted as a hypothesis suggested by the results?

  3. Clarification: consistency of the headline results. Table 6 implies a correctness improvement of 0.6685−0.6016=0.06690.6685-0.6016=0.0669 on LHCpubs-100QA, whereas the abstract and Table 9 report 0.0770.077. Do the aggregate scores, genotype-level differences, and bootstrap intervals use the same predictions for the same 100 questions? Please clarify which results are authoritative and whether correcting the discrepancy changes the confidence interval.

Reviewer_mwbHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ModeRAG, which plans sub-queries in advance and assigns them dense vector, entity-centered, and relationship-centered retrieval tools to evaluate the benefits of multi-mode retrieval orchestration for question answering over high-energy physics literature. It also introduces a cross-paper QA benchmark. Its contributions primarily lie in system integration, domain resources, and diagnostic empirical evaluation.

Critical strengths

1.The paper assembles a corpus containing 2,771 full-text papers and constructs 100 cross-paper questions with reference answers, evidence spans, and source identifiers over a 96-paper indexed subset. The T1–T4 genotypes provide a starting point for analyzing different evidence demands. The value of these resources should not be discounted because the method does not train new parameters.

  1. The authors explicitly state that sub-queries are fixed before execution, dependencies govern scheduling only, and graph tasks receive an additional dense vector retrieval pass. They also report token usage, latency, and the unresolved difference on the mostly single-paper benchmark. These disclosures help readers assess the results and avoid presenting the system as uniformly superior across metrics and settings.
Critical weaknesses
  1. The headline gain is numerically inconsistent, and baseline reporting is incomplete. Table 6 gives 0.6685 − 0.6016 = 0.0669 (6.69 pp), but the abstract, Section 4.2, the conclusion, and Table 9 report 0.077 (7.7 pp). If these results use the same questions, run, and equal weights, a paired bootstrap cannot explain this discrepancy; the authors should clarify runs/samples and reconcile the tables, effect estimates, and confidence intervals. The AI4EIC2023 difference (0.0268 ≈ 0.027) is consistent. In addition, Section 4.1 claims two LightRAG configurations, but Table 5 contains only one LightRAG row without identifying its mode; the authors should provide or clarify both configurations.

  2. LHCpubs-100QA’s cross-paper design is valuable, but Appendix B.1’s T3/T4 wording (“genuinely require different retrieval strategies”; “three-retriever mandatory”) asserts a stronger necessity claim than merely requiring different types of information. The retrievability gate—retain a question if at least half of its eligible evidence items pass snippet or source-paper identifier matching—is only a filtering step and does not establish the validity of retained questions. The paper should report additional checks that all required evidence exists, sufficiently supports the answer, and that reference answers are correct. The title’s conditional claim (“Only Where No Single Query Reaches the Evidence”) is not established: the experiments contrast a cross-paper benchmark with a mostly single-paper benchmark without per-question measurement of single-query sufficient-evidence coverage, and they also confound corpus size, subject matter, and question construction. Cross-paper questions are not necessarily unanswerable with one retrieval call.

Questions for authors

Answer generation and Ragas evaluation use the same DeepSeek backend, which may lead to correlated generation and evaluation errors. The authors disclose this limitation, but the current results do not establish whether the ranking of the main systems is sensitive to the choice of judge. It would be better to reevaluate the main comparisons with another model and conducting a blinded evaluation of a subset of answers by domain experts.

Reviewer_j2QxOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose ModeRAG, a Planner-Executor-Reporter framework that treats Dense Search, Entity-centered Graph Traversal (Local), and Relation-centered Synthesis as tools. The Planner decomposes complex queries into dependency-annotated sub-tasks and assigns specific retrieval modes. To evaluate this, the authors introduce LHCpubs-100QA alongside AI4EIC2023.

Critical strengths
  1. Unlike many RAG papers that claim universal superiority over baseline vector search, this paper adopts a refreshing, sober perspective to evaluate.

  2. The design of a unified Evidence Chain that tracks chunks, graph paths, and entities across execution rounds provides excellent post-hoc explainability and trust calibration.

  3. Releasing 100 high-energy physics questions categorized into 4 distinct cross-paper evidence genotypes with annotated gold evidence spans is a solid contribution to the community.

Critical weaknesses
  1. The authors acknowledge that orchestration costs ~24,268 tokens and 86.7 seconds per question on average. Since 64% of tasks in AI4EIC2023 or simpler queries only require Naive search, running the full Planner-Executor pipeline every single time introduces massive redundancy. The framework lacks an adaptive Router that bypasses heavy graph/agentic pipelines for standard queries.

  2. Missing all the important baselines like HippoRAG, RAPTOR, GFM-RAG, etc.

  3. Both answer generation and RAGAS metric evaluation rely on the same model. This backend coupling might introduce correlated self-preference or judge artifacts (e.g., the 18% non-committal answer penalty described in Section B.4)

  4. In the All Modes configuration, every planned Local or Global task automatically executes an additional dense-vector pass and fuses the results under a score cap. Consequently, ModeRAG issues significantly more dense search calls than the Naive Only baseline. It remains unclear whether the +7.7 point gain stems from graph-structure traversal or simply from issuing multiple dense vector queries under LLM task decomposition.

Questions for authors

Please see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.