acceptodds
Under review as a conference paper at ICLR 2027

DictRAG: Composable Corpus Access for Multi-Hop Question Answering

Abstract

Multi-hop question answering requires combining constraints to locate evidence scattered across a corpus. Ranked retrieval returns limited passage lists, restricting subsequent filtering to retrieved candidates. We introduce DictRAG, which enables agents to compose corpus-level conditions before selecting evidence to read. A bounded SQL interface exposes documents as field–value records and supports exact lexical filtering, set operations, and reusable relation definitions across turns. An indexed backend executes these operations outside model context, returning only selected observations. In corpus-constrained evaluations on HotpotQA, 2WikiMultiHopQA, MuSiQue, and Bamboogle, DictRAG with Qwen3.5-27B achieves a dataset-macro average of 65.70% exact match (EM) and 75.98% F1. Average gains over both agentic retrievers also hold with Qwen3.5-9B and Gemma4-31B. Progressive ablations on MuSiQue show larger observed gains from exact search and condition composition than from cross-turn relation reuse. Under corpus expansion, DictRAG has higher observed EM and F1 than both retrievers on all three scaling datasets at 100K and 500K documents. With a expansion of the HotpotQA corpus to 5.23 million documents, DictRAG retains 65.50% EM and shows a smaller observed decline than both agentic retrievers: 3.0 percentage points, versus 5.2 for Agentic-BM25 and 4.1 for Agentic-BGE. These results support composable corpus access as a useful capability for multi-hop question answering over large document collections.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Hop Question Answering, Retrieval-Augmented Generation, Agentic Retrieval, Information Retrieval, Programmable Corpus Access

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_LLpVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DictRAG, a bounded SQL interface that lets an agent access a corpus as field–value records (id, title, content). The agent can apply exact substring filters over titles/content (trigram-indexed), combine conditions with set operations, and save lazy relation definitions to refine across turns; only small result slices enter the context. With Qwen3.5-27B on HotpotQA, 2Wiki, MuSiQue (HippoRAG2 1k subsets), and Bamboogle (125 questions, project-built corpus), DictRAG reaches a 65.70 macro-average EM vs. 63.35 for an agentic BM25 retriever under the same 30-turn budget, with the same ranking for two other backbones. Gains are largest on MuSiQue (+6.0 EM on the original corpus, +10–14 EM on expanded corpora). A progressive ablation attributes +3.1 EM to exact search, +2.1 to composition, and +0.8 (not significant) to saved relations. Corpus-expansion experiments go up to 5.23M HotpotQA documents.

Critical strengths

S1. A simple, practical idea with a robust MuSiQue result. Replacing top-k ranked lists with corpus-wide exact filtering and set operations is easy to adopt. On MuSiQue the gain is large and Holm-significant in every corpus condition (+6.0 to +13.9 EM; Tables 12–13), consistent in direction across three backbones (Table 7), and obtained with fewer input tokens than deeper ranked retrieval (App. C).

S2. Careful and candid evaluation. The paper uses paired McNemar tests with Holm correction, bootstrap CIs, a progressive ablation, cost reporting, and a candidate-pool diagnostic, and it states explicitly which differences are not significant.

Critical weaknesses

W1. The general claim rests mainly on one dataset, and the scaling test may favor exact matching by design. After Holm correction, only the MuSiQue gains are significant. On 2Wiki (+0.4 EM), HotpotQA (+2.2), and Bamboogle (+0.8, n=125) the differences are within noise (Table 12), and none of the 2Wiki/HotpotQA scaling comparisons is significant (Table 13). The abstract's "smaller decline at 5.23M documents" (3.0 vs. 5.2/4.1 points) compares two single-run differences without a test, and the Full-HotpotQA gaps themselves are not significant (Holm p = 0.080 / 0.326). The 100K/500K corpora add documents drawn from a generic Wikipedia pool; such documents rarely contain the question's exact entity strings, so they barely affect substring filters while adding noise to ranked retrieval. Topically related (hard) distractors would be a more informative test. The paper also does not explain why MuSiQue benefits so much, so it is unclear how far the result transfers beyond title-keyed Wikipedia corpora.

W2. It is unclear which part of the interface drives the gain, and the most natural baseline is missing. The largest increment comes from exact search (+3.1 EM). The distinctive features — saved relations (+0.8, n.s.) and dictionary-style record semantics (untested, Sec. 3.2) — are not shown to matter. General SQL/FTS5 access with joins and Boolean matching scores 49.5 EM, the same as Agentic-BM25 (49.9; Table 6), which suggests that DictRAG's prompt, error feedback, and restricted operation set, rather than composable access per se, may explain much of the gain. The obvious comparator is an agentic retriever that also has a corpus-wide exact-match tool: Table 11 lists such a configuration (Agentic-LexSearch) but reports no accuracy.

Minor: the observation/residual caps and verbatim prompt, which App. A.1 lists as necessary for reproduction, are not given; index size, build time, and latency at 5.23M documents are not reported.

Questions for authors
  1. What is the accuracy of Agentic-LexSearch (Table 11), or of an agent with both ranked search and a corpus-wide exact-match tool, under the same budget? If it matches DictRAG, the contribution reduces to adding exact-match search.

  2. Why are the gains concentrated on MuSiQue? A breakdown by whether question entities appear verbatim in document titles, plus an error analysis of DictRAG failures (e.g., alias or paraphrase misses), would clarify how much the benefit depends on benchmark structure.

  3. Can you report expanded-corpus results with topically related (hard) distractors, together with index size, build time, and query latency at 5.23M documents?

Reviewer_XWydOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DictRAG, an agentic retrieval framework for multi-hop question answering that uses SQL-like operations, including lexical filtering and set operations, to identify relevant documents. Unlike conventional retrieval approaches that return ranked lists of top-k documents, DictRAG enables an agent to progressively construct, combine, and refine candidate document sets through corpus-wide operations before reading the selected documents. The system also supports storing and reusing intermediate candidate relations across agent turns. Experiments on four multi-hop QA benchmarks demonstrate competitive performance, with improvements over several agentic retrieval baselines. Ablation studies examine the contributions of the proposed components.

Critical strengths
  1. Strong empirical results. DictRAG achieves competitive performance across several multi-hop QA benchmarks, with improvements over agentic retrieval baselines. The evaluation considers different language models and corpus sizes, providing useful evidence of the approach's effectiveness.

  2. Composable corpus access. The proposed approach shifts retrieval from ranked document lists toward composable, corpus-wide operations. This is an interesting direction for integrating database-style querying with agentic retrieval, allowing agents to progressively narrow candidate sets and reuse intermediate results.

  3. Component-level evaluation. The ablation studies help isolate the contributions of exact search, composable filtering, and intermediate relation reuse, providing insight into the effectiveness of the individual components.

Critical weaknesses
  1. Limitations of lexical matching. The approach relies heavily on exact lexical matching, which may fail to retrieve relevant documents containing synonyms, aliases, or paraphrases. This is particularly concerning for multi-hop QA, where relevant entities and relationships may be expressed in different ways across documents. The paper does not adequately establish the robustness of the approach under greater lexical variation.

  2. Computational overhead and scalability. Retrieval is a key component of QA systems, and existing retrieval methods have been extensively optimized to handle collections containing billions of documents. DictRAG introduces multiple filtering and query-execution steps that may substantially increase retrieval costs. The reported results already show considerably higher token consumption and execution time than Agentic-BM25. It remains unclear whether the accuracy gains justify these additional costs, particularly at larger corpus scales.

  3. Insufficient baseline documentation. The implementations of some baselines are not sufficiently described. For example, Agentic-BM25 appears to be a custom agentic implementation built around BM25, but its precise design and relationship to existing methods are unclear. More detailed descriptions are needed to assess the fairness and reproducibility of the comparisons.

  4. Presentation and lack of a concrete end-to-end example. The core idea is relatively straightforward, but the presentation makes it more difficult to understand than necessary. Although Figure 2 and Appendix A.1 provide schematic illustrations, the paper would benefit from a concrete, end-to-end example showing how an actual multi-hop question is answered. Such an example should include the generated SQL-like operations, intermediate candidate sets, retrieved evidence, and final answer. This would help readers understand both the execution process and the benefits of composable retrieval.

  5. Modest improvements on several datasets. Although the overall results are positive, improvements over agentic retrieval baselines are relatively small on several datasets and are most pronounced on MuSiQue. Some differences are not statistically significant. This raises questions about the consistency and generalizability of the gains, particularly given the additional computational overhead.

Questions for authors
  1. How would DictRAG handle questions where relevant documents use synonyms, aliases, or paraphrases rather than exact lexical matches? Could semantic retrieval be incorporated into the candidate-generation process?

  2. Can the authors provide a more detailed accuracy–cost comparison with agentic retrieval baselines, accounting for tool calls, token consumption, and end-to-end execution time? How would the approach scale to collections containing hundreds of millions or billions of documents?

  3. Can the authors clarify the implementations of Agentic-BM25 and Agentic-BGE, including their relationship to previously published methods and the availability of code and configurations?

Reviewer_VUjBOpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

DictRAG, retrieval framework is presented whch allows an agent to search and refine document collections through a constrained SQL interface. DictRAG supports saving intermediate results as named relations and applying further filters, while document retrieval and query execution are handled by an indexed SQLite backend outside the model’s context. Evaluated against agent-based retrievers, including Agentic-BM25 and Agentic-BGE, and other published methods on HotpotQA, 2WikiMultiHopQA, MuSiQue, and a constructed Bamboogle corpus. Qwen3.5-27B based DictRAG achieves average exact match of 65.70 and an F1 score of 75.98 across the four datasets. Besides, scalability is also conducted on corpora containing up to 5.23 million documents and includes an ablation study on MuSiQue.

Critical strengths

Intresting Idea: Letting the agent narrow the candidate set before reading evidence is a clear and testable alternative to ranked top-k retrieval, and Figure 2 explains it well. Qwen3.5-27B based DictRAG achieves an average exact match of 65.70 and an F1 score of 75.98 across the four datasets. Besides, scalability is also conducted on corpora containing up to 5.23 million documents and includes an ablation study on MuSiQue. DictRAG is comparedwith decomposition prompting, general SQL over FTS5, and corpus-as-string, which together address whether the gain comes from the interface or just from more elaborate prompting.

Critical weaknesses

DictRAG’s performance gains are concentrated on MuSiQue, which accounts for approximately 64% of its total exact-match improvement over Agentic-BM25 across the four datasets. On HotpotQA and 2WikiMultiHopQA, the differences from competing retrievers are relatively small, and the reported confidence intervals include zero. Hence, claim that DictRAG consistently outperforms existing methods should be qualified.

The ablation studies examining progressive component removal, interface design, and candidate-pool composition are also conducted primarily on MuSiQue with single backbone, leaving it unclear whether the same conclusions hold across other datasets and models. Moreover, reusable relations is central part of proposed framework, observed improvement from cross-turn relation reuse is not statistically significant after multiple-comparison correction, making independent contribution uncertain.

The constructed Bamboogle corpus also raises concerns because its candidate passages are retrieved using the same baseline methods being compared, potentially affecting fairness of evaluation. Since Bamboogle contains only 125 questions but receives equal weight in four-dataset average, better to report aggregate results both with and without it.

Several baseline methods, including LinearRAG, HopRAG, HippoRAG2, IRCoT, and A-RAG, are reimplemented without providing sufficiently detailed prompts and configuration settings, making it difficult to determine evaluation on comparable conditions. Ablation results also do not fully isolate the effects of SQL-based access, dictionary semantics, prompt design, and error feedback; hence, the findings are better interpreted as evidence for the overall system. Equal turn budgets do not guarantee equal computational cost because DictRAG can issue multiple SQL calls within a turn and uses approximately 44.5% more input tokens than Agentic-BM25. The cost-matched comparison is limited to F1 on MuSiQue, so it remains unclear whether DictRAG retains its advantage when both exact match and F1 are compared at matched costs across all datasets.

Questions for authors

Please report cost-matched EM and F1 for all four datasets, not only MuSiQue F1? Can you run the progressive ablation (Table 5) on at least one other dataset and one other backbone? Why were four MuSiQue ranges used, and were they run in separate batches? What are the rates of unsupported-syntax errors, missing final answers, and turn-budget exhaustion for DictRAG? What prompts and settings were used for LinearRAG, HopRAG, HippoRAG2, IRCoT, and A-RAG?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.