- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes DictRAG, a bounded SQL interface that lets an agent access a corpus as field–value records (id, title, content). The agent can apply exact substring filters over titles/content (trigram-indexed), combine conditions with set operations, and save lazy relation definitions to refine across turns; only small result slices enter the context. With Qwen3.5-27B on HotpotQA, 2Wiki, MuSiQue (HippoRAG2 1k subsets), and Bamboogle (125 questions, project-built corpus), DictRAG reaches a 65.70 macro-average EM vs. 63.35 for an agentic BM25 retriever under the same 30-turn budget, with the same ranking for two other backbones. Gains are largest on MuSiQue (+6.0 EM on the original corpus, +10–14 EM on expanded corpora). A progressive ablation attributes +3.1 EM to exact search, +2.1 to composition, and +0.8 (not significant) to saved relations. Corpus-expansion experiments go up to 5.23M HotpotQA documents.
Critical strengths
S1. A simple, practical idea with a robust MuSiQue result. Replacing top-k ranked lists with corpus-wide exact filtering and set operations is easy to adopt. On MuSiQue the gain is large and Holm-significant in every corpus condition (+6.0 to +13.9 EM; Tables 12–13), consistent in direction across three backbones (Table 7), and obtained with fewer input tokens than deeper ranked retrieval (App. C).
S2. Careful and candid evaluation. The paper uses paired McNemar tests with Holm correction, bootstrap CIs, a progressive ablation, cost reporting, and a candidate-pool diagnostic, and it states explicitly which differences are not significant.
Critical weaknesses
W1. The general claim rests mainly on one dataset, and the scaling test may favor exact matching by design. After Holm correction, only the MuSiQue gains are significant. On 2Wiki (+0.4 EM), HotpotQA (+2.2), and Bamboogle (+0.8, n=125) the differences are within noise (Table 12), and none of the 2Wiki/HotpotQA scaling comparisons is significant (Table 13). The abstract's "smaller decline at 5.23M documents" (3.0 vs. 5.2/4.1 points) compares two single-run differences without a test, and the Full-HotpotQA gaps themselves are not significant (Holm p = 0.080 / 0.326). The 100K/500K corpora add documents drawn from a generic Wikipedia pool; such documents rarely contain the question's exact entity strings, so they barely affect substring filters while adding noise to ranked retrieval. Topically related (hard) distractors would be a more informative test. The paper also does not explain why MuSiQue benefits so much, so it is unclear how far the result transfers beyond title-keyed Wikipedia corpora.
W2. It is unclear which part of the interface drives the gain, and the most natural baseline is missing. The largest increment comes from exact search (+3.1 EM). The distinctive features — saved relations (+0.8, n.s.) and dictionary-style record semantics (untested, Sec. 3.2) — are not shown to matter. General SQL/FTS5 access with joins and Boolean matching scores 49.5 EM, the same as Agentic-BM25 (49.9; Table 6), which suggests that DictRAG's prompt, error feedback, and restricted operation set, rather than composable access per se, may explain much of the gain. The obvious comparator is an agentic retriever that also has a corpus-wide exact-match tool: Table 11 lists such a configuration (Agentic-LexSearch) but reports no accuracy.
Minor: the observation/residual caps and verbatim prompt, which App. A.1 lists as necessary for reproduction, are not given; index size, build time, and latency at 5.23M documents are not reported.
Questions for authors
-
What is the accuracy of Agentic-LexSearch (Table 11), or of an agent with both ranked search and a corpus-wide exact-match tool, under the same budget? If it matches DictRAG, the contribution reduces to adding exact-match search.
-
Why are the gains concentrated on MuSiQue? A breakdown by whether question entities appear verbatim in document titles, plus an error analysis of DictRAG failures (e.g., alias or paraphrase misses), would clarify how much the benefit depends on benchmark structure.
-
Can you report expanded-corpus results with topically related (hard) distractors, together with index size, build time, and query latency at 5.23M documents?