- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SAADI builds query-agnostic entity memory and hierarchical summaries through sequential, incremental document indexing. It fuses memory, summary, and base-retriever scores to retrieve original chunks without query-time LLM calls. Experiments across four datasets and five retrievers report improved distributed-evidence retrieval, alongside ablations and analyses of construction costs and amortization.
Critical strengths
- The paper presents a coherent approach to amortizing document understanding: query-agnostic entity records and hierarchical summaries enrich original-chunk retrieval, enabling incremental indexing without adaptive query-time LLM calls or document-wide rebuilding.
- Evaluation spans four datasets, five retrieval backends, context lengths reaching 256K tokens, and multiple evidence-hop counts. Consistent improvements over base retrievers support the method's portability across architectures and document types.
Critical weaknesses
-
Fusion weights are selected using queries retained in the LooGLE and NovelHopQA evaluation sets. This introduces evaluation leakage, weakening confidence in small improvements and claims of generalization on those datasets.
-
Agentic comparisons match only mean chunk exposure, while A-RAG search previews are excluded. These conventions do not establish equivalent per-query evidence access, potentially distorting conclusions about relative multi-hop retrieval effectiveness.
Questions for authors
-
Please exclude all fusion-tuning queries from evaluation or use nested validation. Do improvements, ranking stability, and paired confidence intervals remain consistent on strictly held-out queries across datasets and retrieval backends?
-
Can you compare SAADI against each agent using that query's actual exposure budget, including preview tokens? Please add end-to-end QA results to determine whether retrieval gains translate into better answers.
4 comments
Response to Reviewer mokb (Point 1: Held-out fusion weights, 2/4)
Point 1. Held-out fusion weights
To address your question, we re-selected the fusion weights on strictly held-out queries and compared the results with the published weights in three ways. First, retrieval performance (Table R1). Second, paired confidence intervals for every dataset–retriever pair. Third, SAADI’s ranking in Table 1 of the paper (Table R2). We also report how close the published setting is to the best of all configurations.
Protocol. We searched the four fusion weights of Eq. 2 jointly over 1,296 configurations (the paper examined 14): λM, λS ∈ {0, 0.1, 0.2, 0.3, 0.5, 0.75, 1, 1.5, 2} and μA, μD ∈ {0, 0.3, 0.5, 1}. The grid includes the published setting (λM = 0.5, λS = 0.3, μA = μD = 1). For each of the 20 dataset–retriever pairs (four datasets × five retrievers), we used five-fold cross-validation grouped by document: the configuration with the highest HR@10 on four folds is applied to the fifth (100 selections in total). Every query is therefore scored with weights chosen without it or any other query from its document. Chunks, entity memories, and scoring follow the paper.
Table R1. HR@10 (%), averaged over the five retrievers.
| Weight selection | LooGLE | MuSiQue-32K | QASPER-64K | NovelHopQA | Mean |
|---|---|---|---|---|---|
| Published weights | 88.2 | 85.6 | 28.8 | 20.8 | 55.8 |
| Held-out weights | 88.2 | 86.6 | 29.3 | 20.4 | 56.1 |
The published row is Table 1 of the paper.
- Improvements. Mean HR@10 increases from 55.8% to 56.1%. The average rises on MuSiQue-32K and QASPER-64K, stays at 88.2 on LooGLE, and falls by 0.4 points on NovelHopQA.
- Paired intervals (held-out minus published, 95% paired bootstrap, 10,000 resamples): no significant change in 17 of the 20 pairs, a significant gain in 2 (both on MuSiQue-32K), and a small but significant drop in 1 (NovelHopQA with BM25, −0.9 points). On LooGLE and NovelHopQA, the two datasets the original selection sample came from, 9 of the 10 pairs show no significant change.
Table R2. SAADI's position in Table 1 of the paper (40 cells: five retrievers × four datasets × HR@5 and HR@10). SAADI's cells use its held-out results; the other methods keep their reported scores. Ranks follow Appendix F.1 of the paper (ties share the average rank). One correction to Table 1 is used: Vanilla with Contriever on LooGLE has HR@10 = 86.0, not the printed 73.3; we will fix this in the revision.
| Published weights | Held-out weights | |
|---|---|---|
| Mean rank at HR@5 (lower is better) | 1.78 | 1.73 |
| Mean rank at HR@10 (lower is better) | 1.70 | 1.58 |
Sensitivity. Ranked by mean HR@10 over the 20 pairs, the published setting is 36th of the 1,296 fixed configurations (top 3%). The best one, chosen with all evaluation queries in hand, reaches 56.0% against 55.8%, so no single shared setting does much better than the published one.
Conclusion. SAADI's gains and its ranking hold when the fusion weights are selected without the evaluated queries.
Response to Reviewer mokb (Point 2: Per-query agent comparison, 3/4)
Point 2. Per-query agent comparison
To address your question, we compared SAADI with each agent in two ways. First, on every query, SAADI receives exactly the number of full chunks that the agent read on that query. Second, for A-RAG, we repeated the comparison with its search previews counted on both sides.
Protocol. As in Table 2, an agent's exposure on a query is the number of distinct full chunks placed in its LLM's prompts. Passages it retrieved but did not pass to the LLM, and A-RAG's search previews, are not counted. For every query, SAADI (BM25, published fusion weights) receives exactly the agent's full-chunk count, including zero chunks, and therefore zero retrieval credit, when the agent reads no full chunk. We use the five agents and four datasets of Table 2, with the same queries, evidence labels, and hit criterion. Re-scoring the saved agent traces reproduces Table 2's exposure counts and hit rates in all 20 pairs. Significance is assessed with paired bootstrap 95% confidence intervals over queries (10,000 resamples).
Table R3. Hit rate (%) when SAADI receives each agent's actual full-chunk budget on every query. Bold = significantly higher than the other method in that row. "Mean chunks" is the agent's average exposure; the matching uses each query's own count.
| Dataset | Agent | Mean chunks | Agent HR | SAADI HR |
|---|---|---|---|---|
| LooGLE | RAG-R1 | 4.5 | 71.1 | 90.7 |
| Agentic-R | 3.4 | 54.1 | 88.6 | |
| A-RAG | 2.0 | 68.6 | 69.4 | |
| GRIP | 4.6 | 86.7 | 91.4 | |
| UMA | 3.1 | 87.8 | 87.9 | |
| MuSiQue-32K | RAG-R1 | 8.1 | 86.0 | 71.2 |
| Agentic-R | 5.5 | 75.2 | 63.2 | |
| A-RAG | 2.7 | 45.1 | 44.7 | |
| GRIP | 4.0 | 50.7 | 55.7 | |
| UMA | 19.6 | 90.3 | 91.0 | |
| QASPER-64K | RAG-R1 | 7.6 | 20.0 | 19.7 |
| Agentic-R | 3.8 | 16.1 | 13.1 | |
| A-RAG | 3.5 | 12.6 | 12.2 | |
| GRIP | 3.9 | 10.3 | 14.1 | |
| UMA | 14.8 | 27.3 | 31.1 | |
| NovelHopQA | RAG-R1 | 5.9 | 14.1 | 18.8 |
| Agentic-R | 3.9 | 12.6 | 16.3 | |
| A-RAG | 3.9 | 9.0 | 16.5 | |
| GRIP | 5.0 | 14.2 | 17.7 | |
| UMA | 3.7 | 14.0 | 16.3 |
Results. Even with zero-budget queries included, 17/20 comparisons show either a significant SAADI advantage (11) or no significant difference (6). SAADI significantly outperforms every agent on NovelHopQA. The three comparisons that significantly favor agents are the same three that already favored them in Table 2, and no SAADI lead becomes a significant agent lead.
Why A-RAG on LooGLE changes. This pair shows the largest change from average-budget matching: SAADI's previous lead becomes statistically inconclusive. On 12.1% of the queries, A-RAG sees previews and answers without reading any full chunk, so per-query matching gives SAADI zero chunks. Yet SAADI's first-ranked chunk contains the gold evidence on 87% of these queries, and its top two chunks on 94%. The change therefore reflects A-RAG's stopping decision: SAADI gets no retrieval opportunity on queries where its ranking would supply the evidence. These are retrieval results only; they do not say whether A-RAG's answers from previews are correct.
Including previews. A-RAG previews, on average, 1.3-3.8 further chunks per query beyond those it reads in full, and the main comparison does not add them to SAADI's allowance. When previewed chunks are counted on both sides (toward SAADI's per-query budget and as evidence exposed to A-RAG), SAADI significantly outperforms A-RAG on all four datasets, by 3.0-8.3 HR percentage points.
Conclusion. Per-query matching preserves the comparison of Table 2, also under zero-chunk budgets, and counting A-RAG's previews preserves SAADI's advantage on all four datasets.
Response to Reviewer mokb (Point 3: End-to-end QA, 4/4)
Point 3. End-to-end QA
3.1 One reader, different retrieval methods
Protocol. Qwen3-30B-A3B-Instruct-2507 answers all queries on MuSiQue-32K and NovelHopQA with greedy decoding and the same prompt. Only the context changes. SAADI, Vanilla BM25, and Contextual Retrieval supply their top 10 chunks. RAPTOR and A-RAG supply their selected contexts. Closed-book and gold-evidence inputs provide reference conditions. We report token F1 and paired bootstrap 95% confidence intervals over queries.
Table R4. Answer F1 (%) with each method's usual context. Δ = SAADI minus the method, computed from unrounded scores. Bold differences are significant. A-RAG uses an LLM search loop. The other retrieval methods require no query-time LLM calls for retrieval.
| Reader context | MuSiQue F1 | Δ | NovelHopQA F1 | Δ |
|---|---|---|---|---|
| Closed-book | 9.3 | — | 12.6 | — |
| Vanilla BM25, top 10 | 21.9 | +1.0 | 30.8 | +0.7 |
| Contextual Retrieval, top 10 | 22.8 | 0.0 | 29.9 | +1.7 |
| RAPTOR | 21.7 | +1.1 | 17.6 | +13.9 |
| A-RAG | 22.2 | +0.7 | 21.4 | +10.1 |
| SAADI, top 10 | 22.8 | — | 31.5 | — |
| Gold evidence only | 39.7 | — | 40.7 | — |
MuSiQue denotes MuSiQue-32K throughout. NovelHopQA gold evidence comprises chunks containing its annotated evidence paragraphs.
SAADI achieves the highest or tied-highest F1 among the retrieval methods on both datasets. On NovelHopQA, it significantly exceeds RAPTOR, A-RAG, and Contextual Retrieval. On MuSiQue-32K, all methods are within about one F1 point, without significant differences from SAADI.
Matching context length. RAPTOR and A-RAG produce contexts that vary by query and are shorter on average than SAADI's top 10. For each query, we truncated SAADI's ranked context to exactly the comparator's context-token count, then ran the same reader again. We matched each comparator separately, preserving SAADI's ranking and cutting only the tail.
Table R5. Answer F1 (%) at equal context length per query. Δ = truncated SAADI minus comparator. Bold differences are significant.
| Dataset | Comparator | Mean tokens | Comparator F1 | SAADI F1 | Δ |
|---|---|---|---|---|---|
| NovelHopQA | RAPTOR | 1,701 | 17.6 | 27.7 | +10.1 |
| A-RAG | 2,548 | 21.4 | 28.5 | +7.0 | |
| MuSiQue | RAPTOR | 923 | 21.7 | 21.0 | -0.7 |
| A-RAG | 360 | 22.2 | 15.6 | -6.6 |
SAADI's full top 10 averages 6,447 context tokens on NovelHopQA and 1,326 on MuSiQue-32K.
SAADI's significant NovelHopQA advantages persist at equal length. On MuSiQue-32K, it remains comparable to RAPTOR. A-RAG is more effective at its small budget of 360 tokens on average, while SAADI's full top 10 reaches comparable answer quality without LLM calls for retrieval.
Retrieval and answers. Gains over Vanilla are small and not significant (+1.0 and +0.7 F1). On MuSiQue-32K queries where SAADI retrieves all gold chunks, the reader scores 30.0 F1 with its top 10 versus 45.5 with only the gold passages for those same queries. This leaves room to improve how the reader uses retrieved evidence.
3.2 Long-context reader - full document versus SAADI
Qwen2.5-7B-Instruct-1M reads either the full document or SAADI's top 10, with the same questions, prompt, and greedy decoding. Full documents within the native 262K-token range use exact attention. The six longer books use dual chunk attention with YaRN, as its model card recommends. Every document is read in full. Significance uses paired bootstrap 95% intervals (10,000 resamples).
Table R6. Answer F1 (%) with the same 7B reader. Δ = SAADI minus full document. Bold differences are significant. Token ratio = full-document input / SAADI input.
| Evaluation set | Full document | SAADI | Δ | Token ratio |
|---|---|---|---|---|
| MuSiQue, 42K–52K tokens | 28.6 | 26.3 | -2.3 | 35× |
| NovelHopQA, 10 books, 133K-253K | 23.3 | 30.2 | +6.9 | 30× |
| NovelHopQA, 6 books, 265K-760K | 19.6 | 30.8 | +11.2 | 65× |
| NovelHopQA, all 16 books | 21.7 | 30.5 | +8.8 | 45× |
On MuSiQue-32K, the difference is not significant. Across all 16 novels, SAADI improves F1 by 8.8
Conclusion. SAADI's NovelHopQA gains persist at equal context length, and the long-context reader answers more accurately from its top 10 than from full books.
We hope these results address your concerns about the fusion weights, the agent comparison, and the end-to-end evaluation. We would be glad to clarify any remaining points, and we thank you for your careful and constructive review.
Response to Reviewer mokb (Summary, 1/4)
Thank you for your insightful review and for recognizing SAADI’s approach to amortizing document understanding and its portability across retrieval backends. We ran three new experiments for your questions:
The next three comments give the details and full tables.