acceptodds
Under review as a conference paper at ICLR 2027

Understand Once, Retrieve Many Times: SAADI for Single-Pass Retrieval of Distributed Evidence in Long Documents

Abstract

Long-document retrieval is challenging when relevant evidence is spread across distant passages or multiple hops, especially under a limited retrieval budget. Single-pass top-k retrievers have predictable compute and latency at query time, whereas agentic systems pursue higher recall by reading a document over several steps. We move progressive document understanding to index time, where a single-pass top- retriever can use it without query-time large language model (LLM) calls or adaptive trajectories. SAADI (Stateful Accumulation of Document Information) builds a query-agnostic document state at index time with one state-update call per chunk. Each call extracts the chunk's entities, conditioned on records recalled from the entity memory and on a hierarchical summary of the prefix. The chunk is linked to the resulting records and to the summary covering it, and the original chunks remain the evidence and retrieval units. At query time, two auxiliary scores are computed from a chunk's linked records and its covering summary and fused with any sparse or dense base retriever's score in a single top-k pass. Index-time LLM calls, including summary folds, are fixed before any query arrives, at most 1.25 per chunk in the default configuration. Each call reads a context that grows only logarithmically with the document, and appending chunks extends the state without rebuilding it. Across four datasets and five retrievers, SAADI improves HR@10 over its base retriever in all twenty pairs, by 0.3 to 12.4 points (1.1 to 4.0 under BM25). It ranks first or second in 34 of 40 dataset, retriever, and depth cells and, at matched evidence exposure, is ahead of agentic retrievers in sixteen of twenty pairs. The gains persist to 256K-token contexts and at every evidence-hop count. Agentic costs recur with every query. After a break-even number of queries per document, which we report in model calls and in tokens, they exceed SAADI's one-time construction cost. Ablations show that the entity-memory and summary signals each improve retrieval, and that fusing both is best in every setting. Thus document-level connections behind distributed and multi-hop evidence can be established once at index time, at a cost known before the first query.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Information Retrieval, Long-Document Retrieval, Agentic Retrieval, Multi-Hop Retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mokbOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

SAADI builds query-agnostic entity memory and hierarchical summaries through sequential, incremental document indexing. It fuses memory, summary, and base-retriever scores to retrieve original chunks without query-time LLM calls. Experiments across four datasets and five retrievers report improved distributed-evidence retrieval, alongside ablations and analyses of construction costs and amortization.

Critical strengths
  1. The paper presents a coherent approach to amortizing document understanding: query-agnostic entity records and hierarchical summaries enrich original-chunk retrieval, enabling incremental indexing without adaptive query-time LLM calls or document-wide rebuilding.
  2. Evaluation spans four datasets, five retrieval backends, context lengths reaching 256K tokens, and multiple evidence-hop counts. Consistent improvements over base retrievers support the method's portability across architectures and document types.
Critical weaknesses
  1. Fusion weights are selected using queries retained in the LooGLE and NovelHopQA evaluation sets. This introduces evaluation leakage, weakening confidence in small improvements and claims of generalization on those datasets.

  2. Agentic comparisons match only mean chunk exposure, while A-RAG search previews are excluded. These conventions do not establish equivalent per-query evidence access, potentially distorting conclusions about relative multi-hop retrieval effectiveness.

Questions for authors
  1. Please exclude all fusion-tuning queries from evaluation or use nested validation. Do improvements, ranking stability, and paired confidence intervals remain consistent on strictly held-out queries across datasets and retrieval backends?

  2. Can you compare SAADI against each agent using that query's actual exposure budget, including preview tokens? Please add end-to-end QA results to determine whether retrieval gains translate into better answers.

4 comments
AuthorsOpenReview ↗

Response to Reviewer mokb (Summary, 1/4)

Thank you for your insightful review and for recognizing SAADI’s approach to amortizing document understanding and its portability across retrieval backends. We ran three new experiments for your questions:

  • Point 1. Held-out fusion weights. With weights chosen by 5-fold cross-validation on held-out queries, mean HR@10 goes from 55.8% to 56.1%, so the gains do not depend on tuning on the test queries.
  • Point 2. Per-query agent comparison. Even when SAADI gets exactly as many chunks as each agent read on each query, it is significantly better in 11 of 20 comparisons, and the agents are better in only 3.
  • Point 3. End-to-end QA. With the same reader, SAADI gives the best or tied-best answers among the retrieval methods, and on the 16 novels, its top 10 beats the whole book by 8.8 F1 with 45× fewer tokens.

The next three comments give the details and full tables.

AuthorsOpenReview ↗

Response to Reviewer mokb (Point 1: Held-out fusion weights, 2/4)

Point 1. Held-out fusion weights

To address your question, we re-selected the fusion weights on strictly held-out queries and compared the results with the published weights in three ways. First, retrieval performance (Table R1). Second, paired confidence intervals for every dataset–retriever pair. Third, SAADI’s ranking in Table 1 of the paper (Table R2). We also report how close the published setting is to the best of all configurations.

Protocol. We searched the four fusion weights of Eq. 2 jointly over 1,296 configurations (the paper examined 14): λM, λS ∈ {0, 0.1, 0.2, 0.3, 0.5, 0.75, 1, 1.5, 2} and μA, μD ∈ {0, 0.3, 0.5, 1}. The grid includes the published setting (λM = 0.5, λS = 0.3, μA = μD = 1). For each of the 20 dataset–retriever pairs (four datasets × five retrievers), we used five-fold cross-validation grouped by document: the configuration with the highest HR@10 on four folds is applied to the fifth (100 selections in total). Every query is therefore scored with weights chosen without it or any other query from its document. Chunks, entity memories, and scoring follow the paper.

Table R1. HR@10 (%), averaged over the five retrievers.

Weight selectionLooGLEMuSiQue-32KQASPER-64KNovelHopQAMean
Published weights88.285.628.820.855.8
Held-out weights88.286.629.320.456.1

The published row is Table 1 of the paper.

  • Improvements. Mean HR@10 increases from 55.8% to 56.1%. The average rises on MuSiQue-32K and QASPER-64K, stays at 88.2 on LooGLE, and falls by 0.4 points on NovelHopQA.
  • Paired intervals (held-out minus published, 95% paired bootstrap, 10,000 resamples): no significant change in 17 of the 20 pairs, a significant gain in 2 (both on MuSiQue-32K), and a small but significant drop in 1 (NovelHopQA with BM25, −0.9 points). On LooGLE and NovelHopQA, the two datasets the original selection sample came from, 9 of the 10 pairs show no significant change.

Table R2. SAADI's position in Table 1 of the paper (40 cells: five retrievers × four datasets × HR@5 and HR@10). SAADI's cells use its held-out results; the other methods keep their reported scores. Ranks follow Appendix F.1 of the paper (ties share the average rank). One correction to Table 1 is used: Vanilla with Contriever on LooGLE has HR@10 = 86.0, not the printed 73.3; we will fix this in the revision.

Published weightsHeld-out weights
Mean rank at HR@5 (lower is better)1.781.73
Mean rank at HR@10 (lower is better)1.701.58

Sensitivity. Ranked by mean HR@10 over the 20 pairs, the published setting is 36th of the 1,296 fixed configurations (top 3%). The best one, chosen with all evaluation queries in hand, reaches 56.0% against 55.8%, so no single shared setting does much better than the published one.

Conclusion. SAADI's gains and its ranking hold when the fusion weights are selected without the evaluated queries.

AuthorsOpenReview ↗

Response to Reviewer mokb (Point 2: Per-query agent comparison, 3/4)

Point 2. Per-query agent comparison

To address your question, we compared SAADI with each agent in two ways. First, on every query, SAADI receives exactly the number of full chunks that the agent read on that query. Second, for A-RAG, we repeated the comparison with its search previews counted on both sides.

Protocol. As in Table 2, an agent's exposure on a query is the number of distinct full chunks placed in its LLM's prompts. Passages it retrieved but did not pass to the LLM, and A-RAG's search previews, are not counted. For every query, SAADI (BM25, published fusion weights) receives exactly the agent's full-chunk count, including zero chunks, and therefore zero retrieval credit, when the agent reads no full chunk. We use the five agents and four datasets of Table 2, with the same queries, evidence labels, and hit criterion. Re-scoring the saved agent traces reproduces Table 2's exposure counts and hit rates in all 20 pairs. Significance is assessed with paired bootstrap 95% confidence intervals over queries (10,000 resamples).

Table R3. Hit rate (%) when SAADI receives each agent's actual full-chunk budget on every query. Bold = significantly higher than the other method in that row. "Mean chunks" is the agent's average exposure; the matching uses each query's own count.

DatasetAgentMean chunksAgent HRSAADI HR
LooGLERAG-R14.571.190.7
Agentic-R3.454.188.6
A-RAG2.068.669.4
GRIP4.686.791.4
UMA3.187.887.9
MuSiQue-32KRAG-R18.186.071.2
Agentic-R5.575.263.2
A-RAG2.745.144.7
GRIP4.050.755.7
UMA19.690.391.0
QASPER-64KRAG-R17.620.019.7
Agentic-R3.816.113.1
A-RAG3.512.612.2
GRIP3.910.314.1
UMA14.827.331.1
NovelHopQARAG-R15.914.118.8
Agentic-R3.912.616.3
A-RAG3.99.016.5
GRIP5.014.217.7
UMA3.714.016.3

Results. Even with zero-budget queries included, 17/20 comparisons show either a significant SAADI advantage (11) or no significant difference (6). SAADI significantly outperforms every agent on NovelHopQA. The three comparisons that significantly favor agents are the same three that already favored them in Table 2, and no SAADI lead becomes a significant agent lead.

Why A-RAG on LooGLE changes. This pair shows the largest change from average-budget matching: SAADI's previous lead becomes statistically inconclusive. On 12.1% of the queries, A-RAG sees previews and answers without reading any full chunk, so per-query matching gives SAADI zero chunks. Yet SAADI's first-ranked chunk contains the gold evidence on 87% of these queries, and its top two chunks on 94%. The change therefore reflects A-RAG's stopping decision: SAADI gets no retrieval opportunity on queries where its ranking would supply the evidence. These are retrieval results only; they do not say whether A-RAG's answers from previews are correct.

Including previews. A-RAG previews, on average, 1.3-3.8 further chunks per query beyond those it reads in full, and the main comparison does not add them to SAADI's allowance. When previewed chunks are counted on both sides (toward SAADI's per-query budget and as evidence exposed to A-RAG), SAADI significantly outperforms A-RAG on all four datasets, by 3.0-8.3 HR percentage points.

Conclusion. Per-query matching preserves the comparison of Table 2, also under zero-chunk budgets, and counting A-RAG's previews preserves SAADI's advantage on all four datasets.

AuthorsOpenReview ↗

Response to Reviewer mokb (Point 3: End-to-end QA, 4/4)

Point 3. End-to-end QA

3.1 One reader, different retrieval methods

Protocol. Qwen3-30B-A3B-Instruct-2507 answers all queries on MuSiQue-32K and NovelHopQA with greedy decoding and the same prompt. Only the context changes. SAADI, Vanilla BM25, and Contextual Retrieval supply their top 10 chunks. RAPTOR and A-RAG supply their selected contexts. Closed-book and gold-evidence inputs provide reference conditions. We report token F1 and paired bootstrap 95% confidence intervals over queries.

Table R4. Answer F1 (%) with each method's usual context. Δ = SAADI minus the method, computed from unrounded scores. Bold differences are significant. A-RAG uses an LLM search loop. The other retrieval methods require no query-time LLM calls for retrieval.

Reader contextMuSiQue F1ΔNovelHopQA F1Δ
Closed-book9.3—12.6—
Vanilla BM25, top 1021.9+1.030.8+0.7
Contextual Retrieval, top 1022.80.029.9+1.7
RAPTOR21.7+1.117.6+13.9
A-RAG22.2+0.721.4+10.1
SAADI, top 1022.8—31.5—
Gold evidence only39.7—40.7—

MuSiQue denotes MuSiQue-32K throughout. NovelHopQA gold evidence comprises chunks containing its annotated evidence paragraphs.

SAADI achieves the highest or tied-highest F1 among the retrieval methods on both datasets. On NovelHopQA, it significantly exceeds RAPTOR, A-RAG, and Contextual Retrieval. On MuSiQue-32K, all methods are within about one F1 point, without significant differences from SAADI.

Matching context length. RAPTOR and A-RAG produce contexts that vary by query and are shorter on average than SAADI's top 10. For each query, we truncated SAADI's ranked context to exactly the comparator's context-token count, then ran the same reader again. We matched each comparator separately, preserving SAADI's ranking and cutting only the tail.

Table R5. Answer F1 (%) at equal context length per query. Δ = truncated SAADI minus comparator. Bold differences are significant.

DatasetComparatorMean tokensComparator F1SAADI F1Δ
NovelHopQARAPTOR1,70117.627.7+10.1
A-RAG2,54821.428.5+7.0
MuSiQueRAPTOR92321.721.0-0.7
A-RAG36022.215.6-6.6

SAADI's full top 10 averages 6,447 context tokens on NovelHopQA and 1,326 on MuSiQue-32K.

SAADI's significant NovelHopQA advantages persist at equal length. On MuSiQue-32K, it remains comparable to RAPTOR. A-RAG is more effective at its small budget of 360 tokens on average, while SAADI's full top 10 reaches comparable answer quality without LLM calls for retrieval.

Retrieval and answers. Gains over Vanilla are small and not significant (+1.0 and +0.7 F1). On MuSiQue-32K queries where SAADI retrieves all gold chunks, the reader scores 30.0 F1 with its top 10 versus 45.5 with only the gold passages for those same queries. This leaves room to improve how the reader uses retrieved evidence.

3.2 Long-context reader - full document versus SAADI

Qwen2.5-7B-Instruct-1M reads either the full document or SAADI's top 10, with the same questions, prompt, and greedy decoding. Full documents within the native 262K-token range use exact attention. The six longer books use dual chunk attention with YaRN, as its model card recommends. Every document is read in full. Significance uses paired bootstrap 95% intervals (10,000 resamples).

Table R6. Answer F1 (%) with the same 7B reader. Δ = SAADI minus full document. Bold differences are significant. Token ratio = full-document input / SAADI input.

Evaluation setFull documentSAADIΔToken ratio
MuSiQue, 42K–52K tokens28.626.3-2.335×
NovelHopQA, 10 books, 133K-253K23.330.2+6.930×
NovelHopQA, 6 books, 265K-760K19.630.8+11.265×
NovelHopQA, all 16 books21.730.5+8.845×

On MuSiQue-32K, the difference is not significant. Across all 16 novels, SAADI improves F1 by 8.8

Conclusion. SAADI's NovelHopQA gains persist at equal context length, and the long-context reader answers more accurately from its top 10 than from full books.

We hope these results address your concerns about the fusion weights, the agent comparison, and the end-to-end evaluation. We would be glad to clarify any remaining points, and we thank you for your careful and constructive review.

Reviewer_iVU8OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies chunk retrieval within a single long document when the evidence is distributed or multi-hop. SAADI makes one sequential pass over the chunks at index time. For each chunk, one LLM call (Qwen3-30B-A3B) extracts entities, conditioned on BM25-recalled entity records and a hierarchical summary history that is folded every f=5 units. Chunks are linked to entities by name, so aliases added later propagate back to earlier chunks. At query time, a chunk is scored by text, entity memory, and its level-one summary using the same base retriever; the three scores are z-normalized, combined with hand-tuned weights, and a single top-k is returned with no query-time LLM call. The paper also derives an exact index-time call count (<1.25N) and a break-even query count against agentic retrievers. Experiments on LooGLE, MuSiQue-32K, QASPER-64K, and NovelHopQA with five retrievers show HR@10 gains of 0.3–12.4 points over the base retriever (1.1–4.0 under BM25).

Critical strengths

The paper is careful about cost accounting: index-time cost is analytical and verified, and the evidence-exposure matching rule for agents is clearly stated. The pipeline is simple, retriever-agnostic, and the code is reproducible.

Critical weaknesses

I find the novelty limited. At its core the method is chunking, LLM tagging of each chunk with entities and summaries, and a better index built from those features. This is in the same family as Contextual Retrieval, RAPTOR, GraphRAG/HippoRAG, and LQCA. The ablation even shows that the entity-memory conditioning, which is the main selling point, contributes almost nothing at index time; most of the gain comes from the summary history.

More fundamentally, this only improves index quality and does not address the underlying long-document problem. Chunking itself breaks semantics, there is no optimal chunking strategy (the paper itself uses 100 vs. 500 tokens by dataset), and adding LLM-generated labels on top cannot recover what segmentation has already lost. This is reflected in the absolute numbers: HR@10 stays around 20–26% on QASPER-64K and NovelHopQA, and LQCA is 10–20 points ahead on two of the four datasets.

For long-document QA, RAG is not the most relevant comparison point. The most relevant baselines would be long-context models answering directly (e.g., Qwen2.5-1M), efficient long-context inference such as sparse attention (MInference), context compression with gist or beacon tokens (Activation Beacon), and KV-level memory or retrieval (InfLLM and multi-level KV caches). None of these is compared or seriously discussed. Agentic long-context reading and memory agents (MemWalker, ReadAgent, GraphReader, and the RL-trained MemAgent) are cited but not compared. The agentic baselines that are run (RAG-R1, Agentic-R, etc.) are Wikipedia multi-hop search agents used off the shelf in a setting they were not designed for. Only retrieval HR is reported, with no downstream answer accuracy, so the method cannot be compared with any of these end-to-end approaches.

The evaluation also lacks real long-context benchmarks. The 32K–256K settings are built by concatenating unrelated documents, which the authors admit are lexically separable; under that setup the summary channel may mostly be identifying which sub-document a query belongs to. Standard benchmarks such as LongBench (v2) and InfiniteBench are not used.

Finally, the method is very heuristic. The fusion weights, field weights, folding factor, and recall budget are all hand-tuned, and the fusion weights were selected on 100 queries drawn from the LooGLE and NovelHopQA test sets. Nothing is learned, and there is no evidence that the method generalizes or scales. The cost argument is also weaker than it looks: measured in generated tokens, break-even rises to hundreds or thousands of queries per document.

Questions for authors
  • How does SAADI (plus a reader) compare end-to-end on answer accuracy against a long-context model reading the full document (e.g., Qwen2.5-1M) and against efficient long-context methods such as MInference, Activation Beacon, or InfLLM, at matched compute or latency?

  • Can you evaluate on genuinely long, naturally coherent benchmarks such as LongBench v2 and InfiniteBench rather than concatenated documents? How much of the gain on the expanded MuSiQue/QASPER settings comes simply from identifying the right sub-document?

  • How sensitive are the results to the hand-tuned fusion weights and chunk size when they are selected on held-out data? Is there any path to learning these components so that the method generalizes or scales beyond per-dataset tuning?

7 comments
AuthorsOpenReview ↗

Response to Reviewer iVU8 (Summary and Point 1: Contribution and related methods, 1/4)

Thank you for the detailed review and for recognizing the cost accounting, the evidence-exposure matching, and the retriever-agnostic pipeline. We ran new experiments for your concerns; the main findings:

  • Point 1. Contribution and related methods. SAADI’s memory and summary scores together raise QASPER-64K HR@10 from 22.1 to 26.1, and SAADI has a higher mean HR@10 than Contextual Retrieval on three of four datasets.
  • Point 2. Full-document reading (Q1). Given the same Qwen2.5-7B-Instruct-1M reader, SAADI’s top 10 scores 8.8 F1 higher than the full document on documents of 133K–760K tokens, with 30–65× fewer tokens.
  • Point 3. Answer accuracy (Q1). With one fixed reader, SAADI gives the best or tied-best answers among the retrieval methods, and on long documents it is significantly ahead of RAPTOR, A-RAG and CR.
  • Point 4. Benchmarks (Q2). LongBench v2 and InfiniteBench do not mark the evidence, so retrieval cannot be scored on them; our benchmarks already include natural documents of up to 760K tokens.
  • Point 5. Held-out weights (Q3). Weights chosen by 5-fold cross-validation on held-out queries give a mean HR@10 of 56.1%, against 55.8% with the published weights, so the weights can be fit from data.

Point 1 is detailed below; Points 2, 3, and 4-5 follow in the next three comments.

Point 1. Contribution and related methods

What the document state contributes. SAADI belongs to the index-time enrichment family you list. Its contribution is a sequential construction: each chunk is indexed with the summary history and entity records accumulated while reading, which supplies the cross-chunk context that an isolated chunk lacks. Retrieval combines text, entity-memory and summary scores, returns the original chunks, and makes no LLM calls.

Memory during construction and retrieval. As you observe, conditioning extraction on recalled entity records adds little beyond the summary history (Table 12). Using the records for retrieval is different (Table 34, BM25 HR@10): on QASPER-64K, text alone gives 22.1, adding memory 24.5, adding summary 24.4, and both 26.1. Both together are best on every dataset. Across 100 cross-validated weight selections, memory receives a non-zero weight in 95% and summary in 83%. Later updates also help: on queries whose records gain a query term later, the median memory-channel rank of the evidence improves (e.g., 176 → 90 on QASPER-64K), while fused HR@10 changes stay within ±0.3 points. SAADI’s claim is the robustness of memory and summary together, not the strength of either alone.

Contextual Retrieval. We ran CR with the same indexing LLM, chunks, queries and five retrievers.

Mean HR@10 (%)LooGLEMuSiQue-32KQASPER-64KNovelHopQA
CR86.987.228.119.5
SAADI88.285.628.820.8

Across the 20 settings, four significantly favor SAADI and three favor CR. SAADI’s largest gain is +8.6 HR@10 points (QASPER-64K, Qwen). The two are complementary: adding SAADI’s memory to CR’s chunks significantly improves CR in three settings and significantly lowers it in none.

Indexing input and updates. SAADI’s per-call input grows only logarithmically with the document: 2.5-4.8K input tokens per chunk across our datasets, including documents of up to 760K tokens. CR puts the document in every prompt: 26K tokens per chunk on LooGLE and 204K on NovelHopQA. Appended text costs SAADI about 1.25 calls per chunk, and no earlier chunk is reprocessed. With prefix caching, which CR is designed for, CR’s billed input is lower.

RAPTOR, HippoRAG v2 and LQCA. At HR@10, SAADI is above RAPTOR on all four datasets and above HippoRAG v2 on three. LQCA is ahead on QASPER-64K (by about 15-21 points) and on MuSiQue-32K (by 14 with BM25, but only 0.4-5.3 with the four dense retrievers). SAADI is ahead on LooGLE with four of five retrievers and on NovelHopQA with all five: 18.2-22.1% versus 1.7-2.8%. LQCA resolves coreference and rewrites mentions in the indexed text, while SAADI keeps the original chunks and adds summary and entity-memory scores.

AuthorsOpenReview ↗

Response to Reviewer iVU8 (Point 2: Full-document reading and efficiency, 2/4)

Point 2. Full-document reading and efficiency (Q1)

To address your first question, we gave the same Qwen2.5-7B-Instruct-1M reader either the entire document or SAADI’s top 10 chunks, with identical questions, prompts, and greedy decoding. The comparison covers documents from 42K to 760K tokens, on the two datasets of our end-to-end evaluation. No document is truncated: exact attention is used within the model’s native 262K-token window, and dual chunk attention with YaRN for the six documents beyond it.

Table R1. Answer F1, input reduction, and reader latency, by document length. Differences are SAADI minus full-document reading; bold gains have paired 95% bootstrap intervals excluding zero. Input reduction = full-document / SAADI tokens. Seconds = uncached reader time per question, full / SAADI; indexing and retrieval are excluded.

Document tokensDatasetFull F1SAADI F1Difference [95% CI]Input reductionSeconds
42K–52KMuSiQue-32K28.626.3−2.3 [−5.1, +0.4]35×11.4 / 0.44
133K–253KNovelHopQA, 10 documents23.330.2+6.9 [+4.3, +9.6]30×63.2 / 1.32
265K–760K6 documents19.630.8+11.2 [+8.2, +14.4]65×131 / 1.32
133K–760Kall 16 documents21.730.5+8.8 [+6.8, +10.8]45×—

On par at about 50K tokens, more accurate on longer documents, and cheaper per question at every length. At 42K–52K tokens, the two are on par (95% CI [−5.1, +0.4]), with 35× less input for SAADI. From 133K tokens up, SAADI is more accurate, and the gain grows with length. Documents that fit entirely within the native window still show a significant +6.9 F1, so the benefit is not confined to context extension. Beyond the window the gain is +11.2 F1, although document content and attention configuration also differ there (on the ten in-window documents, the two attention modes give the same F1 within noise). Over all 16 long documents, SAADI gains 8.8 F1; on the longest (760K tokens), SAADI scores 33.9 versus 15.0 F1.

The advantage appears across evidence positions: for the first, middle, and final thirds of the ten in-window documents, F1 is 33.7 / 29.0 / 30.9 with SAADI versus 22.5 / 23.0 / 26.1 with full-document input. These results show that selected chunks can support better answers than the complete document despite segmentation; they do not imply that chunking preserves all information.

Efficiency and scope. At every length, SAADI reduces reader input by 30-65× and has lower uncached reader latency (Table R1). Even with a prefix cache, SAADI remains faster: 1.32 s per question against 1.97 s at 133K-253K tokens, and 0.25 s against 0.74 s at about 50K. Prefix caching reduces full-document prefill costs, and SAADI incurs indexing and retrieval costs, so these measurements are not a matched-compute or total-cost comparison.

We did not evaluate MInference, Activation Beacon, InfLLM or other KV-level memories. They make full-document reading cheaper or extend the window, and we tested full-document reading itself, with the model family you name. The two approaches are complementary; the same 1M reader does better with SAADI’s 6.5K tokens than with the whole document. We will discuss these methods alongside these results in Appendix A.1.

Indexing amortization. Separately, against agentic retrieval, Table 24 gives break-even ranges over the four datasets of 11-457 queries/document in calls, 15-1,844 in input tokens, and 256-7,715 in generated tokens. The higher generated-token threshold, which you point out, reflects writing the state during indexing. The measured indexing call counts match the analytical formula closely (Table 25), with at most 1.25 calls per chunk. SAADI therefore targets repeatedly queried documents; indexing is paid once, retrieval needs no query-time LLM calls, and answer generation retains its per-question cost.

AuthorsOpenReview ↗

Response to Reviewer iVU8 (Point 3: End-to-end accuracy and agentic baselines, 3/4)

Point 3. End-to-end answer accuracy and agentic baselines (Q1)

To address the request for downstream accuracy, we report answer F1 with one fixed reader, the same comparison at equal context length, and the agent comparison under a stricter per-query budget on all four datasets.

Table R2. Answer token F1 (%). One reader (Qwen3-30B-A3B-Instruct-2507, same prompt, greedy decoding) answers all questions; only the context changes. CR = Contextual Retrieval.

Reader contextMuSiQue-32K (42K–52K tokens)NovelHopQA (133K–760K tokens)
Closed-book9.312.6
Vanilla BM25, top 1021.930.8
CR, top 1022.829.9
RAPTOR21.717.6
A-RAG (LLM search loop)22.221.4
SAADI, top 1022.831.5
Gold evidence only39.740.7

Answer accuracy. SAADI has the highest or tied-highest F1 on both datasets, with no query-time LLM calls for retrieval. At about 50K tokens, the retrieval methods lie within about 1 F1 of each other; on the longer documents, SAADI significantly exceeds RAPTOR by 13.9 F1, A-RAG by 10.1, and CR by 1.7 (95% CI [+0.5, +2.9]; +2.2 on the nine documents CR processes whole). Although the strict hit rule (all evidence chunks in the top 10) gives low absolute hit rates on the harder datasets, the retrieved context is useful; it raises the reader from 9.3 to 22.8 F1 and from 12.6 to 31.5 over closed-book. A gap to gold evidence remains: on the MuSiQue-32K questions for which SAADI’s top 10 contains all gold chunks, the reader reaches 30.0 F1 from the top 10 and 45.5 from the gold passages of the same questions. Complete retrieval thus leaves a context-use gap, plausibly due to the surrounding passages; this does not isolate the reader alone.

Equal context length. RAPTOR and A-RAG pass fewer tokens than a top 10, so we truncated SAADI’s ranked list, per question, to their exact token count. On the longer documents SAADI still scores 27.7 versus 17.6 F1 at RAPTOR’s length and 28.5 versus 21.4 at A-RAG’s (both significant). The advantage persists after matching context length, so additional input tokens alone do not explain it. On MuSiQue-32K, the scores are 21.0 versus 21.7 at RAPTOR’s length (the 95% interval includes zero) and 15.6 versus 22.2 at A-RAG’s 360-token budget, where the agent’s compact selection works better. SAADI’s full top 10 reaches 22.8 against A-RAG’s 22.2 without an LLM search loop.

Agentic baselines. Each agent was run as released (policy model, prompts, and search tool unchanged) over the same chunks. We use them as references for query-time LLM reasoning under matched evidence, not as tuned competitors; task-specific adaptation could change their performance. In a new, stricter test, SAADI (BM25) receives exactly the number of chunks the agent read on each query.

Table R3. Hit rate (%), agent / SAADI, at the agent’s per-query budget. Bold = significantly higher (paired 95% bootstrap).

DatasetRAG-R1Agentic-RA-RAGGRIPUMA
LooGLE71.1 / 90.754.1 / 88.668.6 / 69.486.7 / 91.487.8 / 87.9
MuSiQue-32K86.0 / 71.275.2 / 63.245.1 / 44.750.7 / 55.790.3 / 91.0
QASPER-64K20.0 / 19.716.1 / 13.112.6 / 12.210.3 / 14.127.3 / 31.1
NovelHopQA14.1 / 18.812.6 / 16.39.0 / 16.514.2 / 17.714.0 / 16.3

Three comparisons significantly favor agents and 11 significantly favor SAADI, with SAADI wins on every dataset; in the other six the interval includes zero. A-RAG’s search previews are not counted above; with previews counted on both sides, SAADI is significantly ahead of A-RAG on all four datasets (+3.0 to +8.3 points). We did not run MemWalker, ReadAgent, GraphReader, or MemAgent. The closest system we evaluated is UMA, an agent trained end to end with an evolving memory bank: SAADI is significantly ahead of it on QASPER-64K and NovelHopQA, and the intervals include zero on the other two (Table R3). We will add the four systems to the related-work discussion.

Reviewer_zXpiOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes SAADI, which builds an entity memory and a hierarchical summary while reading a document’s chunks in order. Each chunk is processed by one LLM call to extract entities using the recalled records and the prefix summary as context. Query time is LLM-free, where each original chunk is scored by combining (1) the base retriever’s scores on the chunk text, (2) its linked entity records, and (3) its covering summary. The authors also derive the index-time call count and a break-even query count against agentic methods.

Critical strengths

The paper addresses a practical problem for retrieval that query-time reasoning may repeatedly resolve the same references and cross-passage connections. It is useful especially when query-time computation is limited.

Critical weaknesses

The experiments do not establish what SAADI gains from its specific indexing mechanism. Moving query-independent processing to index time is already central to existing methods such as Contextual Retrieval, RAPTOR, and GraphRAG. SAADI’s contribution therefore depends on the benefit of its sequential construction and evolving entity records. Table 12 provides evidence that history-conditioned extraction helps, but conditioning on recalled entity records adds little compared to only using history. The chunk-only extraction ablation can somewhat answer the question, but it still retains cumulative entity records and does not isolate the benefit of propagating later information back to earlier chunks.

Questions for authors
  1. Do updates to entity records from later chunks help retrieve earlier chunks?
  2. Please compare against Contextual Retrieval.
4 comments
AuthorsOpenReview ↗

Response to Reviewer zXpi (Summary and Point 1: Later updates and entity memory, 1/2)

Thank you for your insightful review and for recognizing the value of reducing repeated query-time processing. Below are short answers on later updates and on Contextual Retrieval (CR):

  • Point 1. Later updates and entity memory. Later updates do move earlier evidence up in the memory ranking (median rank 176 → 90 on QASPER), but the final HR@10 changes by at most ±0.3 points.
  • Point 2. Contextual Retrieval. SAADI has a higher mean HR@10 than CR on three of four datasets (up to +8.6 points), and it needs only 2.5-4.8K input tokens per chunk to build its index.
  • Point 3. End-to-end QA. With the same reader, SAADI’s answers beat CR’s by 1.7 F1 on NovelHopQA, and the two are tied on MuSiQue-32K.

Point 1 is detailed below. The following comment covers Point 2 (retrieval and indexing costs) and Point 3 (end-to-end QA).

Point 1. Later updates and entity memory

1.1 Do later updates help retrieve earlier chunks?

We compared final cumulative entity records with the records available when each chunk was indexed. We examined both the memory-channel ranking and the final fused retrieval result.

Table R1. Effect of later updates on queries whose evidence records acquire a query term from a later chunk. BM25, median evidence-chunk rank before → after updates. Lower ranks are better.

DatasetShare of queriesMemory-channel rank
MuSiQue-32K42.0%30 → 16
QASPER-64K34.8%176 → 90
NovelHopQA66.0%85 → 58

The rank statistics describe the selected subset in each row. The corresponding subset on LooGLE contains only 1.6% of queries.

Later information therefore improves the memory-channel ranking of earlier evidence in these subsets. Its effect on final retrieval is smaller. On these queries, the HR@10 change ranges from 0.0 to +0.6 percentage points. Across all queries, changes remain within ±0.3 points, with no statistically significant difference. We will distinguish this limited incremental effect from the broader contribution of entity-memory scoring in the revision.

1.2 What the construction and retrieval ablations establish

Index time, Table 12. Conditioning extraction on recalled records adds little beyond summary history, consistent with your observation. This ablation measures how recalled records affect construction.

Query time, Table 34. Adding entity-memory scoring to text-only retrieval increases BM25 HR@10 on all four datasets, as does adding summary scoring. Combining both gives the highest HR@10 on every dataset. For example, on QASPER-64K, HR@10 is 22.1 with text alone, 24.5 with memory added, 24.4 with summary added, and 26.1 with both.

Independent weight selection supports the use of both channels. Across 100 document-grouped cross-validation selections covering four datasets and five retrievers, the memory weight is non-zero in 95% and the summary weight in 83%.

Together, these results identify complementary contributions from summary history during construction and memory and summary signals during retrieval. SAADI’s claim is the robustness of this combination, not the strength of either part alone. The new experiment further establishes that later updates improve the memory-channel ranking on affected queries, while contributing little additional HR@10 after fusion in the evaluated settings.

The next comment details Points 2 and 3.

AuthorsOpenReview ↗

Response to Reviewer zXpi (Points 2 and 3: Contextual Retrieval and end-to-end QA, 2/2)

Point 2. Contextual Retrieval comparison

To address your question, we ran Contextual Retrieval (CR) as Anthropic describes it, with SAADI’s indexing LLM (Qwen3-30B-A3B-Instruct-2507), chunks, queries, and retrievers. We compare retrieval and indexing costs here, followed by end-to-end QA in Point 3.

2.1 Retrieval

Table R2. HR@10 (%), SAADI / CR, in all 20 settings. SAADI’s cells are Table 1 of the paper. Bold = significantly higher (95% paired bootstrap). Means use unrounded scores.

RetrieverLooGLEMuSiQue-32KQASPER-64KNovelHopQA
BM2596.5 / 96.077.7 / 82.226.1 / 26.922.0 / 21.7
Contriever85.7 / 84.383.7 / 86.125.8 / 26.018.2 / 18.5
BGE-M390.1 / 90.188.3 / 89.730.9 / 33.922.1 / 21.4
Jina82.0 / 79.788.1 / 89.727.2 / 28.121.1 / 19.4
Qwen86.7 / 84.790.0 / 88.634.1 / 25.520.6 / 16.7
Mean88.2 / 86.985.6 / 87.228.8 / 28.120.8 / 19.5
  • SAADI has the higher average HR@10: 55.8 vs 55.5 over all 20 settings, and on three of the four datasets: LooGLE (88.2 vs 86.9), QASPER-64K (28.8 vs 28.1), and NovelHopQA (20.8 vs 19.5). CR is ahead on MuSiQue-32K.
  • SAADI improves on CR by up to +8.6 points (QASPER-64K with Qwen), with further significant gains of +3.9 (NovelHopQA, Qwen), +2.3 (LooGLE, Jina), and +2.0 (LooGLE, Qwen). It is higher or equal in 11 of the 20 settings.
  • The two methods are complementary: adding SAADI’s memory to CR’s chunks significantly improves CR in three settings, with no significant decrease.

We also ran Anthropic’s hybrid setup (each dense retriever merged with BM25 by rank, with the cookbook’s fixed weights), applied in the same way to SAADI. Mean HR@10 over the 16 settings is 59.3 for SAADI and 59.8 for CR. SAADI is significantly higher with Qwen on QASPER-64K (+5.3) and NovelHopQA (+2.2), and CR in four settings, three on MuSiQue-32K (+2.2 to +4.1) and QASPER-64K with BGE-M3 (+3.5). The other ten settings differ by −1.5 to +0.8 points, without significant differences.

2.2 Index-time cost

  • SAADI’s input per chunk grows only logarithmically with the document, averaging 2.5-4.8K tokens across the evaluated datasets. CR’s prompt carries the whole document for every chunk (26K tokens on LooGLE, 204K on NovelHopQA), so without prefix caching, CR processes 28.5 times more input tokens over the four benchmarks.
  • SAADI does not require the full document to fit in the indexing window. Seven novels, counted with their original line breaks, exceed CR’s 262K-token window and are split.
  • Appending is local. A new chunk costs SAADI one bounded call (1.25 amortized, Section 3), and earlier chunks require no additional LLM processing. For CR, the prompt of every new chunk carries the whole updated document, and the contexts written earlier do not reflect the new text unless they are regenerated.

With prefix caching, which CR is designed for, CR’s billed input is lower.

Point 3. End-to-end QA

Protocol. One fixed reader (Qwen3-30B-A3B-Instruct-2507, greedy decoding, same prompt) answers all queries of MuSiQue-32K and NovelHopQA; only the context changes. Closed-book and gold evidence are reference conditions.

Table R3. Answer token F1 (%).

Reader contextMuSiQue-32K F1NovelHopQA F1
Closed-book9.312.6
Vanilla BM25, top 1021.930.8
Contextual Retrieval, top 1022.829.9
SAADI, top 1022.831.5
Gold evidence only39.740.7
  • SAADI gives better answers than CR on book-length documents. On NovelHopQA, SAADI scores 31.5 F1 against 29.9 for CR (+1.7 on unrounded scores, 95% CI [+0.5, +2.9]). On the nine novels that CR processes whole, the F1 gain is +2.2, so the advantage persists when CR sees the whole book.
  • On MuSiQue-32K, SAADI matches CR in F1 (22.8 for both). SAADI has the highest or tied-highest F1 among the retrieval methods on both datasets.
  • SAADI’s top 10 beats reading the whole book, and more so as books get longer. On NovelHopQA, a long-context reader (Qwen2.5-7B-Instruct-1M; dual chunk attention with YaRN beyond its 262K-token window) answers better from SAADI’s top 10 than from the whole book: 30.5 vs 21.7 F1. The gain rises from +6.9 F1 on the ten books of 133K–253K tokens to +11.2 F1 on the six books of 265K-760K tokens, with an input 30 to 65 times smaller.

Conclusion. SAADI complements CR, gives better answers on book-length documents even when CR reads them whole, and keeps indexing input per chunk small as documents grow.

We hope these results address your questions on later updates, the Contextual Retrieval comparison, and end-to-end QA. We would be glad to clarify any remaining points, and we thank you for your careful and constructive review.

Reviewer_zXpiOpenReview ↗

Response to authors

Thank you for your response. I have raised my score accordingly.

Thank you for reading our response and for raising your score. If any concerns remain about the later updates, the Contextual Retrieval comparison, or anything else, we would be glad to address them during the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion for the Area Chair and reviewers

Every method for long documents has to decide how much computation to spend at inference time and how much at indexing. There is a whole spectrum of approaches.

Agentic methods heavily lean towards inference-time cost and discover chunks dynamically as part of an agentic loop. Information from the set of chunks retrieved so far guides what to search for next. This flexibility is powerful, but it also introduces uncertain cost and latency. Long-context readers skip the search but read the whole document for every question. This offers stability and predictability but doesn't leverage the optimization potential of more targeted retrieval. Finally, index-time methods such as Contextual Retrieval, RAPTOR, and GraphRAG move work into indexing, but they need either the whole document in every prompt or a global tree or graph that must be updated when text is added.

SAADI is an index-time method and extends a very well-established family of related works. Nevertheless, we believe that SAADI offers a compelling novelty. Our method asks how much of the agents' flexibility can be moved into indexing at a low and predictable cost. It reads the chunks once, in order, and carries what it learned from earlier chunks into the next ones through an entity memory and a hierarchical summary, so this link between chunks (which agents discover anew each time they process a query) is built once and reused by every question.

The cost of SAADI's preprocessing is known in advance. A core contribution of the present work is a scaling analysis that shows how our design choices make SAADI suitable for corpora of arbitrary scale. A document of N chunks needs fewer than N·f/(f-1) LLM calls (Eq. 3), which is at most 1.25 calls per chunk with our folding factor f = 5, and each call reads only 2.5-4.8K tokens. At query time, there is no LLM call. Instead, retrieval is guided by three complementary scoring functions: the chunk's text, the memory, and the summary, so the right chunk can be reached even when its text alone does not match the question. The chunk text is never changed, and no chunk is removed. The extra channels only raise the right chunks in the ranking. New text can be appended without rebuilding the index.

The main concerns raised in the discussion, and how we addressed them:

  • Fairness of the agent comparison. We gave SAADI exactly the number of chunks each agent read on each query. SAADI is significantly better in 11 of 20 comparisons and the agents in 3.
  • Answer accuracy and Contextual Retrieval. Reviewers asked for answer accuracy and for a comparison with Contextual Retrieval, the closest method. With the same reader, SAADI reaches the same retrieval accuracy (55.8 vs 55.5 mean HR@10) and gives better answers on long documents (+1.7 F1).
  • Concatenated documents. Reviewers asked whether the gains come only from finding the right sub-document. When we rank only inside the correct source document, SAADI still gains about 4 MRR points on MuSiQue at every length.

We would like to thank the reviewers again. The discussion was constructive and helpful. We believe all questions were answered and some additional analysis points have helped us to better establish the unique contributions of SAADI. The revision will include all new experiments and an introduction that presents this framing more directly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.