- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies when an agent should construct persistent memory. Preparing a memory store before any questions arrive can spend effort on unused information, while starting a fresh search for every question discards useful work. MemSeeker keeps cited working notes during search and turns them into persistent memory after answering, without another LLM extraction pass. Later questions can use these memories directly or use them to guide further search.
Critical strengths
I find the construction timing worth studying. The current question tells the agent what information is useful, and the search has already paid the cost of finding and organizing it. MemSeeker uses this work twice, instead of running a separate memory-building process afterward.
Critical weaknesses
The benefit of persistence on general query sequences is not yet well separated from better single-query search.
The full system changes both the organization of the search state and whether knowledge is saved across questions. The dedicated MuSiQue experiment creates explicit reuse opportunities: later questions have earlier anchor questions with equal or nested supporting evidence, and the questions are further filtered. This is a reasonable way to test whether reuse can work, but it does not tell us how much it usually helps without that selection.
The general benchmark suite does not report the same MemSeeker agent with only cross-query persistence turned off. We therefore cannot attribute the full 83.5% token saving to persistent memory; some of it can come from more efficient search within each question.
Questions for authors
On existing corpora with multiple queries, what happens if the search agent is kept unchanged and only cross-query persistence is disabled? Breaking down the result by evidence overlap between questions would help show when saving the working notes is useful.
3 comments
Thank you for running the same search agent with persistence disabled and breaking down the results by evidence overlap. I’m keeping my rating at 2 because enabling persistence saves 6.1% of tokens on the general suite at essentially unchanged accuracy, while the larger gains on MuSiQue come from questions deliberately paired to share evidence.
Interpretation of 6.1% of token saving, MuSiQue reuse design, and the contribution of QTMC
Thank you for the follow-up. We would like to clarify three points about the interpretation of these findings.
- Unchanged accuracy and 6.1% aggregate savings. HET is designed to avoid repeated search and synthesis, so maintaining accuracy at lower token cost demonstrates its intended benefit. The aggregate saving is indeed 6.1%, but it averages queries with and without opportunities for reuse. Our general-suite breakdown already separates these cases and establishes where persistence helps (Table 2).
- What the MuSiQue experiment establishes. MuSiQue deliberately controls reuse opportunities, but paragraph-level evidence overlap does not imply identical information needs or automatic answer reuse. Questions sharing supporting paragraphs may require different facts; others require selecting intermediate conclusions or extending them with new evidence. Successful reuse therefore depends on what knowledge HET retains and how the agent applies it. This experiment isolates those mechanisms, while the general suite examines their benefits under heterogeneous workloads.
- QTMC’s central contribution is constructing memory from the work performed to answer actual queries. In real settings, memory access to one corpus (like one area in the raw history) can be as sparse as LME, CL, etc, or as dense as designed MuSiQue. MaS always incurs construction costs before knowing what will be asked, processing the whole corpus even if only one query will refer to it in the future. Stateless MaA operate when query arrives, but the search cost grows linearly as queries arrive. MemSeeker uses the same structured working notes for efficient search and subsequent memory construction. Single-query corpora benefit from avoiding upfront construction; later queries with reusable evidence can additionally avoid repeated work. Evaluating QTMC solely by the 6.1% persistence increment leaves out this central contribution: making memory construction demand-driven and integrating it with efficient query serving.
If you are satisfied with our analysis, we would greatly appreciate your re-consideration in adjusting the evaluation scores accordingly. We sincerely look forward to your feedback.
Isolating the contribution of cross-query persistence
Thank you for suggesting this comparison. We added MemSeeker w/o HET result on the general suite, keeping the within-query search agent unchanged and disabling cross-query persistence. On the general suite, HET reduces tokens per query from 46.89k to 44.03k (6.1%), with comparable scores (70.38 vs. 70.52, Table 1). This isolates persistence’s incremental benefit; the 83.5% saving over DCI-lite measures the full system’s efficiency.
The 6.1% aggregate averages questions that can reuse prior evidence with questions that cannot; grouping them by overlap separates that average. Table 2 examines 415 questions from multi-query corpora that require agentic search without HET, selected independently of HET-side behavior. The analysis retains 134 zero-overlap cases, which incur a small 2.1% overhead. Nonzero-overlap groups save 12.0%–50.2%, with larger savings generally observed at higher overlap. Notably, 82 questions requiring search without HET are answered directly after initial retrieval with HET, saving 87.1% of tokens.
Reuse opportunities are limited in these benchmark protocols. CL, CL-Life, and LongMemEval provide one query per corpus in our evaluation. Appendix D discusses how other benchmarks tie questions to specific content or evidence, with some construction procedures explicitly limiting redundancy.
The new case studies in Appendix E.2 show how cross-query connections help without deliberately constructing overlapping question pairs. Retained purchase records enable a correct direct answer in Memora (74.3k → 22.7k tokens); a retained event timeline guides retrieval of missing actions in AMA (nine → one additional search round, with a correct answer). These illustrate two benefits of persistence: answering from previous findings and focusing subsequent search.
These observations are consistent with the MuSiQue results. General-suite overlap is measured from agent trajectories; MuSiQue’s gold supporting-evidence annotations allow us to control reuse opportunities and examine these mechanisms more precisely. Together, the experiments distinguish the overall benefit on existing query streams from the gains achievable when successive questions share reusable evidence.