acceptodds
Under review as a conference paper at ICLR 2027

Recall What Is Missing: Composing Evidence from Long-Term Agent Memory

25 views

Abstract

Under a limited context budget, a long-term agent must recall not merely memories that are individually relevant, but evidence that is collectively sufficient for the current query. Conventional memory access retrieves, scores, and truncates memories, often spending multiple context slots on redundant facts while leaving complementary evidence behind. We cast the read phase of long-term memory as *query-conditioned evidence composition*: each memory is valued by what it adds beyond the evidence already selected. We instantiate this operation with MemPursuit, a training-free mechanism that uses the residual of a sparse convex query reconstruction as a representation-space proxy for what the current memory set is missing. By reusing the query and candidate embeddings already available from retrieval, MemPursuit progressively composes nested evidence contexts without invoking a language model or cross-encoder to score each query–memory pair. The effectiveness of our method is verified on 3 popular benchmarks in comparison to strong baselines. In fixed-budget LoCoMo experiments, MemPursuit improves end-to-end accuracy by 7.2 percentage points at Top-10 and 5.5 percentage points at Top-20, with consistent gains across multiple memory frameworks, while achieving a 32.9 speedup in median reranking latency over fully batched BGE-v2-m3.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agent memory, sparse reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_4zYNOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses memory reading for long-term agents: given a candidate pool of memories and a fixed context budget, how should the reader choose which memories to put in context? The authors argue pointwise top-K selection wastes budget on redundant memories and misses complementary evidence. They propose MemPursuit, a training-free reader that casts selection as sparse convex query reconstruction — memories are picked greedily by alignment with the residual, with weights jointly refit after each addition. It comes with a redundancy/coverage analysis and a Frank–Wolfe-gap optimality certificate, and is evaluated on LoCoMo, LongMemEval-S, and BEAM.

Critical strengths

The method is elegant and the framing is right: reading as set-conditioned evidence composition rather than independent scoring. In controlled comparisons the gains are consistent — +7.2pp at Top-10 on LoCoMo (including +12.8pp on multi-hop), better gold-session coverage, and wins in all eight framework–budget configurations.

Critical weaknesses

No error bars anywhere, and several margins are small. The evaluation leans on an LLM judge with an answer-forcing prompt and a "be generous" instruction, which likely inflates absolute numbers and may favor a coverage-maximizing method like this one. The headline system comparison isn't controlled (different backbones and budgets across baselines), and some category-level regressions go unanalyzed — single-session-assistant drops to 57% vs. 95% for RAG, temporal reasoning to 61% vs. 77% for LightMem. Key design choices (ridge λ, refit vs. no-refit, stopping threshold) are also never ablated.

Questions for authors

Which model judges the Qwen3-answerer runs, and has the judge been validated against human labels? What explains the single-session-assistant and temporal-reasoning regressions? Can you ablate the refit and the stopping threshold?

Reviewer_fAFBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MemPursuit, a training-free method for selecting memories in long-term LLM memory systems. It reconstructs the query embedding and iteratively selects memories based on reconstruction residuals to improve evidence coverage. Experiments on LoCoMo, LongMemEval, and BEAM show improvements in question-answering accuracy across multiple settings, while reducing reranking latency.

Critical strengths
  1. The method formulates memory selection as iterative evidence combination, helping reduce redundancy without additional training. It can be readily integrated into existing memory systems.

  2. The paper evaluates multiple benchmarks, memory frameworks, and context budgets, and analyzes both evidence coverage and inference latency. The results are promising.

Critical weaknesses
  1. The method assumes that query reconstruction residuals indicate missing evidence, but semantic embeddings are not necessarily additive. The theoretical guarantees rely on strong assumptions about the embedding space, whose applicability to real-world data remains insufficiently validated.

  2. Key ablation studies separating residual-based selection from joint weight fitting are missing. The comparison with MMR also lacks thorough hyperparameter tuning and statistical significance analysis, making it difficult to determine the source and robustness of the improvements.

Questions for authors
  1. Could you provide ablation studies that separately remove residual-based selection and joint weight fitting? How sensitive are the MMR results to its diversity parameter?

  2. Could you report confidence intervals or paired significance tests for the main experiments, and clarify whether multiple runs were conducted?

  3. To what extent do real-world embeddings satisfy the fact-clustering and separation assumptions required by the theoretical analysis? How strongly does gold-session coverage correlate with coverage of the facts actually needed to answer a question?

Reviewer_89aeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper treats the read step of long-term agent memory as budgeted set selection and proposes MemPursuit, a training-free selector that sits on top of an existing memory system. Given a pool of P=200P=200 candidates and the embeddings the retriever already computed, it greedily reconstructs the query embedding as a convex combination of selected memory embeddings. Each step adds the candidate with the largest inner product with the current residual, then refits all selected weights on the simplex. The selection order is nested, so one run gives a context for every budget, and an optional rule stops once the reconstruction error stops improving. Proposition 1 gives conditions under a fact-cluster model for the first hh picks to cover hh required facts.

The main pipeline runs on Mem0-v3 with K=50K=50 and is compared with nine baselines, seven of them memory systems, on LoCoMo, LongMemEval-S and BEAM, with GPT-4o-mini and Qwen3-30B-A3B as answerers. Controlled LoCoMo experiments compare it with the hybrid order on the same pool, plug it into LightMem, MemoryBank, SimpleMem and Mem0 at K=10K=10 and 2020, and compare accuracy and latency with BGE cross-encoders and Qwen3.5 pointwise scorers. The appendix adds MMR at one setting, five encoders, two residual analyses and case studies.

Critical strengths
  1. The method is cheap and easy to adopt. It needs no training or extra model and reuses the retriever's embeddings. Median selection time is 8.67 ms per question on one CPU thread, against 274 ms for BGE-v2-m3 on a GPU (Table 12), and one run serves every budget. Tables 3 and 4 drop it into four different systems with the store, pool and budget held fixed. Rerankers are still more accurate at Top-20 (87.1 for BGE-v2-m3 against 85.1 in Table 9), so the argument rests on cost, but the cost is low enough to matter for deployed agents.

  2. The paired comparisons, which are the strongest evidence in the paper, show large gains at small budgets. On the same LoCoMo pool, accuracy goes from 73.8 to 81.0 at K=10K=10 and from 79.6 to 85.1 at K=20K=20 (Fig. 4b), with multi-hop going from 63.1 to 75.9. All four frameworks improve at K=10K=10, by 2.47 to 9.29 points, and all five encoders in Table 11 beat the hybrid order. I recomputed Tables 2 to 4 from their per-category numbers and they reproduce the reported overall scores.

  3. The framing is clear and the reporting is open. Treating memory reading as budgeted set selection with a computable objective is a sensible formulation, and the paper reads easily. The appendix gives full prompts, mean, median and P95 latency, batch-size sweeps and case studies that report where each memory ranked under both policies, which is what let me check most of the points below.

Critical weaknesses
  1. The selector is a soft form of MMR, and the paper neither compares it with tuned diversity methods nor ablates its own parts. From Eq. 9 the score is bm−∑sαsGmsb_m - \sum_{s} \alpha_s G_{ms}. With one memory selected, the simplex fixes its weight at 1 (the ridge is 10−1010^{-10}), so the second pick maximizes bm−Gm,m1b_m - G_{m,m_1}, which has the same argmax as MMR with λ=0.5\lambda=0.5. Fig. 2 illustrates exactly this step and the Fig. 8 test runs at ∣S∣=1|S|=1, so neither separates MemPursuit from MMR. Later picks subtract a weighted average of similarities to the selected set where MMR subtracts the maximum, which is a softer penalty. Greedy selection with a full simplex refit is itself standard (fully corrective Frank-Wolfe, OMP), so the case for the method has to be made empirically.

The empirical case is thin. The only diversity baseline is MMR at λ=0.5\lambda=0.5 in Appendix C.5 (81.04 vs 77.08 at K=10K=10, 85.06 vs 83.31 at K=20K=20, 87.08 vs 86.10 at K=50K=50), with no sweep over λ\lambda. DPP, xQuAD and submodular selection are cited and never run, and MMR is absent from the coverage analysis, the cross-framework tables and Table 2. The refit, which the contributions list as a core part of the method, is never ablated against uniform weights, a pursuit without refitting, or nonnegative weights without the sum-to-one constraint. A 4-point gap at K=10K=10 against one MMR setting does not show that reconstruction beats a tuned redundancy penalty, which is the paper's main methodological claim.

  1. The claim that MemPursuit recalls what is missing goes beyond what the score can do and what the experiments show. The residual lies in the span of the query and the selected memories, so a candidate gains only through its own similarity to the query and loses through its similarity to what is already selected. A bridge fact that reaches the query through an entity in a selected memory (the second hop of a chain) is pushed down. Every example in the paper (Fig. 1, Tables 5 to 8) is a set of parallel facts that each resemble the query: three deaths, two ages, two adoption dates, a weight warning and a diagnosis, two event dates. Removing near-duplicates is enough for these. Proposition 1 assumes the same regime. Assumption B.1 requires every required-fact memory to beat every distractor in query similarity by a margin γ\gamma, with nearly equal query similarities across facts (M>0M>0). The required facts are then the top cosine matches already, and the result only rules out one fact's duplicates crowding out the others. The case-study facts sit at hybrid ranks 11, 12, 26 and 41, and the paper never checks how often the assumption holds.

The coverage evidence is at the session level. Fig. 3 counts gold sessions, and swapping 70.8% of the selected memories for memories spread over more sessions raises that count by itself, yet the figure has no MMR or random baseline; Table 8 itself notes that source coverage can miss an omitted fact. By my count of the Fig. 8 histogram, about 53 of the 136 questions favor the semantic comparator. The paper also does not discuss a cost of the redundancy penalty: dated versions of the same fact look redundant in embedding space, and temporal questions need them. On LongMemEval temporal reasoning with GPT-4o-mini, MemPursuit answers 81 of 133 questions, against 99 for Mem0 and 103 for LightMem, and with Qwen it scores 62.41% against 75.94% for Mem0. Without a Mem0-v3-only row (weakness 3) I cannot say how much of this comes from the pursuit, but the paired results lean the same way. The LoCoMo temporal gain at K=10K=10 is 0.9 points (73.2 vs 72.3, Fig. 4a), and MemoryBank loses 4.05 temporal points at K=10K=10.

  1. Table 2 does not isolate MemPursuit, and its budgets are not matched. MemPursuit runs at K=50K=50 on Mem0-v3, while the baselines keep their native budgets: 10 for A-MEM and MemoryOS, 40 for MemOS, 50 for MemoryBank, 60 for LightMem and Mem0, and all memories for SimpleMem. Fig. 4b shows that budget alone moves the hybrid order from 73.8 at K=10K=10 to 84.2 at K=50K=50. That 84.2 is the Mem0-v3 backbone with no pursuit, and with GPT-4o-mini it already matches MemOS (84.48), the strongest baseline. Of the 21.3-point gap between MemPursuit (87.08) and the Mem0 row (65.78), 2.9 points come from the pursuit. Appendix A.3 suggests the Mem0 row is the older paper release (two speaker stores, K=60K=60) and not Mem0-v3. There is no backbone-only row for LongMemEval or BEAM, so the best results on all three benchmarks cannot be credited to MemPursuit; with GPT-4o-mini the LongMemEval lead over Mem0 is 356 against 352 correct answers out of 500.

No result comes with a confidence interval or repeated runs. The gains in all eight framework-budget configurations include LightMem at K=20K=20, where the gain is 0.32 points, or 5 of 1,540 questions. In Table 9, MemPursuit's own per-category accuracies give 84.5 overall instead of the reported 85.1, while every other row in that table reproduces.

Questions for authors
  1. On the same pools at K=10K=10 and 2020, please run MMR over a range of λ\lambda (say 0.3 to 0.9) and at least one of DPP or facility-location selection, and add them to Table 3 and Fig. 3. Please also ablate the refit with uniform weights over the selected set, a pursuit without refitting, and nonnegative least squares without the sum-to-one constraint. Does MemPursuit beat the best-tuned MMR?

  2. Please add MMR and random selection from the pool at the same KK to Fig. 3, and report fact-level coverage on LoCoMo using the judge procedure from Appendix C.3. Does the coverage gain hold at the fact level?

  3. On multi-hop questions where the second fact shares an entity with the first but not with the query, how does the second fact's score change once the first is selected? On how many questions does Assumption B.1 hold, for example with all evidence memories above all distractors in cosine similarity?

  4. On LongMemEval temporal reasoning, please compare the Mem0-v3 hybrid order and MemPursuit at the same KK and look at the memories the pursuit displaces. Are they dated versions of the same fact?

  5. Please add the Mem0-v3 hybrid order at K=50K=50 without MemPursuit to Table 2 on all three benchmarks, and either run the baselines at K=50K=50 or run MemPursuit at each baseline's native budget. Which Mem0 release is in the Mem0 row? Please also report paired bootstrap intervals for Tables 2 to 4 and explain the 84.5 vs 85.1 gap in Table 9.

2 comments

Thank you for your great detailed analysis and responses, the updated work with these results is much stronger and ive raised my score.

Thank you very much for taking the time to carefully read our responses and reconsider your assessment of our work. We sincerely appreciate your positive feedback and continued support. Your thoughtful and constructive suggestions have been very helpful in improving our work. Thank you again for your time and consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow up on Rebuttal Discussion

Dear Reviewers,

We hope this message finds you well.

As the discussion phase is scheduled to conclude in less than 48 hours, we would like to kindly ask whether our rebuttal has adequately addressed your concerns. We would greatly appreciate any further feedback you may have.

Thank you again for your time and effort in reviewing our work. We sincerely appreciate your consideration and look forward to hearing from you.

Best regards,

The authors

AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

We thank you and the reviewers for your time and constructive feedback. We summarize the main additions addressing the shared concerns below.

  1. Controlled comparisons isolate MemPursuit’s contribution. Updated Table 2 compares Mem0-v3, tuned MMR, and MemPursuit using the same memory backbone, candidate pools, and budget. Across three answer-and-judge repetitions with contexts fixed, MemPursuit achieves higher mean scores on all three benchmarks with both answerers. Paired bootstrap analyses further support the gains at small budgets.

  2. Tuned baselines and component ablations establish the value of joint reconstruction. Expanded Table 4 includes an MMR coefficient sweep, tuned DPP, and facility-location selection. MemPursuit achieves the highest accuracy across all eight framework–budget settings. At K=10 on Mem0-v3, it exceeds the best tested MMR setting by 3.96 percentage points. Removing joint refitting reduces accuracy from 81.04% to 79.16%, supporting this design choice.

  3. Fact-level coverage demonstrates more complete supporting evidence. At K=10, MemPursuit achieves 75.4% complete fact coverage, compared with 67.8% for tuned MMR and 67.3% for hybrid retrieval. Added bridge examples demonstrate preservation of connecting evidence beyond duplicate removal. We also clarified the theoretical definitions and added analyses across five encoders supporting local fact clustering and residual preference for uncovered evidence.

  4. Human evaluation supports the gains independently of the LLM judge. A blinded assessment of 300 answers to 100 randomly sampled questions gives MemPursuit 80% accuracy, versus 72% for the backbone and 73% for MMR. Both paired gain intervals exclude zero. The advantage also persists after removing the general-answer generosity instruction from the judge prompt.

  5. The budget clarification distinguishes retrieval counts from actual context size. Some frameworks supplement retrieved memories with linked records or auxiliary context, so matching nominal retrieval counts alone does not equalize context budgets. We report actual memory-context lengths, showing that several baselines use comparable or larger contexts. The matched comparisons in Tables 2–4 hold the backbone, candidate pools, and selection budget fixed within each framework to isolate MemPursuit’s gains.

  6. Matched category analyses clarify the role of memory extraction. The SSA and temporal gaps to other systems already exist before reranking. On the shared Mem0-v3 database, MemPursuit maintains mean temporal accuracy with GPT-4o-mini and improves it with Qwen, while improving overall accuracy with both answerers.

  7. Empirical evidence supports the geometric structure underlying our theoretical analysis. Analyses across five encoders confirm local fact clustering and a consistent residual preference for uncovered evidence. A separate analysis also supports separation from entity-incompatible distractors.

We appreciate Reviewer 89ae’s follow-up recognizing the strengthened results and raising their rating. The added evidence supports MemPursuit’s ability to assemble complementary memories within limited budgets, without training or additional model calls during selection. We sincerely thank you for the time and effort you have devoted to coordinating the entire peer-review and discussion process and carefully considering the reviews, our responses, and the revised manuscript.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.