acceptodds
Under review as a conference paper at ICLR 2027

Memory Navigates, the Record Decides: Auditable Evidence for Million-Token Agents

Abstract

Even when a million-token history fits in the context window, an agent cannot attend to all of it at every step, so it answers from compressed memory. Memory systems that answer from model-written summaries and facts can use outdated or contradictory information and cannot tell when evidence is missing. We observe that compiled memory can afford to be lossy if it only points to source lines that the agent re-reads, so we use it only to navigate to evidence and let the re-read record decide the answer. NavMem, a training-free compiler, builds a multi-resolution continuity view of the history and a dynamics view that tracks updates, conflicts, and event order. Code stamps each item with the address of its source lines, so every item is auditable, and the answer protocol asks the agent to check each claim against the re-read lines and to abstain when they do not support it. When every system answers with Qwen3.5-122B, NavMem scores 0.715 macro on BEAM's 1M tier, 0.100 above the strongest baseline, Mem0 (paired 95% CI [0.076, 0.125]), and 0.755 accuracy on LoCoMo, 0.066 above LIGHT ([0.051, 0.083]).

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, long-term memory, conversational memory, long-context reasoning, provenance, evidence grounding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HyauOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses source traceability, fact updates, and abstention when evidence is insufficient in long-term conversational memory. NAVMEM uses immutable raw conversations as its evidence substrate. A four-layer continuity summary—now, today, recent, and archive—preserves the historical outline, while fact and event ledgers organize dynamic information. The compiler attaches source-line addresses to memory items, and rules generate supersession and conflict edges. At answer time, a keyword router guides candidate retrieval and source rereading. The model is prompted to label each required condition as supported, contradicted, or missing before answering.

Critical strengths

Summaries and fact entries retain source addresses, allowing old and new values to be traced through the dynamics ledger rather than leaving compressed content as the only available material. The interface supports locating errors in retrieval, updating, or answering and distinguishes semantic verification from guarantees about address validity. NAVMEM outperformed Mem0 on 33/35 BEAM conversations. The gain remained 0.103 on conversations 11–35, which were excluded from router tuning, and 0.080 under a second judge. LoCoMo retained a 0.066 gain when all methods answered in one round. These results support the complete system’s practical value beyond BEAM’s multi-turn tool budget.

Critical weaknesses
  1. Table 9 allows NAVMEM up to 30 turns and route-quality retries, while external baselines answer in one round with unmatched input/output budgets. The source-linked methods with source-rereading protocols in Table 7 lack comparisons under the same protocol. LoCoMo supports the complete system in a single-round setting, but does not establish the new design’s added value over these alternatives.

  2. Section 4.4 counts rereads, address citations, and gold-source coverage without checking whether individual assertions are supported or testing answer independence by perturbing summaries while fixing evidence. Proposition 1 assumes this independence. The authors disclose this boundary, but the results do not establish how much rereading reduces unsupported answers or incorrect abstentions.

  3. Supersession edges are absent from 84.2% of facts. Adding dynamics also adds fact and event content, so its gains cannot be attributed to update or conflict links. Determinism and acyclicity in Appendices C/D do not guarantee semantic correctness. Controls retaining identical entries while disabling edges, and checks of how incorrect links affect answers, are missing.

  4. All results rely on prebuilt ledgers. The evaluation does not interleave session capture, consolidation, and queries or report construction cost, index growth, actual answer-token usage, or latency. The results therefore leave accuracy gains under matched resources and the cost of maintaining growing histories unresolved. The 10M scores do not replace these measurements.

Questions for authors
  1. After matching answer-token, tool-turn, and retry budgets, how much gain remains when Mem0 or a related source-linked method receives the same raw-text search and rereading protocol? What changes when update or conflict edges are disabled while ledger content is fixed? These controls would distinguish contributions from answering strategy, ledger content, and link structure.

  2. Could you audit complete answers for assertion-level source support, stale-fact use, and incorrect abstention, and describe the adjudication procedure? With reread evidence fixed, do changes to candidate values in summaries change answers? Please also check whether incorrect update or conflict links cause current values to be missed or old values to be accepted.

  3. Could you repair the tool loops in both pair arms and rerun all 700 questions with identical candidate formats and retry rules? Please report each arm’s actual macro and paired difference on the same questions, explaining how the distribution of earlier failures across questions and conversations affects conclusions about continuity, dynamics, and their complementarity.

  4. Could you report results with session-by-session construction interleaved with queries, construction calls per million history tokens, storage growth, and actual tokens/latency per question? Do the gains remain against a resource-matched baseline, and what are the separate maintenance costs of continuity compression and append-only dynamics?

Reviewer_Q3YWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the unreliability of compiled memory in long-horizon agents, which may answer from stale or contradictory summaries and cannot recognize missing evidence. It proposes NavMem, a training-free memory system in which compiled views serve only to locate evidence, while the raw transcript remains authoritative. Every memory item is linked to its source lines, and the agent is instructed to re-read and verify the source before answering or to abstain when support is missing. Experiments on long-horizon memory benchmarks show gains over existing memory systems, supported by ablation and scaling analyses.

Critical strengths
  1. The paper articulates a clear principle for agent memory: compiled memory should navigate to evidence rather than serve as the answer, so it may be lossy as long as the source record stays authoritative and addressable. This principle targets well-known failure modes of consolidating memory and makes the memory auditable.
Critical weaknesses
  1. The central claim is not supported by the evidence. The paper argues that the re-read record decides the answer, yet verification is only prompt-instructed, the premise of Proposition 1 is never tested, and no ablation removes or enforces the protocol. Meanwhile, an agentic evidence-only arm already matches the strongest baseline, and every baseline answers in a single call without matched budgets. The reported gains therefore cannot be attributed to the proposed mechanism.

  2. The experimental pipeline lacks the control needed for reliable conclusions. Routing rules and retries were tuned on conversations that also enter the main results. Extraction models differ across subsets of conversations. Failed NavMem probes were selectively rerun, and ablation arms whose tool loop failed on a large share of questions were partly excluded. Archived responses are truncated, and whether embedding fusion was active is unrecorded. Each choice is disclosed, but together they leave too many uncontrolled factors for the comparisons to be conclusive, while the heavy formalism adds little that the system actually enforces.

Questions for authors
  1. Can the authors compare against a budget-matched agentic baseline, for example Mem0 or NativeRAG wrapped in the same multi-turn tool loop with transcript read and search tools and the same retries, and report per-question tokens, tool turns, and latency for all systems? Without this comparison, the contribution of the compiled views cannot be separated from that of agentic access to the transcript.

  2. I am curious how much the verification protocol itself contributes. Could the authors run the full system without the re-read, audit, and abstain instructions, or alternatively enforce verification with an entailment check on the cited spans?

  3. The dynamics view is motivated by updates and conflicts, yet the gain over Mem0 on knowledge update is small, and most facts carry no supersession edge. I am wondering whether removing only the supersession and conflict edges, while keeping the fact and event ledgers, measurably changes performance on update and contradiction questions.

  4. Do the results hold under a consistent setup (one extraction model, the same rerun policy for all systems), and how much do they vary across memory builds?

Reviewer_J94mOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

NAVMEM is a training-free memory system built on the premise that compiled memory should only navigate to evidence, while the raw transcript decides the answer. It keeps an append-only record of session transcripts and compiles two lossy, source-addressed views: a continuity view with four time layers (now, today, recent, archive) that an LLM rewrites and compresses, and a dynamics view of fact and event ledgers in which an LLM extracts candidates but deterministic code derives supersession and conflict edges (an LLM pass adds only 1.4% of edges). Code stamps every item with the exact session and line range it came from, so compression can drop content but never its address. At answer time a keyword router picks one of eight retrieval routes, and the agent runs a tool loop (transcript reader, window reader, outline, lexical search) under a prompted protocol that asks it to re-read the cited lines, label each condition as supported, contradicted, or missing, emit an evidence audit, and abstain without support; the paper is explicit that nothing enforces this. With Qwen3.5-122B answering for every system and GPT-5.5 judging, NAVMEM scores 0.715 macro on BEAM's 1M-token tier versus 0.615 for Mem0, the strongest baseline (paired 95% CI [0.076, 0.125], 33 of 35 conversations), leads on 8 of 10 abilities with the largest gains on information extraction, instruction following, temporal reasoning, abstention, and event ordering, holds under a second judge (+0.080), and reaches 0.755 on LoCoMo versus 0.689 for LIGHT. Ablations show raw-evidence access is the most important component (evidence-only 0.629; either compiled view alone scores lower), each pair beats its parts, and the continuity view alone collapses on knowledge update (0.243 vs 0.761). Mechanism diagnostics find the agent re-reads the transcript in 98.9% of episodes, cites source lines in 82.9%, cites a gold source in 58.2%, and episodes that cite a gold source score 0.120 higher, which the authors present as observational and as motivation to enforce verification in future work. Caveats visible in the paper: the main row is a single run, the router and retry ladder were tuned on 10 of the 35 BEAM conversations that are also scored, baselines run with their own readers and some with memories built by a smaller model, and performance drops to 0.575 at the 10M tier.

Critical strengths
  • A clear, defensible design principle. Compiled memory locates evidence; the raw transcript decides. This reframes the memory problem so that lossy compression is safe by construction, because every item keeps a line address that code stamps and the model cannot corrupt. The idea is simple, well motivated by the failure modes in Figure 1, and carried consistently through the system.
  • Large, statistically careful margins on a hard benchmark. On BEAM's 1M tier NAVMEM beats every baseline by 0.10 or more in macro, wins on 33 to 35 of 35 conversations, and the paired conversation-cluster bootstrap intervals exclude zero for every comparison with Holm correction. The result holds under a second judge from a different model family, on the 25 conversations held out from router tuning, with a second answering model, and excluding the event-ordering ability whose metric convention could be disputed.
  • An informative ablation grid. Seven configurations across both benchmarks, each pair outscoring its components, and a telling control: raw-transcript access with no compiled memory reaches 0.629, statistically tied with Mem0. This quantifies how much the views add beyond plain re-reading (0.086) and is reported rather than hidden.
Critical weaknesses
  • The comparison is not budget-matched. NAVMEM runs an agentic loop with up to 30 tool turns, six tools, and three retry types. Every baseline answers in a single call. Appendix F states "input and output budgets are not matched." The evidence-only arm, which shares the tool loop, already reaches 0.629 and is statistically tied with Mem0 (0.615), so about half of the 0.100 headline margin is attributable to multi-turn reading rather than to the compiled views. A baseline that gives Mem0 or LIGHT the same tool loop over their own memory is the missing control.
  • Baselines are handicapped in ways that favor NAVMEM's strongest abilities. Mem0 runs with add-only extraction and no update or delete pass, which directly weakens it on knowledge update and contradiction resolution. LIGHT and A-MEM build memory with a smaller model in some cells, MemoryOS in all. A-MEM packs two messages per note against its default of one. The authors wrote every baseline's answer prompt. LIGHT scoring 0.374 under Qwen3.6-35B, far below NativeRAG, indicates a harness problem that was not investigated.
  • The title claim is not what the system does. Verification is a prompt instruction with no enforcement. The agent cites a gold source in only 58% of episodes with one, covers all reference groups in 52% of multi-state questions, and emits the evidence audit in 73%. In a large minority of cases the record does not decide. Proposition 1 assumes the premise the paper admits is unverified, so it adds nothing.
  • The judge sees NAVMEM's evidence audit. Rubric-nugget judging is done on the complete response, which for NAVMEM includes line citations and a supported/contradicted/missing audit. Baselines emit bare answers. A citation-laden response can read as more complete to an LLM judge independent of correctness. The second judge agrees with the first but has the same exposure.
Questions for authors
  1. Can you run Mem0 or LIGHT with the same 30-turn tool loop, retries, and evidence protocol over their own memory stores? Evidence-only access already ties Mem0, so a budget-matched baseline is needed to show the views add value beyond agentic reading.
  2. Why was Mem0 run with add-only extraction? Can you report Mem0 with its update and delete pass enabled, since knowledge update and contradiction resolution are the abilities under study?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.