acceptodds
Withdrawn from ICLR 2027

HandleMem: Source-Linked Complementary Handles for Adaptive Evidence Acquisition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-horizon agents often compress past interactions into compact memories. These representations play two roles: they guide the retrieval of relevant prior episodes and preserve evidence for later question answering. The difficulty is that compression must decide what to retain before the future use of the memory is known. As a result, it may discard cues needed to locate a relevant interaction, creating an access gap; it may also discard details needed to support an answer, creating an evidence gap. Since richer compression alone cannot eliminate these gaps, we argue that compact memory should instead provide access to retained evidence rather than replace it. We introduce , a long-term memory framework built around this separation. It constructs source-linked complementary event- and detail-level handles that expose different retrieval cues for the same source segment, improving access coverage while preserving links to the original interaction for recovering omitted evidence. At query time, adaptive source resolution performs multi-round retrieval driven by the current information need and newly retrieved cues, using each observation to refine what to search for next, before resolving selected memories back to their sources for final answering. On LoCoMo, achieves 57.40 F1 and 81.69% LLM-as-a-Judge accuracy. On LongMemEval-S, it achieves 80.20% LLM-as-a-Judge accuracy, obtaining the highest overall scores among the compared methods on both benchmarks. Our code is available in an anonymous repository at https://anonymous.4open.science/r/HandleMem-A11Dthis link.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Long-Term Memory, Memory-Augmented Generation, Multi-Granular Retrieval, Evidence Recovery, Source-Grounded Answering

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Wek8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the HandleMem method, which stores raw dialogue segments as evidence for generating answers and indexes them using complementary event-level and detail-level handles. Evaluations on LoCoMo and LongMemEval-S demonstrate that this method achieves improvements over raw-text RAG and previously reported results for memory systems.

Critical strengths

S1. The distinction between retrieved representations and source evidence is clear and of high practical value. Answer control using only handles fixes the selected record, causing accuracy to drop from 80.20% to 73.40%, which directly proves that the recovered source text provides information beyond the selected compressed handles.

S2. Very clearly written.

Critical weaknesses

W1. Ideas and intuition are quite incremental. Furthermore, the scores for Full Context and memory systems are imported from previous reports, while HandleMem and RAG are evaluated here. Sharing the GPT-4o-mini model name does not guarantee that the model, answer prompts, judge prompts, retrieval budget, or preprocessing are exactly the same. Although the internal multi-turn RAG control is helpful, it does not address how large the reported advantages over ES-Mem and other systems are under a unified evaluation process. This limits the claim of superior performance.

W2. The overall cost conclusion heavily depends on how many questions each constructed memory is reused for. Table 4 adds the offline tokens per case to the online tokens per question and concludes that online costs are offset by offline savings. For reuse, the relevant token cost is the construction cost plus the number of queries multiplied by the online cost. HandleMem's online cost is 24.0K tokens, while the listed memory systems are only 0.7–3.7K, and the online latency is also higher. Without specifying the workload and amortization, the reported total savings cannot establish the trade-off for long-term conversational memory. This appears to be a serious flaw for a system that deliberately shifts work to query time.

Questions for authors

Q1. What specifically does the offline "case" in Table 4 refer to? How many queries share this construction? Please report the construction cost per unit and the cost per query, and explain the break-even workload. Is the claimed cost advantage only applicable to single-query scenarios?

3 comments

Dear Reviewer Wek8,

Thank you very much for your thorough review and constructive feedback! We address your concerns regarding cross-paper evaluation comparability and multi-query cost amortization below.

W1: Ideas and intuition are quite incremental. Furthermore, the scores for Full Context and memory systems are imported from previous reports, while HandleMem and RAG are evaluated here. Sharing the GPT-4o-mini model name does not guarantee that the model, answer prompts, judge prompts, retrieval budget, or preprocessing are exactly the same. Although the internal multi-turn RAG control is helpful, it does not address how large the reported advantages over ES-Mem and other systems are under a unified evaluation process. This limits the claim of superior performance.

Our contribution combines source-linked complementary handles with adaptive source resolution to address the Access Gap and Evidence Gap caused by write-time compression. Detail-level handles are generated conditioned on the event-level handle to expose complementary retrieval cues, with both views linked to the same retained source (Section 2.2). During retrieval, OpenSource lets the resolver recover omitted source details and use them to guide subsequent search and selection (Section 2.3). Together, these mechanisms connect complementary memory access with recoverable source evidence.

To address your main concern about cross-paper comparability, we additionally evaluated LightMem and Nemori under both their native protocols and the same evaluation configuration as HandleMem. These controlled experiments show that HandleMem maintains its performance lead over both baselines under the unified protocol, as detailed below.

1. Use of Published Baseline Results. Citing published baseline scores on shared benchmarks under the same LLM backbone is common practice in agent memory research, including Mem0 [1], Nemori [2], and ES-Mem [3]. Following this practice, our paper reports published baseline results obtained with GPT-4o-mini.

2. Controlled Evaluation under Native and Unified Protocols. To directly test your concern that differences in evaluation configuration may affect the reported performance advantage, we evaluated baselines under both their Native protocols and a Unified protocol. The unified protocol uses exactly the same evaluation configuration as HandleMem. Since ES-Mem has not publicly released its source code, we reproduced the two strongest available open-source memory baselines on LoCoMo, LightMem and Nemori, using their official implementations.

Table 1. Controlled comparison between Native and Unified evaluation protocols on LoCoMo (LLJ Accuracy, %).

MethodPaper ReportedNative ProtocolUnified ProtocolProtocol Shift (Unified − Native)
LightMem72.9972.5372.27−0.26
Nemori73.0067.9268.77+0.85
HandleMem81.69—81.69—

As shown in Table 1, switching from the native to the unified protocol changes the scores by only −0.26 points for LightMem and +0.85 points for Nemori. Thus, the tested protocol differences have little effect on these two baselines. Under the unified protocol, HandleMem remains clearly ahead at 81.69 LLJ accuracy, compared with 72.27 for LightMem and 68.77 for Nemori.

References:

  • [1] Chhikara et al. Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory. ECAI 2025.
  • [2] Nan et al. Nemori: Self-Organizing Agent Memory Inspired by Cognitive Science. arXiv:2508.03341, 2025.
  • [3] Zou et al. ES-Mem: Event Segmentation-Based Memory for Long-Term Dialogue Agents. arXiv:2601.07582, 2026.

W2: The overall cost conclusion heavily depends on how many questions each constructed memory is reused for. Table 4 adds the offline tokens per case to the online tokens per question and concludes that online costs are offset by offline savings. For reuse, the relevant token cost is the construction cost plus the number of queries multiplied by the online cost. HandleMem's online cost is 24.0K tokens, while the listed memory systems are only 0.7–3.7K, and the online latency is also higher. Without specifying the workload and amortization, the reported total savings cannot establish the trade-off for long-term conversational memory. This appears to be a serious flaw for a system that deliberately shifts work to query time.

Q1: What specifically does the offline "case" in Table 4 refer to? How many queries share this construction? Please report the construction cost per unit and the cost per query, and explain the break-even workload. Is the claimed cost advantage only applicable to single-query scenarios?

An offline "case" in Table 4 of the main paper refers to one complete multi-session conversation history, which is paired with one evaluated question in the standard benchmark protocol. When a constructed memory store is reused across multiple queries, HandleMem's cumulative token cost advantage extends well beyond single-query scenarios, with estimated break-even points of 35.5, 58.1, and 117.3 queries per conversation history against Mem0, A-MEM, and LangMem, respectively, while achieving 18.0 to 43.0 points higher accuracy.

1. Per-Unit Construction Cost, Per-Query Cost, and Break-Even Workload. In Table 4 of the main paper, one offline "case" is one complete user-assistant conversation history in LongMemEval-S (317.8K offline tokens / 210.4 seconds per history), rather than a single dialogue turn or segment. Because the standard LongMemEval-S benchmark provides 1 evaluated question per conversation history, Table 4 reported the unamortized single-query total (All tokens: 341.8K tokens, comprising 317.8K for construction and 24.0K for one query).

Because HandleMem's offline memory construction is completely query-independent, the handle access layer is built once per history at a cost of 317.8K offline tokens and can serve multiple future queries. Using the measured average of 24.0K online tokens per query, the estimated total token cost for NN queries is: Cost(N)=Cbuild+N⋅Cquery\text{Cost}(N) = C_{\text{build}} + N \cdot C_{\text{query}}

Here, CbuildC_{\text{build}} is the construction cost per history and CqueryC_{\text{query}} is the average online cost per query, both measured in tokens. Equating cumulative token consumption between HandleMem and each baseline gives the break-even query count N∗=Cbuildbase−CbuildoursCqueryours−CquerybaseN^* = \frac{C_{\text{build}}^{\text{base}} - C_{\text{build}}^{\text{ours}}}{C_{\text{query}}^{\text{ours}} - C_{\text{query}}^{\text{base}}}:

Table 2. Multi-query token cost amortization and break-even query count per conversation history on LongMemEval-S.

MethodOffline Tokens (K/history)Online Tokens (K/query)Total for 1 Query (K)Break-Even Queries per HistoryLongMemEval-S LLJ (%)
Mem01144.20.71144.935.553.5
A-MEM1497.13.71500.858.162.2
LangMem3003.91.13005.0117.337.2
LightMem40.71.041.7—69.8
HandleMem317.824.0341.8—80.2

As shown in Table 2, the cost advantage is not limited to single-query scenarios: HandleMem maintains lower cumulative token consumption than Mem0, A-MEM, and LangMem across multi-query workloads of up to 35.5, 58.1, and 117.3 queries per conversation history, respectively, while outperforming them by 18.0 to 43.0 points in accuracy; compared with the lower-cost LightMem, HandleMem trades moderate compute for a 10.4-point accuracy gain.

2. Answer-Quality Benefits of Query-Time Computation. Although HandleMem has higher online latency than the baselines, this additional computation directly addresses the core challenge that write-time compression cannot anticipate future information needs: adaptive retrieval and source inspection at query time recover critical cues and evidence omitted from compressed representations. On LongMemEval-S, HandleMem improves accuracy by 10.4 percentage points over LightMem and by 18.0–43.0 percentage points over Mem0, A-MEM, and LangMem. Controlled ablations further show that restricting retrieval to a single round or removing OpenSource reduces accuracy from 80.20 to 72.80 and 71.20, respectively (Figure 3(b) and Table 3(b) in the main paper). Thus, for applications that prioritize evidence completeness and answer accuracy, this investment in query-time computation provides clear value.

Thank you again for your valuable time and constructive feedback! Please let us know if you have any further questions.

Reviewer_Wek8OpenReview ↗

Thank you for your hard work

Thank you for the author's hard work. I tend to maintain the current assessment.

Reviewer_nLzEOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes HandleMem, a long-term memory framework that separates memory retrieval from answer evidence. It constructs complementary event-level and detail-level handles linked to original conversations and employs adaptive multi-round retrieval to recover relevant evidence. Experiments on LoCoMo and LongMemEval-S demonstrate strong performance over existing memory systems.

Critical strengths
  • Clear presentation. The paper is well organized and makes it easy for the reader to quickly grasp the main idea and method.
  • The paper identifies two important limitations of compressed memory, namely access gaps and evidence gaps.
  • Separating retrieval handles from original evidence provides a simple and well-motivated framework for long-term memory.
  • HandleMem achieves the best overall performance on both benchmarks, with notable improvements in temporal reasoning.
Critical weaknesses
  • The approach mainly combines existing ideas, including multi-granularity memory representations, source-linked retrieval, and iterative RAG. The methodological novelty could be better justified.
  • The largest concern for this paper from my view would be limited compression and scalability. While HandleMem employs compact retrieval handles, it still requires retaining the complete original conversation history, introducing additional storage overhead rather than reducing the overall memory footprint. Moreover, iterative source inspection increases query-time context and computational costs. The paper would benefit from a more detailed scalability analysis of memory storage, context length, and inference cost as conversation histories grow.
  • The experiments are restricted to conversational memory benchmarks. It remains unclear whether HandleMem generalizes to more realistic long-horizon agent scenarios, such as deep research or coding tasks, where memory must support complex workflows and evolving information needs. Evaluations on these tasks would strengthen the paper's practical relevance.
Questions for authors

See weaknesses above

3 comments

Dear Reviewer nLzE,

Thank you very much for your thorough review and constructive feedback! We address your questions on methodological novelty, storage and history-length scaling, and agent-workflow generalization below.

W1: The approach mainly combines existing ideas, including multi-granularity memory representations, source-linked retrieval, and iterative RAG. The methodological novelty could be better justified.

HandleMem is not an ad-hoc combination of existing techniques, but a unified design where source-linked complementary handles and adaptive source resolution work together to mitigate the Access Gap and Evidence Gap of write-time compression.

  • Source-linked complementary handles rather than independent multi-granularity summaries: Standard multi-granularity methods independently summarize text at different granularities. HandleMem first constructs an event-level handle as a broad contextual anchor, then generates detail-level handles conditioned on it to expose concise, source-supported cues insufficiently covered by the event view. Both handle types link to the same retained source segment (Section 2.2).
  • Adaptive source resolution with source-guided inspection rather than standard iterative RAG: Rather than repeating search queries over static chunks, our resolver performs observation-driven multi-round retrieval and selectively calls OpenSource during exploration to recover conditions, relations, or temporal cues omitted from the handles. The inspected source text becomes part of the resolver's observations and can guide subsequent SearchEvent and SearchDetail calls and source selection (Section 2.3).

Controlled ablations further support this combined design: single-round retrieval over the handles achieves 72.80, and multi-round raw-text retrieval under the same action budget achieves 72.20, both below the full HandleMem system's 80.20 on LongMemEval-S (Figure 3(b)).

W2: The largest concern for this paper from my view would be limited compression and scalability. While HandleMem employs compact retrieval handles, it still requires retaining the complete original conversation history, introducing additional storage overhead rather than reducing the overall memory footprint. Moreover, iterative source inspection increases query-time context and computational costs. The paper would benefit from a more detailed scalability analysis of memory storage, context length, and inference cost as conversation histories grow.

HandleMem separates inexpensive offline storage of retained source segments from the active context window: its source-linked complementary handles add only 0.24× text overhead (1.24× total with retained source segments, still smaller than A-Mem's 1.34×), while keeping cumulative online token consumption nearly constant and accuracy around 80% across the tested history lengths.

1. Persistent Memory Storage Overhead. We measured persistent text storage on the LongMemEval-S conversation histories, using the retained raw interaction history size as the 1.00× baseline (Table 1).

Table 1. Persistent memory text footprint relative to retained raw interaction history (1.00×).

MethodStored Memory Text / Raw TextIncludes Original Source Segments?LongMemEval-S LLJ (%)
Raw interaction history (Full Context)1.00×Yes55.0
A-Mem1.34×No62.2
Mem00.09×No53.5
LightMem0.06×No69.8
LangMem0.23×No37.2
HandleMem (w/o OpenSource + Handles-only answer)0.24×No69.4
HandleMem1.24×Yes80.2

Retaining original sources does introduce additional storage overhead, but it also enables HandleMem to recover evidence omitted by write-time compression. HandleMem does not aim to minimize total persistent storage; instead, it retains sources to keep evidence recoverable and uses compact handles for selective access to control query-time context size. To quantify this trade-off, we added a variant that does not retain original sources (w/o OpenSource + Handles-only answer): its stored text is 0.24× the size of the raw history, with an accuracy of 69.4. The full HandleMem system uses 1.24× total text storage and achieves 80.2 accuracy, a gain of 10.8 percentage points. This comparison shows a clear answer-quality benefit from retaining sources.

2. History-Length Scaling of Online Token Consumption and Accuracy. We evaluated LongMemEval-S at three nested history lengths by retaining all gold evidence sessions plus 25%, 50%, or 100% of distractor sessions under identical GPT-4o-mini models, action budgets, and prompts (normalizing cumulative online token consumption to HandleMem at the 25% history length = 1.00×).

Table 2. Scaling behavior on LongMemEval-S as history length increases.

MethodDistractor Sessions RetainedRelative Online Token CostAccuracy (%)
HandleMem25%1.00×79.8
HandleMem50%0.99×79.2
HandleMem100%1.02×80.2
Multi-round raw-text RAG25%2.41×78.0
Multi-round raw-text RAG50%2.41×74.8
Multi-round raw-text RAG100%2.39×72.2

As the proportion of retained distractor sessions increases:

  • Stable online token cost: As the proportion of retained distractor sessions increases, HandleMem's online token consumption remains nearly constant, varying by approximately −1% to +2% relative to the shortest-history setting, and stays consistently below that of multi-round raw-text RAG (Table 2).
  • Robustness against distractor scaling: HandleMem's accuracy stays stable (79.8 to 80.2), whereas Multi-round raw-text RAG drops by 5.8 points (78.0 to 72.2) as distractors accumulate.

W3: The experiments are restricted to conversational memory benchmarks. It remains unclear whether HandleMem generalizes to more realistic long-horizon agent scenarios, such as deep research or coding tasks, where memory must support complex workflows and evolving information needs. Evaluations on these tasks would strengthen the paper's practical relevance.

HandleMem generalizes effectively beyond conversational benchmarks to long-horizon agent workflows, achieving 31.61 accuracy on AMA-Bench and outperforming both RAG (28.97) and LightMem (12.70). To directly evaluate generalization beyond conversational histories to complex agent workflows, we conducted an additional evaluation on AMA-Bench across 6 agent domains, which tests memory systems over recorded long-horizon agent trajectories containing interleaved tool invocations, environment states, and intermediate action-observation steps. We compare HandleMem, RAG, and LightMem on the same trajectories and questions using the exact same GPT-4o-mini answer-generation configuration and the official AMA-Bench evaluator, while preserving each method's own memory construction and retrieval pipeline. All memories are constructed strictly from agent trajectories without access to downstream questions.

Table 3. Generalization to long-horizon agent trajectories on AMA-Bench (Accuracy, %).

MethodEmbodied AIGameOpenWorld QASoftwareText2SQLWebOverall
LightMem5.2832.2214.178.565.5616.1312.70
RAG8.8940.8332.2240.0520.4234.9528.97
HandleMem13.6142.5036.1140.0525.8233.8731.61

Across long-horizon agent trajectories, HandleMem (31.61) outperforms RAG (28.97, +2.64) and LightMem (12.70, +18.91), outperforming LightMem across all 6 domains and matching or exceeding RAG on 5 out of 6 domains (with notable gains on Text2SQL +5.40, Embodied AI +4.72, and OpenWorld QA +3.89). In particular, relying solely on write-time compressed memories (LightMem) suffers severe degradation on dense tool-use and environment-state trajectories, whereas HandleMem's source-linked complementary handles and adaptive source resolution (with OpenSource) recover the fine-grained source evidence needed for downstream agent QA.

Thank you again for your valuable time and constructive feedback! Please let us know if you have any further questions.

Thank you for the detailed rebuttal. While these empirical results help clarify the system's efficiency and behavior across different tasks, my core concerns regarding the limited methodology —as it remains largely an incremental integration of multi-granularity indexing and iterative source-linked retrieval—and the modest performance gains on complex agent workflows remain. Therefore, I will maintain my original rating.

And here is one following question: While HandleMem outperforms standard RAG by over 10 points on conversational benchmarks, the overall gain on AMA-Bench is much smaller (+2.64 points). Designing domain-specific structured handles rather than natural-language summaries could better capture dense tool-use and environment-state trajectories. It would be more valuable to consider some baselines in this benchmark.

Reviewer_PYvsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

this work propose a practical memory system deign to handle the difficulty that compression must decide what to retain before the future use of the memory is known. It constructs source-linked complementary event- and detail-level handles that expose different retrieval cues for the same source segment. the proposal is supported by empirical results and comprehensive ablation study.

Critical strengths

Simple and practical design. The proposed framework combines complementary event-level and detail-level retrieval handles with source-linked evidence recovery. The overall architecture is straightforward and potentially applicable to existing long-term memory systems without requiring complex memory structures.

Critical weaknesses

I am concerned that the central challenge and proposed solution substantially overlap with a broader body of existing work on hierarchical, graph-based, and agentic memory retrieval. For example, REMem (ICLR 2026) organizes episodic memory into a graph connecting event-level gists and fine-grained facts, and employs iterative agentic retrieval. Mnemis (ACL 2026) combines hierarchical memory representations with complementary retrieval routes. H-MEM (EACL 2026) organizes memories at different abstraction levels, with higher-level representations pointing to more detailed memories. APEX-MEM (ACL 2026) further combines structured memory organization, history preservation, and multi-tool retrieval. More fundamentally, using compact representations as retrieval cues to locate detailed source evidence is also well established in hierarchical retrieval and GraphRAG systems. Thus, the paper's central distinction between an access layer and an evidence layer does not appear, by itself, to constitute a fundamentally new architectural insight.

Questions for authors

compared to existing works listed above, please clarify the difference, and relative contribution.

2 comments

Dear Reviewer PYvs,

Thank you very much for your thorough review and constructive feedback! We clarify how HandleMem differs from the cited hierarchical, graph-based, and agentic memory retrieval works below.

W1: I am concerned that the central challenge and proposed solution substantially overlap with a broader body of existing work on hierarchical, graph-based, and agentic memory retrieval. For example, REMem (ICLR 2026) organizes episodic memory into a graph connecting event-level gists and fine-grained facts, and employs iterative agentic retrieval. Mnemis (ACL 2026) combines hierarchical memory representations with complementary retrieval routes. H-MEM (EACL 2026) organizes memories at different abstraction levels, with higher-level representations pointing to more detailed memories. APEX-MEM (ACL 2026) further combines structured memory organization, history preservation, and multi-tool retrieval. More fundamentally, using compact representations as retrieval cues to locate detailed source evidence is also well established in hierarchical retrieval and GraphRAG systems. Thus, the paper's central distinction between an access layer and an evidence layer does not appear, by itself, to constitute a fundamentally new architectural insight.

Q1: compared to existing works listed above, please clarify the difference, and relative contribution.

HandleMem differs from the cited works in its problem formulation (decoupling memory access from answer evidence), memory representation (non-hierarchical, source-linked complementary handles), and agentic retrieval mechanism (adaptive source resolution with mid-search source inspection).

1. Problem Formulation: Decoupling Memory Access from Answer Evidence. Write-time compression must decide what to retain before future queries are known. Using the same compressed representation as both the retrieval target and the answering evidence creates two distinct failure modes: omitting retrieval cues causes an Access Gap, while omitting details needed to answer causes an Evidence Gap (Section 1, Figure 1). Whereas the cited systems organize compressed memories into richer structures that still serve as the primary content for downstream reasoning, HandleMem keeps retained source segments as the evidence layer and uses compact handles strictly for access ("handles provide access; sources provide evidence", Section 2.1).

2. Memory Representation: Source-Linked Complementary Handles. Rather than building relational graphs or multi-level abstraction trees, HandleMem constructs flat, source-linked complementary handles for each source segment: it first generates an event-level handle as a broad contextual anchor, and then conditions detail-level handles on it to capture fine-grained cues omitted from the event view (Section 2.2):

  • vs. REMem and Mnemis: REMem links gists and facts into a hybrid graph for relational recollection, while Mnemis combines local similarity search with global community-graph reasoning. Both reason primarily over retrieved graph nodes. HandleMem requires no graph construction or traversal; its event and detail handles link directly to the same retained source segment, which is recovered and ordered chronologically for answering.
  • vs. H-MEM (Sun et al., EACL 2026) and H-Mem (Yu et al., arXiv:2605.15701) (we clarify both as the review text and link refer to these two works): H-MEM routes top-down across multi-level semantic abstractions, and H-Mem tracks memory evolution via dual temporal-semantic trees and a knowledge graph. HandleMem's event and detail handles are non-hierarchical, parallel access views pointing directly to the source segment, allowing the resolver to flexibly switch between SearchEvent and SearchDetail without top-down routing bottlenecks.
  • vs. APEX-MEM: APEX-MEM maintains an append-only temporal property graph to preserve state updates and synthesizes context summaries for answering. HandleMem instead preserves raw source segments as the evidence layer and links compact handles directly to them.

3. Agentic Memory Retrieval: Source-Guided Resolution over Complementary Handles. Iterative retrieval and tool use are shared with prior agentic memory systems; HandleMem’s distinction lies in how retrieval is organized around complementary handles and retained sources. Whereas REMem organizes retrieval around a gist–fact graph and APEX-MEM around a temporal property graph, HandleMem uses event- and detail-level handles as direct access routes to retained source segments. Its resolver combines SearchEvent, SearchDetail, and OpenSource within the same decision loop: when a retrieved handle lacks a condition, relation, or temporal cue needed to continue, the resolver can inspect the linked source and use the recovered information to decide what to search for next and which sources to select. Removing OpenSource lowers accuracy from 80.20 to 71.20, even though selected sources are still recovered before final answering (Table 3(b)), demonstrating the value of source inspection during retrieval in our design.

4. Architectural Positioning: Beyond Source-Linked Indexing. HandleMem goes beyond source-linked indexing by pairing flat complementary handles with mid-search source inspection. First, rather than building multi-level abstraction trees or cross-segment entity/community graphs, HandleMem constructs flat, complementary event- and detail-level handles for each segment, conditioning detail handles on the event view to expose omitted fine-grained cues (Section 2.2). Second, compared with using compact indexes to retrieve sources for final answering, HandleMem additionally uses source inspection to guide retrieval itself. Through OpenSource, the resolver can inspect linked sources mid-trajectory and use recovered conditions, relations, or temporal cues to guide subsequent search and source selection. In Table 3(b), w/o OpenSource still resolves all selected memories to their original sources before final answering, yet accuracy drops by 9.00 points (80.20 to 71.20). This shows that post-retrieval source lookup alone does not recover the benefit of mid-search source inspection in HandleMem.

Thank you again for your valuable time and constructive feedback! Please let us know if you have any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.