- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces the HandleMem method, which stores raw dialogue segments as evidence for generating answers and indexes them using complementary event-level and detail-level handles. Evaluations on LoCoMo and LongMemEval-S demonstrate that this method achieves improvements over raw-text RAG and previously reported results for memory systems.
Critical strengths
S1. The distinction between retrieved representations and source evidence is clear and of high practical value. Answer control using only handles fixes the selected record, causing accuracy to drop from 80.20% to 73.40%, which directly proves that the recovered source text provides information beyond the selected compressed handles.
S2. Very clearly written.
Critical weaknesses
W1. Ideas and intuition are quite incremental. Furthermore, the scores for Full Context and memory systems are imported from previous reports, while HandleMem and RAG are evaluated here. Sharing the GPT-4o-mini model name does not guarantee that the model, answer prompts, judge prompts, retrieval budget, or preprocessing are exactly the same. Although the internal multi-turn RAG control is helpful, it does not address how large the reported advantages over ES-Mem and other systems are under a unified evaluation process. This limits the claim of superior performance.
W2. The overall cost conclusion heavily depends on how many questions each constructed memory is reused for. Table 4 adds the offline tokens per case to the online tokens per question and concludes that online costs are offset by offline savings. For reuse, the relevant token cost is the construction cost plus the number of queries multiplied by the online cost. HandleMem's online cost is 24.0K tokens, while the listed memory systems are only 0.7–3.7K, and the online latency is also higher. Without specifying the workload and amortization, the reported total savings cannot establish the trade-off for long-term conversational memory. This appears to be a serious flaw for a system that deliberately shifts work to query time.
Questions for authors
Q1. What specifically does the offline "case" in Table 4 refer to? How many queries share this construction? Please report the construction cost per unit and the cost per query, and explain the break-even workload. Is the claimed cost advantage only applicable to single-query scenarios?
3 comments
W2: The overall cost conclusion heavily depends on how many questions each constructed memory is reused for. Table 4 adds the offline tokens per case to the online tokens per question and concludes that online costs are offset by offline savings. For reuse, the relevant token cost is the construction cost plus the number of queries multiplied by the online cost. HandleMem's online cost is 24.0K tokens, while the listed memory systems are only 0.7–3.7K, and the online latency is also higher. Without specifying the workload and amortization, the reported total savings cannot establish the trade-off for long-term conversational memory. This appears to be a serious flaw for a system that deliberately shifts work to query time.
Q1: What specifically does the offline "case" in Table 4 refer to? How many queries share this construction? Please report the construction cost per unit and the cost per query, and explain the break-even workload. Is the claimed cost advantage only applicable to single-query scenarios?
An offline "case" in Table 4 of the main paper refers to one complete multi-session conversation history, which is paired with one evaluated question in the standard benchmark protocol. When a constructed memory store is reused across multiple queries, HandleMem's cumulative token cost advantage extends well beyond single-query scenarios, with estimated break-even points of 35.5, 58.1, and 117.3 queries per conversation history against Mem0, A-MEM, and LangMem, respectively, while achieving 18.0 to 43.0 points higher accuracy.
1. Per-Unit Construction Cost, Per-Query Cost, and Break-Even Workload.
In Table 4 of the main paper, one offline "case" is one complete user-assistant conversation history in LongMemEval-S (317.8K offline tokens / 210.4 seconds per history), rather than a single dialogue turn or segment. Because the standard LongMemEval-S benchmark provides 1 evaluated question per conversation history, Table 4 reported the unamortized single-query total (All tokens: 341.8K tokens, comprising 317.8K for construction and 24.0K for one query).
Because HandleMem's offline memory construction is completely query-independent, the handle access layer is built once per history at a cost of 317.8K offline tokens and can serve multiple future queries. Using the measured average of 24.0K online tokens per query, the estimated total token cost for queries is:
Here, is the construction cost per history and is the average online cost per query, both measured in tokens. Equating cumulative token consumption between HandleMem and each baseline gives the break-even query count :
Table 2. Multi-query token cost amortization and break-even query count per conversation history on LongMemEval-S.
| Method | Offline Tokens (K/history) | Online Tokens (K/query) | Total for 1 Query (K) | Break-Even Queries per History | LongMemEval-S LLJ (%) |
|---|---|---|---|---|---|
| Mem0 | 1144.2 | 0.7 | 1144.9 | 35.5 | 53.5 |
| A-MEM | 1497.1 | 3.7 | 1500.8 | 58.1 | 62.2 |
| LangMem | 3003.9 | 1.1 | 3005.0 | 117.3 | 37.2 |
| LightMem | 40.7 | 1.0 | 41.7 | — | 69.8 |
| HandleMem | 317.8 | 24.0 | 341.8 | — | 80.2 |
As shown in Table 2, the cost advantage is not limited to single-query scenarios: HandleMem maintains lower cumulative token consumption than Mem0, A-MEM, and LangMem across multi-query workloads of up to 35.5, 58.1, and 117.3 queries per conversation history, respectively, while outperforming them by 18.0 to 43.0 points in accuracy; compared with the lower-cost LightMem, HandleMem trades moderate compute for a 10.4-point accuracy gain.
2. Answer-Quality Benefits of Query-Time Computation.
Although HandleMem has higher online latency than the baselines, this additional computation directly addresses the core challenge that write-time compression cannot anticipate future information needs: adaptive retrieval and source inspection at query time recover critical cues and evidence omitted from compressed representations. On LongMemEval-S, HandleMem improves accuracy by 10.4 percentage points over LightMem and by 18.0–43.0 percentage points over Mem0, A-MEM, and LangMem. Controlled ablations further show that restricting retrieval to a single round or removing OpenSource reduces accuracy from 80.20 to 72.80 and 71.20, respectively (Figure 3(b) and Table 3(b) in the main paper). Thus, for applications that prioritize evidence completeness and answer accuracy, this investment in query-time computation provides clear value.
Thank you again for your valuable time and constructive feedback! Please let us know if you have any further questions.
Thank you for your hard work
Thank you for the author's hard work. I tend to maintain the current assessment.
Dear Reviewer Wek8,
Thank you very much for your thorough review and constructive feedback! We address your concerns regarding cross-paper evaluation comparability and multi-query cost amortization below.
Our contribution combines source-linked complementary handles with adaptive source resolution to address the Access Gap and Evidence Gap caused by write-time compression. Detail-level handles are generated conditioned on the event-level handle to expose complementary retrieval cues, with both views linked to the same retained source (Section 2.2). During retrieval,
OpenSourcelets the resolver recover omitted source details and use them to guide subsequent search and selection (Section 2.3). Together, these mechanisms connect complementary memory access with recoverable source evidence.To address your main concern about cross-paper comparability, we additionally evaluated LightMem and Nemori under both their native protocols and the same evaluation configuration as HandleMem. These controlled experiments show that HandleMem maintains its performance lead over both baselines under the unified protocol, as detailed below.
1. Use of Published Baseline Results. Citing published baseline scores on shared benchmarks under the same LLM backbone is common practice in agent memory research, including Mem0 [1], Nemori [2], and ES-Mem [3]. Following this practice, our paper reports published baseline results obtained with GPT-4o-mini.
2. Controlled Evaluation under Native and Unified Protocols. To directly test your concern that differences in evaluation configuration may affect the reported performance advantage, we evaluated baselines under both their Native protocols and a Unified protocol. The unified protocol uses exactly the same evaluation configuration as HandleMem. Since ES-Mem has not publicly released its source code, we reproduced the two strongest available open-source memory baselines on LoCoMo, LightMem and Nemori, using their official implementations.
Table 1. Controlled comparison between Native and Unified evaluation protocols on LoCoMo (LLJ Accuracy, %).
As shown in Table 1, switching from the native to the unified protocol changes the scores by only −0.26 points for LightMem and +0.85 points for Nemori. Thus, the tested protocol differences have little effect on these two baselines. Under the unified protocol, HandleMem remains clearly ahead at 81.69 LLJ accuracy, compared with 72.27 for LightMem and 68.77 for Nemori.
References: