- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TokenGate, a modular framework combining learned write admission with token-budgeted retrieval for long-term LLM agent memory. A lightweight Qwen2.5-0.5B classifier, trained using twelve structured views, decides which memories to retain. At retrieval time, relevance-based ranking and greedy packing enforce a 512-token evidence budget.
Experiments integrate TokenGate with Mem0, A-MEM, and Letta on LoCoMo, with additional admission diagnostics on LongMemEval and RHELM. The method reduces stored memory-body tokens by 41.7%, while the full system exhibits mixed QA and computational-cost outcomes. A separate metadata-aware reranking study improves ordinary-question F1 by 5.21 and 2.66 percentage points on Mem0 and Letta, respectively, under fixed admitted stores.
Critical strengths
S1. Practical separation of memory admission and retrieval budgeting. The framework explicitly controls persistent storage and query-time context consumption, addressing two distinct sources of memory inefficiency. Its integration with three existing memory backends demonstrates practical modularity.
S2. Careful experimental decomposition and transparent reporting. The paper includes matched multi-view training controls, module ablations, fixed-store reranking comparisons, and explicit token accounting. Reporting negative results and separating storage savings from downstream QA improvements make the empirical analysis more credible.
Critical weaknesses
W1. The complete system fails to establish a convincing quality-efficiency improvement. In Table 1, FULL reduces storage but decreases F1 by 1.93 and 4.20 points on A-MEM and Letta, respectively. Its total counted LLM tokens also increase on Mem0 and Letta due to maintenance. Thus, storage compression does not translate consistently into better end-to-end utility.
W2. The retrieval module is not supported by the full-system ablations. In Table 4, removing lexical fusion (NO R) improves F1 across all three backends, by 1.90--4.73 points. Although the separate RST study shows encouraging gains, it uses a different reranker and excludes maintenance. Therefore, the strongest retrieval results do not directly validate the proposed FULL configuration.
W3. Evaluation diversity and statistical reliability are insufficient. The main corrected-input QA results rely on only two LoCoMo test conversations, with limited independent backend repetitions. The LongMemEval results evaluate teacher-label agreement rather than downstream QA, while the RHELM experiments use a different retention protocol. Generalization of the complete system remains unestablished.
W4. The learned admission advantage is modest and potentially fragile. On reviewed LoCoMo candidates, A1 does not establish a statistically reliable AUROC improvement over TF--IDF and has worse Brier error. On RHELM, A1 also underperforms the matched repetition control in overall AUROC. Moreover, the classifier processes candidate text without recent dialogue, potentially discarding context-dependent information.
W5. The cost model omits important components of deployment efficiency. Reported storage savings exclude model weights and indexes, while total LLM-token accounting excludes admission, embedding, and reranker computation. The separate Qwen3 reranker introduces additional inference overhead, making the practical cost-quality trade-off difficult to assess.
Questions for authors
Q1. Given that FULL underperforms BASE in two backends and increases total counted LLM tokens in Mem0 and Letta, under what workload conditions does TokenGate achieve a genuine end-to-end quality-cost advantage?
Q2. Why does removing lexical fusion consistently improve ordinary-question F1 in Table 4? Can the authors identify whether the degradation originates from score calibration, the length penalty, or greedy packing?
Q3. Can the metadata-aware RST reranker be integrated into the complete FULL pipeline, including maintenance, and evaluated across all three backends? This would clarify whether its isolated gains survive full-system integration.
Q4. How sensitive is admission to missing conversational context, especially for pronouns, evolving preferences, corrections, and temporally dependent facts? What fraction of downstream QA failures is attributable to irreversible false discards?
Q5. Can the authors report a unified cost-quality Pareto comparison including admission inference, embeddings, reranking, maintenance, memory indexing, and storage overhead?
Q6. How robust are the reported improvements across independent conversations, alternative answer-generation models, and retrieval budgets? Can the authors provide conversation-level uncertainty estimates on a larger test set?
2 comments
Thank you to the authors for the detailed response and additional analysis. I appreciate the effort to address my concerns and clarify the experimental results. However, I remain concerned about the limited evaluation across independent conversations, which makes it difficult to assess the robustness and generalizability of the findings. Given this remaining concern, I will maintain my original rating.
Thank you for tracing the individual modules through to the deployment decision. Your six questions distinguish several effects that looked similar in our original presentation.
Q1 Workloads that benefit
A costly backend write gives admission a plausible route to savings. In Table 1, turning maintenance off on Mem0 gives both higher ordinary F1 than BASE (0.1831 vs 0.1689) and slightly fewer counted LLM tokens (217.7k vs 219.0k). On A-MEM the counted-token saving is much larger, but NO_M loses 1.53 F1 points. FULL spends 46,091 maintenance tokens for two updates and no deletions. These data support a choice of modules by workload. The all-in break-even point remains unmeasured.
Q2 Why native ranking beats lexical fusion
FULL and NO_R use the same post-maintenance stores, candidate pools and packer. Only the ordering policy changes. Switching back to native order improves annotated-evidence coverage by 9.27, 14.43 and 19.36 points in Mem0, A-MEM and Letta, respectively (Tables A5–A7). This locates the loss in which records reach the 512-token context. Removing just the lexical utility term in a separate fixed-store test actually lowers F1 in both backends (Table A9). We therefore cannot assign the loss to that term alone or distinguish score scaling, the length penalty and their interaction with greedy packing.
Q3 RST with A-MEM and with maintenance
The September corrected-input archive contains a third-backend comparison with maintenance off. With the A-MEM store, candidate pool and budget fixed, native ranking gives F1 of 0.2041 and RST gives 0.2612. The gains on conv-26 and conv-30 are 5.65 and 5.78 points. This extends the Table 3 comparison to A-MEM under the same maintenance setting. RST after maintenance across all three backends remains untested. A rewritten record may combine sources, so a fair test would also specify how the reranker receives speaker and date information for those records.
Q4 Context and irreversible discards
A reply such as “Actually, next Friday” illustrates the risk. The teacher saw up to five preceding turns when judging durable value, whereas A1 scores the utterance alone. On original reviewed inputs, the filter and A1 discarded 26 of 209 KEEP-labelled candidates (Table A4). We have no breakdown by pronouns, corrections or temporal references. To get closer to your attribution question, we checked the saved corrected-input outputs for the two single-run backends, using gated storage and native retrieval at 512 tokens. Among ordinary questions with traceable evidence and F1 of zero, the store lacked every annotated source for 20 of 121 questions on Mem0 and 21 of 103 on Letta. These are pooled question counts across the two test conversations. They identify failures worth replaying. Restoring the missing records and checking whether the answer changes would be necessary to attribute a failure to the write decision.
Q5 What a full cost comparison would require
Table 1 counts backend LLM processing, maintenance and answering. Appendix C measures local admission at 162.9 ms per candidate on average. Appendix D records 110.54 minutes for 149,332 reranker pair requests across four development/test views, including score writing, and 1.21 GB of model files. These are different workloads and units. An end-to-end Pareto comparison would also need embedding and indexing time, physical storage including indexes, and latency or cost at a common write-to-query ratio.
Q6 Robustness across conversations models and budgets
The archived corrected-input budget sweep covers BASE and FULL. On Mem0, FULL vs BASE ordinary F1 is 0.1395 vs 0.1385 at 256 tokens, 0.1781 vs 0.1689 at 512, and 0.1818 vs 0.2043 at 1,024. Even the direction depends on the budget. Our RST comparison uses 512 tokens and Qwen2.5-7B throughout. Its ordinary F1 gain has the same sign on both conversations in all three backends. A-MEM’s three seeds vary store construction. With only two independent test conversations, the paired differences are more informative than a population interval. Other answerers and retrieval budgets for RST remain open.