acceptodds
Under review as a conference paper at ICLR 2027

TokenGate: Write Admission and Budgeted Retrieval for Long-Term Agent Memory

Abstract

Long-term memory enables language model agents to sustain continuity across interactions, yet accumulating histories impose escalating storage demands and recurrent token expenditure during inference. The central challenge is to regulate persistent memory growth while allocating a finite context budget to useful evidence. Conventional top- retrieval imposes a cardinality constraint rather than an explicit token bound, leaving memory accumulation at write time unaddressed. We introduce TokenGate, a modular memory controller that couples learned admission control with query-conditioned evidence selection under a hard token budget. The admission model is trained on twelve structured views of each source candidate, while retrieval ranking combines question–memory relevance, semantic similarity, and a length penalty. On two LoCoMo test conversations with corrected inputs, filtering and learned admission reduce stored memory-body tokens by 41.7% relative to retaining all candidates, before offline maintenance. Compared with the store-all baseline (BASE), the full configuration with offline maintenance (FULL) has lower ordinary-question F1 in A-MEM and Letta and higher total counted LLM input and output tokens in Mem0 and Letta. In a separate comparison limited to Mem0 and Letta, with admitted stores and candidate pools fixed, reranking with speaker and observation-date metadata (RST) improves ordinary-question F1 over native ranking by 5.21 and 2.66 percentage points, respectively, within a 512-token memory budget. On 5,000 teacher-labelled admission examples derived from LongMemEval, twelve-view training achieves an AUROC of .823, compared with .811 for a matched native-example repetition control. The results support separate control of stored text and evidence selection within a fixed retrieval budget, with backend-dependent effects on answer quality and counted LLM-token cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, agent memory, long-term memory, memory management, memory admission, budgeted retrieval, token efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_69dTOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TokenGate, a modular framework combining learned write admission with token-budgeted retrieval for long-term LLM agent memory. A lightweight Qwen2.5-0.5B classifier, trained using twelve structured views, decides which memories to retain. At retrieval time, relevance-based ranking and greedy packing enforce a 512-token evidence budget.

Experiments integrate TokenGate with Mem0, A-MEM, and Letta on LoCoMo, with additional admission diagnostics on LongMemEval and RHELM. The method reduces stored memory-body tokens by 41.7%, while the full system exhibits mixed QA and computational-cost outcomes. A separate metadata-aware reranking study improves ordinary-question F1 by 5.21 and 2.66 percentage points on Mem0 and Letta, respectively, under fixed admitted stores.

Critical strengths

S1. Practical separation of memory admission and retrieval budgeting. The framework explicitly controls persistent storage and query-time context consumption, addressing two distinct sources of memory inefficiency. Its integration with three existing memory backends demonstrates practical modularity.

S2. Careful experimental decomposition and transparent reporting. The paper includes matched multi-view training controls, module ablations, fixed-store reranking comparisons, and explicit token accounting. Reporting negative results and separating storage savings from downstream QA improvements make the empirical analysis more credible.

Critical weaknesses

W1. The complete system fails to establish a convincing quality-efficiency improvement. In Table 1, FULL reduces storage but decreases F1 by 1.93 and 4.20 points on A-MEM and Letta, respectively. Its total counted LLM tokens also increase on Mem0 and Letta due to maintenance. Thus, storage compression does not translate consistently into better end-to-end utility.

W2. The retrieval module is not supported by the full-system ablations. In Table 4, removing lexical fusion (NO R) improves F1 across all three backends, by 1.90--4.73 points. Although the separate RST study shows encouraging gains, it uses a different reranker and excludes maintenance. Therefore, the strongest retrieval results do not directly validate the proposed FULL configuration.

W3. Evaluation diversity and statistical reliability are insufficient. The main corrected-input QA results rely on only two LoCoMo test conversations, with limited independent backend repetitions. The LongMemEval results evaluate teacher-label agreement rather than downstream QA, while the RHELM experiments use a different retention protocol. Generalization of the complete system remains unestablished.

W4. The learned admission advantage is modest and potentially fragile. On reviewed LoCoMo candidates, A1 does not establish a statistically reliable AUROC improvement over TF--IDF and has worse Brier error. On RHELM, A1 also underperforms the matched repetition control in overall AUROC. Moreover, the classifier processes candidate text without recent dialogue, potentially discarding context-dependent information.

W5. The cost model omits important components of deployment efficiency. Reported storage savings exclude model weights and indexes, while total LLM-token accounting excludes admission, embedding, and reranker computation. The separate Qwen3 reranker introduces additional inference overhead, making the practical cost-quality trade-off difficult to assess.

Questions for authors

Q1. Given that FULL underperforms BASE in two backends and increases total counted LLM tokens in Mem0 and Letta, under what workload conditions does TokenGate achieve a genuine end-to-end quality-cost advantage?

Q2. Why does removing lexical fusion consistently improve ordinary-question F1 in Table 4? Can the authors identify whether the degradation originates from score calibration, the length penalty, or greedy packing?

Q3. Can the metadata-aware RST reranker be integrated into the complete FULL pipeline, including maintenance, and evaluated across all three backends? This would clarify whether its isolated gains survive full-system integration.

Q4. How sensitive is admission to missing conversational context, especially for pronouns, evolving preferences, corrections, and temporally dependent facts? What fraction of downstream QA failures is attributable to irreversible false discards?

Q5. Can the authors report a unified cost-quality Pareto comparison including admission inference, embeddings, reranking, maintenance, memory indexing, and storage overhead?

Q6. How robust are the reported improvements across independent conversations, alternative answer-generation models, and retrieval budgets? Can the authors provide conversation-level uncertainty estimates on a larger test set?

2 comments

Thank you for tracing the individual modules through to the deployment decision. Your six questions distinguish several effects that looked similar in our original presentation.

Q1 Workloads that benefit

A costly backend write gives admission a plausible route to savings. In Table 1, turning maintenance off on Mem0 gives both higher ordinary F1 than BASE (0.1831 vs 0.1689) and slightly fewer counted LLM tokens (217.7k vs 219.0k). On A-MEM the counted-token saving is much larger, but NO_M loses 1.53 F1 points. FULL spends 46,091 maintenance tokens for two updates and no deletions. These data support a choice of modules by workload. The all-in break-even point remains unmeasured.

Q2 Why native ranking beats lexical fusion

FULL and NO_R use the same post-maintenance stores, candidate pools and packer. Only the ordering policy changes. Switching back to native order improves annotated-evidence coverage by 9.27, 14.43 and 19.36 points in Mem0, A-MEM and Letta, respectively (Tables A5–A7). This locates the loss in which records reach the 512-token context. Removing just the lexical utility term in a separate fixed-store test actually lowers F1 in both backends (Table A9). We therefore cannot assign the loss to that term alone or distinguish score scaling, the length penalty and their interaction with greedy packing.

Q3 RST with A-MEM and with maintenance

The September corrected-input archive contains a third-backend comparison with maintenance off. With the A-MEM store, candidate pool and budget fixed, native ranking gives F1 of 0.2041 and RST gives 0.2612. The gains on conv-26 and conv-30 are 5.65 and 5.78 points. This extends the Table 3 comparison to A-MEM under the same maintenance setting. RST after maintenance across all three backends remains untested. A rewritten record may combine sources, so a fair test would also specify how the reranker receives speaker and date information for those records.

Q4 Context and irreversible discards

A reply such as “Actually, next Friday” illustrates the risk. The teacher saw up to five preceding turns when judging durable value, whereas A1 scores the utterance alone. On original reviewed inputs, the filter and A1 discarded 26 of 209 KEEP-labelled candidates (Table A4). We have no breakdown by pronouns, corrections or temporal references. To get closer to your attribution question, we checked the saved corrected-input outputs for the two single-run backends, using gated storage and native retrieval at 512 tokens. Among ordinary questions with traceable evidence and F1 of zero, the store lacked every annotated source for 20 of 121 questions on Mem0 and 21 of 103 on Letta. These are pooled question counts across the two test conversations. They identify failures worth replaying. Restoring the missing records and checking whether the answer changes would be necessary to attribute a failure to the write decision.

Q5 What a full cost comparison would require

Table 1 counts backend LLM processing, maintenance and answering. Appendix C measures local admission at 162.9 ms per candidate on average. Appendix D records 110.54 minutes for 149,332 reranker pair requests across four development/test views, including score writing, and 1.21 GB of model files. These are different workloads and units. An end-to-end Pareto comparison would also need embedding and indexing time, physical storage including indexes, and latency or cost at a common write-to-query ratio.

Q6 Robustness across conversations models and budgets

The archived corrected-input budget sweep covers BASE and FULL. On Mem0, FULL vs BASE ordinary F1 is 0.1395 vs 0.1385 at 256 tokens, 0.1781 vs 0.1689 at 512, and 0.1818 vs 0.2043 at 1,024. Even the direction depends on the budget. Our RST comparison uses 512 tokens and Qwen2.5-7B throughout. Its ordinary F1 gain has the same sign on both conversations in all three backends. A-MEM’s three seeds vary store construction. With only two independent test conversations, the paired differences are more informative than a population interval. Other answerers and retrieval budgets for RST remain open.

Thank you to the authors for the detailed response and additional analysis. I appreciate the effort to address my concerns and clarify the experimental results. However, I remain concerned about the limited evaluation across independent conversations, which makes it difficult to assess the robustness and generalizability of the findings. Given this remaining concern, I will maintain my original rating.

Reviewer_gWprOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents TokenGate, which decides what information to store in an agent's long-term memory and selects relevant memories under a fixed token budget. The authors test it with Mem0, A-MEM, and Letta on LoCoMo, and also evaluate the admission classifier on LongMemEval. The method reduces the number of stored memory tokens, but the effects on answer quality and overall LLM-token cost are mixed.

Critical strengths

The paper tackles two practical parts of memory management: deciding what to keep and choosing what to retrieve when the context budget is limited. The experiments also report costs and include ablations across several memory backends.

Critical weaknesses

The results do not show a clear overall benefit. TokenGate reduces stored memory tokens by about 42%, but the full system performs worse than the store-all baseline on ordinary-question F1 for A-MEM and Letta. It also uses more total LLM tokens for Mem0 and Letta once maintenance is included. On the reviewed LoCoMo candidates, the admission classifier does not show a statistically clear AUROC gain over TF--IDF, and its Brier error is higher.

The evaluation is also quite limited. The main QA results come from only two LoCoMo test conversations, and the separate reranking study reuses those conversations with a single logical repeat. Its gains on ordinary questions are accompanied by worse scores on the adversarial set. The LongMemEval experiment measures agreement with teacher-generated labels, rather than whether the method improves downstream question answering. Given these limitations, the paper does not yet provide convincing evidence that TokenGate offers a reliable improvement over simpler approaches.

Questions for authors

Do the results hold on a larger set of independent conversations? After including the costs of admission, reranking, and maintenance, when does TokenGate offer a net benefit over simpler baselines?

2 comments
AuthorsOpenReview ↗

esponse to Reviewer gWpr

Thank you for asking when a user would actually come out ahead. The clearest example in the submitted paper is Mem0 with maintenance disabled. It also illustrates how much depends on what one counts as cost.

When the savings can pay for the extra work

At the same 512-token evidence budget, Mem0 NO_M raises ordinary F1 from 0.1689 to 0.1831 compared with store-all. It cuts stored memory text by 41.73%, while counted LLM input and output fall only from 219.0k to 217.7k tokens (Table 1). A-MEM shows a different trade-off. NO_M reduces counted tokens from 2,146.9k to 1,170.9k, mostly by avoiding native memory-processing calls, while F1 falls from 0.1922 to 0.1769. FULL adds a maintenance charge for two accepted edits. These results favor leaving maintenance off for this workload.

For deployment, a gate has the greatest opportunity to pay for itself when each avoided write saves substantial backend work. Local inference also has a cost. We measured 162.9 ms on average per candidate decision with three resident A1 models on an RTX 4050 Laptop GPU (Appendix C). The Mem0 token saving is small enough that inference could outweigh it. A net deployment saving depends on write and query volume, hardware time, index costs and the price of backend calls. Our measurements establish storage and counted-token trade-offs, while a break-even price across the full system remains unmeasured.

How far the QA evidence extends

The corrected pipeline has two independent held-out conversations, with 304 questions between them. The descriptive analysis across all ten LoCoMo conversations also includes the seven training conversations and the development conversation. It cannot enlarge the held-out sample. A related result comes from historical RHELM. At 25% retention, A1 with BM25 gives F1 of 0.1300 versus 0.0913 averaged over 15 approximately token-matched random stores on 864 conversation-answerable questions (Table A14). A separate historical comparison favors the repeat-trained C0 slightly over A1 at the same storage cap. The evidence for selective storage is stronger than the evidence for choosing this particular classifier. LongMemEval adds an admission diagnostic based on teacher labels.

Ordinary questions and abstention

Fixing the store reveals a trade-off. RST improves ordinary F1 by 5.21 and 2.66 points on Mem0 and Letta, with gains on both conversations. Adversarial scores fall by 3.19 and 6.43 points. On Letta’s conv-26, correct adversarial responses drop from 28 to 20 out of 47. Inspection found both speaker-attribution errors and cases where an abstention flag conflicted with the answer text (Table A11). We would retain native ranking as a comparison whenever reliable abstention is a deployment priority.

Thank you for the clarifications. The authors have explained the limitations of maintenance and provided more details about the cost and evaluation settings. However, the main concern remains that the paper does not show whether the learned admission classifier is better than simpler alternatives for downstream QA under a matched storage budget. The evaluation is also still based on only two independent conversations, and the overall cost benefit has not been established. Therefore, the rebuttal does not change my assessment.

Reviewer_jUqROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TokenGate combines a learned record-level admission classifier with query-conditioned retrieval under a hard token budget. The classifier uses twelve-view augmentation, and retrieval combines relevance, similarity and a length penalty. Experiments on corrected inputs report 41.7 percent fewer stored memory-body tokens. A separate experiment with fixed stores finds gains from metadata-aware reranking. The full system’s effects on quality and cost vary across backends.

Critical strengths
  1. The controls help separate the effects of augmentation and reranking. C0 matches training volume when testing augmentation, while the reranking experiment fixes both admitted stores and candidate pools. In Table 3, RST improves ordinary-question F1 over native ranking by 5.21 and 2.66 percentage points for Mem0 and Letta, respectively. The paper also reports components that did not help and states which costs are included.
Critical weaknesses
  1. The reported results leave the choice of the full configuration hard to justify. In Table 1, removing maintenance improves ordinary F1 and reduces counted LLM tokens in every backend. FULL spends 46,091 maintenance tokens on two updates and no deletions. Table 4 also shows higher ordinary F1 without lexical fusion in all three backends. The results give little reason to combine all the modules on the reported objectives.

  2. It remains unclear what the learned admission classifier adds over cheaper alternatives on downstream QA. On reviewed original-input LoCoMo, A1’s AUROC advantage over TF–IDF is uncertain: the paired difference interval is [−0.0082, 0.0350] (Table A4). The LongMemEval augmentation gain concerns teacher-labelled admission, while historical RHELM AUROC is lower than C0’s (Table A12). The corrected end-to-end evaluation covers only two conversations. Moreover, the strongest reranking comparison fixes A1’s store, so those gains cannot establish that the learned gate or twelve-view training improves QA over simpler admission methods.

Questions for authors
  1. Clarification: Which teacher, annotation rubric and information context were used to produce the 4,352 source training labels? Could the teacher see future questions or reference evidence, and how were conversation and question boundaries enforced? The purpose is to understand what the admission labels encode.

  2. Clarification: Were TF–IDF or C0 evaluated on corrected-input downstream QA with the same retained-token budget and retrieval policy? If those results exist, how much of A1’s quality advantage remains?

2 comments
AuthorsOpenReview ↗

Response to Reviewer jUqR

Thank you for distinguishing the value of the write decision from the value of ranking an already fixed store. Your questions helped us pinpoint which comparison can support each claim.

W1 Which configuration we would use

FULL switches on every module so the ablations can measure the effect of removing one at a time. For these conversations, we would turn maintenance off. It spent 46,091 counted tokens on two updates, added 53 stored tokens and deleted no records. Removing it raised ordinary F1 in all three backends (Tables 1 and 4). With maintenance retained, native ranking also gave higher F1 than lexical fusion in all three. A practical starting point is therefore admission followed by native ranking and budgeted packing. It retains 353 of 788 records. The fixed-store RST comparison then asks whether the remaining records can answer ordinary questions more effectively.

Q1 Where the 4,352 training labels came from

We used deepseek-v4-flash at temperature 0 with one fixed prompt. Each request contained the speaker, the current utterance and up to five immediately preceding turns from the same conversation. We checked this boundary against all 5,882 archived records. Future benchmark questions, answers, evidence IDs and reference counts were absent from the request. The teacher judged whether the utterance would be worth remembering for future personalized interaction. Enduring preferences, relationships, commitments and corrections qualified for KEEP. Greetings, acknowledgments and passing reactions qualified for DISCARD.

We split whole conversations before augmentation (seven training, one development, two test). Selecting training candidates that passed the filter and had a valid teacher label gave 4,352 examples. Each of the twelve views inherited its core label, yielding 52,224 training rows. Distractors came from DISCARD candidates in other training conversations. The teacher could use the preceding turns to interpret a correction. At deployment A1 sees the candidate text alone.

Q2 and W2 What the learned gate adds over cheaper gates

The corrected-input LoCoMo evaluation has no TF–IDF or C0 run that matches both the retained-token budget and retrieval policy. A1’s incremental QA gain under the requested conditions remains unknown. Two archived comparisons narrow the question.

On historical RHELM, A1 and C0 used the same per-persona storage caps and BM25 retrieval over 864 questions. C0 had slightly higher F1 at both caps. At 25%, A1/C0 scored 0.1323/0.1356, and at 50% they scored 0.1358/0.1376. This comparison uses the earlier pilot series. Its A1 records lack a model-endpoint digest, and Table A14 uses a separately verified A1 run for the comparison with random storage. An earlier Mem0 study on 100 LoCoMo questions gave the predecessor gate F1 of 0.2438 versus 0.2129 for TF–IDF, with 33,478 versus 42,666 tokens stored. Those unequal store sizes leave your matched-budget question open.

The clearest evidence for twelve-view training concerns admission scoring. On teacher-labelled LongMemEval, varying the views raises AUROC from 0.811 to 0.823 relative to C0 at matched training volume (95% interval for the gain 0.004 to 0.019). Reviewed LoCoMo favors TF–IDF on Brier error, 0.172 vs 0.215, and the paired AUROC interval spans zero (Table A4). We therefore limit the augmentation claim to the measured admission task.

Thank you for the clear and candid response. The clarification of the teacher, annotation rubric, and context boundaries addresses my question about label provenance. I also appreciate the acknowledgment of maintenance’s limited value and the narrower augmentation claim. However, the central concern remains: the corrected-input evaluation does not establish a downstream QA benefit from the learned gate or twelve-view training over simpler gates under matched storage budgets and retrieval policies. The fixed-store reranking results are valuable but do not resolve this issue. I therefore maintain my rating of weak rejection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.