acceptodds
Withdrawn from ICLR 2027

Every Token Counts: Online Memory Worth for Resource-Constrained Lifelong Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Lifelong agents continually accumulate and reuse experience to improve after deployment, yet in practice they often operate under tight generation, interaction, and memory budgets. Under such stringent joint constraints, we find that existing lifelong-memory methods can become ineffective, with several sophisticated approaches offering little advantage over simple retrieval or even no-memory agents. This reveals a more fundamental problem: resource scarcity makes each memory decision more consequential. When only a small amount of past experience can be retained or reused and opportunities for reasoning and recovery are limited, retrieving a relevant but ineffective memory can consume scarce context and hinder downstream execution. The key challenge is therefore not simply to retrieve relevant experience, but to determine which experiences are actually worth retaining and reusing. To address this challenge, we propose **TokenWorth**, a lightweight online memory framework that learns query-dependent memory worth from sparse execution feedback and applies it consistently to memory retrieval and bounded memory management. Across three LifelongAgentBench environments and both local and API-based deployments, TokenWorth achieves the highest task success rate while using the fewest output tokens among the evaluated methods. These results suggest that effective lifelong adaptation under resource constraints depends not on remembering more, but on learning what is worth remembering.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Lifelong Agents, Agent Memory, Resource-Constrained Learning, Online Learning, Memory Retrieval and Management

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_EELkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes TokenWorth, an online memory management framework for resource-constrained lifelong agents. It combines an online linear scorer with a Recursive Feature Machine (RFM)-based residual learner to estimate query-dependent memory worth from execution feedback. The learned worth is used for both memory retrieval and long-term memory management. Experiments on the DB, OS, and KG environments of LifelongAgentBench show that TokenWorth achieves higher task success rates than various memory baselines while reducing output-token usage.

Critical strengths
  1. Practically relevant problem. The paper investigates how lifelong agents can effectively reuse past experiences under limited generation, interaction, and memory resources, addressing practical constraints in real-world deployment.
  2. Clear overall framework. TokenWorth applies learned memory worth to both retrieval and long-term memory management, with continuous updates from execution feedback, forming a coherent online memory learning framework.
  3. Strong main experimental results. The paper compares TokenWorth against multiple memory baselines across three LifelongAgentBench environments and two backbone models. TokenWorth achieves the highest task success rate in all six main settings while reducing output-token usage, with particularly substantial gains on DB.
Critical weaknesses

Weakness 1: Insufficient evidence for resource robustness

The paper emphasizes joint constraints on generation, interaction, and memory resources, but the experiments do not adequately validate its central claim of resource robustness.

  • Generation Budget: Although the paper reports reduced output-token usage, it does not evaluate task success under different generation budgets.
  • Interaction Budget: The experiments impose fixed limits on interaction rounds but do not report actual interaction counts or evaluate performance under different interaction budgets.
  • Memory Budget: All three main environments set (K_{\max}=1), reducing memory selection to choosing a single high-value memory. The method neither considers combining multiple memories nor explicitly accounts for memory token costs. Moreover, the injection-length constraint may be inactive when memories are sufficiently short. Without reporting actual injected memory tokens or evaluating different memory budgets, it is unclear whether this constraint meaningfully affects the experiments.

Overall, the results demonstrate improved task success under specific fixed budgets, but provide insufficient evidence for the broader claim of resource robustness.

Weakness 2: Limited generalization

In Appendix C, TokenWorth achieves only marginal improvements over the strongest baselines on BigCodeBench and ALFWorld (0.2 and 0.3 percentage points, respectively, with the local model). It even underperforms RAG on API-based BigCodeBench. These results contrast sharply with the substantial gains on DB, suggesting that the method's effectiveness may be task-dependent. The paper does not adequately investigate this discrepancy.

Weakness 3: Limited gains from the complex design

In Figure 3(b), the simple linear scorer already achieves 64.2% success. Adding RFM reduces performance to 63.0%, while the full method reaches only 65.0%—a gain of just 0.8 percentage points over the linear baseline. This raises questions about whether the added RFM and AGOP complexity is justified. Furthermore, component ablations are not provided for the other environments, leaving their broader necessity unclear.

Questions for authors

See weaknesses.

Reviewer_WDB3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies lifelong agents under joint constraints on generation, interaction, and memory. It proposes TokenWorth, an online memory framework that estimates query-dependent memory utility from task feedback. The method combines a prior-regularized linear scorer with a residual Recursive Feature Machine, and uses the learned scores for both retrieval and bounded memory management. Experiments on database, operating-system, and knowledge-graph tasks from LifelongAgentBench show higher success rates and lower output-token usage than the evaluated baselines. Additional results on BigCodeBench and ALFWorld show smaller and less consistent advantages.

Critical strengths
  • The paper studies a practical setting in which memory construction and reflection must share a limited resource budget with task execution. Accounting for these costs makes the evaluation relevant to constrained deployments.

  • The controller uses task feedback for both retrieval and retention without backbone updates or additional language-model calls. The reported improvements across LifelongAgentBench tasks and deployment settings, particularly on database tasks, make this a useful approach to investigate.

Critical weaknesses
  • The necessity of the main technical components is not convincing. In Figure 3(b), the base scorer reaches 64.2% on DB versus 65.0% for the full framework, while adding the RFM alone gives 63.0%. Most of the gain over RAG at 33.0% is therefore already present without the residual learner. The core comparison also lacks uncertainty across independent runs or task orders; Appendix D’s hyperparameter intervals do not establish the reliability of this small incremental gain.

  • The evidence for resource robustness and efficiency is incomplete. Table A1 uses one joint budget per environment, with 128 generation tokens for DB and KG, 512 for OS, and one retrieved memory. These settings may strongly constrain reflection-based baselines. Output-token counts also omit input processing and the controller’s CPU overhead, while Appendix A.4 does not provide the mentioned runtime and resident-memory measurements. Consequently, the practical advantage under different resource limits is not yet clear.

Questions for authors

Please refer to the weaknesses.

Reviewer_AwaTOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes TokenWorth, a lightweight online memory framework that learns memory's value from execution feedback and can use it to memory retrieval and management under constraint cases. It tackles the main limitation on most existing memory frameworks that their memory budget is unbounded and use and manage memories inefficiently, i.e. using more capacity than it should. From the results, TokenWorth is especially stronger compared with existing memory and lifelong agent methods, while maintaining low token usage, and further it demonstrates memory-usage efficiency is important beyond capacity.

Critical strengths
  1. The problem is well-motivated and method is sound and impressive. The framing is convincing that memory efficiency weighs more than memory capacity lone. Also, authors study the lifelong agents under joint constraint in generation, interactions, retrieval, memory capacity, etc, which covers diverse modules in memory-system. The two-scale design on memory adaptation is especially reasonable for tackling memory "overfitting" to a specific feedback. Constraining auxiliary calls for memory operations also makes result convincing.

  2. The advantages margin is large between TokenWorth and the best baseline, which shows that this method also empirically promising. Especially on database tasks, accuracy is mostly doubled over the best baseline. The output token efficiency is also demonstrated.

  3. The writing and delivery is very clear, convincing and easy to follow.

Critical weaknesses
  1. My main concern is TokenWorth's generalization ability. The main evaluation uses only one constraint budget configuration, and there is no budget sweep to test if TokenWorth is budget-sensitive. 2-3 more budget configuration results can make the paper stronger at demonstrating the robustness of proposed framework. Further, in appendix, the generalization test on BigCodeBench and ALFWorld shows much smaller advantages over baselines.

  2. The ablation argument is weak now. Ablation Figure 3b shows very small component advantages (3 variants are all within 63-65), especially there is a drop from base to RFM. The effectiveness or roles RFM played in the system needs other evaluation or argument to support.

Questions for authors
  1. How sensitive the proposed framework is on memory budget? And how does it perform under sufficient budget? Would it remain comparable or better compared with other baselines?

  2. Is this method also generalizable to difficulty tasks and reasoning?

Reviewer_LEzjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies lifelong agents operating under joint constraints on model output, interaction rounds, memory capacity, and memory injection. It proposes TokenWorth, an online framework that learns which past experiences to retrieve and retain without updating the language-model backbone.

TokenWorth combines a linear memory scorer with a nonlinear correction model, a Recursive Feature Machine (RFM). Prediction and feature-weight updates operate at different frequencies to balance adaptation and stability. Execution feedback updates memory scores, which also guide admission and eviction.

Evaluation covers database, operating-system, and knowledge-graph tasks from LifelongAgentBench using two backbones. TokenWorth achieves the highest success rate and lowest output-token usage among the evaluated methods in all six main settings (Table 1; Figure 3(a)). Additional experiments cover BigCodeBench and ALFWorld, where the improvements are substantially smaller (Table A2).

Critical strengths
  • Substantial gains in the main evaluation, particularly on database tasks. Table 1 reports 65.0% success versus 33.0% for the strongest baseline with the local model, and 62.2% versus 32.2% with the API model. These large improvements across two backbones are the strongest evidence supporting the paper.

  • A practically relevant focus on the cost of maintaining and using memory. Counting auxiliary language-model calls within the generation budget is appropriate, and TokenWorth improves success while reducing output usage (Figure 3(a)). Its ability to update memory without additional language-model calls makes the approach potentially useful in constrained deployments.

Critical weaknesses

The evidence does not establish the contribution of the proposed learning mechanisms.

  • Figure 3(b) reports 64.2% for “Base Only,” 63.0% for “Base + RFM,” and 65.0% for the full method. The full method therefore improves over the simpler scorer by only 0.8 percentage points, equivalent to four additional successes on the 500-task database stream. Without uncertainty estimates or a clearer explanation of the differences between these variants, the necessity of RFM and its feature adaptation remains unclear.
  • This matters for novelty: MemRL (Zhang et al., 2026) already learns memory utility from execution feedback, while Hajimiri et al. (2026), (Are Online Skill and Memory Modules Always Worth Their Tokens?), already examines memory augmentation under inference-budget constraints. Both are cited in Section 2, but the experiments need to establish the advance beyond these existing ideas.
  • The additional benchmarks offer limited supporting evidence: the local improvements over the strongest baselines are only 0.2 and 0.3 percentage points, and API-based BigCodeBench trails RAG, 50.6% versus 50.9% (Table A2). These results support competitiveness, but provide little evidence that the added mechanisms generalize reliably.

The evaluation leaves important questions about the budget and feedback assumptions unresolved.

  • Section 3 defines a total per-task generation budget, including auxiliary calls, while Table A1 sets this budget to 128 tokens for database and knowledge-graph tasks, with interaction limits of 3 and 15 rounds respectively. These are unusually restrictive settings. The implementation and allocation of this budget need clarification, and one configuration per environment is insufficient to establish whether the advantage persists under moderately different limits.
  • The method requires task-success feedback for learning and externally verified success for admitting new memories (Sections 4.2 and 4.4). This is reasonable for automatically verifiable tasks, but the paper does not sufficiently explain which deployment settings provide equivalent feedback without access to benchmark answers. This limits the practical scope of its claims.
  • The utility update adds 0.25 after success and subtracts 0.45 after failure (Appendix B.1, pp. 16–17), without explaining how these values were selected. Appendix D examines several hyperparameters, but does not test these update magnitudes.
Questions for authors
  • How is the 128-token task-level budget enforced across interaction and auxiliary calls? Please explain allocation and truncation, particularly for the knowledge-graph setting allowing up to 15 interaction rounds, and justify the chosen limits.

  • What success information is exposed to the memory learner? Does it require benchmark reference answers, and which intended deployment settings provide comparably reliable feedback?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.