acceptodds
Under review as a conference paper at ICLR 2027

Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents

Abstract

Large Language Models (LLM) agents increasingly rely on external memory systems to accumulate experience across tasks. Yet many existing approaches, from graph-structured memories to reflective insight stores, access memory through fixed, hand-designed heuristics. We argue that this static view of memory is a core bottleneck for agentic learning because optimal memory behavior is fundamentally context-dependent. The early stages of tasks benefit from minimal retrieval because memory is sparse; recurring goal types benefit from plan reuse rather than generic nearest-neighbor lookup; stuck agents benefit from re-retrieval with alternative queries; and across long task streams, the memory store itself must be consolidated and pruned to remain useful. We present MEMCON (Memory as a Controlled Process), a framework that models memory operations as a Markov Decision Process and learns an online policy that adaptively decides when, what, and how much to retrieve, when to inject a distilled plan, and when to consolidate or forget. MEMCON is backend-agnostic: it wraps compatible memory implementations through a common interface, learns from task-by-task binary feedback with no controller pretraining or additional LLM calls for policy selection, and uses a lightweight tabular contextual bandit with UCB exploration that adapts during deployment. Across 6 benchmarks, 3 agent frameworks, and 3 LLM backbones, MEMCON improves average performance over the evaluated memory baselines; on GPT-4.1-mini, it raises six-benchmark mean success by 2.6–4.2 points over G-Memory while reducing mean input-token use by 10.9–24.1%.

Primary area:
reinforcement learning
Keywords:
memory

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.50 over 2 reviews, above 99% of the others.
Reviewer_UTaYOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MemCon, a learned backend-agnostic wrapper that formulates LLM agent memory management as a sequential decision problem. Memory operations and their parameters are selected based on task progress and memory state. The policy is learned online using a tabular contextual bandit with UCB exploration, initialized with manually-specified heuristic priors and updated from task-level feedback. The authors evaluate MemCon across six benchmarks, three agent frameworks, and three LLM backbones and show improvements in task success and token efficiency over baselines.

Critical strengths

The paper’s strengths are (1) its generalizable approach to adaptive agent memory management, and (2) its comprehensive empirical evaluation and ablation studies showing the controller’s contribution.

First, MemCon’s backend-agnostic controller learns its policy online without additional LLM training or inference for policy selection. It addresses the limitations of fixed memory access methods that may either retrieve information too aggressively or too conservatively. Learning memory operations is not new, but MemCon provides a lightweight approach that considers both task progress and memory status to adaptively select memory operations and parameters.

Second, the authors conduct extensive comparisons against memory control baselines across six benchmarks, three agent frameworks, and three LLM backbones. These experiments show MemCon’s improvement in task success and token efficiency. The component ablation on ALFWorld shows that the learned controller alone improves performance from 59.7% to 64.9%, which accounts for most of the system’s improvement.

Critical weaknesses

Although the authors show improvements over fixed and several adaptive memory access methods, they do not sufficiently compare against simpler alternatives. For example, the authors could try epsilon-greedy exploration with the same states, actions, and Q-value updates. Or try a non-learning, state-dependent heuristic policy using the same states and actions.

I also found the paper’s figures difficult to parse because they were either crowded with text or had small fonts.

Questions for authors

Could the authors please compare their method against simpler controllers, such as epsilon-greedy exploration or a non-learning, state-dependent heuristic policy using the same states and actions?

Reviewer_HuugOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors model memory as a Markov Decision Process conjoined with an online policy which decides the retrieval, injection, and consolidation/forgetting process. They describe this framework as backend-agnostic and and able to learn at inference-time. They demonstrate improved results with GPT-4.1-mini across 6 benchmarks with reduced mean input-token use.

Critical strengths

I really appreciate how this paper focuses on a key bottleneck of agentic architecture. The previous works are well-researched and relevant. In addition, the described method of selection via MDPs and online policy learning serves as a unique way to navigate the action space at inference-time.

Critical weaknesses

A few key weaknesses/concerns are as listed below:

  1. Benchmarks like ALFWorld and WebQA seem saturated by current suites of literature, why did the authors choose these benchmarks and what makes them good fits for the types of tasks that MEMCON would be good at?
  2. The paper reports that policy hyperparameters are selected on a held-out 30-task ALFWorld subset. Is this too small of a subset to adjust the methodology on and is there empirical evidence to support this sample size?
Questions for authors

A few areas of concern are listed below:

  1. The fact that average token use decreased is interesting. Do the authors have any intuition for why this is so? Were there any inference-time optimizations that were relevant beyond beyond the MDPs and the online policy learning?
  2. The authors completed this research with GPT-4.1-mini, Claude Sonnet-4, and DeepSeek-V3.2 -- as these models seem a bit outdated, I'm curious as to how the architecture performs with more recent model releases. Do the authors have these results?

Suggestions for future work:

I'm curious as to why MEMCON outperforms graph-based memory approaches (GMemory). It would be really awesome if the authors could provide an intuition for this!

Comments on the paper

2 comments
Area_Chair_jVfUOpenReview ↗

Emergency Review from AC (Paper was desk rejected, but got reverted later, therefore there were only 2 assigned reviewers)

The paper proposes a lightweight online controller for agent memory. I like the practical motivation: retrieval should adapt to the task and the available experience. The gains in both success rate and input-token use are encouraging, and the ablation shows that learned control contributes beyond the added planning heuristics.

My main reservation is that the method is fairly incremental, combining tabular UCB with hand-designed states and actions. Most main results also come from single runs, so the smaller improvements are hard to assess. The MDP framing is broader than what the implemented credit assignment supports, particularly for maintenance whose benefits appear in later tasks.

Overall, I lean toward weak accept. The contribution is useful and reasonably well supported, though stronger evidence across repeated runs would make the case more convincing.

AuthorsOpenReview ↗

Response to AC

Dear Area Chair,

Thank you very much for taking the time to provide an emergency review, especially given the unusual circumstances surrounding our submission! :)

We sincerely appreciate your thoughtful assessment and your recognition of the practical value of adaptive memory control. We would also like to briefly clarify three points regarding novelty, experimental reliability, and the MDP formulation.

1. Novelty and contribution. We agree that tabular UCB itself is not a new algorithm. Our main contribution is instead the formulation and implementation of memory management as an adaptive decision process, where retrieval, plan reuse, and memory maintenance can be jointly controlled online using task feedback. To the best of our knowledge, MEMCON is the first framework to jointly learn these memory-management decisions through a unified, lightweight online controller for LLM agents. Unlike fixed memory strategies, MEMCON learns when and how to access memory without requiring additional model training or LLM calls for policy selection. Importantly, our ablation shows that the learned controller alone improves ALFWorld success from 59.7% to 64.9%, before introducing the additional planning components. We have also added comparisons against random selection, fixed heuristics, and ε-greedy exploration. These results help isolate the contribution of learned memory control rather than attributing improvements solely to additional heuristics.

2. Experimental reliability. We appreciate your concern regarding single-run evaluations. To provide additional evidence, we report three-trial results in Table 22. On ALFWorld, MEMCON achieves 67.9 ± 2.1%, compared with 59.7 ± 2.8% for G-Memory. On ScienceWorld, the corresponding results are 38.0 ± 2.4% versus 34.0 ± 3.1%. We agree that repeated evaluations across every configuration would further strengthen the evidence, particularly for smaller improvements, and we do not claim statistical significance from the single-run comparisons. Additionally, our expanded experiments with newer models, including GPT-6-Luna and Qwen-3.8-27B, provide further evidence of the framework's applicability across model backbones.

3. MDP formulation and credit assignment. We agree with your observation that the general MDP formulation is broader than the implemented learning mechanism. In particular, the current contextual-bandit controller updates its policy using task-level feedback and does not explicitly solve long-horizon credit assignment for memory maintenance decisions whose effects may emerge over subsequent tasks. We will make this distinction clearer in the manuscript. Our intended contribution is a lightweight and practical approach to online memory control, rather than a general solution to delayed-reward memory optimization.

Thank you again for your constructive and encouraging assessment! Your feedback helps us clarify both the contribution and the limitations of our work, and we sincerely appreciate your consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.