acceptodds
Under review as a conference paper at ICLR 2027

Toward Adaptive Memory Injection in Agent Systems: A Structure-driven Approach

Abstract

Memory-augmented large language model (LLM) agents have largely focused on storing and retrieving memories. Yet retrieved candidates may contain redundant or task-irrelevant information, while answering a question may require complementary information distributed across multiple memories. Injecting all candidates increases context cost, whereas selecting them solely by individual relevance can omit information needed to cover the task. We introduce Learnable Memory Injection (LMI), which selects from already-retrieved candidates based on the task-relevant information missing from the current selection. It identifies task requirements and links them to source-bound facts in a task–memory relation graph. Using this graph, LMI assesses what information each candidate adds beyond the selected memories and prioritizes memories that supply missing task-relevant information under a memory-token budget. It then organizes the selected memories into the model context. LMI also supports learning context-dependent memory utility from downstream feedback to guide subsequent selections. Experiments across three LLM backbones show that LMI maintains or improves answer accuracy. The largest relative gain is 20.43%: with Llama-3.1-8B and Top-200 candidates, accuracy increases from 64.60% to 77.80%, while injected memory tokens decrease by 83.90% compared with directly injecting all candidates.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, agent memory, memory injection, memory selection, long-term memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uaVHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Learnable Memory Injection (LMI), a post-retrieval memory-injection module placed between memory retrieval and model inference. It addresses two limitations of existing approaches: injecting all retrieved candidates introduces substantial contextual redundancy, while selecting memories solely according to individual relevance may omit complementary evidence required by the task. LMI extracts task requirements and source-traceable facts from the query and candidate memories, constructs a task–memory relation graph, and jointly selects complete memory items according to the task-relevant information missing from the current selection under item-count and token-budget constraints. It further trains a context-dependent utility scorer using downstream performance differences observed when individual memories are removed and organizes the selected memories using a fixed context template. Experiments on LongMemEval and LoCoMo with Qwen2.5-7B, Llama-3.1-8B, and GPT-5 show that LMI generally maintains or improves answer accuracy while substantially reducing the number of injected memory tokens.

Critical strengths

Structured and interpretable selection mechanism: LMI constructs source-traceable task–memory relations and selects memories according to their marginal coverage of missing information. This design is more interpretable than purely embedding-based ranking and preserves the original memory text for provenance.

Strong empirical effectiveness and efficiency: LMI is evaluated across two benchmarks and three LLM backbones. It generally improves answer accuracy while substantially reducing injected memory tokens; for example, Llama-3.1-8B improves from 64.6% to 77.8% on LongMemEval Top-200 while using 83.9% fewer memory tokens.

Critical weaknesses

The methodological novelty is limited. Graph-based retrieval and reasoning, requirement-aware context filtering, budgeted set-dependent selection, and feedback-based utility estimation have all been explored in prior work adaptive memory-routing systems. Some woud combine these established components into a post-retrieval memory-injection pipeline. Although the source-linked requirement–fact graph provides an interpretable implementation, the paper does not clearly demonstrate that it constitutes a fundamentally new selection principle rather than a task-specific reformulation of existing coverage-based context selection.

Questions for authors

see weakness

Reviewer_gcbhOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Learnable Memory Injection (LMI), a post-retrieval module that selects and organizes a subset of already retrieved memories before they are passed to an LLM agent. LMI parses a query into task requirements, extracts source-linked facts from candidate memories, constructs a task–memory relation graph, and selects memories according to requirement coverage, fixed selection rules, and a learned utility score trained from paired memory-removal feedback. The selected memories are then rendered into a structured context.

Critical strengths
  1. The motivation is straightforward and practically relevant.

  2. Well organized.

Critical weaknesses
  1. The technical novelty is limited and not clearly separated from engineering complexity.

  2. Direct injection is not a token-matched baseline, while a simple budget-matched Top-k selector is not reported. In addition, the GPT-5 improvements over Direct are only 0–1.8% relative, yet the paper reports no repeated generations, confidence intervals, or significance analysis.

  3. The efficiency claim is weaker than the token-reduction figures suggest. The measurements exclude retrieval, grading, and offline utility training; therefore, they do not establish lower end-to-end latency or monetary cost.

  4. The method is heavily rule-based, and its generalization is unclear. The semantic frontend uses spaCy together with hand-written dependency patterns, lexical rules, seven relation checks, protected-rank rules, replacement constraints, and several fallback branches.

  5. The central task–memory graph contribution is not isolated experimentally. Table 4 explicitly states that all ablation variants retain the graph. The independent-selection ablation only tests set-dependent selection, the rendering ablation tests presentation, and the utility ablation tests the learned score. None of these results shows that constructing the proposed graph is better than using ordinary query–memory features under the same selector and compiler.

Questions for authors

Please see the Weaknesses section. I would appreciate clarification regarding the technical novelty of LMI, the absence of a budget-matched Top-k baseline and uncertainty estimates, the complete end-to-end computational cost, the generalizability of the heavily rule-based design, and the lack of an ablation that directly isolates the contribution of the task–memory graph.

Reviewer_QsxyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a structured memory post-retrieval processing pipeline to deal with the issue of long retrieved memory that are not only token inefficient, but also can contain irrelevant materials that would decrease the performance of agents. The core technique is quite smart, it first leveraged rules to compose a graph of documents based on retrieval, and then use a regression model to compare it with the query to decide final documents sent to the agents for QA. This pipeline is rather lightweight, and can leverage question to both ensure structural completeness of documents and find questino related documents with the scorer.

Critical strengths
  1. A lightweight approach balances the structural constraints (based on rules) and document embedding (based on the scorer that can be lightly trainable) so that it gains from existing post-processing pipeline to get documents with good coverage yet trainable in a simple way (vs MMR and SC).
  2. I think the technique is quite applicable both for large memory and small / local memory setup given it's simplicity.
  3. evaluation demonstrates that the performance gain.
Critical weaknesses
  1. The rule-based construction and filtering criteria seems a bit adhoc (though quite good in practice). Despite the overall performance gains, it remains unclear how robust these rules are beyond the evaluated personal-memory QA settings.

  2. The post processing increases the response time quite a lot, esp it's CPU time before sending documents to the agent (50% over simple methods). This generally means the decrease in interactivity for user interaction.

Questions for authors
  1. I'm wondering if authors can further dive into the issues that might arise from the adhoc rules? Esp an ablation isolating the rule-derived graph’s contribution to help establish the approach’s generalizability.

  2. Can some of the pipeline done offline to reduce the processing time?

  3. Since the scorer is a quite simple regression and I'm curious about it is stable enough to capture task-related score, I'm wondering if the authors can present an abalation for different filtering threshold would affect the performance? Is it stable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.