Toward Adaptive Memory Injection in Agent Systems: A Structure-driven Approach

Memory-augmented large language model (LLM) agents have largely focused on storing and retrieving memories. Yet retrieved candidates may contain redundant or task-irrelevant information, while answering a question may require complementary information distributed across multiple memories. Injecting all candidates increases context cost, whereas selecting them solely by individual relevance can omit information needed to cover the task. We introduce Learnable Memory Injection (LMI), which selects from already-retrieved candidates based on the task-relevant information missing from the current selection. It identifies task requirements and links them to source-bound facts in a task–memory relation graph. Using this graph, LMI assesses what information each candidate adds beyond the selected memories and prioritizes memories that supply missing task-relevant information under a memory-token budget. It then organizes the selected memories into the model context. LMI also supports learning context-dependent memory utility from downstream feedback to guide subsequent selections. Experiments across three LLM backbones show that LMI maintains or improves answer accuracy. The largest relative gain is 20.43%: with Llama-3.1-8B and Top-200 candidates, accuracy increases from 64.60% to 77.80%, while injected memory tokens decrease by 83.90% compared with directly injecting all candidates.