Remember the Gist, Recover the Details: Task-Adaptive Agent Memory Generation

With the rise of agent applications, effectively compressing large volumes of interaction history has become an important research problem. An ideal memory compression method should preserve as much task-relevant information as possible for downstream tasks. However, because relevance depends on the task—details useful for one task may be distractions for another— existing methods often struggle to retain task-specific details while filtering out task-irrelevant noise. We propose GiDe, a two-stage method that efficiently extracts both the main task information and task-specific details under a limited memory budget, reducing the interference of task-irrelevant noise with detail extraction. In the first stage, task-agnostic trajectory backbone extraction identifies key information directly from the original trajectory to produce a reusable semantic skeleton. In the second stage, task-specific detail supplementation actively queries the original trajectory for details missing from the skeleton, and combines them with the task information already captured in the first stage to form the final memory. Experiments on LoCoMo and LongMemEval-S with multiple backbone models, including GPT-4o-mini and Qwen, show that GiDe consistently outperforms existing memory compression methods, substantially improving both question-answering accuracy and F1 score while keeping total token consumption for compression low. The efficiency advantage is even more pronounced when the same trajectory is reused across multiple tasks.