From Relevance to Decision Value: Low-Overhead Trajectory-Grounded Memory Valuation and Selection for LLM Agents

Memory selection for large language model (LLM) agents hinges on a fundamental question: How should the value of a memory unit be defined? Existing methods often approximate memory value through relevance or novelty, while generative selectors and learned memory policies can require additional training and interaction. A practical observation is that many real-world agents operate repeatedly within bounded workflow domains, naturally accumulating execution histories over time. Motivated by this setting, we take a decision-centric view and define memory value by its query-conditioned contribution to downstream task utility. Based on this definition, we propose a practical trajectory-grounded Value-aware Memory (VAM) estimator, which directly induces a low-overhead Top-$K$ memory selection method without additional learned selectors and rollouts. We further analyze two key questions about the proposed estimator. First, when does the estimated value meaningfully reflect downstream memory contribution? Second, how much trajectory evidence is enough to make the valuation reliable? We answer the first through an average marginal-contribution interpretation, and the second by deriving a Valuation Reliability Assessment (VRA) method to quantify how strongly the available trajectory evidence supports the reliability of the resulting valuation. Experiments on ToolBench, HotpotQA, and ALFWorld demonstrate a favorable performance–cost trade-off across tool use, multi-hop question answering, and interactive decision making. The experiments further validate the practical utility of VRA and show that VAM can progressively improve downstream utility as execution history accumulates.