Different Strokes for Different Folks: Memory Transformation Routing for LLM Agents

Long-term memory enables language agents to retain information beyond a bounded context window. Existing systems typically transform interaction histories through consolidation, augmentation, or relational structuring, applying a fixed transformation across queries. We show that these transformations offer complementary benefits: each improves some queries while degrading others, and no single transformation consistently dominates. This finding motivates MemRx, a memory-aware router that selects among raw and transformed memory views for each query. However, routing based solely on the query and answer-level supervision is insufficient: the query does not reveal how relevant evidence is organized in memory, while answer scores overlook evidence quality and can be sensitive to wording. MemRx addresses these limitations through two components. Probe-Conditioned Routing (PCR) combines query representations with memory-state signals extracted from a lightweight retrieval probe to estimate each view’s utility. Dual-Level Utility Supervision (DUS) integrates LLM judgments of evidence sufficiency and answer correctness with gold-answer F1, giving greater weight to judge signals when F1 poorly distinguishes candidate views. At inference time, MemRx executes only on the selected view and requires a single LLM call. Experiments on LoCoMo and PerLTQA demonstrate the benefits of adaptive transformation selection, with gains of up to 2.05 average F1 points over the strongest fixed transformation on LoCoMo.