No Universal Memory: Query-Aware Routing for Episodic Retrieval in LLM Agents

Long-term LLM agents require episodic memory to reason over interaction histories spanning many sessions, yet no single retrieval strategy serves all query types equally well. Prior graph memory systems propose universal typed-edge topologies and report end-to-end improvements over flat retrieval-augmented generation (RAG), but none has tested whether a single topology generalizes across heterogeneous query classes. We introduce the Causal Temporal Episodic Graph (CTEG), a directed typed-edge memory graph evaluated through 11 ablation variants in the first per-component study at scale, to our knowledge (n=500, LongMemEval-S, full conversation haystack, McNemar significance tests). The central finding is that no fixed topology dominates: the full CTEG system loses to some ablation on every single question type; oracle routing over per-type optimal variants achieves 86.8% (+2.2 pp over the best fixed system), while the best structured memory variant outperforms flat RAG by 14.4 pp (p<0.001). These results establish memory retrieval as substantially query-type-dependent on this benchmark: cross-session CAUSAL BFS shows a directional precision penalty (−2.8 pp, p=0.022, non-significant after family-wise correction), a routing policy recovers most of the per-type gains (+10.0 pp on preference via LATEST dispatch), within 2.8 pp of the oracle ceiling.