Which Experiences Should Be Replayed? Causal Selection for Latent-Memory Agents

The latent-memory paradigm in self-evolving agents, exemplified by MemGen, augments reasoning with trainable Weaver latent vectors; however, it still exhibits substantial forgetting in the tested four-task staged No-replay sequential-learning setting. Replaying historical experience can mitigate this forgetting; the key is to identify examples worth replaying. To this end, we diagnose Weaver event-slot trajectories and find that static saliency signals can nominate candidates but cannot independently provide the bilateral helpful/non-harmful evidence required for replay admission; a slot's intervention effect also depends on its specific injection event. We therefore propose Causal Experience Selection (Causal replay), which uses static signals to narrow the event-slot candidate set, independently masks each candidate at the incoming checkpoint, and determines sample admission from bilateral teacher-forced evidence before constructing a fixed-budget replay set. In four-stage continual learning with Qwen2.5-1.5B-Instruct and a MemGen Weaver, Causal replay improves final four-task mean accuracy (Macro) from No replay's $25.41\pm 0.38$ to $45.88\pm 0.71$, indicating that replay mitigates forgetting in the tested MemGen setting. Under strictly matched actual per-source replay counts and training volume, differing only in historical-example identity, Causal replay obtains a mean paired gain of $+2.19$ percentage points (pp) over quantity-matched Random across three seeds (95% paired-$t$ interval $[+1.13,+3.25]$). This supports selection based on event-slot evidence over random selection in this controlled comparison; task-order permutation and Qwen3-4B also show same-direction results.