ReCoverMem: Risk-Controlled Trust in Compressed Agent Memory

Long-horizon LLM agents compress growing interaction histories into long-term memory, but always trusting compressed memory risks acting on insufficient evidence, while always recovering additional evidence forgoes selective memory use. We introduce ReCoverMem, a Trust/Recover controller that ranks decisions with a learned memory-sufficiency score and applies unit-level conformal risk control. Our audit shows that treating dependent decisions as independent examples weakens the finite-sample correction. Across three workloads and four downstream backbones, always trusting memory yields false-safe risk of 45–72%. At $\alpha =.10$, ReCoverMem routes 10–22% of decisions to Trust, achieving lower false-safe risk and lower exceedance in all 12 configurations. When recovery is substantially more costly, selective trust also reduces execution cost: on NarrativeXL, ReCoverMem saves 8.5–13.6% of executed-branch tokens at $\alpha =.10$ and 3.9% at $\alpha =.05$.