Beyond Similarity: Trustworthy Memory Search for Personal AI Agents

Personal AI agents increasingly rely on long-term memory to personalize their behavior across sessions. Existing memory pipelines, however, are largely driven by semantic similarity: memories close to the current query are retrieved and injected into the model context. This creates a trustworthiness gap, because a semantically related memory may still be contextually inappropriate, leading to cross-domain leakage, sycophancy, tool-call drift, and memory-induced jailbreaks. To mitigate these failures, we propose MemGate, a lightweight, deployable plug-in for trustworthy memory search with only 9M parameters and a 35.1MB footprint. MemGate sits between the vector memory store and the backbone LLM and requires no LLM modification, memory-database rewriting, or inference-time LLM judge. It applies a query-conditioned neural gate to candidate memory representations, turning raw similarity search into task-conditioned memory ranking. Across multiple mainstream memory frameworks, a real-world agent, and diverse LLM backbones, MemGate reduces memory-induced failures while preserving long-term memory utility.