MemSAGE: Asymmetric Memory Beamforming for Contextual Integrity in LLM Memory

Modern Large Language Models (LLMs) are evolving to incorporate persistent memory, enabling them to retain user information across interactions and thereby improve user experience and task performance. However, this capability can be a double-edged sword. Recent studies have revealed the contextual integrity problem of persistent memory, where LLMs utilize retained information indiscriminately without assessing its contextual appropriateness, resulting in unwarranted disclosure of sensitive information. Existing approaches to this problem either regulate memory use externally via explicit prompting and rigid retrieval control, or internalize contextual reasoning into LLMs, leaving unmet the asymmetric, context-specific demands on retained memories. To address this gap, we propose MemSAGE, a novel plug-in framework for enforcing the contextual integrity of LLM MEMory via Stiefel-based Asymmetric Geometric bEamforming. MemSAGE presents a unique twist: it trains a lightweight hypernetwork to dynamically generate Stiefel frames for each context and the retained memories, forming a nested permission–demand geometry that directs memory use during generation. To operationalize this structure, MemSAGE performs asymmetric memory beamforming over LLM attention, suppressing context-inappropriate memories while directionally amplifying obligatory ones through geometry-aligned query–key interactions. Notably, MemSAGE is applied at inference time without altering the retained memories or updating the base model parameters. Extensive experiments across various LLM backbones demonstrate that MemSAGE significantly enhances the contextual integrity of LLM memory compared with existing baselines, simultaneously reducing memory violations and improving coverage.