DualMem: Bridging Semantic and Episodic Memory for Contextual-Aware Privacy in Large Language Models

As large language models (LLMs) increasingly act as persistent assistants, they are expected to use personal information to help users accomplish tasks. However, the appropriateness of such information varies by situation. Contextual Integrity (CI) characterizes this distinction through the actors involved, the information shared, the action taken, and the surrounding context. An LLM must therefore reason over these relationships before deciding how personal information should be used. We propose DualMem, a privacy-aware memory architecture that supports this process through two complementary memory types: Public Semantic Memory and Private Episodic Memory. Public Semantic Memory stores reusable knowledge about contextual privacy norms, while Private Episodic Memory captures situation-specific information states and relationships. Both memories are represented as event-centric hypergraphs to preserve higher-order relationships among CI elements. Given a task, DualMem retrieves relevant norms and grounds them against private information items using the current information flow and knowledge states, enabling context-aware privacy reasoning. We evaluate DualMem on three privacy benchmarks spanning probing, task-oriented generation, and CI compliance. Results show that structured semantic and episodic memory improves privacy recognition and compliance while preserving task helpfulness. Our analysis further reveals a gap between recognizing privacy constraints and consistently applying them during task execution.