RAMem: Role-Aligned Hidden-State Memory for Multi-Agent Systems

Large language model (LLM)-based multi-agent systems (MAS) accumulate experience as their roles collaborate, and memory is the mechanism that turns this experience into guidance for later tasks. However, existing memory pipelines select and reuse it at mismatched granularities: a trajectory can be globally relevant yet hold little that the role acting next can reuse, and compressing its mixed-role content into a fixed latent budget lets irrelevant or conflicting material crowd out what that role needs. To address this, we propose RAMem, a role-aligned hidden-state memory. Instead of storing text, RAMem archives the final-layer hidden states of each role's responses in successful trajectories, and it selects experience by both the task and the target role. A role-conditioned composer then compresses the selected states into a fixed-length latent memory while the agent backbone stays frozen. We train the composer in two stages, first aligning it to a text-memory teacher and then optimizing it for task reward with Hidden Memory Advantage Policy Optimization (HMAPO). Across six benchmarks and three MAS architectures, one unseen during memory construction, RAMem attains the best average score in every architecture, 3.1 to 5.9 points above LatentMem.