acceptodds
Under review as a conference paper at ICLR 2027

RAMem: Role-Aligned Hidden-State Memory for Multi-Agent Systems

Abstract

Large language model (LLM)-based multi-agent systems (MAS) accumulate experience as their roles collaborate, and memory is the mechanism that turns this experience into guidance for later tasks. However, existing memory pipelines select and reuse it at mismatched granularities: a trajectory can be globally relevant yet hold little that the role acting next can reuse, and compressing its mixed-role content into a fixed latent budget lets irrelevant or conflicting material crowd out what that role needs. To address this, we propose RAMem, a role-aligned hidden-state memory. Instead of storing text, RAMem archives the final-layer hidden states of each role's responses in successful trajectories, and it selects experience by both the task and the target role. A role-conditioned composer then compresses the selected states into a fixed-length latent memory while the agent backbone stays frozen. We train the composer in two stages, first aligning it to a text-memory teacher and then optimizing it for task reward with Hidden Memory Advantage Policy Optimization (HMAPO). Across six benchmarks and three MAS architectures, one unseen during memory construction, RAMem attains the best average score in every architecture, 3.1 to 5.9 points above LatentMem.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-Agent Systems, Latent Memory, Role-Aligned Retrieval, Experience Reuse

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EjNuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a technically meaningful extension of latent memory for multi-agent systems, with promising empirical results. However, concerns remain regarding the attribution of performance gains, computational efficiency, and the robustness of cross-architecture generalization.

Critical strengths

This work presents a well-motivated, role-aligned memory design to address the granularity mismatch in trajectory-level memory retrieval, where globally relevant experiences contain abundant information irrelevant to an agent’s functional role. Comprehensive evaluations across six benchmarks and three multi-agent system architectures with informative ablations show its effectiveness.

Critical weaknesses
  1. While ablation studies are provided, RAMem jointly modifies memory representation, retrieval strategy and composer training. It remains unclear whether hidden-state representations yield inherent advantages over text-based memory under controlled retrieval, training and compute budgets.

  2. Efficiency evaluation only reports per-agent token usage. RAMem incurs unmeasured overhead from token-level hidden-state storage, LLM-powered role-aligned reranking and a standalone composer module. End-to-end latency, memory bank footprint and total computational cost should be quantified to support efficiency claims.

  3. All experiments adopt Qwen3-4B as the backbone. Thus results only show transfer across communication structures, not heterogeneous agent models.

Questions for authors

See Weakness.

Reviewer_KErmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose that MAS memory dilutes what the acting role needs, and to address this, the authors propose RAMem that stores each role’s final-layer hidden states and retrieves them by task and role similarity. In detail, this paper trained by distillation and then with an RL objective (HMAPO), compresses them into latent tokens. Across six benchmarks and three MAS architectures with Qwen3-4B, it beats LatentMem by 3.1 to 5.9 points on average.

Critical strengths
  1. Organizing memory around the role that uses it is a clear idea, and it helps even on an architecture unseen during memory construction.

  2. The ablations are comprehensive and clear. HMAPO alone does not explain the gain (Table 6), and latent memory beats text injection while using fewer tokens (Table 3).

Critical weaknesses
  1. On CAMEL, hidden states without role-aligned retrieval score below LatentMem (54.55 vs 55.46). This contradicts the claim that hidden states drive the improvement.

  2. For the evaluations, only Qwen3-4B is tested, and the stored hidden states cannot be reused with any other backbone. The gains also concentrate on closed-book QA, with little change on BigCodeBench.

Questions for authors

On CAMEL, how often does task-only retrieval return trajectories with no record for the target role, which would leave the memory empty?

Reviewer_bxnkOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes RAMem, a Role Signed Hidden State Memory method for LLM based Multi Agent Systems (MAS). The author believes that traditional travel level memory may introduce information unrelated to the target Role. Therefore, it is proposed to store the hidden states generated during the historical generation process by Role, retrieve them by combining Task and Role similarity, compress them into fixed length late memory through Composer, and optimize task performance using HMAPO. The experiment covered six benchmarks and three MAS architectures, and RAMem performed better on average than the closest baseline LatentMem.

Critical strengths
  1. The research motivation is reasonable and the method design is complete. Different agent roles require different historical experiences, and introducing role awareness into retrieval and storage has clear practical significance. Role Signed Retrieval, Hidden State Composition, and reward based training form a relatively complete Memory framework.

  2. The experimental results are strong. RAMem scores 3.12, 5.88, and 4.15 points higher than LatentMem on AutoGen, MacNet, and CAMEL, respectively. The author conducted a comprehensive ablation, including Role specific retrieval, Hidden State input, and HMAPO, and tested unseen architecture and different model scales to enhance the persuasiveness of the experiment.

Critical weaknesses
  1. The core innovation and independent contributions of each component are not yet clear enough. RAMem is highly correlated with LatentMem, and its main improvement is to extend Role awareness to retrieval and storage, and use Hidden States instead of text input. Role Signed Retrieval is relatively straightforward, while HMAPO is mainly based on existing GRPO/DAPO methods, and its independent algorithm innovation is limited. Although Table 2 and Appendix C have provided some matched comparisons, there is still a lack of key controls, such as Full Trajectory Hidden States and a comparison between contextualized hidden states and response only encoded hidden states. Therefore, the independent contributions of Role Local Selection and Hidden State Representation have not been fully distinguished, and the core hypothesis of Hidden States retaining additional source content information has not been directly verified.

  2. The actual mechanism for performance improvement has not been fully validated. Appendix C shows that in the PopQA experiment of Qwen3-1.7B, RAMem increased the search ratio of agents from 14.2% -28.4% of text memory baselines to 93.4% -99.6%. This suggests that performance improvement may mainly come from behavioral steering, such as encouraging agents to actively search and verify, rather than necessarily reusing specific historical experiences more effectively. Suggest adding shuffled/irrelevant memory, empty memory with trained composer, and simple Search First prompting baselines to distinguish between Historical Knowledge Transfer, Procedural Guidance, and Memory Independent Behavioral Steering.

  3. The evidence for actual efficiency and generalization ability is still insufficient. Although RAMem ultimately only provides 8 latest tokens to the agent, the system still needs to store the complete token level hidden states and perform additional retrieval, LLM reranking, and Composer inference. Table 3 mainly reports the token consumption of the agent, but lacks complete end-to-end latency, storage footprint, and computation cost analysis, so its overall efficiency advantage cannot be confirmed yet. In addition, the main cross architecture experiments use the same Qwen3-4B backbone, and transfer benchmarks also allow for online write backs. Although the author provided supplementary experiments, there is still a lack of sufficient validation for cross model Hidden State Transfer and practical scenarios that do not rely on reliable success verifiers.

Questions for authors
  1. Can you provide factory ablation for Full/Mole Local Trajectory × Text/Hidden States and control retrieval, Composer capacity, and training budget to more clearly isolate the contributions of core components?

  2. To what extent does the performance improvement of RAMem come from actual retrieved historical experience rather than general behavioral strategies formed through training? Can it be verified through Shuffled Memory and Search First baselines?

  3. After taking into account Hidden State Storage, Composer inference, Retrieval, and Tool Calls, what is the complete accuracy cost trade-off of RAMem compared to LatentMem?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.