acceptodds
Under review as a conference paper at ICLR 2027

Rosetta Memory: Adaptive Memory for Cross-LLM Agents

Abstract

Memory is the key component for transforming a stateless LLM into a persistent, evolving agent through experience accumulation, long-horizon planning, and continual self-improvement. Existing memory systems typically take the LLM as the center and design memory operations tailored to a specific backbone. In practice, however, users frequently switch between LLMs, for example using Claude for coding and GPT for writing across tasks, or routing different steps to different backbones within a single task for cost-effective trade-offs. As a result, memory written by one model often needs to be consumed by another. Making upstream memory effectively adapt to and activate downstream LLMs remains a critical yet underexplored problem. To bridge this gap, we shift the perspective from LLM-centric memory design to memory-centric LLM adaptation. Specifically, we approach the above upstream-downstream memory adaptation problem from both the write and read sides, and design two profile-conditioned operators that are jointly trained to optimize how memory is stored and presented for better task completion. To ensure the learned operators generalize across a broad set of LLMs, we propose a minimum-gain sampling curriculum that prioritizes the least-served LLMs during training. To better measure the operators' actual contribution rather than the LLM's own capability, we design a performance-gap reward that compares against a naive memory baseline. Experiments on HotpotQA, 2WikiMultihopQA, and MuSiQue demonstrate that our model consistently outperforms baselines and remains robust under unseen-model replacement.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Memory, Cross-Model Memory Transfer, Memory Adaptation, Heterogeneous LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gu9pOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to adapt memory to different LLMs. The authors approach the problem from both sides of memory — writing and reading — design the corresponding operators (RoMem), and implement them by training google/flan-t5-small. Experimental results show that RoMem delivers consistent improvements over several methods on HotpotQA, 2Wiki-MultihopQA, and MuSiQue. Moreover, RoMem retains a reasonable degree of adaptation robustness on models whose data was not seen during training.

Critical strengths
  1. The paper is well organized and readable. Researchers working on LLM memory can take away a useful perspective from it: considering cross-model memory adaptation through the lens of memory reads and writes.
Critical weaknesses
  1. My biggest concern is the emphasis on "Agent" in the title. In fact, the whole study is still carried out on specific base models. Over the past six months, the view that an agent = base model + harness (plus other components, e.g., function calling) has been emerging as a new consensus. If the authors wish to foreground agent-side memory, should they not consider whether the different harnesses themselves have already optimized what this paper studies? As written, no such comparison is taken into account.
Questions for authors

While reading the paper, the following questions caused me confusion:

  1. Could the experiment in Figure 1 be affected by capability differences among the models beyond architecture, or by the ordering of the models? Is it possible that the weaker models introduce noise that throws the overall picture off balance?
  2. Why is HotpotQA not included in the comparison in Figure 2?
  3. Section 5.3 states that profiles are an important component of RoMem. Should the earlier text (e.g., the preliminaries and the method section) give a more detailed account of them?
1 comment

Although the authors’ response came later in the discussion period, my overall impression of the work remains positive. I found the paper clearly written, and I encourage the authors to continue refining the work.

Reviewer_EA8POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies cross-LLM memory mismatch when heterogeneous LLM agents share memory. Even on the same task, different LLMs may produce different actions, writing styles, evidence orderings, and memory formats. Conversely, semantically correct memories may lead to different outcomes when consumed by target LLMs with different contextual preferences. Rosetta Memory decomposes this issue into write-side and read-side mismatch and introduces a source-profile-conditioned writer f_write and a target-profile-conditioned reader f_read. The two operators use separate FLAN-T5-small backbones conditioned by profile soft prefixes constructed from model identity, natural-language descriptions, and metadata. The writer normalizes source-model observation-action traces into shared memory, while the reader converts the shared memory into prompt-ready context tailored to the target LLM. Training follows an Expert Iteration-style procedure with mixed-LLM rollouts: trajectories are ranked by terminal reward and the top-α fraction is retained, while a counterfactual gain ΔR drives minimum-gain sampling so that low-gain and behaviorally diverse LLMs receive more rollout exposure. Mean results on HotpotQA, 2WikiMultihopQA, and MuSiQue show that RoMem outperforms several prompting, raw-buffer, and structured-memory baselines. The w/o f_write, w/o f_read, and w/o profiles ablations also support the relevance of the major components.

Critical strengths
  1. The paper separates write-side mismatch from read-side mismatch. The pilot study indicates that source models produce materially different memories and that semantically related presentations can elicit different answers from target models. Modeling the writer and reader as distinct sources of variation extends the usual formulation of agent memory.
  2. The experimental analysis includes w/o f_write, w/o f_read, and w/o profiles ablations, a minimum-gain coefficient sweep, held-out model replacement, writer-reader compatibility, worst-reader evaluation, and noisy-memory tests. Despite the small samples used in several diagnostics, these experiments provide information about component behavior and failure boundaries.
Critical weaknesses
  1. This approach requires gathering some experience prior to testing. The study does not evaluate a "single-pass" testing paradigm—where an agent processes tasks sequentially, receives feedback, stores the results in memory, and leverages that memory to assist with subsequent tasks. Instead, the final experiments in the paper rely on a "memory bank" that is pre-collected from the training set—a process that is computationally expensive. This methodology could potentially grant memory-based approaches an unfair advantage over memory-free methods, in terms of either resource utilization or access to information.
  2. The paper does not report performance on Code-Agent or code/software-development agent benchmarks. This makes it unclear whether cross-LLM memory adaptation improves code-agent task completion or remains beneficial when memory entries capture code, execution feedback, tool calls, and long-horizon debugging state.
Questions for authors
  1. Why not evaluate the complete evaluation set under a one-pass test-time augmentation protocol in which the agent processes one task at a time, receives feedback, writes the resulting memory, and uses that memory for the next task? The current terminal experiments pre-collect a sufficiently large memory bank from the training split at substantial cost before formal evaluation. Does this create an unfair resource or information advantage over no-memory methods, and how do the results change under a matched online budget?
  2. Can the authors report Code-Agent performance? In particular, how does RoMem improve code-agent task success or code quality relative to no-memory and raw-memory baselines when memory is written from code, execution feedback, tool results, or debugging state?
Reviewer_fEgrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Rosetta Memory, a framework for adapting memory across heterogeneous LLMs. It trains profile-conditioned write and read operators using reward-filtered mixed-model trajectories, together with a minimum-gain sampling strategy to prioritize underserved models. Experiments on three multi-hop QA datasets show improvements over several prompting and memory baselines, with additional ablations and held-out model replacement evaluations.

Critical strengths

For the strengths,

  • The problem of transferring memory across heterogeneous LLMs is practically relevant, particularly for agents that dynamically switch between different backbones.

  • The paper reports consistent improvements on three multi-hop QA datasets. The component ablations and held-out model replacement experiments provide some evidence that model-aware memory adaptation can be useful.

Critical weaknesses

For the weaknesses,

  • Limited methodological novelty and insufficient comparisons with recent memory-training methods. The proposed framework mainly combines learned write/read operators, profile conditioning, reward-filtered imitation, and heuristic model sampling. Several of these ideas have already been explored in prior work on learned memory management and utilization, such as Mem-builder and Memory-R1/R2/T1. The minimum-gain sampling strategy is also not clearly connected to the stated worst-assignment objective, and its contribution is not isolated from diversity-based or uniform sampling. More importantly, the experiments mainly compare against prompting and non-trained memory baselines, without including representative learned memory methods or an equally trained model-agnostic write/read baseline. It is therefore difficult to determine whether the improvements come from the proposed cross-model adaptation or simply from additional task-specific training.

  • The evaluation is too limited to support the broader claims about memory adaptation. All three datasets are converted into short, fixed-length multi-hop QA episodes, with only 159, 67, and 62 test examples, respectively. This setting primarily evaluates short-term evidence accumulation rather than persistent memory management. Recent memory research has introduced more comprehensive benchmarks, such as LoCoMo, LongMemEval, and MemoryAgentBench, covering longer interaction histories, memory updates, and cross-session retrieval. While these benchmarks would need to be adapted to the cross-model setting, evaluation on more realistic memory tasks would substantially strengthen the paper. The relatively small test sets and lack of uncertainty estimates also make it difficult to assess the reliability of the reported improvements.

Questions for authors

For the questions,

  • How does RoMem compare with recent trained memory methods, under a comparable setup? Can the authors include an equally trained write/read baseline without model-specific profiles to better isolate the contribution of cross-model adaptation?

  • What is the specific contribution of minimum-gain sampling compared with uniform or diversity-only sampling under the same training budget? How does the proposed sampler approximate the worst-assignment objective in Eq. (5), given that the gain in Eq. (7) is measured using homogeneous model assignments?

  • Can the authors evaluate RoMem on longer-horizon or cross-session memory tasks, or provide uncertainty estimates for the existing results? This would help establish whether the improvements generalize beyond short multi-hop QA episodes.

Please also refer the weakness part.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.