acceptodds
Under review as a conference paper at ICLR 2027

SaliMory: Orchestrating Cognitive Memory for Conversational Agents

Abstract

Recall and personalization, enabled by memory, are pivotal to a lifelong conversational agent. However, simply expanding context windows with raw retrieval degrades reasoning quality, while training memory agents via standard reinforcement learning creates a severe credit assignment bottleneck in a multi-stage pipeline. To solve this, we introduce SaliMory, a framework that learns to manage and utilize a cognitively-structured memory—spanning user facts, preferences, and working memory. By introducing novel stage-wise process rewards and reward-decomposed contrastive refinement, SaliMory provides isolated supervision for distinct memory operations (selective filtering, consolidation, and cue-driven recall) end-to-end. SaliMory cuts memory-attributed failures by one-third, outperforms the state-of-the-art by over 10% in end-to-end recall accuracy, and more than doubles the Good Personalization rate across 4 datasets.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Conversational AI, LLM Memory, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_DAQsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies user-memory management in long-term conversations and proposes the SaliMory framework. It organizes memory into factual memory, preference memory, and working memory, and uses the Saliency Filter, MemBooster, and MemUtilizer for memory filtering, updating, and query-time utilization. The paper combines stage-wise rewards, GRPO, and role-specific pairwise training, and reports promising results on LoCoMo, LoCoMo-P13n, LongMemEval, and internal data. However, the current results are not yet sufficient to support the paper’s central claims regarding role-level credit assignment and the effective isolation of optimization signals across different stages. The main concerns involve gradient attribution under a shared policy, the state consistency of contrastive examples, reward coverage, data splitting and evaluation reliability, and the fairness of the baselines and statistical reporting.

Critical strengths

(1) The problem is clearly important in practice and covers three key aspects of long-term memory: memory formation, maintenance, and utilization. The paper evaluates not only QA performance, but also personalization quality, memory factuality, vagueness, and utilization, which is more comprehensive than reporting final answer accuracy alone.

(2) Extensive experiments across multiple datasets, reward and contrastive-training ablations, efficiency analysis, and partial validation using an independent evaluator.

Critical weaknesses

(1) Role-level credit assignment is not sufficiently established by the current optimization objective.

The three roles use a shared policy (πθ\pi_\theta), while GRPO computes the group-relative advantage for each complete rollout based on the total reward: Rtotal = R1 + R2 + R3. Therefore, the current description does not explain why the main GRPO gradient would not simultaneously affect the Filter, Booster, and Utilizer. Nor does it demonstrate that the role-specific contrastive losses are sufficient to eliminate cross-role interference caused by shared parameters. In addition, the final (R2R_2) measures the quality of the complete memory bank, which is not equivalent to the local contribution of each intermediate memory-writing action.

(2) The contrastive pairs may not be constructed under matched conditions, weakening the causal-isolation argument.

Different Booster rollouts may produce different memory states at early steps, while the paper constructs winner/loser pairs based on the final R2. A better final result does not necessarily imply that every intermediate memory write was better. For the Utilizer, a Jaccard threshold of τsim=0.15 only guarantees a certain degree of set overlap. It does not guarantee that the key facts, item ordering, prompt length, or context conditions are identical. Consequently, the difference in utilization quality may still be caused by upstream memory differences rather than by the Utilizer itself.

In addition, the current objective is more accurately described as a weighted DPO-style pairwise logistic loss; the paper does not show the reference-policy log-ratio term used in the standard DPO formulation.

(3) The data split and personalization-label definitions do not sufficiently rule out leakage and evaluation bias.

The paper reports 1,108 additional questions and a 60/20/20 split, but does not clearly explain whether the split is performed by user, conversation, session, or QA pair. If the same user or conversation history appears across different splits, the reported test results may overestimate generalization to unseen users.

The generation process for the additional questions, the proportion subjected to human review, and the quality-control procedure are also insufficiently transparent. Furthermore, whether Underuse and NA are determined using a shared original-history reference or each method’s own memory bank directly affects the fairness of the comparison.

(4) LLM-based evaluation, statistical robustness, and baseline fairness remain insufficient.

GPT-4o is used as the primary judge, while another judge is used only for cross-validation on a limited subset. Category-level confusion patterns, the number of human-evaluated examples, the sampling procedure, and confidence intervals are not sufficiently reported. The paper also mainly reports point estimates and lacks results across multiple random seeds, user- or conversation-level bootstrap estimates, and paired significance tests.

In addition, the policy model, frozen generator, judge, prompts, token budget, memory budget, and serving environment may not be consistent across methods. Therefore, the reported gains may not be attributable solely to the memory architecture.

Questions for authors

(1) The authors should clearly specify parameter sharing, token-level loss masks, gradient routing, and the update scope of each objective. They should also conduct ablations comparing global GRPO, role-specific rewards without contrastive training, role-specific contrastive training, and the full method, in order to identify the source of the reported gains.

(2) The authors should report the construction procedure and acceptance rate of the preference pairs. They should also add controlled experiments with fixed preceding states, shared prefixes, fixed memory contents, and different similarity thresholds, together with an ablation without the similarity gate.

(3) The authors should clearly specify the split granularity, user-level isolation protocol, memory-reset rules, and human-review procedure. They should also use a method-independent personalization-opportunity reference.

(4) The authors should report baselines using matched generators and matched computational budgets, provide the complete training and inference costs, and clearly distinguish official implementations from author reimplementations.

(5) QA is not the highest for SaliMory on every LongMemEval subset; the paper should avoid claiming universal superiority. In the R1-only ablation, Vague is 28.3 while Bad Personalization is 4.5; these values should not be conflated. Public-benchmark and internal-traffic Vague/Factuality values differ substantially and should be discussed separately. The semantics of Apply, memory insertion/deletion/merging, conflict handling, capacity limits, pair filtering, and the relationship between β and temperature η need to be specified for reproducibility.

5 comments

Thank you for the encouraging assessment. We address each point below.

In Responding to Weakness and Questions

W1 / Q1. Role-level credit assignment under a shared policy.

In SaliMory, memory is not optimized in isolation. Following the cognitive view that memory has value only through use, the design enables a loop: good memory leads to better utilization and better downstream responses, and downstream performance in turn guides what to store and how to use it. The roles are intentionally coupled through the downstream task, and separately supervised where attribution is needed. Memory formation, maintenance, and utilization matter only through the final response, so the shared GRPO term ties all roles to that outcome. Each role additionally receives a signal that measures its own output.

  • Parameters: one policy π_θ (Qwen3.5-9B); the roles differ only by instruction prompt. There are no role-specific parameters.
  • GRPO term: clipped surrogate + KL over all traces of a rollout, with the rollout's group-normalized advantage from R_total. This term updates every role by design.
  • Role-specific signals: filter — saliency reward against ground-truth labels (recall-weighted); booster — contrastive loss over booster traces only, pairs chosen by R2; utilizer — contrastive loss over utilizer traces only, pairs chosen by the utilization score. Each contrastive loss masks all tokens except its role's outputs.
LoCoMoQA ↑Good Pers. ↑
Global GRPO (Table 2 in the paper)68.626.8
Role-specific advantages, no contrastive70.432.1
Global GRPO + booster contrastive (Table 2 in the paper)69.129.5
Full SaliMory72.939.8

Role-specific-advantages row: new experiment. Other rows: from Table 2 in the paper.

On R2 being bank-level: booster pairs are formed per turn. For each utterance, the outputs of the R2-best and R2-worst rollouts are paired, and the loss is weighted by their R2 gap. Turns where both rollouts wrote the same output contribute nothing, so the update concentrates on the write decisions that differ. We will add this specification to §3.4.

W2 / Q2. Matched conditions; τ_sim; DPO terminology.

Why decompose the winner selection (illustrative episode, G = 4).

RolloutR1 (response)R2 (memory)R3 (utilization)Total
g01.000.080.121.20
g11.000.020.141.16
g20.00−0.150.05−0.10
g31.45−0.100.081.43

Illustrative numbers to explain the selection rule, not measured results.

Selecting by total reward would pair g3 against g2 and teach the booster g3's vague memories, even though g3 won only because the frozen generator answered well. Instead, SaliMory selects by each role's own score: the booster pairs g0 vs. g2 (gap 0.23), and the utilizer pairs g1 vs. g2 only if their banks overlap (Jaccard > τ_sim).

Matched conditions. Booster pairs share the episode, the conversation, and the utterance, and the loser's output is re-scored under the winner's prompt, so both writes are compared on the same input. Utilizer pairs share the question and conversation, and are kept only when the two memory banks overlap. Low-overlap pairs are dropped, and the utilizer still receives its continuous utilization reward through GRPO.

Pair statistics (per training step)Value
Booster: utterances matched between winner and loser412
Booster: pairs formed (outputs differ)251 (61%)
Booster: mean R2 gap0.31
Utilizer: candidate pairs passing τ_sim = 0.1538%
Utilizer: mean Jaccard of accepted pairs0.42

New: pair statistics logged from our training runs.

τ_simPair acceptanceQA ↑Good Pers. ↑
No gate100%70.835.4
0.15 (submitted)38%72.939.8
0.3026%72.238.7
0.5017%71.536.9

New τ_sim ablation. The 0.15 row is the submitted setting, from Table 1 in the paper.

Terminology. We agree: the objective is a gap-weighted pairwise logistic loss (DPO-style, without the reference log-ratio). The reference policy enters through the GRPO KL term (coefficient 0.08). We will rename it. η in Table 7 of the paper is the same parameter as β in Eqs. 4 and 6 (β = η = 0.15, scaling the pairwise logit); we will use a single symbol. Rollouts are sampled at temperature 0.5, which we will add to Table 7. For reproducibility, we also released the code: https://anonymous.4open.science/r/SaliMory-4D59.

W3 / Q3. Split, LoCoMo-P13n construction, Underuse/NA reference.

Split granularity and LongMemEval's role. The split is over episodes, so questions about the same LoCoMo history can appear in both train and test. This does not bias our results. (i) Questions over the same history span different types (single-hop, multi-hop, temporal, open-domain, recommendation, implicit personalization) that draw on different evidence and require different reasoning, and memory is rebuilt from scratch in every episode, so no answer or memory carries over from training. (ii) LongMemEval-s gives every question its own user history, so no test history is seen in training; it is trained and evaluated separately, with no cross-training, and SaliMory's gains hold there (69.1 QA / 20.8 Good, Table 8 in the paper). (iii) On a new split of LoCoMo by conversation, where test users and histories are entirely unseen, SaliMory's gain over the untrained policy holds (+10.8 QA / +18.2 Good). Hyperparameters and checkpoints are selected on the validation partition only.

The 1,108 Cat 6–7 pairs were generated by the 4-step semi-automatic pipeline of Appendix A.1 in the paper via Llama3.3-70b-instruct. 100% were manually reviewed by a linguistic expert, who rejected or corrected pairs not supported by the conversation. Good Personalization is computed over a fixed question set, so its denominator does not depend on what is stored and a method cannot raise it by storing less: missing memories turn into NA or Underuse and lower Good. Memory is reset between episodes for every method. We will add these details to Appendix A.1.

W4 / Q4. Statistics, judges, matched budgets, cost.

The gains come from the training algorithm, not only the architecture: with the same architecture, prompts, generator, and budget, the full method improves over GRPO-only by +4.3 QA / +13.0 Good (Table 2 in the paper), and over our new untrained-policy baseline (same architecture, untrained Qwen3.5-9B) by +12.5 QA / +21.2 Good (72.9 vs. 60.4; 39.8 vs. 18.6).

Comparison (LoCoMo)Δ QA95% CIΔ Good95% CI
SaliMory vs. Mem-R1+10.2
+7.9,+12.6+7.9, +12.6

| +23.5 |

+20.1,+26.8+20.1, +26.8

| | SaliMory vs. GRPO only | +4.3 |

+2.1,+6.4+2.1, +6.4

| +13.0 |

+10.2,+15.9+10.2, +15.9

| | SaliMory vs. Zero-shot Agentic | +2.0 |

−0.5,+4.5−0.5, +4.5

| +33.2 |

+29.8,+36.5+29.8, +36.5

|

New analysis: bootstrap 95% CIs over test episodes. The Δ values are computed from Tables 1 and 2 in the paper.

Judges. Appendix A.3 in the paper reports human agreement for the final-response judge, and Table 5 in the paper shows identical rankings under a second judge, Claude 3.5 Sonnet (ρ = 1.00). We have now also validated the intermediate judges (new analysis): across judge families (GPT-4o vs. Gemini 3.1 Pro), response-level signals agree with κ = 0.62–0.72 and utilization with r = 0.70; against human ratings, utilization reaches r = 0.78 and the memory-quality signals r = 0.61–0.66 . Budgets. All baselines in Table 1 of the paper are run with the same frozen generator, judge, and prompts. The one exception is Zero-shot Agentic, which uses Gemini-3-Flash for memory operations; we will flag this in the table caption.

MethodTraining computeInference tokens / episodeLatency / episode
Mem-R1Not reported (4× H100, two RL agents), est 3.4 hrs per epoch12~18knot reported per episode. Per-call p50 only (Appendix G in the paper): manager ≈2.0s, answer 0.34s
Zero-shot Agentic—12~18k38.7s (38s formation + 0.69s/query,including network latency)
SaliMory8× A100, 1.14 s per episode (Table 3 in the paper)8~10k0.76s (0.5s formation + 0.26s/query)

New: inference tokens measured in our setup. SaliMory training cost and all latencies: Table 3 in the paper. Mem-R1 entries: from the Mem-R1 paper and our estimate.

Q5. Claim scoping and specification.

We will make all of these changes: (i) scope the QA claim to “best QA among trained memory methods, on par with a frontier zero-shot agent” and fix the bold formatting in Table 8 in the paper; (ii) separate the R1-only ablation's Vague rate (28.3) from Bad Personalization (4.5); (iii) discuss public-benchmark and internal-traffic Vague/Factuality separately; (iv) specify Apply semantics (insertion, update, conflict handling, capacity limits), pair filtering, and a single symbol for β/η in §3.2–3.4 and Appendix A.6.

We sincerely thank you for a precise set of questions; addressing them has tightened the paper. Looking forward to your responses!

Thanks for the detailed response.

The explanation on shared GRPO update, new ablations, and the conversation-disjoint LoCoMo experiments are helpful to address my concerns.

However, my remaining concern lies in whether the preference pairs can isolate each role’s contribution. For the Booster, a higher final memory-bank score may reflect better writes at other turns, so it is unclear whether the preferred write at the paired turn is actually better or not. For the Utilizer, a higher score may come from the access to more relevant facts or from the use of same facts. In both cases, the comparison leaves open issue whether the preferred output earned its higher score through the role’s own decision.

Additionally, when deciding whether there was an opportunity to personalize the answer, does the judge use the original conversation history, or the memories each method actually stored?

Reviewer_xTuXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how an AI agent can retain truly useful information during long-term conversations and appropriately utilize it when generating responses. The core approach involves defining three distinct roles for memory management and retrieval, and employing reinforcement learning to train a memory management model that effectively coordinates these roles. Each component is optimized using specific training signals tailored to its respective function. The proposed method demonstrates strong performance on datasets such as LoCoMo.

Critical strengths
  1. Structured memory and its management design: The paper proposes a complementary memory structure that distinguishes between user facts, long-term preferences, and short-term working memory, while defining three distinct roles: information filtering, memory updating, and memory utilization. I like this elegant and coherent design.

  2. Fine-grained reward designs: Reward Beyond training based on the quality of the final response, the paper incorporates fine-grained process rewards and contrastive learning signals tailored to each role. This provides the model with more direct intermediate supervision, helping to mitigate the credit assignment challenge inherent in multi-stage memory management.

  3. Strong Performance: The proposed method achieves strong performance across several long-conversation memory datasets, with notable excellence in personalized response generation and memory quality. Ablation studies further validate the effectiveness of the individual reward mechanisms and contrastive learning components.

Critical weaknesses

1. Limitations in Reward Design. The "Memory Quality" reward primarily evaluates faithfulness and specificity, without explicitly penalizing the omission of important information. Under the definitions provided in the appendix, an empty memory could theoretically receive the maximum reward. Since Booster's contrastive learning objective relies entirely on preferences derived from this reward, it may inadvertently encourage the model to discard useful information.

2. Lack of Comparison with Direct Role-Specific Advantages. Given that the paper already defines separate rewards for different roles, a simpler alternative would be to compute role-specific GRPO advantages directly, without introducing additional contrastive losses. However, the paper does not compare against this approach, making it difficult to determine whether contrastive learning provides benefits beyond direct role-specific RL supervision.

3. Potential Conflicts Between Optimization Objectives. The role-specific contrastive objectives may not always align with the global GRPO objective. For example, removing information could improve the "Memory Quality" reward while negatively affecting downstream response quality. The paper does not investigate potential conflicts between these objectives through gradient alignment analysis or sensitivity studies on loss weights, leaving it unclear how they interact during optimization.

4. Insufficient Validation of Intermediate-Stage Evaluation and Credit Assignment. The paper relies on an LLM judge to assess memory quality and utility, as well as to attribute errors to intermediate stages. However, the human evaluation and cross-model verification primarily focus on the final responses, rather than directly validating the accuracy of intermediate rewards and error attribution. This raises concerns about the reliability of the proposed process supervision and credit assignment mechanisms.

5. Limited Novelty and Missing Key Baselines. Memory management and step-level GRPO for memory optimization have already been explored in prior work. Although the paper compares against Memory-R1, Mem-α, and Mem1, it omits several closely related methods, including MemBuilder, MemCoE, and DeltaMem. In particular, MemBuilder incorporates multi-type memory, dense rewards, and contribution-aware gradient weighting, while MemCoE employs structured process rewards to optimize memory updates. Without comparisons against these methods, it remains unclear whether the proposed process reward and credit assignment mechanisms offer meaningful improvements over existing approaches.

Questions for authors

Please refer to the weaknesses section. Meanwhile, here's the additional question I'd like to ask:

  1. For the memory structure and the management roles, why did the author decide on such designs? Are they comprehensive enough, and also concise enough without unnecessary designs (for example, why not unify the filtering process into the update "no-op" operator)?

I'll raise my score if the authors can justify the weakness and answer my questions.

4 comments

Thank you for the encouraging assessment. We address each point below.

In Responding to Weakness

W1. R2 has no coverage term; risk of excessive pruning.

Thanks for bringing this out! In SaliMory, memory is not optimized in isolation. Following the cognitive view that memory has value only through use, the design is a loop: good memory leads to better utilization and better downstream responses, and downstream performance in turn guides what to store and how to use it. The reward implements this loop:

  • R2 scores the memory state produced in every rollout, i.e. the intermediate state of the pipeline, not a standalone bank. We will clarify this in Section 3.3.
  • The same rollout is also scored by R1 on the downstream answer and by the utilization score. R2 spans 0.8 (−0.5 to +0.3), while R1 spans 1.45 (0 to 1.45; Eqs. 1–2 and Table 7 in the paper). Even an empty bank gains at most +0.3 in R2 but can lose up to 1.45 in R1, so a pruned bank that loses answers is strongly penalized in the total reward.
  • The filter's saliency reward is recall-weighted (w_r = 0.6), which directly penalizes dropping salient turns.

Empirically, on the LoCoMo-P13n test set, the booster contrastive prevents pruning instead of causing it:

ModelEntries / bank ↑Empty banks ↓Utility ↑Good Pers. ↑
Untrained policy (same architecture)5.14.1%0.71218.6
GRPO only (Table 2 in the paper)3.612.5%0.79326.8
SaliMory7.90%0.91739.8

New analysis. Entries per bank and empty banks are new for all rows; Utility and Good Pers. are from Table 2 in the paper for GRPO only and SaliMory, and from our new untrained-policy run for the untrained row.

Without contrastive training, GRPO alone shrinks the bank and leaves some conversations with an empty bank, which is the failure the reviewer anticipates. With the booster contrastive, the bank grows, no conversation ends with an empty bank, and utility and Good Personalization rise. We will add this analysis and state R2's role explicitly in the revision.

W2. Why not role-specific advantages directly?

Thanks for raising this interesting question. We ran this ablation: GRPO with a separate group-normalized advantage per role (filter: R1 + saliency; booster: R1 + R2; utilizer: R1 + utilization), with no contrastive loss.

LoCoMoQA ↑Good ↑Vague ↓Factuality ↑Utility ↑
GRPO, global advantage (Table 2 in the paper)68.626.831.790.90.793
GRPO, role-specific advantages70.432.124.892.10.861
SaliMory (global GRPO + role contrastive)72.939.819.693.90.917

Role-specific-advantages row: new experiment. Other rows: from Tables 1 and 2 in the paper.

Two reasons the contrastive objective carries information a role-specific advantage cannot: (i) a role-specific advantage is still one scalar shared by every action of that role in the episode, whereas booster pairs compare the two rollouts' outputs on the same utterance, so the update targets the writes that actually differ; (ii) an advantage only re-weights what was sampled, whereas the pairwise loss pulls the policy toward a specific better output, weighted by the reward gap.

W3. Conflicts between objectives; sensitivity.

The downstream objective sets the direction: R1 dominates the total reward, and the contrastive weights are small (λ_b = 0.15, λ_u = 0.1). The specific conflict raised (better memory scores by omission vs. downstream quality) does not occur in practice: the contrastive term increases bank size (3.6 → 7.9 entries per bank vs. GRPO-only) and utility (0.793 → 0.917).

We ran two new analyses to test the conflict directly. Here, booster contrastive is the pairwise loss on the booster's memory-update outputs, and utilizer contrastive is the pairwise loss on the utilizer's profile-synthesis outputs. (i) Gradient alignment: both contrastive gradients point in the same direction as the GRPO (downstream) gradient in most steps (positive mean cosine), so they do not pull against the downstream objective.

Gradient alignment (GRPO vs. contrastive)Mean cosineSteps with cosine > 0
Booster contrastive0.2178%
Utilizer contrastive0.1771%

(ii) Sensitivity: halving or doubling either contrastive weight changes QA by at most 1.3 and Good Personalization by at most 3.6, so the results do not hinge on the weights. The default row is from Table 1 in the paper; the other rows are new runs.

Loss weightsQA ↑Good Pers. ↑
λ_b × 0.571.636.2
λ_b × 272.138.5
λ_u × 0.572.337.6
λ_u × 271.838.9
Default (λ_b = 0.15, λ_u = 0.1)72.939.8

W4. Validation of intermediate judges.

We validated the intermediate signals against humans and against a second judge model from a different family:

SignalHuman vs. judgegpt-4o vs. gemini-3.1-pro
Response correctness (R1)agreement 0.91, κ = 0.81agreement 0.87, κ = 0.72
Personalization (R1)agreement 0.91, κ = 0.71 (Appendix A.3)agreement 0.87, κ = 0.62
Utilization (R3)Pearson r = 0.78Pearson r = 0.70
Memory vagueness (R2)Pearson r = 0.66Pearson r = 0.43
Memory factuality (R2)Pearson r = 0.61Pearson r = 0.22

New analysis of the intermediate judges. The human personalization result is from Appendix A.3 in the paper (LoCoMo-P13n); all other cells are new.

All main claims rest on response-level signals, and these agree strongly with human ratings (κ = 0.71–0.81) and across judge families (κ = 0.62–0.72). Utilization also agrees well (r = 0.70–0.78). The memory-quality signals (R2) agree less across LLM judges (r = 0.22–0.43) but track human ratings (r = 0.61–0.66). This is expected given R2's role: it is a small dense shaping term (range 0.8 vs. 1.45 for R1), so judge noise in R2 cannot dominate training, and no headline result relies on R2 alone. We will report these agreements and the human study in the appendix.

W5. Memory RL is not new; MemBuilder, MemCoE, DeltaMem.

Thank you for these references. We agree memory RL is an active line of work, and §1–2 already discuss it. Our contribution is not RL for memory per se. Grounded in cognitive science, we propose (i) a structured memory of facts, preferences, and working memory managed by distinct filter, booster, and utilizer roles, and (ii) on top of standard memory RL, stage-wise process rewards plus reward-decomposed contrastive refinement that gives each role its own pairwise signal. We will discuss MemBuilder, MemCoE, and DeltaMem in §2 and include them as baselines in the revision. We use their open-sourced code with the same experimental settings as SaliMory to conduct the experiments over LoCoMo, and present the results as below:

LoCoMo / LoCoMo-P13nQA ↑Good Pers. ↑
MemBuilder65.318.9
MemCoE53.89.2
DeltaMem62.112.4
SaliMory72.939.8

**Reference**

- https://github.com/Zh1yuShen/MemBuilder

- https://github.com/Applied-Machine-Learning-Lab/ACL2026_MemCoE

- https://github.com/declare-lab/delta-Mem

Q1. Why this memory structure and these roles? Why not fold filtering into the "no-op" update?

The roles act on different inputs, at different times, under different signals. The filter is a streaming per-turn decision over raw dialogue (most turns never become memory candidates), scored by recall-weighted saliency. The booster's no-op acts later, on candidate updates to an existing bank, scored by R2. Folding filtering into the no-op would (i) push every raw turn through the more expensive update prompt and (ii) merge two signals (saliency vs. bank quality) into one operator, which reintroduces the credit-assignment ambiguity the design reduces.

The stores serve different functions: working memory drives QA (66.2%), and the preference bank drives personalization (22.7%) (Figure 3 in the paper). The utilizer's synthesis across them raises Good Personalization from 28.6% to 39.8% compared with raw concatenation.

We hope this resolves your concerns, please do let us know if you have further questions! Thanks again for your reviews and looking forward to your response!

Thanks for the point-to-point rebuttal. The newly added baselines and ablation studies have partially addressed my concerns. I still have the following concerns:

  1. This method relies heavily on an LLM Judge to determine which memory update is superior. However, the cross-model consistency is generally not high enough, with a minimum value of only 0.22. If the ranking itself is unreliable, the contrastive signal learned from it may also be incorrect.

  2. The new experiments actually suggest that using role-specific GRPO alone is sufficient to achieve strong results, which leaves me unconvinced regarding the necessity of incorporating such complex contrastive learning design with more hyperparameters. Furthermore, given a large party of the existing works is working on reward designs on memory RL, the overall novelty of this approach appears limited.

  3. The revisions included a lot of changes, but they felt more like allowing the AI to make arbitrary defensive additions rather than the authors making necessary and reasonable revisions to address specific problems. There is even an instance on page 17 where text and images overlap, which suggests a lack of human supervision in the revision process.

Therefore, I am inclined to maintain my original rating.

AuthorsOpenReview ↗

Response to the follow-up

Thank you for the follow-up and for acknowledging the new baselines and ablations. We address the remaining concerns below.

1. Reliability of the judge-based ranking.

The minimum value of 0.22 is one cell: cross-LLM agreement on memory factuality. It does not describe the signals that drive the ranking or the results:

  • The judge used in training tracks humans on the memory-quality signals (r = 0.61–0.66). The lower cross-LLM value reflects disagreement between two LLM judges, not disagreement with humans.
  • The contrastive signal is a ranking, not an absolute score, and it is gap-weighted. A pair contributes in proportion to the score gap between winner and loser, so close calls, where judge noise matters most, carry little weight. Noise that does not systematically favor one output averages out over the 251 pairs per step, while consistent quality differences accumulate.
  • Utilizer pairs are ranked by utilization, which agrees well both across judges (r = 0.70) and with humans (r = 0.78).
  • R1 dominates the total reward (range 1.45 vs. 0.8 for R2), and the contrastive weights are small (λ_b = 0.15, λ_u = 0.1), so judge noise in R2 cannot steer training.

Most importantly, the outcome is verified independently of the training judge. The final results are scored on response-level signals that agree with humans (κ = 0.71–0.81) and across judge families (κ = 0.62–0.72). Under a second judge, Claude 3.5 Sonnet, the method ranking is identical (ρ = 1.00, Table 5 in the paper). If the contrastive signal were systematically wrong, these independently judged outcomes would not improve. Instead, SaliMory improves significantly over GRPO-only (+4.3 QA, 95% CI [+2.1, +6.4]; +13.0 Good, [+10.2, +15.9]).

2. Is the contrastive design necessary beyond role-specific GRPO?

Role-specific GRPO is a strong baseline that we added at your request, and SaliMory still improves on it by +2.5 QA and +7.7 Good Personalization (72.9 / 39.8 vs. 70.4 / 32.1, a 24% relative gain in Good). It also lowers the Vague rate (24.8 → 19.6) and raises utility (0.861 → 0.917). This is the largest single gain over that baseline among all components, and it follows from what the pairwise loss provides that an advantage cannot: a role-specific advantage is still one scalar shared by all of a role's actions in an episode, whereas a pair compares two outputs on the same input and pulls the policy toward the better one.

The added hyperparameters do not make the method fragile. In our sensitivity runs, every setting tested still outperforms role-specific GRPO:

  • halving or doubling either contrastive weight: lowest 71.6 / 36.2;
  • every τ_sim from no gate to 0.50: lowest 70.8 / 35.4.

The contrastive term adds no sampling and no judge calls: pairs are drawn from the same G rollouts and the same scores GRPO already computes.

Novelty. We agree reward design for memory RL is an active area. Our contribution is not a new reward term but three things existing work does not provide:

  1. a cognitively grounded memory structure (facts, preferences, working memory) managed by three distinct roles;
  2. reward-decomposed contrastive refinement, in which each role is paired and weighted by its own score. This is a different mechanism from reward shaping, as the role-specific-advantage ablation shows;
  3. LoCoMo-P13n and a two-level evaluation protocol that scores both responses and intermediate memory quality, with 1,108 new, fully human-reviewed personalization queries.

Against recent memory-RL methods run under our setup (same generator, judge, data, and prompts), SaliMory leads the strongest, MemBuilder, by +7.6 QA and +20.9 Good Personalization (72.9 / 39.8 vs. 65.3 / 18.9).

3. On the revision.

Each is a requested experiment or specification, not a general addition. We have the AI used for polish the text but all the additions are drafted by us. The text–figure overlap on page 17 is a LaTeX float-placement artifact from inserting the new appendix tables, and it has been fixed. I have updated the revision, thanks for pointing the rushy layout out.

We hope this clarifies the remaining points, and we thank you again for the careful review.

Reviewer_LKzQOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

SaliMory addresses memory management for long-term conversational personalization. It organizes user information into factual, preference, and working-memory stores. A shared Qwen3.5-9B policy performs saliency filtering, memory updates, and query-conditioned profile synthesis. Training combines GRPO with stage-wise rewards and role-specific contrastive losses. The default response generator is a frozen Qwen3-235B-A22B model; the 9B designation therefore describes the memory policy rather than the complete system.

Evaluation covers LoCoMo/LoCoMo-P13n, LongMemEval variants, and an internal conversational dataset. The studied capabilities are conversational recall and personalized responses, with broader agent capabilities outside the demonstrated scope.

Critical strengths
  • Substantial personalization gains, supported by component ablations. On LoCoMo/LoCoMo-P13n, SaliMory achieves 39.8% Good Personalization, compared with 21.9% for A-Mem and 16.3% for Memory-R1 (Table 1). Adding both contrastive components improves Good Personalization from 26.8% under GRPO-only to 39.8%, while QA accuracy increases from 68.6% to 72.9% (Table 2). These results provide meaningful evidence that the proposed training additions improve performance within the reported setting.

  • Useful controls for generator capacity and evaluator dependence. The smaller-generator experiment retains an advantage over the included baselines (Table 10), and the alternative-judge evaluation preserves method rankings on a sampled subset (Table 5). These controls strengthen the empirical results and partially address concerns that the gains arise entirely from the large frozen generator or a particular evaluator.

Critical weaknesses

The experimental controls do not sufficiently establish generalization or isolate the contribution of training.

  • The reported 60/20/20 split does not specify whether partitioning occurs by question, conversation, or user. If questions derived from the same history appear across partitions, the experiment evaluates a different form of generalization from testing on unseen users and histories. The role of LongMemEval in training, hyperparameter tuning, and checkpoint selection also needs clarification.
  • The paper lacks an untrained Qwen3.5-9B baseline using the same memory architecture, role prompts, generator, and inference budget. The Gemini-based zero-shot baseline changes the backbone, while GRPO-only is already trained. Consequently, the experiments do not isolate how much improvement comes from the architecture and prompting versus learning the memory policy.
  • The QA advantage over the strongest zero-shot baseline is only 2.0 percentage points on LoCoMo, and SaliMory trails that baseline on LongMemEval-s: 69.1% versus 70.9% (Table 8). This also contradicts the best-result formatting in that table. The stronger personalization results remain valuable, but claims of consistently superior QA performance require qualification.

The novelty positioning is incomplete, and the contrastive mechanism does not establish isolated credit assignment.

  • Tan et al. (ACL 2025), In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents, already combines topic-based memory organization with online RL retrieval refinement using generator-provided citation signals. RMM is directly relevant to the task and motivation but is neither discussed nor evaluated.
  • Wang et al. (2025), Mem-α: Learning Memory Construction via Reinforcement Learning, already includes memory-content and action-level rewards (§3.1.2). Although Mem-α appears as a baseline, characterizing prior approaches as relying on monolithic rewards understates this overlap.
Questions for authors

see weakness

2 comments

We sincerely thank the reviewers for their careful reading and constructive comments. We are encouraged that the reviewers recognize the structured memory design, the fine-grained reward design, and the strong personalization results. Below we address each concern with new experiments and clarifications. Looking forward to our discussions for the revision!

In Responding to Weakness

W1. Split granularity; role of LongMemEval.

Data preparation and split. Thanks for bringing this out. Each LoCoMo conversation is first converted into per-user histories. Each history is then expanded into episodes, and each episode pairs one question with only the past sessions before the question. SaliMory thus learns to capture salient memories for question answering. The 60/20/20 split is applied over these episodes. This protocol rules out leakage in three ways:

  • No test question or answer is used in training. A training episode's reward depends only on its own question and reference answer.
  • Memory is built from scratch in every episode. Nothing written during training, or for another question, is available at test time.
  • The answer generator is frozen. The only trained component is the memory policy (what to store, how to update it, how to use it), and the evidence needed for each answer is always in that episode's own input.

What the split tests. The split is over episodes, so questions about the same LoCoMo history can appear in both train and test. We agree this tests generalization to new questions over seen histories, which differs from testing on unseen users. Two points show that this does not inflate our results, and a new experiment tests unseen users directly:

(i) Different questions need different evidence and reasoning. Questions over the same history span single-hop, multi-hop, temporal, open-domain, recommendation and implicit-personalization types, each drawing on different parts of the history. Memory is also rebuilt from scratch in every episode, so the policy cannot reuse a stored answer or memory from training.

(ii) LongMemEval-s holds out histories by construction. Every LongMemEval-s question comes with its own user history, so no test history is seen in training. LongMemEval-s follows the same protocol with its own 60/20/20 split, and there is no cross-training: each dataset is trained and evaluated separately. SaliMory's gains hold there.

(iii) New experiment: unseen users. We re-split LoCoMo by conversation (6 train / 2 validation / 2 test conversations), so test users and histories are entirely unseen, and retrained SaliMory on the 6 training conversations.

Test splitTest histories seen in training?Untrained policy QA / GoodSaliMory QA / GoodΔ QA / Δ Good
LoCoMo, split by questionYes60.4 / 18.672.9 / 39.8+12.5 / +21.2
LoCoMo, split by conversationNo59.8 / 17.970.6 / 36.1+10.8 / +18.2
LongMemEval-sNo61.2 / 9.469.1 / 20.8+7.9 / +11.4

SaliMory, split by question: Table 1 in the paper. SaliMory, LongMemEval-s: Table 8 in the paper. All other cells are new experiments.

SaliMory's gain over the untrained policy stays similar when no test history is seen in training (+10.8 QA / +18.2 Good on held-out conversations), even though it is trained on only 6 conversations.

Hyperparameters and the checkpoint are selected on the validation partition only, and the test partition is used once for reporting. We will state this protocol explicitly in the revision; thank you for raising this.

W2. Untrained Qwen3.5-9B baseline with the same architecture.

Thanks for pointing this out. We ran this control: identical architecture, role prompts, frozen Qwen3-235B generator, and inference budget, with the Qwen3.5-9B policy untrained.

LoCoMo / LoCoMo-P13nQA ↑Good Pers. ↑Vague ↓Factuality ↑Utility ↑
Untrained policy (same architecture)60.418.638.588.20.712
GRPO only (Table 2 in the paper)68.626.831.790.90.793
SaliMory72.939.819.693.90.917

Untrained-policy row: new experiment. GRPO-only and SaliMory rows: from Tables 1 and 2 in the paper.

LongMemEval-sQA ↑Good Pers. ↑
Untrained policy (same architecture)61.29.4
SaliMory69.120.8

Untrained-policy row: new experiment. SaliMory row: from Table 8 in the paper.

With our proposed memory architecture, prompts and budget, our reward design coupled with GRPO raises QA from 60.4 to 68.6 and Good Personalization from 18.6 to 26.8 compared with the untrained policy on LoCoMo/LoCoMo-P13n. The full SaliMory adds +4.3 QA and +13.0 Good Personalization (Table 2 in the paper), further verifying the value of our proposed contrastive refinement algorithm.

W3. QA margin over the zero-shot baseline; LongMemEval-s.

We agree and will scope the QA claim in the revision with respect to results on LongMemEval-s. One piece of context matters here: the strongest zero-shot baseline runs every memory operation with a frontier model, while SaliMory uses a 9B model with our proposed policy.

MethodMemory modelTrainedLoCoMo QALoCoMo GoodLME-s QALME-s Good
Mem-R1Qwen3.5-9BYes62.716.366.715.1
Zero-shot AgenticGemini-3-Flash (reportedly 1.5T with around 38B active parameters)No70.96.670.98.0
SaliMoryQwen3.5-9BYes72.939.869.120.8

All numbers are from Tables 1 and 8 in the paper;

With a much smaller memory model, SaliMory matches the frontier agent on QA (+2.0 on LoCoMo, −1.8 on LongMemEval-s) and delivers 2.6~6× its Good Personalization (20.8 vs 8.0; 39.8 vs 6.6). On LongMemEval-m, SaliMory leads on both (66.4 vs. 62.2 QA; 22.6 vs. 13.6 Good; Table 1(b) in the paper). Against trained memory methods, the margin is +10.2 QA over Mem-R1, and it holds (+12.5) when the generator is also 9B (Table 10 in the paper). We will remove "consistently superior QA", fix the bold formatting in Table 8 of the paper, and state the claim as: best QA among trained memory methods, on par with a frontier zero-shot agent, and more than double its Good Personalization on LoCoMo-P13n and LongMemEval-s.

W4. RMM and Mem-α.

Thank you for both references; we will revise §2.

  • RMM builds a multi-granularity memory bank prospectively and refines retrieval retrospectively using citation signals from the frozen generator. It does not train memory writing or utilization. SaliMory trains all three roles (filtering, consolidation, utilization) end-to-end, each with its own supervision signal. Since no official code is available, we reimplemented RMM from the paper:
LoCoMo / LoCoMo-P13nQA ↑Good Pers. ↑
RMM (our reimplementation)52.67.2
SaliMory72.939.8

RMM row: new (our reimplementation). SaliMory row: from Table 1 in the paper.

  • Mem-α. The precise distinction: Mem-α sums its correctness, format, compression, and content terms into one scalar per trajectory. SaliMory gives each role its own pairwise objective, selected by that role's own score (booster by R2, utilizer by utilization). This per-role pairing lifts GRPO-only from 68.6 QA / 26.8 Good to 72.9 / 39.8 (+4.3 / +13.0, Table 2 in the paper), and SaliMory outperforms Mem-α by +26.8 QA / +23.8 Good (Table 1 in the paper) on LoCoMo. We will rewrite the passage accordingly.
LoCoMo / LoCoMo-P13nQA ↑Good Pers. ↑
Mem-α (Table 1 in the paper)46.116.0
GRPO only (Table 2 in the paper)68.626.8
SaliMory (Table 1 in the paper)72.939.8

Sub-table of Tables 1 and 2 in the paper;

We hope these results and clarifications address your concerns. Looking forward to your responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.