acceptodds
Under review as a conference paper at ICLR 2027

JanusMem: Learning Long-Term Memory Edits from Utility Change Across Time

Abstract

Long-term memory is essential for LLM agents that interact with an environment across many turns over time, where a memory manager must decide how to add, update, or delete entries as each turn arrives. Existing methods learn these decisions from the task outcomes observed after an edit. This signal overlooks both the utility change the edit causes, which separates its contribution from the utility already in memory, and the edit's effect on past and future queries: an edit that helps now may erase knowledge that earlier queries rely on or miss information that future queries need. To address both limitations, we present JanusMem, a reinforcement learning framework that measures the actual effect of each memory edit across time. On the current query, JanusMem scores an edit by both its post-edit utility and its change from the pre-edit memory, which isolates the edit's own contribution. JanusMem further uses a bidirectional window to credit improvement on nearby future queries and penalize deterioration on nearby past queries. Since utility changes can be subtle, all utilities are measured by a fixed LLM judge with continuous semantic feedback rather than lexical matching. Trained on only 152 QA pairs from a single LoCoMo conversation, JanusMem achieves state-of-the-art performance on the 8 held-out conversations, improving the LLM-Judge score over Memory-R1 by 11.6%. Without further training, the same model transfers zero-shot to LongMemEval and MemoryAgentBench, outperforming all baselines with gains of 7.6% and 2.9%, respectively.

Primary area:
reinforcement learning
Keywords:
Agent memory,Memory Manager,Long-term memory

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MgwEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper trains an LLM memory manager to add, update, or delete memories using rewards based on answer quality before and after each edit, while penalizing harm to past questions and rewarding support for future questions during training. Trained on 152 QA pairs, it reports improved LLM-judged accuracy on LoCoMo and two transfer benchmarks.

Critical strengths

It is an interesting idea to treat memory editing as preserving useful knowledge across time, rather than optimizing only the answer to the current question. It is also nice to have cross-benchmark generalization experiment results demonstrating improvement.

Critical weaknesses

My main concern is that the paper uses a very specific shaped reward

0.7s+0.3tanh⁡((s−b)/0.25)0.7s+0.3\tanh((s-b)/0.25)

This introduces non-linearity in addition to the difference s−bs-b. An ablation should be added to isolate the effect.

Questions for authors
  1. Can you report how often an edit makes a previously answerable question unanswerable, compared with the baselines? This would directly test the preservation mechanism rather than infer it from final QA accuracy.
1 comment

W1 Reward shaping and utility change

Thank you for asking us to disentangle utility change from the nonlinear transformation. The comparison in our original Table 6 measures their combined effect. We have added ablations of the current-query reward, keeping both temporal windows enabled at K=3K=3.

Let s+s^+ and s−s^- denote the answering utilities after and before an edit on the same question. All scores are percentages.

Current-query rewardLoCoMo LLM-JLongMemEval LLM-J
αs++(1−α)tanh⁡(s+−s−τ)\alpha s^+ + (1-\alpha)\tanh\left(\frac{s^+-s^-}{\tau}\right)44.1037.60
s+s^+39.1034.20
αs++(1−α)tanh⁡(s+τ)\alpha s^+ + (1-\alpha)\tanh\left(\frac{s^+}{\tau}\right)38.1133.80
tanh⁡(s+−s−τ)\tanh\left(\frac{s^+-s^-}{\tau}\right)40.0835.00
s+−s−s^+-s^-40.0034.60

The nonlinear post-edit control uses s+/τs^+/\tau inside tanh⁡\tanh. The full reward uses (s+−s−)/τ(s^+-s^-)/\tau and retains the same linear term αs+\alpha s^+. The full reward exceeds this control by 5.99 percentage points on LoCoMo and 3.81 on LongMemEval.

Applying tanh⁡\tanh to (s+−s−)/τ(s^+-s^-)/\tau raises LLM-J by 0.08 and 0.40 points over s+−s−s^+-s^- alone. The nonlinear post-edit control scores 0.94 and 0.41 points below s+s^+ alone. The full reward achieves the highest LLM-J score on both benchmarks, exceeding tanh⁡((s+−s−)/τ)\tanh((s^+-s^-)/\tau) alone by 4.02 and 2.60 points.

Q1 How often does an edit make a previously correct answer incorrect?

Across all eight LoCoMo test conversations, we sampled up to three non-adversarial questions per turn for JanusMem. A question was eligible if all its annotated evidence appeared before that turn. We used a fixed-seed hash of the conversation, turn, and question identifiers to select questions independently of model answers. This yielded 15,062 turn–question pairs. For each pair, we used the same Answer Agent and retrieval settings before and after the edit, and scored both answers with Qwen2.5-7B-Instruct as a binary judge.

For both methods, we count only turn–question pairs from turns that changed the memory bank. A turn where the model emits only NOOP leaves the bank unchanged and is excluded.

MethodPairs with memory changesWrong → WrongWrong → CorrectCorrect → WrongCorrect → CorrectDamage rate
JanusMem14,6417,2981331457,065145/7,210 = 2.01%
Memory-R114,2338,0181152755,825275/6,100 = 4.51%

We define the damage rate as the fraction of initially correct answers that become incorrect. JanusMem's observed rate was 2.50 percentage points lower.

Reviewer_ZA56OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents JanusMem, a reinforcement learning (RL) framework to teach LLM agents about memory edits based on the utility change instead of purely relying on task outcomes. It employs a bidirectional window on both past queries and future queries and scores the utility using a fixed LLM with continuous semantic feedback. On the long-term memory benchmarks, including LoCoMo, LongMemEval, and MemoryAgentBench, it achieves state-of-the-art performance compared to the existing baselines such as Memory-R1.

Critical strengths
  • This paper designs a novel reward model by crediting the memory editing for supporting future queries and penalizing the deterioration of past queries, using a bidirectional window.
  • It also designs an LLM judge to provide continuous feedback based on the semantic change in the answer instead of using exact match.
Critical weaknesses
  • My major concern is the inconsistency of the evaluation metrics when comparing against the Memory-R1 baseline; please see the questions for authors section below.
  • The contribution of the proposed JanusMem is somewhat limited, as it simply replaces the reward model in Memory-R1 with a new reward model including (1) a continuous semantic change score and (2) future credit and past penalty. The Answer Agent and Memory Manager follow the same design as in the Memory-R1 paper. Using LLM-as-a-judge to provide continuous feedback and report a score to two decimal places has the risk of introducing noise or bias.
Questions for authors
  • From the main results in Table 3, the proposed JanusMem significantly outperforms the Memory-R1 baseline (which is based on EM reward). However, in Table 12 in the Appendix, Memory-R1 outperforms JanusMem on the LoCoMo dataset using the F1 metric. The reported metrics for Memory-R1 also appear lower than those reported in the Memory-R1 paper [1]. For example, with Qwen-2.5-7B-Instruct base models, the LLM-as-judge accuracy on LoCoMo in the Memory-R1 paper is 62.74; however, this number is only 32.5 in this paper. Could the authors explain the discrepancies in the reported metrics?

[1]: Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie, Zifeng Ding, Zonggen Li, Xiaowen Ma, Jinhe Bi, Kristian Kersting, Jeff Z. Pan, Hinrich Schuetze, Volker Tresp, and Yunpu Ma. 2026. Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12805–12825, San Diego, California, United States. Association for Computational Linguistics.

2 comments

W1&Q1 — Memory-R1 reproduction and metric differences

Why our Memory-R1 score differs from the published result

The submitted Memory-R1 baseline used the third-party implementation by pradyutnair-prosus. The authors released their training and evaluation code on November 8, 2026 (UTC+8), after the review release. We subsequently ran Memory-R1 and JanusMem in the official implementation with Qwen2.5-7B-Instruct. Third-party repository (https://github.com/pradyutnair-prosus/memory-r1), official release (https://github.com/yansikuan/memory-r1/commit/f5ca10a45a08320b60575010c622fed6d3b3c686).

In Memory-R1 Table 1, the 62.74% LLM-J score is for Llama-3.1-8B-Instruct. The corresponding GRPO result for Qwen2.5-7B-Instruct is 61.51%. Our submitted baseline of 32.5% used the earlier implementation and Qwen2.5-14B-Instruct as the judge. Memory-R1, Table 1 (https://arxiv.org/html/2508.19828v5).

The new runs use the same official LoCoMo test set and evaluation prompt with Kimi K3 as judge. The reproduction uses all-MiniLM-L6-v2 embeddings. The official defaults are GPT-4o-mini and text-embedding-3-small. Scores are percentages, and B1 is BLEU-1. Official evaluation instructions (https://github.com/yansikuan/memory-r1/blob/main/eval/README.md).

Method and sourceEvaluation judgeF1B1LLM-J
Memory-R1, published Qwen2.5-7B GRPO resultGPT-4o-mini43.1436.4461.51
Memory-R1, our official-code reproductionKimi K342.4037.1559.37
JanusMem, implemented in the official codeKimi K332.5534.6168.97

The reproduced Memory-R1 baseline differs from the published result by −0.74 points on F1 and +0.71 on B1. Its LLM-J score is numerically 2.14 points lower under a different judge. Under the shared Kimi K3 judge, JanusMem scores 68.97% versus 59.37%, a gain of 9.60 percentage points. We will identify the implementation and judge for each set of results in the revision.

W2 — Contribution beyond Memory-R1

Contribution and applicability across agent frameworks

JanusMem builds on the Memory Manager and Answer Agent architecture of Memory-R1. Our contribution is a training objective for memory edits that measures their effect on answering utility across time. It uses the pre-edit memory as a reference for the current question, penalizes deterioration on earlier questions, and credits support for later questions. This specifies which consequences of an edit enter the learning signal.

This objective can be implemented wherever a training procedure can evaluate memory states before and after an operation. We tested that portability in AgeMem, whose unified policy manages long-term memory and short-term context through tool actions. This gives us a second implementation with a different organization of memory management and answering. AgeMem (https://arxiv.org/abs/2601.01885).

We adapted AgeMem to LoCoMo and evaluated both variants with Qwen2.5-3B-Instruct. Both rows below are from our experiments. Scores are percentages.

MethodExact matchF1LLM-J
AgeMem, our LoCoMo experiment5.9326.5541.02
AgeMem with the JanusMem objective7.0132.5549.10
Absolute change+1.08+6.00+8.08

The improvement across all three metrics provides evidence that the training objective is useful in a unified-agent framework as well as the Memory-R1 architecture.

Q1 & W2 — How are training rewards scored, and which metric agrees most closely with human judgments of answer correctness?

Thank you for raising the question of training reward granularity. LLM5 uses five semantic quality bands. The judge first selects a band based on the correctness and completeness of the answer, then assigns a score within that band to two decimal places. For example, 0.62 and 0.64 both fall in the fourth band, meaning that most key information is correct, with minor omissions or imprecision.

We introduced fine-grained scoring within each band to reduce reward ties during GRPO training. This introduces some noise into within-band scores. In the Memory Manager training runs reported in Table 8, switching from LLM3 to LLM5 reduced the proportion of groups in which all candidate edits received the same reward from 43.8% to 21.6%. With the Answer Agent reward fixed at LLM5 and only post-edit utility enabled, LoCoMo LLM-J accuracy increased from 38.9% to 40.2%.

Agreement with human judgments

For final-answer correctness, we compared EM, token F1, and LLM-J against human judgments on the same answers. Zheng et al. used human preferences to evaluate LLM judges in MT-Bench and Chatbot Arena. For LoCoMo, our reference was the majority judgment of three annotators on whether each answer was correct. Zheng et al., NeurIPS 2023 (https://arxiv.org/abs/2306.05685).

We selected 200 questions from eight LoCoMo test conversations, with 50 questions per category. Three annotators each assessed all 400 answers from JanusMem and Memory-R1. They agreed unanimously on 300 answers (75.0%, Krippendorff’s α = 0.639). To convert token F1 into a binary judgment, we selected its threshold on seven conversations and applied it to the held-out eighth conversation in each fold. AUROC and Spearman’s ρ use the original scores. We estimated 95% confidence intervals using 10,000 paired bootstrap resamples at the conversation level.

MetricAgreement with human majorityCohen’s κ [95% CI]False negative rateFalse positive rateAUROCSpearman’s ρ
LLM-J85.50% (342/400)0.681 [0.614, 0.761]20.71%11.15%0.8410.684
EM72.75% (291/400)0.270 [0.179, 0.376]77.86%0.00%0.6110.412
Token F178.75% (315/400)0.544 [0.440, 0.652]25.00%19.23%0.8730.674

LLM-J had the highest agreement with the human majority on individual answers (85.50%), exceeding token F1 by 6.75 percentage points and EM by 12.75 points. It also had the highest chance-corrected agreement (Cohen’s κ = 0.681).

Human evaluation and LLM-J gave the same system ranking. On the same 200 questions, human annotators judged 41.50% of JanusMem answers and 28.50% of Memory-R1 answers correct, giving JanusMem a 13-point lead by majority vote. LLM-J scored the two systems at 42.50% and 27.50%, respectively, giving JanusMem a 15-point lead.

LLM5 scoring prompt

You are an impartial judge. Given a question and a gold (reference) answer, assess the correctness of the predicted answer in two steps.

Step 1. Select one of the following five bands based on the correctness and completeness of the predicted answer.

- Band 1 (0.00–0.19, wrong): The answer contradicts the reference, misses the key information entirely, is unrelated to the question, or is empty.
- Band 2 (0.20–0.39, mostly wrong): The answer has only incidental overlap with the reference.
- Band 3 (0.40–0.59, partially correct): Some key information is correct, but parts are missing, imprecise, or slightly wrong.
- Band 4 (0.60–0.79, mostly correct): Most key information is correct, with only minor omissions or imprecision.
- Band 5 (0.80–1.00, fully correct): The answer conveys the same key information as the reference. Paraphrases and different formats of the same date or number count as fully correct.

Step 2. Within the selected band, assign a specific score based on the correctness and completeness of the predicted answer.

Question: {question}
Gold answer: {gold_answer}
Predicted answer: {predicted_answer}

Respond with exactly one number to two decimal places (e.g., 0.73).
Reviewer_YMrYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JanusMem, a reinforcement learning framework for long-term memory editing in LLM agents. It evaluates memory edits using post-edit utility and utility change, incorporating a bidirectional window that considers past and future queries. A fine-grained LLM judge provides feedback for training the Memory Manager. Experimental results on LoCoMo, LongMemEval, and MemoryAgentBench demonstrate that this approach outperforms existing memory management methods.

Critical strengths
  1. The paper addresses an important problem in long-term memory management by evaluating the utility change caused by individual memory edits rather than relying solely on post-edit task outcomes.
  2. The method combines post-edit utility with nonlinear utility change, supported by an analysis of GRPO group normalization. The bidirectional window further considers both past and future queries, providing a clear motivation for the overall design.
  3. The experiments cover LoCoMo, LongMemEval, and MemoryAgentBench, with evaluations across different model backbones and ablation studies, showing promising generalization across benchmarks.
Critical weaknesses
  1. The contribution of utility change is not sufficiently isolated. The proposed reward combines post-edit utility with a nonlinear transformation of utility change, making it unclear which component drives the improvements. A comparison with a nonlinear transformation of post-edit utility would help clarify its contribution.
  2. The effectiveness of the bidirectional window is not fully established. Appendix Table 14 shows that at (K=3), the past-only setting outperforms the combined past-and-future setting on all three benchmarks. Further future-only ablations and analysis are needed to establish the benefit of combining both windows.
  3. The reported results contain inconsistencies. Table 4 and Appendix Table 14 report different MemoryAgentBench scores (28.8% vs. 29.8%) for the default bidirectional setting with (K=5), while Figure 3 and Appendix Table 17 report different LoCoMo scores at (G=32) (40.78% vs. 43.2%). These discrepancies require clarification.
  4. The evaluation lacks independent validation. The same Qwen2.5-14B-Instruct judge is used for training rewards and evaluation. Moreover, JanusMem achieves lower EM and token-level F1 scores than Memory-R1 on LoCoMo despite higher LLM-Judge accuracy. Evaluation with a different judge or human assessment would strengthen the conclusions.
Questions for authors
  1. Can the authors clarify whether the improvement attributed to utility change comes from comparing pre-edit and post-edit utility or from the nonlinear transformation?
  2. Why does the past-only setting outperform the bidirectional setting at (K=3), and what is the independent contribution of the future window?
  3. Can the authors explain the numerical discrepancies between the main paper and appendix, particularly in Table 4, Table 14, Figure 3, and Table 17?
  4. Do the improvements in LLM-Judge accuracy remain consistent when using an independent judge or human evaluation?
2 comments

W1&Q1 — Utility change and nonlinear shaping

Table 6 evaluated the pre-edit reference and nonlinear shaping together. To isolate their contributions, we added ablations varying only the current-query reward, with both temporal windows enabled at K=3K=3. Here s+s^+ and s−s^- denote post-edit and pre-edit utility, respectively. The table reports LLM-J accuracy (%).

Current-query rewardLoCoMo LLM-JLongMemEval LLM-J
αs++(1−α)tanh⁡(s+−s−τ)\alpha s^+ + (1-\alpha)\tanh\left(\frac{s^+-s^-}{\tau}\right)44.1037.60
s+s^+39.1034.20
αs++(1−α)tanh⁡(s+τ)\alpha s^+ + (1-\alpha)\tanh\left(\frac{s^+}{\tau}\right)38.1133.80
tanh⁡(s+−s−τ)\tanh\left(\frac{s^+-s^-}{\tau}\right)40.0835.00
s+−s−s^+-s^-40.0034.60

The nonlinear post-edit control scores 0.94 percentage points below s+s^+ on LoCoMo and 0.41 points below on LongMemEval. The full reward improves LLM-J accuracy over s+s^+ by 5.05 and 3.40 points, and over the nonlinear post-edit control by 5.99 and 3.81 points, respectively. The bounded and linear change-only variants differ by 0.08 points on LoCoMo and 0.40 points on LongMemEval. The full reward has the highest LLM-J score on both benchmarks.

W2&Q2 — Past and future windows at K=3K=3

Past-only led on all three benchmarks in the runs originally reported in Table 14. We repeated the comparison at K=3K=3 using three new random seeds, and added a future-only variant. The table reports mean LLM-J accuracy ± sample standard deviation across the three seeds, in percentage points.

Temporal windowLoCoMoLongMemEvalMemoryAgentBench
Past only42.21 ± 0.8735.20 ± 1.3128.17 ± 0.38
Future only41.91 ± 0.5435.30 ± 0.1027.73 ± 0.61
Past + future43.95 ± 0.5438.93 ± 1.1030.03 ± 0.32

The original past-only lead likely reflects run-to-run variability. In the new experiments, the combined window outperformed both single-window variants on all three benchmarks at each seed. Compared with past-only, it improved mean accuracy by 1.74 points on LoCoMo, 3.73 on LongMemEval, and 1.87 on MemoryAgentBench.

W3 & Q3 — Discrepancies between the main text and appendix

For MemoryAgentBench, Appendix Table 14 correctly reports 29.8% for the K=5K=5 setting with both past and future windows enabled. The 28.8% reported in Table 4 was a reporting error. We compute the overall accuracy by dividing the total number of correct answers across all four categories by the total number of evaluated examples. Each category therefore contributes in proportion to its sample count. We will use the K=3K=3 past+future checkpoint consistently for the main results in the revised manuscript.

For LoCoMo at G=32G=32, the correct score is 40.78%, as shown in Figure 3. We will correct the 43.2% entry in Appendix Table 17 to 40.78%. We thank the reviewer for pointing out these discrepancies.

W4&Q4 — Validation with a different judge and human evaluation

Evaluation with DeepSeek V4.1 Flash

The submitted evaluation used Qwen2.5-14B-Instruct, which also supplied the training reward scores. We evaluated JanusMem with DeepSeek V4.1 Flash as an alternative judge and compared its scores with the JanusMem row of Tables 3 and 4. Scores are percentages, and changes are in percentage points.

BenchmarkCategory or taskQwen2.5-14B-Instruct (Tables 3–4)DeepSeek V4.1 FlashChange
LoCoMoSingle-hop54.255.2+1.0
LoCoMoMulti-hop36.437.1+0.7
LoCoMoOpen-domain26.525.4−1.1
LoCoMoTemporal28.727.9−0.8
MemoryAgentBenchAccurate Retrieval44.144.0−0.1
MemoryAgentBenchConflict Resolution20.121.5+1.4
MemoryAgentBenchLong-Range Understanding52.153.0+0.9
MemoryAgentBenchTest-Time Learning5.24.7−0.5
LongMemEvalOverall36.435.8−0.6

Across the nine entries, JanusMem's mean absolute score difference is 0.79 points, and the largest difference is 1.4 points.

Human evaluation and metric agreement

Three annotators each rated 400 answers from JanusMem and Memory-R1 to 200 LoCoMo questions (eight conversations, 50 questions per category). They agreed unanimously on 300 answers (75%) and split 2–1 on 100. Pairwise Cohen’s κ ranged from 0.579 to 0.714, and Krippendorff’s α was 0.639 (95% CI [0.574, 0.712]). The majority vote classified 140 answers as correct and 260 as incorrect.

LLM-J agreed most often with the human majority on answer correctness (342/400, 85.50%). For binary F1 judgments, we selected the threshold on seven conversations and applied it to the eighth (thresholds of 0.26 or 0.31). AUROC uses the raw scores, and Spearman’s ρ compares them with the share of annotators voting correct. Confidence intervals use 10,000 paired bootstrap resamples of conversations.

MetricAgreementCohen's κ [95% CI]False negative rateFalse positive rateAUROCSpearman ρ
LLM-J85.50%0.681 [0.614, 0.761]20.71%11.15%0.8410.684
EM72.75%0.270 [0.179, 0.376]77.86%0.00%0.6110.412
Token F178.75%0.544 [0.440, 0.652]25.00%19.23%0.8730.674

LLM-J’s κ exceeded EM’s by 0.411 (95% CI [0.326, 0.507]) and thresholded F1’s by 0.137 ([0.061, 0.227]). EM missed 109 of the 140 answers accepted by the majority. Raw F1 had the highest AUROC, exceeding LLM-J by 0.033 (95% CI [−0.003, 0.066]).

All three annotators ranked JanusMem higher, by 19.0, 10.5, and 15.5 percentage points. By majority vote, only JanusMem was correct on 46 questions and only Memory-R1 on 20.

EvaluationJanusMemMemory-R1JanusMem − Memory-R1 [95% CI]
Human majority vote41.50% (83/200)28.50% (57/200)+13.00 [+5.33, +20.43]
LLM-J accuracy42.50%27.50%+15.00 [+7.73, +23.50]
EM4.50%11.00%−6.50 [−9.95, −3.20]
Mean token F1 (0–100)25.2029.16−3.96 [−9.01, +2.23]

JanusMem led by 13 points under human evaluation and 15 points under LLM-J. The two estimates differed by 2.00 points (95% CI [−1.84, +5.56]).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.