- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper trains an LLM memory manager to add, update, or delete memories using rewards based on answer quality before and after each edit, while penalizing harm to past questions and rewarding support for future questions during training. Trained on 152 QA pairs, it reports improved LLM-judged accuracy on LoCoMo and two transfer benchmarks.
Critical strengths
It is an interesting idea to treat memory editing as preserving useful knowledge across time, rather than optimizing only the answer to the current question. It is also nice to have cross-benchmark generalization experiment results demonstrating improvement.
Critical weaknesses
My main concern is that the paper uses a very specific shaped reward
This introduces non-linearity in addition to the difference . An ablation should be added to isolate the effect.
Questions for authors
- Can you report how often an edit makes a previously answerable question unanswerable, compared with the baselines? This would directly test the preservation mechanism rather than infer it from final QA accuracy.
W1 Reward shaping and utility change
Thank you for asking us to disentangle utility change from the nonlinear transformation. The comparison in our original Table 6 measures their combined effect. We have added ablations of the current-query reward, keeping both temporal windows enabled at K=3.
Let s+ and s− denote the answering utilities after and before an edit on the same question. All scores are percentages.
The nonlinear post-edit control uses s+/τ inside tanh. The full reward uses (s+−s−)/τ and retains the same linear term αs+. The full reward exceeds this control by 5.99 percentage points on LoCoMo and 3.81 on LongMemEval.
Applying tanh to (s+−s−)/τ raises LLM-J by 0.08 and 0.40 points over s+−s− alone. The nonlinear post-edit control scores 0.94 and 0.41 points below s+ alone. The full reward achieves the highest LLM-J score on both benchmarks, exceeding tanh((s+−s−)/τ) alone by 4.02 and 2.60 points.
Q1 How often does an edit make a previously correct answer incorrect?
Across all eight LoCoMo test conversations, we sampled up to three non-adversarial questions per turn for JanusMem. A question was eligible if all its annotated evidence appeared before that turn. We used a fixed-seed hash of the conversation, turn, and question identifiers to select questions independently of model answers. This yielded 15,062 turn–question pairs. For each pair, we used the same Answer Agent and retrieval settings before and after the edit, and scored both answers with Qwen2.5-7B-Instruct as a binary judge.
For both methods, we count only turn–question pairs from turns that changed the memory bank. A turn where the model emits only NOOP leaves the bank unchanged and is excluded.
We define the damage rate as the fraction of initially correct answers that become incorrect. JanusMem's observed rate was 2.50 percentage points lower.