- Rating
- 3(+1)
- Confidence
- 4(+1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper argues that current LLM memory systems keep memory outside the model as text records that are retrieved on demand. Inference itself therefore stays stateless. The authors propose LiveMem, which attaches a parallel Gated DeltaNet-2 recurrent branch to every attention layer of a frozen Qwen3-4B. The attention path covers a bounded working context that is turned over FIFO by evicting KV chunks, while the recurrent state is meant to carry information forward after its source tokens have been evicted. Only the branch is trained, with SFT followed by GRPO-style RL, and eviction is applied during training so the model has to rely on the state. Evaluation covers 18 datasets in four groups: multi-hop and multi-question Wiki QA, conversational memory, test-time learning and long-document QA, under 8K/32K active budgets. LiveMem reaches the best overall average (0.519 vs. 0.458). Supporting analyses condition on how much evidence has been evicted (LongMemEval) and compare a zero-state truncated input (Trunc) against the full state.
Critical strengths
Training under real eviction on a frozen pretrained backbone. The recurrent branch is zero-initialized, so training starts exactly from the base model. Training-time masking matches inference-time KV release, which stops the model from bypassing the state through attention. This is a practical way to add memory without pretraining a hybrid architecture, and it is the main reason I see value in the work.
Critical weaknesses
The gains are not shown to come from the memory state. In Table 2, the zero-state Trunc variant already recovers most of the improvement over Qwen3-4B. For example, on 2Wiki-MQ the base model scores 0.336, Trunc 0.545 and the full state 0.587. Across tasks the state itself adds only about 1–4 points. The training mixture also contains MuSiQue, 2Wiki, NarrativeQA and classification data that overlap with the evaluation tasks, while none of the baselines were trained on this data. Without a control that fine-tunes Qwen3-4B on the same data and active budget but without the branch, the headline result may mainly reflect in-distribution post-training. This bears directly on the paper's central claim.
Baseline protocols make the comparison unfair.
δ-Mem is run under a protocol that discards out-of-window input. It never sees the evicted history, so it behaves like a truncation baseline (3.1% on the fully-evicted subset). RAG uses top-1 retrieval for multi-hop QA (2Wiki 0.117 vs. 0.817 for the base model) and is omitted from MQ and from the eviction analysis in Fig. 2. This is despite RAG outperforming LiveMem on LongMemEval-S (0.548 vs. 0.302). The Recurrent baseline is untrained and scores 0.000 on Banking and CLINC. The strongest baseline overall is the base model with no memory mechanism at all. Taken together, the claim of "leading performance over RAG and intrinsic memory methods" is not established.
Questions for authors
1: How does Qwen3-4B score when trained on the same data and active-context budget, but without the recurrent branch?
2: How does δ-Mem perform under a protocol where it streams the full input? How does RAG perform with per-question retrieval and a larger top-k (e.g., top-5) on multi-hop, MQ and the eviction-conditioned subsets of Fig. 2?
3:On single-question Wiki QA, the inputs fit in the window and no eviction happens, yet LiveMem underperforms the base model (HotpotQA 0.846→0.772, MuSiQue 0.461→0.383). Why does the branch hurt when it is not needed?
3 comments
Baseline Improvements—RAG & -Mem
We agree with your suggestion that, on wiki-style multi-hop tasks, retrieving only one document with RAG is indeed unreasonable. We investigated the number of supporting evidence documents in 2Wiki, HotpotQA, and MuSiQue, and confirmed that top-k=3 is an appropriate value. In addition, we switched to Qwen3-Embedding-4B, a stronger embedding model. We evaluated the RAG baseline under this more reasonable setting than before and present the evaluation results.
| old | new | |
|---|---|---|
| 2Wiki | 0.117 | 0.639 |
| HotpotQA | 0.399 | 0.724 |
| MuSiQue | 0.117 | 0.318 |
The improved RAG scores are overall more reasonable.
In paper we used the trunc. setting for -mem, while we agree with you suggestion that -mem should be aligned with LiveMem here. We have added the aligned evaluation results.
| -mem | -mem (trunc.) | diff | |
|---|---|---|---|
| 2Wiki | 0.801 | 0.824 | -2.3% |
| 2Wiki-MQ | 0.246 | 0.232 | +1.4% |
| HotpotQA | 0.852 | 0.853 | -0.1% |
| Hotpot-MQ | 0.281 | 0.226 | +5.5% |
| MuSiQue | 0.496 | 0.483 | +1.3% |
| MuSiQue-MQ | 0.148 | 0.151 | -0.3% |
| LoCoMo | 0.290 | 0.256 | +3.4% |
| LongMemEval-s | 0.258 | 0.230 | +2.8% |
| FactCons | 0.189 | 0.171 | +1.8% |
| Banking | 0.840 | 0.770 | +7.0% |
| Clinc | 0.880 | 0.890 | -1.0% |
| NLU | 0.790 | 0.770 | +2.0% |
| Trec-C | 0.700 | 0.740 | -4.0% |
| Trec-F | 0.310 | 0.510 | -20.0% |
| Redial | 0.000* | 0.146 | -14.6% |
| infBenchQA | 0.168 | 0.114 | +5.4% |
| EventQA | 0.425 | 0.433 | -0.8% |
| NarrQA | 0.349 | 0.326 | +2.3% |
*: -mem directly outputs eos for all Redial samples, we probe the eos token's probability, which is >0.99 for all samples
The results show that -mem improves on some tasks and drops on others, with little overall change (overall -0.57%). In the final version, these results will replace the -mem scores in the main table.
Other Issues
Why does LiveMem in Table 1 perform worse than Qwen3-4B on the Wiki QA datasets 2Wiki, HotpotQA, and MuSiQue?
This is a very good question! We state the conclusion directly: this is the performance loss caused by YaRN. Qwen3-4B is a native 32k-length model and needs YaRN to be extended to 256k-length, and YaRN usually brings performance loss to the model. LiveMem is trained under YaRN, whereas the Qwen3-4B shown in Table 1 is the capability of Qwen3-4B under its native 32k without YaRN enabled. To control this configuration to be the same (i.e., comparing Qwen3-4B with YaRN enabled against LiveMem trained under YaRN), one should actually compare the Qwen3-4B in Table 6 with the LiveMem in Table 1, where the performance of Qwen3-4B on Wiki QA when YaRN is enabled and the length is extended to 256k is shown. We directly paste the results here for reference:
| Qwen3-4B (256k YaRN) | LiveMem-RL | |
|---|---|---|
| 2Wiki | 0.683 | 0.784 |
| HotpotQA | 0.632 | 0.772 |
| MuSiQue | 0.400 | 0.383 |
Thanks to author
Thank you for your detailed response. My concerns have been addressed.
I will update my score to 3 with 4 points of confidence. I think this is a good job and deserves to be accepted as a poster.
Rebuttal
Overall Response
Thank you for your review comments, we will respond to your concerns in the rebuttal. We strengthen the attribution of performance gains to memory state and decompose the gains from the post-training part. At the same time, we also re-adjust the baseline configurations to address the baseline setup. The adjusted overall results still support the conclusion that LiveMem is comprehensively leading.
Strengthening the Attribution of Performance Gains
We summarize your concern as follows: the gains of LiveMem come from the introduction of memory state and from post-training, but these two parts are mixed in the table without separate attribution. Therefore, we further refine this attribution. We select representative recall-style QA tasks (LongMemEval, LoCoMo, 2Wiki-MQ, HotpotQA-MQ, MuSiQue-MQ), filter samples whose supporting evidence is entirely outside the context window, and rerun generation with LiveMem after clearing the memory state on these samples, so as to compare the gain brought by Memory State. Then, on the same samples, we compare the QA scores of LiveMem without memory state and the Qwen3-4B backbone under the same conditions, in order to distinguish the gain brought by post-training.
The table shows that the gains of LiveMem overall come from two parts: one is the gain brought by memory state, and the other is the gain brought by post-training.