- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Q-mem is a bounded recurrent memory module for frozen decoder language models. A gated query component orthogonal to the incoming key helps select old content for correction, while the key remains the sole update address. The module also uses retention, separate correction and write gates, and a temporary state for each request. Q-mem achieves the best reported aggregate LoCoMo F1 and BLEU-1 among the compared methods across three backbones. Additional experiments examine knowledge updating, context recovery, structural components, and initialization sensitivity.
Critical strengths
-
The method offers a coherent way to separate correction-content selection from the write address. Independent correction and write gates provide flexible control, while the temporary request state lets the current question condition memory access without changing persistent history. The equations and state lifecycle are clearly described.
-
The experiments cover three frozen backbones, several memory baselines, knowledge updating, context recovery, structural ablations, and initialization sensitivity. Q-mem consistently improves over delta-mem in aggregate LoCoMo results, and the analyses report both useful gains and limitations of the design.
Critical weaknesses
-
Table 2 shows only small gains from the query-dependent direction over a fixed random direction: 0.45 F1 and 0.37 BLEU-1 under a single seed. The random-direction control is also insufficiently specified, including how the vector is sampled, normalized, gated, and projected against the key. Given the small gap and lack of uncertainty estimates, the results do not firmly establish the benefit of query dependence beyond adding an extra correction direction.
-
Resetting memory across independent training episodes is reasonable, but Q-mem targets assistants that maintain user-specific memory across sessions. Although LoCoMo contains multi-session histories, the paper does not systematically evaluate retention and knowledge revision as the same persistent state undergoes progressively more updates across sessions. It therefore remains unclear whether memory quality remains stable as interactions accumulate.
Questions for authors
-
How is the fixed random direction constructed and applied in Table 2? Can the authors report results across multiple training seeds and random-direction draws, with uncertainty, to establish whether the query-dependent gain is reliable?
-
How is the persistent state maintained across LoCoMo sessions and queries? Can the authors evaluate retention and knowledge revision over progressively longer interactions with the same user, continuously maintaining the persistent state across sessions?
4 comments
Replies to Reviewer h3G3: second part
W2&Q2: Retention and knowledge revision over progressively longer interactions with the same user
Thank you for distinguishing multi-session QA from evaluation with a continuously updated persistent state. In the original LoCoMo evaluation (A), we reset the state before replaying each conversation, but maintain it across the sessions within that conversation. We now also evaluate a continuous stream (B and C): we process all 10 LoCoMo conversations in their stored order, carrying the same recurrent state across conversation boundaries without carrying over the attention KV cache. Questions are answered from a snapshot taken at the end of the corresponding conversation. Evaluation questions and generated answers are not written back to the persistent state.
Retention without explicit history. In C, the answer prompt contains the instruction and question but no conversation text. Table R2 compares δ-mem and Q-mem with the no-memory backbone on the same 1,540 questions.
Table R2. LoCoMo results without explicit history.
| Method | State policy | Multi-hop F1 | Multi-hop B1 | Temporal F1 | Temporal B1 | Open-domain F1 | Open-domain B1 | Single-hop F1 | Single-hop B1 | Overall F1 | Overall B1 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-4B-Instruct backbone | None | 8.34 | 6.94 | 2.27 | 2.47 | 10.07 | 9.42 | 8.00 | 6.03 | 7.00 | 5.67 |
| δ-mem | Carry (C) | 10.18 | 8.13 | 2.57 | 2.17 | 12.57 | 9.31 | 11.41 | 8.94 | 9.41 | 7.40 |
| Q-mem | Carry (C) | 11.37 | 9.21 | 2.57 | 2.44 | 12.66 | 9.74 | 12.17 | 9.48 | 10.05 | 7.98 |
To examine performance as writes accumulate, we also report F1 for the early, middle, and late portions of the stream, ending after 70, 187, and 272 sessions, respectively. Q-mem scores 10.06, 10.69, and 9.22, compared with 6.28, 6.88, and 7.70 for the backbone on the corresponding questions. The carried state therefore remains useful without explicit history even late in the stream.
Performance under continued memory updates. A and B both expose only the current conversation’s full history in the answer prompt. Table R3 compares resetting the state before each conversation (A) with carrying it across conversations (B).
Table R3. LoCoMo results with the current conversation visible in the prompt.
| Method | State policy | Multi-hop F1 | Multi-hop B1 | Temporal F1 | Temporal B1 | Open-domain F1 | Open-domain B1 | Single-hop F1 | Single-hop B1 | Overall F1 | Overall B1 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| δ-mem | Reset (A) | 33.44 | 26.12 | 35.64 | 30.78 | 12.55 | 11.24 | 52.29 | 43.17 | 42.89 | 35.47 |
| δ-mem | Carry (B) | 35.92 | 27.02 | 30.97 | 27.46 | 14.60 | 12.84 | 53.26 | 47.34 | 43.03 | 37.32 |
| Q-mem | Reset (A) | 38.54 | 29.55 | 38.35 | 34.81 | 13.80 | 12.53 | 56.40 | 49.23 | 46.71 | 40.33 |
| Q-mem | Carry (B) | 37.76 | 28.11 | 37.87 | 33.82 | 13.93 | 12.44 | 56.47 | 49.11 | 46.52 | 39.79 |
Under carryover, Q-mem remains close to its reset result and above δ-mem. This shows that Q-mem maintains most of its QA performance when it continues updating a state that already contains earlier conversations. This indicates that, throughout increasingly long interactions with the same user, Q-mem retains a high capacity for knowledge modification.
Thank you for the clarifications and additional experiments. The random-direction controls and continuous-state experiments help address my questions. I will maintain my original positive score of 3.
Thank you for your positive feedback and for taking the time to review our additional experiments. We appreciate your constructive comments, which have helped us strengthen the paper. We will incorporate these clarifications and results into the revised manuscript.
Replies to Reviewer h3G3
W1&Q1: Random-direction control and reliability of query-dependent gains
In Table 2 of the original paper, we sample an eight-dimensional standard Gaussian vector for each adapted layer and memory head at initialization, normalize it to unit ℓ₂ norm, and keep it frozen throughout training and evaluation. This vector replaces the memory query only in the additional correction term and the normal query-based memory readout is unchanged. With fixed direction r, correction gate γt, normalized key kt, and learned strength η, the correction selector is
$ a_t=\gamma_t\odot k_t+\eta(I-k_tk_t^\top)(\gamma_t\odot r). $
We apply the gate before projection and do not renormalize afterward. Thus, the sampled vector is fixed, but its gated and projected contribution still varies with the token. The update address remains kt.
To quantify variability, we retrained the fixed-random control with three direction draws for each of three training seeds (42, 43, and 44). Each draw samples a new set of per-layer, per-head vectors using a separate random generator, while the other parameter initializations are held fixed within each training seed. The same LoCoMo evaluation settings are used for all nine runs.
Table R1. Fixed-random control across training seeds and random-direction draws.
Averaging the three direction draws within each training seed, the fixed-random control achieves 46.05 ± 0.52 F1 and 39.37 ± 0.39 B1 (mean ± sample SD across the three seed means). The pooled within-seed SDs across direction draws are 0.44 and 0.48 points, respectively.
The query-dependent result in Table 2 of the original paper (46.71 F1, 40.33 B1) exceeds the fixed-random mean for each of the three control training seeds on both metrics, and exceeds 8 of 9 individual runs on F1 and all 9 on B1. Thus, the observed aggregate gain is robust to the tested choices of random direction: it does not depend on the particular draw used in Table 2. The query-dependent model was evaluated from a single training run, so this result addresses sensitivity to the random control.