acceptodds
Under review as a conference paper at ICLR 2027

STAMP: Learning Cross-Session Memory in Native Recurrent States

13 views

Abstract

Hybrid attention-recurrent language models process each session as a token sequence, using attention caches for local context while compressing part of the history into fixed-size native recurrent states. These states provide bounded memory within a sequence, but long-term memory requires information from a completed session to initialize later independent calls after the source tokens and attention key–value cache are discarded. We introduce STAMP, a learned cross-session interface that makes the backbone’s existing recurrent states persistent and reusable across independent calls. A learned write mechanism encodes future-useful information into terminal recurrent states, and a lightweight state adapter maps them into initial states for future sessions. STAMP is trained end-to-end over multi-session trajectories from a general-purpose context–question–answer corpus, but at inference it requires no per-session optimization. Experiments show that STAMP transfers without benchmark-specific retraining to LoCoMo, NextMem, PersonalMem, and LongMemEval, improving state-only recall over backbone-matched Metis while keeping the memory footprint fixed as sessions accumulate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Linear attention, recurrent model, persistent memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_8VXjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STAMP learns a cross-session interface for the existing recurrent states of frozen Qwen3.5 backbones. Learned write tokens and write-only LoRA prepare the terminal state, a low-rank carrier maps it to the next session's initializer, and norm-controlled carry supports repeated updates. Source context and attention KV caches are removed before answering. On the static evaluation, macro F1 improves over backbone-matched Metis by 6.15 points at 4B and 12.19 points at 27B. The paper also reports 2/4/8-write trajectories, ten-segment component and geometry diagnostics, multi-state reading, and post-write SVD compression. The reported results establish state-only memory construction from supplied relevant evidence more directly than retention of complete interaction histories for unknown future questions.

Critical strengths

S1. A concrete adaptation interface that reuses the backbone's own recurrent states. Unlike the dedicated memory pathway of Metis, STAMP writes into and reads from the existing recurrent matrices of a frozen backbone, and under source-excluded answering it improves macro F1 over backbone-matched Metis at both scales (+6.15 at 4B, +12.19 at 27B). This is a potentially useful, low-footprint way to make native state persist across independent calls.

S2. Candid disclosure of the evaluation's limits. The paper states gold-evidence conditioning, overlap between evaluation suites, growing LoCoMo endpoint question sets, repeated-write interference, and the storage/compression trade-off. Together with retrained ablations and carrier-only results, this makes the evidence easy to scope correctly.

Critical weaknesses

W1. The main evidence does not cover complete interaction memory, and retention is not separated from acquisition. Section 5 and Appendix B reset memory per LoCoMo question and write its annotated evidence sessions; LongMemEval merges each question's gold evidence into a packet. This is a valid memory-construction setting, not a leakage allegation, but the future question has already determined which history must be stored. The 2/4/8-write trajectories and ten-segment diagnostics still do not evaluate writing an entire conversation once and answering previously unknown queries. In Table 3, STAMP-27B LongMemEval F1 falls from 29.55 after two writes to 15.82 after eight, below Metis-27B's 17.33; at 4B the eight-write scores are 13.29 versus 17.24. Gains on three of four suites do not establish uniformly stronger long-term retention, and LoCoMo's growing endpoint question sets are a confound the authors acknowledge. Either a bounded complete-history evaluation with an unchanged fact set and age-stratified retention is needed, or the abstract and conclusions should be narrowed to supplied-evidence compression and recall.

W2. Architectural and resource advantages, and the carrier/stabilization mechanism, are not isolated. The Aux.=0 column excludes STAMP's 48/144 MiB native states (disclosed in the caption), so total persistent bytes should be shown; the 16-soft-token baseline uses only 0.078 MiB and is not capacity-matched. Metis checkpoints, training data, supervision and tuning budgets are not documented, and there is no direct state-carry or simple normalization/scaling baseline trained with the same QA supervision. Appendix C.1 removes deployment stabilization together with the transformation, so the carrier's own contribution is not separable; default SLERP is not the best setting in Figure 4, and Table 7's static PersonalMem carrier-only score (18.61) exceeds the full model's (17.01). Eq. 11 states an intent to preserve general capabilities, but no memory-on/off general-task results or preservation-loss ablation are reported.

Questions for authors
  1. How much does performance change when the complete conversation is written once and all subsequent questions are answered, compared with gold-evidence conditioning? If this cannot be run, will the abstract be narrowed to supplied-evidence recall?
  2. What are the results for simple state carry/normalization and for Metis under matched QA training, total persistent state bytes, and write/read budgets?
  3. What remains of the carrier's benefit once stabilization is separated, and how sensitive are age-stratified retention and later writes to the first state's norm?
  4. Does the KL preservation loss (Eq. 11) measurably protect general capabilities, and how much memory-induced behavioral drift remains?
Reviewer_LM4tOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STAMP addresses the limitation in hybrid attention-recurrent models, which lack a mechanism to persist and reuse recurrent states across sessions. In STAMP, a learned write mechanism encodes future-useful information into terminal recurrent states, and a lightweight state adapter maps them into initial states for subsequent sessions. Also, STAMP can be trained end-to-end on multi-session trajectories, with no per-session optimization required at inference. Experiments show STAMP's transferability and improved cross-session recall, while keeping the memory footprint fixed.

Critical strengths
  1. The "session-boundary gap" in hybrid linear attention is reasonably identified with numerical evidence.
  2. The solution via a learned cross-session interface is simple and practical, with a lightweight 17M/118M parameter footprint.
  3. Evaluation covers 4 benchmarks, relevant baselines, 2 model sizes, and diverse conditions including static/continual/distractor settings.
  4. Writing is clear and the story is clean.
Critical weaknesses
  1. STAMP is trained on MS MARCO-style data while Metis and others use original paper checkpoints. Thus, the impact of differences in training data and budget on results is unclear.
  2. In Fig. 3, the longest condition in static evaluation is around evidence + 2K tokens, so behavior under realistic long-term conversation history lengths remains unverified.
  3. Performance degrades significantly after 8 writes, yet on LongMemEval Metis improves as writes increase (4B: 15.16→17.24). There might be a possible reversal beyond 8 writes. The authors should provide the analysis of this for appropriate applicability.
  4. Could the main tables also include the performance of practical upper bounds such as full context and summarization-based alternatives?
  5. No ablations are provided for the three losses and curriculum, which leaves the training objective formula unsupported by experiments.
Questions for authors

Please see above.

2 comments

Thank you for your clarification and additional experiments. Your new results help address many of my concerns. However, I still think the design choices—in particular, the role and effect of each component (native recurrent state, learned write, and adapter)—are not well validated/clarified due to insufficient controlled comparisons. I hope this can be more thoroughly justified in the revision.

Overall, I believe the strengths outweigh the weaknesses, so I will raise my score to Weak Accept.

Dear Reviewer LM4t,

Thank you for your thoughtful feedback. For the revised manuscript, we are conducting further controlled ablation experiments to clarify how the native recurrent states and the learned write mechanism contribute to STAMP. We aim to publish our next release as soon as possible, with new experimental results and open-source checkpoints for models of different scales.

Best regards,

The Authors

Reviewer_XFGmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

STAMP learns a cross-session interface for frozen hybrid attention-recurrent backbones. Learned write tokens and write-only LoRA prepare terminal native recurrent states, while a low-rank carrier and norm-controlled carry rule turn them into initializers for later calls after source text and attention caches are removed. The interface is trained on MS MARCO-derived multi-session context–QA trajectories and reused across four evaluation suites without benchmark-specific retraining. Results include question-selected static memory construction, short continual-write trajectories, retrained component ablations, and post-write state compression.

Critical strengths
  1. The combined interface makes frozen native recurrent states useful across independent calls. Source tokens and attention caches are removed, and write-LoRA is disabled during answering. On LoCoMo, the full interface improves over raw carry from 7.55 to 25.40 static F1 and from 8.87 to 15.87 F1 in the ten-segment diagnostic, using the same native state substrate (Tables 4/7). This establishes the usefulness of the complete write-and-carry design.
  2. The interface transfers with meaningful state-only gains. At 4B and 27B, static macro F1 exceeds backbone-matched Metis by 6.15 and 12.19 points, respectively (Table 1).
Critical weaknesses
  1. The carrier ablation confounds low-rank adaptation with stabilization. Appendix C.1 states that removing the carrier also removes deployment-time stabilization, so Tables 4/7 cannot isolate the contribution of the low-rank transforms in Eq. 9. Figure 4 varies stabilization while retaining the carrier. Without identity or scalar-only carrier controls using the same SLERP rule, the independent benefit of the proposed key/value-axis adaptation remains unclear.
Questions for authors
  1. Can you compare the full carrier with identity and layer-wise scalar-only carriers under the same retraining protocol, extractor architecture, and SLERP rule? This would clarify the independent benefit of the low-rank adaptation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.