acceptodds
Under review as a conference paper at ICLR 2027

LSTMem: Hierarchical Long Short-Term Online Memory for Large Language Models

Abstract

Large language models increasingly serve as long-horizon assistants and agents, where they must both accumulate information across interactions and make the relevant parts available when later requests depend on them. Existing compact online memories typically use a single persistent state both to accumulate history and to serve readout, so what the memory stores cannot be controlled separately from what it exposes to the current computation. We propose LSTMem, an LSTM-inspired online memory that instead equips each layer of a frozen LLM with two matrix-valued states: a cell state that accumulates history and a hidden state whose readouts correct the backbone's attention. Input and forget gates control what the cell stores, while an output gate separately controls what the cell exposes through the hidden state. LSTMem further connects memory across depth through forward hidden-state propagation and block-end feedback, and uses higher-layer reconstruction gradients to refine lower-layer cell states before rebuilding hidden states from shallow to deep layers. Across memory benchmarks on Qwen3-4B-Instruct, LSTMem consistently improves MemoryAgentBench, LoCoMo, and HotpotQA over the plain backbone. Comparisons further show that the LSTM-based memory formulation outperforms an associative-memory counterpart, while removing cross-layer hidden-memory propagation degrades performance. These results demonstrate the benefits of separating memory accumulation from memory expression and organizing memory hierarchically across model depth.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM memory, LSTM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_31NqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces LSTMem, where they add a two matrix-valued states to each layer of a frozen LLM. Comparisons on memory benchmarks show that the proposed method outperforms an associative memory counterpart.

Critical strengths

The idea of having a skip connection from the previous layer's matrix state to the current layer is interesting. On the TTL task, it seems to show benefit.

Critical weaknesses

Motivation unclear. The intro/abstract says: "Keeping the full history in context becomes increasingly costly as interactions accumulate," "Compact online memory offers another way to retain history in a fixed-size state that persists across interactions", but it seems that the method retains all KV cache.

Rather, the motivation seems closer to: add small tunable architecture blocks on top of a frozen LM and do SFT to train those parts, to improve downstream performance. This is far too general, and there are too many possible methods to achieve this, including simply fine-tuning the backbone; adding a few more attention layers and fine-tuning those; to add a few SSM blocks and fine-tuning them. These simple, standard, and more scalable approaches were not mentioned or compared against.

Questions for authors

Shouldn't having two separate matrix states be compared against simply inserting SSM layers with 2x more heads for a fair comparison in terms of memory usage during inference? Retrofitting with inserted SSM blocks is common, eg https://arxiv.org/abs/2508.15884.

In Tab. 4, benefits seem to concentrate on TTL but not on other tasks. Can you explain why, and why despite this the result is significant?

Reviewer_Dt9uOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LSTMem augments a frozen Transformer with a cell matrix that accumulates history and a hidden matrix that supplies low-rank attention corrections at each memory layer. Gated updates, forward propagation across layers, and reconstruction feedback at history-block boundaries govern these states. The adapter is trained on QASPER and synthetic long histories, then evaluated on three backbones using memory benchmarks and, for Qwen3-4B, general-capability tasks. The paper reports improved aggregate memory scores, with evaluation retaining the historical KV cache.

Critical strengths
  1. The explicit hidden formulation and feedback recurrence provide a concrete design for coordinating memory accumulation, readout, and communication across model depth.
  2. Under matched two-stage training, LSTMem improves LoCoMo F1 over δ-Mem from 50.06 to 53.60, providing evidence for the complete adapter's practical value. On Qwen3-4B, the 36 KiB of persistent memory state and 5.9%–6.5% additional end-to-end time over δ-Mem for 2,048-token answers make this a potentially useful addition to a frozen backbone within the measured setting.
Critical weaknesses
  1. Appendix C.5 retains historical text and its KV cache during evaluation, leaving recall after history eviction untested. The reported gains therefore establish benefits when history remains accessible, without showing how much information the stored C/D states can recover on their own.
  2. On Qwen3-4B, long-range understanding falls to 37.59 from 47.08 for the backbone and 41.55 for δ-Mem, while accurate retrieval contributes 6.16 of the 6.23-point MAB gain (Table 1). This concentration of gains and the regression on long-range understanding limit the method's usefulness across memory tasks.
  3. None of Tables 4–7 isolates cell/hidden separation, and the depth ablation in Table 4 also removes feedback. These controls cannot establish the independent benefit of state separation or attribute the MAB depth-ablation gains specifically to forward propagation.
  4. Table 1 compares LSTMem trained on QASPER and Long with δ-Mem trained only on QASPER; the matched second-stage comparison is reported only for LoCoMo (Table 6). The MAB and HotpotQA margins therefore do not isolate architectural improvements from additional training.
  5. Appendix D reports 5.9%–6.5% end-to-end overhead with 2,048-token answers, whereas HotpotQA and LoCoMo cap answers at 32 and 50 tokens. Since longer answers amortize the cost of feedback during history processing, these timings do not establish the overhead for those short-answer workloads.
Questions for authors
  1. For the Qwen3-4B δ-Mem checkpoint after Stage 2 in Table 6, what are the MAB average and HotpotQA F1 under the Table 1 evaluation protocol?
  2. Beyond the complete-system comparison in Table 6, is there a controlled comparison that isolates C/D separation from the changed write rule, depth coupling, and feedback?
Reviewer_GiLwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LSTMem, an LSTM-inspired online memory adapter for frozen language models. Each memory layer maintains a cell state for accumulating history and a separately gated hidden state for memory readout, which produces low-rank corrections to attention queries and outputs. The method couples memories across depth through hidden-state propagation and introduces block-end feedback that refines lower-layer cells using higher-layer reconstruction errors. Experiments on MemoryAgentBench and HotpotQA cover three backbones, with additional LoCoMo and general-capability evaluations on Qwen3-4B-Instruct. Ablations examine depth coupling, memory placement, training stages, and feedback, while efficiency measurements quantify the incremental inference cost.

Critical strengths
  1. Separating historical accumulation from memory expression is well motivated. Combining this distinction with cross-layer propagation and reconstruction-based feedback provides an interesting extension of existing online memory adapters.

  2. The empirical evaluation covers multiple backbones and complementary tasks. LSTMem improves average memory-benchmark performance, while largely preserving the evaluated general capabilities on Qwen3-4B-Instruct.

  3. The implementation and feedback procedure are described in detail. The matched second-stage training comparison on LoCoMo and end-to-end efficiency measurements strengthen the experimental evidence.

Critical weaknesses
  1. Appendix C.5 states that evaluation retains the historical KV cache, so the backbone can still attend directly to the history. These results establish the usefulness of memory-based attention corrections, but do not isolate what the compact state retains independently. A memory-only or restricted-context evaluation would clarify its value beyond explicit context. The 36 KiB figure describes the additional persistent state, rather than total inference memory.

  2. Most baseline results are imported from prior work, and LSTMem additionally receives long-history training. Table 6 usefully controls this difference on LoCoMo, but equivalent comparisons are absent for MemoryAgentBench and HotpotQA. Repeating the matched-training comparison on these benchmarks would strengthen architectural attribution. Confidence intervals would also help assess the smaller component gains, particularly given the reported retraining variability.

  3. The ablations support depth coupling and feedback, but do not directly isolate the central accumulation–expression separation. Comparisons with δ-Mem change several mechanisms simultaneously, while the depth ablation removes both coupling and feedback. A controlled variant that ties the states or removes the output gate, with comparable training and parameter capacity, would provide more direct evidence for this design choice.

  4. Improvements are not uniform across memory capabilities. On Qwen3-4B-Instruct, long-range understanding falls from 47.08 to 37.59, and accurate retrieval contributes approximately 6.16 of the 6.23-point average improvement over δ-Mem. The paper acknowledges this tradeoff, but further analysis would help explain whether it reflects capacity limitations, training specialization, or interference from memory corrections.

Questions for authors

How sensitive are performance and feedback stability to memory-head width, history-block length, and the number of sampled reconstruction positions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.