- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces LSTMem, where they add a two matrix-valued states to each layer of a frozen LLM. Comparisons on memory benchmarks show that the proposed method outperforms an associative memory counterpart.
Critical strengths
The idea of having a skip connection from the previous layer's matrix state to the current layer is interesting. On the TTL task, it seems to show benefit.
Critical weaknesses
Motivation unclear. The intro/abstract says: "Keeping the full history in context becomes increasingly costly as interactions accumulate," "Compact online memory offers another way to retain history in a fixed-size state that persists across interactions", but it seems that the method retains all KV cache.
Rather, the motivation seems closer to: add small tunable architecture blocks on top of a frozen LM and do SFT to train those parts, to improve downstream performance. This is far too general, and there are too many possible methods to achieve this, including simply fine-tuning the backbone; adding a few more attention layers and fine-tuning those; to add a few SSM blocks and fine-tuning them. These simple, standard, and more scalable approaches were not mentioned or compared against.
Questions for authors
Shouldn't having two separate matrix states be compared against simply inserting SSM layers with 2x more heads for a fair comparison in terms of memory usage during inference? Retrofitting with inserted SSM blocks is common, eg https://arxiv.org/abs/2508.15884.
In Tab. 4, benefits seem to concentrate on TTL but not on other tasks. Can you explain why, and why despite this the result is significant?