- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
I provide a review of “Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers”. This paper suggests the research question that storing information does not necessarily mean storing useful information. It means that CoTFormer can store information in the KV cache, but old information is not useful and harms performance. The authors have revealed this using the Rule 30 cell automaton problem and suggest metrics for depth extrapolation and delayed recall. The authors also suggest that LSTM-like cells address this problem by smarter handling of stored information.
Critical strengths
This paper suggests a very important problem of remembering in transformers:
- The proposed metric for remembering is very informative, and the experiments can capture the usefulness of stored information.
- The proposed method has extensive improvements beyond CoTFormer.
Critical weaknesses
I think the research question is very impressive; however, the presentation, experiments, and research design have several problems.
-
This is very important. I think the experiments and explanation of this paper do not strongly support the proposed method. LSTM-like methods are known to be weak for long-term remembering tasks, and Mamba and other methods address this long-term remembering problem. In this research context, introducing an LSTM-like cell is not naive. I hope the authors will try other Transformer variants and Mamba variants to show that this improvement is a gain of the proposed method.
-
The authors should add other tasks to make the results robust. Other automaton tasks or other long-term remembering tasks, and they should try two metrics.
-
The presentation seems to come from an LLM, as the authors stated in the AI use statement. It is not bad; however, too many numeric and dense explanations in the main text are hard for human readers.
-
Related work should be extended. Only 8 references are too few in this research context.
-
No reproducibility statement. It is not mandatory, but the authors release code, and this should be stated in this section.
Questions for authors
Please see the weakness. I request additional comparisons with existing methods and additional tasks. I also request that the authors explain why an LSTM-like cell is better than other mechanisms that employ linear transformers or Mamba.
Thank you for the review, and for the encouragement about the research question.
1. LSTM-like cells are known to be weak for long-term memory; compare with Mamba and other variants. The recurrence here is over depth, not over the sequence: the cell carries a state from one application of the block to the next, at each token, for 12 to 60 steps. LSTMs are weak, and Mamba strong, along sequences of thousands of tokens. We are not aware of a state-space model whose recurrence runs over depth;.
The revision also no longer attributes each gain to the LSTM cell as such. We removed parts of LSTM-UT to find what each task needs (Section 3, Table 1):
So for stable rollouts the LSTM cell is not special; a decaying state is. For recall, input-dependent gates help and fixed decays do not (Table 2). LSTM-UT remains the only model here that leads on both. A fixed-size state limits recall in any of these models (Jelassi et al., 2024, prove it for state-space models), and ours agree: no model recalls the oldest rows (Figure 2). We did not train a full Mamba block.
2. Add other automaton tasks or other long-term remembering tasks. Few elementary automata can pose this task. It needs a rule whose rows keep changing, that keeps different inputs apart, that spreads a one-cell error, and that has no closed form. We checked all 256 rules on our setting (64 periodic cells, 60 steps), which form 88 classes under reflection and complement:
Rule 30 is the standard chaotic member of that set. Rules 45 and 106 would be valid further tasks and we did not train on them; other tasks remain the main limitation (Section 7).
3. The text is dense, with too many numbers. We rewrote the introduction, added a Models section that defines every variant in one place, and gave each results paragraph a heading that states its finding. The results still carry many numbers; we kept those a reader needs to check a claim.
4. Related work is too short. The paper now has 58 references.
5. No reproducibility statement. Added after the conclusion. It points to the data seeds, training settings, checkpoint-selection rules and the definitions of every model and diagnostic, and commits to releasing the code and the scripts that produce each table and figure.