acceptodds
Under review as a conference paper at ICLR 2027

Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers

Abstract

Recurrent-depth Transformers reuse one block across many steps, so information needed later must survive repeated rewriting of the hidden state. A natural remedy is to keep more history. We show that, in controlled cellular-automaton tasks, making history available is not the same as making it usable. Using Rule 30, where the correct state is known at every recurrent step, we test depth extrapolation and delayed recall, the recovery of an earlier state after further computation. CoTFormer, which caches keys and values from every earlier step, extrapolates less far and recalls less accurately than a Block Universal Transformer (BUT) that keeps only its current state. Interventions show that its retained history can pull a corrected trajectory back toward failure, and that the cache block written at the requested step is neither necessary nor sufficient for recall. We introduce LSTM-UT, which adds a small, bounded, gated cell state to the shared block. Trained to depth 12, LSTM-UT keeps \(99.7%\) exact-row accuracy at depth 60, where BUT gets no row fully correct, and one checkpoint stays above \(99.95%\) at depth 1,000. It also improves delayed recall over both baselines, and the advantage largely persists at near-matched parameter counts. On these tasks, a small state under learned control proved more useful than a complete but unaddressed history. In OpenWebText2 language modelling, LSTM-UT outperforms BUT and comes within 0.6 perplexity of a parameter-matched CoTFormer, which needs up to 91% more training time per step.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Machine Learning, Recurrent-Depth Transformers, Universal Transformers, Gated Memory, Cellular Automata, Depth Extrapolation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_91mLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

I provide a review of “Storing Is Not Remembering: LSTM-UT and Bounded Gated Memory for Looped Transformers”. This paper suggests the research question that storing information does not necessarily mean storing useful information. It means that CoTFormer can store information in the KV cache, but old information is not useful and harms performance. The authors have revealed this using the Rule 30 cell automaton problem and suggest metrics for depth extrapolation and delayed recall. The authors also suggest that LSTM-like cells address this problem by smarter handling of stored information.

Critical strengths

This paper suggests a very important problem of remembering in transformers:

  1. The proposed metric for remembering is very informative, and the experiments can capture the usefulness of stored information.
  2. The proposed method has extensive improvements beyond CoTFormer.
Critical weaknesses

I think the research question is very impressive; however, the presentation, experiments, and research design have several problems.

  1. This is very important. I think the experiments and explanation of this paper do not strongly support the proposed method. LSTM-like methods are known to be weak for long-term remembering tasks, and Mamba and other methods address this long-term remembering problem. In this research context, introducing an LSTM-like cell is not naive. I hope the authors will try other Transformer variants and Mamba variants to show that this improvement is a gain of the proposed method.

  2. The authors should add other tasks to make the results robust. Other automaton tasks or other long-term remembering tasks, and they should try two metrics.

  3. The presentation seems to come from an LLM, as the authors stated in the AI use statement. It is not bad; however, too many numeric and dense explanations in the main text are hard for human readers.

  4. Related work should be extended. Only 8 references are too few in this research context.

  5. No reproducibility statement. It is not mandatory, but the authors release code, and this should be stated in this section.

Questions for authors

Please see the weakness. I request additional comparisons with existing methods and additional tasks. I also request that the authors explain why an LSTM-like cell is better than other mechanisms that employ linear transformers or Mamba.

1 comment

Thank you for the review, and for the encouragement about the research question.

1. LSTM-like cells are known to be weak for long-term memory; compare with Mamba and other variants. The recurrence here is over depth, not over the sequence: the cell carries a state from one application of the block to the next, at each token, for 12 to 60 steps. LSTMs are weak, and Mamba strong, along sequences of thousands of tokens. We are not aware of a state-space model whose recurrence runs over depth;.

The revision also no longer attributes each gain to the LSTM cell as such. We removed parts of LSTM-UT to find what each task needs (Section 3, Table 1):

  • Linear memory is a diagonal linear recurrence with a learned constant decay per channel and no gates; with an input-dependent decay this is the recurrence inside Mamba. It extrapolates as well as LSTM-UT.
  • Decay BUT has no memory cell at all, only a fixed decay on the stream, and also extrapolates.

So for stable rollouts the LSTM cell is not special; a decaying state is. For recall, input-dependent gates help and fixed decays do not (Table 2). LSTM-UT remains the only model here that leads on both. A fixed-size state limits recall in any of these models (Jelassi et al., 2024, prove it for state-space models), and ours agree: no model recalls the oldest rows (Figure 2). We did not train a full Mamba block.

2. Add other automaton tasks or other long-term remembering tasks. Few elementary automata can pose this task. It needs a rule whose rows keep changing, that keeps different inputs apart, that spreads a one-cell error, and that has no closed form. We checked all 256 rules on our setting (64 periodic cells, 60 steps), which form 88 classes under reflection and complement:

  • 72 classes become periodic, 69 of them in at least 99% of rollouts.
  • The additive rules (60, 90, 105, 150) have a closed form at any depth.
  • Others merge inputs or lose balance (Rules 18, 22, 126), and Rule 110 spreads errors more weakly and erases some.
  • Three classes pass every check: Rules 30, 45 and 106.

Rule 30 is the standard chaotic member of that set. Rules 45 and 106 would be valid further tasks and we did not train on them; other tasks remain the main limitation (Section 7).

3. The text is dense, with too many numbers. We rewrote the introduction, added a Models section that defines every variant in one place, and gave each results paragraph a heading that states its finding. The results still carry many numbers; we kept those a reader needs to check a claim.

4. Related work is too short. The paper now has 58 references.

5. No reproducibility statement. Added after the conclusion. It points to the data seeds, training settings, checkpoint-selection rules and the definitions of every model and diagnostic, and commits to releasing the code and the scripts that produce each table and figure.

Reviewer_uGjkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes LSTM-UT, a method that combines looped transformers with depth-wise LSTMs. On Rule 30 cellular automata, LSTM-UT shows better depth extrapolation than the baselines: CoTFormer and the vanilla looped transformer.

Critical strengths

Depth-wise LSTMs, to my knowledge, have not been tried. Depth extrapolation is important for looped models, which they argue that the proposed depth-wise LSTM architecture helps.

Critical weaknesses

Missing prior work. The idea that depth can be viewed as a sequence dimension, and thus the sequence modeling idea can be applied to the depth axis, is not new. See "DeepCrossAttention: Supercharging Transformer Residual Connections" and "Attention Residuals," for instance. Within this view, applying LSTM, a kind of sequence model, to the depth axis seems natural. However, I'm not convinced that depth-wise LSTM is better than depth-wise attention explored in these prior works.

Lack of analysis. Why should adding LSTM to the depth improve depth extrapolation?

In Fig. 1(a), the CoTFormer baseline appears undertrained. Its accuracy within the grey area is already worse. It's unclear whether CoTFormer fails to depth-extrapolate or it just underfits train distribution due to suboptimal implementation.

"DELAYED CELLULAR-AUTOMATON RECALL" I don't see why this task is meaningful. Hidden activations of a deep neural net need not correspond to some concrete information you can later retrieve. And why does LSTM-UT not fail in a similar way to CoTFormer does?

Fig. 3 (left): The ranking between BUT and LSTM-UT seems to flip multiple times during training. How do we know whether the ranking will flip again if we had trained longer?

Questions for authors

See weaknesses.

1 comment

Thank you for the review. The second weakness you raised, the lack of analysis, is what the revision is now built around.

1. Missing prior work on depth as a sequence axis. We now cite DeepCrossAttention, Attention Residuals and related work, and place LSTM-UT within the Depth Memory framing of explicit mixers (attention over a growing cache) and implicit mixers (a fixed-size state) (Section 2.2). We agree that a depth-wise LSTM is a natural idea in that view, and we do not claim it is better than depth-wise attention in fixed-depth models, where every depth is seen in training; the paper now says this explicitly. Our setting is different: a loop run past its trained depth, where a growing history must be read at depths never trained on.

2. Why should an LSTM across depth improve extrapolation? Because its forget gate makes the carried state decay, and decay makes errors contract. We show this by intervention, not by comparison of architectures:

  • Multiplying the plain block's residual stream by a fixed 0.9 each repeat, with identical parameters, turns failure at depths 18 to 31 into exact rollouts at depth 60 in every seed (Table 1).
  • Removing the forget gate from LSTM-UT leaves two of five seeds reaching depth 60.
  • The measured error contraction rate of the decayed block is -0.098 per repeat against ln 0.9 = -0.105, so nearly all of it comes from the decay.
  • Gates, saturation, input re-injection and extra parameters are each shown not to be needed.
  • One exception is reported: a gated stream with the same fixed decay contracts small errors yet reaches depth 60 in only two of five seeds (general response; Table 1, Appendix Table 24).

The plain block's update is the identity plus a learned term, so directions the block does not act on keep multiplier 1 and errors along them never shrink. A decay replaces that 1 with a number below 1 (Section 4.3).

3. CoTFormer in Figure 1 looks undertrained. You are right that it does not fit the supervised depths, and we now treat this as two separate findings. First, CoTFormer is hard to fit on this task: with five seeds under the same protocol it fits the trained depths in one (Appendix B). Second, where it fits it does not extrapolate: no fitted run stays above 95% for more than three steps past its trained depth. The cause is the cache and not our implementation: the same model with its attention cut to the most recent repeat fits in every run and lasts four to eight steps past its trained depth. This holds over every training range we tried (Appendix B.1).

4. Why is delayed recall a meaningful task? We agree that hidden activations in general need not hold retrievable information. In this task they are trained to: every repeat is supervised to decode to its automaton row, so the state the model is later asked for is one it was trained to compute, and the question is only whether it can still retrieve it after further steps. The motivation is world models, which carry a recurrent state so that information survives many rollout steps (Section 1). As to why LSTM-UT does not fail as CoTFormer does: the revision finds that input-dependent gating is what helps, with or without a cell (point 1 in the general response), and that CoTFormer's difficulty is largely one of fitting its rollout at longer horizons.

5. Figure 3 (left): the ranking between BUT and LSTM-UT flips during training. You are right, and at matched parameter count the two are not distinguishable. That panel (Appendix Figure 10, left, in the revision) compares width-384 LSTM-UT with a width-408 BUT. The gap between them changes sign several times, with swings of up to 0.9 perplexity, and ends at 0.23; we do not claim a difference there.

The comparison the text relies on is at equal width, and there the ranking never flips. At equal width LSTM-UT has 12% more parameters, and with one seed the matched-parameter comparison is inconclusive. The summary of contributions now says LSTM-UT "is ahead of BUT" in place of "beats BUT", and the paper states that differences of a few tenths are indicative only.

Reviewer_zXsfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper shows that an expanding depth-wise KV cache in looped Transformers (CoTFormer) fails at Rule 30 depth extrapolation and delayed recall, underperforming even a memoryless looped block (BUT). The authors propose LSTM-UT, adding a small gated cell state that extrapolates much further, improves recall, and avoids CoTFormer's cache overhead in OpenWebText2 pretraining.

Critical strengths
  1. The causal interventions on CoTFormer clearly show that storing latent history is not the same as using it: masking the queried cache block barely changes recall, and stale cached states pull clean hidden states back toward failure.

  2. Adding a small gated cell works well on chaotic Rule 30 dynamics, extrapolating from depth 12 to 60+ and beating a near-parameter-matched BUT on delayed recall with constant memory overhead.

Critical weaknesses

1.It is unclear whether LSTM-UT really needs a separate cell state or just depth-wise gating; comparing against a simpler GRU or highway-gated residual loop (and recent persistent-state loops like MeSH) is needed to isolate the gain.

  1. Unlike the CoTFormer analysis, LSTM-UT's cell state is treated as a black box, with no causal checks (such as zeroing or freezing the cell before recall) to verify whether it stores past states or just stabilizes residual drift.

  2. A few protocol choices cloud the baseline comparisons: Rule 30 checkpoints are picked using validation accuracy at an unseen depth (K + 3), delayed recall stops at 5,000 steps before CoTFormer's forward pass converges, and the OpenWebText2 runs use only one seed.

Questions for authors
  1. How does LSTM-UT compare against a simpler GRU or highway-gated residual loop without a separate cell state?

  2. What happens to delayed-recall accuracy if you zero out or freeze the cell state right before the query step?

  3. Do the Rule 30 extrapolation gains hold when checkpoints are selected strictly on trained depths (1 to K), and does CoTFormer's recall improve if trained until its forward pass converges?

1 comment

Thank you for a review that asked exactly the questions the paper needed.

1. Is the separate cell needed, or only depth-wise gating? We added the highway-gated loop you suggested ("gated BUT": the same gate inputs as LSTM-UT, no cell). The two tasks separate cleanly:

  • On delayed recall the gate is what matters. At horizon 20 gated BUT recalls 70.5% of rows exactly and LSTM-UT 67.6%, against 53.8% for the plain block (Table 2). The cell is not needed for recall.
  • On depth extrapolation the gate does not help. Gated BUT stops where the plain block does (reach 19 to 23 against 18 to 31), and every seed is at 0% exact rows at depth 60 (Table 1).

What extrapolation needs is a decaying carried state, and a cell is one way to get it but not the only one: a fixed decay on the plain block's stream is enough. So the cell is needed for neither task taken alone. Whether it is needed to get both in one model is open. The simplest combination without a cell, a fixed decay on the gated stream, does not manage it: gated decay BUT reaches depth 60 in two of five seeds and its recall at horizon 20 falls to the plain block's (general response; Table 1, Table 2, Appendix Table 24). LSTM-UT is the only model here in the leading group on both tasks, and the paper says the two ingredients alone are not enough (Section 7). We cite MeSH and persistent-memory recursive Transformers as related designs but did not train them.

2. The cell was a black box. What happens if it is zeroed or frozen before the query? We ran these on the six 25,000-step LSTM-UT checkpoints (Section 5.3).

  • Zeroing the cell just before the recall step costs 2.8 and 5.6 cell points on average but takes exact rows from 86.3% to 58.6% (horizon 12) and from 67.6% to 38.7% (horizon 20). The current-row query, which needs no memory, stays at 99%.
  • Swapping the cell between two unrelated inputs shows what it holds. At some ages the answer follows the donor input: in one horizon-20 model the row from two steps back is the donor's row exactly for 99.6% of test inputs (Figure 3). At other ages the answer ignores the foreign cell, so those rows are carried by the residual stream.
  • Freezing the cell or closing the output gate also breaks the current-row query, so it does not isolate memory (Appendix C.3).

So the cell does store earlier rows, for some ages, and which ages differs between trained models. In the extrapolation models, which are never asked to recall, the cell behaves differently: it is a fast-decaying channel read at every step (Section 4.3).

3a. Checkpoints were selected at an unseen depth. We repeated the comparison selecting on trained depths only (Appendix Table 9). LSTM-UT selects the same checkpoint in 5 of 5 seeds and reaches depth 60 either way. BUT reaches 18 to 24 under in-distribution selection against 18 to 31, and gated BUT 18 to 23 against 19 to 23. Every BUT and gated BUT seed is at 0% exact rows at depth 60 under either rule.

3b. Recall stopped at 5,000 steps, before CoTFormer converged. We retrained every model for 25,000 steps (Table 2). CoTFormer gained 22 to 35 cell points and remains last in exact rows at both horizons (52.1% and 32.2%). Longer training did not make it fit: at the last trained depth its rollout gets 27.6% of final rows exactly right at horizon 12 and none at horizon 20, against 68 to 93% for BUT, LSTM-UT and gated BUT. We now report this fitting difficulty as a result in its own right. LSTM-UT's lead over the plain block also shrank with training, from 10.7 cell points to 1.8 at horizon 12; it remains 9.5 and 13.9 exact-row points at horizons 12 and 20.

3c. One seed for language modelling. This is unchanged, and the paper says differences of a few tenths are indicative only.

A correction that concerns a strength you listed. You noted that masking the queried cache block barely changes recall. That test addressed the wrong block.

At repeat q the controller adds its step embedding, the keys and values are cached, and only then do attention and the MLP compute row q. So the block that carries repeat q's signature holds the row before update q. The completed row q is written to the cache one repeat later, in block q+1, and that is the block CoTFormer uses: at horizon 6, in two of three seeds, giving block q+1 another input's values takes recall from 100% to chance at most queries. The third seed answers without the cache.

At horizon 12 no well-fitted checkpoint depends on block q or q+1, and at horizon 20 no seed fits its rollout, so nothing can be concluded there (Section 5.3, Appendix B.3). "Neither necessary nor sufficient" is replaced by "looked up at horizon 6 and not at horizon 12". The finding that stale cached states pull a corrected state back toward failure on Rule 30 used a different intervention and is unaffected.

Comments on the paper

1 comment

We thank the three reviewers. Their questions changed the paper more than a normal revision would, so we first say plainly what is different.

What the paper now claims. The submitted version presented LSTM-UT as a better memory than CoTFormer's cache. Reviewers zXsf and uGjk asked which part of LSTM-UT does the work and why it should help extrapolation, so we took the model apart. Each ability has a simpler source, and the two do not simply add:

  1. Decay gives stable rollouts. Trained on depths 1 to 12, all 30 seeds of six model families with a decaying state are exact at depth 60, and all 15 seeds of three families without decay stop between depths 18 and 31 (Table 1). A fixed decay of 0.9 on the plain block's stream, with nothing else changed, is sufficient. The decaying models are the ones in which small errors contract (Section 4.3).
  2. Input-dependent gates help recall; no fixed decay does. At horizon 20 a gated stream with no cell recalls as well as LSTM-UT, and no fixed-decay model improves on the plain block (Table 2).
  3. LSTM-UT is the only model in the leading group on both tasks. The simplest combination without a cell, a gated stream with a fixed decay, does neither reliably (see below). In LSTM-UT the decay is itself input-dependent, through the forget gate.

The paper keeps LSTM-UT as its model; the title is now "LSTM-UT: Bounded Gated Memory for Looped Transformers".

What was added.

  • New models and controls under the same protocol (Section 3, Appendix D):
    • gated BUT (a highway gate, no cell), and the same with a fixed decay;
    • LSTM-UT without its forget gate, and with a linear read;
    • linear memory (a cell with a constant decay and no gates);
    • decay BUT at 0.9 and 0.5;
    • BUT with input injection;
    • the residual scaling of Fixed-Point Reasoners, with and without input re-injection;
    • capacity controls: wider BUTs and a narrower LSTM-UT;
    • CoTFormer with its cache restricted to recent repeats (Appendix B).
  • Lyapunov rates and perturbation experiments on the Rule 30 models (Section 4, Appendix D).
  • Delayed recall retrained for 25,000 steps instead of 5,000, for every model (Section 5).
  • Causal interventions on LSTM-UT's cell: zeroing, freezing, and swapping it between two inputs (Section 5.3).
  • Checkpoint selection on trained depths only (Appendix Table 9).
  • Related work extended to 58 references, and reproducibility and AI use statements.

A correction to the submitted version. The submission said that the CoTFormer cache block written at the requested step is "neither necessary nor sufficient for recall". That test addressed the wrong block. The block written during repeat q holds the state before update q; the completed row q is in block q+1. Testing block q+1, we find that CoTFormer does look the row up at the shortest horizon: at horizon 6, in two of three seeds, giving that block another input's values takes recall to chance. At horizon 12 the well-fitted checkpoints show no such dependence, and at horizon 20 no CoTFormer seed fits its rollout, so the test says nothing there. The claim is now "a stored history can be used as addressable memory, and we found that solution only where the task was easiest" (Section 5.3, Appendix B.3). The state-and-cache interventions on Rule 30 extrapolation are unchanged.

One family that does not fit the pattern. Reviewer zXsf asked whether gating without a cell is enough. Gated BUT answers that for a stream with no decay; we also trained it with decay BUT's fixed 0.9 on the stream ("gated decay BUT", five seeds). It is the one decaying family that does not reach depth 60 in every seed: two seeds are exact there, two fit and stop at depth 24, and one does not fit (41.3 ± 51.8% exact rows at depth 60; Table 1 and Appendix Table 24). The diagnostics do not explain it. It halves a small error in 7 repeats and contracts at -0.08 to -0.10 per repeat while its rows are exact, like decay BUT, and the two fitted seeds that stop keep roughly balanced rows instead of saturating (Appendix D.4, Appendix Table 3). On recall at horizon 20 it is level with the plain block, not with the gated models (53.0% exact rows against 53.8%; Table 2). So contraction of small errors goes with extrapolation in the other families but does not guarantee it, and the paper now says so (Section 4.3).

What we did not do. We did not add tasks beyond Rule 30 and language modelling, we did not train a Mamba or GRU variant, and the language-modelling runs still use one seed per configuration. These are stated as limitations (Section 7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.