acceptodds
Under review as a conference paper at ICLR 2027

Continuous Memory Machines

Abstract

Recurrent neural networks typically compress information into a single vector-valued recurrent state, forcing short-term computation and long-term retention to share the same representation. Past extensions alleviate this bottleneck by increasing the memory capacity or separating timescales, but lack the combination of rapid neuron-level processing and longer-term retention found in biology. To that end, we introduce the Continuous Memory Machine (CMM), a recurrent architecture with matrix-valued short- and long-term memory states serving distinct functional roles. Building on the Continuous Thought Machine (CTM), the CMM's short-term memory tracks recent neural activity, with uniquely parameterized neuron-level models learning to use these activity patterns for computation. A persistent long-term memory stores information for later use, with a Transformer jointly updating both memory stores, providing an expressive bidirectional read–write mechanism such that each store can reorganize its own contents and both read from and write to the other. Across algorithmic, in-context learning, and recurrent reasoning tasks, the CMM outperforms a broad suite of baselines, exhibiting stronger generalization than prior memory-augmented networks while preserving the CTM's interpretable attention patterns.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
long-term memory, memory-augmented neural networks, recurrent neural networks, memory architectures, biologically inspired AI, neural dynamics

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_mRBKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work straightforwardly extends the prior work on Continuous Thought Machine to include a "long-term memory" bank, similar to prior work on memory-augmented neural networks, that is indexed by a Transformer and used to update the "short-term memory" that is used as the input to the "neuron-level modules" function which produces the model's next state. The authors evaluate this architecture on a wide range of tasks including algorithmic sequence memory tasks (copy, recall, sort, parity), maze solving, few-shot learning, and classification, demonstrating strong performance across the evaluated tasks.

Critical strengths

Taking the prior CTM work as given, this paper represents a simple and performant extension to enhance its long-term memory capacity. The mechanism is intuitive and easy to understand, and has good empirical performance across a diverse set of tasks. The ablations are meaningful and thoughtfully designed, and the limitations are considered.

Critical weaknesses

Although it has good performance overall, CMM is not the absolute winner on many of the individual tasks, and there are some unclear results about the necessity/value of the LTM. For instance, CMM performs worse than its ablations on Copy, indicating that LTM is potentially unhelpful even for explicit memory manipulation tasks. Similarly, the Maze performance is not entirely convincing due to the extremely high variability and small number of seeds -- indeed, CMM-STM seems to outperform CMM on 2 out of 3 seeds (Table 9). Additionally, the memory overhead is much larger than competing methods, while the performance gains are only modest for some tasks, putting into question whether the overhead is worth it, and whether it's scalable.

A minor suggestion for the authors that would slightly improve readability is to group the algorithmic tasks bar graph in Fig 2 by task rather than by model -- this makes comparing models easier.

Questions for authors

Could the authors provide additional evidence, e.g. additional seeds, for the robustness of the Maze results, particularly the improvement of CMM over CMM-STM?

3 comments

We thank the reviewer for their assessment of our work. We are glad that the reviewer found the Continuous Memory Machine architecture easy to understand and recognised its good empirical performance across a broad range of tasks. We address their questions and concerns below.

W1. Relative Performance vs Baselines

The reviewer highlights that the CMM is not the absolute winner on every task, pointing to copy as an example. The ablations CTM-Ext and CMM-STM have extended short-term memories long enough to hold the entire input sequence. This makes sequential recall easier, since the entire input history remains available, much as it does for the causal Transformer. The CMM, with its much shorter STM, must instead learn to store and recall the sequence through its LTM.

The extended STM ablations therefore have an inductive bias well suited to this task. The CMM can also learn to copy accurately, with one seed achieving a loss close to CMM-STM. Extending STM also has a cost, since the number of parameters in the neuron-level models grows with the memory length at a fixed model width. The sort and parity results show that extending STM alone does not match the CMM when the task requires more than sequential recall.

W2. Variability in Maze Solving

The reviewer highlights the variance in maze performance and that CMM-STM outperforms the CMM in two of three seeds. We agree that full-maze accuracy varies considerably. This is a strict metric as a single incorrect action makes an otherwise correct solution count as incorrect.

All three CMM seeds achieve over 90% action accuracy. Figure 4 (bottom) shows that the CMM maintains long correct prefixes, with errors becoming more common near the end. On 100-action targets, accuracy over the first 75 actions is 96–99% across seeds. The CTM, on the other hand, shows much more variation over these earlier actions. This helps explain why full-maze scores vary so much even when most actions are correct. Additional maze seeds are still running, but are unlikely to finish before the end of the discussion period.

W3. Scalability

The reviewer raises concerns about memory overhead and scalability. We acknowledge the increased computational and memory costs as a limitation. As discussed in future work, one way to reduce this cost is to run the memory Transformer less frequently.

Although our experiments are already parameter matched, we also ran a sort comparison with matched training-time budgets, as suggested by reviewer FuaF. We trained the CMM and wider CTM-Ext and RMT models for 30 minutes per seed under a common memory limit. Models were trained on lengths 4–16 and evaluated on the same examples.

Results below show bit accuracy (%) as mean ± standard deviation across three seeds.

ModelLength 20Length 30Length 40Length 50
CMM95.1 ± 0.489.6 ± 0.581.7 ± 1.274.0 ± 1.5
CTM-Ext89.9 ± 0.164.0 ± 1.855.7 ± 0.353.6 ± 0.9
RMT92.1 ± 1.884.6 ± 3.276.1 ± 5.270.4 ± 4.4

The CMM achieves the highest mean accuracy at every unseen length, showing that its generalisation advantage on sort remains under these resource constraints. These results have been added to the paper (Appendix C.1, Table 8).

Regarding scalability more broadly, our motivation is to study how memory should be organised and what capabilities different memory mechanisms provide. We believe there is a distinction between developing useful computational building blocks and making them efficient at scale. For example, xLSTM revisits LSTM with architectural changes that improve performance and scalability [1]. If scalability is the main criterion from the outset, we risk overlooking ideas that could support future improvements.

We have also regrouped Figure 2 by task to make the model comparisons easier to read. We thank the reviewer for this suggestion.

We thank the reviewer again for their careful consideration of our work. If the reviewer believes we have addressed their critical concerns, we kindly ask them to consider raising their score.

[1] Beck, Maximilian, et al. "xlstm: Extended long short-term memory." Advances in Neural Information Processing Systems 37 (2024): 107547–107603.

Thank you for the detailed response and additional experiments, I appreciate the authors’ efforts to address my concerns. The additional results somewhat strengthen the paper, but my overall assessment remains, I will maintain my score.

As we wrap up the author-reviewer discussion period we would like to thank reviewer mRBK again for their helpful review of our work. It's much appreciated.

Reviewer_FuaFOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper extends the Continuous Thought Machine with a persistent long-term memory. In the CTM each neuron has its own small network over a sliding window of its recent pre-activations, and the model reads out from neuron-pair synchronization. The CMM keeps that window as a short-term memory and adds a matrix of long-term memory tokens that is carried across steps. At every step a single Transformer processes the LTM tokens, the STM tokens and a sink token whose value vector is set to zero, so each memory can read from and write to the other, or ignore both by attending to the sink. The updated STM then goes through the neuron-level models as in the CTM. The model is compared with ten recurrent and memory-augmented baselines, a causal Transformer and three ablations on copy, recall, sort, parity, sort length generalization, few-shot regression, 39×39 mazes and ImageNet-100, at matched parameter budgets, 500K by default, and three seeds. The authors also inspect where the Transformer's attention goes and report that the CMM uses its memories differently depending on the task.

Critical strengths

The architecture is a sensible combination of known parts, and the ablations are set up to test the right thing. Stripped down, the CMM is the CTM with an RMC-style slot memory, where the CTM's window and a set of persistent memory tokens go through one Transformer together. CTM-Ext. gives the CTM a window as large as the CMM's total memory, and CMM-STM keeps the Transformer but drops the LTM, so the Transformer and the LTM are added one at a time. On the two tasks where this matters, the LTM clearly helps. On sort the CMM has the lowest test loss of all models, several times below CMM-STM (Fig. 2). On few-shot regression with K=20K=20 it more than doubles the accuracy of both ablations, although their windows already cover the whole episode (Table 2, Appendix D.5). That is good evidence that a recurrently updated memory helps the computation itself and not only access to past inputs, and it is the main reason I lean toward acceptance.

Critical weaknesses

Outside sort and few-shot regression, the headline claims are stronger than the evidence. The abstract says the CMM "outperforms a broad suite of baselines" with "stronger generalization than prior memory-augmented networks". On the maze, board accuracy varies widely across seeds (Table 9), which the paper acknowledges. The CMM has two strong runs and one weak one, the CTM one strong and two weak, and the best CTM run beats two of the three CMM runs. By my calculation a Welch test gives p≈0.47p \approx 0.47, so the advantage is not established, although with three runs this cannot show that the models are equal either. The gain may also not be a memory effect. The ablation without an LTM is not clearly worse, STM queries attend only to the zero-valued sink (Fig. 6), and the paper suggests that the gains arise from "the expressivity of the memory Transformers' feed-forward networks". This does not show that memory played no role during training, but it is why I ask for the FFN-only control in Question 3. On few-shot regression the CMM ties RMT, on ImageNet-100 all models except the three with very large seed variance are within half a point, and on copy the two long-window ablations beat it. The generalization claim rests mainly on Fig. 3, where the lead over RMT at the longest lengths is about the size of the error bars and DNC is better inside the training range, and parity is reported only at the longest training length, although longer lengths were evaluated (Appendix D.4). I would ask the authors to narrow the abstract to what sort and few-shot regression show.

The comparison is parameter-matched but not compute-matched, and it is not clear how the memory sizes were chosen. On Copy-100 (Table 5), at about the same parameter count the CMM is about twice as slow per recurrent step, forward plus backward, as the CTM and needs about 35 times its peak memory. CMM-STM needs almost as much, so the cost comes mainly from running the memory-update Transformer at every step. Training memory is a known issue for recurrent memory-token models, and it is usually reduced by storing only the recurrent state and recomputing the rest during backpropagation, as in gradient checkpointing or the replay-based training of models such as RMAAT [R1]. The paper lists the cost as a limitation and suggests updating the memory less often (Section 5), but does not discuss whether something like this would apply to the CMM, so it is hard to tell how much of the memory cost is inherent to the design. The memory sizes also change from task to task (Table 7), the only sweeps reported (Fig. 9) are on Sort test loss, with two seeds and very wide error bars at several settings, and the paper does not say how the per-task values were picked. Some baseline details also need fixing. The maze footnote says GDN, RMT and LSAM "could not train stably", while Appendix D.7 says that of the three only RMT was unstable and GDN and LSAM were dropped for memory and runtime reasons. The ImageNet-100 backbone is "the first two blocks" of a frozen ViT in Section 4.4 but its penultimate block in Appendix D.6, which matters for reading a task where the leading models are within half a point of each other.

References:

[R1] Md Zesun Ahmed Mia, Malyaban Bal, and Abhronil Sengupta. RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers. In International Conference on Learning Representations (ICLR), 2026. https://openreview.net/forum?id=sTkJdbVxsI

Questions for authors
  1. Clarification: How were MLM_L, MSM_S, JJ and the synapse depth DD chosen for each task, and on which split?
  2. Clarification: Can the maze be rerun with more seeds? With outcomes this spread out across seeds (Table 9), the CMM–CTM difference cannot be separated from seed variation.
  3. Clarification: Since the maze model's STM queries attend only to the sink, does a memory update with the FFN alone, without attention or an LTM, reproduce the maze result? On copy, sort and few-shot regression, what happens if the long-term memory is zeroed at test time?
  4. Suggestion: A comparison at matched wall-clock time or memory, for example a wider CTM or RMT, would show whether the gains hold at equal compute.
  5. Suggestion: The paper says the CMM separates timescales (Section 3), but both memories pass through the same Transformer at every step, so it is not clear how different their effective timescales are in practice. A measurement of how long information persists in each memory would make the claim concrete, and it would help to position the CMM against recurrent memory-token models with explicit short- and long-term timescales, such as RMAAT [R1].

References:

[R1] Md Zesun Ahmed Mia, Malyaban Bal, and Abhronil Sengupta. RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers. In International Conference on Learning Representations (ICLR), 2026. https://openreview.net/forum?id=sTkJdbVxsI

6 comments

We thank the reviewer for their assessment of our work. We are glad that the reviewer found the paper clear, considered the proposed architecture sensible, and recognised the evidence that recurrent memory helps the computation itself. We address each of the questions and weaknesses below.

Q1. Hyperparameter Selection

The reviewer asks how MLM_L, MSM_S, JJ and the synapse depth DD were chosen for each task. In general, we kept the architecture as simple as possible.

  • DD (synapse model): For simplicity, we used a two-layer MLP for all tasks except maze. For maze, we used a U-Net synapse with depth setting D=16D=16, matching the original CTM paper [1].
  • JJ (memory Transformer depth): We used one layer by default, with the only exception being the sort experiment, where we used two layers because this improved performance (on freshly sampled evaluation examples). The depth sweep in Appendix B.1 and Figure 9 shows the effect of increasing J.
  • MLM_L (LTM size): Following the external-memory setup of [2], we chose enough capacity to store the input information on tasks such as copy and sort. For example, copy uses 128 LTM tokens for 100 input steps. Appendix B.1 and Figure 9 also examine how increasing LTM capacity affects sort performance.
  • MSM_S (STM length): We chose short windows informed by the original CTM paper [1].

Q2–Q3. Maze Variability and Control Experiments

We agree that full-maze accuracy varies considerably across seeds. However, this masks more consistent performance at the action level. The CMM achieves approximately 92% action accuracy or higher in all three seeds, compared with one of the three CTM seeds. Full-maze accuracy is much stricter, as a single incorrect action makes the entire solution count as incorrect.

On 100-action targets, accuracy over the first 75 actions is 96–99% across the CMM seeds, with much less variation than the CTM. Figure 4 (bottom) shows that errors become more common towards the end. The weaker full-maze seed therefore learns much of the route but struggles to complete it. These results suggest that the CMM learns good action-level solutions more consistently than the full-maze scores alone indicate. Additional maze seeds are still running, but are unlikely to finish before the end of the discussion period.

We do not attribute the maze gains to LTM. We see the flexibility of the CMM's memory system as a strength. As suggested, we ran an experiment zeroing the LTM or resetting it to its learned initial state before every memory update. This substantially degrades copy, sort and few-shot regression, while leaving maze accuracy unchanged across all three seeds. This provides further evidence that the CMM learns task-dependent memory strategies that can make use of the LTM.

TaskMetricZeroed LTM ChangeReinitialized LTM Change
CopyQuery MSE+4.30 ± 0.59+1.25 ± 0.64
SortSequence accuracy (%)-94.7 ± 0.4-94.7 ± 0.4
Regression (K=10)MSE+0.74 ± 0.006+0.73 ± 0.004
Regression (K=20)MSE+0.96 ± 0.006+1.01 ± 0.08
MazeFull-maze accuracy (%)0 ± 00 ± 0

Mean ± sample standard deviation across three seeds. Intervention columns show paired changes from intact, in percentage points for accuracy. LTM is zeroed or reset to its learned initial state before every memory update.

For the requested FFN-only control, we removed memory self-attention and LTM reads/writes while retaining FFN updates to STM, with the model weights fixed. This substantially degrades copy, sort and few-shot regression, while both action and full-maze accuracy remain unchanged on all 5,000 test mazes for each seed (Appendix B.3, Table 5).

TaskMetricFFN-only Change
CopyQuery MSE+4.27 ± 0.55
SortSequence accuracy (%)-94.7 ± 0.4
Regression (K=10)MSE+0.74 ± 0.01
Regression (K=20)MSE+0.97 ± 0.02
MazeAction accuracy (%)0 ± 0
MazeFull-maze accuracy (%)0 ± 0

Mean ± sample standard deviation across three seeds. FFN-only changes are paired against intact, in percentage points for accuracy. Models are evaluated without retraining on 1,024 examples per synthetic task and 5,000 test mazes per seed.

These experiments suggest that the CMM adapts its use of memory to the task, relying on LTM for copy, sort and few-shot regression while retaining strong maze performance with FFN-only memory updates. We see this flexibility as a strength of the architecture.

Q4. Comparison at Matched Compute

The reviewer asks whether the gains hold at matched wall-clock time or memory, including against a wider CTM or RMT. To perform a matched compute experiment, we trained each model for 30 minutes per seed on the same NVIDIA H100 GPU. CTM-Ext and RMT were widened within a common peak GPU-memory budget. All models were trained on lengths 4–16 and evaluated on the same examples.

Bit accuracy (%), mean ± standard deviation across three seeds:

ModelLength 20Length 30Length 40Length 50
CMM95.1 ± 0.489.6 ± 0.581.7 ± 1.274.0 ± 1.5
CTM-Ext89.9 ± 0.164.0 ± 1.855.7 ± 0.353.6 ± 0.9
RMT92.1 ± 1.884.6 ± 3.276.1 ± 5.270.4 ± 4.4

The CMM achieves the highest mean accuracy at every unseen length, showing that its generalisation advantage on sort remains under equal training-time and memory budgets. We thank the reviewer for this suggestion. The setup and results are in Appendix C.1, Table 8.

Q5. Separate Memory Timescales

The reviewer asks how memory timescales differ when both stores are updated at every step. We thank the reviewer for raising this point. Although the memory Transformer runs at every step, some tokens can change little while others change substantially, since attending to the sink can suppress the attention contribution, while the feed-forward network produces token-dependent residual updates.

To investigate this, we measured the relative L2 norm change of LTM tokens at each timestep in the few-shot regression task. Averaged over three seeds, the highest-update quarter changes nearly twice as much as the lowest-update quarter. We also counted peaks in each token’s update magnitude and found that tokens differ in how often they undergo larger updates. This suggests that substantial memory updates occur at different frequencies across tokens, despite the shared Transformer update schedule. Figures 10 and 11 in Appendix B.4 show the relative L2 changes and peak counts. We have also added RMAAT [3] to the related work section.

As discussed in Section 5, the CMM could be trained with a hyperparameter N that runs the memory Transformer once every N recurrent steps, carrying LTM forward unchanged between updates while short-term computation continues at every step. This would explicitly separate the update frequencies and reduce the cost of Transformer processing. The CMM could also use activation checkpointing, recomputing intermediate activations during backpropagation to reduce training-memory use.

W1. Scope of Performance Claims

The reviewer highlights that the abstract makes a broader performance claim than the results support. We agree that the strongest evidence for the benefit of persistent LTM comes from sort and few-shot regression, where the CMM improves over the ablations, even though their memories cover the full episode. We have narrowed the abstract to reflect these findings and describe generalisation results by task.

W2. Experimental Details

The reviewer raises concerns about missing experimental details. We provide clarification:

Maze Baselines: At the maze batch size of 1,024, GDN and LSAM exceeded the memory available on a single H100 80GB GPU. Matching this batch size through gradient accumulation would have made the 9,000-epoch schedule impractical within our compute budget. RMT training suffered from large gradient spikes across all three seeds despite gradient clipping and hyperparameter tuning, including a tenfold reduction in the learning rate. These runs ended before completing the schedule, so we excluded this baseline. We have clarified these exclusions in the main-text footnote. We believe this is the fairest way to present the results.

ViT: For ImageNet-100, the features come from the penultimate block of the frozen ViT, as stated in Appendix D.6. The reference to the first two blocks in the main text was a mistake. We have corrected it in the revised paper.

We again thank the reviewer for their thoughtful assessment and constructive suggestions. If the reviewer believes we have adequately addressed their concerns, we kindly ask them to consider raising their score.

[1] Darlow, Luke, et al. "Continuous thought machines." Advances in Neural Information Processing Systems 38 (2026): 21548-21594.

[2] Graves, Alex, Greg Wayne, and Ivo Danihelka. "Neural turing machines." arXiv preprint arXiv:1410.5401 (2014).

[3] Mia, Md Zesun Ahmed, Malyaban Bal, and Abhronil Sengupta. "RMAAT: Astrocyte-Inspired Memory Compression and Replay for Efficient Long-Context Transformers." International Conference on Learning Representations (2026).

Thank you for the detailed response and the revision. The new test-time experiments answer the question I cared most about. Zeroing or resetting the LTM breaks copy, sort and few-shot regression and leaves maze accuracy unchanged (Table 4), and the FFN-only control also leaves maze accuracy unchanged (Table 5). This is clear evidence that the trained CMM depends on its LTM on these memory tasks. I also appreciate the narrowed abstract, the parity table, the hyperparameter selection paragraph, the compute-matched runs, and the fixes to the maze footnote and the ViT backbone.

A few things remain, mostly about wording:

  1. Section 4.2 still says the CMM "outperforms all baselines" on the maze, and Section 5 makes a similar claim. Since you now agree the gain does not come from the LTM, and three runs cannot yet separate it from seed variation, I would ask you to describe the maze result more modestly until the new seeds are in. It would also help to report the learned residual gate values of the maze models. The STM attends only to the zero-valued sink and each block starts as the identity map, so it is not clear how much the Transformer changes the STM on this task.

  2. Figs. 10 and 11 show how much LTM tokens change per step and how often they make larger updates, not how long information stays in them. I think it is simpler to say in Section 3 that the STM has forced turnover through its sliding window while the LTM is carried forward without being evicted, instead of "separates timescales".

  3. The compute-matched runs test sort length generalization, while the cost gap in Table 7 is at the 500K Copy-100 setting. I don't expect new runs. A sentence in Section 5 on what the comparison covers, and the widths used for CTM-Ext. and RMT, would be enough.

Reviewer_1TpYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The Continuous Memory Machine (CMM) extends the Continuous Thought Machine (CTM) with a persistent, learned, matrix-valued long-term memory (LTM). At every internal tick, a shared Transformer runs over [sink token; LTM; STM]. The STM is the CTM's sliding window of recent pre-activations, and the sink has its value zeroed. The updated STM feeds the per-neuron models, and the updated LTM is carried forward. The paper reports gains over the CTM and argues that the model learns task-dependent memory strategies. This argument rests on attention shares to the sink, LTM and STM.

Critical strengths
  • Memory designs are well-motivated, even though just limited to CTM.
  • Good evaluation against other methods on multiple tasks that demand good memory designs.
Critical weaknesses
  • Novelty is limited. At this stage, it is well-established that memory has to be expressive enough (e.g., matrix-valued). The novelty therefore resides primarily in a specific architectural combination. Persistent attention-updated memory alongside a fast local processor already appears in RMC, RMT and, closer to this design, Titans and ARMT.
  • Ablations do not establish whether improvements arise from the hypothesised short-term/long-term memory interactions or simply the additional nonlinear computation introduced by the Transformer.
Questions for authors
  • If you zero the LTM, or swap it between episodes, at test time on the trained CMM, which of copy, sort, regression and maze degrade? A collapse on the storage tasks with maze unchanged would turn the "task-dependent strategy" claim into a causal finding, and I would raise my score. A maze drop would mean the sink-attention reading of "no memory use" is misleading.
  • What are the parity results at lengths 50–1000, which Appendix D.4 says were evaluated?
  • What separates the maze seeds that reach 70-90% from those near 2-23%? Does the LTM or the Transformer change the fraction of runs that converge? A positive answer would be a more defensible maze claim (optimization reliability) than the current mean.
  • How does the CMM differ from Titans and ARMT in mechanism, beyond the CTM substrate?
4 comments

We thank the reviewer for their thoughtful review of our paper. We appreciate that the reviewer found the memory design well-motivated and highlighted our extensive evaluation. We address their questions and concerns below.

Q1. Test-Time Long-Term Memory Interventions

The reviewer asks whether disrupting the trained CMM's long-term memory degrades copy, sort and regression while leaving maze performance unchanged. We thank the reviewer for this suggestion as we believe it helps strengthen our claims regarding the flexibility of the CMM’s memory usage. We tested this by zeroing the long-term memory or resetting it to its learned initial state before every memory update on the algorithmic, regression and maze tasks. Both interventions considerably degrade all three memory tasks, while maze accuracy remains unchanged. Results are shown below as changes from the intact model (Mean ± sample standard deviation across three seeds)

TaskMetricZeroed LTM ChangeReinitialized LTM Change
CopyQuery MSE+4.30 ± 0.59+1.25 ± 0.64
SortSequence accuracy (%)-94.7 ± 0.4-94.7 ± 0.4
Regression (K=10)MSE+0.74 ± 0.006+0.73 ± 0.004
Regression (K=20)MSE+0.96 ± 0.006+1.01 ± 0.08
MazeFull-maze accuracy (%)0 ± 00 ± 0

This provides further evidence that the CMM learns task-dependent memory strategies that can make use of the LTM. These results are included in Appendix B.2, Table 4. We hope this addresses the reviewer's question about task-dependent memory use.

Q2. Parity Length Generalisation

The reviewer points out that Appendix D.4 states that parity was evaluated up to length 1,000, but these results were not included. These were meant to be included. We apologise for the omission. We provide these values below and in Appendix D.4, Table 12. Models were trained on lengths 1–40. Values are accuracy (%) and show means ± standard deviations over three seeds.

Model40501002005001000
Transformer57.6 ± 1249.8 ± 3.150.5 ± 1.750.6 ± 4.649.5 ± 1.649 ± 3
RNN100 ± 0100 ± 0100 ± 0100 ± 0100 ± 0100 ± 0
LSTM100 ± 0100 ± 0100 ± 0100 ± 0100 ± 0100 ± 0
FS-LSTM100 ± 0100 ± 0100 ± 0100 ± 0100 ± 0100 ± 0
GDN99.7 ± 0.699 ± 1.891.6 ± 12.266.9 ± 10.648.1 ± 2.949.2 ± 1.6
DNC100 ± 0100 ± 0100 ± 095.2 ± 4.149.7 ± 0.748.6 ± 1.2
RMC100 ± 0100 ± 0100 ± 0100 ± 0100 ± 099.9 ± 0.2
LSAM100 ± 0100 ± 097.9 ± 2.463.5 ± 1551.5 ± 0.248.6 ± 2
NAM-TM100 ± 0100 ± 099.7 ± 0.596.4 ± 5.262.5 ± 10.553.3 ± 5.8
RMT49.5 ± 1.647.5 ± 0.550.4 ± 3.151 ± 1.548.7 ± 2.450.6 ± 1.9
CTM95.5 ± 493.6 ± 5.985.7 ± 12.572.1 ± 24.667.6 ± 28.367.1 ± 28.5
CMM-LSTM100 ± 0100 ± 0100 ± 0100 ± 0100 ± 0100 ± 0
CTM-Ext.97.9 ± 3.696.3 ± 6.492.5 ± 1386.3 ± 23.783 ± 29.483.5 ± 28.5
CMM-STM97.9 ± 195 ± 2.180.9 ± 4.955.9 ± 1.549.4 ± 2.748.2 ± 0.3
CMM99.7 ± 0.699.4 ± 1.198.2 ± 3.295.5 ± 7.891.3 ± 1587.4 ± 21.9

The RNN, LSTM, FS-LSTM and CMM-LSTM solve the task perfectly through length 1,000, and RMC remains close to perfect. GDN, DNC, LSAM and NAM-TM perform well within the training range but degrade towards chance on longer sequences. The Transformer and RMT remain near chance.

The CMM retains 87% accuracy at length 1,000, compared with 48% for CMM-STM. CMM-LSTM, which also retains persistent long-term memory, reaches 100%. The CMM generalises well beyond the training lengths, although there is variation between seeds.

Q3. Maze Seed Variability

The reviewer asks what separates the maze seeds with high and low full-maze accuracy, and whether the LTM or Transformer changes the fraction of runs that converge.

In the maze task, the strongest models fall into two groups. The CTM-based architectures, including CMM, CTM and CMM-STM, perform the best, followed by MANNs such as DNC and NAM-TM. We see this pattern as motivation for combining neuron-level computation with expressive read-write operations, as the CMM does.

All three CMM seeds achieve approximately 92% action accuracy or higher, compared with one of the three CTM seeds. We interpret this as evidence of more reliable action-level learning in these runs. The strong performance of CMM-STM in two seeds also supports the contribution of Transformer processing.

Full-maze accuracy is of course stricter, as a single incorrect action makes the entire solution count as incorrect. On 100-action targets, accuracy over the first 75 actions is 96–99% across the CMM seeds, with much less variation than the CTM. Figure 4 (bottom) similarly shows that the CMM maintains long correct prefixes before performance drops towards the end. The CMM seed with lower full-maze accuracy therefore successfully learns much of the task, but struggles to complete the entire route.

The main difference between the CMM seeds is therefore in completing the final part of the route. All three reach high action accuracy, compared with one CTM seed, suggesting more reliable learning with the CMM.

Q4. Titans and ARMT

The reviewer asks how the CMM differs from Titans and ARMT beyond the CTM substrate. We explain these differences below and have added both models to the related work.

Titans: Titans models long-term memory with a separate neural network. At test time, it stores new information by updating this network’s weights using gradient descent with momentum and adaptive forgetting. In the CMM, long-term memory is a set of tokens updated jointly with the short-term history through a Transformer. The model learns during training how to update this memory, then applies that learned mechanism at test time while keeping its weights fixed.

ARMT: Beyond the CTM substrate, CMM differs from ARMT in how persistent memory is represented and updated. ARMT stores key–value associations in layerwise matrices, using memory tokens to generate gated delta-style updates between input segments. CMM maintains persistent long-term memory tokens and jointly processes them with a rolling short-term history through a Transformer at each recurrent step. In the CMM, the long-term slots can attend to one another and exchange information with the short-term representations.

W1. Novelty

The reviewer notes that matrix-valued memory is already well established and that our novelty lies in the architectural combination. We agree that the CMM builds on the CTM and prior memory-augmented and matrix-valued recurrent models. Our contribution is the combination of short-term neural dynamics (modelled with NLMs) and a persistent long-term store, with a learned mechanism that allows both memories to interact and update their contents. Ablations show this combination is effective for tasks that require memory.

W2. Ablations

The reviewer asks whether the improvements come from STM–LTM interaction or simply the additional Transformer computation. In addition to the new test-time experiment described in Q1 that provides evidence that long-term memory is useful for computation, we describe how the ablations can further clarify the source of the performance gains.

Ablation CTM-Ext. has an extended STM matching the CMM's combined STM and LTM capacity. CMM-STM has the same extended STM, but adds the Transformer. Comparing these two ablations therefore allows us to examine the contribution of the Transformer directly. For example, on sort, adding the Transformer substantially improves CMM-STM over CTM-Ext., but it still does not reach the CMM's performance. This suggests that although Transformer processing over the STM helps, combining it with a distinct persistent LTM provides a further benefit, as also noted by reviewer FuaF.

To further understand the contribution of Transformer processing, we also ran an FFN-only intervention on the trained models, as suggested by reviewer FuaF. We removed memory self-attention and LTM reads and writes while retaining FFN updates to STM, with the model weights fixed. Copy, sort and few-shot regression performance degrades substantially, while both action and full-maze accuracy remain unchanged across all three seeds (Appendix B.3, Table 5). Results are shown below as changes from the intact model (mean ± sample standard deviation across three seeds).

TaskMetricFFN-only Change
CopyQuery MSE+4.27 ± 0.55
SortSequence accuracy (%)-94.7 ± 0.4
Regression (K=10)MSE+0.74 ± 0.01
Regression (K=20)MSE+0.97 ± 0.02
MazeAction accuracy (%)0 ± 0
MazeFull-maze accuracy (%)0 ± 0

This further emphasises the CMM's task-dependent use of memory, since FFN updates alone preserve maze performance but are insufficient for copy, sort and few-shot regression.

We again thank the reviewer for their thoughtful assessment of our work. We believe that we have addressed all of the reviewer's concerns. In particular, we have shown that LTM-zeroing substantially degrades performance on copy, sort and regression while leaving maze performance unchanged. In light of this, we kindly ask the reviewer to consider raising their score.

We thank the reviewer again for their review of our work.

We believe we have addressed the reviewers' concerns. As there are only a few days of discussion left, we kindly ask the reviewer to let us know if there is anything else we can clarify. If the reviewer is satisfied with our comments and updated manuscript we kindly ask the reviewer to consider raising their score.

Reviewer_ph3LOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose the Continuous Memory Machine (CMM), an improved model built on the Continuous Thought Machine (CTM), which distinguishes long-term from short-term memory by separating short-term memory from long-term memory. On this basis, the work introduces a Transformer to handle cross-region interactions between long-term and short-term memory. It validates and compares the effects of different baselines on four typical tasks—copy, recall, sort, and parity—demonstrating its advantages. The paper also shows certain advantages on maze tasks, few-shot regression, and image recognition.

Critical strengths
  1. The paper's experimental design covers multiple aspects, the compared baselines and their corresponding explanations are sufficient, and reproducibility is relatively strong.

  2. Separating long-term and short-term memory helps improve performance, and the performance on different tasks is also relatively good.

Critical weaknesses
  1. The experiments proposed in the paper all focus on small tasks. Although the conclusions show advantages for CMM, it is doubtful whether it can continue to generalize after scaling up. Scaling up also affects the RNN-based recurrent computation part in the paper's basic design, and training parallelism becomes a computational difficulty that directly limits the method's potential for effective scaling. The memory comparisons in Tables 4 and 5 of the supplementary material clearly show disadvantages compared with other methods.

  2. The ImageNet results in the experiments are not sufficiently differentiated, showing no advantage compared with other methods. For work such as GDN, the noticeable variance mainly comes from the seed and learning rate themselves, and the comparison of the actual optimal results of individual models is unclear.

  3. The corresponding maze baseline experiment in the paper is not completed; the supplementary material mentions that this is due to insufficient memory, which should not serve as a corresponding reason.

  4. The experiment mentions that on the maze task almost all attention goes to the sink token, which seems to make it impossible to attribute CMM's improvement over CTM on this task to long-term memory or interaction; the gain comes from the Transformer itself, which conflicts with the memory interaction that the paper mainly intends to express.

  5. The biologically inspired conclusions are insufficient; parts of the current paper are difficult to correspond to the mentioned different frequency scales, consolidation, and robustness to interference, and such explanatory writing should be avoided.

Questions for authors

see weakness

4 comments

We thank the reviewer for their thoughtful review of our paper. We are glad that the reviewer found our work interesting and acknowledged that the Continuous Memory Machine achieves strong performance across a range of tasks. We address each of the concerns below.

W1. Scaling and Computational Cost

The reviewer raises concerns about the scale of our experiments and whether the CMM can scale given its recurrent computation and memory costs. The reviewer is correct that the CMM, like other nonlinear RNNs, cannot be trained in parallel across time. We recognise this architectural tradeoff and do not claim that the CMM should replace parallel models when training throughput is the main objective. We report this limitation in the paper.

However, parallel training is only one consideration when studying recurrent architectures. Our experiments ask how memory should be organised and what capabilities different memory mechanisms provide. For example, while the Transformer has high training throughput, it fundamentally struggles in the parity task. Similarly, the GDN struggles in the sort task. The CMM, on the other hand, excels at both of these.

More broadly, our motivation is to study intelligence and the computational mechanisms that may support it. We take inspiration from biology, where different memory systems retain and process information over different timescales. The CMM translates this idea into an architecture with short-term neural dynamics and persistent long-term memory, then studies what computational behaviour emerges from their interaction. We understand that the lack of time parallelism is a limitation, but believe that understanding these mechanisms has scientific value beyond scalability. The DNC and CTM are examples of non-parallelisable architectures that study memory and reasoning through sequential recurrent computation. The CMM builds on this line of work where sequential dynamics are part of the computational system we aim to study.

W2. ImageNet-100 Results

The reviewer highlights that the ImageNet-100 results do not clearly differentiate the architectures. We agree. This was not the purpose of this experiment. The goal of the ImageNet-100 experiment was to understand if the CMM can maintain the interpretable attention patterns of the CTM, not to build a state of the art image classification model, as mentioned in the paper. As we show, the CMM does maintain the attention patterns of the CTM, suggesting that the change to the model's memory has not removed one of the CTM's interesting emergent capabilities [1].

W3. Missing Maze Baselines

The reviewer raises concerns that some maze baselines were not completed, in particular the GDN, LSAM and RMT. We explain why these are not included below.

We found that performance greatly improves for all models when the batch size is increased, which led to us using a batch size of 1,024. For most models, this could fit comfortably on our NVIDIA H100 80GB GPU. However, for GDN and LSAM, their large matrix-valued recurrent states lead to out-of-memory errors. One option was to use gradient accumulation. However, this would have dramatically increased the time taken to run the 9,000 epochs, making it infeasible. The other option was to run these architectures with a lower batch size, which makes the comparison unfair. We therefore determined the most fair and honest approach was to explicitly state that GDN and LSAM are excluded.

For RMT, training suffered from large gradient spikes across all three seeds. We tried to address this through hyperparameter tuning, including a tenfold reduction in the learning rate, but the runs still ended before completing the schedule despite gradient clipping. We therefore excluded this baseline.

The paper evaluates seven task settings and compares against ten recurrent baselines, a causal Transformer and three architectural ablations. We believe this is an extensive evaluation, and that the absence of three baselines from one task does not invalidate the results across the other tasks.

W4. Sink Attention and the Role of Long-Term Memory

The reviewer questions whether the maze gains can be attributed to LTM, given that STM attention goes to the sink. We agree that the CMM does not rely on LTM in this task. As noted in the paper, our ablations indicate that these gains come from the combination of the Transformer and neuron-level models. We argue that this is a strength of the CMM; it adapts its use of memory to the task, relying on LTM for copy and sort, while attending to the sink and still achieving strong and interpretable performance on mazes.

Following reviewer 1TpY's suggestion, we zeroed the LTM before every memory update at test time to show if the long-term memory is explicitly required. Performance drops sharply on copy, sort and few-shot regression, while maze accuracy remains unchanged. This provides further evidence that the CMM learns task-dependent memory strategies that can make use of the LTM.

TaskMetricZeroed LTM ChangeReinitialized LTM Change
CopyQuery MSE+4.30 ± 0.59+1.25 ± 0.64
SortSequence accuracy (%)-94.7 ± 0.4-94.7 ± 0.4
Regression (K=10)MSE+0.74 ± 0.006+0.73 ± 0.004
Regression (K=20)MSE+0.96 ± 0.006+1.01 ± 0.08
MazeFull-maze accuracy (%)0 ± 00 ± 0

Mean ± sample standard deviation across three seeds. LTM is zeroed or reset to its learned initial state before every memory update. The protocol and results are in Appendix B.2, Table 4.

W5. Biological Inspiration

The reviewer questions how the CMM relates to biological timescales, consolidation and robustness to interference. Our motivation is to take ideas from biological memory and study their computational value. As discussed in the introduction and related work, complementary learning systems motivate separating memory into interacting stores with different roles. The CMM uses this idea to combine short-term neural dynamics with persistent long-term memory.

We do not aim to reproduce the biological process itself. Both stores are processed by the Transformer at every step, but the STM has forced turnover through its sliding window, while the LTM is carried forward without forced eviction. This is the distinction between short- and long-term memory in our architecture, not a claim that the model reproduces hippocampal–neocortical consolidation or biological update frequencies.

We again thank the reviewer for their review of our paper. We believe we have addressed the concerns raised and kindly ask the reviewer to consider raising their score.

[1] Darlow, Luke, et al. "Continuous thought machines." Advances in Neural Information Processing Systems 38 (2026): 21548-21594.

Reviewer_ph3LOpenReview ↗

Reviewer Comments

I thank the authors for their detailed response. The reply has addressed some of my concerns, but several issues remain worth attention.

  1. Scalability as the primary requirement. Scalability is the single most important requirement here. Otherwise, this work is essentially a combination of a Transformer and an SSM-style state method, with updates performed via a nonlinear mechanism, and the novelty of the paper is diminished accordingly.

  2. Training of GDN in the maze setting. I remain confused about the GDN training issue. It may be that I have misunderstood, but if the requirement is to update over the entire sequence, GDN could switch to a parallel computation method to avoid memory problems. That is, since the Transformer can be trained, a GDN with comparable computational cost should likewise be trainable. The current implementation may be serial and should be adjusted

We thank the reviewer for their timely response.

1. Scalability and Scientific Contribution

We agree that scalability is important. We do not agree, however, that it is the single most important requirement when assessing the contribution of a research paper. For example, [1,2,3,4,5] all present interesting non-linear RNNs without focussing on scalability. These recent works have nevertheless been recognized by our community (including conference orals and spotlights at top ML conferences).

We also push back on the description of this work as “essentially a combination of a Transformer and a SSM”, as this overlooks how the CMMs computation and memory are organised and interact. The CMMs memory Transformer processes both recent neural activation histories and persistent long-term memory tokens. The updated short-term memory is then fed as input to the neuron-level models and the updated long-term memory is carried forward.

The novelty of the CMM is therefore this specific combination of neuron-level temporal processing and interacting long- and short-term memory. Ablations show that simply extending the CTMs short-term memory or adding Transformer processing does not recover the CMMs performance. We therefore believe the contribution is both the proposed memory organisation and the experimental evidence showing what this combination adds beyond its individual components.

2. Why GDN cannot be trained in this maze setting

We provide more clarity on the maze task. The maze task is exactly as in the CTM paper [1], which aimed to probe how models reason over time. The task is set up as follows: at each timestep, the model uses its current state to attend to patches in the maze via cross-attention. This means that at each timestep, the model’s input/observation depends on its hidden state at the previous timestep. Therefore, the inputs are not available in advance, and training cannot be parallelised across time. All models are trained with backpropagation through time, including the Transformer, whose attention query from the maze attention comes from the previous token’s state. The memory issues then arise with the GDN due to parameter matching, with the 10M parameter GDN state being too large to be trained with BPTT. We realise this task setup is not made clear enough in the paper and will improve it accordingly.

We thank the reviewer again for their questions and for raising their concerns. If the reviewer believes we have now adequately addressed their main concerns we would kindly ask them to consider raising their score.

[1] Darlow, Luke, et al. "Continuous thought machines." Advances in Neural Information Processing Systems 38 (2026): 21548-21594.

[2] Miyato, Takeru, et al. "Artificial kuramoto oscillatory neurons." International Conference on Learning Representations. Vol. 2025. 2025.

[3] Dai, Jiawen, and Yue Song. “Winfree Oscillatory Neural Network.” NeurIPS (2026). Spotlight.

[4] Liu, Zhuo, and Tao Chen. "Toward practical equilibrium propagation: Brain-inspired recurrent neural network with feedback regulation and residual connections." International conference on learning representations. Vol. 2026. 2026.

[5] Liu, Fenghao, et al. "LIF Recurrent Memory Enables Long-Horizon Spiking Computation." Forty-third International Conference on Machine Learning.

Comments on the paper

1 comment

We thank the reviewers for their consideration of our work. Below we summarize both the strengths and main concerns raised by the reviewers, as well as our responses.

Strengths

Generally, the reviewers found the paper clear and easy to follow. The memory design was described as well motivated, sensible and intuitive. Reviewers praised the evaluation across tasks and baselines, and highlighted the good empirical performance. Reviewers also recognised the ablations as well designed. In particular, they highlighted that adding persistent long-term memory to the CTM’s short-term computation improves performance on sort and few-shot regression.

Weaknesses

  • Reviewers questioned if the short-term/long-term memory interactions actually affected performance. To address this, we added an additional experiment where we zero the long-term memory at test time and see how the performance changes. We find for tasks that require memory, such as sequence copying, sorting, regression etc, that the performance drops significantly. In maze solving, performance remained unchanged, consistent with our attention analysis. Thus, this test-time long-term memory intervention gives further evidence that the CMM is learning task-specific memory interactions.
  • Although all baselines were already parameter matched, reviewers questioned if results would hold at equal compute. In response to this, we ran the sort task with matched compute and found the CMM outperforms baselines.
  • Reviewers were concerned about variance in the maze runs. We clarified that the CMM is the only architecture with consistently high action accuracy across all seeds, and the variance mostly comes from solving the final few steps of the maze. Additional maze seeds are still running, but are unlikely to finish before the end of the discussion period.
  • Some reviewers were concerned about scalability. We acknowledge the computational costs of the CMM (shared by other non-linear RNNs), while emphasising that our goal is to study computational mechanisms inspired by biological memory. Developing useful mechanisms and making them scalable need not happen at the same time, as illustrated by LSTM and later developments such as xLSTM [1]. We believe that judging new architectures primarily by their initial scalability risks overlooking useful directions for future research.

We have added the new experiments and corrections to the paper, with changes in red. We again thank the reviewers for their work.

[1] Beck, Maximilian, et al. "xlstm: Extended long short-term memory." Advances in Neural Information Processing Systems 37 (2024): 107547–107603.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.