- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This work straightforwardly extends the prior work on Continuous Thought Machine to include a "long-term memory" bank, similar to prior work on memory-augmented neural networks, that is indexed by a Transformer and used to update the "short-term memory" that is used as the input to the "neuron-level modules" function which produces the model's next state. The authors evaluate this architecture on a wide range of tasks including algorithmic sequence memory tasks (copy, recall, sort, parity), maze solving, few-shot learning, and classification, demonstrating strong performance across the evaluated tasks.
Critical strengths
Taking the prior CTM work as given, this paper represents a simple and performant extension to enhance its long-term memory capacity. The mechanism is intuitive and easy to understand, and has good empirical performance across a diverse set of tasks. The ablations are meaningful and thoughtfully designed, and the limitations are considered.
Critical weaknesses
Although it has good performance overall, CMM is not the absolute winner on many of the individual tasks, and there are some unclear results about the necessity/value of the LTM. For instance, CMM performs worse than its ablations on Copy, indicating that LTM is potentially unhelpful even for explicit memory manipulation tasks. Similarly, the Maze performance is not entirely convincing due to the extremely high variability and small number of seeds -- indeed, CMM-STM seems to outperform CMM on 2 out of 3 seeds (Table 9). Additionally, the memory overhead is much larger than competing methods, while the performance gains are only modest for some tasks, putting into question whether the overhead is worth it, and whether it's scalable.
A minor suggestion for the authors that would slightly improve readability is to group the algorithmic tasks bar graph in Fig 2 by task rather than by model -- this makes comparing models easier.
Questions for authors
Could the authors provide additional evidence, e.g. additional seeds, for the robustness of the Maze results, particularly the improvement of CMM over CMM-STM?
3 comments
Thank you for the detailed response and additional experiments, I appreciate the authors’ efforts to address my concerns. The additional results somewhat strengthen the paper, but my overall assessment remains, I will maintain my score.
As we wrap up the author-reviewer discussion period we would like to thank reviewer mRBK again for their helpful review of our work. It's much appreciated.
We thank the reviewer for their assessment of our work. We are glad that the reviewer found the Continuous Memory Machine architecture easy to understand and recognised its good empirical performance across a broad range of tasks. We address their questions and concerns below.
W1. Relative Performance vs Baselines
The reviewer highlights that the CMM is not the absolute winner on every task, pointing to copy as an example. The ablations CTM-Ext and CMM-STM have extended short-term memories long enough to hold the entire input sequence. This makes sequential recall easier, since the entire input history remains available, much as it does for the causal Transformer. The CMM, with its much shorter STM, must instead learn to store and recall the sequence through its LTM.
The extended STM ablations therefore have an inductive bias well suited to this task. The CMM can also learn to copy accurately, with one seed achieving a loss close to CMM-STM. Extending STM also has a cost, since the number of parameters in the neuron-level models grows with the memory length at a fixed model width. The sort and parity results show that extending STM alone does not match the CMM when the task requires more than sequential recall.
W2. Variability in Maze Solving
The reviewer highlights the variance in maze performance and that CMM-STM outperforms the CMM in two of three seeds. We agree that full-maze accuracy varies considerably. This is a strict metric as a single incorrect action makes an otherwise correct solution count as incorrect.
All three CMM seeds achieve over 90% action accuracy. Figure 4 (bottom) shows that the CMM maintains long correct prefixes, with errors becoming more common near the end. On 100-action targets, accuracy over the first 75 actions is 96–99% across seeds. The CTM, on the other hand, shows much more variation over these earlier actions. This helps explain why full-maze scores vary so much even when most actions are correct. Additional maze seeds are still running, but are unlikely to finish before the end of the discussion period.
W3. Scalability
The reviewer raises concerns about memory overhead and scalability. We acknowledge the increased computational and memory costs as a limitation. As discussed in future work, one way to reduce this cost is to run the memory Transformer less frequently.
Although our experiments are already parameter matched, we also ran a sort comparison with matched training-time budgets, as suggested by reviewer FuaF. We trained the CMM and wider CTM-Ext and RMT models for 30 minutes per seed under a common memory limit. Models were trained on lengths 4–16 and evaluated on the same examples.
Results below show bit accuracy (%) as mean ± standard deviation across three seeds.
The CMM achieves the highest mean accuracy at every unseen length, showing that its generalisation advantage on sort remains under these resource constraints. These results have been added to the paper (Appendix C.1, Table 8).
Regarding scalability more broadly, our motivation is to study how memory should be organised and what capabilities different memory mechanisms provide. We believe there is a distinction between developing useful computational building blocks and making them efficient at scale. For example, xLSTM revisits LSTM with architectural changes that improve performance and scalability [1]. If scalability is the main criterion from the outset, we risk overlooking ideas that could support future improvements.
We have also regrouped Figure 2 by task to make the model comparisons easier to read. We thank the reviewer for this suggestion.
We thank the reviewer again for their careful consideration of our work. If the reviewer believes we have addressed their critical concerns, we kindly ask them to consider raising their score.
[1] Beck, Maximilian, et al. "xlstm: Extended long short-term memory." Advances in Neural Information Processing Systems 37 (2024): 107547–107603.