- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes a computational framework for assessing whether memory-related cognitive phenomena are model-, task-, or data-dependent. The authors trained six variants of gated RNNs extended with a stack memory (called "episodic memory") on a multi-objective, synthetic memorization task and evaluated whether they exhibit patterns that parallel human performance. The authors classified documented cognitive phenomena based on whether they are replicated from specific model configurations or data distributions.
Critical strengths
The high-level concept of identifying the roots of cognitive phenomena via model/task comparison is appealing.
Critical weaknesses
Limited Representativeness of the Explored Models
The biggest concern lies in the representativeness of the experimented models. While the authors emphasize that their architecture generalizes major models including Transformer, LSTM, and GRU, the actual experimental results are based on just five specific configurations. Accordingly, there are architectural variations left untested such activation functions and the number of heads. Although it is impossible to exhaust all combinations, it is misleading to draw general conclusions like "this cognitive phenomenon roots in data statistics, not in model architectures" from the narrow exploration.
The authors should at least discuss several neural network models that claimed to have replicated biological memory effects (e.g., Gosmann & Eliasmith, 2021, Psychol. Rev.). These biologically oriented models are necessarily formalized as a gated RNN.
The Evaluation Metric is Not Justified Enough
This paper evaluated their models based on Cohen's , which is derived from two statistics . Table S3 explains how and were computed.
First of all, I do not think the main text is comprehensive enough to understand the analysis. It just says "Effect sizes were measured using Cohen’s " in the caption of Figure 2, without explaining what "effect size" means or how/why is computed.
Moreover, the specific choice of and is not justified in the paper. For example, the set size effect was evaluated between the recall probability conditioned on 4/5/6 vs. 10/11/12 study items. What is the philosophy behind these specific thresholds?
Similarly, the referenced human baselines (Table S4 and S5) appear arbitrary. The authors should justify why these specific studies are selected from the long history of psychological research, explaining in what ways they align with the models and/or tasks explored here.
Overall, I do not think the diverse phenomena discussed in this study can be uniformly evaluated by Cohen's . Instead, the standard evaluation/visualization specialized for each phonomenon is much friendlier to readers. For example, why don't you simply plot the accuracy or recall probability against presentation time of study items? Such visualization must be easier than force-unified evaluation of multiple phenomena.
Task Design is Not Justified Enough
The task adopted in this paper combines four subtasks. I do not understand what motivated the authors to choose such a complex design rather than simpler, individual experiments aligning with psychological experiments. (In Figure S2, results from "single-task training" are reported. However, they do not report which single task is reported there.) While I agree that "the brain was not shaped to solve any one task in isolation", the combination of four tasks is far apart from a real-world learning. If the authors are truly interested in naturalistic settings, they should rather test few-shot adaptation of vision–language–action models for example. I also think a single-task setting is even more natural for modeling animal experiments.
Additionally, the design of the "semantic structure" is misleading. While the semantic information represented in word embeddings originates in co-occurrence statistics of words, this "co-occurrence" is based on the neighborhood within serialized word sequences, not the simultaneous presentation of multiple items adopted in this study.
Citation
The authors report that they "used generative AI tools [...] to help find related works". Consequently, some of the cited studies have incorrect or non-standard publication years. For example, the Vaswani et al.'s "Attention is All You Need" is usually considered to have been published in 2017, not 2023. I also think the NeurIPS version is more appropriate than the preprint.
Questions for authors
My questions are embedded in the Critical Weaknesses section.
6 comments
Response to reviewer LF2M (weaknesses 3 & 4)
Weakness 3: task design is not justified enough
The reviewer asks two questions: why train the models on multiple tasks, and why choose these tasks? We answer both in a new paragraph in the beginning of the "training and testing" section:
Models are given sequences of study observations (Section 2.3), followed by recall cues which determine the tasks that should be performed. The training tasks define the loss against which the model parameters are optimized, and therefore shape the behavior of the model. Ideally, they should reflect the pressures under which biological memory systems were shaped. We considered four candidate tasks, each corresponding to a basic and plausible memory function:
- Recognition: deciding, given a (possibly partial) cue, whether a matching observation has been seen before. This is useful, for instance, to recognize a familiar place, a known individual or a threat met in the past.
- Cued recall: recovering the missing part of a partial cue. For example, coming back to a place where danger was encountered should bring back the full memory of that encounter.
- Backward and forward recall: retrieving the observation that preceded or followed a given cue. Knowing what came before or after an event can support causal learning, prediction and planning.
Implementation details for each task are given in Section A.3. Each task can be switched on or off independently, so that models can be trained on any single task or on any combination of them. Being able to combine tasks matters for two main reasons. First, memory in the brain is unlikely to have been shaped by a single objective, and previous work has shown that training a network on several tasks changes the representations and behavior it develops (Li et al., 2026; Yang et al., 2019). Second, the phenomena we study are measured with different tasks in the human literature (recognition for some, cued recall for others; see Table S5 and Table S6), so a single model can only show all of them if it can solve several tasks.
Another reason why we included forward and backward tasks is their role in the emergence of the contiguity effect, as shown in figure 4(a).
In the following, we answer to other minor comments from the reviewer:
- We agree that the single-task plots do not show clearly which task the model was trained on. Thanks for pointing this out, we now explain it in the caption and we will highlight on it on the figure.
- It is true that an animal in the lab is typically trained on one task, but it starts with a memory system already shaped by many other processes.
- Regarding the semantic structure, we chose that name because this structure is meant to create a form of semantic memory. However, it is true that "semantic" typically applies to words and language, while our notion aims to be more general. We could switch it to "schematic structure".
Weakness 4: Citation
The reviewer noticed a mistake in the date of a reference. This mistake happened because the date of the latest arXiv version of the paper was saved to Zotero. We thank the reviewer for spotting this, we now reference the Neurips version. We want to clarify that AI tools were only used in the process of finding papers, but no AI tool was used to generate in-text citations or reference entries. All reference entries come from a .bib file generated from Zotero.
Conclusion
We thank the reviewer again for their insightful feedback. We believe the modifications have improved the clarity our manuscript without changing the conclusions, and we hope to have answered the reviewer's comments. We would be grateful if the reviewer would re-evaluate their ratings in light of these improvements.
I appreciate the authors' effort on the revisions. I have raised my score to 2. I cannot give them a better score mainly because of Weakness 3; it is true that "memory in the brain is unlikely to have been shaped by a single objective", but I think the ultimate scientific goal is to find the necessary and sufficient components. I don't think starting with a handful of mixed learning objectives helps us step toward it in comparison to atomic analyses based on single objectives.
Response to reviewer LF2M (weaknesses 1 & 2)
We thank the reviewer for their detailed and useful comments. We ran additional simulations, and updated the manuscript accordingly.
Weakness 1: limited representativeness of the explored models
The specific models we used were chosen as representatives of the main families of configurations that can be expressed in the unified model space (working-memory-only, episodic-memory-only, both components with built-in vs learned mechanism). Standard models each belong to one of these families, and in our experiments we found that they behave similarly to their respective representatives. To make this clearer, we have extended our analysis to include additional models (LSTM, GRU, Transformer, and TCM/CMR variants):
In all these new analyses, we compare models across different hidden dimensions and show that the results remain qualitatively similar within each group. As suggested, we also conducted additional analyses on hyperparameters:
Overall, results remain stable across these hyperparameter changes. We will refer to these analyses in the main text, and we will explain intuitively the difference between our Constrained model and TCM/CMR (which can already be assessed from entries in Table 1 and Table S7). We will also discuss the CUE model by Gossman & Eliasmith, 2021. As it is heavily inspired by the TCM model we expect it to behave similarly to the TCM variants.
Weakness 2: the evaluation metric is not justified enough
The reviewer questions the choice of Cohen's h as our primary metric for assessing the presence of phenomena. We agree that this choice was not well explained in the main text, and that detailed plots would be a good addition. We modified the manuscript: