- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper compares two models of episodic memory based inference: integrative encoding (operationalized as query blind) and retrieval-based inference (operationalized as query conditioned, +q, where the network processes the query along with the memories). These two approaches are compared in three architecture-matched models using DeepMind's symbolic Alchemy causal worlds, since this allows exact characterization of the posterior distribution (an exact oracle). Even though the models all achieve similar accuracy, there are key differences in their internal representations: the +q models do not generalize well (R < 1) attend preferentially to query-relevant memories (particularly in early layers), and blind models show higher pairwise cosine similarities while +q models keep memories more separate, consistent with the IE and RBI interpetations. +q models also have lower query-level KL divergence and are more parameter efficient and have world posterior that are closer to the oracle as measured by KL divergence, with the advantage shrinking for queries that depend on latent structure. The authors also show that perturbing query-relevant memories by replacing them with their sample mean hurts performance the most and that representational differences in both models do not translate into behavioral differences in the planning, exploration and the full Alchemy task. Finally, the biggest result is posisbly that as query number (N) increases encourages the model to have more IE-like representations which are, in turn, more reusable (R -> 1) and cheaper to compute. This shows that there is a continuum along which IE and RBI are endpoints, with a model’s position on this spectrum determined by how many queries a single representation must support.
Critical strengths
The paper is well written and easy to follow. The setup is logically consistent and clean, and the different architectures/capacities/seeds make the results much more compelling. Using the exact oracle as a comparison point is also a very nice ground truthing which is usually not available in these kinds of papers. The general point, that matched behavior can hide different computations, is interesting and well explored, and the multi-query analysis is an interesting normative explanation for why one would prefer one representation to the other.
Critical weaknesses
-
The relevance of these results to hippocampal computation is very overstated. There are no neural or behavioral predictions, or comparison to data or to previous neuroscience results. The links to the hippocampal literature are by analysis. The results, while interesting, are more descriptive than anything and strongly constrained to ML, so presenting them as a 'computational account' of memory mechanisms is not a fair characterization. Moreover, the +q/-q operationalization doesn't really line up with IE vs. RBI, where the big difference is when integration happens (at retrieval, or during learning when there is overlap with previously stored memories). It's not clear if anything is ever stored about the current world, because all memories are supplied as input and so there are no capacity limits or interference as you would expect in a biological system. Both models are processing the full evidence and answering the query in one pass, so the real question is about whether processing the evidence is or is not guided by the query, which would be a different question from actual hippocampal memory integration. I would suggest either reframing these claims or discussing them.
-
Some of the central results arise from construction without any discussion. For example, a query-blind model is approximated by a model where N -> 48 (any query is possible), whereas a query-specific model is one where N = 1. This kind of forces the continuum result to arise, so an interpolation of R and KL divergence is expected. It would be good to discuss this in the text. Moreover, the tasks in 4.5 seem to be insensitive to uncertainty information. The argmax transition is used in the planning and Alchemy tasks which doesn't explicitly use uncertainty. Here the VMPO comparison also seems a bit apples to oranges since the models compared are supervised vs. unsupervised. Also, pattern completion tasks (where the answer is certain so oracle posterior has no uncertainty) make up most of the test set, and there the +q advantage is the largest. It's unclear why the advantage would come from representations of uncertainty themselves. It would be good to probe internal representations of uncertainty, and not just output performance.
I have to also add that the conclusion is strongly lacking. In general results are presented descriptively, and the framing does not fit the evidence, so the biological analogies should either be scaled back or substantiated. This would need to be addressed before I can recommend acceptance.
Questions for authors
1.Can uncertainty be decoded from internal states? For example decoding the oracle posterior from intermediate layers? If so does decodability differ between-q/+q models? 2. Is the pairwise similarity in IE models the same across all memory pairs, or specific to memories with overlap (e.g.with the same potion)? The IE would predict that memories with overlap have higher similarity 3. Does the composition of query types in the multi-query model affect results? Is training done on queries of all one type, or mixed? I couldnt find the training details for 4.6
3 comments
Response to reviewer 7zrN (2/3)
4. “the tasks in 4.5 seem to be insensitive to uncertainty information."
We agree that the planning analysis tests whether a model places most of its confidence on the correct transition. Meanwhile, the exploration analysis tests whether a model places most of its uncertainty on the most uncertain transition. Together, planning and exploration allows the model to navigate through uncertain causal structures and to perform the Alchemy task.
Section 4.5 has 2 aims: to validate that the learned representations support downstream behavior, and to test whether the matched accuracy (Table 1) extends to behavior. Despite different computations and uncertainty representations, the two model types are behaviorally indistinguishable. The more accurate uncertainty representation does not yield better downstream behavior because only the most certain and uncertain transitions are necessary for the behavior. The difference between both model types resides across the uncertainty distribution, beyond those two most extreme points. Previous works were comparing RBI and IE through inference accuracy and behavior (Zeithamova & Preston, 2010; Zeithamova et al., 2012a; Banino et al., 2016; Koster et al., 2018). Our results suggest that it would be difficult to measure their representational difference this way in a setting with causal uncertainty. We revised Section A.9 to make this more clear.
5. "Here the VMPO comparison also seems a bit apples to oranges"
We agree, and we do not intend it as a matched comparison. Like the oracle, it serves only as a reference point. It shows that the learned representations are useful and gives a sense of the task’s difficulty. We added a brief mention that it is a RL-trained gated transformer-XL to prevent confusion.
6. Pattern completion and uncertainty
At sufficient capacity, there is no advantage in task performance between query-conditioned and query-blind models for any query types, including PC (see Fig.A11). The advantage is specifically in representation of uncertainty, and it is true across all query types (Figure 4a). The advantage in uncertainty representation during PC queries is explained by query-conditioned models having access to the exact memory that answers the query. By contrast, query-blind models must compress all evidence into a fixed-size representation before knowing the query. Thus, the technically certain answer is no longer directly retrievable, resulting in some uncertainty in query-blind’s representation. This advantage shrinks as the inference depends more on reconstructing the latent structure from many memories, instead of retrieving one or few specific memories. We have revised Section A.5 to clarify this.
7. "the conclusion is strongly lacking."
We agree. We replaced it with a more comprehensive discussion section.
Response to reviewer 7zrN (3/3)
8. Decoding uncertainty from model intermediate representation
We thank the reviewer for the suggestion and agree that this is an interesting addition. We carried out this experiment in 20x models and added it to the revised manuscript (Sec 4.2, Fig.A8). In query-conditioned models, top-1 accuracy approaches the final accuracy by layer 2. Yet, its uncertainty representation continues to improve. This both matches the query retrieval peak at layer 2 (Figure 2b) and echoes that similar performance doesn’t imply similar uncertainty representation. In comparison, query-blind’s accuracy gradually increases and only matches the query-conditioned at the final layer.
9. Cosine similarity between memory pairs with different levels of similarity
In IE paradigms, related memories are integrated together into a cohesive representation. In Alchemy, every memory constrains the same causal structure and thus, even memories that do not directly overlap are informative about each other and contribute to the integrated representation. We conducted the Figure 2c analysis with memory pairs separated by initial similarity. We find that similar memories stay similar, and initially dissimilar memories gradually become more similar, as memories get integrated into a cohesive representation of the world. Regardless of the initial similarity, they reach comparable similarity at the final layer. This result suggests that the result in Figure 2c is not driven by already similar memory pairs. We added the results to the revised manuscript (Sec. 4.1, Fig.A9).
10. Composition of query types in the multi-query model
The training of multi-query models is exactly the same as the single-query model training, except that the loss averages over N queries per sample. Training is always done on all query types, as the training samples are generated with the original reinforcement learning framework, where the actions are performed by a random policy instead of a learned one. The different types of inference naturally occur in the Alchemy task, and we only check post hoc how they affect uncertainty representation. We revised Section A.10 to clarify that. We added a per query type decomposition for the multi-query models, and we find the same overall trend across query types (Figure A.19).
We thank the reviewer for their feedback. Their comments on the framing and the conclusion led to the most substantial changes in this revision. The paper now presents the work as a computational comparison, and a new Discussion sets out its predictions and limitations. We hope the revised manuscript addresses their concerns, and we would be grateful if they would consider revising their score. We are happy to discuss any remaining questions.
Response to reviewer 7zrN (1/3)
We thank the reviewer for the engaged review. We are glad that the reviewer found the paper “well written and easy to follow”, the setup “logically consistent and clean”, and the general point “interesting and well explored”.
1. "Relevance of these results to hippocampal computation"
We do not intend to claim the models as models of hippocampus. The hippocampal literature motivates the question, but our aim is to better understand the computational implications of RBI and IE computations in a causal setting. Our claims are at the computational level, not at the biological level. To our knowledge, previous studies have not directly investigated how RBI / IE affects uncertainty representations. Our computational comparison reveals a trade-off between uncertainty representation and representational reuse, and provides several model-derived findings opening up future experiments. We have revised the introduction and replaced the conclusion with a more comprehensive discussion to better frame the scope of this work.
2. Mapping +q/-q operationalization with IE vs. RBI
We have revised section 3.2 to better clarify the operationalization. The main property of IE is that memories are integrated together into a general-purpose representation, before receiving a query. In query-blind models, the core model likewise first compresses all memories into a limited-capacity representation (Fig 1), denoted as context vector c in the text (for exact size see “Hidden dim. H” in Table 2 in A.1). This general purpose representation c is computed once per causal world. The integrated representation can then be recalled later to answer as many queries as desired, through the query head. Moreover, when the core model is GRU or Causal transformer, memories are processed sequentially in a temporal order, such that each memory can only be processed in relation to previously acquired memories (Figure 1, right panel). Interestingly, the findings tend to be consistent between the Causal transformer and ESWM (which integrates memories non-sequentially).
In comparison, the main property of RBI is that, when the query is received, each memory remains individually accessible, as separate tokens in our query-conditioned models. Similar to IE’s integrated representation, our implementation assumes that each memory can be perfectly recalled. The GRU results provide a direct example where GRU+q receives the query, but processes it only after all memories are integrated into the fixed-size hidden state. It cannot revisit individual memories and thus behaves IE-like (Fig A.17). In contrast, ESWM+q and Transformer+q, which can attend to individual memories, behave RBI-like. Thus, RBI and IE-like computations are not merely whether the query is part of the input, but whether memories remain individually accessible or are integrated together before the query arrives.
We also added a discussion about the limitations in the new discussion section, which notes that this work assumed perfect storage and recall of both the integrated representation (IE) and the individual memories (RBI). We probed at the latter with perturbations during inference (Fig 4b), and now, as a response to reviewer vXBH, we also probed at memory perturbation during training (see response “3. Memory perturbation during training”). Overall, this perfect recall assumption entails that there is no unrelated memory from other causal worlds recalled, and thus dynamics such as interference across worlds is not modeled.
3. "This kind of forces the continuum result to arise, so an interpolation of R and KL divergence is expected."
We agree that the endpoints are expected due to design. N=1 is the query-conditioned model and a N=48 would cover all possible queries, making it akin to the more general query-blind model. The interpolation between the endpoints, however, is not obvious. One reasonable expectation is that because N-query models are query-conditioned, they would always have an advantage in representing query-specific uncertainty over a query-blind model. Another reasonable expectation would be that increasing pressure toward a more general representation would cause an abrupt switch between the two computation modes at some threshold. Instead of a single query-conditioned mode or two distinct modes matching the RBI vs IE dichotomy suggested from literature, we find a gradual spectrum in uncertainty and transferability. We have revised the result section 4.6 to clarify.