acceptodds
Under review as a conference paper at ICLR 2027

Timing of memory integration shapes representations of causal uncertainty

Abstract

Two questions have largely been studied separately in hippocampal research: how uncertainty over hidden structure is represented and when episodic memories are integrated relative to a query. We bring them together by extending the Episodic Spatial World Model (ESWM) framework from spatial mapping to inference over latent causal structure. We compare matched architectural instantiations of integrative encoding, which compresses ambiguous evidence before a query is known, and retrieval-based inference, which combines stored episodes after the query identifies the relevant causal relation. DeepMind's Symbolic Alchemy provides an enumerable space of causal worlds, allowing us to compute the exact posterior over structures and the posterior predictive for every query. This provides a normative reference for comparing the beliefs expressed by models with nearly indistinguishable behavioral accuracy. At sufficient capacity, the two modes achieve nearly indistinguishable behavioral performance while relying on different internal computations and uncertainty representations: retrieval-based-inference models preserve more separable memories, preferentially retrieve query-relevant evidence, and represent uncertainty more faithfully, whereas integrative-encoding models form representations that generalize better across queries. As a single representation is required to support more queries, models smoothly shift from query-specific toward reusable, integrative-encoding-like representations. These results characterize integrative encoding and retrieval-based inference as endpoints of a trade-off between query-specific inference and representational reuse.

Primary area:
applications to neuroscience & cognitive science
Keywords:
hippocampus, uncertainty, integrative encoding, retrieval-based inference, cognitive map, neuroscience

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7zrNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper compares two models of episodic memory based inference: integrative encoding (operationalized as query blind) and retrieval-based inference (operationalized as query conditioned, +q, where the network processes the query along with the memories). These two approaches are compared in three architecture-matched models using DeepMind's symbolic Alchemy causal worlds, since this allows exact characterization of the posterior distribution (an exact oracle). Even though the models all achieve similar accuracy, there are key differences in their internal representations: the +q models do not generalize well (R < 1) attend preferentially to query-relevant memories (particularly in early layers), and blind models show higher pairwise cosine similarities while +q models keep memories more separate, consistent with the IE and RBI interpetations. +q models also have lower query-level KL divergence and are more parameter efficient and have world posterior that are closer to the oracle as measured by KL divergence, with the advantage shrinking for queries that depend on latent structure. The authors also show that perturbing query-relevant memories by replacing them with their sample mean hurts performance the most and that representational differences in both models do not translate into behavioral differences in the planning, exploration and the full Alchemy task. Finally, the biggest result is posisbly that as query number (N) increases encourages the model to have more IE-like representations which are, in turn, more reusable (R -> 1) and cheaper to compute. This shows that there is a continuum along which IE and RBI are endpoints, with a model’s position on this spectrum determined by how many queries a single representation must support.

Critical strengths

The paper is well written and easy to follow. The setup is logically consistent and clean, and the different architectures/capacities/seeds make the results much more compelling. Using the exact oracle as a comparison point is also a very nice ground truthing which is usually not available in these kinds of papers. The general point, that matched behavior can hide different computations, is interesting and well explored, and the multi-query analysis is an interesting normative explanation for why one would prefer one representation to the other.

Critical weaknesses
  1. The relevance of these results to hippocampal computation is very overstated. There are no neural or behavioral predictions, or comparison to data or to previous neuroscience results. The links to the hippocampal literature are by analysis. The results, while interesting, are more descriptive than anything and strongly constrained to ML, so presenting them as a 'computational account' of memory mechanisms is not a fair characterization. Moreover, the +q/-q operationalization doesn't really line up with IE vs. RBI, where the big difference is when integration happens (at retrieval, or during learning when there is overlap with previously stored memories). It's not clear if anything is ever stored about the current world, because all memories are supplied as input and so there are no capacity limits or interference as you would expect in a biological system. Both models are processing the full evidence and answering the query in one pass, so the real question is about whether processing the evidence is or is not guided by the query, which would be a different question from actual hippocampal memory integration. I would suggest either reframing these claims or discussing them.

  2. Some of the central results arise from construction without any discussion. For example, a query-blind model is approximated by a model where N -> 48 (any query is possible), whereas a query-specific model is one where N = 1. This kind of forces the continuum result to arise, so an interpolation of R and KL divergence is expected. It would be good to discuss this in the text. Moreover, the tasks in 4.5 seem to be insensitive to uncertainty information. The argmax transition is used in the planning and Alchemy tasks which doesn't explicitly use uncertainty. Here the VMPO comparison also seems a bit apples to oranges since the models compared are supervised vs. unsupervised. Also, pattern completion tasks (where the answer is certain so oracle posterior has no uncertainty) make up most of the test set, and there the +q advantage is the largest. It's unclear why the advantage would come from representations of uncertainty themselves. It would be good to probe internal representations of uncertainty, and not just output performance.

I have to also add that the conclusion is strongly lacking. In general results are presented descriptively, and the framing does not fit the evidence, so the biological analogies should either be scaled back or substantiated. This would need to be addressed before I can recommend acceptance.

Questions for authors

1.Can uncertainty be decoded from internal states? For example decoding the oracle posterior from intermediate layers? If so does decodability differ between-q/+q models? 2. Is the pairwise similarity in IE models the same across all memory pairs, or specific to memories with overlap (e.g.with the same potion)? The IE would predict that memories with overlap have higher similarity 3. Does the composition of query types in the multi-query model affect results? Is training done on queries of all one type, or mixed? I couldnt find the training details for 4.6

3 comments
AuthorsOpenReview ↗

Response to reviewer 7zrN (1/3)

We thank the reviewer for the engaged review. We are glad that the reviewer found the paper “well written and easy to follow”, the setup “logically consistent and clean”, and the general point “interesting and well explored”.

1. "Relevance of these results to hippocampal computation"

We do not intend to claim the models as models of hippocampus. The hippocampal literature motivates the question, but our aim is to better understand the computational implications of RBI and IE computations in a causal setting. Our claims are at the computational level, not at the biological level. To our knowledge, previous studies have not directly investigated how RBI / IE affects uncertainty representations. Our computational comparison reveals a trade-off between uncertainty representation and representational reuse, and provides several model-derived findings opening up future experiments. We have revised the introduction and replaced the conclusion with a more comprehensive discussion to better frame the scope of this work.

2. Mapping +q/-q operationalization with IE vs. RBI

We have revised section 3.2 to better clarify the operationalization. The main property of IE is that memories are integrated together into a general-purpose representation, before receiving a query. In query-blind models, the core model likewise first compresses all memories into a limited-capacity representation (Fig 1), denoted as context vector c in the text (for exact size see “Hidden dim. H” in Table 2 in A.1). This general purpose representation c is computed once per causal world. The integrated representation can then be recalled later to answer as many queries as desired, through the query head. Moreover, when the core model is GRU or Causal transformer, memories are processed sequentially in a temporal order, such that each memory can only be processed in relation to previously acquired memories (Figure 1, right panel). Interestingly, the findings tend to be consistent between the Causal transformer and ESWM (which integrates memories non-sequentially).

In comparison, the main property of RBI is that, when the query is received, each memory remains individually accessible, as separate tokens in our query-conditioned models. Similar to IE’s integrated representation, our implementation assumes that each memory can be perfectly recalled. The GRU results provide a direct example where GRU+q receives the query, but processes it only after all memories are integrated into the fixed-size hidden state. It cannot revisit individual memories and thus behaves IE-like (​​Fig A.17). In contrast, ESWM+q and Transformer+q, which can attend to individual memories, behave RBI-like. Thus, RBI and IE-like computations are not merely whether the query is part of the input, but whether memories remain individually accessible or are integrated together before the query arrives.

We also added a discussion about the limitations in the new discussion section, which notes that this work assumed perfect storage and recall of both the integrated representation (IE) and the individual memories (RBI). We probed at the latter with perturbations during inference (Fig 4b), and now, as a response to reviewer vXBH, we also probed at memory perturbation during training (see response “3. Memory perturbation during training”). Overall, this perfect recall assumption entails that there is no unrelated memory from other causal worlds recalled, and thus dynamics such as interference across worlds is not modeled.

3. "This kind of forces the continuum result to arise, so an interpolation of R and KL divergence is expected."

We agree that the endpoints are expected due to design. N=1 is the query-conditioned model and a N=48 would cover all possible queries, making it akin to the more general query-blind model. The interpolation between the endpoints, however, is not obvious. One reasonable expectation is that because N-query models are query-conditioned, they would always have an advantage in representing query-specific uncertainty over a query-blind model. Another reasonable expectation would be that increasing pressure toward a more general representation would cause an abrupt switch between the two computation modes at some threshold. Instead of a single query-conditioned mode or two distinct modes matching the RBI vs IE dichotomy suggested from literature, we find a gradual spectrum in uncertainty and transferability. We have revised the result section 4.6 to clarify.

AuthorsOpenReview ↗

Response to reviewer 7zrN (2/3)

4. “the tasks in 4.5 seem to be insensitive to uncertainty information."

We agree that the planning analysis tests whether a model places most of its confidence on the correct transition. Meanwhile, the exploration analysis tests whether a model places most of its uncertainty on the most uncertain transition. Together, planning and exploration allows the model to navigate through uncertain causal structures and to perform the Alchemy task.
Section 4.5 has 2 aims: to validate that the learned representations support downstream behavior, and to test whether the matched accuracy (Table 1) extends to behavior. Despite different computations and uncertainty representations, the two model types are behaviorally indistinguishable. The more accurate uncertainty representation does not yield better downstream behavior because only the most certain and uncertain transitions are necessary for the behavior. The difference between both model types resides across the uncertainty distribution, beyond those two most extreme points. Previous works were comparing RBI and IE through inference accuracy and behavior (Zeithamova & Preston, 2010; Zeithamova et al., 2012a; Banino et al., 2016; Koster et al., 2018). Our results suggest that it would be difficult to measure their representational difference this way in a setting with causal uncertainty. We revised Section A.9 to make this more clear.

5. "Here the VMPO comparison also seems a bit apples to oranges"

We agree, and we do not intend it as a matched comparison. Like the oracle, it serves only as a reference point. It shows that the learned representations are useful and gives a sense of the task’s difficulty. We added a brief mention that it is a RL-trained gated transformer-XL to prevent confusion.

6. Pattern completion and uncertainty

At sufficient capacity, there is no advantage in task performance between query-conditioned and query-blind models for any query types, including PC (see Fig.A11). The advantage is specifically in representation of uncertainty, and it is true across all query types (Figure 4a). The advantage in uncertainty representation during PC queries is explained by query-conditioned models having access to the exact memory that answers the query. By contrast, query-blind models must compress all evidence into a fixed-size representation before knowing the query. Thus, the technically certain answer is no longer directly retrievable, resulting in some uncertainty in query-blind’s representation. This advantage shrinks as the inference depends more on reconstructing the latent structure from many memories, instead of retrieving one or few specific memories. We have revised Section A.5 to clarify this.

7. "the conclusion is strongly lacking."

We agree. We replaced it with a more comprehensive discussion section.

AuthorsOpenReview ↗

Response to reviewer 7zrN (3/3)

8. Decoding uncertainty from model intermediate representation

We thank the reviewer for the suggestion and agree that this is an interesting addition. We carried out this experiment in 20x models and added it to the revised manuscript (Sec 4.2, Fig.A8). In query-conditioned models, top-1 accuracy approaches the final accuracy by layer 2. Yet, its uncertainty representation continues to improve. This both matches the query retrieval peak at layer 2 (Figure 2b) and echoes that similar performance doesn’t imply similar uncertainty representation. In comparison, query-blind’s accuracy gradually increases and only matches the query-conditioned at the final layer.

9. Cosine similarity between memory pairs with different levels of similarity

In IE paradigms, related memories are integrated together into a cohesive representation. In Alchemy, every memory constrains the same causal structure and thus, even memories that do not directly overlap are informative about each other and contribute to the integrated representation. We conducted the Figure 2c analysis with memory pairs separated by initial similarity. We find that similar memories stay similar, and initially dissimilar memories gradually become more similar, as memories get integrated into a cohesive representation of the world. Regardless of the initial similarity, they reach comparable similarity at the final layer. This result suggests that the result in Figure 2c is not driven by already similar memory pairs. We added the results to the revised manuscript (Sec. 4.1, Fig.A9).

10. Composition of query types in the multi-query model

The training of multi-query models is exactly the same as the single-query model training, except that the loss averages over N queries per sample. Training is always done on all query types, as the training samples are generated with the original reinforcement learning framework, where the actions are performed by a random policy instead of a learned one. The different types of inference naturally occur in the Alchemy task, and we only check post hoc how they affect uncertainty representation. We revised Section A.10 to clarify that. We added a per query type decomposition for the multi-query models, and we find the same overall trend across query types (Figure A.19).

We thank the reviewer for their feedback. Their comments on the framing and the conclusion led to the most substantial changes in this revision. The paper now presents the work as a computational comparison, and a new Discussion sets out its predictions and limitations. We hope the revised manuscript addresses their concerns, and we would be grateful if they would consider revising their score. We are happy to discuss any remaining questions.

Reviewer_nU9XOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a framework motivated by hippocampal memory system to study two possible mechanisms previously discussed in neuroscience, one using general representations reusable for multiple queries and the other using specific representation for a given query. The paper builds on existing memory models and trains it with different modes corresponding to these possibilities. Then the paper shows a series of simulations on this model of each mode and reveals its computational properties.

Critical strengths

The method that trains a memory model with the different modes is somewhat new. The direction of the empirical tests on the model, revelant to the hippocampal memory capability, is also not found before.

Critical weaknesses

The goal of the study is rather obscure. Is it mainly for contributing to neuroscience or AI? If it's for neuroscience, what are biological insights obtained from this study? Are the results interpretable as explanation or prediction of neural properties? Is there any implication to neural circuitry? If it's for AI, what kind of AI components are discovered that can be beneficial for potential applications, e.g., performance, in comparison with prior models?

The main findings claimed in intro are some distinct properties of memory when the model is trained in different modes. These findings appear to be not so surprising. If these are related to actual biological experiments in a quantitative manner, it would be much more interesting.

Questions for authors

See above.

3 comments
AuthorsOpenReview ↗

Response to reviewer nU9X

We thank the reviewer for their assessment. We believe their two main concerns, the goal of the paper and the significance of its findings, stem largely from the framing, which we have revised. We address both below.

1. Goal of the study: neuroscience or AI?

The primary contribution is to computational neuroscience. We ask what two computations proposed for the hippocampus, integrative encoding (IE) and retrieval-based inference (RBI), imply when inference is over uncertain causal structure. Our question brings both IE / RBI and uncertainty representation together: if IE or RBI is performed, how should we expect it to represent uncertainty over hidden structure? This is difficult to answer directly in the brain, where the timing of integration cannot be controlled independently of everything else. The matched models with an exact Bayesian reference let us do that.

We show that the distinct internal representations of IE- and RBI-like models do not produce distinct behavior (Table 1, Fig. 5). However, we also identify signatures that do distinguish them: agreement with the exact posterior, generality of the representation, and selective sensitivity to perturbation, etc. These findings inform what future experiments should investigate (see point 3). The new Discussion section makes this explicit.

The results also quantify a trade-off relevant to memory-based ML models, which often make the query-blind/query-conditioned choice implicitly, though this is not our primary aim.

2. Significance of the findings

We agree that some properties follow from the design. For example, R = 1 for query-blind models holds by construction, and the endpoints of the N-query continuum correspond to the two model types. However, to our knowledge, how IE and RBI affect uncertainty representation has not been directly studied before, and the results did not have to turn out as they did. For example:

  • Matched models, different beliefs. At sufficient capacity, identical models except for the timing of integration reach nearly identical accuracy and behavior. A natural expectation would be that they would express similar beliefs. Instead, they show consistent differences in uncertainty representation across capacities.
  • World-level coherence. A model forced to build a general representation of the evidence might be expected to hold the more coherent belief about the world. We find the opposite. Although per-query predictions of query-conditioned models share no representation, they jointly imply a world posterior closer to the oracle.
  • A gradual trade-off. One reasonable expectation is that, because N-query models are query-conditioned, they would always keep an advantage in representing query-specific uncertainty over a query-blind model. Another expectation is that increasing pressure toward a general representation would cause an abrupt shift between the two computation modes, at some threshold. However, instead of a single query-conditioned mode or two distinct modes matching the RBI vs IE dichotomy suggested from literature, we find a gradual spectrum.

3. Relation to biological experiments

We agree that quantitative comparison with biological data is an important next step. To our knowledge, previous studies have not directly investigated how RBI / IE affects uncertainty representations. Closing that gap is the motivation for this work. This paper provides quantified, model-derived predictions that guide future experiments. The revised Discussion states these predictions explicitly:

  • Inference type. RBI's advantage in uncertainty representation should be largest for directly retrievable inferences and shrink for those that require reconstructing the global latent structure.
  • Behavior alone may not separate them. Since the two model types are behaviorally indistinguishable, accuracy and behavioral measures used in previous work (Zeithamova & Preston, 2010; Zeithamova et al., 2012a; Banino et al., 2016; Koster et al., 2018) may miss their representational differences under causal uncertainty.
  • Timing of retrieval. RBI should reactivate query-relevant evidence early in processing.
  • Perturbation. Selectively disrupting query-relevant memories should impair performance more under RBI than under IE.
  • Representational similarity. Memory associated with the same latent structure should be more similar in their representations under IE.
  • Task demands. Anticipating many queries about the same structure may be a method to control whether a subject tends to use RBI or IE.

We hope this clarifies the goal and significance of the work. If our responses address your concerns, we would be grateful if you would consider revising your score. We are happy to discuss any remaining questions.

Reviewer_nU9XOpenReview ↗

Reply

I thank the authors for clarification. However, even with these computational results, my concern about relevance to biology still remains because of lack of quantitive comparison with experimental data. I keep my score.

We understand that the remaining concern is the lack of quantitative comparison with experimental data. We address the reviewer’s concerns below.

  1. Existing experimental data cannot cleanly distinguish RBI from IE. RBI and IE are at their core theoretical hypotheses of how the hippocampus performs its function, and neither can be directly observed or isolated in the brain. Existing experimental paradigms cannot force biological subjects to adopt either RBI or IE to solve the same task. Evidence for either mode comes from indirect behavioral or neural signatures, which do not rule out the other. For example, Zeithamova & Preston (2010) interpreted hippocampal activity during encoding of overlapping pairs as evidence for integration, whereas Kumaran & McClelland (2012) reproduced the pattern in a purely retrieval-based model, without integration. Comparing our model against experimental data interpreted as supporting IE would not validate the model's implementation of IE, since the subject may have relied on RBI. The difficulty of experimentally isolating RBI and IE is exactly one of our core motivations.

  2. Our findings concern computational questions that haven't been directly studied experimentally. As recognized by the reviewer, “The direction of the empirical tests on the model […] is also not found before”. Indeed, our contribution lies in providing model-derived predictions in underexplored scenarios where experimental data are limited. By isolating the timing of integration from other factors, we derive what each theory implies in a causal setting. Previous work has compared the two modes through inference accuracy, neural activation, memory representational similarity, and behaviors (Zeithamova & Preston, 2010; Banino et al., 2016; Koster et al., 2018). One of our findings is that some measures used in prior work, inference accuracy and behavior, fail to distinguish the two modes in a causal setting, even though their internal computations and uncertainty representations differ (Table 1, Fig. 5).

  3. Our findings are consistent with existing results. Where prior data exist, our results are qualitatively consistent with them and extend them. For example, indirectly related memories become more similar under integration (Schlichting et al., 2015; Fig. 2 c; Fig. A.9), query-relevant retrieval feeds later computation under RBI (Kumaran & McClelland 2012; Koster et al., 2018; Fig. 2b, 4b), and balance between modes depends on task demands (Zeithamova & Bowman, 2020; Fig. 6).

  4. Our approach is also consistent with established work at ICLR that investigates biologically motivated computational principles without direct quantitative comparison with experimental data. Examples motivated by hippocampal functions include:

  • Banino et al., MEMO: A deep network for flexible combination of episodic memories (ICLR 2020): a RBI model
  • Krotov & Hopfield, Large Associative Memory Problem in Neurobiology and Machine Learning (ICLR 2021): a theoretical account of associative memory
  • Arani et al., Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System (ICLR 2022): continual learning motivated by complementary learning systems theory
  • Wijmans et al., Emergence of Maps in the Memories of Blind Navigation Agents (ICLR 2023, Outstanding paper award): models how pure egomotion can result in map-like spatial representation.
  • He et al., Building spatial world models from sparse transitional episodic memories (ICLR 2026): models how spatial maps can be built from sparse memories.

Like these works, our contribution is at the computational level. Our aim is to better understand the computational implications of RBI and IE on uncertainty representation in a causal setting. We hope this clarifies the aim of our project and addresses the reviewer’s concern.

Reviewer_vXBHOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work explores two modes of hippocampal-like memory representation (query-blind vs. query-dependent) in architecture/capacity-matched settings on a state traversal task (Alchemy), examining both under behavioral and uncertainty representation differences. Key findings include mostly equal behavioral performance while confirming internal representational strategies largely differ. Extended experiments further show that query-conditioned models output distribution better match a bayesian oracle, including under different retrieval types. Advantages of the query-blind representations are also discussed, overall offering an integrated framework to study these issues.

Critical strengths
  • This work is extremely comprehensive -- comparing the two model types across multiple model body and model sizes, and comparing them at multiple levels spanning performance, task variants (e.g. retrieval types/demands), against oracle distributions, representation analyses, verifying utility of learned representations/predictions via downstream search and policy execution.
  • I find the work offering a balanced yet quantitative view of the relative advantages/differences of the two methods. This experimental setup opens up ways to study many interesting further questions.
  • The writing is dense, but clear and approachable.
Critical weaknesses
  • Figure legibility needs to be improved.
Questions for authors

Below are mostly questions out of curiosity:

  • Do the two models learn/asymptote performance at different speed? Relatedly, another way of comparing the two models is if query-based embedding can be front-loaded, it in theory needs fewer layers to achieve equal performance (at least in the default setup), which looks to be true in Table 1 (1x +q vs. 5x blind with transformers).
  • Do you also see equal behavioral accuracy under different query types?
  • Which method might withstand memory perturbations more if that's introduced during training? Or adapts to environment structure changes? E.g., if you mask the query occasionally during training, do you predict the learning pressure to push the model more quickly converge to query-blind representation or does query-dependent representations remain advantageous to a large degree?
  • In general, any thoughts on further telling the two theories apart? E.g., would you predict results to be different if models were trained under RL?
2 comments
AuthorsOpenReview ↗

Response to Reviewer vXBH

We’re glad that the reviewer found our work comprehensive, balanced, quantitative, easy to read, and that it paves the way to study future questions. We are thankful that our aim to provide a comprehensive comparison between the computational instantiation of RBI and IE in a causal structure was positively received.

We resonate with the reviewer’s curiosity and we address them to the best of our efforts.

1. Learning speeds

We added the training curve (Fig A.7). Query-conditioned ESWM and Transformer approach their validation loss asymptote faster than their query-blind variants, with the gap narrowing as capacity increases. Therefore, query-conditioned models train in fewer steps (Fig A.7), are more parameter efficient, and require less observations to reduce their uncertainty (Fig 3). We agree with the reviewer’s front-loading intuition, which offers a natural explanation for those efficiency advantages, especially at lower capacity. In addition to N-query models with larger N requiring fewer PFLOPs for both training and inference, the convergence rate adds an additional piece to the efficiency comparison and we thank the reviewer for raising it.

2. Equal accuracy for different query types

Yes, we found that both model types are largely indistinguishable at sufficient capacity. We added the accuracy across query types in (Fig A.11). This further supports that similar behaviour does not imply similar uncertainty representation, and we thank the reviewer for highlighting this.

3. Memory perturbation during training

Since memories are not always stored/recalled perfectly in biological systems, we agree that it is a particularly interesting question. To probe this, we trained ESWM and Transformer at 10x capacity where 5%-20% of the memories are injected with Gaussian noise (variance sampled uniformly between 1-3) and reran the analysis for fig 2, fig3, fig4a assuming memories are also noisy at test time. We added Table 5, Fig. A.14, Fig. A.15, Fig. A.16. Overall, injecting noise in the memories did not selectively impact one model type, and mainly replicated the results with clean memories. Both model types’ accuracy remained nearly indistinguishable (Table 5) despite different uncertainty representation (Fig. A.15, Fig. A.16). However, training under noisy memory did noticeably decrease the advantage of query-conditioned models on uncertainty representation during pattern completion queries (from 175x better to 70x better, see Fig A.16 vs Fig A.12). While those preliminary results still leave the question open to more naturalistic memory perturbations, they align with the previous results and we thank the reviewer for this complement to the inference time memory perturbations.

4. Adaptation to structural changes

Since every training sample comes from a different world, the models already learn to adapt to any structure implied by the evidence set. Therefore, both model types could handle a world that changes by automatically replacing outdated memories conflicting with new memories. This would let the model adapt to the new structure as if it never changed. Biological agents, however, must manage memories continually, which makes the problem much harder. Here the two modes may differ. RBI could down-weight outdated memories at retrieval, whereas IE would have to revise an already integrated representation. This is definitely an interesting direction to expand in the future.

5. Occasional masking

We find this suggestion very interesting. Early in this project, we explored training a model to solve both query-conditioned and query-blind samples by masking the query in 50% of the training samples. This did not create sufficient learning pressure to push the model to exclusively adopt a computation mode. When the query was available, this hybrid model represented uncertainty like the exclusively query-conditioned model. When it was masked, it behaved like the exclusively query-blind model. Overall, this seems to support that a single network can support both computations and switch between them depending on task demands, in line with the complementary view in the literature (Zeithamova & Bowman, 2020).

6. RL training

This is a great question. Since section 4.5 shows that behavior does not differentiate the computations and RL rewards behavior, it could shrink the difference in uncertainty representation. However, in a task like Alchemy where uncertainty must guide exploration, RL could still encourage faithful general uncertainty representation. Intuitively, a query-specific representation seems less serviceable for policy learning, since a policy must compare many candidate actions about the same structure. Thus, RL-training may pressure learning toward a more general-purpose representation, diminishing the query specificity similarly to the N-query models.

We thank the reviewer again for their genuine engagement and thoughtful questions.

I thank the authors for the follow up on my questions. As I already gave the highest score, I'll maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.