acceptodds
Under review as a conference paper at ICLR 2027

Spatial representations that transfer by remapping emerge from meta-reinforcement learning

Abstract

Animals and artificial agents navigating partially observable environments construct spatial representations from limited egocentric input. In artificial agents and in models of the hippocampal cognitive map alike, such representations are typically learned with predictive or auxiliary objectives decoupled from reward, then treated as a fixed basis for value learning. Spatial representations can emerge from reward alone, but how they are organised to transfer across environments is unknown. Here we train a recurrent agent with meta-reinforcement learning across grid-worlds with varying home and reward locations, using no spatial priors or auxiliary objectives, and analyse its hidden units as a neural population. With weights fixed, the agent solves novel environments in-context, returning home by a shortest path on 99% of trajectories, even with sensory cues removed. Its hidden states form a spatial representation: units develop localised fields, position is linearly decodable, and hidden-state similarity tracks spatial distance when the agent navigates from memory but not when its goal is in view. Across unmarked home locations this representation remaps selectively, with a subset of units tracking the home while others maintain a largely stable spatial code. These remapping units determine where the agent goes: ablating them degrades home decoding far more than ablating any other units, and overwriting their state once, with the pattern for a different home, sends it to that false home. These properties resemble the hippocampus, where dedicated cells follow a remembered goal while the rest of the map is largely unchanged, and which is required for navigation from memory but not towards a visible goal. Reward maximisation alone thus yields a spatial representation that transfers across environments, so a spatial basis need neither be fixed in advance nor relearned in new environments.

Primary area:
applications to neuroscience & cognitive science
Keywords:
meta-reinforcement learning, spatial representations, remapping, hippocampus, cognitive map, in-context learning, emergent representations, spatial navigation, recurrent neural networks

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZbzKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper trained a recurrent neural network agent to do a home-reward task across multiple grid environments using reinforcement learning. The paper reported units tuned to task-relevant locations, population activity tracking task variables (e.g., agent, home, reward locations), units that saliently encode home location and causally affect the agent’s behaviour, and population activity reflecting spatial distances in the network’s hidden state.

Critical strengths
  1. The paper fills a genuine research gap by showing that unit-level and population-level spatial representation can emerge from reinforcement learning, and that the units organize into home-location-tracking units that remap and relatively stable units. The results are meaningful contributions to the broader NeroAI and hippocampus community.

  2. The causal effect of manipulating units encoding home locations is compelling and insightful.

Critical weaknesses

Major weaknesses

  1. Why do the exploration paths in Fig. 2b look almost as optimal as the shortest paths (i.e., as if the agent knows where to go)? There are more turns than in the optimal paths, but still, I would expect more suboptimal paths. Why does Fig. 2d only measure the number of excess forward steps but not rotation steps? I would assume that rotation is also a big part of exploration.

  2. For Fig. 3, the authors should first show that the spatially tuned units are actually spatially tuned and not visually tuned (views of boundaries can inform location, even when sensory cues are removed). The authors could compare the learned representation against an observation-only encoder to test whether spatial tuning persists after controlling for visual similarity. Second, the authors need to show that these units arise from training and do not emerge in untrained networks.

  3. For Fig. 4, the authors need to show that position/reward/home could not be decoded from an observation-only model and could not be decoded from an untrained network.

  4. For Fig. 6, the authors should first demonstrate that the latent space structure indeed reflects spatial similarity and not visual similarity, and second, demonstrate that this structure arises from training. I recommend two controls: 1) compare trained and untrained networks on matched trajectories, and 2) compare spatial RSA against visual-similarity RSA. These controls are important to support the claim that “Hidden-state similarity therefore tracks spatial distance“.

  5. The environment size is 10 x 10 while the agent's FOV is 7 x 7. This means the agent, at any time, can see the majority of the environment. This would weaken the requirement for memory-based navigation. Why didn’t the authors consider also running the analysis with a smaller FOV or a larger environment (like the models in Fig. A2)?

  6. How well do the major representation and intervention results replicate across models trained with different random seeds?

Minor weakness

  1. Fig. 5 is interesting. However, a comparison with a random-policy agent to show that the agent preferentially goes to the false home after intervention is recommended. A comparison with an agent that optimally goes to the false home is also recommended.

  2. Fig. 1c could be made clearer by indicating that each environment corresponds to different trials.

  3. Fig. 3b could benefit from showing what the remapping looks like for Wasserstein-distance = 0.2, 0.4, 0.8, etc., so that readers have a better idea of what these values mean. Similarly, Fig. 3c would also benefit from showing what place fields look like for Spatial Information = 0.5, 1.0, 2.0, etc.

  4. The abstract says “units develop a localised field”. This could be made more specific by saying “units develop a localised field at home locations” unless the authors can demonstrate that the units with high SI track locations other than the home locations.

Questions for authors
  1. Fig. 2e is unclear. What are the x and y axes? Is the heatmap over home locations or reward locations? I wasn’t able to understand how this shows that “>99% follow a shortest path to every home location”.

  2. Fig. 3b and Fig. 3c independently show that there is a small portion of units that remap and have high SI. Are they the same group of units? Fig. 3d shows some qualitative examples, but it would be stronger to demonstrate this quantitatively.

  3. The experimental procedure in Fig. 3a is unclear. I wasn’t able to understand what the analysis is and why it is done. Can the authors clarify?

  4. Fig. 4a, c, e, and g are unclear. What are the red box and red dot? What’s the blue box? What’s the long tail showing? What does probability state mean?

  5. In L454–455, the authors say “while the others retain a largely stable spatial code”. Presumably, this refers to the units with low Wasserstein distance. Why are they called “spatial”? These units have not been shown to have high Spatial information.

  6. L462 says “a continuous trajectory” while the experiment uses a discrete environment. Can the authors clarify what they mean by “continuous”?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ZbzK (1/2)

Thank you for the detailed and constructive review. The suggested controls have significantly strengthened the manuscript. The revised manuscript now includes an untrained network on matched trajectories (new Figures A4 to A6 and A9), the removal of every visual cue including the wall colours (new Figures A4 and A5), a control for visual similarity (Appendix A.3.6), a 5 × 5 observation window for all main results, replication across multiple seeds (new Figure A10), and comparisons with random-action and optimal agents (new Figure A8). The conclusions hold in each case. In light of these changes, we would be grateful if you would reconsider your score. This comment covers the major weaknesses, and a second covers the minor weaknesses and questions. Figure numbers refer to the revised manuscript.

Major weaknesses

"Why do the exploration paths in Fig. 2b look almost as optimal as the shortest paths … Why does Fig. 2d only measure the number of excess forward steps but not rotation steps?"

You are right on both counts. With a 7 × 7 view the agent could see most of the arena, so the new reward was usually already visible when it reached home. Explore trajectories contain ~2.24 excess turns and ~1.87 excess forward steps, whereas home trajectories average only 0.10 excess actions. We have now replaced the main results with an agent that has a 5 × 5 view, and the new Figure 2e reports excess turns and excess forward steps separately.

"…the authors should first show that the spatially tuned units are actually spatially tuned and not visually tuned … Second, the authors need to show that these units arise from training and do not emerge in untrained networks."

We have added both controls. To test for visual tuning, we removed every remaining visual cue, including the colours of the arena walls, so that many locations look identical. Visually tuned units would lose their spatial information. Instead, the units kept their fields and their spatial information (new Figures A4b and A5a, e). They are also tuned to the home, which the view does not determine. To test the effect of training, we ran an untrained network on trajectories matched to those of the trained agent, and no such units appeared (new Figures A4b and A5b, d).

"For Fig. 4, the authors need to show that position/reward/home could not be decoded from an observation-only model and could not be decoded from an untrained network."

Position, reward and home cannot be decoded accurately from an untrained network on matched trajectories (new Figure A6c). This network receives the same observations as the trained one, so the visual input passed through an untrained copy of the architecture does not support the decoding. In addition, all analyses are run with the sensory cues removed, so only the arena boundary and the reward are visible, and the home is unmarked: it never appears in the observation, yet it is decoded even while the agent explores away from it (Figure 4g–h).

"I recommend two controls: 1) compare trained and untrained networks on matched trajectories, and 2) compare spatial RSA against visual-similarity RSA."

For the first control, spatial RSA is far lower in an untrained network on matched trajectories: about 0.16 in the home phase against 0.50 in the trained network, and close to zero in the explore phase (new Figure A9f). For the second, we repeated the analysis with a visual distance, the Euclidean distance between the feature vectors that the recurrent network receives. Among pairs of time steps with identical feature vectors at different locations, hidden-state distance still followed spatial distance (explore phase ρ = 0.40 ± 0.05, home phase ρ = 0.55 ± 0.02; mean ± s.e.m. across home locations). Spatial RSA was also largely preserved after partialling out visual distance (explore phase 0.34 → 0.27, home phase 0.47 → 0.44), and the partial correlation was positive for every home location in both phases. These results are in Appendix A.3.6.

"The environment size is 10 x 10 while the agent's FOV is 7 x 7 … This would weaken the requirement for memory-based navigation."

We agree, and this prompted the main change to the manuscript. The main results now come from a network with a 5 × 5 view, which covers at most a quarter of the arena, and the 7 × 7 results have moved to the appendix. With either view the home is unmarked, so it is never visible and can only be reached from memory.

"How well do the major representation and intervention results replicate across models trained with different random seeds?"

They replicate across 3 seeds (new Figure A10). Two further seeds are still training, and we will add them when they finish.

AuthorsOpenReview ↗

Response to Reviewer ZbzK (2/2)

This second comment covers the minor weaknesses and questions.

Minor weaknesses

"…a comparison with a random-policy agent to show that the agent preferentially goes to the false home after intervention is recommended. A comparison with an agent that optimally goes to the false home is also recommended."

We have added both (new Figure A8a–c). The injected agent spends more time near the false home, and less near the real one, than a random-action agent. Unlike an agent that navigates optimally to the false home, it also visits the locations rotationally symmetric to it.

"Fig. 1c could be made clearer by indicating that each environment corresponds to different trials."

Thank you for this suggestion. Figure 1c now states that a new trial is sampled, with different sensory patterns, home and reward locations.

"Fig. 3b could benefit from showing what the remapping looks like for Wasserstein-distance = 0.2, 0.4, 0.8, etc. … Similarly, Fig. 3c would also benefit from showing what place fields look like for Spatial Information = 0.5, 1.0, 2.0, etc."

Thank you for this suggestion. We now show example units across the range of Wasserstein distances, from 0.09 to 0.67, and across a range of spatial information (Figures 3d and 5a, and new Figures A4 and A7a).

"The abstract says 'units develop a localised field'. This could be made more specific by saying 'units develop a localised field at home locations'…"

Agreed. The abstract now reads "units develop a localised field at home locations".

Questions

"Fig. 2e is unclear. What are the x and y axes? Is the heatmap over home locations or reward locations?"

We agree that this panel was unclear. It was a heatmap over home locations, coloured by the mean excess steps on returns to each one. We have replaced it: Figure 2e now shows excess actions for explore and home trajectories, and Figure 2f shows excess steps on successive returns home. The proportion of returns that follow a shortest path (more than 98%) is given in the text.

"Fig. 3b and Fig. 3c independently show that there is a small portion of units that remap and have high SI. Are they the same group of units?"

They overlap. Units that remap more tend to have higher spatial information (Spearman ρ = 0.32, p = 1.2 × 10⁻⁷), and the new Figure 3c shows the measures together for every unit.

"The experimental procedure in Fig. 3a is unclear. I wasn't able to understand what the analysis is and why it is done."

Thank you for flagging this. We have rewritten Section 5.1. The analysis asks whether the representation is shared across home locations or rebuilt for each one. Because the arena is symmetric, a shared map could be reused by rotating it, so we compared the population rate map for each home location with rotated copies of a reference map. Each home location closely matched one rotation, giving four clusters, one per quadrant.

"Fig. 4a, c, e, and g are unclear. What are the red box and red dot? What's the blue box? What's the long tail showing? What does probability state mean?"

Apologies for the confusion. The colour of each cell shows the probability that the decoder assigns to that location, which is what "probability state" meant. The dot is the agent and the tail is its path so far.

The boxes were not separate markers: they were cells to which the decoder assigned high probability. We have redrawn the figure with the agent and its path in grey, and the caption now names them.

"In L454–455, the authors say 'while the others retain a largely stable spatial code' … Why are they called 'spatial'? These units have not been shown to have high Spatial information."

You are right, and we now describe these units as a "largely stable code". What we can say about space concerns the population: position can still be decoded when either the remapping or the stable units are ablated (new Figure A7).

"L462 says 'a continuous trajectory' while the experiment uses a discrete environment."

Thank you, we meant uninterrupted, not continuous in space. The agent is never reset or teleported between episodes, so each trial is a single unbroken path. We have reworded this in the revised manuscript.

If there is a further analysis you would like to see, please tell us and we will try to add it during the discussion period.

I appreciate the author's detailed responses and additional experiments. My previous criticisms are largely resolved, and the manuscript has improved substantially. I'm therefore happy to increase my rating to 3.

However, I do agree with reviewer GKmn that the study would benefit from more complex environments. For example, introducing random obstacles across environments in the current grid setup would help establish whether the learned spatial representation can transfer across different spatial structures, making the study more compelling. Nonetheless, I think the study in its current form still provides interesting insights.

Reviewer_U1A4OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Neurons in the brain show highly structured representations of space, including place and grid cells. How do these cells arise? What computational principles do they serve? There is a big literature on this topic, ranging from authors handcrafting networks that show such representations, to normative theory that derive such cells from various optimality principles.

Yet another line is training networks. People in the past have shown that trained networks both on unsupervised or reward objectives develop such representations.

This paper takes the path of a reward objective, and shows that spatial representations developed in such networks also show partial remapping as in the hipppocampus.

Critical strengths

The papers main strength is its careful analysis of the spatial representations learned by the model in Section 5.

The paper is also very well written and easy to read. I want to thank the readers for that, my reviewer batch is full of unreadable papers, this was a breath of fresh air.

Critical weaknesses

In my opinion, this is a nice study, however its findings are not sufficient for a paper. One major weakness is lack of quantitative comparisons with data. The paper's main motivation of spatial representations and their remapping are motivated by experimental findings in neuroscience. The analyses presented in Section 5 are also motivated by neuroscience findings, otherwise why would one look at such things? However, because no quantitative comparisons to real data are made, it is not clear what the conclusion from these analyses are. Did we succeed or not in understanding something about the brain? There is some small paragraph in the discussion section, but this is not enough.

If the authors want to stay with only qualitative comparisons, then their model should provide something more than just qualitatively reproducing what has already been seen. For example, they can make some new predictions about experiments, or provide some new mechanistic insight.

It is also possible that I am not getting the main motivation here, I am open to hearing from the authors.

Authors should remove "Spatial representations emerge from reward alone" from a contribution. By their own literature review, this was shown by previous work.

Questions for authors

My main questions are above.

In addition, authors could provide some more results about the robustness of their conclusions to hyperparameter choices of their simulations, e.g. why 7x7 view but not something smaller or larger?

8 comments
AuthorsOpenReview ↗

Response to Reviewer U1A4

Thank you for your review, and for the kind words about the writing. The suggestions have significantly improved the manuscript. Three predictions for hippocampal recordings have been added to the Discussion, the first contribution now distinguishes itself from previous work, and the main results now use a smaller observation window, replicated across multiple seeds (new Figure A10). In light of these changes, we would be grateful if you would reconsider your score. We reply to each point below. Figure numbers refer to the revised manuscript.

"One major weakness is lack of quantitative comparisons with data … Did we succeed or not in understanding something about the brain? … It is also possible that I am not getting the main motivation here, I am open to hearing from the authors."

This is a fair challenge, and you are right that we make no quantitative comparison with neural data. Our main motivation was a question about reinforcement learning: how can a spatial representation be reused in a new environment without being fixed in advance or relearned? We used analyses from neuroscience because remapping studies ask the same thing, namely which cells change between environments and which do not. What the model offers neuroscience is a mechanism and a set of testable predictions, which we set out under the next point.

"…their model should provide something more than just qualitatively reproducing what has already been seen. For example, they can make some new predictions about experiments, or provide some new mechanistic insight."

We agree, and thank you for the suggestion. The mechanistic insight is that a goal can be stored by remapping a small subpopulation of units while the rest of the map is left unchanged, and that overwriting this subpopulation alone redirects behaviour (Figure 5). From this we have added three predictions to the Discussion:

  • A dedicated population of cells should follow a remembered goal in two-dimensional environments while the remaining cells stay largely stable (Figure 3).

  • Targeted reactivation of these cells should be sufficient to redirect an animal towards a different goal (Figure 5d–e).

  • The noise in the spatial code should depend on task demands: the excess variability of place cell firing, termed overdispersion, should be lower when an animal navigates from memory than when it is guided by a visual cue (Figure 6).

"Authors should remove 'Spatial representations emerge from reward alone' from a contribution."

We agree that this was too broad. The contribution now reads "Spatially tuned units emerge from reward alone". Previous work showed that spatial information can be decoded from the memory of agents trained on reward (Wijmans et al., 2023), but decodability does not by itself show that any single unit is tuned to space. We show this tuning directly, in the rate maps of individual units, and show that these units support transfer across environments.

"…why 7x7 view but not something smaller or larger?"

Thank you for this suggestion. The main results now come from a network with a 5 × 5 view, and they match those of the 7 × 7 network, which has moved to the appendix (new Figures A5, A6, A8 and A9). The results also reproduce across 3 seeds (new Figure A10). Two further seeds are still training, and we will add them when they finish.

If there is a specific comparison with data that you would like to see, please tell us and we will try to add it during the discussion period.

Thanks for the rebuttal. I will take some time to think about the rest of the rebuttal and respond to it, but just wanted to quickly reply to this point below to give the authors a chance to respond further, since this was my main point of criticism.

"If there is a specific comparison with data that you would like to see, please tell us and we will try to add it during the discussion period."

I think it is primarily the authors' responsibility to tell the readers what quantitative comparisons could have been made with existing data in the literature and have been omitted, and why. Was that because the model wasn't predictive at that level?

Dear Reviewer U1A4,

Thank you for the follow-up and clarification. Quantitative comparisons with existing data are possible in principle, for example the proportion of goal-coding cells (Gauthier & Tank, 2018; Sarel et al., 2017) or the degree of overdispersion (Fenton & Muller, 1998). We did not make them because the model is not designed to match these quantities. The agent is trained on a single task, whereas hippocampal cells are recorded in animals with a far broader behavioural repertoire, so the fraction of units devoted to the goal in the model need not match the fraction of goal cells in an animal. Likewise, model units have no spiking noise, so their firing statistics cannot be compared directly with those of recorded neurons. We have added this to the Discussion (L500).

What we do show is how spatial representations that emerge from reward alone, without auxiliary objectives, are organised to support goal-directed behaviour in unseen environments. Previous models have typically learned spatial representations through predictive or auxiliary objectives separate from reward and then held them fixed, so they cannot show how the representations reorganise when the goal moves. This leads to predictions that go beyond reproducing known observations, set out in Predictions (L486): the goal-following population should be sufficient to redirect navigation, and activity should be less variable when navigation is guided by memory.

Reviewer_GKmnOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies spatial representations in a recurrent meta-RL agent navigating a grid world. It identifies neurons whose activity changes with the home location and shows that modifying their activity can redirect the agent's homing behavior.

Overall, the causal result is interesting, but I do not find the contribution sufficiently substantial or well supported in its current form.

Critical strengths
  • The intervention provides useful causal evidence connecting the representations to behavior.
Critical weaknesses
  • I am not convinced that the paper offers a sufficiently substantial contribution beyond prior work, such as [1]. Learning without explicit coordinate inputs is a useful difference, but the paper needs to establish more clearly what additional insight selective remapping provides.
  • All representation and intervention analyses use a single trained network. The behavioral results across multiple seeds do not establish that the reported organization of the representations is reproducible.
  • The main analyses are limited to a small, open grid world. This substantially limits what we can conclude about the role of these representations in navigation and transfer.
  • I found Section 5.1 particularly difficult to follow. The construction and interpretation of the population rate maps and their rotations as well as the overall motivation need clearer explanations.
  • The presentation could also be improved. The figures often contain too many small panels squashed together. Together with their overly long captions, they take up substantial space that could be used for further analysis, intuition, and explanation.

[1] Wijmans et al., Emergence of Maps in the Memories of Blind Navigation Agents, ICLR 2023.

Questions for authors
  1. Can position still be decoded accurately using only the neurons whose activity remains similar across home locations?
  2. When replacing activity with activity recorded for a different home, the positions are matched. Is the direction the agent is facing also matched?
4 comments
AuthorsOpenReview ↗

Response to Reviewer GKmn

Thank you for the review and the clear, specific questions. These comments have significantly improved the manuscript, which now states more clearly what selective remapping adds beyond previous work and shows that the main results reproduce across multiple seeds (new Figure A10). We reply to each point below. If any concern remains, please tell us and we will try to address it during the discussion period. Otherwise, we would be grateful if you would reconsider your score. Figure numbers refer to the revised manuscript.

Weaknesses

"I am not convinced that the paper offers a sufficiently substantial contribution beyond prior work, such as [1] … the paper needs to establish more clearly what additional insight selective remapping provides."

We agree that the original text did not make this clear enough. In Wijmans et al., the agent receives its position (relative to the start) and the location of its goal as input at every step. In our task, the agent receives neither its position nor the location of its home, which is unmarked and has to be held in memory. This makes it possible to ask how a goal is stored and what changes when the goal moves, which their setting cannot address. A subpopulation of units encodes the home location and remaps with it while the other units change little, and overwriting this subpopulation once is sufficient to send the agent to a false home (Figure 5). This provides a mechanism for reusing a spatial representation across environments, beyond showing that a map can be decoded from memory. The first contribution now makes this comparison explicitly clear (line ~58).

"All representation and intervention analyses use a single trained network."

This was a fair concern. The main representation and intervention results now reproduce across 3 seeds (new Figure A10). Two further seeds are still training, and we will add them when they finish.

"The main analyses are limited to a small, open grid world."

We chose a simple environment so that the representations are easy to interpret and the causal interventions are clean. Previous meta-reinforcement learning studies of navigation either used mazes with limited movement options (Duan et al., 2016; Mirowski et al., 2017) or gave the agent its position directly (Zintgraf et al., 2020). To make the task more demanding, we have now reduced the agent's observation window from 7 × 7 to 5 × 5, so that it sees at most a quarter of the arena at a time. Despite this increasing the effective size of the environment relative to what the agent can observe, the main results are unchanged, and the 7 × 7 results are now in the appendix. The agent's behaviour also carries over without retraining to trials twice as long and to arenas of other sizes (Appendix A.3.2, Figures A2 and A3).

"I found Section 5.1 particularly difficult to follow."

Thank you for flagging this. We have rewritten the section, and the idea is as follows. Because the arena is symmetric, a single map could be reused across home locations by rotating it. To test this, we compared the population rate map for each home location with rotated copies of a reference map. Each home location closely matched one rotation, and home locations fell into four clusters, one per quadrant of the arena (Figure 3a). The representation is therefore shared across home locations, not rebuilt for each one.

"The figures often contain too many small panels squashed together. Together with their overly long captions, they take up substantial space…"

We have simplified the figures where we could. Figure 3 now has four panels instead of six, supporting examples have moved to the appendix, and most captions are shorter.

Questions

"Can position still be decoded accurately using only the neurons whose activity remains similar across home locations?"

Yes. When the remapping units are ablated, position decoding is impaired no more than when the same number of random units are ablated, which has been added as new Figure A7. The stable units therefore carry enough information about position on their own. This has been made more clear in the revised manuscript (line ~365).

"When replacing activity with activity recorded for a different home, the positions are matched. Is the direction the agent is facing also matched?"

No. The injected pattern is averaged over all the directions the agent faced at the matched position. Therefore redirection does not depend on matching heading. This has been made more clear in the revised manuscript (line ~370).

[1] Wijmans et al. (2023). Emergence of Maps in the Memories of Blind Navigation Agents. ICLR.

I appreciate the revisions and I apologize for the late response. The addition of evaluations with multiple seeds is valuable and I believe it is crucial for the paper's claims to be appreciable. That being said, I remain unconvinced by the broader contribution, given the paper's title "Spatial Representations That Transfer by Remapping Emerge from Meta-Reinforcement Learning". To the best of my knowledge, even much earlier meta-RL work, such as RL^2, studied navigation in more complicated, procedurally generated mazes with varying layouts. Other RL work has also studied spatial representations in substantially more challenging environments (e.g., Wijmans et al.).

Comparatively, the main experiments here use a fixed, open 10x10 gridworld. Although home and reward locations vary, the underlying spatial geometry remains unchanged. In such a setting, "homing" could be solved by maintaining relative displacement rather than a reusable spatial map. As such, finding home and position information in the network's representations is not particularly surprising.

My main concern is that, given a fixed geometry, a reusable goal encoding does not necessarily imply a reusable spatial map. The observed representations could reflect something much simpler, such as relative position, goal vectors, and the symmetry of the gridworld. While the selective remapping and causal interventions are interesting, I find it difficult to see how the results establish the broader claim of transferable spatial representations beyond the highly restricted gridworld setting studied here. Given that other reviewers have also raised similar concerns about the depth of the contributions, I maintain my score.

We thank the reviewer for their response. Although the reviewer is maintaining their score, we will take the opportunity to clarify the points raised and address their remaining concerns.

"even much earlier meta-RL work, such as RL^2, studied navigation in more complicated, procedurally generated mazes with varying layouts. Other RL work has also studied spatial representations in substantially more challenging environments (e.g., Wijmans et al.)."

Our contribution is not to claim that our navigation task is more challenging than these environments, nor that the meta-RL agents' ability to navigate is itself novel. Rather, our focus is on what spatial representations emerge as a consequence of learning to navigate without explicit spatial inputs or priors. To reiterate, in Wijmans et al., the agent receives its position (relative to the start) and the location of its goal as input at every step. In our task, the agent receives neither its position nor the location of its home, which is unmarked and has to be held in memory.

"My main concern is that, given a fixed geometry, a reusable goal encoding does not necessarily imply a reusable spatial map. The observed representations could reflect something much simpler, such as relative position, goal vectors, and the symmetry of the gridworld.

"As such, finding home and position information in the network's representations is not particularly surprising."

A reusable goal encoding is itself a form of spatial representation. The fact that it might be implemented as a relative displacement or goal vector does not make it non-spatial. Wijmans et al. (2023) illustrate that spatial representations in reinforcement-learning agents can take different forms, ranging from collision-detection neurons (Fig. 1C) to decodable metric (occupancy) maps of the environment (Fig. 4). We therefore do not consider a goal-related or relative-displacement representation to be incompatible with the broader notion of a spatial representation. Our contribution is not simply to show that spatial or goal-related information is encoded in the network. Rather, we show that this information is represented through individual units whose activity selectively remaps with changes in the home location.

"Comparatively, the main experiments here use a fixed, open 10x10 gridworld. Although home and reward locations vary, the underlying spatial geometry remains unchanged."

"I find it difficult to see how the results establish the broader claim of transferable spatial representations beyond the highly restricted gridworld setting studied here".

We agree that our experiments do not establish that the learned representation will transfer to arbitrary environment geometries. However, this is not the sense of transferability we intend to claim. Instead, our claim is that a spatial representation learned through reward maximisation can be reused across different instances of the task distribution, without changing the network weights.. Specifically, despite changes in home and reward locations, the network preserves a largely shared code while selectively remapping a subset of units to track the current home.

We acknowledge that testing whether this organisation generalises to substantially different geometries, such as richer environments and procedurally generated mazes, is an important direction for future work. Nevertheless, it is worth putting the geometry of our environment in the context of previous meta-RL navigation studies. For example, Duan et al. use 5 × 5 and 9 × 9 mazes (RL^2, Fig. 4), but these consist of narrow corridors separated by walls. While these environments introduce greater topological complexity, movement is restricted to the corridor network, making the effective navigable state space more limited than the grid dimensions alone suggest. By contrast, our open 10 × 10 grid allows movement throughout the environment, providing a larger accessible spatial area and a broader range of possible transitions. Our task is designed to capture the demands of navigating this space through both exploration and memory-guided homing. Thus, the use of a gridworld does not in itself imply a particularly restricted navigational setting.

Dear Reviewer GKmn,

Although home and reward locations vary, the underlying spatial geometry remains unchanged

Before the rebuttal period ends, we kindly ask you to review our most recent comment to all readers (https://openreview.net/forum?id=i0eQJlvGmv&noteId=NK37YZiH5T). It includes new preliminary results that address this point directly (Appendix A.3.8, Figure A11). In short, the way remapping units are organised across home locations carries over to environments with internal barriers, and their spatial fields respect those barriers.

"I am not convinced that the paper offers a sufficiently substantial contribution beyond prior work, such as [1] … the paper needs to establish more clearly what additional insight selective remapping provides more clearly."

We would also like to point you to our recent response to Reviewer U1A4 (https://openreview.net/forum?id=i0eQJlvGmv&noteId=Bj97IPLDZs) which clarifies what additional insight selective remapping provides more clearly.

We hope these additions address your concerns and that you might consider revisiting your score.

Comments on the paper

4 comments
AuthorsOpenReview ↗

General response and summary of changes

We thank all three reviewers for their careful and constructive reviews, which have improved the manuscript. We were pleased that reviewers found the causal intervention "compelling and insightful" (ZbzK) and "useful causal evidence" (GKmn), the manuscript "very well written and easy to read" with a "careful analysis of the spatial representations" (U1A4), and that the work "fills a genuine research gap" (ZbzK).

Several points were raised by more than one reviewer, so we summarise the main changes here before replying to each review. Figure numbers in our replies refer to the revised PDF.

  1. A smaller observation window. The main results now come from a network with a 5 × 5 view instead of 7 × 7, so the agent sees at most a quarter of the arena at any time. The 7 × 7 results have moved to the appendix.

  2. Replication across seeds. The main results reproduce across 3 seeds (new Figure A10). Two further seeds are still training, and we will add them when they finish.

  3. Controls. An untrained network following matched trajectories shows no spatially informative units, does not support position/reward/home decoding, and shows much weaker similarity structure (new Figures A4 to A6 and A9). Spatially tuned units also keep their fields when every visual cue is removed, including the colours of the arena walls (new Figures A4 and A5). Spatial structure in the latent space also persists when controlling for visual similarity in the observations (new Section A.3.6).

  4. A clearer contribution. The first contribution now reads "Spatially tuned units emerge from reward alone" and is set against Wijmans et al. (2023). We have also added predictions for hippocampal recordings to the Discussion.

  5. Clearer presentation. We rewrote Section 5.1 and simplified the figures and their captions.

Dear reviewers,

We are now about halfway through the discussion period. Please read through the authors' rebuttal and respond if there are points that remain unclear (or to let them know they have addressed your concerns).

Sincerely,

AC

Dear reviewers,

We have uploaded a new revision in which the main results reproduce across 5 seeds (new Section A.3.7; Figure A10). To make the rebuttal process easier to follow, the main changes made in response to each reviewer are now colour-coded:

GKmn: Blue\color{blue}{\textbf{Blue}}
U1A4: Brown\color{brown}{\textbf{Brown}}
ZbzK: Red\color{red}{\textbf{Red}}

Dear reviewers,

Thank you to GKmn and ZbzK for their thoughtful responses, and in particular to ZbzK for increasing their score.

"However, I do agree with reviewer GKmn that the study would benefit from more complex environments. For example, introducing random obstacles across environments in the current grid setup would help establish whether the learned spatial representation can transfer across different spatial structures."

In our most recent response to GKmn, we acknowledged that "testing whether this organisation generalises to substantially different geometries, such as richer environments and procedurally generated mazes, is an important direction for future work."

We would therefore like to ask our remaining reviewer, U1A4, and, if appropriate, GKmn, to reconsider their scores in light of some last-minute analyses we are conducting before the rebuttal period concludes.

We now have preliminary evidence (Appendix A.3.8, Figure A11) suggesting that the organisation of units across home locations generalises to grid-world environments containing internal barriers, provided that such environments are represented in the training distribution. More specifically, we have trained an additional model across three different environments. Our preliminary results show that the remapping of units across home locations remains consistent across these environments: units that remap do so to the same home locations across environments, even when internal barriers are present. Furthermore, the spatial fields of these units respect the internal barriers rather than extending through them (Gustafson and Daw, 2011).

We emphasise that these preliminary results are currently from a single trained network. If given the opportunity to continue this work beyond the rebuttal period, we intend to repeat these experiments across five random seeds.

These results provide initial evidence that the learned spatial organisation is not restricted to a single, fixed geometry, but can accommodate changes in spatial structure and observations (i.e., line-of-sight constraints from internal barriers), provided these are represented in the training distribution. We believe this directly addresses the concern raised regarding the transfer of learned spatial representations across environments with more complex spatial structures and perceptual constraints. However, we stress that this is not the main focus of our contributions, and we leave a more detailed analysis for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.