- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper trained a recurrent neural network agent to do a home-reward task across multiple grid environments using reinforcement learning. The paper reported units tuned to task-relevant locations, population activity tracking task variables (e.g., agent, home, reward locations), units that saliently encode home location and causally affect the agent’s behaviour, and population activity reflecting spatial distances in the network’s hidden state.
Critical strengths
-
The paper fills a genuine research gap by showing that unit-level and population-level spatial representation can emerge from reinforcement learning, and that the units organize into home-location-tracking units that remap and relatively stable units. The results are meaningful contributions to the broader NeroAI and hippocampus community.
-
The causal effect of manipulating units encoding home locations is compelling and insightful.
Critical weaknesses
Major weaknesses
-
Why do the exploration paths in Fig. 2b look almost as optimal as the shortest paths (i.e., as if the agent knows where to go)? There are more turns than in the optimal paths, but still, I would expect more suboptimal paths. Why does Fig. 2d only measure the number of excess forward steps but not rotation steps? I would assume that rotation is also a big part of exploration.
-
For Fig. 3, the authors should first show that the spatially tuned units are actually spatially tuned and not visually tuned (views of boundaries can inform location, even when sensory cues are removed). The authors could compare the learned representation against an observation-only encoder to test whether spatial tuning persists after controlling for visual similarity. Second, the authors need to show that these units arise from training and do not emerge in untrained networks.
-
For Fig. 4, the authors need to show that position/reward/home could not be decoded from an observation-only model and could not be decoded from an untrained network.
-
For Fig. 6, the authors should first demonstrate that the latent space structure indeed reflects spatial similarity and not visual similarity, and second, demonstrate that this structure arises from training. I recommend two controls: 1) compare trained and untrained networks on matched trajectories, and 2) compare spatial RSA against visual-similarity RSA. These controls are important to support the claim that “Hidden-state similarity therefore tracks spatial distance“.
-
The environment size is 10 x 10 while the agent's FOV is 7 x 7. This means the agent, at any time, can see the majority of the environment. This would weaken the requirement for memory-based navigation. Why didn’t the authors consider also running the analysis with a smaller FOV or a larger environment (like the models in Fig. A2)?
-
How well do the major representation and intervention results replicate across models trained with different random seeds?
Minor weakness
-
Fig. 5 is interesting. However, a comparison with a random-policy agent to show that the agent preferentially goes to the false home after intervention is recommended. A comparison with an agent that optimally goes to the false home is also recommended.
-
Fig. 1c could be made clearer by indicating that each environment corresponds to different trials.
-
Fig. 3b could benefit from showing what the remapping looks like for Wasserstein-distance = 0.2, 0.4, 0.8, etc., so that readers have a better idea of what these values mean. Similarly, Fig. 3c would also benefit from showing what place fields look like for Spatial Information = 0.5, 1.0, 2.0, etc.
-
The abstract says “units develop a localised field”. This could be made more specific by saying “units develop a localised field at home locations” unless the authors can demonstrate that the units with high SI track locations other than the home locations.
Questions for authors
-
Fig. 2e is unclear. What are the x and y axes? Is the heatmap over home locations or reward locations? I wasn’t able to understand how this shows that “>99% follow a shortest path to every home location”.
-
Fig. 3b and Fig. 3c independently show that there is a small portion of units that remap and have high SI. Are they the same group of units? Fig. 3d shows some qualitative examples, but it would be stronger to demonstrate this quantitatively.
-
The experimental procedure in Fig. 3a is unclear. I wasn’t able to understand what the analysis is and why it is done. Can the authors clarify?
-
Fig. 4a, c, e, and g are unclear. What are the red box and red dot? What’s the blue box? What’s the long tail showing? What does probability state mean?
-
In L454–455, the authors say “while the others retain a largely stable spatial code”. Presumably, this refers to the units with low Wasserstein distance. Why are they called “spatial”? These units have not been shown to have high Spatial information.
-
L462 says “a continuous trajectory” while the experiment uses a discrete environment. Can the authors clarify what they mean by “continuous”?
3 comments
Response to Reviewer ZbzK (2/2)
This second comment covers the minor weaknesses and questions.
Minor weaknesses
"…a comparison with a random-policy agent to show that the agent preferentially goes to the false home after intervention is recommended. A comparison with an agent that optimally goes to the false home is also recommended."
We have added both (new Figure A8a–c). The injected agent spends more time near the false home, and less near the real one, than a random-action agent. Unlike an agent that navigates optimally to the false home, it also visits the locations rotationally symmetric to it.
"Fig. 1c could be made clearer by indicating that each environment corresponds to different trials."
Thank you for this suggestion. Figure 1c now states that a new trial is sampled, with different sensory patterns, home and reward locations.
"Fig. 3b could benefit from showing what the remapping looks like for Wasserstein-distance = 0.2, 0.4, 0.8, etc. … Similarly, Fig. 3c would also benefit from showing what place fields look like for Spatial Information = 0.5, 1.0, 2.0, etc."
Thank you for this suggestion. We now show example units across the range of Wasserstein distances, from 0.09 to 0.67, and across a range of spatial information (Figures 3d and 5a, and new Figures A4 and A7a).
"The abstract says 'units develop a localised field'. This could be made more specific by saying 'units develop a localised field at home locations'…"
Agreed. The abstract now reads "units develop a localised field at home locations".
Questions
"Fig. 2e is unclear. What are the x and y axes? Is the heatmap over home locations or reward locations?"
We agree that this panel was unclear. It was a heatmap over home locations, coloured by the mean excess steps on returns to each one. We have replaced it: Figure 2e now shows excess actions for explore and home trajectories, and Figure 2f shows excess steps on successive returns home. The proportion of returns that follow a shortest path (more than 98%) is given in the text.
"Fig. 3b and Fig. 3c independently show that there is a small portion of units that remap and have high SI. Are they the same group of units?"
They overlap. Units that remap more tend to have higher spatial information (Spearman ρ = 0.32, p = 1.2 × 10⁻⁷), and the new Figure 3c shows the measures together for every unit.
"The experimental procedure in Fig. 3a is unclear. I wasn't able to understand what the analysis is and why it is done."
Thank you for flagging this. We have rewritten Section 5.1. The analysis asks whether the representation is shared across home locations or rebuilt for each one. Because the arena is symmetric, a shared map could be reused by rotating it, so we compared the population rate map for each home location with rotated copies of a reference map. Each home location closely matched one rotation, giving four clusters, one per quadrant.
"Fig. 4a, c, e, and g are unclear. What are the red box and red dot? What's the blue box? What's the long tail showing? What does probability state mean?"
Apologies for the confusion. The colour of each cell shows the probability that the decoder assigns to that location, which is what "probability state" meant. The dot is the agent and the tail is its path so far.
The boxes were not separate markers: they were cells to which the decoder assigned high probability. We have redrawn the figure with the agent and its path in grey, and the caption now names them.
"In L454–455, the authors say 'while the others retain a largely stable spatial code' … Why are they called 'spatial'? These units have not been shown to have high Spatial information."
You are right, and we now describe these units as a "largely stable code". What we can say about space concerns the population: position can still be decoded when either the remapping or the stable units are ablated (new Figure A7).
"L462 says 'a continuous trajectory' while the experiment uses a discrete environment."
Thank you, we meant uninterrupted, not continuous in space. The agent is never reset or teleported between episodes, so each trial is a single unbroken path. We have reworded this in the revised manuscript.
If there is a further analysis you would like to see, please tell us and we will try to add it during the discussion period.
I appreciate the author's detailed responses and additional experiments. My previous criticisms are largely resolved, and the manuscript has improved substantially. I'm therefore happy to increase my rating to 3.
However, I do agree with reviewer GKmn that the study would benefit from more complex environments. For example, introducing random obstacles across environments in the current grid setup would help establish whether the learned spatial representation can transfer across different spatial structures, making the study more compelling. Nonetheless, I think the study in its current form still provides interesting insights.
Response to Reviewer ZbzK (1/2)
Thank you for the detailed and constructive review. The suggested controls have significantly strengthened the manuscript. The revised manuscript now includes an untrained network on matched trajectories (new Figures A4 to A6 and A9), the removal of every visual cue including the wall colours (new Figures A4 and A5), a control for visual similarity (Appendix A.3.6), a 5 × 5 observation window for all main results, replication across multiple seeds (new Figure A10), and comparisons with random-action and optimal agents (new Figure A8). The conclusions hold in each case. In light of these changes, we would be grateful if you would reconsider your score. This comment covers the major weaknesses, and a second covers the minor weaknesses and questions. Figure numbers refer to the revised manuscript.
Major weaknesses
You are right on both counts. With a 7 × 7 view the agent could see most of the arena, so the new reward was usually already visible when it reached home. Explore trajectories contain ~2.24 excess turns and ~1.87 excess forward steps, whereas home trajectories average only 0.10 excess actions. We have now replaced the main results with an agent that has a 5 × 5 view, and the new Figure 2e reports excess turns and excess forward steps separately.
We have added both controls. To test for visual tuning, we removed every remaining visual cue, including the colours of the arena walls, so that many locations look identical. Visually tuned units would lose their spatial information. Instead, the units kept their fields and their spatial information (new Figures A4b and A5a, e). They are also tuned to the home, which the view does not determine. To test the effect of training, we ran an untrained network on trajectories matched to those of the trained agent, and no such units appeared (new Figures A4b and A5b, d).
Position, reward and home cannot be decoded accurately from an untrained network on matched trajectories (new Figure A6c). This network receives the same observations as the trained one, so the visual input passed through an untrained copy of the architecture does not support the decoding. In addition, all analyses are run with the sensory cues removed, so only the arena boundary and the reward are visible, and the home is unmarked: it never appears in the observation, yet it is decoded even while the agent explores away from it (Figure 4g–h).
For the first control, spatial RSA is far lower in an untrained network on matched trajectories: about 0.16 in the home phase against 0.50 in the trained network, and close to zero in the explore phase (new Figure A9f). For the second, we repeated the analysis with a visual distance, the Euclidean distance between the feature vectors that the recurrent network receives. Among pairs of time steps with identical feature vectors at different locations, hidden-state distance still followed spatial distance (explore phase ρ = 0.40 ± 0.05, home phase ρ = 0.55 ± 0.02; mean ± s.e.m. across home locations). Spatial RSA was also largely preserved after partialling out visual distance (explore phase 0.34 → 0.27, home phase 0.47 → 0.44), and the partial correlation was positive for every home location in both phases. These results are in Appendix A.3.6.
We agree, and this prompted the main change to the manuscript. The main results now come from a network with a 5 × 5 view, which covers at most a quarter of the arena, and the 7 × 7 results have moved to the appendix. With either view the home is unmarked, so it is never visible and can only be reached from memory.
They replicate across 3 seeds (new Figure A10). Two further seeds are still training, and we will add them when they finish.