acceptodds
Under review as a conference paper at ICLR 2027

Sensory Geometry Sets the Reference Frame of Multimodal Spatial Codes

Abstract

Recurrent networks trained to predict their upcoming sensory input develop spatial representations resembling those of the hippocampus, constructing a world-centred (allocentric) map from egocentric input. Such models have almost always been given a single sense, usually vision, yet hippocampal representations draw on many senses whose signals depend differently on position and heading. Here we asked how the available senses determine the reference frame of the learned representation. An agent explored a virtual arena with first-person vision and binaural audio from four speakers; recurrent networks received one or both senses plus self-motion and were trained to predict their input one second ahead. Because egocentric signals change with the direction the agent faces, we measured spatial allocentricity: the consistency of each unit's spatial tuning across headings (1, identical at every heading; 0, unrelated). All trained networks became more allocentric, but audition far more than vision (median allocentricity vs ). This difference reflected the geometry of the signals rather than the senses themselves: what the agent sees depends on heading as well as position, whereas what it hears from the surrounding speakers depends mainly on position. Rearranging the speakers so that sound also depended on heading along one axis reduced auditory allocentricity by 40%, removing interaural cues at test time increased allocentricity. Moreover, with visual input fixed, making the prediction target heading-invariant or heading-dependent moved allocentricity up or down. With both senses, vision resolved the positional ambiguity created by rearranging the speakers without raising allocentricity, and removing vision at test time could raise or lower allocentricity, depending on the geometry of the signals the network had to predict. The reference frame of a learned spatial representation is therefore set by how its afferent signals depend on heading, not by their modality. For predictive models, this suggests that latent invariances are inherited from the geometry of inputs and targets and cannot be read from decoding accuracy alone; for the brain, that place field directionality should track how the cues an animal relies on depend on heading.

Primary area:
applications to neuroscience & cognitive science
Keywords:
hippocampus, spatial learning, recurrent network, computational neuroscience

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_m1ptOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks what shapes the reference frame of spatial representations learned by predictive recurrent networks. They let an agent explore a 2×2 m virtual arena with first-person vision (120° FOV) and/or binaural audio from four looping speakers. Each sense is embedded by a frozen convolutional autoencoder, and a 400-unit RNN receiving the embedding(s) plus self-motion is trained to predict the embedding 1 s ahead. The authors define a unit-level spatial allocentricity index (the mean correlation of a unit’s rate maps across four heading bins, insensitive to heading-dependent gain) and summarise each network by the median over reliable units.

Trained auditory networks are much more allocentric than visual ones (0.68 vs 0.36), with multimodal networks in between. The authors attribute this to how signals depend on heading rather than to modality, supported by (i) relocating the speakers onto a line (with retraining), which lowers auditory allocentricity; (ii) removal of interaural level during test or of one sense, and scaling of recurrent weights; (iii) retraining with heading-averaged or heading-residual “oracle” targets while keeping the input fixed; and (iv) a negative correlation between allocentricity and the target’s position–heading interaction share.

They further show that position decodability and allocentricity can dissociate, and derive the prediction that place-field directionality in vivo should align with how the available cues depend on heading.

Critical strengths
  1. The question is relevant and timely. Predictive models of the hippocampal formation have mostly used vision; this work examines systematically how the geometry of multiple senses, including binaural audio, shapes the reference frame that a predictive network learns.
  2. The oracle-target manipulation is a good design: with the input fixed, it separates the contribution of target geometry from that of input geometry.
  3. The dissociation between position decodability and heading invariance is a useful caution for a field that often infers a cognitive map from decoding accuracy or single-unit tuning alone. Meanwhile, the paper offers a normative hypothesis that connects several empirical observations (strong directionality in vision-dominated VR, Acharya et al. 2016; largely omnidirectional fields in open arenas, Muller et al. 1994; reduced directionality with local cues, Battaglia et al. 2004), and offers a testable in vivo prediction.
Critical weaknesses
  1. The central claim is overstated. The paper claims that signal geometry, not modality, sets the reference frame. However, after the frozen encoders, modality has no dedicated structure in the RNN: the only difference between senses is the statistics of their embeddings. Within this architecture, “geometry rather than modality” therefore holds almost by construction and cannot really be tested. What the paper does establish cleanly is that the prediction target is sufficient to set the heading invariance of the units.
  2. The encoders are not validated, so the dependence of the results on their inductive biases is unknown. No reconstruction quality is reported, nor whether heading-dependent cues survive in the embeddings (e.g., whether ILDs or source azimuth can be decoded from the auditory embedding, or heading from the visual one). The choice of 200 dimensions is not justified, and there are no encoder ablations. This matters because, in the quadrant arena, much of the auditory allocentricity already arises at the encoder stage (0.111 → 0.423, Fig. 2d).
  3. The allocentricity index has conceptual issues. It scores partial heading gating high but complete gating near zero, although field location is fixed in both. Sargolini et al. (2006) conjunctive cells, cited as low-allocentricity examples, have fixed fields. Since in vivo directionality is mostly gating measured by rate differences, the index does not match the predicted biological quantity. Only unit-level measures are reported; population-level analyses (e.g., cross-heading decoder generalisation) might be needed.
  4. Test-time manipulations and statistics. Removing vision or ILDs and scaling recurrence are out-of-distribution test-time manipulations without retrained controls, and several effects are small (+0.047, +0.013, −0.028).
  5. Oracle-target controls are incomplete. The heading-averaged target uses true position, making it close to supervised regression. There is no μ (b, h) control to separate binning from heading dependence, and position decoding of oracle-trained networks is not reported.
  6. Presentation and writing. Key details (interaction share, oracle targets, removal procedures) appear only in appendices, and retrained versus test-time manipulations are not clearly distinguished. Fig. 2d,e and Fig. 3 use unexplained marker sizes; Fig. 4f does not define open versus filled markers; Fig. 4d,e reuse the modality color legend with a different meaning. The introduction contains inaccuracies (“prediction alone”, the developmental order in Abrate et al., “remains unexplored”), and there are typos (e.g., “auditon”, “consquence”, “therfore”).
Questions for authors
  1. Could you report prediction performance for each condition (including oracle targets and test-time removal of a sense or of ILDs), relative to simple baselines such as copying the current embedding or using self-motion alone? This would help determine whether differences in allocentricity partly reflect differences in learning quality.
  2. Could you add a μ (b, h) target condition to separate the effects of binning from those of heading dependence? Can position be decoded from μ (b) itself, and what is the position decoding R² of networks trained on oracle targets? Were the lookup tables computed from training or test recordings?
  3. Unimodal networks predict 200-dimensional targets and multimodal networks 400-dimensional ones, with 400 hidden units in all cases. Could this difference in capacity per target dimension affect the comparison between unimodal and multimodal networks?
  4. Have you considered an index that treats partial and complete heading gating consistently, or reporting the fraction of units in each functional class (place-like, heading-tuned, conjunctive, egocentric)?
3 comments
AuthorsOpenReview ↗

Authors reply: Reviewer m1pt (part 1/2)

Thank you for a careful review and many constructive suggestions. We address the weaknesses first, then the questions in a separate part due to characeter limits. For the critical weaknesses, in order:

  1. We respectfully disagree that the result holds by construction. The hippocampal formation receives sensory input only after extensive processing in sensory cortex, as convergent input through the entorhinal cortex. Our frozen autoencoders model this stage, so each sense arrives as a population of bounded firing rates and the recurrent network has no modality-specific circuitry. This is the relevant setting for the brain, and within it the outcome was not fixed in advance. The two senses could have produced the same reference frame, or differed for reasons unrelated to heading. The claim is also testable, as we tested by changing geometry while holding the sense fixed. We agree that the oracle-target experiment shows the prediction target alone can set heading invariance, and we will say so explicitly.
  2. We will add reconstruction examples for both autoencoders to Appendix B (reconstruction R^2: auditory 0.74, and 0.98 for time-averaged spectrum, visual 0.94), and show that the auditory embedding preserves ILDs almost exactly (ILD decoding R^2 = 0.99; heading decoding from the visual embedding R^2 0.46). We chose 200 dimensions to give two recurrent units per target dimension, and this does not limit the representation: 95% of the embedding variance lies in 48 (auditory) and 124 (visual) principal components.
  3. Our index deliberately ignores changes in firing rate with heading and asks only whether a field stays in the same place. For units that fall silent at some headings, the gating-filtered index that you suggested uses only the headings at which a unit is active, and audition remains more allocentric than vision on this index too. We also computed the mean vector length of in-field firing against heading. Visual units were more directional than auditory units (mean vector lengths 0.314 ± 0.018 vs 0.123 ± 0.009 in the quadrant arena and 0.307 ± 0.015 vs 0.120 ± 0.006 in the line arena), the same order as our index. At the population level, the paper already reports decoding (Figs 2d and 3e; Appendix E; Fig. S2), and we will add the cross-heading test you suggested. A position decoder trained at one heading and tested at the others generalised well in auditory networks ratio of R^2 0.98–0.99 in the quadrant arena, 0.90–0.93 in the line arena) but less well in visual networks (0.61–0.64 and 0.62–0.63). We agree that the conjunctive cells of Sargolini et al. (2006) are an example of pure rate modulation by heading: their fields stay in the same place and only the firing rate changes with direction. We will therefore remove them as an example of low allocentricity.
  4. Removing a sense or the ILDs does not take the networks out of distribution. Manipulated inputs and hidden states were indistinguishable from held-out intact data, as we measured the distance of manipulated points from the intact distribution (Mahalanobis distance, 0.47–1.00 times that of held-out intact points; 0–2.2% of manipulated points lay beyond the intact 99th percentile, against 1% expected). We decoded the next spectrogram, which also stayed accurate ( intact network scored R^2 = 0.64, against 0.61 without vision and 0.58 without ILDs). The paper also includes a re-trained control (line 265, Fig 4a), an auditory network that reached the same allocentricity as test-time removal (median 0.677 vs 0.680), so the effect is not an artefact of the manipulation. We designed the recurrence scaling as a lesion of the trained network, in the same way that pathways are inactivated in adult animals. Of the effects listed, +0.013 is the null result we report. The others held in all five seeds and remained significant after Holm correction (+0.047, 95% CI [0.023, 0.066]; −0.028, 95% CI [−0.047, −0.016]), and their opposite signs cannot be produced by an artefact of the manipulation.
  5. The heading-averaged target uses true position because it is a test of principle: it shows that, with the input unchanged, the prediction target alone can set heading invariance. We do not propose it as a model of how the hippocampus learns. We address the binning control and position decoding in our answer to question 2 below.
  6. We will ensure to use the additional page to move the definitions of the interaction share, the oracle targets, and the removal procedures into the main text, and correct minor presentation and layout issues. Thank you for pointing these out.
AuthorsOpenReview ↗

Authors reply: Reviewer m1pt (part 2/2)

To answer the questions:

  1. Indeed. The trained networks predict better than simply copying the current embedding, and beat this baseline even after a single epoch, in every condition: | Network | Condition | Loss below copy | Below self-motion alone | |---|---|---|---| | Modality | intact | 66% | 58% | | | ILDs removed | 57% | 48% | | Visual | intact | 33% | 17% | | Audiovisual | intact | 49% | 37% | | | vision removed | 43% | 30% | | | audition removed | 24% | 7% | | | ILDs removed | 44% | 32% | | Visual, oracle targets | heading-averaged | 76% | 32% | | | heading-residual | 59% | 19% |

  2. Most of the effect of the heading-averaged target comes from removing heading dependence rather than from binning. As requested, we re-trained visual networks to predict the smoothed μ(b,h) and μ(b) targets:

TargetAllocentricity, quadrantAllocentricity, linePosition R², quadrantPosition R², line
Original0.363 ± 0.0330.336 ± 0.0400.935 ± 0.0040.933 ± 0.005
μ(b,h)0.461 ± 0.0190.482 ± 0.0210.970 ± 0.0040.971 ± 0.003
μ(b)0.789 ± 0.0090.822 ± 0.0120.977 ± 0.0020.985 ± 0.000

Predicting μ(b,h) raised allocentricity only slightly, likely due to smoothing, whereas predicting μ(b) raised it far more. Position decoding was high for all oracle-trained networks. The tabled numbers were computed from training recordings.

  1. Target dimension does not limit capacity (please see weakness 2 in part 1). We also varied the number of recurrent units (200 for the unimodal networks, 800 for the separate multimodal network; three seeds each) to compare all networks at the same number of units per target dimension. The order is the same at both ratios, and doubling the multimodal network leaves its allocentricity unchanged:
AuditoryVisualMultimodal (separate)Units per target dimension
0.677 ± 0.010 (400 units, original)0.363 ± 0.033 (400 units, original)0.628 ± 0.025 (800 units, new)2
0.717 ± 0.031 (200 units, new)0.218 ± 0.032 (200 units, new)0.633 ± 0.012 (400 units, original)1
  1. Yes. On a gating-filtered index, which uses only the headings at which a unit is active and its map is reliable, audition remains more allocentric than vision (0.73 ± 0.01 vs 0.38 ± 0.03 in the quadrant arena; 0.53 ± 0.03 vs 0.36 ± 0.04 in the line arena). Gain modulation with the field preserved is also a minority (15.7% in the visual networks). We addressed this further in the overall response – please kindly navigate to find more information there. We also classified units by function, as suggested, using spatial information (SI, Skaggs et al., 1994), head-direction tuning (Taube et al., 1990), and to where the walls or arena centre lie relative to the head (egocentric tuning; Alexander et al., 2020). A unit was egocentric if its egocentric tuning was significant against time-shifted null and an egocentric model predicted its activity better, under cross-validation, than allocentric models of the same size. The remaining units were place-like (position only), heading-tuned (head direction only) or conjunctive (both). The proportions (auditory / visual) were: place-like 19% / 11%, heading-tuned 8% / 0.5%, conjunctive 69% / 59%, egocentric 2% / 29%. Auditory networks have more place-like units and almost no egocentric units, whereas visual networks have many egocentric units.

We hope these additions addressed your concerns; if so, we would be grateful if you would consider raising your score.

Thank you for the detailed response.

I look forward to seeing these analyses incorporated into the revised version.

I have increased my score from weak reject to weak accept.

Reviewer_iqNaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes to investigate the question of why and how much neural units develop allocentric versus directional firing maps while receiving multimodal (audio and vision) inputs.

The paper is very clear, well structured, logical and easy to follow. The figures are also very clean and informative.

A series of experiments support the main argument: that sensory geometry, not sensory type, determines whether units are allocentric or directional. The discussion on how these results can inform computational neuroscience is interesting.

Critical strengths

The paper is well structured and very clear. The figures are also very clean and informative.

It focuses on one concrete question and designs relevant experiments to investigate this question.

Each RNN is trained with 5 seeds, and results are reported with std.

The introduction and discussion motivate the main question within neuroscience.

Critical weaknesses

I have four critics:

  1. whether the argument is in fact obvious: if a task doesn’t depend on heading, isn’t it expected that the latent representation also won’t depend on it?

  2. the experiments only test a 400 units sigmoid RNN: what about other architectures, in particular those with nonlinearities such as tanh/relu that are more common in computational neuroscience: would the results survive? Why training only with L1 loss, would the results survive with L2 loss?

  3. why making the distinction between vision and audio? wouldn't it be easier/cleaner to stick with vision and include vs remove visual landmarks in the arena, or stick with audio and place the speakers in different places, rather than mixing both?

  4. the paper would be stronger if the predicted, quantitative allocentricity would match actual allocentricity observed in real biological network. Currently, the introduction and discussion only mention qualitatively how certain units are (or not) allocentric. Can the trained RNNs reproduce the allocentricity observed in real data?

Questions for authors

My main question is: Could the trained RNNs reproduce, quantitatively, the allocentricity observed in real data?

3 comments
AuthorsOpenReview ↗

Authors reply: Reviewer iqNa

Thank you for a thorough review. We answer each of the 4 critis below: (1) whether the result is obvious, (2) the choice of activation function, loss and network size, (3) why senses are combined, and (4) reproducibility of the allocentricity of biological cells.

(1) We do not think the result is obvious, because how much a signal depends on heading does not predict allocentricity. Moving the speakers onto a line barely changed the share of auditory variance that depends on heading (0.53 to 0.56), yet allocentricity fell from 0.68 to 0.41. What matters is whether position and heading combine separably, which we measure as the interaction share of the target (Section 3.5). Moreover, neither sense is independent of heading. Both are sampled at the head, and heading can still be decoded from the auditory network in the quadrant arena (R^2 = 0.87). Finally, Appendix F shows that position can be read out exactly from units that all change with heading.

(2) The results do not depend on the activation function, loss, or network size: with tanh or ReLU units, an L2 loss, or 200- or 800-dimensional RNNs (as Reviewer m1pt raised), the order of auditory and visual networks was unchanged for both allocentricity and position decoding. We will place a table in the revised submission Appendix C. We chose sigmoid units because they most closely resemble biological neurons. Like firing rates, their activity is non-negative and bounded, unlike tanh units (which can be negative) or ReLU units (which are unbounded), and both senses are therefore represented on the same scale. L1 and L2 losses gave the same ordering when training the RNN. When training the sensory autoencoders, L2 gives imbalanced loss magnitude in vision and auditory due to the spectrogram dominated by a small number of high-energy bins.

(3) We agree that changing the geometry of one sense can change allocentricity, and the paper does this within each sense, by moving the speakers and by changing the prediction target. Combining senses answers a different question, namely whether a sense that adds missing positional information also changes the reference frame. Our results show that vision supplied position without raising allocentricity (Section 3.4), which is relevant because animals navigate with several senses at once (Jeffery, 2007; Nguyen et al., 2024) and it offers an explanation to why place fields are strongly directional in vision-dominated virtual reality but largely omnidirectional in open arenas (Discussion, lines 461-470).

(4) Yes. Applying the same index to recorded cells from CA1, CA2, and medial entorhinal cortex (median allocentricity of each dataset 0.655–0.790, 10-90th percentile 0.410-0.928), the auditory and audiovisual networks fell within the recorded range, whereas the visual network was lower. We provide further details in the overall response (“Biological data”).

We hope these additions address your concerns; if so, we would be grateful if you would consider raising your score.

Reviewer_iqNaOpenReview ↗

Thank you for your rebuttal and additional experiments

I thank the authors for addressing my comments.

I believe that the additional real experiments, in particular, will make the paper much stronger.

However, I believe that such a large amount of new experiments (new ablations with activation functions + new, biological experiments) requires another round of peer-review.

Hence, I maintain my score.

AuthorsOpenReview ↗

Thank you for your response

Thank you for reading our rebuttal.

We want to gently note that this is what the rebuttal period is for: reviewers request additional experiments to settle open questions, and assess the additional results. We reported the results in full and have amended our manuscript for robustness and clartiy. Crucially, none of the experiments is a new line of work, as they only validate our existing claims and conclusions, which remain unchanged in the paper.

The public recordings we analysed are available at https://crcns.org/data-sets/hc. If any specific aspect needs further clarification, we are committed to responding before the rebuttal deadline.

Reviewer_91V6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how spatial representations in RNNs are shaped by the geometry of sensory inputs and prediction targets, particularly their dependence on heading, rather than sensory modality alone. The authors train networks in two simulated environments across visual, auditory, and multimodal settings, and test this hypothesis through speaker rearrangement, test-time removal of ILDs, and retraining with transformed targets. The experiments provide complementary support for this geometry-based account. They also show that position decodability can dissociate from unit-level spatial-field invariance, and that changing prediction targets can reverse the effect of visual ablation.

Critical strengths
  1. The intervention design directly addresses the geometry-based account. Rearranging speakers changes auditory cue geometry while preserving modality, visual input, and trajectories. Removing ILDs tests reliance on interaural differences without retraining, while target manipulation examines the effect of changing what the network predicts with input held fixed. These complementary interventions provide persuasive evidence that modality identity alone does not determine spatial-field invariance.

  2. The dissociation between positional information and spatial-field invariance is consequential. Removing interaural cues can increase allocentricity while reducing position decoding, and vision can supply missing positional information without increasing allocentricity. The reversal of the visual ablation effect under different targets further demonstrates that these quantities capture distinct properties of a representation. Appendix F supports the possibility of exact nonlinear position recovery alongside low invariance in the position-dependent units, although its description of the reference frame needs correction.

Critical weaknesses
  1. The mechanistic and reference-frame interpretations extend beyond the evidence. Position–heading interactions can be expressed through gain modulation, h(p,θ)=c+ϵf(p)g(θ)h(p,\theta)=c+\epsilon f(p)g(\theta) with g(θ)>0g(\theta)>0, without reducing the allocentricity index. Thus, the linear-readout argument does not establish why residual targets should lower spatial invariance. Appendix F also includes world-referenced heading through cos⁡θ\cos\theta and sin⁡θ\sin\theta, supporting decodability without invariant position fields rather than a population devoid of world-referenced information. These distinctions limit the interpretation, while leaving the empirical dissociations intact.

  2. The final evaluation pipeline remains insufficiently documented. It is unclear whether intact and ablated conditions use the same eligible units, and the missing analysis modules prevent verification. Recording assignments also differ: the scripts suggest checkpoint selection on Test1 and subsequent analysis on Test2, whereas the notebooks use both for checkpoint evaluation. These gaps leave uncertainty about unit-selection effects and evaluation independence, although they do not establish bias.

Questions for authors
  1. How do you distinguish between actual shifts in spatial field structure versus mere gain modulation? Since gain-modulated units can produce position-heading interactions without dropping the allocentricity index, is there direct evidence that the networks undergo structural field changes, or should the drop in allocentricity simply be treated as an empirical observation rather than a direct mathematical consequence of the linear readout?

  2. Does your central claim about reference frames refer to individual units or the population as a whole? In Appendix F, world-referenced heading signals like cosine and sine are used to recover position. Does this mean your conclusion should be limited strictly to position decodability without requiring invariant position fields, and how does this distinction shape the broader claims in the main text?

  3. Were your inclusion criteria and reliability thresholds applied independently to both intact and ablated conditions, or did you track the exact same set of units across both? If the unit sets differed, do the main interaural level difference and modality ablation effects still hold when you restrict the median calculations to units that satisfy the criteria in both conditions? Could you also clarify the unit retention counts and point us to the missing allocentricity and extraction modules?

  4. Which exact code entry points and recordings were used to generate the final results? The scripts imply that checkpoint selection happened on Test1 and recurrent lesion analysis on Test2, but the notebooks appear to mix both recordings for evaluation. Were the final representation and decoding analyses performed on completely held-out recordings that were never touched during checkpoint selection? If any data was reused, which specific findings are affected, and do the main conclusions still hold on a strictly independent recording?

2 comments
AuthorsOpenReview ↗

Authors reply: Reviewer 91V6

Thank you for a careful and constructive review. We are pleased you enjoyed the paper, and we answer the points on gain modulation (weakness 1; question 1), unit-level versus population measures (question 2), reliability criteria (question 3), and the code and test recordings (weakness 2; question 4) below.

  1. We designed the allocentricity index deliberately so that it does not measure gain modulation. It asks only whether a field stays in the same place across headings, so a unit whose firing rate changes with heading but whose field does not move counts as allocentric. We agree that such units can still carry position–heading interactions, and that the linear readout alone does not explain why heading-residual targets reduce allocentricity. We will therefore present the fall in allocentricity with heading-residual targets as an empirical result, and revise the sentence at line 361. We have also measured gain modulation directly. In visual networks, only 15.7 ± 1.2% of units were gain-modulated with the original target and 9.8 ± 0.6% with the residual target, and in most units heading changed the shape of the rate map rather than its gain (overall response, “Gain modulation”, which also answers Reviewer m1pt). You are also right about Appendix F. The cosine and sine of heading are world-referenced, so the example shows only that position can be read out from units whose fields change with heading. We will correct the text.
  2. Our claim concerns the population. We compute the index for each unit, show the full distribution (Figs 2c and 3g), and use the median nuymber as a summary. All decoding used population vectors.
  3. We applied the split-half reliability criterion separately in each condition. We have now repeated the analysis using only units that pass in both the intact and the lesioned condition. Every effect keeps its direction and significance, and allocentricity changes by at most 0.012 (ILDs removed), 0.008 (audition removed), and 0.019 (vision removed). On average 93% of units pass the criterion in both conditions (85-97% across networks), and 96-100% of the units that pass in the intact network also pass after the manipulation. The allocentricity and extraction modules will be updated in the anonymised code submissions.
  4. Thank you for checking the code. The notebooks are interactive copies of the training scripts, used only for prototyping and debugging. All reported networks were trained with the command-line scripts, and Test1 was used only to monitor the loss and select checkpoints. All results were then computed with an additional evaluate_spatial_code.py, which runs the selected checkpoint modelson Test2. The train recordings (3 x 2 hours, labeled Train1, Train2 and Train3) were never used for checkpoint selection. We will update the anonymised code submissions zip with this zip. We hope this answers your questions, and we are happy to clarify anything further.

Thank you for your detailed response. It has addressed most of my concerns, and I will keep my positive rating unchanged.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overall comment: gain modulation and comparison with recorded cells

We thank all reviewers for their careful reading, and welcome their constructive suggestions and critical questions. Here we answer two points raised by several reviewers.

  1. Gain modulation. Reviewers 91v6 and m1pt noted that the allocentricity index does not take into account gain modulation. This is deliberate as shown in Appendix D in the original manuscript. We designed the index so that it does not measure gain modulation, and it asks only whether a field stays in the same place across headings; a unit whose firing rate changes with heading but whose field does not move therefore counts as allocentric. We agree that gain-modulated units can still carry position–heading interactions, and will revise the sentence at line 361. To address reviewer 91v6's concerns, we then measured gain modulation directly in all visual networks (2 arenas × 3 targets × 5 seeds × 400 units): a. Only a small fraction of units were gain-modulated (95th percentile of the gain ratio as in Leutgeb et al., 2005): 15.7 ± 1.2% with the original target and 9.8 ± 0.6% with the residual target. By contrast, 62.6 ± 2.3% and 73.4 ± 1.0% of units changed their rate map with heading (allocentricity < 0.5). b. The positive gain model, as proposed by Reviewer 91v6, left 53 ± 1% of the heading-dependent variance unexplained with both targets, so more than half reflects a change in the shape of the map. In addition, 49–58% of units had at least one pair of heading maps that were negatively correlated, which positive gain cannot produce. c. We also computed a gating-filtered allocentricity index, suggested by reviewer m1pt, which only includes a heading-positional map in the pairwise correlation if the map is reliable (split-half correlation > 0.5) and active (mean activity > 10% of the most active heading). In this version, the ordering remains identical. audition stayed above vision: 0.73 ± 0.01 vs 0.38 ± 0.03 in the quadrant arena and 0.53 ± 0.03 vs 0.36 ± 0.04 in the line arena (five seeds, p < 0.001).
  2. Biological data. Reviewers asked whether the networks reproduce the allocentricity of recorded cells. We applied the same index to public datasets (hc-16, mouse CA1: Sparks et al., 2017; Talbot et al., 2018; hc-20, rat medial entorhinal cortex: Park & Fenton, 2018; Park et al., 2018) and to preliminary Neuropixels recordings from our lab. The auditory and audiovisual networks fall within the range of allocentricity of recorded cells (table below; cells with split-half reliability > 0.5).
DatasetCellsMedian10th-90th percentileShare with allocentricity > 0.5
Mouse CA1 (CRCNS hc-16, wild type)640.7490.579-0.87795%
Rat MEC, spatial cells (hc-20)210.7900.449-0.92881%
Rat MEC, conjunctive cells (hc-20)650.6660.410-0.86777%
Our preliminary hippocampal recordings*610.6550.467-0.89285%

The network models the hippocampal formation as a whole rather than a single subregion, so we compare it with cells from CA1, CA2, and medial entorhinal cortex. Under the same criterion, the auditory and audiovisual networks had medians of 0.74, 0.67 (separate encoders), and 0.65 (joint encoder), within the range of the recordings, whereas the visual network had a median of 0.36. Rate maps were binned and smoothed in the same way for recorded cells and network units. *Histology for the preliminary recordings is not yet complete; the probe was most likely in CA2. We expect to publish this dataset once more analysis is complete.

We will address the remaining points in the individual responses below and will add the new analyses to the revised manuscript. We would like to sincerely thank the reviewers once again for their constructive comments, which have substantially improved our work.

AuthorsOpenReview ↗

Overall comment: Thank you to all reviewers

We would like to once again thank all reviewers for their comments, which have meaningfully strengthened our findings.

We have uploaded a revised manuscript with all changes marked in blue.

We would like to point out that every analysis requested in the reviews confirmed the original findings. Main figures and conclusions of the submitted paper have not been altered. Taking advantage of the allowed additional page, our revision adds supporting controls and clarifications, including definitions moved into the Methods (Section 2.5), a comparison with publicly available recorded cells (Section 3.1), encoder reconstructions (Appendix B), a gain-modulation analysis (Appendix D), and a binning control for the oracle targets (Appendix G).

We remain available and committed to further discussion during the rebuttal period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.