- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks what shapes the reference frame of spatial representations learned by predictive recurrent networks. They let an agent explore a 2×2 m virtual arena with first-person vision (120° FOV) and/or binaural audio from four looping speakers. Each sense is embedded by a frozen convolutional autoencoder, and a 400-unit RNN receiving the embedding(s) plus self-motion is trained to predict the embedding 1 s ahead. The authors define a unit-level spatial allocentricity index (the mean correlation of a unit’s rate maps across four heading bins, insensitive to heading-dependent gain) and summarise each network by the median over reliable units.
Trained auditory networks are much more allocentric than visual ones (0.68 vs 0.36), with multimodal networks in between. The authors attribute this to how signals depend on heading rather than to modality, supported by (i) relocating the speakers onto a line (with retraining), which lowers auditory allocentricity; (ii) removal of interaural level during test or of one sense, and scaling of recurrent weights; (iii) retraining with heading-averaged or heading-residual “oracle” targets while keeping the input fixed; and (iv) a negative correlation between allocentricity and the target’s position–heading interaction share.
They further show that position decodability and allocentricity can dissociate, and derive the prediction that place-field directionality in vivo should align with how the available cues depend on heading.
Critical strengths
- The question is relevant and timely. Predictive models of the hippocampal formation have mostly used vision; this work examines systematically how the geometry of multiple senses, including binaural audio, shapes the reference frame that a predictive network learns.
- The oracle-target manipulation is a good design: with the input fixed, it separates the contribution of target geometry from that of input geometry.
- The dissociation between position decodability and heading invariance is a useful caution for a field that often infers a cognitive map from decoding accuracy or single-unit tuning alone. Meanwhile, the paper offers a normative hypothesis that connects several empirical observations (strong directionality in vision-dominated VR, Acharya et al. 2016; largely omnidirectional fields in open arenas, Muller et al. 1994; reduced directionality with local cues, Battaglia et al. 2004), and offers a testable in vivo prediction.
Critical weaknesses
- The central claim is overstated. The paper claims that signal geometry, not modality, sets the reference frame. However, after the frozen encoders, modality has no dedicated structure in the RNN: the only difference between senses is the statistics of their embeddings. Within this architecture, “geometry rather than modality” therefore holds almost by construction and cannot really be tested. What the paper does establish cleanly is that the prediction target is sufficient to set the heading invariance of the units.
- The encoders are not validated, so the dependence of the results on their inductive biases is unknown. No reconstruction quality is reported, nor whether heading-dependent cues survive in the embeddings (e.g., whether ILDs or source azimuth can be decoded from the auditory embedding, or heading from the visual one). The choice of 200 dimensions is not justified, and there are no encoder ablations. This matters because, in the quadrant arena, much of the auditory allocentricity already arises at the encoder stage (0.111 → 0.423, Fig. 2d).
- The allocentricity index has conceptual issues. It scores partial heading gating high but complete gating near zero, although field location is fixed in both. Sargolini et al. (2006) conjunctive cells, cited as low-allocentricity examples, have fixed fields. Since in vivo directionality is mostly gating measured by rate differences, the index does not match the predicted biological quantity. Only unit-level measures are reported; population-level analyses (e.g., cross-heading decoder generalisation) might be needed.
- Test-time manipulations and statistics. Removing vision or ILDs and scaling recurrence are out-of-distribution test-time manipulations without retrained controls, and several effects are small (+0.047, +0.013, −0.028).
- Oracle-target controls are incomplete. The heading-averaged target uses true position, making it close to supervised regression. There is no μ (b, h) control to separate binning from heading dependence, and position decoding of oracle-trained networks is not reported.
- Presentation and writing. Key details (interaction share, oracle targets, removal procedures) appear only in appendices, and retrained versus test-time manipulations are not clearly distinguished. Fig. 2d,e and Fig. 3 use unexplained marker sizes; Fig. 4f does not define open versus filled markers; Fig. 4d,e reuse the modality color legend with a different meaning. The introduction contains inaccuracies (“prediction alone”, the developmental order in Abrate et al., “remains unexplored”), and there are typos (e.g., “auditon”, “consquence”, “therfore”).
Questions for authors
- Could you report prediction performance for each condition (including oracle targets and test-time removal of a sense or of ILDs), relative to simple baselines such as copying the current embedding or using self-motion alone? This would help determine whether differences in allocentricity partly reflect differences in learning quality.
- Could you add a μ (b, h) target condition to separate the effects of binning from those of heading dependence? Can position be decoded from μ (b) itself, and what is the position decoding R² of networks trained on oracle targets? Were the lookup tables computed from training or test recordings?
- Unimodal networks predict 200-dimensional targets and multimodal networks 400-dimensional ones, with 400 hidden units in all cases. Could this difference in capacity per target dimension affect the comparison between unimodal and multimodal networks?
- Have you considered an index that treats partial and complete heading gating consistently, or reporting the fraction of units in each functional class (place-like, heading-tuned, conjunctive, egocentric)?
3 comments
Authors reply: Reviewer m1pt (part 2/2)
To answer the questions:
-
Indeed. The trained networks predict better than simply copying the current embedding, and beat this baseline even after a single epoch, in every condition: | Network | Condition | Loss below copy | Below self-motion alone | |---|---|---|---| | Modality | intact | 66% | 58% | | | ILDs removed | 57% | 48% | | Visual | intact | 33% | 17% | | Audiovisual | intact | 49% | 37% | | | vision removed | 43% | 30% | | | audition removed | 24% | 7% | | | ILDs removed | 44% | 32% | | Visual, oracle targets | heading-averaged | 76% | 32% | | | heading-residual | 59% | 19% |
-
Most of the effect of the heading-averaged target comes from removing heading dependence rather than from binning. As requested, we re-trained visual networks to predict the smoothed μ(b,h) and μ(b) targets:
| Target | Allocentricity, quadrant | Allocentricity, line | Position R², quadrant | Position R², line |
|---|---|---|---|---|
| Original | 0.363 ± 0.033 | 0.336 ± 0.040 | 0.935 ± 0.004 | 0.933 ± 0.005 |
| μ(b,h) | 0.461 ± 0.019 | 0.482 ± 0.021 | 0.970 ± 0.004 | 0.971 ± 0.003 |
| μ(b) | 0.789 ± 0.009 | 0.822 ± 0.012 | 0.977 ± 0.002 | 0.985 ± 0.000 |
Predicting μ(b,h) raised allocentricity only slightly, likely due to smoothing, whereas predicting μ(b) raised it far more. Position decoding was high for all oracle-trained networks. The tabled numbers were computed from training recordings.
- Target dimension does not limit capacity (please see weakness 2 in part 1). We also varied the number of recurrent units (200 for the unimodal networks, 800 for the separate multimodal network; three seeds each) to compare all networks at the same number of units per target dimension. The order is the same at both ratios, and doubling the multimodal network leaves its allocentricity unchanged:
| Auditory | Visual | Multimodal (separate) | Units per target dimension |
|---|---|---|---|
| 0.677 ± 0.010 (400 units, original) | 0.363 ± 0.033 (400 units, original) | 0.628 ± 0.025 (800 units, new) | 2 |
| 0.717 ± 0.031 (200 units, new) | 0.218 ± 0.032 (200 units, new) | 0.633 ± 0.012 (400 units, original) | 1 |
- Yes. On a gating-filtered index, which uses only the headings at which a unit is active and its map is reliable, audition remains more allocentric than vision (0.73 ± 0.01 vs 0.38 ± 0.03 in the quadrant arena; 0.53 ± 0.03 vs 0.36 ± 0.04 in the line arena). Gain modulation with the field preserved is also a minority (15.7% in the visual networks). We addressed this further in the overall response – please kindly navigate to find more information there. We also classified units by function, as suggested, using spatial information (SI, Skaggs et al., 1994), head-direction tuning (Taube et al., 1990), and to where the walls or arena centre lie relative to the head (egocentric tuning; Alexander et al., 2020). A unit was egocentric if its egocentric tuning was significant against time-shifted null and an egocentric model predicted its activity better, under cross-validation, than allocentric models of the same size. The remaining units were place-like (position only), heading-tuned (head direction only) or conjunctive (both). The proportions (auditory / visual) were: place-like 19% / 11%, heading-tuned 8% / 0.5%, conjunctive 69% / 59%, egocentric 2% / 29%. Auditory networks have more place-like units and almost no egocentric units, whereas visual networks have many egocentric units.
We hope these additions addressed your concerns; if so, we would be grateful if you would consider raising your score.
Thank you for the detailed response.
I look forward to seeing these analyses incorporated into the revised version.
I have increased my score from weak reject to weak accept.
Authors reply: Reviewer m1pt (part 1/2)
Thank you for a careful review and many constructive suggestions. We address the weaknesses first, then the questions in a separate part due to characeter limits. For the critical weaknesses, in order: