- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
The authors investigate whether representations supporting relational generalization can develop in a model of the visual pathways upstream of the hippocampal-entorhinal system. ReGraph combines asymmetric dorsal/ventral inputs and task-specific pretrained feature extractors with recurrent self-attention layers and dorsal-to-ventral cross-attention. It is trained on a balanced 33-class Something-Something V2 subset. The proposed Relational Primitive Analysis combines spatial HOSVD of pre-softmax attention scores, gridness tests, object-disjoint linear probing, basis ablation, and reconstruction of unseen attention scores. The principal findings are an increasing fraction of grid-like bases in the full model's dorsal hierarchy, improved late ventral probing performance with dorsal modulation, and greater late-layer sensitivity to high-gridness basis ablation. The authors interpret these findings as evidence for reusable routing templates supporting context-invariant representations.
Critical strengths
-
The paper makes an interesting empirical connection between visual-stream modeling and grid-like structure in learned attention. Its distinctive contribution is the investigation of spatial connectivity bases in a video action-recognition model, extending earlier work on dual visual streams and structural representations in hippocampal models. The dorsal grid-like fraction rises from 0.0% to 15.3% across four layers, with the qualitative trend retained at 3%, 5%, and 7% singular-direction retention (Tables 3 and 14).
-
The analysis goes beyond visual resemblance. Appendix L.1 intervenes on attention scores during the forward pass and evaluates the resulting representations with newly fitted OOD probes. At zero added noise, high-gridness ablation produces accuracies 7.0 and 6.0 percentage points below low-gridness ablation at L3 and L4, respectively. Complementary reconstruction results favor high-gridness bases in these layers across the reported 30-50% candidate pools and k=5/9 settings (Table 18). These experiments provide meaningful evidence that the identified structures are functionally relevant.
Critical weaknesses
Appendix L.2 shows that training-derived subspaces capture unseen score structure, with coefficients computed from each unseen matrix. It does not directly establish how periodic geometry supports transfer through normalized attention and transmitted values. The forward-pass ablation provides functional evidence, but reports newly trained probes' OOD accuracy without an ID comparison establishing transfer specificity. Reconstruction and ablation also use different candidate pools and selection rules, with no reported overlap of the selected bases. Thus, the combined analyses do not yet establish reuse and causal relevance of the same templates. The stronger reusable-routing interpretation of the gridness in bases needs additional support or narrower wording.
The claim that the selected biological biases are critical prerequisites for relational representations exceeds the comparisons. Table 2 already shows strong OOD decoding in the unmodulated dorsal stream: 96.9% at L4 versus 97.4% in full ReGraph, despite their different grid-like fractions (6.3% versus 15.3%, Table 3). Thus, strong cross-object action decodability must be distinguished from the particular joint gridness/probing pattern reported in ReGraph. The D-to-V removal is an informative controlled comparison, but the other trait ablations bundle changes, and single-stream references differ in capacity, token budget, and training recipe (Table 1; Appendix H.2). The strongly specialized pretrained front ends further limit attribution of the findings specifically to retinal asymmetry. Appendix M also acknowledges one training seed per configuration. These results support a hypothesis about this model's organization, but not the stronger prerequisite claim or a firm account of where generalization originates.
The architectural positioning is also incomplete. SlowFast is cited, but its M/P-inspired temporal asymmetry and fast-to-slow fusion are not discussed as precedents. The manuscript does not cite the Tolman-Eichenbaum Machine 2020, which links structure/content factorization to generalization across environments, or its transformer formulation in 2022 ICLR, which produces grid-like recurrent positional representations. These are close conceptual precedents, and better connected with the biological observations. These results do not duplicate ReGraph's gated cross-attention, but warrant a more precise account of which design principles and computational explanations are new.
The missing references: TEM 2020: Whittington, J. C. R., et al. (2020). The Tolman-Eichenbaum Machine: Unifying Space and Relational Memory through Generalization in the Hippocampal Formation. Cell, 183, 1249-1263.e23. https://doi.org/10.1016/j.cell.2020.10.024 The transformer variant: Whittington, J. C. R., Warren, J., and Behrens, T. E. J. (2022). Relating transformers to models and neural representations of the hippocampal formation. ICLR. https://openreview.net/forum?id=B8DVo9B1YE0
Questions for authors
-
Were the probe-test object identities excluded from the 33-class action-recognition training of ReGraph and the associated fine-tuning of its feature extractors, or only from probe training? Please specify how object identities and the train/test sets were constructed. This affects whether the evaluation establishes transfer to objects unseen by the representation learner or by the probe.
-
How many high-gridness reconstruction bases are also in the actual high-gridness ablation set, separately for source and target modes? I guess the expanded pool may reasonably improve gridness separation, but the concern is the correspondence between the sets supporting the two conclusions.
-
Please distinguish the contribution from TEM and its transformer formulation (see above). Do the authors claim that these subspaces support generalizable features, or that their periodic geometry implements reusable routing? What evidence distinguishes the latter interpretation? A direct analysis of attention-output changes using a common selected subspace, or an ID-versus-OOD comparison of the existing intervention, would help.
-
Scope of the prerequisite claim: How should the abstract's claim be reconciled with the strong unmodulated dorsal OOD results? Is the intended claim specifically about the observed combination of increasing gridness and strong decoding within the tested model family?
2 comments
(3) Q1: In our original experimental setup, an object-disjoint OOD split was used only for training and evaluating the linear probes, whereas the ReGraph representation learner itself was trained using the standard SSV2 split. We acknowledge that this leaves open the possibility that the representation learner may have indirectly acquired information about the probe-test objects during its training. To address this limitation and more rigorously evaluate genuine context invariance, we conducted additional cross-dataset linear probing experiments using four entirely new motion/action datasets that ReGraph had never encountered during training: KTH Actions, Moving-MNIST, TartanAir, and LIBERO (Table 1). These datasets are entirely different from SSV2 in object appearance, background characteristics, and types of motion. The “D–V (avg)” column in Table 1 represents the performance gap between the average performance of the four dorsal-stream layers and that of the ventral stream for each dataset. As shown in Table1, the dorsal stream consistently outperformed the ventral stream across all four unseen datasets (+3.2%p to +38.1%p). These findings demonstrate that the dorsal stream is particularly specialized in extracting context-invariant spatiotemporal structure, even when evaluated on entirely unseen datasets.
Table 1: Motion / Action Probes (Dorsal > Ventral).
| Dataset | Chance (%) | Stream | L1 | L2 | L3 | L4 | Avg | D–V (avg) |
|---|---|---|---|---|---|---|---|---|
| KTH actions (Schüldt et al., 2004) | 16.7 | D | 81.4 | 79.9 | 81.5 | 79.6 | 80.6 | +8.0 |
| V | 75.8 | 68.6 | 75.5 | 70.7 | 72.6 | |||
| Moving-MNIST heading (Srivastava et al., 2015) | 12.5 | D | 96.5 | 86.4 | 85.1 | 77.8 | 86.5 | +38.1 |
| V | 47.0 | 41.1 | 60.5 | 44.8 | 48.3 | |||
| TartanAir rotation sign (Wang et al., 2020) | 50.0 | D | 78.5 | 74.1 | 77.4 | 74.4 | 76.1 | +14.0 |
| V | 59.9 | 59.6 | 67.8 | 60.9 | 62.0 | |||
| LIBERO z-displacement sign (Liu et al., 2023) | 50.0 | D | 99.8 | 99.6 | 99.6 | 99.5 | 99.6 | +3.2 |
| V | 95.7 | 96.5 | 98.0 | 95.5 | 96.4 |
(4) W5: "Complementary Stream Asymmetry vs. Capacity and Pretrained Front-End Confounds?" We agree that our ablation studies involve changes in multiple factors, including model capacity and token budget, and therefore should not all be interpreted as strictly one-to-one controlled comparisons. However, these changes are inherent and necessary to the design required to test our hypothesis. Our central question is how the holistic synergy of biologically motivated properties—M/P-driven asymmetry, parallel dorsal/ventral processing, and cross-stream interactions—supports the emergence of relational primitives. Because ablating into a single-stream model naturally reduces both the parameter count and token budget, we adopted a slightly higher learning rate for these smaller models (Appendix H.2); rather than confounding the evaluation, this customized training recipe was specifically necessary to ensure their optimal convergence and provide a genuinely fair comparison. We also agree that the specialized pretrained front-ends make it difficult to attribute the observed effects exclusively to retinal asymmetry. However, we would like to clarify that our proposed M/P-driven asymmetry is an integrated biological inductive bias, incorporating both the complementary spatiotemporal properties of retinal inputs and the functional differentiation of early visual processing. Our hypothesis concerns how these complementary properties jointly shape downstream representations, rather than whether retinal sampling alone is sufficient to produce relational structure.
Nevertheless, to address the reviewer's concern more directly, we conducted additional experiments with two new trait-symmetric dual-stream variants: Dual-Dorsal and Dual-Ventral. In these models, both pathways receive identical retinal-style inputs and employ the same pretrained front-end, thereby removing the complementary specialization between streams while preserving the dual-stream framework. Both variants therefore have parameter capacities and token budgets equal to or greater than those of ReGraph.The results further support our hypothesis: neither variant exhibited the characteristic progressive increase in significant grid-like bases observed along ReGraph's extended dorsal stream (0.0% → 5.1% → 8.8% → 15.3%). In Dual-Ventral, the corresponding proportions were 0.20% → 6.70% → 6.53% → 5.60% in the first stream and 0.29% → 5.69% → 5.50% → 5.53% in the second stream. Similarly, neither stream in Dual-Dorsal exhibited a progressive increase (1st: 0.00 → 4.97 → 4.01 → 5.56 → 3.63, 2nd: 2.45 → 5.23 → 2.18 → 4.56 → 3.61). These results provide critical evidence that complementary M/P-driven dorsal–ventral asymmetry plays a key role in supporting the emergence of relational primitives and the improved performance observed in ReGraph.
Dear Reviewer dpx4,
Thank you very much for your insightful critique and helpful feedback. In the following responses, we have tried our very best to comprehensively address every point you raised. We hope our clarifications meet your expectations and look forward to any further discussion.
(1) W6: “Single-seed limitation”
We agree with the reviewer that evaluations based on a single training seed may be statistically insufficient to support our strong claim. To address these concerns, we evaluated both Top-1 performance and layer-wise grid-like code ratios across all seven model variants using multiple seeds. Table1 reports the mean results for both metrics (averaged across n = 3 seeds for ReGraph, and n=2 for the others due to computational constraints). We are actively running additional seeds and will provide the updated results as soon as they become available, in addition to incorporating the full statistics into the revised manuscript.
Across all seven variants, ReGraph achieved the highest task performance. Furthermore, an increasing pattern of grid-like code ratios across successive layers of the extended dorsal stream (reaching a maximum of 12.20% at L4, with the highest mean of 5.30) was observed only in the full model, in which all biological inductive biases implemented across the primary and extended streams were combined. These results demonstrated the robustness of our findings for the contribution of the proposed architectural design to both task performance and the hierarchical development of grid-like representations.
[Table1. Top-1 Accuracy and Layer-wise Grid-like Code Ratio across all variants]
(2) W3 & Q2: “Correspondence between reconstruction and ablation bases for reusable routing.”
We verified that 97% to 99% of the grid-like bases used in Bases-Ablation are indeed included in the Reconstruction's Ghigh set in L3–L4 (L3: Target 98.9% / Source 98.7%, L4: Target 96.9% / Source 98.1%). However, to further completely rule out any residual effects from other bases introduced by expanding the candidate pool, we re-performed the Connectivity-Reconstruction analysis using strictly the exact same grid-like and non-grid-like bases as those used in the Bases-Ablation. As a result, the reconstruction score (R2) of Ghigh (0 → 0 → 0.103 → 0.116) exhibited a distinct surge in the later layers (L3–L4) compared to Glow (0 → 0 → 0.013 → 0.073), clearly reproducing the exact same trend as our original conclusion.