- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper leverages fixed behavior encoding for neural decoder training, and proposed BeRG to match the distributional space between neural and behavior states of similar tasks. BeRG can leverage behavior information explicitly, with trained encoder from scratch and with pretrained model, and is compatible with various neural decoder backbone showing improved performance.
Critical strengths
- The use of frozen behavior encoder and distribution matching is novel
- Method demonstrate improved performance over many decoder backbones
- Conducted future probing and representation analysis
Critical weaknesses
- The writing of the paper makes it difficult to gauge the performance of the proposed methods, especially in the results section. Experiment details are not clear.
- In some of the experiments, the improvement of using BeRG over base is relatively small or slightly worse, more comprehensive investigations are likely needed.
- Code is not provided
Questions for authors
- Since both decoding and relational guidance are in the loss function (both leverage behavior targets), a key question is whether the relational guidance mainly acts as a regularizer. Can the proposed method operates without decoding loss?
- Can the author elaborate how the source-training bank was selected? What is the definition of the same task for random-target sessions?
- How bank size affects the performance of the proposed method?
- Please clarify split details on both datasets. For example for Perich datasets, what are development endpoints and how do the 19 sessions represent it.
- Can the author provide citation for all the baseline, for example I am not clear if POYO-light is just a variant of POYO methods proposed in Azabou 2023
- There seems to be only POYO as a recent neural decoder baseline, can BeRG be applied to other popular decoder methods like NDT? In addition, the choice of biGRU is non-common among neural decoder, have the authors tried normal GRU as well?
- Please explain the training details for POYO-light. What do the authors mean by “these runs do not use an official pretrained checkpoint” (line 259) and “BeRG arm resumes from weights at epochs 35 and 50 without restoring optimizer or RNG states” (appendix D3)
- What are the potential reasons for a worse performance in MC maze? I understand this is from different subject and conducting maze tasks. I am curious if this is a dataset / lab difference or task difference. More finetuning datasets could likely provide some insights
Format related:
- Submission pdf is likely not generated properly, all cross-references to figures/tables/citations are disabled.
- Figure 3 is being cut-off
- Citation on Perich dataset is likely wrong in date, I believe it should be 2025
8 comments
Response to Reviewer LpMq (2/6): Source-training bank, task definition, and bank size
Q2–Q3, concern summary. How is the source-training bank constructed, what counts as the same task for random-target sessions, and how does bank size affect performance?
Cache, bank, and per-anchor candidates.
The full source-training cache, dynamic bank, and per-anchor candidate set serve different roles. The cache contains valid paired neural and behavioral states from source-training intervals. Each active task uses a session-balanced, cyclically refreshed bank of 256 states. For each anchor, its own session is excluded before selecting 64 distinct candidates from near-to-distant behavioral-similarity ranks, with at most four states per session. In Perich, “same task” denotes the CO or RT paradigm; target changes within RT do not create new task labels. Behavioral history and available target and phase context distinguish movement states within each paradigm. We have clarified the task definition in Section 3.3 and reorganized Appendix G.3 into a six-step procedure from cache construction and anchor selection to relational-loss computation.
Bank size and candidate-quality evidence.
Bank size determines the comparison pool, whereas the 64 candidates support an individual anchor’s relational distribution. All 256 bank states undergo neural forward passes and contribute to session centering, so changing bank size also affects computation and the centering estimates. Table R2 summarizes the earlier candidate-quality diagnostic.
Table R2. Candidate quality in an early source-training diagnostic.
| Candidate construction | Mean best-candidate teacher cosine similarity | Median best-candidate rank in the full eligible pool |
|---|---|---|
| Session-fixed sets of 64 candidates | 0.8991 | 4,073.5 |
| Refreshed 256-state bank with 64 candidates selected per anchor | 0.9484 | 1,101 |
Source-training diagnostic using 3,168 anchors from 99 sessions; 32 banks were evaluated for the dynamic construction. Higher cosine similarity and lower rank indicate closer behavioral candidates.
The dynamic construction provides closer behavioral neighbors in this diagnostic. The comparison jointly changes bank refresh and candidate selection; the independent effect of bank size on decoding performance under the final recipe has not been systematically evaluated. We report these diagnostics and their scope in Appendix G.3.
Response to Reviewer LpMq (3/6): Data splits and development endpoints
Q4, concern summary. How are the two datasets split? In particular, what are the Perich development endpoints and the role of the 19 sessions?
Perich teacher fitting, selection, and development.
The 19 Perich sessions form a teacher-selection set stratified by animal and task, covering the five animal–task combinations present in the source data. Learned behavioral teachers are fitted on training intervals from the other 80 sessions, with checkpoints selected using validation intervals from these 19 sessions. Neural pretraining and the relation bank use training intervals from all 99 source sessions. The 19 sessions are held out from teacher fitting while their training intervals still participate in neural pretraining. “Development endpoints” refers to source-validation evaluations used during development, because the 99-session aggregate includes validation data used for teacher selection. Evaluations on held-out TEST intervals within source sessions are reported separately in Appendix C.1.
Table R3. Perich session split for behavioral-teacher fitting and selection.
| Animal | Task | Source sessions | Teacher fitting | Teacher selection |
|---|---|---|---|---|
| C | CO | 53 | 43 | 10 |
| C | RT | 15 | 12 | 3 |
| J | CO | 3 | 2 | 1 |
| M | CO | 22 | 18 | 4 |
| M | RT | 6 | 5 | 1 |
| Total | 99 | 80 | 19 |
LIGHT and MOMENT use the same session split. Teacher fitting uses TRAIN intervals from 80 sessions; selection uses VALID intervals from the other 19. Neural pretraining and bank construction use TRAIN intervals from all 99 sessions.
LINK source, development, and final dates.
LINK is divided chronologically into 94 source, 17 development, and 45 final dates. Within source dates, retained trials 1–270 are used for training and 271–300 for source validation. The separate 17 development dates determine the training and calibration protocol. Final dates are excluded from source-model fitting and protocol selection, but recent50 evaluation uses labeled trials 251–300 from each final date for calibration and scores trials 301–375. Source validation, development dates, and final-date calibration and scoring therefore serve distinct roles. We have clarified these roles in the experimental setup in Section 4 and Appendix A.1, including the stratified session counts; target calibration and source-session TEST protocols are detailed in Appendices B.1 and C.1, respectively.
Response to Reviewer LpMq (1/6): Relational supervision and training without the decoding loss
Thank you for the careful review. We address the methodological, experimental, and presentation concerns below.
Q1, concern summary. Does relational guidance mainly act as a regularizer when both objectives use behavior, and can the method learn representations without the decoding loss?
What the two objectives constrain.
Relational guidance supplies behavior-dependent structural supervision and can also regularize decoder training. The decoding loss constrains current behavioral predictions at each state, whereas the relational objective constrains how neural states from different sessions preserve neighborhoods defined by the behavioral reference. The two objectives therefore constrain prediction accuracy and cross-session behavioral organization along the same decoding path.
Training without decoding supervision.
To test representation learning without the decoding loss, we removed this loss during neural representation training, froze the resulting models, and fitted linear ridge readouts from their final hidden states using a common protocol. Table R1 reports the resulting readouts alongside random initialization, decoding-only Base, and joint LIGHT training.
Table R1. Frozen behavioral readouts with and without decoding supervision.
Window Transformer final-hidden ridge readouts on Perich source validation. Scores are session-macro R2, reported as mean ± sample SD across neural-training seeds 11, 22, and 33, conditional on the original frozen LIGHT teacher. Each query session is excluded from readout fitting; its training data remain included in neural pretraining. These scores differ from trained-decoder pooled R2.
The relational objective can learn behaviorally informative representations without direct decoding supervision during representation training, while the joint objective gives stronger current-behavior readout under the tested recipe. We summarize this finding in Section 4.1 and report the ablation protocol and unified readout results in Appendix D.1.