- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the performance advantage of the pretrained BCIM in neural decoding tasks. Comparing readouts with different temporal window sizes and local smoothing shows that simply widening the readout window for raw neural signals yields values close to those of the frozen BCIM model on MC-RTT and MC-Maze datasets. This indicates that the decoding gains of frozen features primarily stem from temporal integration rather than encoding abstract behavioral structures.
Critical strengths
-
- The research question is clearly defined as the relationship between frozen-feature’s decoding gains and temporal context.
-
- The manuscript provides detailed reporting of experimental setups, including dataset descriptions, network architecture, readout protocols, and conducts comprehensive ablation studies.
Critical weaknesses
-
- The claim that frozen-feature gains stem from temporal integration is drawn from MC-RTT and MC-Maze datasets with similar data formats, prediction targets, and evaluation metrics. The key advantage of a pretrained BCIM is expected to lie in its generalization across diverse data, tasks, and subjects. However, the decoding tasks employed here may be solvable with simple temporal smoothing, thereby limiting the potential benefits in other scenarios.
-
- Motor decoding from macaque neural signals is inherently tied to temporal dynamics; hand velocity does not change abruptly but is strongly autocorrelated with past states. Hence, it is unsurprising that enlarging the readout’s temporal context substantially improves performance. Furthermore, the fact that simple Gaussian smoothing achieves competitive performance suggests that the chosen tasks may not sufficiently challenge the model’s representational capacity, thereby limiting the demonstration of a large-scale pretrained network's advantages.
Questions for authors
-
- Could the similar performance between frozen features and locally smoothed readouts be due to the simplicity of the current task setup, where a simple network with adequate temporal context suffices, making it difficult to demonstrate the value of the pretrained BCIM?
-
- Has consideration been given to comparing frozen features with other readout protocols under more challenging and generalization scenarios, such as cross tasks, cross species, or few shot learning, rather than only predicting hand velocity? These are typically regarded as the strengths of pretrained models.
-
- Table 1 shows that frozen features are more robust to randomized context than raw inputs. Does this indicate that their advantage lies in robustness and generalization, rather than merely predictive accuracy in a single scenario?
-
- Even with an enlarged temporal window, frozen features maintain a stable advantage over raw inputs (Figure 1 and Table 2). How can the remaining advantage be attributed solely to temporal integration?
-
- Since the manuscript claims that frozen-feature’s decoding gains may be unrelated to whether it encodes abstract behavioral structures, why not compare the neural manifolds generated by frozen features with those from other readout protocols, such as the linear model in Figure 2b? Such a comparison could provide more direct evidence.
2 comments
Thank you for your response. However, several fundamental issues remain inadequately addressed. First, the paper’s core conclusion rests on two highly similar datasets, making it difficult to demonstrate the key advantages of a pretrained model. Second, the explanation for the residual advantage of frozen features is insufficient. Accordingly, I maintain my original rating.
Reply to Reviewer Uush
Thank you for the review. Tables A and B are in the general response.
W1, W2 and Q1. The tasks may be solvable with temporal smoothing. We agree that these decoding tasks "may be solvable with simple temporal smoothing", which is the central point of the paper. MC-RTT and MC-Maze were chosen because they are standard benchmarks on which frozen-feature gains are reported. In our window comparisons, widening both readout windows removes most of the mean advantage of frozen features. This is a statement about what a comparison on these benchmarks can show, rather than the value of pretraining in other settings.
Q2. Harder settings: across tasks, across species, few-shot. The main comparisons concern velocity decoding, but two results bear on the question. Appendix C varies the training set from 25 to 100 percent of the training split. Since submission, we also evaluated an NDT2 checkpoint pretrained on human recordings and tested it on both macaque tasks. Across readout windows, its largest mean difference from causal Gaussian smoothing is +0.0037 [-0.0135, +0.0203] on MC-RTT and -0.0003 [-0.0032, +0.0028] on MC-Maze.
Q3. Does Table 1 indicate robustness and generalization? The control tests a narrower property. First, the backbone processes the intact trial. Neighbouring tokens in the readout window are then replaced with tokens from other positions in the same trial, with both readouts refitted. The smaller drop for frozen features is consistent with the backbone integrating temporal context before the readout. Robustness across tasks or sessions would need a different experiment.
Q4. Can the remaining advantage be attributed solely to temporal integration? We agree that these results do not establish temporal integration as the sole source of the remaining advantage. Section 4.1 states that the reductions "quantify changes in the decoding gap, not the causal contribution of temporal integration". At the widest window, the remaining difference has mixed signs across replicates on MC-RTT and stays positive in all five on MC-Maze.
Q5. Why not compare neural manifolds? We chose a functional comparison because the paper's question concerns decoding through the readout. Section 4.3 predicts the projected frozen features linearly from raw spikes and trial position under index alignment. Those predictions replace the projected features in the trained head, without retraining the head. A geometric comparison would complement this analysis.