- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper introduces CAPT, a continuous autoregressive Transformer for modeling calcium imaging population activity across datasets and species. The proposed framework combines continuous patch tokenization, a dual-axis Transformer backbone, dynamic residual gating, and neuron-aware QKV modulation. The model is pretrained on a large-scale mouse calcium imaging dataset using an autoregressive forecasting objective and subsequently evaluated on independent mouse, larval zebrafish, and C. elegans datasets. During transfer, the pretrained backbone remains frozen, while dataset-specific neuron and session embeddings are optimized for forecasting, followed by task-specific readout training for behavior decoding.
The authors report improved forecasting and decoding performance over several existing baselines, including POCO, POYO+, and CalM. They further investigate the biological relevance of the learned representations using NeuroPAL annotations in C. elegans, showing that the adapted neuron embeddings support cross-dataset cell-identity classification.
The central contribution of this work is an empirical demonstration that a mouse-pretrained autoregressive backbone can be reused across heterogeneous calcium imaging datasets with limited parameter adaptation. However, the extent to which these results establish the transfer of neural dynamics, rather than generic calcium signal statistics or architectural inductive biases, remains unclear. Similarly, the NeuroPAL analysis provides evidence of identity-related information in the adapted embeddings, but does not directly establish that conserved neural dynamics are represented across species.
Critical strengths
- Biologically motivated evaluation beyond predictive performance
One notable strength of this work is the incorporation of NeuroPAL annotations to investigate whether learned neuron embeddings capture biologically meaningful information. Rather than relying exclusively on forecasting and decoding performance, the authors examine whether the representations support cell-identity classification across independent C. elegans datasets. I particularly appreciate this attempt to connect neural representation learning with biological interpretability, as such evaluations remain relatively uncommon in neural foundation model research. Although the results do not directly establish the transfer of conserved neural dynamics, they provide an interesting empirical observation and a potentially useful direction for future investigations.
- Broad empirical evaluation across heterogeneous calcium imaging datasets
The evaluation spans multiple independently collected datasets across mice, larval zebrafish, and C. elegans, covering different recording conditions, sampling rates, and experimental paradigms. This breadth is valuable because cross-dataset generalization remains an important challenge in neural population modeling. The frozen-backbone adaptation protocol also provides a useful empirical setting for investigating model reusability. While the results do not fully establish the underlying mechanism of transfer, the extensive evaluation offers a potentially valuable benchmark for future work.
Critical weaknesses
- The central claim of cross-species neural dynamics transfer is not sufficiently substantiated.
My primary concern is the substantial gap between the paper’s central scientific claim and the evidence provided. The authors demonstrate that a mouse-pretrained backbone can be adapted to calcium imaging datasets from other species by optimizing target-specific neuron and session embeddings. However, successful adaptation does not necessarily imply that representations of neural population dynamics have been transferred across species.
In particular, the observed performance could be explained by generic temporal properties of calcium fluorescence signals, such as autocorrelation, smoothness, or indicator-dependent kinetics, rather than conserved neural dynamics. This distinction is especially important given the substantial differences in sampling rates, calcium indicators, and biological timescales across the evaluated datasets. The paper does not provide sufficient controlled experiments or mechanistic analyses to distinguish these competing explanations.
The NeuroPAL experiments are an interesting attempt to establish biological relevance, but they remain indirect evidence. Cross-dataset cell-identity classification demonstrates that the adapted embeddings contain identity-related information; it does not establish that this information originates from mouse pretraining or reflects conserved neural dynamics. Furthermore, the failure of independently trained models to produce directly comparable embeddings is not sufficient evidence that the shared coordinate system learned through the frozen backbone captures biologically conserved structure.
Consequently, the experiments establish the practical reusability of a pretrained backbone, but do not sufficiently substantiate the stronger scientific interpretation of cross-species neural dynamics transfer. This distinction is central to my assessment because cross-species transferability is presented as one of the paper’s primary contributions.
- The methodological contributions and the role of pretraining are insufficiently isolated.
My second concern is the limited evidence establishing which components of CAPT are responsible for its reported improvements, particularly under cross-species transfer. The proposed framework largely combines existing modeling techniques, including continuous patch tokenization, dual-axis attention, dynamic residual gating, neuron-aware QKV modulation, and residual autoregressive prediction. While their combination yields competitive empirical performance, the paper does not convincingly establish a new methodological principle underlying transferability.
More importantly, the current experimental design does not adequately distinguish the contribution of pretrained representations from that of the architecture itself. For example, a randomly initialized frozen backbone with trainable target-specific embeddings would provide an important control for determining whether the observed transfer performance genuinely depends on knowledge acquired during mouse pretraining. Similarly, target-only training and controlled comparisons of adaptation strategies would help clarify the contribution of source-domain pretraining.
Although the authors provide architectural ablations, these primarily examine forecasting performance rather than the cross-species transfer settings that motivate the paper. Therefore, the reported improvements cannot be confidently attributed to the proposed components or to the autoregressive pretraining strategy.
Without these controls, it remains unclear whether CAPT introduces a genuinely transferable representation learning approach or primarily offers an effective architecture for adapting to heterogeneous calcium imaging datasets. This substantially limits the methodological novelty and the strength of the paper’s conclusions.
Questions for authors
Q1. What evidence supports the transfer of neural dynamics rather than generic calcium fluorescence statistics?
The paper’s central claim concerns transferable representations of neural population dynamics across species. However, calcium fluorescence signals contain substantial temporal structure arising from indicator kinetics, autocorrelation, and recording-specific characteristics. How do the authors distinguish the transfer of biologically meaningful neural dynamics from the reuse of generic temporal statistics shared across calcium imaging datasets?
In particular, have the authors considered controls that preserve basic signal statistics while disrupting population-level neural interactions? Such comparisons would help determine whether CAPT transfers neural population structure or primarily models generic fluorescence dynamics.
Q2. How much of the reported transfer performance actually depends on source-domain pretraining?
Could the authors clarify whether they evaluated a randomly initialized, frozen CAPT backbone with the same target-specific embedding adaptation procedure?
This comparison is particularly important because the current results do not isolate the contribution of mouse pretraining from the architectural inductive biases of CAPT. A comparison against target-only training would further clarify whether source-domain pretraining provides a meaningful advantage over learning directly from the target dataset.
Q3. What is the evidence that the proposed architectural components specifically improve cross-species transferability?
The paper introduces several architectural modifications, including dynamic residual gating, neuron-aware QKV modulation, and residual-last prediction. However, the reported architectural ablations primarily focus on forecasting performance within a limited number of sessions.
Do these components also improve transfer performance on zebrafish and C. elegans datasets? In particular, is the complete architecture consistently superior to a simpler continuous autoregressive dual-axis Transformer under the same frozen-backbone adaptation protocol?
This distinction would materially affect my assessment of the methodological contribution, as the central motivation of the paper is transferability rather than source-domain forecasting accuracy.
Q4. How do the authors account for differences in physical timescales across species and recording conditions?
The datasets differ substantially in sampling frequency and calcium indicators. For example, the mouse pretraining dataset is sampled at 6 Hz, whereas the C. elegans datasets are sampled at approximately 1.67 Hz. Consequently, a fixed-length temporal patch corresponds to substantially different physical durations across datasets.
How does CAPT account for these differences when transferring temporal representations? Were any experiments conducted to determine whether the reported transfer performance is robust to changes in sampling frequency or temporal resampling?
Without such controls, it is difficult to determine whether the backbone learns transferable temporal dynamics or simply adapts to dataset-specific temporal statistics through newly optimized embeddings.
Q5. What evidence establishes that the NeuroPAL results reflect biologically meaningful shared representations?
I appreciate the NeuroPAL-based analysis, as it attempts to evaluate learned representations beyond predictive performance. However, cross-dataset cell-identity classification does not necessarily establish that the embeddings encode conserved neural dynamics.
Could the authors clarify what specific evidence supports this interpretation? In particular, would comparable identity-classification performance emerge from a randomly initialized frozen backbone or a backbone pretrained on temporally matched surrogate calcium signals?
Additionally, the independently trained CAPT models used in Table 15 do not necessarily share a common embedding coordinate system. Therefore, their poor cross-dataset classification performance may reflect representational misalignment rather than the absence of biologically meaningful information.
How do the authors distinguish these possibilities?
Q6. Are the improvements over existing baselines attributable to the pretrained backbone or to differences in adaptation and decoding protocols?
CAPT employs a two-stage adaptation procedure, first optimizing target-specific neuron and session embeddings and subsequently training a nonlinear behavior decoder. The compared baselines differ in their architectures, objectives, and adaptation strategies.
Could the authors clarify whether the baselines were evaluated under comparable adaptation budgets and decoder capacities? In particular, would CAPT retain its advantage if the same decoding architecture and comparable trainable parameter budgets were used across methods?
This clarification is important for attributing the observed performance improvements to the proposed representation learning framework rather than differences in downstream adaptation capacity.
9 comments
Rebuttal part II
[W1 & Q2]:
We further evaluate a randomly initialized, frozen CAPT backbone using the same target-specific embedding adaptation protocol. As shown below, the pretrained backbone consistently outperforms the random backbone across transfer settings, demonstrating that source-domain pretraining provides substantial benefits beyond the architectural inductive biases alone.
| Dataset | randomly initialized, frozen CAPT backbone | Pretrained backbone | Δ Corr |
|---|---|---|---|
| AHB | 0.1900 ± 0.0830 | 0.4003 ± 0.0197 | +0.2102 |
| Brysch | 0.1616 ± 0.0278 | 0.2129 ± 0.0249 | +0.0513 |
| CaLiAli | 0.2074 ± 0.0102 | 0.2970 ± 0.0082 | +0.0895 |
| Park | 0.1804 ± 0.0102 | 0.2576 ± 0.0173 | +0.0771 |
| Atanas | 0.1554 ± 0.0825 | 0.2575 ± 0.1062 | +0.1021 |
| Copper | 0.1937 ± 0.0835 | 0.2823 ± 0.1298 | +0.0886 |
To directly assess the contribution of source-domain pretraining, we compare mouse-pretrained CAPT with the same architecture trained from scratch on each target dataset. The pretrained model is evaluated using frozen-backbone adaptation, whereas the target-only model is trained with all backbone parameters learnable. As shown below, frozen-backbone CAPT generally achieves comparable or better performance than target-only training, despite keeping the pretrained backbone fixed. Especially, the largest forecasting improvements are observed on the two C. elegans datasets.
| Dataset | Train from scratch | Embedding adaptation |
|---|---|---|
| AHB | 0.4088 ± 0.0757 | 0.4003 ± 0.0197 |
| Brysch | 0.2094 ± 0.0047 | 0.2129 ± 0.0249 |
| CaLiAli | 0.2956 ± 0.0077 | 0.2970 ± 0.0082 |
| Park | 0.2467 ± 0.0132 | 0.2576 ± 0.0173 |
| Atanas | 0.2291 ± 0.0913 | 0.2575 ± 0.1062 |
| Copper | 0.2257 ± 0.1137 | 0.2823 ± 0.1298 |
| Dataset / behavior | Train from scratch | Embedding adaptation | Δ R2 |
|---|---|---|---|
| Atanas / head angle | 0.5668 ± 0.1615 | 0.5918 ± 0.1522 | +0.0250 |
| Copper / velocity | 0.7429 ± 0.0929 | 0.7503 ± 0.0948 | +0.0074 |
| Brysch / eye position | 0.5029 ± 0.1673 | 0.5250 ± 0.1503 | +0.0221 |
[W2 & Q3]:
We conduct architectural ablations under the same frozen-backbone transfer protocol as required. The full CAPT consistently outperforms the simplified continuous autoregressive Transformer without dynamic gating, QKV modulation, or residual-last prediction across transfer settings. These results demonstrate the robust benefit of the proposed architectural components for transfer performance.
| Dataset | Session | Simplified CAPT | CAPT | Δ Corr |
|---|---|---|---|---|
| AHB | 3 | 0.3818 ± 0.0177 | 0.4003 ± 0.0197 | +0.0185 |
| Brysch | 29 | 0.1895 ± 0.0246 | 0.2129 ± 0.0249 | +0.0234 |
| CaLiAli | 3 | 0.2833 ± 0.0079 | 0.2970 ± 0.0082 | +0.0136 |
| Park | 3 | 0.2473 ± 0.0174 | 0.2576 ± 0.0173 | +0.0103 |
| Atanas | 80 | 0.2372 ± 0.1041 | 0.2575 ± 0.1062 | +0.0203 |
| Copper | 27 | 0.2634 ± 0.1185 | 0.2823 ± 0.1298 | +0.0188 |
[W1 & Q4]:
To examine the effect of different sampling rates, we resample the same three sessions of the Tseng and Sun dataset at 0.5× and 2× the original sampling rate. We transfer the same pretrained CAPT backbone to each version following the embedding-only fine-tuning protocol without any sampling-rate-specific modifications to the model architecture or patch length. During test evaluation, we adjust the context and forecasting horizon proportionally to cover the same physical durations across different sampling rates. We observe model stays robust across different sampling rates.
Together with the comparison against a simple autoregressive baseline in [W1 & Q1], these results suggest that CAPT benefits from population-level structure beyond simple individual-neuron statistics, while remaining robust to changes in sampling rate.
| Dataset | Downsample ×0.5 | Original ×1 | Upsample ×2 |
|---|---|---|---|
| Context → Horizon | 24 → 16 | 48 → 32 | 96 → 64 |
| Tseng 3 sessions | 0.3418 ± 0.0726 | 0.3502 ± 0.0649 | 0.3483 ± 0.0764 |
| Sun 3 sessions | 0.3850 ± 0.1144 | 0.3966 ± 0.1067 | 0.4085 ± 0.1029 |
Rebuttal part III
[W2 & Q5]:
To address this concern, we repeat the NeuroPAL classification experiments using a shared, randomly initialized backbone and backbones pretrained on the 3-sessions surrogate Tseng data described in Q1. For a controlled comparison, we also pretrain a backbone on the same three original Tseng sessions. The backbone pretrained on real calcium recordings substantially outperforms both the randomly initialized backbone and those pretrained on surrogate data that preserve single-neuron temporal statistics. These results demonstrate that our advantage cannot be explained solely by the constraint of an arbitrary shared backbone or by single-neuron calcium signal statistics, but also depends on population-level temporal structure learned during pretraining.
We agree that the result in Table 15 does not necessarily indicate an absence of biologically meaningful information. However, in practical cross-dataset applications, where no prior cell correspondences or external alignment reference is available, such misalignment is itself a substantial limitation, as the learned representations cannot be directly compared or transferred. CAPT addresses this issue by adapting different datasets into a shared pretrained backbone, providing a common representation space while retaining cross-dataset cell-identity-related structure.
| Balanced Acc/ Macro F1 | 189-session pretrained | Random backbone |
|---|---|---|
| Pooled | 52.47 / 53.40 | 27.04 / 26.19 |
| Atanas → Copper | 36.45 / 38.14 | 24.42 / 22.12 |
| Copper → Atanas | 35.35 / 37.23 | 20.56 / 20.83 |
| Balanced Acc/ Macro F1 | 3-session pretrained | 3-session IAAFT | 3-session trial shuffle |
|---|---|---|---|
| Pooled | 44.71 / 45.16 | 34.66 / 35.28 | 34.17 / 33.45 |
| Atanas → Copper | 35.25 / 35.39 | 20.16 / 20.43 | 19.94 / 20.48 |
| Copper → Atanas | 31.57 / 33.77 | 20.66 / 21.39 | 27.45 / 26.44 |
[W2 & Q6]:
We thank the reviewer for this important question. POYO+ is primarily designed for supervised neural decoding, while CAPT employs a self-supervised forecasting backbone followed by a task-specific decoder.
We first compare the two methods with matched model sizes (27M parameters) and comparable training budgets. As shown below, CAPT consistently outperforms POYO+ on both the pretraining and cross-species transfer datasets, suggesting that the performance advantage cannot be explained by model size alone.
| Setting | 27M POYO+ | 27M CAPT |
|---|---|---|
| Tseng held-in decoding macro R² | 0.7127 | 0.7941 |
| Atanas head-angle R² | −0.0008 | 0.5918 |
| Copper velocity R² | 0.0528 | 0.7503 |
To further examine whether these improvements arise primarily from differences in decoder capacity or adaptation procedures, we conducted an additional common-decoder experiment. Following each method's original pretraining and target-adaptation pipeline, we extracted the features from the final backbone outputs and froze the adapted feature extractors. We used PCA to reduce the CAPT feature dimensionality to match that of POYO+, and trained newly initialized decoders with the same POYO+-style cross-attention decoder. Despite this dimensionality reduction and the replacement of its native decoder, CAPT retains a substantial decoding advantage under this shared readout architecture.
These results show that CAPT's decoding advantage persists even when both methods use an identical, newly initialized decoder architecture, indicating that the improvement cannot be explained solely by differences in their native decoder designs or capacities.
| Dataset | 27M CAPT + common decoder | 27M POYO+ + common decoder |
|---|---|---|
| Atanas head-angle | 0.3235 ± 0.2319 | −0.0317 ± 0.0700 |
| Copper velocity | 0.7244 ± 0.0952 | 0.0588 ± 0.2649 |
We sincerely thank the reviewer again for the careful feedback. We hope our responses have addressed your concerns, and would be happy to provide further clarification if needed.
Rebuttal part I
We thank the reviewer for the detailed comments and questions with sufficient explanation.
[W1 & Q1]:
We thank the reviewer for this important question. To investigate whether CAPT benefits from population-level information beyond individual-neuron temporal statistics, we conduct three additional experiments: (1) removing neuron-axis attention to prevent cross-neuron communication; (2) pretraining CAPT on surrogate source data with disrupted cross-neuron relationships; and (3) introducing an AR(2) baseline to evaluate forecasting based on simple temporal dependencies.
We construct two surrogate-data controls: (1) IAAFT (iterative amplitude-adjusted Fourier transform), independently generating surrogate traces that preserve the amplitude distribution and approximately preserve the power spectrum; (2) trial shuffle, independently permuting trial assignments for each neuron. These controls disrupt different aspects of cross-neuron temporal structure while preserving selected single-neuron signal statistics. All three pretrained models use the same three Tseng source sessions and identical training budgets.
For IAAFT, we first independently shuffle each neuron’s trace within each trial. At each iteration, we take its FFT, replace the magnitudes with the original trace’s magnitudes while keeping the current phases, and apply the inverse FFT. We then replace the smallest resulting value with the original minimum, the second smallest with the original second-smallest value, and so on. We repeat this up to 200 times, retaining the surrogate with the smallest Fourier-magnitude error.
For the AR(2) baseline, we independently fit each neuron's autoregressive coefficients with xt=b+α1xt−1+α2xt−2+ϵt using least squares on the training trials only. At test time, the model is initialized with the observed context and recursively predicts the forecasting horizon.
We evaluate all models on three Tseng sessions and three Sun sessions under source-domain and cross-dataset transfer settings, respectively. For the surrogate-pretraining controls, only the source pretraining data are modified, while the target adaptation and evaluation data remain unchanged. This allows us to examine the contribution of source-domain population structure to subsequent transfer.
As shown below, CAPT consistently outperforms all ablation baselines on both Tseng and Sun. In particular, removing neuron-axis attention or disrupting cross-neuron temporal structure during pretraining leads to substantial performance degradation. These results suggest that population-level temporal information contributes to CAPT's forecasting and transfer performance beyond simple individual-neuron autoregressive statistics.