CAPT: A Multi-task Continuous Autoregressive Transformer enabling Cross-dataset and Cross-species Transfer for Calcium Population Dynamics

Calcium imaging has enabled large-scale recordings of neural population activity across diverse animals, laboratories, and experimental paradigms. However, these datasets remain highly fragmented: Existing approaches are often designed for individual datasets or tasks, making it unclear whether their learned representations are reusable for new calcium imaging trace datasets. To tackle this gap, we present CAPT, a Continuous Autoregressive Population Transformer for calcium population dynamics. CAPT models continuous calcium traces directly through a continuous patch tokenization strategy and is trained autoregressively with a mean squared error objective, enabling end-to-end training and adaptation to diverse downstream tasks. We first pretrain CAPT on a large-scale mouse calcium imaging dataset and evaluate its transferability across independent mouse, larval zebrafish, and C. elegans datasets collected by different laboratories. In these transfer settings, the pretrained backbone is frozen and only adaptation modules, i.e., neuron and session embeddings or task-specific heads, are updated. Across neural population forecasting and behavior decoding tasks, CAPT consistently outperforms specialized and general-purpose baselines, demonstrating that a single continuous autoregressive backbone can learn representations that transfer beyond the original pretraining distribution. Beyond predictive performance, multimodal analyses using NeuroPAL annotations in C. elegans datasets show that CAPT embeddings form a shared functional space across datasets and capture cell-identity-related structure. Together, these results suggest that continuous autoregressive modeling provides an effective framework for learning reusable calcium population representations that transfer across datasets, experimental paradigms, and species. Code is available at https://anonymous.4open.science/r/project_code_2026-0AC8/