acceptodds
Under review as a conference paper at ICLR 2027

FreeCAB: A Benchmark for Neural Encoding and Decoding from Freely Moving Hippocampal Calcium Imaging

Abstract

Neural encoding and decoding are commonly evaluated in constrained or trial-structured settings with curated targets and relatively homogeneous recordings, leaving continuous neural–behavior modeling under freely moving conditions less systematically characterized. We introduce FreeCAB, a benchmark built from 58 curated session-level recordings that pair dorsal CA1 miniscope calcium imaging with synchronized multi-view behavior. FreeCAB defines matched bidirectional sequence tasks under a unified session-based interface, evaluating both 2D position and reconstructed 3D pose in single-session neural encoding and decoding, together with explicit tracking-quality control. Position-based extensions further evaluate behavioral-state dependence, held-out adaptation after multi-session pretraining, and joint encoding–decoding training. Across five general-purpose baselines, position decoding is consistently more reliable than position-to-neural encoding. Using 3D pose as the encoding input yields higher Pearson correlation than 2D position for all five baselines, although explained variance does not improve uniformly; conversely, 3D pose remains decodable from neural activity but is more difficult to decode than 2D position. Position decoding is also consistently stronger during locomotion across tested speed thresholds. Multi-session pretraining and joint training produce model-, metric-, and task-dependent effects rather than uniformly reliable gains. Together, FreeCAB provides a benchmark for studying how behavioral representation, behavioral state, and session heterogeneity shape neural–behavior modeling under continuous free behavior.

Primary area:
datasets and benchmarks
Keywords:
neural encoding and decoding, benchmark, hippocampal calcium imaging, freely moving behavior

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aKN2OpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

FreeCAB is a benchmark pairing freely moving dorsal CA1 miniscope calcium imaging with synchronized 8-camera behavioral video across 58 curated sessions. It defines matched encoding and decoding tasks on 2D position and reconstructed 3D pose, and evaluates behavioral-state dependence, cross-session adaptation, and joint encoding–decoding training across general-purpose and neural-domain baselines.

Critical strengths

The paired multi-view video and calcium data is a valuable resource for neural–behavior modeling under free behavior.

The quality of the 3D tracking appears to be high, with low cross-session variance.

Critical weaknesses

Table 3 says nine of the ten held-out sessions come from animals that also contribute held-in sessions, so the cross-session evaluation is largely within-animal. An included leave-animal-out split would be a stronger test of transfer.

The main text spends most of its space on the simple general-purpose baselines. I would prefer to see that material condensed and the foundation-model benchmarks the work is primarily concerned with moved into the main text.

Section 3.3 describes exclusion of sessions for insufficient behavioral coverage and alignment. It would be helpful to retain or co-release a less-filtered version of the session list, as this would enable additional analyses, particularly for self-supervised methods that do not depend on behavioral labels. This is a very mild weakness.

Questions for authors

Do you split over mouse identity, or only over sessions?

Do you have any setup for zero-shot or forecasting settings that are becoming more popular? (e.g. SPINT, Le et al. 2025, arXiv:2507.08402, for zero-shot; POCO, Duan et al. 2025, arXiv:2506.14957, for forecasting)

Will you release the unfiltered/complete data? The multiview camera data may be useful for methods like BEAST3D, Wang et al. 2026, arXiv:2606.02937.

3 comments
AuthorsOpenReview ↗

Response to Reviewer aKN2 (Part 1/3)


W1: Table 3 says nine of the ten held-out sessions come from animals that also contribute held-in sessions, so the cross-session evaluation is largely within-animal. An included leave-animal-out split would be a stronger test of transfer.

Q1: Do you split over mouse identity, or only over sessions?

Thank you for raising this question. Our original split is session-disjoint, so held-in and held-out sessions may come from the same mouse. This evaluates adaptation to new recording sessions, whereas animal-disjoint evaluation tests transfer to previously unseen animals.

We have therefore added an animal-disjoint NEDS 2D-position decoding experiment. All recordings from three target mice (M01, M04, and M10) are excluded from source training and validation, leaving 41 source sessions from 11 mice. Scratch and Pretrain+FT are compared on identical target-session splits using three downstream seeds.

Pretraining improves decoding on two target mice (M01: ΔR2=+0.0205\Delta R^2=+0.0205; M04: +0.0059+0.0059), while performance decreases on the third (M10: −0.0136-0.0136), yielding a mean gain of +0.0043+0.0043. These results extend our evaluation to unseen animals and show that transfer benefits vary across target recordings.

We have clarified the original split in Section 4.2 and added the animal-disjoint results in Appendix E.5 (Table 18).

AuthorsOpenReview ↗

Response to Reviewer aKN2 (Part 2/3)


W2: The main text spends most of its space on the simple general-purpose baselines. I would prefer to see that material condensed and the foundation-model benchmarks the work is primarily concerned with moved into the main text.

Q2: Do you have any setup for zero-shot or forecasting settings that are becoming more popular? (e.g. SPINT, Le et al. 2025, arXiv:2507.08402, for zero-shot; POCO, Duan et al. 2025, arXiv:2506.14957, for forecasting)

Neural-domain baselines (W2). We have moved all six neural-domain baselines—LFADS, NeuroFormer, NDT2, CEBRA, POYO+, and NEDS—into the main results (Section 5.1, Table 2), covering 2D-position and 3D-pose decoding alongside available NEDS encoding results. These evaluations use ten reporting sessions and three seeds.

Additional prediction tasks (Q2). Following your suggestion, we have added supplementary evaluations with SPINT [1] and POCO [2] to explore label-free transfer and neural forecasting within FreeCAB.

Label-free adaptation with SPINT. We adapted SPINT to FreeCAB's 30-frame neural-to-position decoding task, using 41 source sessions from 11 mice and three animal-disjoint target sessions. Each target session provides 300 unlabeled neural frames for calibration, without behavioral labels or parameter updates. SPINT achieves a mean R2R^2 of −0.1071, compared with −0.0354 for a source-global position-mean baseline. Since calibration uses unlabeled target neural activity, this is a label-free adaptation setting rather than strict zero-shot transfer. Our evaluation also uses whole-window decoding instead of SPINT's original last-timestep protocol.

Neural forecasting with POCO. We evaluated POCO on a 48-to-16 neural forecasting task, predicting 16 future neural frames from 48 context frames. Across ten sessions and three seeds, POCO achieves R2=0.5304±0.0299R^2=0.5304\pm0.0299 and Pearson =0.7885±0.0016=0.7885\pm0.0016, outperforming the persistence baseline (R2=0.3737R^2=0.3737). The mean R2R^2 improvement is +0.1568 (paired 95% CI: [0.0737, 0.2165]).

These experiments extend FreeCAB to additional neural prediction settings while keeping encoding and decoding as its primary benchmark tasks. Full protocols and results are provided in Appendix E.8 (Table 21 for POCO).

References

[1] Le et al. SPINT: Spatial Permutation-Invariant Neural Transformer for Consistent Intracortical Motor Decoding. NeurIPS 2025.

[2] Duan et al. POCO: Scalable Neural Forecasting through Population Conditioning. NeurIPS 2025.

AuthorsOpenReview ↗

Response to Reviewer aKN2 (Part 3/3)


W3: Section 3.3 describes exclusion of sessions for insufficient behavioral coverage and alignment. It would be helpful to retain or co-release a less-filtered version of the session list, as this would enable additional analyses, particularly for self-supervised methods that do not depend on behavioral labels. This is a very mild weakness.

Q3: Will you release the unfiltered/complete data? The multiview camera data may be useful for methods like BEAST3D, Wang et al. 2026, arXiv:2606.02937.

Thank you for your interest in the broader FreeCAB dataset and its potential applications to methods such as BEAST3D.

Yes. Upon acceptance, we plan to publicly release the additional unfiltered recordings and corresponding multi-view behavioral videos, alongside the curated 58-session benchmark. This will enable researchers to explore alternative preprocessing and tracking pipelines, develop new behavioral representations, and evaluate methods beyond the benchmark's predefined tasks.

The curated benchmark will remain available with its standardized preprocessing, quality-control criteria, and evaluation splits to support reproducible comparisons.

Reviewer_DT6aOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces FreeCAB, a benchmark for neural encoding and decoding using freely moving hippocampal calcium imaging data. The dataset comprises 58 recording sessions from 19 mice, combining dorsal CA1 miniscope recordings with synchronized multi-view behavioral tracking. The authors establish a unified framework for bidirectional neural–behavioral modeling using 2D position and reconstructed 3D pose, with additional evaluations of behavioral-state dependence, cross-session adaptation, and joint encoding–decoding training.

The benchmark is evaluated using five general-purpose sequence models (MLP, DLinear, TCN, Transformer, and PatchTST), alongside several neural-domain baselines. The main findings are that (1) position decoding is substantially more reliable than neural encoding, (2) 3D pose improves encoding correlation but is more difficult to decode than 2D position, (3) position decoding performs better during locomotion than during low-mobility periods, and (4) multi-session pretraining and joint encoding–decoding training do not consistently improve performance.

The authors position FreeCAB as a standardized resource for evaluating neural–behavioral modeling under continuous, naturalistic conditions. However, the empirical findings are primarily descriptive, and the extent to which the proposed benchmark identifies novel machine learning challenges remains unclear.

Critical strengths

S1. Valuable multimodal dataset with carefully designed acquisition and alignment procedures.

The integration of dorsal CA1 calcium imaging with synchronized multi-view behavioral recordings represents a substantial data collection and preprocessing effort. The resulting dataset provides temporally aligned neural activity, 2D position, and reconstructed 3D pose, accompanied by explicit tracking-quality assessments.

Such recordings are technically demanding to acquire and process, and the resulting resource could be useful for researchers investigating neural representations during naturalistic behavior. In particular, the availability of richer behavioral measurements beyond conventional position tracking may support future investigations of neural–behavioral relationships.

This constitutes the paper’s strongest contribution, although the value of the dataset as a scientific resource should be distinguished from its contribution as a machine learning benchmark.

S2. A unified evaluation framework for bidirectional neural–behavioral modeling.

The benchmark provides a consistent interface for evaluating neural encoding and decoding across behavioral representations, behavioral states, and recording sessions. The inclusion of both general-purpose sequence models and neural-domain baselines facilitates comparisons across different modeling approaches.

The evaluation of cross-session adaptation is particularly relevant given the heterogeneity of neural populations across recording sessions. Although the current experiments do not fully characterize the underlying challenges, the proposed framework could serve as a useful starting point for developing and evaluating transferable neural representations.

Critical weaknesses

W1. Limited novelty and insufficiently established machine learning contribution.

My primary concern is that the manuscript does not sufficiently establish what new machine learning challenges or insights are introduced by FreeCAB beyond previously recognized properties of hippocampal neural coding and cross-session neural modeling.

Several of the main empirical findings are broadly consistent with existing knowledge. For instance, the relationship between hippocampal activity and locomotion has been extensively investigated, while session heterogeneity and the difficulty of transferring neural representations across recordings are well-established challenges in neural population modeling.

Importantly, the manuscript primarily reports these phenomena through differences in baseline performance without systematically investigating their underlying computational mechanisms. The observation that multi-session pretraining produces inconsistent improvements is potentially interesting, but the experiments do not clarify whether these outcomes arise from neural population heterogeneity, recording quality, limited pretraining diversity, architectural constraints, or adaptation protocols.

Similarly, the reported interference between encoding and decoding objectives is not accompanied by sufficient analyses to distinguish optimization-related limitations from fundamental incompatibilities between the tasks.

As a result, the benchmark currently appears more successful as a curated neuroscience dataset than as a diagnostic machine learning benchmark. It remains unclear which specific methodological limitations FreeCAB is designed to expose, or how its findings could guide the development of new neural representation learning approaches.

This distinction is consequential to my assessment because the primary contribution of a benchmark paper should extend beyond providing standardized evaluations to establishing meaningful challenges that advance methodological research.

W2. Insufficient validation of the neural encoding task and potentially confounded interpretations of benchmark results.

A second major concern is whether the proposed evaluation protocol adequately validates the neural encoding task and supports the interpretations drawn from the reported performance differences.

In Table 1, all five general-purpose baselines achieve negative R² values for neural encoding using both 2D position and 3D pose. This indicates that the evaluated models perform worse than a constant-mean predictor under the reported metric, raising questions about the suitability of the task formulation, preprocessing procedures, temporal alignment, and selected model architectures.

Although negative explained variance does not necessarily indicate poor data quality, the manuscript does not sufficiently establish whether these results reflect intrinsic limitations of behavior-based neural prediction, inadequate model specifications, or properties of the processed neural recordings. In particular, comparisons against established neuroscience encoding approaches, such as place-field-based models or regularized GLMs, would be important for determining whether the benchmark provides a meaningful assessment of neural encoding capability.

Furthermore, the interpretation of behavioral-state-dependent decoding performance requires additional caution. The reported improvements during locomotion may be influenced by differences in spatial coverage, target variance, or behavioral distributions between locomotion and low-mobility periods. Since R² is sensitive to the variance of the evaluation targets, higher R² during locomotion does not necessarily establish that more spatial information can be decoded from neural activity in this state.

These issues directly affect the validity and interpretability of the benchmark’s principal empirical findings. Without stronger controls and validation, it is difficult to determine whether the reported differences reveal meaningful properties of neural representations or arise from characteristics of the evaluation protocol.

Questions for authors

I would appreciate clarification on the following points, as the answers could materially affect my assessment of the benchmark’s validity and contribution.

Q1. What specific machine learning challenges does FreeCAB reveal that are not already captured by existing neural benchmarks?

The manuscript emphasizes behavioral representation, locomotion dependence, and cross-session heterogeneity. However, several of the reported findings appear consistent with established neurophysiological observations or previously recognized challenges in neural population modeling.

Could the authors clarify which findings constitute genuinely new methodological insights, rather than empirical confirmations of known phenomena? More specifically, what limitations of existing neural representation learning approaches can be uniquely identified through FreeCAB, and how should these findings inform future model development?

Q2. How do the authors explain the consistently negative R² values observed in neural encoding?

All five general-purpose baselines achieve negative R² values when predicting neural activity from either 2D position or 3D pose.

Could the authors clarify whether these results have been compared against established neuroscience encoding baselines, such as place-field-based models or regularized GLMs?

Additionally, have the authors examined whether encoding performance depends on neuron-level signal quality, temporal alignment, calcium signal preprocessing, or the temporal context available to the models?

These clarifications are important for determining whether the negative explained variance reflects intrinsic task difficulty, limitations of the evaluated architectures, or potential issues with the benchmark formulation.

Q3. Could the observed locomotion-dependent improvements in position decoding be explained by differences in behavioral distributions rather than neural information?

The manuscript reports consistently higher position-decoding R² during locomotion than during stationary or low-mobility periods.

However, these behavioral states may differ substantially in spatial coverage, position variance, and temporal autocorrelation. Such differences could influence regression metrics independently of changes in neural information content.

Have the authors controlled for these factors, for example through matched spatial distributions or comparisons of absolute prediction errors under comparable behavioral conditions?

Without such controls, how do the authors distinguish genuine state-dependent changes in neural decoding capability from differences in evaluation difficulty?

Q4. What factors explain the inconsistent benefits of multi-session pretraining?

The manuscript reports substantial variability in cross-session adaptation gains, including cases where pretraining degrades performance relative to training from scratch.

Have the authors investigated whether these outcomes correlate with measurable session characteristics, such as neuron count, recording quality, behavioral coverage, animal identity, or the amount of available adaptation data?

In particular, does multi-session pretraining improve sample efficiency when the target-session training budget is limited, even if improvements are not consistently observed under the current adaptation protocol?

Clarifying these points would help establish whether the benchmark reveals specific barriers to transferable neural representation learning or primarily reflects variability across the evaluated models and sessions.

Q5. How do the authors distinguish objective interference from optimization or model-capacity limitations in joint encoding–decoding training?

The manuscript suggests that differences in task difficulty and convergence rates may explain the degradation in decoding performance under joint training.

However, this interpretation does not appear to be supported by experiments that isolate the effects of loss weighting, shared model capacity, or optimization dynamics.

Have the authors examined whether the observed performance degradation persists under alternative objective-weighting strategies or comparable model-capacity controls?

This distinction is important because the implications for unified neural–behavioral representation learning differ substantially depending on whether the observed degradation reflects an inherent conflict between encoding and decoding objectives or a limitation of the particular training configuration.

9 comments
AuthorsOpenReview ↗

Response to Reviewer DT6a (Part 1/4)


W1 & Q1: Machine Learning Contribution and Diagnostic Value

Thank you for recognizing the value of the resource and for asking us to clarify its contribution as a machine learning benchmark.

We respectfully distinguish established neurophysiological knowledge from established model capabilities. Knowing that hippocampal activity relates to behavior, or that richer behavioral measurements contain additional information, does not determine which modeling approaches can exploit that information, which predictive metrics improve, or whether learned representations transfer across recordings. FreeCAB provides a reproducible setting in which these questions can be evaluated systematically on continuous, freely moving neural–behavioral data without externally imposed trial boundaries.

FreeCAB's diagnostic contribution is organized around three complementary questions:

  1. Do neural-domain modeling approaches provide consistent empirical advantages? Our common-session comparison evaluates general-purpose and neural-domain models using the same behavioral targets, chronological data partitions, and evaluation metrics. Under the evaluated configurations, neural-domain approaches do not automatically outperform general-purpose baselines in decoding. This motivates evaluating domain-specialized methods under shared data and evaluation conditions rather than assuming an advantage from their design.
  2. Do richer behavioral representations improve prediction consistently across objectives and metrics? FreeCAB evaluates 2D position and 3D pose in both encoding and decoding. Higher encoding correlation does not necessarily translate into improved (R2R^2), while decoding performance differs across behavioral representations. These comparisons establish behavioral representation and predictive fidelity as explicit evaluation axes. Additional classical spatial encoding references and neuron-level analyses further distinguish aggregate model performance from heterogeneous neural predictability.
  3. Are apparent performance benefits robust to evaluation distributions and recording identity? Spatial-occupancy-matched decoding and physical-distance errors examine whether behavioral-state differences persist under more comparable spatial distributions. Our additional animal-disjoint NEDS evaluation finds a small mean pretraining gain (ΔR2=+0.0043\Delta R^2 = +0.0043) across three target animals, with improvements on two and a decrease on the third. This illustrates the importance of evaluating transfer across individual targets rather than relying solely on aggregate gains.

These analyses provide concrete directions for methodological development: better utilization of richer behavioral information, improvements across complementary predictive metrics, robust evaluation under behavioral distribution shifts, and more reliable adaptation to heterogeneous neural populations. They distinguish reproducible observations of model performance from hypotheses about their underlying causes. In particular, we will describe joint-training results as configuration-dependent findings rather than evidence of fundamental objective incompatibility.

We will revise the main text around these diagnostic questions, foregrounding neural-domain comparisons and the additional validation controls. FreeCAB's contribution is therefore a reproducible framework for identifying and comparing model capabilities and limitations under naturalistic recording conditions, independently of whether the underlying physiological relationships are already known.

AuthorsOpenReview ↗

Response to Reviewer DT6a (Part 2/4)


W2 / Q2 / Q3: Neural Encoding Validity and Behavioral-State Controls

Thank you for these questions. We added classical encoding baselines, neuron-level analyses, and behavioral-state controls to examine both findings.

1. Negative encoding R2R^2 (Q2). FreeCAB uses per-neuron macro R2R^2, so negative aggregate scores do not imply that every neuron is unpredictable. We evaluated classical spatial encoders across 58 sessions:

ModelMSE ↓Macro R2R^2 ↑Pearson ↑
Train-mean1.1310−0.13480.0000
Place-field1.1203−0.13870.0971
Spatial GLM1.1211−0.15170.1015

Place-field encoding reduces MSE relative to train-mean by 0.0108 (95% CI: [0.0069, 0.0139]). More importantly, 43.3% of neurons achieve positive held-out R2R^2, and 63.3% outperform train-mean. The new Figure 6 shows the full distribution across 4,435 session–neuron instances and example neural activity predictions (R2=0.218R^2=0.218–0.3590.359). These results demonstrate measurable neuron-level spatial predictability despite negative population averages.

2. Locomotion-related decoding (Q3). After matching spatial occupancy across 58 sessions, all five models retain positive mean locomotion-associated decoding gains (ΔR2=+0.070\Delta R^2=+0.070 to +0.143+0.143), with four of five 95% confidence intervals excluding zero. Mean physical position errors are also 0.32–0.63 cm lower during locomotion.

Excluding 11 synchronization-flagged sessions yields the same pattern on the remaining 47 sessions: all five models retain positive mean ΔR2\Delta R^2 (+0.0817 to +0.1613), with four confidence intervals excluding zero.

These results support the validity of both findings: encoding predictability varies substantially across neurons, while the locomotion-associated decoding advantage persists after spatial-occupancy matching and synchronization screening (Appendices A.2 and E.2–E.3).

AuthorsOpenReview ↗

Response to Reviewer DT6a (Part 3/4)


Q4: Multi-Session Transfer and Sample Efficiency

Thank you for this question. We have expanded the transfer experiments to examine how pretraining gains vary across models, target sessions, animal identities, and downstream data budgets.

Transfer reliability. We analyzed pretraining gains at the session and seed levels (Appendix E.4). CEBRA 2D decoding has a mean ΔR2\Delta R^2 of −0.0119, with positive gains across all three seeds in 0/10 target sessions. In comparison, NEDS 2D encoding achieves +0.0682, with consistent positive gains in 9/10 sessions, while NEDS 3D decoding achieves +0.1007, with positive gains in 10/10 sessions across all three seeds. This reveals substantial differences in transfer consistency across models and tasks that are not apparent from mean performance alone.

Animal-disjoint transfer. We also evaluated NEDS 2D decoding on three held-out mice (M01, M04, and M10), excluding all their recordings from source training and validation (Appendix E.5). The experiment uses 41 source sessions from 11 mice and three downstream seeds. Pretrain+FT achieves an average improvement of +0.0043 in ΔR2\Delta R^2, with gains on two target mice and a decrease on the third. This extends our transfer evaluation to previously unseen animals and shows how adaptation gains vary across individual targets.

Sample efficiency. We evaluated NEDS 2D decoding with 10%, 25%, 50%, and 100% of the target-session training data (Appendix E.6). The corresponding mean gains in ΔR2\Delta R^2 are +0.0100, +0.0060, +0.0172, and +0.0163, respectively. Pretraining produces positive mean gains at all four budgets, with the largest observed gain at 50% of the available training data.

These experiments provide a more detailed characterization of multi-session transfer, covering gain consistency, cross-animal adaptation, and target-data efficiency. They also show why evaluating pretraining across individual recordings and data budgets is informative beyond reporting a single average improvement.

Reviewer_UHg6OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper provides a new benchmark based on freely moving hippocampal calcium imaging. It provides 58 curated dorsal CA1 miniscope sessions from 15 mice with synchronized behavior videos. Using this dataset, the authors test encoding and decoding with general-purpose baselines and neural-domain baselines. The reported findings are that decoding is easier than encoding, 3D pose is harder to decode than 2D position, 3D pose improves encoding Pearson correlation but not R², locomotion improves position decoding, and multi-session / joint training give mixed, model- and metric-dependent gains.

Critical strengths

A valuable and non-trivial resource. This paper presents a neural dataset in a freely moving setting, containing multiple individuals and multiple sessions. Naturalistic neural data are always more valuable and promising; they represent real neural activity rather than responses under carefully designed paradigms, which are often accompanied by overtraining.

Critical weaknesses

Using the FreeCAB dataset, the authors do not reach new conclusions or major supplements/corrections. The main results reported in the paper do not go beyond my existing understanding of the field—for example, decoding is easier than encoding, and 3D poses contain richer information, which is beneficial for encoding but detrimental for decoding. The comparison between general-purpose sequence models and neural-domain baselines is an interesting and valuable point, especially for the Neuro-AI community, but the results and discussion in this paper almost abandon this point. Only partial neural-domain baseline results are presented in the Appendix, and comparison is still lacking.

Questions for authors
  1. neural-domain baselines in the main results. Section 4.4 claims to test six neural-domain baselines “to test whether neural-specialized architectures offer advantages over general-purpose sequence models.” However, in the main results, only five general-purpose baselines are reported, except for the adaptation gains in Fig. 3d. The neural-domain baseline analysis in the Appendix is also incomplete: it only covers 2D-position decoding (encoding only for NEDS) and, as far as I can tell, is on the held-out 10 sessions rather than the 58-session pool. I suggest completing the relevant analysis and discussing the comparison in the main text.

  2. Held-in/held-out split across the same animal Table 3 shows that the same mouse (e.g., M01, M03, M10, M11, M13) contributes both held-in and held-out sessions. Although the paper emphasizes that the benchmark is defined at the session level, cross-session adaptation and generalization may be affected by individual consistency from the same animal. I suggest adding animal-grouped held-out results or discussing this limitation.

  3. Neural-behavior alignment error Appendix A.2 mentions that the synchronized intervals from the two systems are not always exactly identical in length and that residual mismatches of less than 1 s can occur. Please quantify this and clarify whether alignment error may correlate with behavioral state—for example, whether it is larger during locomotion—and whether this could affect the locomotion vs. stationary conclusion in the paper.

4 comments
AuthorsOpenReview ↗

Response to Reviewer UHg6 (Part 1/4)


Weakness: Using the FreeCAB dataset, the authors do not reach new conclusions or major supplements/corrections. The main results reported in the paper do not go beyond my existing understanding of the field—for example, decoding is easier than encoding, and 3D poses contain richer information, which is beneficial for encoding but detrimental for decoding. The comparison between general-purpose sequence models and neural-domain baselines is an interesting and valuable point, especially for the Neuro-AI community, but the results and discussion in this paper almost abandon this point. Only partial neural-domain baseline results are presented in the Appendix, and comparison is still lacking.

Thank you for recognizing the value of FreeCAB and for highlighting the general-purpose versus neural-domain comparison. We have substantially strengthened this part of the evaluation and clarified the broader insights offered by the benchmark.

1. A more complete comparison of general-purpose and neural-domain models. We have moved the results of all six neural-domain baselines—LFADS, NeuroFormer, NDT2, CEBRA, POYO+, and NEDS—into the main text (Section 5.1, Table 2), extending the evaluation to 3D-pose decoding alongside 2D-position decoding and available encoding results.

We also conducted a direct comparison of five general-purpose models with CEBRA and NeuroFormer on the same ten sessions, using aligned targets, chronological splits, and common metrics. Under this evaluation, MLP achieves decoding R2R^2 of 0.3012 / 0.1504 for 2D position / 3D pose, compared with 0.1555 / 0.0384 for CEBRA and 0.0819 / −0.0100 for NeuroFormer. All five general-purpose baselines outperform these two neural-domain models under the tested configurations. This makes neural-domain performance an explicit part of FreeCAB's main analysis, rather than an appendix-only comparison.

2. Behavioral representation matters differently across prediction directions and metrics. FreeCAB evaluates both 2D position and multi-view reconstructed 3D pose in neural encoding and decoding. Although 3D pose produces higher encoding Pearson correlation across all five general-purpose models, it does not consistently improve R2R^2, while 3D decoding remains harder than 2D decoding. These results show that richer behavioral representations do not translate uniformly into improved prediction, and that conclusions depend on both the modeling direction and evaluation metric.

3. Additional analyses reveal differences that aggregate scores can miss. After matching spatial occupancy, all five general-purpose models retain positive mean locomotion-associated decoding differences (ΔR2=+0.070\Delta R^2=+0.070 to +0.143+0.143). Our transfer-reliability analysis also shows substantial model-dependent variation: CEBRA's 2D decoding gains change sign across downstream seeds in every target session, whereas NEDS achieves positive 3D decoding gains in all 30 session–seed comparisons. These findings extend the benchmark analysis to behavioral-state robustness and the consistency of cross-session adaptation.

FreeCAB's contribution goes beyond confirming individual performance trends: it provides a shared evaluation of how model performance changes with behavioral representation, prediction direction, behavioral state, and recording session. The expanded neural-domain comparisons and robustness analyses make these differences more explicit and provide concrete evidence for evaluating neural–behavioral models under freely moving conditions.

Relevant results are reported in Sections 5.1–5.3, Tables 1–2, and Appendices E.3–E.4.

AuthorsOpenReview ↗

Response to Reviewer UHg6 (Part 2/4)


Q1: neural-domain baselines in the main results. Section 4.4 claims to test six neural-domain baselines “to test whether neural-specialized architectures offer advantages over general-purpose sequence models.” However, in the main results, only five general-purpose baselines are reported, except for the adaptation gains in Fig. 3d. The neural-domain baseline analysis in the Appendix is also incomplete: it only covers 2D-position decoding (encoding only for NEDS) and, as far as I can tell, is on the held-out 10 sessions rather than the 58-session pool. I suggest completing the relevant analysis and discussing the comparison in the main text.

Thank you for highlighting the need for a more complete neural-domain comparison. We have expanded the evaluation to 3D-pose decoding, moved all six neural-domain baselines into the main results (Section 5.1, Table 2), and added a direct comparison with general-purpose models.

Expanded neural-domain evaluation. Table 2 now reports LFADS, NeuroFormer, NDT2, CEBRA, POYO+, and NEDS on 2D-position and 3D-pose decoding, together with the available NEDS encoding results. These experiments cover ten reporting sessions and three seeds (0, 7, and 42), with model-specific training protocols documented in the appendix.

Direct model comparison. To compare both model families on the same data, we evaluated five general-purpose baselines alongside CEBRA and NeuroFormer on the same ten sessions, using aligned targets, chronological splits, and identical macro R2R^2 metrics.

Method2D decoding R2R^23D decoding R2R^2
MLP0.30120.1504
DLinear0.21050.0890
TCN0.25160.1322
Transformer0.25840.1046
PatchTST0.20130.1162
CEBRA0.15550.0384
NeuroFormer0.0819−0.0100

Results are session-equal means using seed 42, separate from the three-seed results in Table 2. All five general-purpose models outperform CEBRA and NeuroFormer on both decoding tasks under these configurations.

The original general-purpose benchmark still covers 58 sessions; the neural-domain evaluation and this direct comparison use the same ten-session subset, avoiding comparisons between different session pools.

We also distinguish single-session predictive performance from pretraining benefits. The latter are evaluated separately through Scratch-versus-Pretrain+FT comparisons, including new 3D adaptation and transfer-reliability analyses (Section 5.3, Appendix E.4).

These additions directly address the reviewer's concern about task coverage and visibility of the neural-domain results, while providing a clearer basis for comparing general-purpose and neural-domain approaches in FreeCAB.

AuthorsOpenReview ↗

Response to Reviewer UHg6 (Part 3/4)


Q2: Held-in/held-out split across the same animal Table 3 shows that the same mouse (e.g., M01, M03, M10, M11, M13) contributes both held-in and held-out sessions. Although the paper emphasizes that the benchmark is defined at the session level, cross-session adaptation and generalization may be affected by individual consistency from the same animal. I suggest adding animal-grouped held-out results or discussing this limitation.

Thank you for raising this concern. Our original split is session-disjoint, allowing different sessions from the same mouse in the held-in and held-out pools. This evaluates adaptation to new recording sessions, while animal-disjoint evaluation tests transfer to previously unseen animals.

To examine the effect of shared animal identity, we added an animal-disjoint NEDS 2D-position decoding experiment (Appendix E.5, Table 18). We excluded all recordings from three target mice (M01, M04, and M10) from source training and validation, leaving 41 source sessions from 11 mice. Scratch and Pretrain+FT were compared on identical target-session splits using three downstream seeds.

Pretraining improves decoding for M01 (ΔR2=+0.0205\Delta R^2=+0.0205) and M04 (+0.0059+0.0059), while M10 shows a decrease (−0.0136-0.0136), giving a mean gain of +0.0043+0.0043. These results show that transfer gains remain heterogeneous even when source and target animals do not overlap.

We have clarified the distinction between session-disjoint and animal-disjoint evaluation in Section 4.2, with the additional protocol and per-mouse results reported in Appendix E.5.

AuthorsOpenReview ↗

Response to Reviewer UHg6 (Part 4/4)


Q3: Neural-behavior alignment error Appendix A.2 mentions that the synchronized intervals from the two systems are not always exactly identical in length and that residual mismatches of less than 1 s can occur. Please quantify this and clarify whether alignment error may correlate with behavioral state—for example, whether it is larger during locomotion—and whether this could affect the locomotion vs. stationary conclusion in the paper.

Thank you for raising this concern. Our additional analyses show that the identified synchronization discrepancies do not materially change the locomotion-related decoding results. We clarify the original mismatch description and provide two checks supporting this conclusion.

What does the “less than 1 s” mismatch mean? It refers to differences in the lengths of synchronized intervals extracted from the two recording systems, not a 1-second frame-level neural–behavior alignment error. Our pipeline aligns frames through explicit mappings rather than assuming identical recording durations. Across 55 synchronization-marker pairs, the median absolute interval discrepancy is 42.76 ms (P95: 639.77 ms; maximum: 1,025.89 ms).

Does the mismatch vary with behavioral state? Across 58 sessions, the locomotion-minus-low-mobility difference in absolute internal miniscope–webcam timestamp residuals is +1.56 ms (95% CI: [−0.89, +3.75] ms). When comparing states within the same 60-second blocks, the difference falls to +0.12 ms (95% CI: [−0.82, +0.78] ms), with no clear locomotion-associated increase in these measured residuals.

Does it affect the locomotion conclusion? We excluded 11 sessions flagged by synchronization diagnostics, independently of decoding performance, and repeated the 5-cm occupancy-matched evaluation on the remaining 47 sessions from 14 mice. All five models still show higher mean decoding R2R^2 during locomotion (ΔR2=+0.0817\Delta R^2=+0.0817 to +0.1613+0.1613), with four of five confidence intervals excluding zero. Mean Euclidean position errors are also 0.40–0.73 cm lower during locomotion.

The locomotion advantage remains after spatial-occupancy matching and exclusion of sessions with potential synchronization concerns. The measured timestamp residuals also show no clear dependence on locomotion. These results support the robustness of our original finding to the identified synchronization issues (Appendices A.2 and E.3).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

As the discussion period draws to a close, we would like to highlight the substantive experimental evidence and clarifications provided in our responses. We have addressed the reviewers' major concerns through expanded baseline evaluations, additional controlled analyses, and clarification of FreeCAB's benchmark contributions and research scope. We hope the following summary will facilitate a final assessment of the revised work.

1. Expanded neural-domain evaluation. We have moved all six neural-domain baselines—LFADS, NeuroFormer, NDT2, CEBRA, POYO+, and NEDS—into the main results (Section 5.1, Table 2), extending the evaluation to both 2D-position and 3D-pose decoding alongside available NEDS encoding results. We also provide matched-session comparisons between general-purpose and selected neural-domain models. Neural-domain evaluation is now an integral part of the main empirical analysis rather than an appendix-only supplement.

2. Additional validation of the benchmark and its findings. We have substantially expanded the experimental evidence:

  • Tracking and synchronization: Quantitative multi-view tracking quality control across all 58 sessions, additional alignment diagnostics, and synchronization-screened sensitivity analyses (Section 3.2, Appendix A).
  • Encoding predictability: Classical spatial encoding references and neuron-level analyses show that 43.3% of neurons achieve positive held-out place-field R2R^2, while 63.3% outperform their corresponding train-mean predictors, despite negative aggregate scores (Section 5.1, Appendix E.2).
  • Behavioral-state robustness: After matching spatial occupancy, all five general-purpose baselines retain positive mean locomotion-associated decoding differences (ΔR2=+0.070\Delta R^2=+0.070 to +0.143+0.143), with animal-cluster confidence intervals excluding zero for four models (Section 5.2, Appendix E.3).
  • Transfer reliability: Expanded session-level and seed-level analyses, animal-disjoint evaluation, and target-data-budget experiments characterize when pretraining succeeds and where its gains remain inconsistent (Section 5.3, Appendices E.4–E.6).
  • Additional modeling settings: We have extended the joint encoding–decoding analysis to 3D pose and added supplementary evaluations with SPINT for label-free adaptation and POCO for neural forecasting (Appendices E.7–E.8).

3. FreeCAB's benchmark contribution and research scope. FreeCAB introduces a unified evaluation paradigm for bidirectional neural–behavioral modeling under continuous free behavior, moving beyond task-specific, single-direction evaluations toward systematically studying neural encoding and decoding within a shared benchmark. By evaluating both directions on aligned neural–behavioral sequences, across 2D position and 3D pose, FreeCAB enables direct investigation of encoding–decoding asymmetry, behavioral representation, state dependence, and cross-session transfer under consistent task definitions and evaluation protocols.

This paradigm is not simply a collection of baseline comparisons. It establishes a common experimental foundation for examining how neural–behavioral relationships are modeled in both directions under naturalistic conditions, while providing controlled, reproducible tests of representation sensitivity, state-dependent decoding, and transfer reliability. These constitute concrete methodological challenges for future ML research.

Importantly, a benchmark evaluates whether existing methods succeed under defined conditions; it does not presume that they must. Negative encoding scores and inconsistent pretraining gains therefore reveal meaningful modeling challenges rather than invalidate the benchmark. We welcome further investigation of the underlying computational and biological mechanisms, but maintain that such investigations are complementary to, rather than prerequisites for, recognizing FreeCAB's methodological and empirical contributions.

We appreciate the reviewers' time and engagement throughout the discussion. We would be grateful if reviewers could revisit the revised manuscript and our detailed responses, and share any remaining feedback or an updated assessment before the discussion deadline. We also welcome the Area Chair's consideration of the complete revised evidence and discussion.

Thank you for your consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.