acceptodds
Withdrawn from ICLR 2027

The Variance Brain Foundation Models Forgot: A Third-Order Basis for Predicting Individual Differences

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Brain foundation models (BFMs) are self-supervised Transformers pretrained on fMRI data. We posit that these models should capture each subject's cognitive performance from their fMRI signal. Yet across three published BFMs and every readout we test, they predict cognition worse than a linear regression from the 80K parameters of the functional connectivity matrix (FC). Scale does not help: BrainLM's 650M model predicts cognition worse than its 111M. We attribute this to a variance allocation problem: BFM pretraining captures the variance components that dominate fMRI but not the higher-order structure that predicts cognition. Our per-cumulant analysis of the reconstructed signal shows that the second-order covariance is partially preserved, while the third-order co-skewness tensor is largely destroyed. To recover what BFMs lose, we design a linear pipeline that projects the fMRI signal into the subspace that best preserves its co-skewness and computes FC there. This improves over raw FC and exceeds every pretrained BFM on both datasets and both parcellations we test ( from to on AOMIC AAL-424), matching or exceeding the closest published FC results on comparable targets with no pretraining and no GPU. We recover the raw-FC ceiling on BrainLM's forward pass by finetuning with a loss targeted at this same subspace, which the same loss outside the subspace does not reach. This points to the pretraining objective, not the model size, as the bottleneck.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI, functional connectivity, brain foundation models, self-supervised learning, higher-order statistics, Tucker decomposition, pretraining objectives

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_2JgLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies whether pretrained brain foundation models capture information relevant to individual cognitive differences. The authors first show that the foundation models fall short and even worse than FC baselines, then they propose a pretraining objective that prioritize high-variance signals, along with an imporved FC baseline with Tucker decomposition.

Critical strengths
  1. The research question is important. The paper shows the foundation model may not perform as they are expected on cognition-related tasks, and that they may even worse than FC baselines.
  2. The proposed FC-Tucker is efficient without no large-scale pretraining.
Critical weaknesses
  1. The writing and presentation is unclear. The paper is difficult to follow. For example, the authors discusses kappa2kappa_2 and kappa3kappa_3 without defining them. The empirical findings, datasets, and methods are mixed in section 3, the narrative shifts between variance allocation, higher-order information, and FC preservation.

  2. The foundation models the authors studied are outdated. They are limited to ROI-based input (while recent focus more on atlas-free), and they are not tuned for cognitive-related tasks. So the comparison with the FC baseline is not fair. Also, the near-zero reconstruction fidelity of BrainLM-650M and the poor HCP reconstruction results are surprising relative to previously reported BrainLM performance.

  3. The authors attribute BFM underperformance to the loss of cognition-relevant third-order information. However, showing that a co-skewness-derived subspace improves FC prediction does not establish that third-order interactions are responsible for the improvement or for BFM failures.

  4. The statistics used in this paper, such as third-order statistics, co-skewness analysis, and Tucker decomposition are well-established techniques. The novelty is fundamentally limited toe combination of them.

  5. The baseline is not sufficient. Some other classical ML baselines, such as random orthogonal projections and ICA should also be studied. It's unknowned whether improvements arise from third-order information or more general changes in feature representation.

  6. The HCP target is defined as the first principal component of 58 behavioral measures, which may not correspond to the cognitive factor used in prior studies. The authors should validate this target and clarify the comparability of their results with published benchmarks.

  7. The proposed loss preserves second-order FC within a third-order-selected subspace, rather than directly preserving third-order statistics. Also, finetuning improves reconstructed FC but not the model embeddings, so it's not known whether this can produce a better generic representation.

Questions for authors
  1. Is the key finding that third-order interactions contain additional cognitive information, or that a third-order-derived spatial basis improves conventional FC prediction? These are distinct claims, and the paper does not clearly distinguish them.

  2. How is the Log-Cholesky metric computed for rank-deficient FC matrices when the number of ROIs exceeds the number of timepoints?

Reviewer_u6cJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors argued that the feature readouts of existing brain fMRI foundation models are less correlated to the first principle component of behavioral measurements than the raw Functional Connectivity (FC) matrix extracted from the fMRI. They present a method replacing FC matricies with a new FC derived from BOLD that is projected to a third-order space. The experimental results shown the proposed method can replicate the correlation between raw FC and behavioral measurements on HCP (n=955 subjects) and AOMIC (n=876 subjects) datasets, while the readouts of BOLD-input brain foundation model cannot correlate.

Critical strengths
  1. The authors tested three published BFMs against parcellation-matched classical FC baselines under a replicated nested cross-validation protocol, with leakage controls, fully nested rank selection, and FDR-corrected significance testing.
Critical weaknesses
  1. The core target, i.e., the Pearson's correlation between Brain Foundation Model (BFM) readouts and "cognitive-ability composite", is not rigorously defined or referenced. From Lines 217-219, "cognitive-ability composite" looks like the 1st principal component from PCA. While in contributions claimed by the authors that the core target is "predicting cognitive labels", there is no evidence and analysis present in the paper to support that the 1st principal component is the representation of "cognitive labels", which is also not defined.

  2. Contributions are unclear. The authors made a fundamental claim about the weaknesses of existing brain foundation models (BFM), yet the evidence covers only BOLD-input models. The sole FC-input BrainMass was claimed "as a control rather than a direct comparison" (Sec. 4.1). The BFM readouts of BOLD feature representation comparing to the proposed FC-input method is not fair. The core result that FC-Tucker beats raw FC fails the FDR-corrected significance test (q = 0.092 and q = 0.153, Sec. 4.4) for HCP.

  3. The paper relevance with BFM is the claim that the pretraining objective is the bottleneck instead of the model size, and the key evidence is a new cumulant-informed finetuning loss that increases the correlation between BrainLM and behavioral measurements. But this experiment is demonstrated only on BrainLM, the oldest BFM with BOLD as the input and the simplest objective (masked reconstruction).

Questions for authors

None

Reviewer_VS9SOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies why brain foundation models underperform functional connectivity (FC) in cognitive prediction. It proposes co-skewness-informed Tucker FC features and an associated finetuning loss, evaluated against matched baselines on AOMIC and HCP.

Critical strengths
  • The comparisons include useful controls for readout choice. Parcellation- and input-length-matched FC baselines, multiple readouts, and alternative prediction heads make it harder to dismiss the observed performance gaps as a consequence of one poorly chosen prediction head.

  • The proposed FC representation is simple and retains positive nested mean gains on the main datasets. It requires no neural pretraining. The statistical significance of those gains still needs to be established with paired inference for the nested procedure.

Critical weaknesses
  1. The central explanation of BFM failure is not established by the experiments. Table 7 supports preferential reconstruction of dominant variance primarily for AOMIC-111M (B1 fidelity 0.749). AOMIC-650M and both HCP models instead have band-wise temporal reconstruction fidelity between -0.013 and 0.036 throughout. These three settings show broadly poor temporal reconstruction, without direct evidence of selective variance allocation. Native-pipeline checks are needed before attributing this failure to the pretraining objective.

    For a fixed Tucker basis, the predictive features are determined by the subject's covariance. The successful objective likewise consists of second-order FC distances, predominantly within that basis, rather than an explicit constraint on κ3\kappa_3 (Section 3.7). The direct κ3\kappa_3 loss fails to optimize meaningfully under the reported settings (validation-loss reduction below 1%; FC-reconstruction r=0.032/0.030r=0.032/0.030, Table 21). The recovery is demonstrated by converting reconstructed signals back into FC; it does not establish improved cognitive readout from the pooled representation or identify third-order preservation as the cause.

    The numerical comparison also needs clarification. With 424 regions and 200 timepoints, centered empirical FC has rank at most 199, whereas the Log-Cholesky loss requires positive-definite matrices. I could not find a specification of the regularization used before Cholesky factorization. Comparing this singular full-space problem with an 80-dimensional subspace can confound basis selection with regularization and conditioning. A matched-rank PCA or random-basis loss comparison under the same numerical treatment is needed to isolate the proposed mechanism.

  2. The evaluation leaves test-subject exposure unresolved and selects the headline ranks on outer results. Appendix U describes fixed finetuning splits of 703/173 subjects for AOMIC and 763/192 for HCP, while Table 3 reports 200 fold-level evaluations. I cannot determine whether every outer test subject was excluded from representation fitting. If an outer test subject's unlabeled input was used for finetuning or adaptation, that evaluation would not measure generalization to an entirely unseen subject. Avoiding test-label use alone would not resolve this concern; the overlap between these partitions needs to be specified.

    Selecting ranks using outer evaluation performance introduces optimistic selection bias into Table 2. This conflicts with the Reproducibility Statement's assertion that no reported number depends on post-hoc hyperparameter choice. The AOMIC-AAL and HCP-Schaefer results are 0.368 and 0.571, whereas nested selection on the coarser grid gives 0.346 and 0.558 (Table 27). The positive nested mean gains need paired corrected tests and confidence intervals for the gains themselves. External validation is inconclusive: the fixed-rank meta-analysis gives 0.024±0.0250.024\pm0.025 (p=0.34p=0.34), and three of seven cohorts change sign between rank-selection protocols (Appendix F).

Minor comments. Please check the AOMIC sample count in Figure 16: it is 877 there, versus 876 in Section 3.3.

Questions for authors
  1. Were the finetuned checkpoints trained separately for each outer fold? Please specify any subject overlap between each outer test fold and the sets used for finetuning, basis estimation, target PCA, early stopping, and loss-weight selection.

  2. What exact construction and parameters make the FC matrices positive definite before Cholesky factorization? How does the full-space versus subspace loss comparison change with this regularization?

  3. To test the mechanistic claim, do matched-rank PCA or random-subspace FC losses recover similar performance under the same optimization and numerical settings? Does successful finetuning reduce directly measured third-cumulant error?

  4. Could you provide paired corrected tests and confidence intervals for the fully nested gains, and reconcile the Reproducibility Statement with the sweep-selected results in Table 2?

  5. What explains Table 7's near-zero temporal reconstruction fidelity in three model-dataset settings? Please provide a sanity check of the released checkpoints under their native input conditions and specify TR handling, window length, normalization, and input-output alignment in the reported evaluations.

  6. Why were BrainLM and Brain-JEPA chosen as the main time-series baselines? How do you delimit the conclusions relative to newer approaches such as Brain-Semantoks and the function-only BrainHarmonix-F?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.