acceptodds
Under review as a conference paper at ICLR 2027

A Connectome Interpreter for Brain Foundation Model: From Direct to Directed Connectivity

Abstract

Brain foundation models (BFMs) recently demonstrate strong predictive performance across various clinico-demographic traits; yet, what they internally represent remains a black box. Here, we introduce a data-driven connectome interpreter by training a canonical BFM with bias-free learnable embedding tokens on large-scale human lifespan fMRI datasets. This framework allowed us to investigate biological implication of whole-brain spatiotemporal self-attention, a core operation of BFM. Our analyses revealed that the self-attention, driven by learnable-embedding representations naturally emerging from the data, is far more than an intermediate product for downstream prediction but intrinsically embraces key information to recover the brain's "direct" connectivity (unmediated, non-spurious communication backbone). Moreover, integrating these attention weights with an established effective-connectivity method, a framework we formalized as AXON (Attention-informed eXtraction Of directed Networks), accurately resolves "directionality" (sender-receiver relationships) of these connections with high accuracy in both mouse and macaque tract-tracing connectomes as well as in the human-brain simulations where the ground-truth network is synthesized based on the empirical data. Finally, when leveraging the AXON-derived effective connectivity to predict comprehensive downstream tasks including demographics, phenotypes (58 behavioral traits) and brain states (8 task conditions), our approach broadly outperformed all state-of-the-art BFMs as well as classic yet powerful linear regression models that have recently rivaled the BFM accuracy, yielding top performance in most individual traits. These results suggest that extracting a biologically meaningful interpretation of what foundation models internally compute—rather than scaling or hand-designing their architecture—is a critical next step for developing robust BFM-based imaging markers.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI, neuroscience, brain foundation model, interpretability, effective connectivity, connectome

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_bsvHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AXON, which uses attention learned through masked reconstruction of fMRI signals as an edge-specific regularization prior for VAR connectivity estimation. Experiments cover connectome recovery, reproduction of empirical dynamics, and downstream prediction, with AXON achieving a mean behavioral prediction correlation of 0.210, compared with 0.169 for FC and 0.197 for VAR.

Critical strengths
  1. AXON is simple and practically appealing, combining a learned attention backbone with each subject’s own timeseries through edge-specific regularization.

  2. The evaluation spans animal tract-tracing connectomes, human simulations, lifespan cohorts, and multiple downstream tasks, providing complementary evidence for the approach.

  3. The ablations on region embeddings, encoder layers, attention activations, and causal masks provide useful insights into the attention representations learned under the proposed setup.

Critical weaknesses
  1. Step-1 may attribute to attention a gain that stems from population-level information. The pooled attention is learned across the pretraining cohort and is largely subject-invariant: with the BOLD input ablated, it still correlates with the original at r = 0.71–0.96 (Fig. 10B). FC and partial correlation, by contrast, appear to be estimated from individual held-out recordings. The mismatch is most consequential in the Step-1 simulations, where all subjects share the same structural connectome and differ only in noise seed (App. B.1). Population-average FC and a tuned regularized partial correlation estimated on the pretraining data are needed for a like-for-like comparison.

  2. The contribution of the edge-specific prior is not isolated. Appendix D argues that because γ = 0 recovers VAR at the same λ₀, any difference “arises solely from the attention prior.” This does not follow. Under Eq. 5, each row of A sums to 1/P, so every entry is well below 1 and (A_ij + δ)^(−γ) is large. Unless A is rescaled, AXON therefore applies a much stronger overall penalty than VAR at the same λ₀, not merely a differently structured one. In addition, γ is tuned while VAR’s λ₀ does not appear to be. An independently tuned VAR, a uniform prior matched in penalty scale, and priors built from population FC or shuffled attention are needed to show that the edge-specific structure itself drives the gains.

  3. The directionality claims need qualification. Figure 2B suggests higher direction (r) on two atlases but slightly lower d-accuracy on all three relative to VAR, while the animal directionality results claimed in the abstract are not reported in the presented evaluations.

  4. The mechanistic conclusions extend beyond the evaluated setup. The ablations do not establish a fundamental limitation of self-attention, and unmasked patches from the same region remain available, weakening the argument that reconstruction must rely on other regions.

  5. The downstream advantage over VAR is limited, and key evaluation details remain unclear. The gains are not significant for age or in five of eight behavioral categories, and brain-state balanced accuracy is identical to VAR at the reported precision. Of particular concern, each condition has its own pretrained encoder (l. 340); if brain-state features are extracted with the encoder matching each scan’s condition, label information enters the features. The split protocol and the configurations of several near-chance BFM baselines (e.g., 0.17–0.20 on 8-class decoding, vs. 0.97 for FC) also require clarification.

Questions for authors
  1. Are Step-1 FC and partial correlation estimated from individual held-out recordings? How would population-average FC and tuned regularized partial correlation using the pretraining data compare?

  2. Is (A) rescaled before Equation 7? Could the authors compare AXON with independently tuned VAR, a penalty-scale-matched uniform prior, and priors based on population FC or shuffled attention?

  3. How are brain-state features extracted without knowing the test condition? Please also clarify the relationship between pretraining and the ten evaluation splits, family grouping, and the treatment of dependence across splits in statistical testing.

  4. Could the authors provide numerical results and uncertainty estimates for Figure 2B, and clarify whether animal directionality was evaluated? How are connections with zero ground-truth asymmetry handled in d-accuracy?

  5. Do the current ablations distinguish the effects of the reconstruction objective, patch resolution, and attention pooling on directionality? If not, could the conclusion be restricted to the evaluated setup?

  6. Which checkpoints, input representations, spatial/TR adaptations, feature readouts, and probe-tuning procedures were used for each BFM baseline?

  7. The animal direct (r) values in Figure 2A appear to differ from Table 9. Could the authors clarify whether the models, evaluation subsets, or aggregation procedures differ?

10 comments
AuthorsOpenReview ↗

Response to Q1 and W1 (1/8)

We thank the Reviewer for recognizing the simplicity and practicality of AXON, the breadth of our evaluation, and the insights from our ablations. We ran all the analyses the Reviewer requested and answered the 7 questions below in order. At each answer, we also discussed the corresponding "Critical Weakness" raised by the Reviewer.

Q1: Are Step-1 FC and partial correlation estimated from individual held-out recordings? How would population-average FC and tuned regularized partial correlation using the pretraining data compare?

W1: The pooled attention is learned across the pretraining cohort and is largely subject-invariant ... FC and partial correlation, by contrast, appear to be estimated from individual held-out recordings.

Key summary: "Step-1 may attribute to attention a gain that stems from population-level information."

Response for Q1 and W1:

The Reviewer is correct that FC and partial correlation in Fig. 2A were estimated from each held-out subject's recording, whereas the attention is learned from the pretraining subjects and, as shown in Fig. 10B (Fig. 11b in the revised PDF), is largely consistent across subjects. Fig. 2A alone therefore cannot determine whether the attention's advantage stems from its learned structure or simply from averaging across multiple subjects.

For a like-for-like comparison, we computed the population-average FC and the tuned regularized partial correlation as the Reviewer suggested (the latter with two standard estimators, Ledoit-Wolf shrinkage and graphical lasso) from the same 100 pretraining subjects whose attention was averaged, and scored each against the ground-truth connectome (values are DK-68 / Glasser-360 / Schaefer-400; higher is closer to the ground truth; † significantly lower than the attention, tests in Appendix C.5):

methodF1AUCdirect r
Attention0.695 / 0.699 / 0.7040.885 / 0.957 / 0.9590.800 / 0.847 / 0.869
population-average FC0.506† / 0.605† / 0.683†0.726† / 0.916† / 0.944†0.554† / 0.762† / 0.831†
partial correlation (Ledoit-Wolf)0.630† / 0.581† / 0.577†0.777† / 0.820† / 0.822†0.795 / 0.795† / 0.810†
partial correlation (graphical lasso)0.650† / 0.677† / 0.675†0.812† / 0.874† / 0.874†0.822 / 0.815† / 0.829†

Even FC averaged over 100 subjects did not match the attention: the attention outperformed it on F1, AUC and direct r at all three atlases (e.g., DK-68 F1 0.695 vs 0.506), and exceeded both partial correlations on F1 and AUC at all three atlases. The one exception was direct r at DK-68, where the attention was similar to the Ledoit-Wolf partial correlation (0.800 vs 0.795) and slightly, though not significantly, below the graphical lasso (0.822). Averaging over subjects alone therefore does not explain most of the attention's advantage. Revised PDF: Appendix C.5, Table 11.

AuthorsOpenReview ↗

Response to Q2 and W2, part 1 (2/8)

Q2: Is (A) rescaled before Equation 7? Could the authors compare AXON with independently tuned VAR, a penalty-scale-matched uniform prior, and priors based on population FC or shuffled attention?

W2: Unless A is rescaled, AXON therefore applies a much stronger overall penalty than VAR at the same λ₀, not merely a differently structured one. In addition, γ is tuned while VAR’s λ₀ does not appear to be.

Key summary: "AXON's gain over VAR may reflect a stronger overall penalty or unequal tuning rather than the edge-specific structure of the attention."

Response for Q2 and W2:

We thank for this critical comment. The Reviewer pointed out that AXON and VAR may differ not only in the edge-specific structure of the attention but also in the overall penalty strength and in tuning; it therefore needs to be shown that the edge-specific structure itself drives the gain once these two factors are matched. Below, we first clarify that the penalty strength was already matched in our implementation, which answers the Reviewer's question on rescaling; we then report the results of new analyses in which we tuned λ0 independently for each prior and, with both factors matched, varied only the edge-specific structure.

  1. Penalty strength: In our implementation, the penalty strength was already matched, because A is rescaled before Eq. 7: Specifically, A is scaled to a maximum of 1, and the penalties are normalized by their off-diagonal mean, so that every prior has the same mean penalty as VAR; VAR thus corresponds to the scale-matched uniform prior. At the same λ0, AXON therefore does not penalize more strongly than VAR. We now state this rescaling explicitly below Eq. 7 (Section 3.2) and in Appendix D.1, and have removed the claim that any difference "arises solely from the attention prior" in the revised manuscript.

  2. Tuning: In the original comparison, AXON and VAR shared the same λ0; with the mean penalty matched, the two estimators therefore differed only in how the penalty was distributed across connections. As the Reviewer requested, however, we further tuned λ0 independently for VAR and for every other prior (see the results in 3) below and Appendix D.1).

  3. Edge-specific structure: With penalty strength and tuning matched, we compared four priors that have the same average penalty but differ in which connections are penalized more or less: (i) VAR, which applies the same penalty to every connection; (ii) shuffled attention, which uses the attention values but assigns them to random connections (1,000 permutations under each of 2 null models); (iii) a population-average FC prior, which uses the group-average FC of the pretraining subjects in place of the attention; and (iv) AXON, which uses the attention itself. We evaluated each prior on the directed simulations (100 subjects), scoring both the detection of direct connections (direct r, F1) and their direction (direction r, d-accuracy). The table below lists values for DK-68 / Glasser-360 / Schaefer-400, with higher values indicating better performance; † and ‡ mark values significantly lower and higher than AXON, respectively (tests in Appendix D.5). For shuffled attention, we report the best of the 2,000 permutations for each metric, i.e., the most favorable case for this control:

methoddirect rF1direction rd-accuracy
AXON0.633 / 0.680 / 0.6950.535 / 0.498 / 0.4980.186 / 0.172 / 0.1630.594 / 0.611 / 0.614
VAR (tuned)0.447† / 0.339† / 0.333†0.461† / 0.299† / 0.295†0.194‡ / 0.084† / 0.080†0.609‡ / 0.606† / 0.608†
Shuffled attention prior0.494 / 0.401 / 0.3940.476 / 0.347 / 0.3390.151 / 0.078 / 0.0770.593 / 0.607 / 0.609
Population-average FC prior0.624† / 0.576† / 0.6950.534 / 0.425† / 0.502‡0.179† / 0.120† / 0.153†0.598‡ / 0.609† / 0.614
AuthorsOpenReview ↗

Response to Q2 and W2, part 2 (3/8)

Response for Q2 and W2 (continued)

  • Even with its own tuned λ0, VAR had lower direct r and F1 than AXON at all three atlases, and this pattern was largely preserved for the direction r and d-accuracy, except for DK-68. At this DK-68 parcellation, VAR achieved rather higher performance than AXON. We attribute this, at least in part, to the small number of regions: with 68 predictors per regression and about 1,200 time points, VAR is already well determined, leaving little room for an edge-specific prior. At Glasser-360 and Schaefer-400, where VAR must estimate roughly 130,000 and 160,000 coefficients, the problem becomes ill-posed, which is the regime AXON was designed for; here AXON roughly doubled VAR's direction r (0.172 vs 0.084 and 0.163 vs 0.080).
  • Shuffled attention, with the same values in random positions, fell below AXON on every metric, even when its highest value among the 2,000 shuffles was taken (1,000 from each of 2 shuffling schemes, node permutation and row shuffle), indicating that the gain arises largely from where the values are placed rather than from the values themselves. The population-average FC prior matched AXON's F1 at DK-68 (0.534 vs 0.535) and was slightly but significantly higher at Schaefer-400 (0.502 vs 0.498), but had lower direction r at all three atlases. Thus, AXON's gain over VAR in recovering direct connections does not stem from tuning, and most of it is lost when the attention values are shuffled. Against the population-average FC prior, the advantage was clear for direction r but not for F1, suggesting that while a population-level prior captures part of the benefit for detecting connections, the attention prior yields better direction estimates. Revised PDF: Section 3.2 and Appendix D.1 (penalty scaling); Appendix D.5, Table 15.
Reviewer_dzbkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed AXON, utilizing the self-attention of a masked-reconstruction brain foundation model (BFM) as a structural prior to estimate directed connectivity from fMRI. The claimed contributions are that the attention recovers structural connectivity (SC) in mouse (n=253), macaque (n=19), and simulated human data better than FC, and that AXON recovers their directionality and outperforms some baselines and existing BFMs in predicting demographics, behavior, and brain states on HCP-YA.

Critical strengths
  1. The validation coverage is extensive. It covers three species with independent ground truths, i.e., mouse and macaque tract-tracing and simulations driven by human structural connectomes across three atlases. The downstream benchmark compares 14 BFM variants, 4 deep learning models, FC, and VAR under an identical frozen-encoder linear probe with Holm–Bonferroni-corrected paired tests.

  2. The analyses of what the attention encodes are informative, including its emergence during pretraining (Fig. 10A), the region-embedding ablation (Fig. 10B), the layer-wise increase in symmetry (Tables 9–10), and the mask-by-activation ablation showing that neither causal masking nor the choice of activation yields directionality (Table 11), although all of these analyses are pushed back to the appendix.

Critical weaknesses
  1. The presentation of the proposed methods uses uncommon terms. For example, "direct connectivity" actually refers to the structural or tract-tracing connectome used as the ground truth (Table 1), and the "connectome interpreter" in the title is never formally defined, and its attention is largely reproduced when the model receives only the region embedding (r = 0.71–0.96, Fig. 10B). Meanwhile, this problem also happens in the motivation, where "mechanistic interpretability" (Sec. 2) is used for reading attention weights averaged over layers, heads, and time windows (Eq. 5), making the claim of revealing what the black box of BFMs not demonstrated. Besides, the description of Figure 1 is not present in the main text.

  2. The novelty of AXON is not clear. AXON is relevant to weighted ridge regression and structurally informed effective connectivity (EC) methods according to Eq. 7, which scales each VAR edge's ridge penalty by the attention. The authors do not discuss these methods, stating that existing EC methods "fundamentally rely on unconstrained temporal modeling of short timeseries" without any reference (Sec. 2), and plain VAR is the only EC baseline. Meanwhile, the attention in Eq. 7 was not replaced by a prior other than the BFM, such as group-averaged FC or partial correlation, so whether the gain over VAR needs the BFM is unknown.

  3. The validation of the contribution lacks thoroughness. The authors made a fundamental claim that AXON resolves directionality "with high accuracy in both mouse and macaque tract-tracing connectomes" (Abstract), yet Fig. 2A reports only the attention, FC, and partial correlation on the mouse and macaque data, whose ground truths are directed (L.321–323). The directionality of AXON is evaluated only on human simulations that has no hemodynamic model (Table 1, Fig. 2B, Eq. 8–9). Additionally, the paper omits computational complexity analysis.

Questions for authors

None

5 comments
AuthorsOpenReview ↗

Response to W1

We thank the Reviewer for recognizing the extensive validation across three species with independent ground truths and a controlled downstream benchmark, and the analyses of what the attention encodes. We addressed all weaknesses raised by the Reviewer point-by-point.

W1: The presentation of the proposed methods uses uncommon terms. For example, "direct connectivity" actually refers to the structural or tract-tracing connectome used as the ground truth (Table 1), and the "connectome interpreter" in the title is never formally defined, and its attention is largely reproduced when the model receives only the region embedding (r = 0.71-0.96, Fig. 10B). Meanwhile, this problem also happens in the motivation, where "mechanistic interpretability" (Sec. 2) is used for reading attention weights averaged over layers, heads, and time windows (Eq. 5), making the claim of revealing what the black box of BFMs not demonstrated. Besides, the description of Figure 1 is not present in the main text.

Key summary: "Key terms are uncommon or undefined, and reading the averaged attention does not demonstrate mechanistic interpretability, and Figure 1 is not described."

Response for W1:

We apologize that the original manuscript did not define these terms clearly. We addressed the four points of W1 as below.

(i) "Direct connectivity": As the Reviewer correctly noted, it refers to the connections of the structural or tract-tracing connectome used as the ground truth. In our perspective, however, those terms could be re-loaded based on the two scientific questions: whether two regions are functionally connected without 3rd-party nodes (thus 'Direct'; Step-1) and in which directionality they are connected ('Directed'; Step-2). The title "From Direct to Directed Connectivity" reflects this motivation, and moreover "Direct" follows the literature that separates direct connections from indirect ones through other regions (Honey et al., 2009; Smith et al., 2011).

(ii) "Connectome interpreter": The Reviewer pointed out that the term is not defined properly, and that the attention is largely reproduced from the region embedding alone (Fig. 10B; Fig. 11b in the revised PDF).

Regarding the region-embedding issue first, the fact that the attention is largely reproduced from the region embedding alone does not mean that it is arbitrary or unrelated to the fMRI data. The region embedding is the only part of the brain token that tells the encoder which region a token belongs to, and no connectivity is given as input. Masked reconstruction asks the model to predict the signal of a masked region from the other regions. To do so, the model must learn which regions predict which, and it encodes this information in each region's embedding. In this way, the signal itself gives each region a unique functional identity in its embedding. The attention read from this embedding (see the input ablation in Appendix C.2, Figure 11) is therefore still derived from the data, not computed arbitrarily.

The "connectome interpreter" in the title refers to exactly this: through the region embedding, the attention weights, originally a black box, could be translated into a biologically interpretable quantity, the brain connectome. The term therefore points to a broader direction, interpreting the internal processes of BFMs in terms of information flow through such connectome information.

(iii) "Mechanistic interpretability": We agree with the Reviewer that probing the attention averaged over layers, heads and time windows does not clearly disentangle their joint effects, therefore lacking the mechanistic interpretability. As we also recognized this issue, in the original manuscript, we have further analyzed each layer and head separately (p. 28, Figure 14 in the revised PDF). Our post-hoc analyses showed that single layers indeed could recover the ground-truth connectome (direct r of the best layer 0.38-0.84), and direct r increased in deeper layers in the macaque and at all three human atlases (Appendix C.3, Table 8). Taken together, what we call mechanistic interpretability is not merely probing the averaged attention alone, but decomposing their layer- and head-wise representation where the connectome information lies in the BFM, thereby elucidating the black box of BFMs.

(iv) Figure 1: The revised PDF adds a sentence in the Introduction that describes the pipeline summarized in Figure 1: "each subject's fMRI is passed through our BFM, and its attention provides the scaffold on which the VAR model fitted to the same recording yields a directed connectome". Revised PDF: Introduction (Figure 1); Appendix C.2-C.4.

References: Honey et al., PNAS 2009; Smith et al., NeuroImage 2011.

AuthorsOpenReview ↗

Response to W2, part 1

W2: The novelty of AXON is not clear. AXON is relevant to weighted ridge regression and structurally informed effective connectivity (EC) methods according to Eq. 7, which scales each VAR edge's ridge penalty by the attention. The authors do not discuss these methods, stating that existing EC methods "fundamentally rely on unconstrained temporal modeling of short timeseries" without any reference (Sec. 2), and plain VAR is the only EC baseline. Meanwhile, the attention in Eq. 7 was not replaced by a prior other than the BFM, such as group-averaged FC or partial correlation, so whether the gain over VAR needs the BFM is unknown.

Key summary: "AXON's novelty over structurally informed EC methods is unclear, plain VAR is the only EC baseline, and whether the gain over VAR needs the BFM is unknown."

Response for W2: We thank the Reviewer for this critical comment. As the Reviewer correctly noted, Eq. 7 is a weighted ridge regression that scales each VAR edge's ridge penalty by the attention. Because the penalty is set by connectivity information, AXON belongs to the same family as structurally informed EC methods, and its novelty does not lie in the estimator. These methods take their prior from diffusion MRI, for example dynamic causal modeling with tractography-based priors (Stephan et al., 2009) and the multivariate Ornstein-Uhlenbeck model constrained by the structural connectome (Gilson et al., 2016), so they require diffusion MRI. AXON instead takes its prior from the attention of a BFM pretrained on fMRI alone. Although no connectivity was given as input, this attention recovered the ground-truth connectome better than FC and partial correlation (Section 4.1). The revised Appendix D.1 now discusses AXON in relation to these methods, and the sentence the Reviewer quoted in Section 2 now states that most methods fit a temporal model to each subject's short time series alone, with references (Friston et al., 2003; Frässle et al., 2017).

We then addressed the two remaining points the Reviewer raised: how AXON compares with EC methods other than plain VAR, and whether the gain over VAR needs the BFM.

(i) Other EC methods: We used plain VAR as the main baseline because we developed AXON based on this algorithm, thus it reduces to VAR under a uniform prior. To further compare AXON with other classes of EC methods, we took representative 7 EC methods: acyclic graphical models (BOSS, Patel's tau), cyclic graphical models (FASK, CCD, pairwise LiNGAM) and dynamical-system methods (VAR, regression DCM). Each was run at its authors' default settings, and none was tuned with the ground truth. We scored each method in the directed simulations (100 subjects) for direct connections (direct r, F1) and directionality (direction r, d-accuracy). The table below lists values for DK-68 / Glasser-360 / Schaefer-400, with higher values indicating better performance; † and ‡ mark values significantly lower and higher than AXON, respectively (two-sided paired Wilcoxon, Holm-corrected over methods):

methoddirect rF1direction rd-accuracy
AXON0.704 / 0.706 / 0.7330.624 / 0.593 / 0.5830.170 / 0.180 / 0.1710.585 / 0.603 / 0.604
VAR0.613† / 0.473† / 0.467†0.567† / 0.438† / 0.433†0.174 / 0.080† / 0.078†0.605‡ / 0.606‡ / 0.609‡
regression DCM0.352† / 0.289† / 0.303†0.461† / 0.295† / 0.304†0.165 / 0.077† / 0.075†0.603‡ / 0.604 / 0.607‡
FASK0.428† / 0.403† / 0.408†0.358† / 0.140† / 0.123†-0.019† / -0.002† / 0.000†0.500† / 0.500† / 0.500†
CCD0.538† / 0.504† / 0.513†0.381† / 0.188† / 0.176†0.011† / 0.041† / 0.033†0.505† / 0.506† / 0.505†
BOSS0.527† / 0.536† / 0.555†0.412† / 0.235† / 0.226†-0.058† / 0.009† / 0.036†0.487† / 0.504† / 0.507†
pairwise LiNGAM0.174† / 0.276† / 0.292†0.396† / 0.284† / 0.278†-0.002† / -0.003† / -0.002†0.501† / 0.497† / 0.499†
Patel's tau0.518† / 0.471† / 0.478†0.377† / 0.175† / 0.161†0.000† / -0.002† / 0.000†0.500† / 0.499† / 0.500†

AXON had significantly higher direct r and F1 than every other method at all three atlases (e.g., Schaefer-400 direct r 0.733, vs 0.555 for the next best, BOSS). The graphical models were at or near chance on directionality, and apart from AXON only the dynamical-system methods (VAR, regression DCM) recovered directionality. Their direction r was similar to AXON's at DK-68 and significantly lower at Glasser-360 and Schaefer-400, while their d-accuracy was significantly higher in 5 of the 6 method-atlas pairs, by at most 0.020.

AuthorsOpenReview ↗

Response to W2, part 2

Response for W2 (continued)

Regarding this lower d-accuracy, we would like to reiterate the sentence that the attention prior carries no direction of its own (Section 4.1), thus it does not add any directional information to the VAR fit. The opposite scenario could occur, however (i.e., the prior may confuse the estimated direction of some connections). It is because, with the mean penalty matched to VAR's, a true connection with low attention is penalized more heavily than in VAR, and equally in both directions (i→j and j→i). Both weights then shrink toward zero, so the difference between them (from which d-accuracy is derived) becomes small and its sign can be easily flipped by noise. This may be the likely source of the small loss in d-accuracy, which was largest at DK-68, where VAR is already well determined.

(ii) Priors other than the BFM: In the submission, we showed that the attention itself recovered direct connections better than FC and partial correlation (direct r and AUC; Fig. 2A). We did not, however, test whether it also performs better than these as a prior combined with VAR, so, as the Reviewer pointed out, whether the gain over VAR needs the BFM was unknown. As the Reviewer suggested, we therefore replaced the attention in Eq. 7 with group-averaged FC and partial correlation (computed with Ledoit-Wolf and with the graphical lasso). Here the ridge penalty strength was selected for each prior without the ground truth, so the values of AXON and VAR differ from those in (i). The table below lists values for DK-68 / Glasser-360 / Schaefer-400 in the same directed simulations, with higher values indicating better performance; † and ‡ mark values significantly lower and higher than AXON, respectively (two-sided paired Wilcoxon, Holm-corrected over priors):

priordirect rF1direction rd-accuracy
AXON0.633 / 0.680 / 0.6950.535 / 0.498 / 0.4980.186 / 0.172 / 0.1630.594 / 0.611 / 0.614
VAR0.447† / 0.339† / 0.333†0.461† / 0.299† / 0.295†0.194‡ / 0.084† / 0.080†0.609‡ / 0.606† / 0.608†
group-averaged FC0.624† / 0.576† / 0.6950.534 / 0.425† / 0.502‡0.179† / 0.120† / 0.153†0.598‡ / 0.609† / 0.614
partial correlation (Ledoit-Wolf)0.499† / 0.672† / 0.711‡0.473† / 0.430† / 0.463†0.183 / 0.178‡ / 0.217‡0.606‡ / 0.613‡ / 0.618‡
partial correlation (graphical lasso)0.496† / 0.670† / 0.711‡0.472† / 0.428† / 0.459†0.184 / 0.176‡ / 0.214‡0.606‡ / 0.613‡ / 0.618‡
  • Priors computed without the BFM also improved on VAR. At Schaefer-400, for example, the group-averaged FC prior had the same direct r as AXON (0.695), but it had significantly lower direction r at all three atlases.
  • The two partial correlation priors, conversely, had significantly higher direction r at Glasser-360 and Schaefer-400 (Schaefer-400 0.217 / 0.214 vs 0.163), but significantly lower F1 at all three atlases (Glasser-360 0.430 / 0.428 vs 0.498).

No prior therefore matched AXON on both direct connections and directionality at any atlas. Taken together, part of the gain over VAR can be obtained without the BFM, but only the attention prior improved both. Because the partial correlation priors improved directionality at the two larger atlases, how to build a prior that improves both direct connections and directionality is left for future work. Revised PDF: Section 2 (sentence with references); Appendix D.1 (structurally informed EC methods); Appendix D.5, Table 15; Appendix D.6, Tables 16 and 17, Figure 18.

References: Friston et al., NeuroImage 2003; Frässle et al., NeuroImage 2017; Stephan et al., NeuroImage 2009; Gilson et al., PLoS Comput. Biol. 2016.

Reviewer_24UrOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether self-attention in a masked-reconstruction brain foundation model captures meaningful brain connectivity. The authors find that attention recovers aspects of the connectivity backbone but does not reliably recover directionality. They propose AXON, which uses attention-derived connectivity to define edge-specific regularization in a vector autoregressive model fitted to individual fMRI time series. Evaluation includes animal tract-tracing references, simulations with known connectivity, reproduction of human resting-state dynamics, and downstream prediction of demographics, behavioral traits, and brain states.

Critical strengths
  1. The authors test whether attention captures the direction of brain connections instead of assuming that it does. Their simulations show that attention alone cannot reliably recover direction. Analyses of different layers and changes to the embeddings help explain what the model learns. This adds value beyond prediction accuracy.

  2. AXON uses attention to guide regularization, combining patterns learned across subjects with each person’s time-series data. The authors test the method using animal connectivity data, simulations with known connections, brain dynamics, and behavioral prediction. These different tests strengthen the evidence. The increase in average behavioral prediction correlation from 0.197 for VAR to 0.210 for AXON is promising.

Critical weaknesses
  1. ection 3.3 states that separate models are pretrained for the different task conditions, while the downstream evaluation includes classification of those same conditions. It is therefore essential to explain how an encoder is selected for a test scan whose condition is unknown. Selecting it using the true condition would introduce target information into feature extraction; using a shared encoder or evaluating every scan through all encoders would avoid this issue. Separately, the relationship between the stated pretraining split and the ten downstream splits is unclear, as is whether related HCP participants are kept together. These are questions about the protocol, not established evidence of leakage, but they need resolution before the predictive results can be fully assessed.

  2. Appendix D fixes the base regularization strength and describes VAR as AXON with gamma equal to zero. Holding the base value fixed does not ensure an equally well-tuned VAR baseline, because the attention transformation changes both the distribution and overall magnitude of the edge penalties. Independently tuned VAR and matched alternative priors are needed to establish that the gain comes from meaningful attention structure. Furthermore, structural tract tracing is not a direct ground truth for physiological EC, and the human directionality evidence comes from a particular simulated dynamical system. These results support the proposed representation, but do not establish general recovery of human causal influence.

Questions for authors
  1. For eight-class brain-state decoding, how is the encoder selected without knowing the test condition? If condition-specific encoders are used, is every scan processed through all of them, or is another label-independent procedure used?

  2. Across the ten downstream splits, is pretraining repeated within each training partition, or are the downstream partitions restricted to a fixed pretraining/held-out division? Are HCP family members grouped, and are all scans from the same participant kept in the same partition?

  3. Does AXON retain its advantage over a VAR baseline with independently selected regularization strength and lag order? How does it compare with FC-based or shuffled-attention penalties matched for their overall regularization scale?

  4. How sensitive is direction recovery to regional hemodynamic delays, observation noise, and a different data-generating model? If these conditions have not been evaluated, can the claims be explicitly limited to the tested simulation setting?

9 comments
AuthorsOpenReview ↗

Response to Q1 and W1

We thank the Reviewer for recognizing that we tested whether the attention captures the direction of brain connections instead of assuming it, and that the animal, simulation, dynamics and behavioral tests together strengthen the evidence. We ran the analyses the Reviewer requested and answered the questions below in order. At each answer, we also discussed the corresponding "Critical Weakness" raised by the Reviewer.

Q1: For eight-class brain-state decoding, how is the encoder selected without knowing the test condition? If condition-specific encoders are used, is every scan processed through all of them, or is another label-independent procedure used?

W1: Section 3.3 states that separate models are pretrained for the different task conditions, while the downstream evaluation includes classification of those same conditions. It is therefore essential to explain how an encoder is selected for a test scan whose condition is unknown.

Key summary: "Brain-state features may leak the condition label."

Response for Q1 and W1:

We apologize that the original manuscript did not precisely explain the model training procedure. While the sentence in Section 3.3 ("Each condition was pretrained as an independent model") indeed described the default setting for most analyses, brain-state decoding was the exception: to avoid exactly the issue the Reviewer raises, it used one of the options the Reviewer suggested, a single shared brain foundation model (hereafter encoder), pretrained once on REST and all seven tasks jointly. In AXON, each test scan is passed through this same encoder to obtain its own attention map, which then serves as the prior for that scan's VAR fit. Because every scan is processed by the same encoder regardless of its condition, the condition label of a test scan cannot enter its features.

All other analyses used the per-condition encoders, and none of them predicts the condition label; for example, the task EC analysis asks whether the EC estimated during each task predicts performance on that task. Revised PDF: Section 3.3 and Appendix E.4.

AuthorsOpenReview ↗

Response to Q2 and W1

Q2: Across the ten downstream splits, is pretraining repeated within each training partition, or are the downstream partitions restricted to a fixed pretraining/held-out division? Are HCP family members grouped, and are all scans from the same participant kept in the same partition?

W1: Separately, the relationship between the stated pretraining split and the ten downstream splits is unclear, as is whether related HCP participants are kept together.

Key summary: "The split protocol seems unclear."

Response for Q2 and W1:

We addressed the three points of Q2 as below.

(i) Pretraining and the 10 downstream splits: Pretraining was not repeated within each split; the downstream partitions were restricted to a fixed pretraining/held-out division. The encoder was pretrained once on approximately 80% of the subjects, and the downstream probes were trained on the same 80%. In each of the 10 splits, the remaining 20% was split equally into validation and test sets, so that no test subject was included in either pretraining or downstream training.

(ii) Family grouping: As the Reviewer raised, the original splits did not account for family structure. Because family members, particularly twins, share genetic and environmental factors, placing them in both the training and test sets can inflate prediction accuracy. To rule out this leakage issue, we added an evaluation with family-aware splits. The table reports the mean r (Pearson correlation between predicted and observed scores of the test subjects, averaged over the 58 behavioral measures and the 10 splits; for the paper splits, this is the "All" column of Table 2); under both split schemes, every method was significantly lower than AXON (corrected resampled t-test of Nadeau & Bengio (2003) with Holm correction across methods, p ≤ 0.019):

methodpaper splitsfamily-aware splits
ConnFC0.1690.147
VAR0.1970.172
DLBrainNetCNN0.1230.104
BrainGNN0.0620.046
BolT0.0910.077
BNT0.0650.057
BFMSwiFT0.1180.099
BrainLM0.0310.035
BrainMass0.1050.086
Brain-JEPA0.0570.058
CSM0.0690.054
BrainGFM0.1120.096
NeuroSTORM0.0920.081
BrainHarmonix0.1130.107
CortexMAE-P0.1080.093
CortexMAE-F0.0930.077
CortexMAE-V0.1160.114
Brain-Semantoks0.1530.129
Omni-fMRI0.0720.061
Brain-DiT0.0520.047
OursGAP0.0920.085
ATTN0.1380.118
AXON0.2100.186

GAP and ATTN are the pooled embedding and the attention of our BFM, respectively. Family-aware splits lowered most methods by about 0.02, and AXON remained above every method.

(iii) All scans of the same participant: Yes. The splits were made by subject, so all scans of a participant fall in only one of the training, validation and test sets. Revised PDF: Section 3.4.

AuthorsOpenReview ↗

Response to Q3 and W2, part 1

Q3: Does AXON retain its advantage over a VAR baseline with independently selected regularization strength and lag order? How does it compare with FC-based or shuffled-attention penalties matched for their overall regularization scale?

W2: Holding the base value fixed does not ensure an equally well-tuned VAR baseline, because the attention transformation changes both the distribution and overall magnitude of the edge penalties. Independently tuned VAR and matched alternative priors are needed to establish that the gain comes from meaningful attention structure.

Key summary: "AXON's gain over VAR may reflect a different overall penalty or unequal tuning rather than the attention structure."

Response for Q3 and W2:

We thank for this critical comment. The Reviewer pointed out that AXON and VAR may differ not only in the edge-specific structure of the attention but also in the overall penalty magnitude and in tuning; it therefore needs to be shown that the edge-specific structure itself drives the gain once these two factors are matched. Below, we first clarify that the penalty magnitude was already matched in our implementation; we then report the results of new analyses in which we selected the regularization strength and the lag order independently for each method and, with both factors matched, varied only the edge-specific structure.

  1. Penalty magnitude: In our implementation, the overall magnitude was already matched, because A is rescaled before Eq. 7: Specifically, A is scaled to a maximum of 1, and the penalties are normalized by their off-diagonal mean, so that every prior has the same mean penalty as VAR; VAR thus corresponds to the scale-matched uniform prior. At the same base value, the attention therefore changes only the distribution of the penalties, not their overall magnitude. We now state this rescaling explicitly below Eq. 7 (Section 3.2) and in Appendix D.1.

  2. Tuning: In the original comparison, AXON and VAR shared the same base regularization strength λ0; with the mean penalty matched, the two estimators therefore differed only in how the penalty was distributed across connections. As the Reviewer requested, however, we further selected λ0 and the lag order (1 or 2) independently for each method by one-step-ahead prediction error on held-out time points, without the ground truth (Appendix D.1; Bergmeir et al., 2018). Both methods chose lag 1 at all three atlases. With lag 2 imposed, the pattern was the same (DK-68 / Glasser-360 / Schaefer-400; higher is better):

lag 2 imposeddirect rF1direction rd-accuracy
VAR0.416 / 0.341 / 0.3360.453 / 0.308 / 0.3050.158 / 0.071 / 0.0700.596 / 0.597 / 0.602
AXON0.562 / 0.630 / 0.6380.497 / 0.445 / 0.4480.144 / 0.140 / 0.1350.580 / 0.601 / 0.605

AXON had higher direct r and F1 at all three atlases, and higher direction r and d-accuracy at Glasser-360 and Schaefer-400, as with lag 1.

Reference: Bergmeir, Hyndman & Koo, Comput. Stat. Data Anal. 2018.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and revisions

Dear Area Chair,

We thank the Area Chair and the three Reviewers for their careful reading and constructive comments.

In this paper, we examined what BFMs represent internally by analyzing the self-attention that couples brain regions. Our main findings were threefold: (i) the attention of a BFM pretrained on fMRI alone, with no connectivity given at the input, recovers the brain's direct connections but not their direction; (ii) AXON, which combines this attention with each subject's own time series, further recovers directionality; and (iii) the resulting connectome predicts phenotypes better than existing BFMs. During the discussion, the Reviewers raised a concern about each finding: (i) whether the advantage of the attention arises from its learned structure, (ii) whether the gain of AXON is due to the attention and whether its directionality is reliable, and (iii) whether the predictive evaluation was fair. We performed the requested analyses to address each concern, and the results further supported all three findings.

(i) The attention recovers the brain's direct connections. Although no connectivity was given at the input, the attention recovered the structural backbone of the connectome more accurately than FC and partial correlation across three species. Reviewer bsvH noted, however, that this advantage might simply reflect averaging over subjects, because the attention is learned across the pretraining subjects. The advantage was largely maintained against population-average FC and regularized partial correlation computed from the same pretraining subjects, indicating that averaging over subjects alone does not explain most of it (Appendix C.5). Notably, the attention did not carry the direction of connections, a result that was consistent across 8 encoders with different training objectives and patch lengths and thus did not depend on these design choices (bsvH; Appendix C.4).

(ii) AXON recovers directionality from the attention-derived backbone. AXON therefore combines the backbone provided by the attention with each subject's own time series to estimate direction. Compared with 7 representative EC methods based on different principles, AXON recovered direct connections most accurately at all three atlases and showed the highest direction r at Glasser-360 and Schaefer-400 (dzbk; Appendix D.6). The Reviewers questioned whether this advantage was due to the attention and whether the directionality was reliable. AXON's gain over VAR in direct connections was preserved when VAR was tuned independently, but was largely lost when the attention values were shuffled, indicating that the gain arises largely from the edge-specific structure of the attention (24Ur, bsvH; Appendix D.5). This gain also held under observation noise, regional hemodynamic delays and two additional generative models (24Ur; Appendix D.7). Moreover, the directionality estimated by AXON was supported not only by the simulations but also by empirical data from both animals and humans. In the mouse and the macaque, AXON's directed connections were consistent with laminar pathways (dzbk, bsvH; Appendix D.4), and in humans, the whole-brain model driven by AXON reproduced the regional time-lag structure of empirical fMRI more closely than that driven by VAR (Section 4.3).

(iii) The resulting connectome predicts phenotypes better than existing BFMs. In the mean accuracy over 58 behavioral measures, AXON outperformed 14 BFMs, four deep-learning models, FC and VAR (mean r 0.210). Regarding the fairness of this evaluation, we clarified that brain-state decoding used a single shared encoder for all scans, so the condition label cannot enter the features (24Ur, bsvH; Section 3.3). AXON also remained significantly above every method under the corrected resampled t-test, which accounts for the dependence between repeated splits, and with family-aware splits (24Ur, bsvH; Table 2 and our response to 24Ur Q2).

Taken together, the analyses performed during the discussion strengthened each of the three findings: the direct connections recovered by the attention, the directionality added by AXON, and the predictive power of the resulting connectome. These results support our central claim that the predictive potential of a BFM is better realized through an interpretable connectome than through black-box features. We thank the Area Chair again for overseeing the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.