- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes BRIO for cross-site generalization of resting-state fMRI classifiers. Its central motivation is that agreement between source-site representations does not necessarily imply stability under re-estimation of functional connectivity from the same scan. BRIO generates FC re-estimates through temporal block resampling, maps them into consistently indexed connectome factors, and measures changes in each factor’s predictive contribution. Predictive relevance and task-calibrated re-estimation support jointly determine factor weights and the strength of class-conditional cross-site alignment. Leave-one-site-out experiments cover four datasets and an additional parcellation on ABIDE.
Critical strengths
-
FC is estimated from finite, temporally dependent observations, yet cross-site alignment often treats it as fixed. BRIO evaluates changes in predictive contributions rather than raw edge variability alone, and combines stability with task relevance rather than assuming that every stable feature is useful. The shared factor representation makes these comparisons well defined within the model.
-
The paper evaluates unseen-site performance across ASD, depression, and ADHD datasets, with an additional atlas on ABIDE. The controlled alignment-strength sweep, factor-retention analysis, block-length sensitivity, and reduced-scan experiments help test the proposed mechanism. In particular, the comparison at similar site discrepancy provides evidence that the method offers more than simply stronger alignment.
Critical weaknesses
-
The ablation studies show that BRIO’s components are useful. However, it remains unclear whether the full weighting method works better than simpler approaches, such as using re-estimated FC graphs for data augmentation or encouraging consistent predictions across them. These methods should be compared using the same backbone and computational budget. The theoretical results explain the weights but do not guarantee better performance at unseen sites.
-
Table 1 shows how performance varies across test sites, but not how consistently BRIO improves over the strongest baseline. The accuracy gains over CORE range from 1.17 to 2.27 percentage points. Please report paired comparisons on the same test sites, results from multiple random seeds, and details of model selection. The results should distinguish variation across sites from variation across random seeds.
Questions for authors
- How does BRIO compare with the same backbone trained using block-resampled FC augmentation or a prediction-consistency loss, with matched re-estimation and tuning budgets? This would help determine whether selective alignment is necessary for the reported gains.
- Can the authors provide paired per-site improvements over CORE and the strongest BRIO ablation, including uncertainty across repeated seeds? Please also specify the number of seeds and how they are aggregated in Table 1.
- How are the two validation sites selected in each LOSO fold, and what hyperparameter-search and checkpoint-selection procedures are used for BRIO and its baselines? Please clarify whether the same validation protocol and comparable tuning budgets are applied throughout.
- Equation 16 assigns higher support when the reference scale increases, which can occur through larger within-class variability as well as stronger class separation. Can the authors explain or empirically test when this normalization distinguishes genuinely stable predictive evidence from noisy, heterogeneous factors, particularly at small sites?
- Table 5 reports the default block as exactly 12.0 s on ABIDE, SRPBS and ABCD. Yet ABIDE and SRPBS have heterogeneous TRs across sites, and the block length is defined in time points at the fold level. A single duration in seconds is therefore ill-defined, and the paper’s TR-transfer argument does not hold as stated. How is the block length handled across sites with different TRs? Why is a single duration in seconds reported?
- Several values look unusually regular. In Table 4 (ID results), almost every mean and SD ends in the digit 5. BRIO’s SDs are smaller than every baseline’s in nearly every column. Could you confirm how they were computed and release the per-run logs and code?
14 comments
Response to Reviewer j4pB (2/10)
As shown in the above table, the full weighting outperforms the simpler approaches under a matched backbone and budget. UA+Aug+Cons, the strongest variant, uses the same number of re-estimates as BRIO and a longer training time of 0.197 hours on ABIDE, compared with 0.162 hours for BRIO, and BRIO exceeds it by 2.4 to 3.2 points in every column. ERM+Aug and ERM+Cons train slightly faster than BRIO and improve ERM by only 0.4 to 1.1 points, and ERM+Aug passes the most re-estimates through the network without a larger gain. Across the methods, performance grows with the use of alignment and qualification and does not follow the number of re-estimates or the training time: on all three datasets and both metrics, both training-signal variants outperform ERM, UA outperforms both of them, UA+Aug+Cons outperforms UA, and BRIO performs best.
To test whether this improvement holds across target sites, we computed the paired gain of BRIO over UA+Aug+Cons on the same target sites and seeds, with seeds paired across methods. For each dataset and metric, the table below reports the mean over target sites of the seed-averaged gain in percentage points and an unadjusted 95% confidence interval (CI) from a two-level bootstrap that resamples target sites and then seeds within each site with 10,000 replicates [2]. It also reports the number of target sites with a positive gain and the -value of an exact Wilcoxon signed-rank test over target sites, Holm-corrected across the six dataset–metric tests [3, 4]. SD${\mathrm{site}}$ is the SD of the seed-averaged gain across target sites, and SD${\mathrm{seed}}$ is the SD of the gain across seeds within a target site, averaged over target sites.
| Setting | Metric | Gain | 95% CI | Sites > 0 | SD$_{\mathrm{site}}$ | SD$_{\mathrm{seed}}$ | |
|---|---|---|---|---|---|---|---|
| ABIDE | ACC | 2.41 | [1.52, 3.36] | 19/20 | <0.001 | 1.98 | 1.36 |
| ABIDE | AUC | 2.54 | [1.43, 3.67] | 16/20 | 0.002 | 2.57 | 1.27 |
| REST-meta-MDD | ACC | 3.15 | [2.32, 4.01] | 24/25 | <0.001 | 2.14 | 0.92 |
| REST-meta-MDD | AUC | 2.96 | [2.01, 3.90] | 23/25 | <0.001 | 2.36 | 1.14 |
| SRPBS | ACC | 2.40 | [0.71, 3.93] | 9/11 | 0.019 | 2.60 | 1.28 |
| SRPBS | AUC | 2.50 | [1.06, 3.98] | 9/11 | 0.014 | 2.54 | 1.31 |
As shown in the above table, the gain of BRIO over UA+Aug+Cons is positive on 80% to 96% of the target sites, its CI excludes zero in all six dataset–metric settings, and it remains significant after Holm correction with . The gain varies more across target sites than across seeds in every setting, which indicates that site differences account for most of the remaining variation. We have added these results to Appendix J.7 (Tables 8 and 9) of our manuscript.
The theoretical results characterize what the weights estimate and how they respond to re-estimation changes. The performance at unseen sites is established empirically, through the LOSO experiments, where each target site is excluded from training and model selection, and through the paired comparisons above. The evidence for unseen-site performance is empirical. It comes from the LOSO experiments, where each target site is excluded from training and model selection, and from the paired comparisons above.
Response to Reviewer j4pB (3/10)
W2: Table 1 shows how performance varies across test sites, but not how consistently BRIO improves over the strongest baseline. The accuracy gains over CORE range from 1.17 to 2.27 percentage points. Please report paired comparisons on the same test sites, results from multiple random seeds, and details of model selection. The results should distinguish variation across sites from variation across random seeds.
A2: Thanks for your comment. To show how consistently BRIO improves over the strongest baseline, we computed paired comparisons with CORE on the same target sites and seeds, separated the variation across sites from that across seeds, and describe the model-selection protocol.
(1) Paired comparison with CORE. Table 1 of the paper reports one run per target site with a single seed, and its ± is the SD across target sites. For this response, we ran BRIO and CORE on all five settings with three new seeds shared by all methods. The LOSO splits and the two validation sites of each fold are shared by all methods and seeds, so all results are paired by target site and seed. For each setting and metric, we computed the gain of BRIO over CORE for every target site and seed. The table below reports the mean over target sites of the seed-averaged gain in percentage points and an unadjusted 95% CI from a two-level bootstrap that resamples target sites and then seeds within each site with 10,000 replicates [2]. It also reports the number of target sites with a positive gain and the -value of a two-sided exact Wilcoxon signed-rank test over target sites [3], Holm-corrected across the ten settings [4]. Following [5], we separate the two sources of variation. SD${\mathrm{site}}$ is the SD of the seed-averaged gain across target sites, and SD${\mathrm{seed}}$ is the SD of the gain across seeds within a target site, averaged over target sites.
| Setting | Metric | Gain | 95% CI | Sites > 0 | SD$_{\mathrm{site}}$ | SD$_{\mathrm{seed}}$ | |
|---|---|---|---|---|---|---|---|
| ABIDE | ACC | 1.29 | [−0.16, 2.72] | 13/20 | 0.46 | 3.19 | 1.76 |
| ABIDE | AUC | 1.89 | [0.29, 3.43] | 14/20 | 0.24 | 3.61 | 1.30 |
| ABIDE (CC200) | ACC | 2.22 | [1.11, 3.35] | 15/20 | 0.011 | 2.47 | 1.27 |
| ABIDE (CC200) | AUC | 2.52 | [1.24, 3.79] | 15/20 | 0.010 | 2.92 | 1.32 |
| REST-meta-MDD | ACC | 1.52 | [0.19, 2.91] | 17/25 | 0.24 | 3.48 | 1.54 |
| REST-meta-MDD | AUC | 1.35 | [−0.41, 3.38] | 15/25 | 1.00 | 4.94 | 1.09 |
| SRPBS | ACC | 1.08 | [−1.97, 4.21] | 6/11 | 1.00 | 5.24 | 1.86 |
| SRPBS | AUC | 1.25 | [−0.64, 3.49] | 6/11 | 1.00 | 3.66 | 1.13 |
| ABCD (ADHD-task) | ACC | 1.35 | [−1.05, 3.82] | 6/10 | 1.00 | 4.06 | 1.18 |
| ABCD (ADHD-task) | AUC | 1.97 | [−1.01, 5.25] | 6/10 | 1.00 | 5.39 | 1.40 |
As shown in the above table, the mean gain over CORE is positive in all ten settings and ranges from 1.1 to 2.5 points, in line with the single-seed gains in Table 1. Its CI excludes zero in four settings, and both metrics on ABIDE (CC200) remain significant after Holm correction. The variation of the gain comes mainly from the target sites. SD$_{\mathrm{site}}$ is 1.8 to 4.5 times SD$_{\mathrm{seed}}$ in every setting, and the intervals are widest on SRPBS and ABCD, which have 11 and 10 target sites. When the 66 target sites of the four AAL116 settings are pooled, the gain over CORE is 1.35 points in ACC with a 95% CI of [0.43, 2.27], positive on 42 sites and with an unadjusted Wilcoxon , and 1.59 points in AUC with a 95% CI of [0.56, 2.63], positive on 41 sites and with . The single-seed results in Table 1 also show a lower across-site SD for BRIO than for CORE in all ten settings, by a factor of 1.3 to 2.2. We have added these results to Appendix J.8 (Table 16) of our manuscript.
Response to Reviewer j4pB (1/10)
We are truly grateful for the time you have taken to review our paper and your insightful review. Here we address your comments in the following:
A1: Thanks for your comment. To test whether the full weighting works better than simpler approaches under the same backbone and computational budget, we compared BRIO with five variants that use the same functional connectivity (FC) re-estimates for data augmentation, prediction consistency, or both on top of unqualified alignment, on the same network, and recorded the number of re-estimates and the training time of every method. In BRIO, the re-estimates are used only to compute the qualification. At each refresh, the calibration subjects are passed through the network with their re-estimates without gradients, and training mini-batches contain only full-scan FC. The variants are constructed as follows:
All variants share the BRIO network, edge support, fold-level block rule, re-estimate generator, warm-up, optimizer, number of training epochs, validation sites and seeds. In each fold and seed, the hyperparameters shared with BRIO take the values selected for BRIO, and each variant tunes its own parameter over three values with the same selection metric, with k∈{1,2,4} for ERM+Aug and λcons∈{0.3,1,3} for ERM+Cons and UA+Aug+Cons. A run of BRIO generates R=NcalKnrefresh re-estimates, where Ncal is the number of calibration subjects and nrefresh is the number of refreshes. We ran all methods, including BRIO, on ABIDE, REST-meta-MDD and SRPBS with three new seeds shared by all methods. The table below reports the leave-one-site-out (LOSO) area under the ROC curve (AUC) and accuracy (ACC) in %. For each seed, we average the metric over all target sites of a dataset, and each entry is the mean ± standard deviation (SD) of the three seed-level values. Re-est. is the number of re-estimates passed through the network per training run on ABIDE, and Wall-h is the wall-clock training time in hours of one configuration and seed on ABIDE. Both are averaged over folds, with eight runs sharing two GPUs.