acceptodds
Under review as a conference paper at ICLR 2027

Inter-Regional Structural Self-Supervised Learning for Brain MRI: Individual and Population Formulations

Abstract

Brain structure is characterized not only by the properties of individual anatomical regions but also by the relationships among them. These inter-regional relationships constitute a fundamental aspect of brain organization. However, existing self-supervised learning (SSL) methods for structural brain MRI predominantly rely on voxel reconstruction or instance-level consistency, without explicitly modeling such inter-regional structural organization. We introduce Inter-Regional Structural Self-Supervised Learning (IRS-SSL), a framework that establishes inter-regional structural organization as its primary self-supervisory principle and inductive bias. Rather than incorporating inter-regional relationships as an auxiliary constraint on a conventional SSL objective, IRS-SSL derives its primary supervisory signal directly from these relationships. We instantiate this principle through two formulations. The individual formulation, IRS-SSL-Ind, characterizes pairwise similarities between regional structural profiles within each subject. The population formulation, IRS-SSL-Pop, captures coordinated variation between brain regions across subjects. The two formulations encode distinct relational information: IRS-SSL-Ind describes how regions relate within a particular brain, whereas IRS-SSL-Pop captures structural regularities shared across individuals. We systematically evaluate both formulations across diverse downstream neuroimaging tasks and compare them with state-of-the-art SSL frameworks. Both formulations demonstrate the value of inter-regional structural organization as a source of self-supervision. Notably, IRS-SSL-Ind yields strong and consistent downstream performance. These findings establish inter-regional structural organization as an effective inductive bias for SSL and provide new insights into the relative value of individual- and population-derived relationships for learning representations from structural brain MRI.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Self Supervised Learning, Magnetic Resonance Imaging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nZQ8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IRS-SSL, a self-supervised learning framework for structural brain MRI that learns representations by modeling inter-regional structural similarities at both individual and population levels.

Critical strengths

The paper is well structured and clearly written, with extensive experiments across multiple downstream tasks. Incorporating inter-regional structural similarities into self-supervised learning is an interesting direction that goes beyond conventional voxel-level or instance-level learning.

Critical weaknesses
  1. The proposed method relies on a small set of T1 intensity statistics to construct inter-regional structural relationships. Are these features sufficient to capture meaningful anatomical information? Have the authors considered comparing them with more conventional morphological features, such as cortical thickness, regional volume, or surface area?

  2. The individual formulation consistently achieves better performance, but the underlying reason is not well explained. Is this because individual-level relationships provide more informative supervision, or because the two formulations differ in the amount and type of structural information they preserve? A more detailed analysis would help clarify this finding.

  3. The current comparison between HO and Desikan-Killiany is rather limited. First, the authors could evaluate different parcellation resolutions, such as varying the number of ROIs within the same atlas family. Second, a random parcellation baseline would be particularly informative. If randomly defined brain regions achieve comparable performance, it would raise questions about whether anatomically meaningful structural relationships are actually responsible for the improvements. The existing random region pairing experiment does not fully address this concern.

Questions for authors

See Weakness

8 comments
AuthorsOpenReview ↗

Comment for Reviewer nZQ8 (1/3)

“Critical Strengths: Incorporating inter-regional structural similarities into self-supervised learning is an interesting direction that goes beyond conventional voxel-level or instance-level learning.”

We thank the reviewer for the positive comment. Indeed our study conceptualizes inter-regional structural organization as the primary self-supervision principle, going beyond conventional voxel-level or instance-level learning.

W1: The proposed method relies on a small set of T1 intensity statistics to construct inter-regional structural relationships. Are these features sufficient to capture meaningful anatomical information? Have the authors considered comparing them with more conventional morphological features, such as cortical thickness, regional volume, or surface area? Are these features sufficient to capture meaningful anatomical information?

We thank the reviewer for this comment. Our set of T1-intensity statistics provide a compact, computationally tractable description of regional intensity patterns, suitable for large-scale pre-training. IRS-SSL uses these statistics to derive inter-regional relationships. In addition to the strong downstream performance and radiologist evaluation reported in the paper, here we additionally conducted a cross-session individual-identification experiment (see below). This result indicates that the descriptors contain reproducible, subject-specific structural information and supports their utility as a self-supervisory signal. We do not claim that they provide a complete characterization of brain morphology or replace conventional morphological features (Note that conventional morphological features could be impractical for our large-scale pretraining. See below.)

Individual identification experiment. To provide additional support that our set of T1 intensity statistics retain meaningful information for inter-regional structural relationships, we conducted an individual identification experiment on the HCP test–retest dataset N=45N = 45, following the R2SN protocol[c1]. We use A(k)\mathbf{A}^{(k)} as described in our paper as the input for this identification experiment.

The results support that our set of intensity statistics preserve meaningful, subject-specific anatomical information, as evidenced by significant test-retest identification performance. We evaluated all (5n)\binom{5}{n} combinations of our five intensity statistics for n=3,4, and 5, and report the identification accuracy averaged across combinations for each value of n. We also evaluated the set of 25 statistics described in R2SN [1] on the Harvard-Oxford Atlas as a reference.

N-statistics combinationAveraged Subject Identification AccuracyP-value
30.630<0.001
40.758<0.001
50.789<0.001
25 from R2SN [c1]0.778<0.001

As shown in the results, using our set of intensity statistics achieved accuracy comparable to that obtained with the 25 R2SN statistics (77.8%). These results support that our set of intensity statistics retains reproducible, subject-specific information consistent with anatomical variation. We have added this analysis and clarified its interpretation in Appendix G of the revised manuscript.

Have the authors considered comparing them with more conventional morphological features, such as cortical thickness, regional volume, or surface area?

We agree that conventional morphological descriptors such as cortical thickness, regional volume, and surface area are important anatomical baselines. However, our objective is not to propose the selected T1 intensity statistics as substitutes for these descriptors, or to claim that they contain more anatomical information. Rather, we seek a compact structural representation that is suitable for large-scale pre-training.

As discussed in Sec 2.3, conventional morphological features could be impractical for our pretraining task, because of the heavy computational overhead due to repeated anatomical segmentation and regional cortical thickness estimation in every pretraining iteration. To assess the potential computational overhead of using conventional descriptors during pretraining, we extracted cortical thickness and regional volumes from 10 subjects from the A4 dataset using the standard two-stage voxel-based pipeline comprising DeepSCAN [c2] for tissue and anatomical segmentation and CortexMorph [c3] for registration-based regional cortical thickness estimation. Assuming one extraction per scan, the mean runtime of 18.7 seconds per scan on our pretraining dataset of 12541 scans for 200 epochs would be prohibitive for pretraining. In contrast, as discussed in Sec 2.3, A(k)\mathbf{A}^{(k)} based on intensity (radiomics) statistics would not require repeated anatomical segmentation and regional cortical thickness estimation, and is suitable for large-scale pretraining. ( to be continued )

AuthorsOpenReview ↗

Comment for Reviewer nZQ8 (2/3)

The subject-identification experiment provides additional support for its retention of subject-discriminative anatomical information, rather than establishing equivalence to conventional morphological features.

Table tc1

MethodRuntime (1 scan) mean ± SD
DeepSCAN segmentation + CortexMorph18.7 ± 1.2 sec

W2: The individual formulation consistently achieves better performance, but the underlying reason is not well explained. Is this because individual-level relationships provide more informative supervision, or because the two formulations differ in the amount and type of structural information they preserve? A more detailed analysis would help clarify this finding.

We thank the reviewer for this insightful comment. We would like to clarify that IRS-SSL-Ind and IRS-SSL-Pop are based on two distinct, well-established neuroscientific constructs, namely individualized structural similarity [c5,c1] and population-level structural covariance [c4, c6]. Both have been associated with clinically relevant variation, but they encode different types of information (See detailed discussion in Sec 2.1).

In the self-supervised task formulations, an important distinction lies in how supervision and gradients are derived. IRS-SSL-Ind constructs a separate structural similarity matrix for each subject, providing subject-specific supervisory signals and gradient contributions. In contrast, IRS-SSL-Pop computes covariance matrices from regional features aggregated across the mini-batch. Consequently, its gradients are coupled across subjects and driven by batch-level structural relationships rather than independent subject-specific targets. The pretraining loss graph (Appendix. I.1 Fig 8) also suggests that IRS-SSL-pop saturates much quicker than IRS-SSL-ind.

IRS-SSL-Ind provides independently defined subject-level supervision, whereas IRS-SSL-Pop provides shared batch-level supervision with cross-subject gradient coupling. This difference in supervisory granularity may help explain the stronger performance of IRS-SSL-Ind on individual-level downstream tasks, where preserving subject-specific anatomical variation is particularly relevant.

W3: The current comparison between HO and Desikan-Killiany is rather limited. First, the authors could evaluate different parcellation resolutions, such as varying the number of ROIs within the same atlas family. Second, a random parcellation baseline would be particularly informative. If randomly defined brain regions achieve comparable performance, it would raise questions about whether anatomically meaningful structural relationships are actually responsible for the improvements. The existing random region pairing experiment does not fully address this concern.

First, the authors could evaluate different parcellation resolutions, such as varying the number of ROIs within the same atlas family

We thank the reviewer for suggesting an evaluation across different parcellation resolutions. While the Harvard-Oxford (HO) anatomical atlas does not offer multi-scale parcellations like functional atlases (e.g., Schaefer or Craddock), we evaluated ROI granularity within the HO family by comparing the coarse 21-ROI HO atlas against a finer 63-ROI parcellation obtained by combining the HO Cortical and Subcortical atlases [c7]. Specifically, whereas the 21-ROI atlas represents the entire cerebral cortex as just two coarse hemisphere-level ROIs alongside subcortical structures, merging it with the HO Cortical atlas subdivides the cortical mantle into 48 finer-grained anatomical parcels alongside the 15 subcortical gray-matter structures (yielding 63 total ROIs after excluding overlapping whole-cortex, white-matter, and ventricular masks).

Comparing our model across both the coarse (21-ROI) and fine-grained cortical (63-ROI) parcels shows that the coarser (HO 21) atlas performs slightly better than the (HO 63) atlas for many tasks. For the brain-tumor grading task, the finer granularity of the (HO 63) atlas would preserve localized lesion cues better than the coarser atlas.

Table tc2: Classification Tasks (Bal. Acc ↑ / AUC ↑ / F1 ↑)

TaskSettingIRS-SSL-Ind (HO 63)IRS-SSL-Ind (HO 21)*
GenderID82.903 / 0.908 / 0.82985.47 / 0.918 / 0.856
MCI / CNID70.793 / 0.808 / 0.71775.290 / 0.826 / 0.757
FTDID79.683 / 0.886 / 0.79580.593 / 0.856 / 0.803
Tumor GradeID64.21 / 0.788 / 0.6661.763 / 0.692 / 0.625
AD / MCI / CNOOD60.353 / 0.852 / 0.55964.433 / 0.834 / 0.605
MCI / CNOOD73.537 / 0.83 / 0.71175.067 / 0.893 / 0.718

Table tc3: Regression & Segmentation

ModelAge (ID) L1 ↓Age (OOD) L1 ↓Seg. DSC ↑
IRS-SSL-Ind (HO 63)9.03310.73165.653
IRS-SSL-Ind (HO 21)*9.23810.60566.6
*denotes results from the manuscript.
(to be continued)
AuthorsOpenReview ↗

Comment for Reviewer nZQ8 (3/3)

Second, a random parcellation baseline would be particularly informative. If randomly defined brain regions achieve comparable performance, it would raise questions about whether anatomically meaningful structural relationships are actually responsible for the improvements.

To evaluate whether anatomically meaningful boundaries drive our improvements rather than arbitrary spatial partitioning, we generated a spatially contiguous random parcellation (R=21) within the identical Harvard-Oxford foreground mask using Poisson disk sampling [c8] (Appendix H. Figure 6). We use the exact same model architecture, pretraining hyperparameters, and relational supervision loss function as described in the paper, with the sole difference being the atlas.

As shown in the table tc4 and tc5, replacing anatomically defined regions with a random parcellation (IRS-SSL-Ind(HO Random Parcel), R=21) causes a substantial performance drop across every single task and metric compared to both HO 21 (e.g., OOD AD/MCI/CN balanced accuracy drops from 64.43% to 43.28%, OOD MCI vs. CN balanced accuracy drops from 75.07% to 53.96%, Age MAE worsens from 9.24 to 11.13, and Segmentation DSC drops from 66.60 to 56.32). This confirms that our relational self-supervision specifically relies on anatomically meaningful brain structures and their inter-regional relationships rather than arbitrary spatial partitioning.

Table tc4: Classification Tasks (Bal. Acc ↑ / AUC ↑ / F1 ↑)

TaskSettingIRS-SSL-Ind (HO Random Parcel)IRS-SSL-Ind (Original)
GenderID75.26 / 0.897 / 0.76685.47 / 0.918 / 0.856
MCI / CNID63.437 / 0.757 / 0.63575.290 / 0.826 / 0.757
FTDID73.37 / 0.837 / 0.73380.593 / 0.856 / 0.803
Tumor GradeID53.447 / 0.662 / 0.51461.763 / 0.692 / 0.625
AD / MCI / CN (OASIS-3)OOD43.283 / 0.675 / 0.42664.433 / 0.834 / 0.605
MCI / CN (OASIS-3)OOD53.96 / 0.624 / 0.53775.067 / 0.893 / 0.718
MCI / CN (AIBL)OOD52.357 / 0.618 / 0.52664.336 / 0.72 / 0.625
AD / MCI / CN (AIBL)OOD41.378 / 0.625 / 0.45253.987 / 0.765 / 0.534

Table tc5: Regression & Segmentation

ModelAge (ID) L1 ↓Age (OOD) L1 ↓Seg. DSC ↑
IRS-SSL-Ind (HO Random Parcel)11.13212.53156.32
IRS-SSL-Ind (Original)9.23810.60566.6

We thank the reviewer for the constructive feedback.

References :

[c1] Zhao, K., et al. (2021). Regional radiomics similarity networks (R2SNs) in the human brain: Reproducibility, small-world properties and a biological basis. Network neuroscience (Cambridge, Mass.), 5(3), 783–797.

[c2] Rebsamen, M., et al. (2020). Direct cortical thickness estimation using deep learning-based anatomy segmentation and cortex parcellation. Human brain mapping, 41(17), 4804–4814.

[c3] McKinley, R., & Rummel, C. (2023). CortexMorph: fast cortical thickness estimation via diffeomorphic registration using VoxelMorph. Medical image computing and computer-assisted intervention : MICCAI 2023 (pp. 730–739.)

[c4] Dong, Chao, et al. "Brain structural covariances in the ageing brain in the UK Biobank." Brain Structure and Function 229.5 (2024).

[c5] Liu, Han, et al. "A radiomics-based brain network in T1 images: construction, attributes, and applications." Cerebral Cortex 34.2 (2024)

[c6] Mongay-Ochoa, Neus, et al. "Structural covariance analysis for neurodegenerative and neuroinflammatory brain disorders." Brain 148.9 (2025)

[c7] Crofts, Jonathan J., et al. "Network analysis detects changes in the contralesional hemisphere following stroke." NeuroImage 54.1 (2011): 161-169.

[c8] Arslan, Salim, et al. "Human brain mapping: A systematic comparison of parcellation methods for the human cerebral cortex." NeuroImage 170 (2018): 5-30.

Reviewer_d89tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Inter-Regional Structural Self-Supervised Learning, which introduces inter-regional structural relationships as the primary inductive bias for 3D brain MRI self-supervised learning. The framework encompasses two formulations, namely individual morphological similarity and population structural covariance, which learn representations by predicting relationships among regional radiomic features under perturbed inputs. Following pretraining on 12,541 multi-site MRI scans, the authors evaluate the learned representations on downstream tasks, including disease classification, brain age prediction, and tumor segmentation.

Critical strengths
  1. The paper conceptualizes Morphometric Similarity Networks and Structural Covariance Networks from neuroscience as primary supervisory signals, moving beyond conventional paradigms that rely solely on voxel reconstruction or instance contrastive learning.
  2. The experimental evaluation spans over six clinical prediction tasks alongside few-shot settings, and further includes a blinded forced-choice preference assessment by six certified radiologists on Grad-CAM attention maps, providing valuable evidence of clinical interpretability.
  3. Empirical results demonstrate that subject-specific inter-regional similarity is more robust than population-level covariance, consistently outperforming standard self-supervised baselines in linear probing and few-shot evaluation regimes.
Critical weaknesses
  1. The pretraining corpus incorporates the OASIS cohort (424 scans), whereas the external evaluation includes OASIS-3. Because OASIS-3 represents a longitudinal extension of the original OASIS study, substantial subject overlap is expected. Without rigorous subject-level deduplication, the reported generalization gains may be invalid.
  2. The authors explicitly state that the MAE pixel reconstruction loss is completely discarded after the initial 50 epochs, leaving the high-capacity 3D decoder guided solely by a scalar loss computed on a 21 by 21 regional similarity matrix. Without voxel-level spatial constraints, the decoder risks mode collapse or generating adversarial image patterns that fulfill low-order intensity statistics while displaying severe anatomical distortion.
  3. Preprocessing applies only rigid registration to the MNI template, which fails to account for interindividual morphological and cortical folding variations. Applying a static 21-region Harvard-Oxford atlas to rigidly aligned brains introduces substantial spatial misalignment along region of interest boundaries, raising the concern that the model may capture registration artifacts rather than genuine neuroanatomy.
  4. The pretraining data aggregates seven cohorts acquired across diverse scanners and protocols without intensity standardization such as Nyul matching or Z-score normalization. Because raw MRI intensities lack absolute physical units, simple intensity statistics such as regional mean and variance are susceptible to scanner-induced batch effects rather than authentic biological variation.
Questions for authors
  1. Could the authors provide representative slices of decoded volumes x^(k) obtained during the final 200 epochs to demonstrate that optimizing solely with the inter-regional loss does not lead to visual collapse or non-biological artifacts?
  2. Could the authors explicitly clarify whether subject identifiers were screened to ensure complete independence between the pretraining OASIS subset and the downstream OASIS-3 cohort, confirming that no longitudinal scans of the same subjects were shared across splits?
  3. What specific intensity harmonization or scaling strategy was applied across the seven pretraining datasets before computing regional intensity statistics, ensuring that the features reflect anatomical properties rather than scanner discrepancies?
  4. Appendix Table 10 specifies training a 3D ViT-B/16 with an 8-layer decoder and a batch size of 64 on a single 48GB GPU. Could the authors clarify the memory optimization techniques used, such as gradient accumulation or mixed precision, to ensure reproducibility?
  5. Please confirm whether references to "NAISSL" in the caption of Table 10 and "IRS-SSN" in Line 887 represent legacy drafting artifacts that should be reconciled with the proposed IRS-SSL nomenclature.
6 comments
AuthorsOpenReview ↗

Comment for Reviewer d89t (1/3)

Critical Strengths: The paper conceptualizes Morphometric Similarity Networks and Structural Covariance Networks from neuroscience as primary supervisory signals, moving beyond conventional paradigms that rely solely on voxel reconstruction or instance contrastive learning.

We thank the reviewer for the careful assessment and for accurately characterizing our central goal: conceptualizing morphometric similarity and structural covariance networks as the primary self-supervisory signals, moving beyond existing voxel-reconstruction or instance-level contrastive paradigms.

W1 The pretraining corpus incorporates the OASIS cohort (424 scans), whereas the external evaluation includes OASIS-3. Because OASIS-3 represents a longitudinal extension of the original OASIS study, substantial subject overlap is expected. Without rigorous subject-level deduplication, the reported generalization gains may be invalid.

Q2 Could the authors explicitly clarify whether subject identifiers were screened to ensure complete independence between the pretraining OASIS subset and the downstream OASIS-3 cohort, confirming that no longitudinal scans of the same subjects were shared across splits?

We thank the reviewer for this observation. We acknowledge the concern. Because OASIS data-use policy prevents verification of complete participant-level separation, for this task, we include additional results using another dataset AIBL (Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing) in which complete participant-level separation can be verified. The results consistently support clear gains of our method. Details are discussed below, and we have also updated the manuscript (Sec. 3.2) Table 1, 2, and 3.

Alzheimer’s classification task. Our previous setup for the Alzheimer’s classification task mostly follows BrainIAC. BrainIAC includes OASIS-3 in its pretraining data pool (Supplementary Data Table 4) while evaluating the MCI versus cognitively normal (CN) task on OASIS-1 (195 scans for training/validation and 40 scans for the held-out test set; Supplementary Methods A.2.5). Thus, while OASIS-3 contributes to pretraining, the specific MCI-CN evaluation is performed on OASIS-1.

OASIS-3 is an extension of the original OASIS series and the OASIS data-use policy prohibits linking of subject identifiers across releases. Therefore, it is not possible to verify complete participant-level independence between the OASIS data used in pretraining and any OASIS-derived evaluation sets.

Additional dataset and experiment. To ensure complete participant-level separation, we add the AIBL (Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing) dataset for another external Alzheimer’s classification evaluation. Identical to the external OASIS-3 protocol reported in the paper, we use the exact same frozen linear-probing head trained on ADNI when evaluating AIBL, ensuring complete participant-level separation.

With this new dataset, IRS-SSL-Ind continues to maintain clear gains over other pretraining methods and SOTA models. On MCI vs CN, IRS-SSL-Ind achieves 64.33% balanced accuracy and 0.720 AUC, outperforming the next best method (IRS-SSL-Pop) as well as MAE, contrastive baselines (BarlowTwins, SimCLR), AnatCL, BrainSegFounder, and MedicalNet.

Updated Table 1: AIBL Comparison of SSL pretext strategies evaluated under linear probing

ModelMCI/CN BAcc↑MCI/CN AUC↑MCI/CN F1↑AD/MCI/CN BAcc↑AD/MCI/CN AUC↑AD/MCI/CN F1↑
MAE57.0660.6420.56548.5930.6660.472
BarlowTwins52.3030.5550.54935.2100.5310.347
SimCLR55.0170.5490.52737.5180.5600.393
BarlowTwins (scratch)54.9100.5830.51737.8130.5910.399
SimCLR (scratch)57.1330.5760.54545.0400.6870.446
IRS-SSL-Pop (Ours)62.6670.6860.59550.9930.6910.538
IRS-SSL-Ind (Ours)64.3360.7200.62553.9870.7650.534

Updated Table 2: AIBL Comparison of SOTAs evaluated under linear probing

ModelMCI/CN BAcc↑MCI/CN AUC↑MCI/CN F1↑AD/MCI/CN BAcc↑AD/MCI/CN AUC↑AD/MCI/CN F1↑
AnatCL60.4830.6320.6132.8330.580.326
BrainSegFounder53.4630.5520.50342.0270.6840.442
MedicalNet57.6630.5840.58247.210.6840.431
IRS-SSL-Pop (Ours)62.6670.6860.59550.9930.6910.538
IRS-SSL-Ind (Ours)64.3360.7200.62553.9870.7650.534

Updated Table 3: AIBL Comparison of SOTAs evaluated under layer decay

ModelMCI/CN BAcc↑MCI/CN AUC↑MCI/CN F1↑AD/MCI/CN BAcc↑AD/MCI/CN AUC↑AD/MCI/CN F1↑
AnatCL63.8420.6540.62635.5660.6070.348
BrainSegFounder56.8330.5850.52845.4040.7020.463
MedicalNet60.4030.6090.61051.2580.7090.460
IRS-SSL-Pop (Ours)66.0770.7130.63855.0230.7150.560
IRS-SSL-Ind (Ours)67.0870.7510.65257.3750.7960.566

( To be continued )

AuthorsOpenReview ↗

Comment for Reviewer d89t (2/3)

W2 The authors explicitly state that the MAE pixel reconstruction loss is completely discarded after the initial 50 epochs, leaving the high-capacity 3D decoder guided solely by a scalar loss computed on a 21 by 21 regional similarity matrix. Without voxel-level spatial constraints, the decoder risks mode collapse or generating adversarial image patterns that fulfill low-order intensity statistics while displaying severe anatomical distortion.

Q1 Could the authors provide representative slices of decoded volumes x^(k) obtained during the final 200 epochs to demonstrate that optimizing solely with the inter-regional loss does not lead to visual collapse or non-biological artifacts?

Clarification of the decoder pipeline. In the image-space realized architecture, the decoder maps the encoder embeddings into an intermediate representation, from which the regional feature operator is applied to compute the regional features Z\mathcal{Z}. Importantly, the intermediate representation produced by the decoder is not intended to be a reconstruction of the input.

To make this clear, we include an additional figure (Fig. 3; Sec 3.4). In the image-space realization, the decoder and the regional feature operator together form a mapping from encoder embeddings to regional features Z\mathcal{Z}. This corresponds to the “Compute Statistics” block in Fig. 1(b), and is functionally analogous to the MLP in the embedding-space realized architecture, which maps directly from the embeddings to regional features. Empirically, we observe that the decoder together with the regional feature operator performs better than a direct MLP mapping alone.

The reviewer’s comment is accurate and precise:

“The authors explicitly state that the MAE pixel reconstruction loss is completely discarded after the initial 50 epochs, leaving the high-capacity 3D decoder guided solely by a scalar loss computed on a 21 by 21 regional similarity matrix.”

The main aim of our study is to conceptualize inter-regional structural organization as the primary self-supervision principle, and to understand the quality of the resulting pre-trained features when reconstruction (and instance-level consistency) is discarded.

Visualization to address mode collapse, adversarial patterns, and distortion concerns. Here we provide qualitative and quantitative visualization to support that there is no mode collapse, adversarial patterns, or distortion in the regional features Z\mathcal{Z}. We clarify that the regional features Z\mathcal{Z} are what matter, as they are used to compute our inter-regional structural loss (Sec. 2.2)

Qualitative visualization (Sec. L): Visual inspection of the regional feature matrices (5 features × 21 regions) produced by the decoder pipeline for different scans shows that they closely track the corresponding original regional feature matrices. The regional feature matrices from the decoder pipeline also vary substantially across scans, supporting the absence of mode collapse.

Quantitative visualization (Sec. L): We compare the means and standard deviations of the features for (I) the original features and (II) the features obtained from IRS-SSL. We observe that the regional features from the decoder pipeline closely track the original regional features. The standard deviations are comparable, further supporting the absence of mode collapse.

W3 Preprocessing applies only rigid registration to the MNI template, which fails to account for interindividual morphological and cortical folding variations. Applying a static 21-region Harvard-Oxford atlas to rigidly aligned brains introduces substantial spatial misalignment along region of interest boundaries, raising the concern that the model may capture registration artifacts rather than genuine neuroanatomy.

We thank the reviewer for the comment. Below we (1) cite prior work supporting the preprocessing pipeline, (2) provide an empirical ablation.

Prior anatomical analysis. While the registration strategy is a preprocessing choice and not part of the proposed IRS-SSL objective, our pipeline follows prior anatomical analyses that extract regional intensity features after linear registration to a static atlas. For example, Giraldo et al. (2018) [b1] registered brains to the Harvard–Oxford atlas with linear registration and noted that residual inter-subject displacement has only a small effect on regional intensity distributions.

Similar to that work, our IRS-SSL relies on regional intensity (radiomic) features rather than cortical thickness or regional volumes, enabling scalable pretraining without repeated subject-specific segmentation. We therefore view residual misalignment after linear registration as a known and previously accepted trade-off for this class of regional intensity analysis.

( To be continued )

AuthorsOpenReview ↗

Comment for Reviewer d89t (3/3)

Empirical ablation. In addition, we provide an empirical ablation to test if IRS-SSL relies on gross alignment artifacts. We conducted a new pretraining experiment, replacing rigid registration with affine registration [b2]. In the table below, it describes that by reducing boundary misalignment, IRS-SSL still maintains strong performance across all 10 tasks. In fact when comparing against other SOTA models, IRS-SSL-Ind with affine registration achieves top-1 for 16/26 metrics (when excluding our models). This supports that the original IRS-SSL pipeline using rigid registration may capture genuine neuroanatomy.

Classification Tasks (Bal. Acc ↑ / AUC ↑ / Macro F1 ↑)

TaskSettingIRS-SSL-Ind (rigid)*IRS-SSL-Ind (affine)
GenderID85.47 / 0.918 / 0.85683.743 / 0.915 / 0.831
MCI / CNID75.290 / 0.826 / 0.75773.303 / 0.831 / 0.746
FTDID80.593 / 0.856 / 0.80379.313 / 0.875 / 0.793
Tumor GradeID61.763 / 0.692 / 0.62558.947 / 0.646 / 0.602
AD / MCI / CNOOD64.433 / 0.834 / 0.60562.617 / 0.825 / 0.6
MCI / CNOOD75.067 / 0.893 / 0.71871.273 / 0.854 / 0.704
MCI / CN (AIBL)OOD64.336 / 0.72 / 0.62561.003 / 0.682 / 0.597
AD / MCI / CN (AIBL)OOD53.987 / 0.765 / 0.53450.993 / 0.711 / 0.521

Regression & Segmentation

ModelAge (ID) L1 ↓Age (OOD) L1 ↓Seg. DSC ↑
IRS-SSL-Ind (rigid)*9.23810.60566.6
IRS-SSL-Ind (affine)9.31910.94365.433

*original configuration from manuscript

W4 The pretraining data aggregates seven cohorts acquired across diverse scanners and protocols without intensity standardization such as Nyul matching or Z-score normalization. Because raw MRI intensities lack absolute physical units, simple intensity statistics such as regional mean and variance are susceptible to scanner-induced batch effects rather than authentic biological variation.

Q3 What specific intensity harmonization or scaling strategy was applied across the seven pretraining datasets before computing regional intensity statistics, ensuring that the features reflect anatomical properties rather than scanner discrepancies?

We clarify that intensity normalization using Brain-mask-based Z-score normalization was applied consistently across all the pretraining datasets before computing regional intensity statistics. Specifically, each MRI volume was normalized during data loading using the mean and standard deviation of its brain voxels, excluding the background, before regional feature extraction.

The intensity normalization can be found in our code submitted previously: Pretrain>utils>data_utils.py (line 605) NormalizeIntensityd(keys=["image"], nonzero=True, channel_wise=True).

This reduces scanner-dependent global intensity variations while preserving relative regional intensity patterns. We have clarified this in the updated manuscript (Appendix C.1).

Q4 Appendix Table 10 specifies training a 3D ViT-B/16 with an 8-layer decoder and a batch size of 64 on a single 48GB GPU. Could the authors clarify the memory optimization techniques used, such as gradient accumulation or mixed precision, to ensure reproducibility?

Thank you for the question. No gradient accumulation was used during the training of IRS-SSL; the model was trained directly with a batch size of 64 on a single 48GB GPU. To fit within the memory budget, we utilized FP16 automatic mixed precision (AMP). The details for the optimization and caption of table 10 has also been updated.

Q5 Please confirm whether references to "NAISSL" in the caption of Table 10 and "IRS-SSN" in Line 887 represent legacy drafting artifacts that should be reconciled with the proposed IRS-SSL nomenclature.

We have fixed the typo.

Thank you very much for the constructive comments.

References :

[b1] Giraldo, Diana L., et al. "Characterization of brain anatomical patterns by comparing region intensity distributions: Applications to the description of Alzheimer's disease." Brain and Behavior. 2018.

[b2] Greve, D. N., & Fischl, B. (2009). Accurate and robust brain image alignment using boundary-based registration. NeuroImage, 48(1), 63–72.

Reviewer_bkkcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

IRS-SSL pretrains brain MRI encoders by matching within-subject regional radiomic similarities or cross-subject regional covariance. Evaluation spans clinical and demographic prediction, segmentation, and external cohorts.

Critical strengths
  1. Multiple downstream tasks and external cohorts are evaluated with three-run uncertainty estimates, linear probing, and fine-tuning.
  2. Matched training budgets, MAE warm-up controls, two atlases, and population-pairing ablations strengthen the evaluation.
Critical weaknesses
  1. Relational supervision is not isolated from regional feature supervision. A matched model directly predicting the same ROI statistics is missing; comparisons with different anatomy-aware methods do not resolve this attribution (Sections 2.3, 3.3).
  2. The individual objective correlates heterogeneous descriptors, including mean, variance, and dimensionless moments. Within-vector normalization does not remove feature-unit dependence. Unspecified descriptor scaling and intensity normalization leave potential acquisition sensitivity unresolved (Equation 1; Appendix C.1).
  3. Pretraining includes OASIS, while external evaluation uses OASIS-3 under the same citation. Despite the explicit sample-separation claim, participant-level separation and longitudinal-scan handling are undocumented, leaving external-validation independence unclear (Tables 8–9).
  4. The clinician study treats 90 responses from six readers evaluating 15 comparisons per baseline as independent binomial trials. Repeated readers and cases create dependence; the reported p-values require a clustered analysis (Table 4; Appendix F).
Questions for authors
  1. Does IRS-SSL outperform direct ROI-statistic prediction under matched training, and how are descriptor scales and MRI intensities standardized?
  2. Can you provide participant-level overlap checks and repeated-scan handling for OASIS/OASIS-3?
  3. Do clinician preferences remain significant after accounting for reader and case dependence?
6 comments
AuthorsOpenReview ↗

Comment for Reviewer bkkc (1/3)

W1 Relational supervision is not isolated from regional feature supervision. A matched model directly predicting the same ROI statistics is missing; comparisons with different anatomy-aware methods do not resolve this attribution (Sections 2.3, 3.3).

Q1 Does IRS-SSL outperform direct ROI-statistic prediction under matched training…

In this regional feature supervision ablation, we use the exact same model architecture, pretraining hyperparameters, and relational supervision loss function as described in the paper, with the sole difference being the pre-training loss objective. We define Z(k)Z(k) as the ROI statistics S×RS \times R extracted from each subject. The brain MRI encoder is pretrained by minimizing the L1 objective ∥Z(k)−Z^(k)∥1\|Z(k) - \hat{Z}(k)\|_1. Downstream performance is then evaluated using the exact same linear probing protocol as in the paper.

IRS-SSL-Ind (relational supervision) consistently outperforms the regional feature supervision across all evaluation tasks, with the largest gains on the more challenging out-of-distribution settings (ADMCICN (OASIS - 3) balanced accuracy: 64.4% vs 47.5%; MCICN (OASIS - 3): 75.1% vs 51.3%). This shows that pretraining guided by the regional feature supervision does not provide the same gains as our proposed relational supervision.

Table ta1 Classification Tasks (Bal. Acc ↑ / AUC ↑ / F1 ↑)

TaskSettingIRS-SSL-IndRegional Features
GenderID85.470 / 0.918 / 0.85683.613 / 0.896 / 0.837
MCI / CNID75.290 / 0.826 / 0.75770.137 / 0.794 / 0.711
FTD / CNID80.593 / 0.856 / 0.80374.980 / 0.823 / 0.741
Tumor GradeID61.763 / 0.692 / 0.62557.190 / 0.687 / 0.580
AD / MCI / CN (OASIS - 3)OOD64.433 / 0.834 / 0.60547.737 / 0.815 / 0.470
MCI / CN (OASIS - 3)OOD75.067 / 0.893 / 0.71851.290 / 0.740 / 0.511
MCI / CN (AIBL)OOD64.336 / 0.720 / 0.62558.313 / 0.662 / 0.575
AD / MCI / CN (AIBL)OOD53.987 / 0.765 / 0.53449.126 / 0.679 / 0.496

Table ta2 Regression & Segmentation

ModelAge (ID) L1 ↓Age (OOD) L1 ↓Tumor Seg. DSC ↑
IRS-SSL-Ind9.23810.60566.600
Regional Features11.09711.05860.143

We have added these results to the updated manuscript under ( Appendix K; Table 19 )

W2 The individual objective correlates heterogeneous descriptors, including mean, variance, and dimensionless moments. Within-vector normalization does not remove feature-unit dependence. Unspecified descriptor scaling and intensity normalization leave potential acquisition sensitivity unresolved (Equation 1; Appendix C.1).

Q1 …and how are descriptor scales and MRI intensities standardized?

Unspecified descriptor scaling and intensity normalization leave potential acquisition sensitivity unresolved

We thank the reviewer for this question. We clarify that intensity normalization using Brain-mask-based Z-score normalization and separate normalization for each descriptor across ROIs were applied as shown in our submitted program code. Below, we provide more details.

We clarify that intensity normalization using Brain-mask-based Z-score normalization was applied consistently. Specifically, each MRI volume was normalized during data loading using the mean and standard deviation of its brain voxels, excluding the background, before regional feature extraction.

The intensity normalization can be found in our code submitted previously: Pretrain>utils>data_utils.py (line 605) NormalizeIntensityd(keys=["image"], nonzero=True, channel_wise=True).

This reduces scanner-dependent global intensity variations while preserving relative regional intensity patterns. We have clarified this in the updated manuscript (Appendix C.1).

Within-vector normalization does not remove feature-unit dependence

Radiomic descriptor scaling. For each scan, we normalize each descriptor separately across the ROIs: the mean-intensity values across all ROIs form one column, the variance values form another, and so on.

The descriptor scaling can be found in our code submitted previously:

Pretrain/entropy_loss.py (lines 331-334) where x is the statistics tensor of [Statistics,Regions]

331: feature_mean = x.mean(dim=1, keepdim=True)

332: feature_std = x.std(dim=1, keepdim=True, unbiased=False)

333:

334: x = (x - feature_mean) / (feature_std + eps)

Each column is standardized across ROIs to have zero mean and unit variance. Thus, normalization is performed across ROIs separately for each descriptor, rather than across the heterogeneous descriptors within each ROI. We apply this operation consistently to the regional features before computing the inter-regional similarity matrices. This makes the descriptor scales comparable in the similarity calculation.

We have clarified this in the updated manuscript (Sec 2.2).

( to be continued )

AuthorsOpenReview ↗

Comment for Reviewer bkkc (2/3)

W3 Pretraining includes OASIS, while external evaluation uses OASIS-3 under the same citation. Despite the explicit sample-separation claim, participant-level separation and longitudinal-scan handling are undocumented, leaving external-validation independence unclear (Tables 8–9).

Q2 Can you provide participant-level overlap checks and repeated-scan handling for OASIS/OASIS-3?

We thank the reviewer for this observation. We acknowledge the concern. Because OASIS data-use policy prevents verification of complete participant-level separation, for this task, we include additional results using another dataset AIBL (Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing) in which complete participant-level separation can be verified. The results consistently support clear gains of our method. Details are discussed below, and we have also updated the manuscript (Sec. 3.2) Table 1, 2, and 3.

Alzheimer’s classification task. Our previous setup for the Alzheimer’s classification task mostly follows BrainIAC. BrainIAC includes OASIS-3 in its pretraining data pool (Supplementary Data Table 4) while evaluating the MCI versus cognitively normal (CN) task on OASIS-1 (195 scans for training/validation and 40 scans for the held-out test set; Supplementary Methods A.2.5). Thus, while OASIS-3 contributes to pretraining, the specific MCI-CN evaluation is performed on OASIS-1. OASIS-3 is an extension of the original OASIS series and the OASIS data-use policy prohibits linking of subject identifiers across releases. Therefore, it is not possible to verify complete participant-level independence between the OASIS data used in pretraining and any OASIS-derived evaluation sets.

Additional dataset and experiment. To ensure complete participant-level separation, we add the AIBL (Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing) dataset for another external Alzheimer’s classification evaluation. Identical to the external OASIS-3 protocol reported in the paper, we use the exact same frozen linear-probing head trained on ADNI when evaluating AIBL, ensuring complete participant-level separation. With this new dataset, IRS-SSL-Ind continues to maintain clear gains over other pretraining methods and SOTA models. On MCI vs CN, IRS-SSL-Ind achieves 64.33% balanced accuracy and 0.720 AUC, outperforming the next best method (IRS-SSL-Pop) as well as MAE, contrastive baselines (BarlowTwins, SimCLR), AnatCL, BrainSegFounder, and MedicalNet.

Updated Table 1: AIBL Comparison of SSL pretext strategies evaluated under linear probing

ModelMCI/CN BAcc↑MCI/CN AUC↑MCI/CN F1↑AD/MCI/CN BAcc↑AD/MCI/CN AUC↑AD/MCI/CN F1↑
MAE57.0660.6420.56548.5930.6660.472
BarlowTwins52.3030.5550.54935.2100.5310.347
SimCLR55.0170.5490.52737.5180.5600.393
BarlowTwins (scratch)54.9100.5830.51737.8130.5910.399
SimCLR (scratch)57.1330.5760.54545.0400.6870.446
IRS-SSL-Pop (Ours)62.6670.6860.59550.9930.6910.538
IRS-SSL-Ind (Ours)64.3360.7200.62553.9870.7650.534

Updated Table 2: AIBL Comparison of SOTAs evaluated under linear probing

ModelMCI/CN BAcc↑MCI/CN AUC↑MCI/CN F1↑AD/MCI/CN BAcc↑AD/MCI/CN AUC↑AD/MCI/CN F1↑
AnatCL60.4830.6320.6132.8330.580.326
BrainSegFounder53.4630.5520.50342.0270.6840.442
MedicalNet57.6630.5840.58247.210.6840.431
IRS-SSL-Pop (Ours)62.6670.6860.59550.9930.6910.538
IRS-SSL-Ind (Ours)64.3360.7200.62553.9870.7650.534

Updated Table 3: AIBL Comparison of SOTAs evaluated under layer decay

ModelMCI/CN BAcc↑MCI/CN AUC↑MCI/CN F1↑AD/MCI/CN BAcc↑AD/MCI/CN AUC↑AD/MCI/CN F1↑
AnatCL63.8420.6540.62635.5660.6070.348
BrainSegFounder56.8330.5850.52845.4040.7020.463
MedicalNet60.4030.6090.61051.2580.7090.460
IRS-SSL-Pop (Ours)66.0770.7130.63855.0230.7150.560
IRS-SSL-Ind (Ours)67.0870.7510.65257.3750.7960.566

( to be continued )

AuthorsOpenReview ↗

Comment for Reviewer bkkc (3/3)

W4 The clinician study treats 90 responses from six readers evaluating 15 comparisons per baseline as independent binomial trials. Repeated readers and cases create dependence; the reported p-values require a clustered analysis (Table 4; Appendix F).

Q3 Do clinician preferences remain significant after accounting for reader and case dependence?

Thank you for the observation. We clarify that we have only repeated clinicians in a given comparison. To account for repeated clinician dependence, we employed separate binomial logistic generalized linear mixed models (GLMMs) to evaluate whether IRS-SSL-Ind was preferred over each baseline method. Each of the six clinicians evaluated 15 comparisons per baseline, yielding 90 ratings per model pair. Preference for IRS-SSL-Ind was coded as 1 and preference for the baseline as 0. A clinician-specific random intercept accounted for repeated ratings from the same clinician. Each case contributed one rating within a given comparison, so no case random intercept was included; dependence involving cases across comparisons was not modeled jointly. Statistical significance was assessed using two-sided Wald tests and parametric-bootstrap likelihood-ratio tests (9,999 simulations), with 95% profile-likelihood confidence intervals. Holm correction was applied to the three bootstrap p-values.

As shown in the table below, all three comparisons remained statistically significant after Holm correction, demonstrating consistent clinician preference for IRS-SSL-Ind over MAE, SimCLR, and Barlow Twins.

Pairwise clinician preference results analyzed using GLMM.

ComparisonPrefer IRS-SSL-IndLog-Odds95% CIWald pBootstrap pHolm p
IRS-SSL-Ind vs. MAE65/900.959[0.440, 1.612]9.23e-50.00990.0198
IRS-SSL-Ind vs. SimCLR77/901.881[1.119, 3.013]3.66e-60.00470.0141
IRS-SSL-Ind vs. Barlow Twins73/901.653[0.708, 3.018]3.93e-40.01870.0198

We have updated Table 4 and Section 3.3 of the manuscript to reflect this revised analysis.

We thank the reviewer for the constructive comments.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank you AC and Reviewers for your valuable time and discussion

Dear Area Chair and Reviewers:

We sincerely thank the AC and reviewers for the discussion. We are encouraged that reviewers recognized the novelty of conceptualizing neuroscience-grounded inter-regional structural relationships as primary self-supervisory signals (d89t), the extensive evaluation across clinical tasks and the value of the blinded radiologist assessment (d89t), the comprehensive benchmarking and controlled pretraining comparisons (bkkc), and the potential of inter-regional structural learning beyond existing voxel-level and instance-level SSL (nZQ8).

To our knowledge, IRS-SSL is the first to conceptualize individualized structural similarity and population-level structural covariance as primary self-supervisory signals for large-scale brain MRI pretraining, going beyond existing paradigms centered on voxel reconstruction or instance-level contrastive learning.

Our comprehensive experiments and six radiologists' evaluations support the effectiveness and biological grounding of this approach.

During the rebuttal, we added experiments and clarifications that the reviewers acknowledged as helpful or as resolving several concerns. Their follow-up comments focused on a narrower set of remaining issues, to which we provided further clarifications.

Reviewer bkkc

Reviewer bkkc initially questioned whether the gains extend beyond direct ROI-feature prediction, feature and intensity normalization, and external validation. We added a matched regional-feature supervision ablation, clarified the normalization procedures in our submitted code, and included AIBL results; the reviewer acknowledged these additions as helpful.

The follow-up focused on clinician-study dependence and participant-level independence for one of our evaluations using OASIS. We clarified that there are no repeated case-level observations within a model (Fig. 11), while repeated clinician ratings are modeled; clinicians' preference for IRS-SSL-Ind remains significant after Holm correction.

We also noted that related BrainIAC protocol (Tak et al., 2026, Nature Neuroscience) uses similar OASIS setup, clarified the limitation in the revised manuscript, and added AIBL evaluation with verifiable participant-level separation, where IRS-SSL shows consistent gains.

No further reviewer questions have been posted following these additional clarifications.

Reviewer d89t

Reviewer d89t initially raised concerns about OASIS overlap, decoder artifacts, atlas alignment, cross-scanner harmonization ("Nyul matching or Z-score normalization"). We added AIBL evaluation, an affine-registration ablation, and clarification.

In the follow-up, the reviewer mentioned "The authors' response substantially mitigates the main concerns ... however, decoder anatomical fidelity, registration/atlas mismatch, and cross-scanner harmonization remain unresolved."

We clarified that the decoded volume is an intermediate for computing regional features, not an anatomical reconstruction, added results of the derived features closely tracking the original features, addressing collapse and distortion concerns at the level relevant to IRS-SSL.

For registration/atlas mismatch, we added experiments to support that the affine-registration model remained strong across all tasks, and highlighted that none of our evaluation tasks uses atlas-based ROIs.

For harmonization, we clarified that, in our submitted code, all seven pretraining cohorts use consistent brain-mask-based Z-score intensity normalization (the one mentioned by this reviewer).

No further reviewer questions have been posted following these additional clarifications.

Reviewer nZQ8

Reviewer nZQ8 initially questioned whether our five T1-intensity descriptors capture meaningful inter-regional structure, IRS-SSL-Ind vs IRS-SSL-Pop, parcellation resolution and random parcellation. We added a cross-session identification experiment, clarified the distinct supervision mechanisms, compared HO-21 and HO-63, and evaluated a spatially contiguous random parcellation.

In the follow-up, the reviewer said the concerns were partially resolved, but remained unconvinced about the descriptors and the methodological contribution.

We highlighted that we did not test the five intensity statistics in isolation. We tested the inter-regional structural matrix constructed from these statistics, which achieves 78.9% cross-session individual identification accuracy on HCP test–retest data. We follow the R2SN protocol for this experiment, and the result shows that the inter-regional structure based on these statistics contains reproducible, subject-specific information. (Sec. G)

Furthermore, we also reaffirmed the methodological contribution.

Thank you very much.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.