acceptodds
Under review as a conference paper at ICLR 2027

Atlas-Guided Regional Representation Adaptation for Brain Foundation Models

Abstract

Brain foundation models provide a powerful starting point for transferring knowledge across neuroimaging tasks, yet their learned representations are often organized around generic image tokens rather than anatomically defined brain regions. This makes it challenging to directly exploit the regional structure of the brain when adapting pretrained representations to downstream applications. We introduce AnatoQ, an efficient anatomy-aware adaptation framework that converts generic foundation-model features into region-specific representations using anatomical priors. At its center is an Atlas-Guided Q-Former (AGQ), which uses atlas-defined regions as queries to selectively aggregate informative features from pretrained image tokens. Rather than altering the architecture or retraining the foundation model, AnatoQ operates as a lightweight adaptation module, allowing anatomical structure to be incorporated while retaining the knowledge encoded by the original backbone. The resulting regional representations provide a common interface for downstream brain imaging tasks and can be combined with different pretrained backbones. Experiments on a broad range of downstream tasks and multi-session neuroimaging datasets show general gains across foundation models and aggregation methods. Additional analyses demonstrate that the learned representations exhibit stable correspondence with anatomically meaningful regions and support task-dependent interpretation. These results suggest that atlas-guided regional adaptation offers an effective and general approach for injecting anatomical structure into pretrained brain representations.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Foundation, Learning Representations, Regional Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_UvgpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AnatoQ adapts frozen brain foundation models to atlas-defined regional representations through an Atlas-Guided Q-Former (AGQ), combining atlas-initialized queries, global cross-attention, and regional-volume pretraining. Experiments on ADNI/OASIS report improvements across four backbones and several tasks, alongside attribution analyses.

Critical strengths
  1. Relevant problem: Bridges generic spatial features and anatomically defined regions.
  2. Practical design: A lightweight adapter preserves pretrained backbones.
  3. Promising results: Improvements across four CNN/Transformer encoders and multiple tasks, particularly MedicalNet.
  4. Useful analyses: Aggregation ablations, attribution visualizations, and stability measurements.
Critical weaknesses
  1. (Major) Attribution faithfulness is not established. Sections 4.3–4.4 interpret saliency as identifying structures driving predictions. Yet global cross-attention allows each region-labeled token to access the entire brain. Moreover, mean pooling gives all region tokens identical output gradients, so Appendix E.1's scores chiefly reflect mean-centered token features weighted by shared channel gradients. Table 3 and Figure 3 show stable, anatomically plausible patterns, not faithful spatial localization. Controlled regional occlusion/perturbation tests are needed.

  2. (Major) Generalization claims exceed the evidence. Four backbones demonstrate architectural compatibility, but evaluation is limited to ADNI and OASIS. OASIS supports only AD classification; all regression tasks use ADNI. This limited evaluation does not substantiate the general-purpose capabilities claimed in the abstract, nor establish robustness under cohort or task-distribution shifts. Please add targeted external-transfer results or narrow the claims.

  3. (Major) Possible pretraining/test-subject overlap. Appendix A.3 states that regional-volume pretraining can use “all subjects” without diagnostic labels. However, it uses subject-specific MRIs and 120 gray-matter regional volumes associated with downstream phenotypes. If test subjects were included, evaluation would be transductive rather than strictly unseen-subject generalization. Please disclose overlap across stages and, if present, report subject-disjoint results.

  4. (Major) Atlas guidance and the spatial-resolution explanation lack controlled support. In Figure 2, MedicalNet's generic Q-Former nearly matches full AGQ despite its fine token grid; coarser BrainIAC/SwinUNETR show greater benefits from atlas initialization, while Static Atlas markedly outperforms AGQ on SAM-Med3D. These results are not convincingly explained by the proposed token-resolution argument. Please test controlled token resolutions and randomized/spatially perturbed atlases with matched capacity, supervision, and optimization to isolate anatomically correct guidance.

  5. Reproducibility inconsistencies. Appendix A.1 lists 1,013 OASIS subjects, but Table A2 totals 813 (451 CN + 362 AD); the remaining 200 are unaccounted for. Table A6 reports ~75k/~133k/~298k regression-head parameters at feature dimensions 384/512/768, despite describing linear heads (a single-layer scalar head would require only d+1). Please specify architectures and counts; the supplementary reference code does not resolve this.

  6. Unexplained main/ablation discrepancies. For MedicalNet/ADNI, Table 1 reports AGQ gains over mean pooling of +8.5 ACC and +5.2 AUC points, whereas Figure 2 suggests ~+4.5 ACC and ~+10.0 AUC. These substantial discrepancies raise concerns about experimental variability or unstable model convergence. Please clarify whether the evaluation protocols differ and provide evidence, such as multi-seed results or convergence analyses, to establish the reliability of the reported improvements.

Questions for authors
  1. What are the exact regression-head architectures and parameter counts? Why do Table 1 and Figure 2 differ—were splits, seeds, checkpoints, and settings identical?
  2. What evidence supports transfer to independent cohorts or substantially different task distributions, beyond backbone compatibility?
  3. Did anatomical pretraining include downstream test subjects? If so, can the authors clarify the setting and provide subject-disjoint evaluation?
3 comments
AuthorsOpenReview ↗

For Reviewer Uvgp with the Question 1

For Question 1, we thank the reviewer for raising these important questions regarding the regression-head implementation, parameter accounting, and experimental reproducibility.

(i) Regression-head architecture and parameter counts. We clarify that the downstream regression head is implemented as a lightweight multilayer perceptron (MLP). Specifically, its architecture is:

BatchNorm1d(d) → Linear(d, h) → GELU → LayerNorm(h) → Linear(h, 1)

where h=max⁡(⌊d/2⌋,32)h = \max(\lfloor d/2 \rfloor, 32) and dd denotes the encoder feature dimension. The corresponding trainable parameter count is dh+2d+4h+1dh + 2d + 4h + 1, including the learnable affine parameters of the BatchNorm and LayerNorm layers.

We will explicitly add the MLP architecture and its hidden-dimension definition to the revised manuscript to improve reproducibility.

(ii) Differences between Table 1 and Figure 2. We clarify that both Table 1 and Figure 2 report average performance over five data splits. However, the two experiments use separately generated sets of five data splits, rather than the same five partitions. Therefore, although the number of splits is identical, the actual data partitions differ, which can lead to numerical differences in the reported average performance.

Importantly, other experimental settings, including random seeds and model checkpoints, are kept consistent between the two experiments. Thus, the numerical differences primarily arise from the independently generated data partitions, rather than changes in the random seed, model initialization, or checkpoint selection.

Table 1 presents the main performance evaluation, whereas Figure 2 focuses on controlled architectural ablation experiments to investigate the contributions of different components of AnatoQ. We will clarify the use of independently generated data splits and the consistency of the other experimental settings in the revised manuscript to improve transparency and reproducibility.

AuthorsOpenReview ↗

For Reviewer Uvgp with the Question 2

For Question 2, we thank the reviewer for this important question. Our evaluation goes beyond compatibility with different pretrained backbones by including an independent cohort evaluation on OASIS and multiple downstream prediction tasks.

Specifically, Stage-1 anatomical pretraining uses only ADNI images without diagnostic or cognitive labels. In addition to the downstream evaluation on ADNI, we further evaluate AGQ on OASIS, an independent external cohort, where it improves AD classification accuracy over mean pooling on three of the four pretrained encoders (+1.5 to +8.3 percentage points; Table 1).

Importantly, this evaluation is intended to examine the generalizability of the proposed aggregation framework beyond the original ADNI cohort, rather than merely demonstrating compatibility with different encoder architectures.

Furthermore, our evaluation covers multiple downstream prediction targets, including AD diagnosis, chronological age, MMSE, ADAS11, ADASQ4, and RAVLT-immediate. As illustrated in Figure 3, AGQ exhibits distinct task-relevant regional attribution patterns across these prediction targets, which are consistent with established neuroimaging findings. These observations suggest that its anatomically structured representations can capture different task-related associations rather than relying exclusively on a fixed set of predictive regions.

Together, the independent OASIS evaluation, multiple downstream tasks, and results across four pretrained encoders provide complementary evidence of the generalizability of AGQ across datasets, prediction targets, and backbone architectures.

AuthorsOpenReview ↗

For Reviewer Uvgp with the Question 3

For Question 3, we thank the reviewer for raising this important question. We would like to explicitly clarify that downstream test subjects are excluded before any model training or Stage-1 anatomical pretraining is performed.

All datasets used in our experiments consist of baseline scans without repeated subjects. The test set is separated first, and no test-subject images or information are used during Stage-1 regional GMV pretraining, Stage-2 downstream training, or model optimization.

Although Stage-1 pretraining does not require diagnostic or cognitive labels, this does not mean that held-out test subjects are included. Anatomical pretraining is performed exclusively on the designated training subjects, ensuring strict subject-level separation throughout the training and evaluation pipeline.

Therefore, our experiments do not follow a transductive adaptation setting, and the reported test results reflect generalization to entirely unseen subjects. We will explicitly clarify this data-splitting procedure in the revised manuscript to eliminate potential ambiguity.

Reviewer_dJMLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper addresses a structural mismatch in brain MRI foundation models: their representations are organized around regular patch/grid tokens rather than anatomically defined brain regions. The authors propose AnatoQ, a lightweight, plug-and-play post-encoder adaptation framework whose core module is the Atlas-Guided Q-Former (AGQ). AGQ operates in three stages: (1) an atlas-to-patch soft assignment that builds a deterministic, feature-independent correspondence matrix W from voxel overlap; (2) anatomy-grounded query initialization, where atlas-pooled regional features Z are combined with a learnable region-identity embedding (Q = Z + E_region); and (3) anatomy-anchored contextual refinement via global cross-attention over the full token sequence. The backbone stays frozen; only the adapter and a linear probe are trained. Experiments on ADNI and OASIS sMRI with four frozen encoders (SAM-Med3D, BrainIAC, MedicalNet, SwinUNETR) and the AAL2 atlas (K=120) cover AD diagnosis and continuous cognitive/demographic regression. The authors report general gains over mean pooling and provide gradient-based regional attribution analyses that align with known AD biomarkers.

Critical strengths
  1. Well-motivated and clearly articulated problem. The patch-vs-region mismatch is a real and underexplored limitation of current volumetric brain foundation models, and the paper frames it precisely (a token may span multiple regions; a region may fragment across tokens).

  2. Practical, parameter-efficient design. Keeping the backbone frozen and inserting a lightweight adapter is attractive for small-scale medical datasets where full fine-tuning overfits. The approach is architecture-agnostic and demonstrated across four heterogeneous encoders.

  3. Interpretability by construction. Because each output token maps to a named atlas region, regional attribution is natural. The reported Top-20 attribution mass (avg 32.7% vs 16.7% uniform baseline) and high stability (ρ = 0.981–0.998) are a reasonable, quantitative attempt at interpretability rather than cherry-picked visualizations.

  4. Reasonable ablation structure. The progression mean pooling → static atlas pooling → atlas query only → generic Q-Former → full AGQ cleanly isolates the contribution of anatomical initialization versus learnable refinement.

Critical weaknesses
  1. Weak baselines; no comparison against anatomy-aware SOTA. The central comparisons are mean pooling and a generic Q-Former — essentially ablation variants. The paper does not compare against existing anatomy-aware methods (ROI-based modeling, graph representation learning, anatomical-constraint approaches) or established adapters (e.g., AdaptFormer), despite citing them as related work. It is therefore hard to judge AnatoQ's advantage over prior regional-prior methods.

  2. Statistical reliability is insufficient. AGQ wins only 23/32 encoder×dataset×metric combinations for diagnosis, and SEN/SPE fluctuate heavily. The reported std values are large (e.g., ACC 91.9±5.7, SEN 89.8±8.9), yet no significance testing is provided. Many claimed improvements plausibly fall within error. The authors themselves attribute variability to small test splits, which exposes a sample-size limitation rather than resolving it.

  3. Inconsistent / contradictory ablation findings. On SAM-Med3D, simpler atlas-based variants outperform the full model. The paper dismisses this as "backbone-dependent" without analysis. This directly undercuts the claim that contextual refinement is a generally beneficial core contribution.

  4. Confounded central claim about token resolution. The strongest gains (MedicalNet) are attributed to its finer 12³ grid. But the four backbones differ simultaneously in pretraining objective, data, and architecture, so the gain cannot be isolated to resolution. A controlled multi-resolution experiment on a single backbone is needed to support this claim.

  5. No atlas sensitivity analysis. AAL2 (K=120) is fixed throughout. The method's robustness to atlas choice and parcellation granularity — central to any atlas-guided approach — is entirely unexamined.

  6. Attribution evidence is post-hoc consistency, not validation. Agreement between salient regions and known AD biomarkers shows the method is "not inconsistent" with prior knowledge, but does not establish that it captures disease-specific causal evidence. The high attribution stability may partly reflect the fixed atlas-initialization prior rather than learned, disease-relevant signal.

  7. Limited scope vs. the "foundation model" framing. The title claims generality for brain foundation models, but experiments are confined to sMRI with AD diagnosis and cognitive regression. No fMRI, segmentation, or other diseases/modalities are tested, so the "general" claim is under-supported.

  8. Limited novelty. The method is largely a thoughtful recombination of existing components: deterministic weighted pooling (W) + Q-Former with atlas-initialized queries + standard cross-attention. The individual mechanisms are not new.

Questions for authors
  1. Can you provide significance tests (e.g., paired tests across seeds/splits) for the AGQ-vs-baseline comparisons, given the large reported variances?
  2. How does AnatoQ compare against at least one established anatomy-aware method and one standard adapter under identical splits and probes?
  3. To disentangle the resolution claim, can you vary token resolution on a single backbone and show the trend holds?
  4. How sensitive are results to the atlas choice and K (e.g., Schaefer, different AAL2 granularity)? Does performance degrade gracefully with coarser parcellations?
  5. On SAM-Med3D, why does contextual refinement hurt? Is there a diagnostic (e.g., attention entropy, token redundancy) that predicts when refinement helps?
  6. What is the exact trainable-parameter count and compute overhead relative to mean pooling and a generic Q-Former (Appendix C is referenced but a summary in the main text would help)?
5 comments
AuthorsOpenReview ↗

For the Reviewer dJML with Question 1 and Question 2, we hope that our responses will provide a clearer and more comprehensive understanding of our work.

We sincerely appreciate the reviewer's careful and insightful assessment of our work.

For Question 1, we thank the reviewer for this important suggestion regarding the statistical reliability of our comparisons.

Our main experimental results in Tables 1 and 2 are already reported as mean performance with standard deviations across five independent data splits, rather than relying on a single train/test partition. Specifically, Table 1 reports the mean and standard deviation of ACC, AUC, sensitivity, and specificity, while Table 2 reports the corresponding statistics for MAE and Pearson correlation across the downstream regression tasks. The values in parentheses represent the standard deviations across data splits.

This evaluation protocol is intended to characterize both the average performance and the variability of AGQ and the mean-pooling baseline under different data partitions. Moreover, the comparisons use matched experimental settings, including the same pretrained encoders and downstream training configurations, to ensure a consistent evaluation. Moreover, the improvements are mostly substantial and consistent across multiple pretrained encoders, providing strong evidence of effectiveness beyond statistical significance alone.

To further examine the robustness of the observed improvements, we additionally conducted a leave-one-out robustness analysis, evaluating whether AGQ maintains its advantage over mean pooling after individual exclusions.

Across four pretrained encoders, AGQ retains improvements in 21/24 ACC comparisons (87.5%) and 22/24 AUC comparisons (91.7%). Overall, 43 out of 48 comparisons (89.6%) continue to show improvements, suggesting that the observed performance gains are not solely driven by individual evaluation subsets.

For Question 2, we thank the reviewer for this valuable suggestion. To the best of our knowledge, existing anatomy-aware foundation model approaches generally differ from AnatoQ in their methodological objectives and integration strategies.

For example, AnatCL (Barbano et al., 2026) incorporates anatomical information into the foundation model's contrastive pretraining to learn anatomy-aware representations. AtlasSegFM (Zhang et al., 2026) integrates atlas-derived contextual guidance with a medical foundation model through registration, prompting, and lightweight prediction fusion, primarily targeting medical image segmentation. BrainGFM (Wei et al., 2026) incorporates multiple brain atlases and parcellations into graph-based foundation model pretraining and prompt-based adaptation. These methods demonstrate the potential value of anatomical priors but differ in their architectures, input representations, training objectives, and downstream tasks.

In contrast, AnatoQ is designed as a lightweight, post-encoder representation adaptation framework. Its primary objective is to enhance the representations of existing pretrained brain foundation models through atlas-guided feature aggregation, without modifying or retraining the pretrained backbone. Consequently, directly reproducing the aforementioned methods under identical backbones, data splits, and downstream prediction heads would require substantial modifications to their original frameworks and would not constitute a strictly equivalent comparison.

Nevertheless, our existing experiments include two representative and controlled baselines (Figure 2 and Table A7): Static Atlas, representing conventional anatomy-aware feature aggregation, and Q-Former, representing a standard learnable attention-based adapter.

Importantly, all methods are evaluated using identical frozen backbone features, data splits, downstream prediction heads, and optimization settings. As shown in Figure 2, Static Atlas improves AUC over Mean Pooling across all four encoders (+2.4 to +4.2 percentage points), while AGQ further outperforms Q-Former in AUC across all four encoders (+0.4 to +0.9 percentage points).

Together, these controlled comparisons demonstrate the benefits of combining anatomical guidance with adaptive feature aggregation, while maintaining the defining advantage of AnatoQ: improving pretrained foundation model representations without modifying or retraining the backbone.

[1] Barbano, C. A., Brunello, M., Dufumier, B., and Grangetto, M. Anatomical Foundation Models for Brain MRIs. Pattern Recognition Letters, 2026. https://arxiv.org/abs/2408.07079

[2] Zhang, Z., Yu, Y., Zhu, S., Aly, A., Gao, Y., Gu, N., and Xue, Y. Atlas is Your Perfect Context: One-Shot Customization for Generalizable Foundational Medical Image Segmentation. ECCV, 2026. https://arxiv.org/abs/2512.18176

[3] Wei, X., Zhao, K., Jiao, Y., He, L., and Zhang, Y. A Brain Graph Foundation Model: Pre-Training and Prompt-Tuning across Broad Atlases and Disorders. ICLR, 2026. https://arxiv.org/abs/2506.02044

AuthorsOpenReview ↗

For the Reviewer dJML with Question 4

For Question 4, we thank the reviewer for this valuable suggestion regarding the sensitivity of AnatoQ to atlas selection and parcellation granularity.

First, the primary contribution of AnatoQ is not the selection or optimization of a particular brain atlas, but the introduction of anatomical guidance into the representations of pretrained brain foundation models without modifying or retraining the backbone. Our framework constructs atlas-to-token soft assignments and uses the resulting anatomical features to initialize region queries, which are subsequently refined through global cross-attention.

Second, we selected the AAL2 atlas with K = 120 regions because it provides a widely adopted and anatomically interpretable parcellation for structural brain MRI analysis, particularly in studies of neurodegenerative diseases. This choice allows us to establish consistent anatomical correspondence across subjects and pretrained encoders while retaining meaningful region-level representations.

Importantly, the proposed framework is not architecturally restricted to AAL2 or K = 120. For a different atlas or parcellation granularity, the atlas-to-token assignment matrix can be recomputed according to the new anatomical definitions and encoder token geometry, and the lightweight AGQ module can be pretrained accordingly. The pretrained foundation model itself remains unchanged.

Third, the effect of anatomical granularity also depends on the spatial resolution of the pretrained encoder. As discussed in Appendix B, coarse encoder token grids may overlap multiple anatomical regions, limiting the ability to distinguish small structures even when a fine-grained atlas is used. Conversely, coarser parcellations may merge anatomically distinct structures and reduce regional specificity. These considerations indicate that atlas granularity should be interpreted in conjunction with the spatial resolution of the backbone.

Furthermore, we have also conducted an additional experiment using one alternative brain atlas AICHA with 384 regions, which differ substantially from the 120 anatomically defined regions in AAL in terms of both parcellation granularity and underlying organizational principles. The results demonstrate that our method still improves performance under this alternative atlas AICHA, with overall results comparable to those obtained using the AAL-based atlas. This suggests that the effectiveness of our approach is not highly sensitive to the specific choice of brain atlas, further supporting our central motivation that the key contribution lies in incorporating anatomical constraints into foundation model training rather than relying on a particular parcellation scheme. We are willing to include the completed experiment in an updated version. We will also incorporate these results into the appendix of the revised manuscript.

AuthorsOpenReview ↗

For the Reviewer dJML with Question 5 and 6

For Question 5, Figure 2 shows a consistent pattern across two readouts. Q-Former underperforms mean pooling on SAM-Med3D, and introducing atlas-based regional queries improves on Q-Former. The clearest evidence comes from the comparison with static atlas pooling. Replacing mean pooling with static atlas pooling raises AUC by 4.4 percentage points, the largest such gain among the four backbones. Adding contextual refinement to the atlas-initialized queries then lowers AUC by 4.5 percentage points and accuracy by 6.0 percentage points relative to static atlas pooling. These results indicate that anatomical partitioning is useful for SAM-Med3D and that, on the Figure 2 splits, contextual refinement gives up the AUC gain achieved by static pooling.

A plausible explanation is that SAM-Med3D’s dense volumetric segmentation pretraining (Wang et al.) produces features whose regional averages already retain information useful for AD classification. In our design, the atlas-pooled representations initialize the queries, but cross-attention re-aggregates the encoder features without an explicit residual connection that preserves the original regional representations. Moreover, refinement is trained to predict regional volumes. An update that favors volume prediction may attenuate regional differences that are useful for diagnosis. The ablations establish where the performance drops, while the roles of the training objective and the missing residual connection remain open.

A reliable, generally applicable diagnostic for predicting when contextual refinement benefits AD classification remains unclear. Attention entropy and regional token redundancy may offer useful clues, but neither directly measures whether refinement preserves information relevant to diagnosis. Their predictive value therefore remains an open question. We plan to release our code publicly and will share any additional validation results in the same repository. We welcome independent evaluations that use or build on our implementation to investigate when refinement helps.

References

H. Wang et al., "SAM-Med3D: A Vision Foundation Model for General-Purpose Segmentation on Volumetric Medical Images," in IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 10, pp. 17599-17612, Oct. 2025, doi: 10.1109/TNNLS.2025.3586694.

For Question 6, we thank the reviewer for this suggestion. We agree that a concise summary of the parameter and computational overhead would improve the clarity of the main manuscript.

(i) Exact trainable-parameter counts. As detailed in Appendix C and Table A6, AGQ consists of a region identity embedding, one multi-head cross-attention layer, and an output layer normalization. The total number of trainable parameters is

P_AGQ = Kd + (4d² + 4d) + 2d = 4d² + 126d, where K = 120 and d denotes the feature dimension of the pretrained encoder.

Accordingly, the exact parameter counts are 638,208 for SAM-Med3D, 2,456,064 for BrainIAC, 1,113,088 for MedicalNet, and 638,208 for SwinUNETR.

Mean Pooling introduces no trainable aggregation parameters, whereas AGQ and the generic Q-Former have exactly the same number of trainable parameters under our matched experimental configuration. The atlas-to-token assignment matrix W is determined solely by atlas geometry and encoder tokenization and therefore introduces no additional trainable parameters.

(ii) Computational overhead. Compared with Mean Pooling, AGQ introduces a lightweight atlas-guided aggregation operation and one cross-attention layer. Compared with the generic Q-Former, AGQ uses the same attention architecture, number of region queries, and feature dimensions, with the only additional operation being the atlas-guided initialization WP.

This additional aggregation has a computational complexity of O(KNd), where N is the number of pretrained spatial tokens. Consequently, AGQ and the generic Q-Former share the same asymptotic computational complexity, dominated by the attention projections and cross-attention operations.

Furthermore, AnatoQ does not modify or retrain the pretrained foundation model. The backbone features can be extracted once and reused, while AGQ is pretrained separately. During downstream adaptation, both the backbone and AGQ remain frozen, and only the lightweight prediction head is optimized.

We will incorporate a concise summary of these parameter counts and theoretical computational costs into the main manuscript, while retaining the detailed component-wise analysis in Appendix C.

Reviewer_F795OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes AnatoQ, an atlas-guided Q-Former that adapts pretrained brain MRI features into anatomically structured region representations. The approach is lightweight and evaluated across multiple pretrained backbones and downstream tasks. While the motivation is reasonable and the results show some improvements, I find the methodological novelty limited and the experimental evidence insufficient to support the central claims.

Critical strengths

The paper is well-motivated and presents a simple, lightweight framework to use anatomical priors with pretrained brain patch features. The evaluation across multiple backbones and downstream tasks demonstrates its potential applicability.

Critical weaknesses
  1. Weak baselines. The main experiments primarily compare against mean pooling, which is a rather weak, non-trainable baseline. Stronger alternatives, such as learnable attention pooling or other lightweight trainable pooling modules, are necessary to establish the effectiveness of the proposed method. Although a generic Q-Former is included in the ablation study, it is not systematically compared across the main experiments.
  2. Unverified anatomical correspondence. The anatomical correspondence of region queries is only at initialization. Since the subsequent cross-attention operates globally without spatial constraints, there is no guarantee that the learned region tokens continue to represent their designated anatomical regions. The attribution visualizations do not sufficiently validate this correspondence. More direct evidence, such as attention localization or region-specific perturbation experiments, is needed to support the interpretability claims.
  3. Inconclusive ablation studies. The ablation results do not convincingly demonstrate the individual contributions of atlas-guided initialization, region embeddings, and cross-attention refinement. In several cases, simpler variants match or outperform the full model. These observations raise questions about which components are actually responsible for the improvements. More systematic controlled experiments are needed to establish the necessity and effectiveness of each component.
Questions for authors
  1. Can the authors provide comparisons with stronger trainable pooling baselines, such as attention pooling, under comparable training settings?
  2. How can the authors verify that the learned region queries remain anatomically aligned with their designated brain regions after global cross-attention?
  3. Can the authors provide more conclusive ablation experiments to isolate the contributions of atlas-guided initialization, region embeddings, and cross-attention refinement?
2 comments
AuthorsOpenReview ↗

For Reviewer F795, we believe that our statement will help clarify for Q1 and Q2.

We sincerely appreciate the reviewer's careful and insightful assessment of our work, particularly their thoughtful understanding of our core motivation and methodological design. The reviewer's constructive questions provide a valuable opportunity to further clarify our methodological choices, underlying rationale, and key contributions. We hope that our responses will provide a clearer and more comprehensive understanding of our work and support a fair and informed reassessment of its contributions and overall evaluation.

For Question 1, we thank the reviewer for this valuable suggestion. Our existing ablation study already includes a strong, established trainable attention-based pooling baseline, namely Q-Former, which performs adaptive feature aggregation through learnable queries and multi-head cross-attention.

Specifically, as detailed in Table A7, the Q-Former baseline employs K = 120 learnable queries to aggregate spatial features through global cross-attention, without anatomical guidance. This provides a direct comparison with AGQ, whose queries additionally incorporate atlas-guided anatomical initialization.

Importantly, Q-Former and AGQ are evaluated under matched experimental conditions, including the same frozen backbone features, data splits, downstream prediction head, optimizer settings, and Stage-1 pretraining protocol. Both methods use the same number of queries and trainable parameters (0.64M–2.46M across the four encoders; Table A6), ensuring that the comparison isolates the contribution of anatomical guidance rather than differences in model capacity or training configuration.

As shown in Figure 2, AGQ consistently outperforms Q-Former in AUC across all four pretrained encoders, with improvements of 0.4–0.9 percentage points, and achieves higher or equal classification accuracy on three of the four encoders.

These results demonstrate that AGQ provides measurable benefits over an established, fully trainable attention-based aggregation approach under comparable training settings. In particular, the improvements highlight the value of anatomy-guided query initialization beyond generic learnable attention pooling, rather than simply reflecting the use of a more expressive pooling mechanism.

For Question 2, we thank the reviewer for this insightful question. AnatoQ maintains anatomical correspondence through atlas-guided query initialization and region-level anatomical supervision, while global cross-attention enables the integration of complementary information from other brain regions. We provide the following evidence supporting the anatomical relevance of the refined region queries.

First, anatomical correspondence is explicitly established during query initialization. Each regional query is initialized using an atlas-derived spatial assignment matrix, which aggregates encoder features according to their overlap with the corresponding predefined brain region. Furthermore, Stage-1 anatomical pretraining supervises each region token using its corresponding atlas-derived gray matter volume, encouraging the refined queries to retain region-specific anatomical information even after global cross-attention.

Second, the learned representations exhibit anatomically meaningful and task-discriminative attribution patterns. As shown in Figure 3, AGQ identifies distinct task-relevant brain regions across cognitive and disease-related prediction tasks. Importantly, the highly attributed regions are consistent with established neuroimaging findings, suggesting that global contextual aggregation does not eliminate meaningful anatomical differentiation.

Together, the atlas-guided initialization, region-specific anatomical supervision, and task-dependent attribution patterns provide complementary evidence that the refined queries retain meaningful anatomical associations after global cross-attention.

AuthorsOpenReview ↗

For Reviewer F795, we believe that our statement will help address the concerns raised in the Q3

For Question 3, we thank the reviewer for this important question. Our existing ablation study (Figure 2 and Table A7) systematically investigates the roles of atlas-guided initialization, region identity embeddings, and cross-attention refinement through five complementary configurations. We clarify the design and findings below.

VariantAtlas-guided initializationRegion embeddingsCross-attention
Mean Pooling–––
Static Atlas✓––
Atlas Query Only✓✓–
Q-Former–✓✓
AGQ (Full)✓✓✓

(i) Contribution of atlas-guided initialization. Comparing AGQ with Q-Former directly isolates the contribution of atlas-guided query initialization while retaining learnable queries and cross-attention. Both variants use identical trainable parameter counts and matched experimental settings, including the same frozen backbone features, data splits, prediction heads, and pretraining protocol. As shown in Figure 2, AGQ achieves higher AUC across all four encoders (+0.4 to +0.9 percentage points) and higher or equal accuracy on three of the four encoders. Furthermore, Static Atlas improves AUC over Mean Pooling across all four encoders (+2.4 to +4.2 percentage points), providing additional evidence for the effectiveness of anatomical guidance.

(ii) Role of region identity embeddings. Region identity embeddings serve as learnable query representations that complement atlas-guided regional features. Specifically, AGQ constructs its queries as Q = WP + E_region, combining subject-specific anatomical information (WP) with learnable region-specific embeddings (E_region). This construction allows the attention mechanism to distinguish different anatomical regions while adaptively aggregating global contextual information. The existing ablations examine this design through complementary configurations: Q-Former uses learnable queries alone, Atlas Query Only combines atlas-guided features and region embeddings without cross-attention, and AGQ integrates all three components. These comparisons characterize how region embeddings participate in query construction.

(iii) Contribution of cross-attention refinement. Comparing AGQ with Atlas Query Only isolates the effect of cross-attention refinement while retaining the same atlas-guided query construction. As shown in Figure 2, removing cross-attention leads to accuracy decreases of 4.5–12.1 percentage points and AUC decreases of 1.8–12.3 percentage points across BrainIAC, MedicalNet, and SwinUNETR, with SAM-Med3D being an exception. These results demonstrate the importance of adaptive contextual refinement beyond anatomically initialized queries.

Together, the existing ablation results provide complementary evidence that AGQ benefits from integrating anatomical priors, learnable region representations, and adaptive cross-attention. In particular, the comparisons demonstrate that anatomy-guided query initialization and contextual refinement offer distinct and complementary benefits, rather than the improvements arising solely from generic learnable attention or static atlas aggregation.

Reviewer_rapbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a framework called AnatoQ and it uses brain atlases to improve the representations learned by pretrained MRI foundation models. The main idea is to organize image patch features into meaningful brain regions. The authors use the AAL2 atlas as standard to build 120 region queries, and then interact with the pretrained features through cross-attention. The backbone is kept frozen, and the proposed module is first trained to predict regional gray matter volumes before being used for downstream tasks. The authors test AnatoQ on ADNI and OASIS with four different pretrained models. They test on Alzheimer's disease diagnosis, age prediction, and several cognitive score predictions. The results show improvements over mean pooling in many settings. The paper also includes ablation experiments and regional attribution maps to examine the role of atlas guidance and how the learned features relate to brain anatomy.

Critical strengths

I think I like the idea of using anatomical information to guide the adaptation procedure of pretrained brain MRI models. Instead of simply averaging all patch tokens, AnatoQ builds region queries from a brain atlas and uses cross-attention to extract features. The method is relatively simple and can be applied to different frozen backbones without changing their architectures. The experiments cover four pretrained models and two MRI datasets, with both Alzheimer's disease classification and cognitive score prediction. The improvements with MedicalNet look quite noticeable.

Critical weaknesses

I am not quite sure where the performance gains actually come from. In Stage 1, AGQ is trained to predict gray matter volumes for 120 brain regions, which are already closely related to Alzheimer's disease, age, and cognitive scores. However, the mean pooling baseline does not receive the same training. This makes it difficult to tell whether the improvement comes from AGQ itself or from the additional anatomical supervision. I think a comparison with a simpler model using the same pretraining would be important. It would also be interesting to see how well a model using only the 120 gray matter volumes performs. Without these comparisons, I find it hard to judge how much the proposed architecture really contributes.

I also have some doubts about the anatomical interpretation. Each region query can attend to the whole brain, and all region tokens are averaged before prediction. So I am not sure whether they still represent their assigned regions. The attribution results do not fully answer this, especially since some unexpected regions appear across different tasks. A random parcellation baseline would be helpful here.

Questions for authors
  1. I am curious how much of the improvement actually comes from the gray matter volume pretraining. Since these volumes are already related to Alzheimer's disease and cognitive scores, have the authors tried using the 120 regional volumes directly as a baseline? I would also like to see whether a simpler pooling method could achieve similar results with the same pretraining objective.
  2. I found the atlas-guided query design interesting, but how important is the choice of AAL2? Would a random parcellation with a similar number and size distribution of regions give comparable results?
  3. Since each query can attend to the whole brain, I wonder whether the attention actually focuses on its assigned region. Have the authors looked at the attention maps? It would be interesting to see whether restricting attention to the corresponding anatomical region changes the performance.
  4. I noticed that Cerebellum 10 and Pallidum appear repeatedly among the top attributed regions, even for quite different tasks. Is there a reason for this? Could these patterns partly reflect differences in region size or feature distributions rather than task-specific anatomical information?
  5. Appendix A.3 mentions that regional pretraining can use all subjects. Does this include the downstream test subjects? If so, I would be interested in seeing whether excluding them from Stage 1 changes the results.
6 comments
AuthorsOpenReview ↗

For reviewer rapb with the Question1

We sincerely appreciate the reviewer's careful and insightful assessment of our work, particularly their thoughtful understanding of our core motivation and methodological design. The reviewer's constructive questions provide a valuable opportunity to further clarify our methodological choices, underlying rationale, and key contributions. We hope that our responses will provide a clearer and more comprehensive understanding of our work and support a fair and informed reassessment of its contributions and overall evaluation.

For Question 1, (i) Gains without Stage-1 pretraining. The Static Atlas (WPWP) baseline performs anatomically structured pooling without trainable aggregation parameters or Stage-1 pretraining. Nevertheless, it improves over mean pooling in AUC across all four encoders (+2.4 to +4.2 pp) and in accuracy on three of the four encoders (+1.5 to +9.0 pp, with equal accuracy on MedicalNet; Figure 2). This indicates that anatomically structured aggregation itself can provide benefits independently of GMV pretraining, although it does not directly quantify the pretraining contribution within full AGQ.

(ii) Comparison under identical GMV pretraining. Our Q-Former baseline uses the same Stage-1 regional GMV pretraining objective and schedule as AGQ, with identical frozen encoder features, trainable parameter counts, and downstream training settings (Appendix D). Despite these controlled conditions, AGQ achieves higher AUC across all four encoders (+0.4 to +0.9 pp) and higher or equal accuracy on three of the four encoders (Figure 2). These results indicate that the shared GMV pretraining objective alone does not explain AGQ's performance, and that atlas-guided aggregation provides additional benefits over atlas-free learnable token aggregation.

(iii) Regional GMVs as direct predictive features. We acknowledge the reviewer's suggestion to use the 120 regional GMV measurements directly as a baseline. Such a comparison would help assess how much predictive information is already captured by regional volumes alone. Importantly, our framework uses GMVs as auxiliary supervision during Stage-1 pretraining, rather than as direct downstream prediction inputs. The downstream predictions are based on representations aggregated from frozen 3D MRI encoder features. Therefore, a GMV-only baseline would provide a complementary assessment of the predictive value of regional volumetric information, rather than a direct replacement for the proposed aggregation mechanism.

(iv) Independent cross-dataset validation. We further evaluate AGQ on OASIS, an independent external dataset, where it improves accuracy over mean pooling on three of the four encoders (+1.5 to +8.3 pp; Table 1). This provides additional evidence of the generalizability of the proposed aggregation framework beyond the original brain MRI data cohort.

Overall, our existing comparisons demonstrate that anatomically structured pooling provides benefits without GMV pretraining, while AGQ retains performance advantages over learnable aggregation under identical pretraining conditions. These findings support the complementary contributions of anatomical organization and adaptive aggregation, rather than attributing the observed improvements solely to regional GMV supervision.

AuthorsOpenReview ↗

For reviewer rapb with the Question2

For the Question2, we thank the reviewer for this valuable question. We would like to clarify that AAL2 serves as an anatomical reference in our framework, rather than an essential or uniquely optimal atlas. There is currently no universally accepted brain parcellation standard in neuroimaging, as different atlases vary in their anatomical definitions, organizational principles, and regional granularity. Therefore, a representative atlas must be selected for experimental evaluation.

(i) Motivation for selecting AAL2. We adopted AAL2 (120 regions) primarily because it is a widely used and well-established anatomical reference in neuroimaging research, particularly in studies of neurodegenerative diseases. Importantly, our research does not aim to identify the optimal brain atlas, but to investigate whether anatomically guided adaptive aggregation can consistently improve representations extracted from different frozen foundation models. Accordingly, our experimental design focuses on evaluating AGQ across four distinct pretrained encoders under a common anatomical reference, rather than optimizing the atlas for a particular model. The consistent improvements across these encoders support the applicability of our approach to different backbone architectures.

(ii) Additional evaluation using AICHA. To further investigate whether our method depends on the specific choice of AAL2, we have conducted an additional experiment using AICHA (384 regions), which differs substantially from AAL2 in both regional granularity (384 vs. 120 regions) and parcellation principles (functional connectivity-based vs. anatomical definitions). Our results indicate that AGQ continues to achieve performance improvements under AICHA, with overall performance comparable to that obtained using AAL2. This provides additional evidence that our framework is not restricted to a particular atlas or a fixed number of anatomical regions.

(iii) Regarding random parcellation and anatomical guidance. We appreciate the reviewer's suggestion of comparing against random parcellations with matched region counts and size distributions. Such an experiment would specifically examine the contribution of anatomically meaningful boundaries relative to arbitrary spatial partitioning. While AICHA is not a random parcellation, the consistent results obtained using two substantially different atlases provide evidence that the effectiveness of AGQ does not rely exclusively on the specific regional definitions of AAL2. Furthermore, our existing ablations (Figure 2 and Appendix D) demonstrate the benefits of atlas-guided aggregation over atlas-free learnable aggregation under matched frozen encoder features and training settings.

We will provide the complete results of the additional AICHA experiment in this rebuttal thread at least two days before the rebuttal deadline and incorporate them into the appendix of the revised manuscript.

Overall, the central contribution of our work is a generalizable atlas-guided adaptive aggregation framework applicable to different frozen foundation models, rather than the selection of a particular brain atlas. AAL2 provides a widely recognized anatomical reference for controlled evaluation, while the additional AICHA results further support the flexibility and generalizability of our approach.

AuthorsOpenReview ↗

For the Review rapb with the Question 4, Question 5

For the Question 4 Response: We thank the reviewer for this insightful observation. Traditionally, cognitive neuroscience research has predominantly focused on cerebral regions, while the cerebellum has received comparatively less attention. However, recent neuroscience studies have increasingly highlighted the important role of the cerebellum in higher-order cognitive functions. For example, Wang et al. (2025) emphasized that the cerebellum has historically been overlooked in neuroimaging research despite its substantial contributions to cognition, while Saadon-Grosman et al. (2024) demonstrated its involvement in multiple cognitive domains, including cognitive control, language, social cognition, and memory-related processing.

Importantly, our framework identifies cerebellar regions as highly attributed across several cognitive tasks, which closely aligns with these recent neuroscientific findings. This agreement with emerging neuroscience evidence provides additional support for the effectiveness and neuroscientific interpretability of our approach. In particular, it suggests that our attention maps can capture neurologically meaningful information consistent with recent advances in understanding the cerebellum's contributions to cognition, rather than merely highlighting traditionally well-studied cerebral regions.

Regarding the repeated appearance of Cerebellum 10 and Pallidum, this may partly reflect the relatively small anatomical extent of the cerebellum and the potential overlap of cerebellar involvement across different cognitive tasks. Nevertheless, the overall attention patterns across brain regions exhibit distinct distributions for different tasks, rather than consistently highlighting the same several regions, as shown in Table A8 and Table A9. Therefore, this task-dependent differentiation based on overall brain regions, together with the consistency of cerebellar findings with recent neuroscience research, further supports the interpretability of our framework and we think these may provide useful insights for future neuroscience investigations.

References

[1] Wang, B., LeBel, A., & D'Mello, A. M. (2025). Ignoring the cerebellum is hindering progress in neuroscience. Trends in Cognitive Sciences. https://doi.org/10.1016/j.tics.2025.01.004

[2] Saadon-Grosman, N., et al. (2024). Within-individual organization of the human cognitive cerebellum: Evidence for closely juxtaposed, functionally specialized regions. Science Advances. https://doi.org/10.1126/sciadv.adq4037

For the Question 5 Response: We thank the reviewer for raising this important question. We would like to clarify that the test set is separated before any model training or anatomical pretraining is performed.

All datasets used in our experiments consist of baseline scans without repeated subjects. After splitting the data, test subjects are completely excluded from all training stages, including Stage-1 anatomical pretraining and downstream task training. No images or information from the held-out test subjects are used during model optimization. Therefore, our evaluation does not follow a transductive adaptation setting. The reported test performance reflects generalization to entirely unseen subjects, with strict subject-level separation between training and testing.

In additonaly, Correction to Appendix A.3. The sentence "region-level pretraining ... can use all 1,416 ADNI subjects" was imprecise. It was meant to say that Stage 1 requires no diagnostic labels, not to describe the data split. In our implementation, the downstream test subjects are excluded from Stage 1, which is trained on the remaining ADNI subjects.

OASIS as independent evidence. No OASIS subject was used in Stage 1. The AGQ evaluated on OASIS was pretrained on ADNI only and kept frozen, so the OASIS results in Table 1 reflect generalization to unseen subjects from a different cohort.

Revision. We will provide further clarification in the Appendix A.3.

Reviewer_dZ14OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AnatoQ, an anatomy-guided representation adaptation framework for structural MRI foundation models. Its core module, the Atlas-Guided Q-Former (AGQ), constructs region-specific queries based on the spatial overlap between a fixed brain atlas and the encoder's spatial tokens. These queries are subsequently refined through cross-attention to extract regional representations from pretrained encoder features. The authors pretrain AGQ using a regional gray matter volume prediction objective. During downstream evaluation, both the foundation model and AGQ are frozen, and only task-specific prediction heads are trained. Experiments are conducted across four pretrained encoders and multiple downstream tasks, including Alzheimer's disease classification, age prediction, and cognitive score prediction. The results show that the proposed approach generally outperforms conventional mean pooling.

Critical strengths

S1. A practically meaningful problem with a flexible adaptation framework. Effectively leveraging spatial representations from existing MRI foundation models is an important research problem. AnatoQ introduces a lightweight adapter to construct region-level representations without modifying or fine-tuning the pretrained backbone. This design offers practical advantages, as it can be applied to different pretrained models with relatively low computational overhead.

S2. Evaluations across multiple encoders and downstream tasks. The authors evaluate their approach using four pretrained encoders with different architectures and multiple downstream prediction tasks, demonstrating notable performance improvements in several settings. The paper also includes comparisons with static atlas pooling, a conventional Q-Former, and several architectural variants, providing useful empirical evidence for understanding the contributions of different components.

Critical weaknesses

W1. The necessity of fixed anatomical priors and their advantages over adaptive spatial organization remain insufficiently justified.

The central motivation of this work is that regularly partitioned patches do not align well with anatomical brain regions, motivating the use of a fixed atlas to reorganize patch-level representations into region-level features. However, this observation does not necessarily imply that predefined anatomical boundaries provide the most suitable spatial organization for representation learning and downstream prediction. In practice, predictive information in neuroimaging may exhibit multi-scale, cross-regional, and subject-specific characteristics. Alternative approaches, such as dynamic patch partitioning, adaptive token aggregation, or data-driven regional organization, may also learn task-relevant representations while preserving fine-grained spatial information.

Although the authors include a conventional Q-Former as a baseline, its queries are shared learnable embeddings, whereas AGQ explicitly conditions its queries on subject-specific input features. Consequently, this comparison does not fully disentangle whether the performance gains originate from the fixed anatomical prior or from input-conditioned query initialization and adaptive feature aggregation. This concern is also reflected in the experimental results. For MedicalNet, the conventional Q-Former achieves the same classification accuracy as the full AGQ model, with only a marginal difference in AUC. These results leave the independent contribution of the fixed anatomical prior unclear.

Furthermore, the experiments exclusively adopt the AAL2 atlas with 120 regions, without systematically investigating the effects of different atlases, parcellation granularities, or adaptive spatial partitions. This limitation is particularly relevant for backbones with coarse spatial token grids, where multiple neighboring anatomical regions may depend on substantially overlapping sets of tokens. Whether fixed region identities genuinely improve regional specificity under such conditions remains to be demonstrated. Overall, the paper would benefit from stronger justification and experimental evidence regarding the unique advantages of fixed atlas-based organization over data-driven aggregation methods under comparable parameter counts and training budgets.

W2. The independent contribution of regional volume pretraining and the anatomical interpretability claims are not sufficiently established.

Before downstream evaluation, AGQ is additionally pretrained using regional gray matter volume prediction as an auxiliary supervision signal. However, gray matter volume is itself associated with Alzheimer's disease, aging, and several cognitive measures. Therefore, the observed downstream improvements may partly arise from additional task-relevant morphological supervision, rather than from learning a fundamentally better organization of regional representations.

The current experiments lack a direct comparison with the full AGQ model without regional volume pretraining, as well as comparisons with alternative pretraining objectives. Consequently, the specific contribution of this auxiliary training stage remains unclear.

Another concern relates to the anatomical interpretability claims. The authors use gradient-based attribution over regional tokens to identify anatomical structures considered important for downstream predictions. However, AGQ allows each region query to aggregate information from the entire brain through global cross-attention. Therefore, the importance of a particular region token does not necessarily indicate that the corresponding anatomical region in the original MRI provides the predictive evidence.

The current attribution experiments primarily demonstrate the concentration of regional importance scores and the stability of their rankings across subject groups. While these findings support the relative consistency of task-related regional representations, they do not establish that the model accurately localizes predictive evidence in the original anatomical space. In particular, a Top-20 attribution mass exceeding the uniform baseline does not, by itself, validate the anatomical correctness of the resulting attribution maps.

Therefore, the claims regarding anatomical interpretability would benefit from more direct experimental validation.

Questions for authors

Q1. Why is a fixed anatomical atlas preferred over adaptive spatial organization?

Could the authors further explain the advantages of fixed AAL2 parcellation over dynamic patch partitioning, learnable spatial aggregation, or input-conditioned query mechanisms? Given that the proposed framework does not modify the pretrained backbone, would it be possible to compare fixed atlas-based aggregation with data-driven adaptive token aggregation using the same frozen encoder features, while controlling for parameter count, pretraining objectives, and training budget?

Q2. How much of the performance improvement comes from anatomical pretraining?

Could the authors provide an ablation of AGQ without regional gray matter volume pretraining, as well as comparisons with alternative auxiliary pretraining objectives? Additionally, has the necessity of the 30% token masking strategy been systematically evaluated?

Q3. How sensitive is the method to atlas selection and parcellation granularity?

The current experiments exclusively use the AAL2 atlas with 120 regions. How sensitive is the performance to the choice of atlas and the number of regions? The authors also attribute the stronger improvements on MedicalNet to its finer spatial token resolution. Could this explanation be further validated by controlling the feature resolution within the same backbone?

Q4. Do regional token attributions faithfully correspond to anatomical evidence in the input MRI?

Since each region query can aggregate information from the entire brain, could the authors validate whether regional attribution scores correspond to the actual importance of the associated anatomical structures in the input MRI? For instance, input-level ROI perturbation or regional masking experiments could help assess the consistency between token-level attribution and anatomical evidence.

Q5. Could the authors clarify the relationship between anatomical pretraining and downstream evaluation splits?

The appendix states that regional volume pretraining does not require diagnostic labels and can therefore use all subjects. Does this include subjects from the downstream validation and test sets? If so, could the authors clarify whether the evaluation follows a transductive adaptation setting and whether the reported performance reflects generalization to completely unseen subjects?

5 comments
AuthorsOpenReview ↗

For Reviewer dZ14, we believe that our statement will help clarify the intended scope and contributions of our work

For the Question 1, We thank the reviewer for this insightful question. We would like to clarify two important points: (i) AAL2 serves as an anatomical reference rather than a rigid spatial constraint, and AGQ combines this anatomical prior with input-adaptive cross-attention; and (ii) our existing ablation study (Figure 2 and Appendix D) already includes a controlled comparison against learnable, atlas-free token aggregation using the same frozen encoder features. We elaborate below.

(i) Why use AAL2 as an anatomical prior?

Our motivation for adopting AAL2 is not to assume that a fixed anatomical parcellation is universally optimal, but to provide a consistent and interpretable spatial reference across subjects, tasks, and pretrained encoders. AAL2 is also a widely adopted anatomical parcellation in neuroimaging research.

Unlike purely data-driven aggregation, where learned tokens do not necessarily maintain stable anatomical correspondence, atlas-guided queries are anchored to predefined brain regions. This provides two practical advantages:

  • Anatomical consistency and interpretability: Each aggregated token retains an explicit anatomical reference, enabling region-level attribution and comparison across subjects and encoders without requiring post-hoc token-to-region matching.

  • Reproducibility: The resulting regional attribution patterns are concentrated rather than uniformly distributed (Top-20 attribution mass: 25.4–40.2%, compared with 16.7% under a uniform distribution; Tables A8–A9) and show high consistency across disjoint subject splits (Spearman ρ\rho = 0.981–1.000; Table 3).

Importantly, AGQ is not restricted to static atlas pooling. The atlas provides an anatomically grounded query initialization (Q=WP+EregionQ = WP + E_{\mathrm{region}}, Eq. 5), while subsequent global cross-attention (Eq. 6) enables input-dependent aggregation over all encoder tokens. Thus, AGQ preserves anatomical correspondence while allowing adaptive integration of spatial information beyond predefined regional boundaries.

(ii) Controlled comparison using the same frozen encoder features

Our ablation study compares AGQ against two complementary alternatives:

  • Q-Former (MHA(E,P)\mathrm{MHA}(E,P), Table A7): Data-driven learnable token aggregation without an anatomical prior.
  • Static Atlas (WPWP): Fixed atlas-based pooling without adaptive cross-attention.

All variants use the same frozen encoder features, data splits, K=120K=120 output tokens, prediction head, optimizer settings, and Stage-1 pretraining objective and schedule (Appendix D). Since WW is parameter-free, the Q-Former variant has the same number of trainable parameters as AGQ (0.64M–2.46M, Table A6).

The following table reports performance differences relative to full AGQ, in percentage points (pp). Negative values indicate lower performance than AGQ.

EncoderQ-Former Δ\DeltaACC / Δ\DeltaAUCStatic Atlas Δ\DeltaACC / Δ\DeltaAUC
BrainIAC−3.0 / −0.9−3.0 / −5.2
MedicalNet0.0 / −0.4−4.5 / −7.6
SwinUNETR−3.0 / −0.7−1.5 / −0.4
SAM-Med3D+3.0 / −0.5+7.5 / +2.1

Compared with the atlas-free Q-Former at identical parameter count, AGQ achieves higher AUC across all four encoders (0.4–0.9 pp) and higher or equal accuracy on three of the four encoders. This suggests that anatomical guidance provides complementary benefits beyond learnable aggregation alone.

Compared with Static Atlas pooling, AGQ improves both ACC and AUC on three of the four encoders (1.5–4.5 pp ACC and 0.4–7.6 pp AUC), supporting the benefit of adaptive cross-attention over purely fixed anatomical pooling.

We acknowledge that SAM-Med3D is an exception, where simpler alternatives are competitive or outperform AGQ on certain metrics. Its mean-pooled features already achieve the strongest performance among the four encoders (Table 1), potentially leaving less room for additional aggregation-based improvements.

Overall, these controlled comparisons suggest that anatomical guidance and adaptive aggregation provide complementary benefits. Our choice of AAL2 is motivated by anatomical consistency, interpretability, and reproducibility, rather than an assumption that fixed spatial organization is universally superior.

AuthorsOpenReview ↗

For the Reviewer dZ14, the Question 2

For Question 2, we thank the reviewer for raising this important question. While our current experiments do not directly isolate the quantitative contribution of Stage-1 pretraining, they provide three complementary pieces of evidence demonstrating that the improvements cannot be attributed solely to anatomical pretraining.

(i) Gains without Stage-1 pretraining. Static Atlas (WPWP) has no trainable aggregation parameters and does not require Stage-1 pretraining. Nevertheless, it improves over mean pooling in AUC across all four encoders (+2.4 to +4.2 pp) and in accuracy on three of the four encoders (+1.5 to +9.0 pp, with equal accuracy on MedicalNet; Figure 2). This demonstrates that anatomically structured aggregation can provide benefits independently of Stage-1 pretraining, although this comparison does not directly quantify the contribution of pretraining within full AGQ.

(ii) Gains with pretraining held constant. The Q-Former variant receives the same Stage-1 pretraining and has the same trainable parameter count as AGQ, using identical frozen encoder features and training settings (Appendix D). Nevertheless, AGQ achieves higher AUC across all four encoders (+0.4 to +0.9 pp) and higher or equal accuracy on three of the four encoders (Figure 2). This demonstrates that the performance improvements cannot be explained solely by the shared pretraining procedure, supporting the additional contribution of atlas-guided aggregation.

(iii) Independent cross-dataset validation. Stage-1 pretraining uses only ADNI training images without diagnostic or cognitive labels. We further evaluate AGQ on OASIS, an independent external dataset, where it improves accuracy over mean pooling on three of the four encoders (+1.5 to +8.3 pp; Table 1). This experiment is intended to evaluate the cross-dataset generalizability of our framework, rather than to isolate the contribution of pretraining. The results provide additional evidence that the proposed aggregation mechanism remains effective beyond the original ADNI cohort.

What's more, for the 30% token masking strategy: Token masking is a regularizer used only in Stage-1 pretraining (Appendix B.3, Table A4). It is not applied in Stage 2 or at inference, and we do not claim it as a contribution or as a necessary component of AGQ.

We set ρ = 0.3 in advance as a moderate default and did not tune it on any encoder or downstream task, so the reported results do not benefit from selection over ρ. For this reason we did not include a sweep in the submission. We will revise the wording in Appendix B.3 accordingly.

AuthorsOpenReview ↗

For the Reviewer dZ14 with Question 3

For the Question3, we thank the reviewer for this valuable suggestion. By far, there is no universally standardized brain atlas in clinical neuroscience, as different atlases vary in anatomical definitions and regional granularity. Importantly, our primary contribution is not the selection or optimization of a specific atlas, but the introduction of anatomical constraints into brain imaging foundation models during encoder training. This encourages the encoder to learn not only voxel-level imaging features but also anatomically informed representations. Notably, the atlas is used only during training and is not required at inference time.

Our choice of the AAL2 with 120-region atlas is motivated by its widespread adoption and practical relevance in AD neuroimaging. AAL-based anatomical parcellation is widely used in brain imaging research, particularly for regional analysis of neurodegenerative diseases. Furthermore, regional brain segmentation schemes involving approximately 120 regions are also commonly adopted in clinical neuroimaging workflows and imaging analysis systems, including those associated with major medical imaging manufacturers such as GE Healthcare and Siemens Healthineers. Therefore, we selected this widely recognized anatomical reference to ensure that our framework is grounded in a clinically relevant and interpretable brain parcellation scheme. More importantly, our experimental design focuses on validating the effectiveness and generalizability of the proposed anatomical constraints across different foundation models, rather than optimizing performance for a particular atlas. The consistent improvements observed across multiple foundation models support the applicability of our approach beyond a single encoder architecture.

Furthermore, we have also conducted an additional experiment using one alternative brain atlas AICHA with 384 regions, which differ substantially from the 120 anatomically defined regions in AAL in terms of both parcellation granularity and underlying organizational principles. The results demonstrate that our method still improves performance under this alternative atlas AICHA, with overall results comparable to those obtained using the AAL-based atlas. This suggests that the effectiveness of our approach is not highly sensitive to the specific choice of brain atlas, further supporting our central motivation that the key contribution lies in incorporating anatomical constraints into foundation model training rather than relying on a particular parcellation scheme.We are willing to include the completed experiment in an updated We will complete the additional experiments and provide the full experimental results in this rebuttal thread at least two day before the rebuttal deadline. We will also incorporate these results into the appendix of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.