acceptodds
Withdrawn from ICLR 2027

STH-Brain: A Hierarchical Spatiotemporal Brain Time-Series Foundation Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Resting-state functional magnetic resonance imaging (fMRI) captures spontaneous fluctuations in blood-oxygen-level-dependent signals across the brain. Differences in temporal sampling complicate consistent modeling across scans, while spatial aggregation can obscure the temporal context of individual brain regions. We propose STH-Brain, a brain time-series foundation model combining physical spatiotemporal encoding with hierarchical representation learning. Physical-Time Rotary Encoding maps elapsed seconds to rotary phases, while Spatial-Aware Encoding incorporates atlas coordinates into spatial attention. The hierarchy models temporal context within each region of interest (ROI) and retains regional states alongside network representations. During self-supervised pretraining, dual-level latent prediction supervises both levels, with visibility-based routing selecting same-ROI or network memory for regional prediction according to the available context. Across seven disorder-classification tasks, STH-Brain achieves the best mean performance among the evaluated baselines, improving accuracy by an average of 5.91 percentage points over the strongest baseline for each task. It also achieves the best mean results for all three age-prediction metrics across seven disease groups and outperforms the compared models on the evaluated external-cohort and zero-shot transfer tasks. Interpretability analyses reveal shared and distinct regional patterns across disorders. Code is available at https://anonymous.4open.science/r/STH-Brain/.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain temporality, resting-state functional MRI, physical spatiotemporal coding, hierarchical brain network representations, self-supervised learning.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gGUXOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes STH-Brain, a self-supervised foundation model for resting-state fMRI time series. Whereas most prior work builds a functional connectivity (FC) matrix from the whole scan, or from sliding windows, STH-Brain operates directly on ROI time series and introduces two modules: Physical-Time Rotary Encoding (PTRE), which maps elapsed time in seconds to rotary phases, and Spatial-Aware Encoding (SAE), which injects atlas coordinates into within-network spatial attention. These are combined with a hierarchy that retains ROI-level states alongside network-level tokens, trained with dual-level latent prediction and visibility-based routing. Across seven disorder-classification tasks, STH-Brain reports the best results over 13 baselines, and it also reports the best restuls for age prediction across seven disease groups, plus gains on an external SCZ/HC cohort and a zero-shot MDD→BP transfer. Interpretability maps are provided.

Critical strengths

S1. Performance is strong across a broad set of benchmarks, and the evaluation covers a sufficient number of tasks and baselines (13 models, 7 classification tasks, age prediction, external and zero-shot evaluation).

S2. The zero-shot transfer setting (MDD/HC → BP/HC without target labels) is an interesting and underexplored evaluation for fMRI foundation models.

S3. The idea of a hierarchy that retains both ROI-level and network-level representations, rather than collapsing ROIs into network tokens, is appealing.

Critical weaknesses

W1. (Minor) Citation formatting is incorrect throughout: in-text citations appear as “Li et al. (2021)” where a parenthetical citation is intended. Please use \citep / \citet appropriately.

W2. All downstream experiments use a single fixed subject-level split (7:1:2), with the 10 seeds varying only training randomness. It is well known that performance on neuro-disorder benchmarks varies substantially across splits, so the reported standard deviations understate the true variance, and the paired t-tests across seeds do not establish robustness to data partitioning. In addition, several datasets (ABIDE-I/II, ADHD-200, REST-MDD) are multi-site, but the paper does not describe how sites are handled in the partitions (site-stratified, or possibly site-leaking across train/test). Without repeated splits and site-aware partitioning details, I am not fully convinced by the current results.

W3. The main text depends heavily on the appendix. For example, the loss-weight sensitivity results discussed at L453–L457 are not shown in the main paper at all (only Figs. 5–8 in Appendix H.5); if results are discussed in the main text, the corresponding figure should appear there. Similarly, the local temporal encoder that underlies PTRE (L167–171) is only described in Appendix A, including key design choices such as the 8-s blocks and 1-s query grid. This makes the method difficult to understand from the main text alone.

W4. The contribution of PTRE is unclear. (a) The model already interpolates every scan onto a fixed 1-s query grid in 8-s blocks (Appendix A.2), which by itself removes TR heterogeneity; it is not clear what PTRE adds over simply resampling all scans to a common TR. (b) Once blocks are fixed-length and non-overlapping, the block-center timestamp is 8b + 4, so PTRE’s phase is a linear function of the block index and the learnable multiplier can absorb the scale. Unless blocks are missing, PTRE appears nearly equivalent to standard RoPE over block indices. (c) The ablation “w/o PTRE” does not specify what replaces it (block-index RoPE, sinusoidal PE, or no position encoding), so it cannot distinguish “physical time vs. frame index” from “rotary vs. no position encoding.” A comparison against block-index RoPE is needed to support the central claim.

W5. The motivation for spherical coordinates in SAE over Euclidean coordinates (or pairwise Euclidean distance) is not explained, and the paper itself notes that the encoding does not reflect Euclidean distance (Appendix B.1). An ablation comparing spherical rotary encoding against a Euclidean-coordinate or distance-based variant is needed.

W6. The four loss terms are not ablated. The paper reports sensitivity to the loss weights (Appendix H.5), but a sweep over weights does not show whether each term is necessary; a leave-one-out ablation would clarify the contribution of each component of the pretraining objective, which is a core part of the claimed contribution.

Questions for authors

Q1. The CC200 atlas (200 ROIs) is coarser than parcellations used by some baselines (e.g., Brain-JEPA uses 450 ROIs). Did the authors try finer parcellations with STH-Brain? Figure 3 shows CC200 outperforming Power264; if the gap persists at higher resolution, what is the plausible explanation, given that the method is designed to preserve ROI-level states?

Q2. All comparisons are with full fine-tuning. Since the goal of pretraining is a generalizable representation space, did the authors evaluate STH-Brain and the baselines under linear probing (or frozen-encoder) protocols, as done in Brain-Semantoks? This would help separate representation quality from fine-tuning capacity.

Reviewer_gtTnOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The study presents STH-Brain, a brain time-series foundation model combining physical spatiotemporal encoding with hierarchical representation learning. The authors introduce Time Rotary Encoding to address temporal-position ambiguity across sampling intervals and Spatial Aware Encoding to incorporates ROI coordinates into spatial attention. They also propose an ROI-preserving hierarchy to model ROI-specific temporal states. The proposed STH-Brain were evaluated on seven disorder-classification tasks, age prediction, external-cohort evaluation, and zero-shot transfer. The authors reported state-of-the-art performance.

Critical strengths

1.The Time Rotary Encoding strategy that addresses temporal-position ambiguity across sampling intervals. 2.The ROI-preserving hierarchy that is capable of modeling ROI-specific temporal states.

Critical weaknesses

1.The motivations are not quite clear, weakened the contributions of the study. 2.The study was not well presented, with a lot of ambiguous descriptions. Please see “Questions for authors” for details.

Questions for authors

1.The authors stated two major limitations that current spatiotemporal fMRI studies face: 1) temporal-position ambiguity across sampling intervals and 2) the loss of direct access to ROI-specific temporal states. For the former one, the authors didn’t justify why it is important and what are the consequences. I am not very sure about the limitation of a relative temporal position and the benefits of an absolute temporal position. For the later, the authors focused on “ROI-specific states”, but they never used it in the evaluation tasks. In addition, the experiments didn’t come back to the two motivations: ablations are not enough, it would be much better to provide intermediate results to prove the necessity of the presented solutions.

2.The presentation should be improved substantially. The Introduction section pulls and drags me between foundation models for “dynamic brain networks” (which takes brain networks as input) and for “spatotemopral fMRI data” (which takes fMRI sequence as input). In general both of them can be described as “dynamic brain states”, however, methodologically they stay apart. The Methods section brings me back to a foundation model for “spatotemopral fMRI data” when I come to the Input Representation and Notation. But some unexplained notations again confused me, e.g., “network features”. I did go Appendix A, but failed to figure it out. Similar ambiguous descriptions are quite often in the manuscript, making it’s very hard to follow.

Reviewer_k864OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces STH-Brain, a self-supervised foundation model for parcellated fMRI. It encodes time in seconds rather than frames, adds atlas coordinates to spatial attention, and keeps ROI-level states next to tokens for 8 fixed networks. Pretraining predicts latents at both the ROI and network level. The model is evaluated on seven disorder classification tasks, age prediction, an external cohort and zero-shot transfer, against 13 baselines.

Critical strengths
  • The approach is simple and still improves over a broad set of baselines and pretrained models.
  • Clear improvement in classification, about 5.5–6.3 accuracy points over the second-best model on all seven tasks (Tables 1 and 8).
  • Defining time in seconds is a sensible way to handle different TRs across sites.
  • Broad evaluation, including an external cohort and zero-shot transfer.
Critical weaknesses
  • The compound effect in the ablation is not clear (Table 5). Every removed component costs about the same on every task, so it is hard to tell how the parts interact. There is also no variant without the ROI branch, which is the main architectural claim.
  • Novelty is incremental. The time encoding is RoPE (Su et al., Neurocomputing 2024) evaluated in seconds, the spatial encoding extends 2D RoPE from vision (Heo et al., ECCV 2024) to 3D spherical atlas coordinates, and the latent prediction follows Brain-JEPA (Dong et al., NeurIPS 2024).
  • The method description (Section 3) is too convoluted. A lot of notation comes before the reader has a picture of the model.
  • Figure 2 could use more of a description. The caption is only "Overall workflow of STH-Brain."
  • Age prediction shows a moderate improvement (0.07–0.16 years MAE), on a task where baselines already perform well.
  • The model relies on a known, fixed network assignment, and how ROIs were mapped to networks is not described.
Questions for authors
  • Can you report ablations that remove components together, and one without the ROI branch?
  • Were the ablated variants pretrained from scratch?
  • How does your time encoding compare with Brain Harmony, or with resampling to a common TR?
  • How sensitive are the results to the network assignment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.