acceptodds
Withdrawn from ICLR 2027

BrainWM: A Multimodal World Model for Representing, Forecasting, and Simulating Whole-Brain Dynamics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Most brain imaging foundation models treat entire fMRI scans as static images, yet the brain is a dynamic world: its state is described by both structure and function, and it evolves, driven by intrinsic spontaneous activity, external stimuli, and long-term processes such as aging. We present BrainWM, a multimodal world model for whole-brain 4D dynamics that unifies representation learning, forecasting, and simulation in a single model with three components: BrainWM-VAE encodes structural and functional MRI into an integrated structure–function latent state; BrainWM-DiT learns to generate the next state from the past in this latent space via flow matching; and BrainWM-BAM learns latent actions for state transitions through prior–posterior modeling. BrainWM is trained progressively in three stages: first unified latent-space learning, then within-run generative forecasting pretraining, and finally post-training for conditional cross-run simulation. On large-scale internal and external test sets, BrainWM achieves performance on downstream tasks that surpasses or is comparable to that of foundation models designed specifically for representation learning, while also being capable of forecasting and simulating functional and structural brain changes—tasks that these models are not designed to address. To our knowledge, BrainWM is the first brain imaging world model systematically evaluated for representing, forecasting, and simulating whole-brain dynamics.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI, multimodal neuroimaging, brain foundation model, world model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_yLhBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces BrainWM, a multimodal model for representing, forecasting, and simulating whole-brain sMRI+fMRI dynamics. It consists of a structure–function VAE, a flow-matching transformer, and a latent Brain Action Model, trained across three phases. Experiments cover competitive downstream performance and forecasting/simulation capabilities.

Critical strengths
  • The paper evaluates representation learning, within-run forecasting, rest-to-task generation, and longitudinal anatomical changes within a common framework. These cover a wide breadth of tasks that are not commonly covered but has the possibility to be important in the future.

  • Beyond temporal correlation, the authors assess connectivity, spectral properties, variance, and individual specificity, providing a more informative assessment of the generated brain dynamics compared to what reconstruction quality alone could tell.

Critical weaknesses
  • For the downstream results in table 1, the best BrainWM variant is inconsistent. In other words, sometimes the phase 2 and/or phase 3 training harms the model performance possibly due to the representation being worse. Besides for downstream task evaluation, benefits of BAM is also questionable for forecasting. In the real setting, Table 5 show that BAM is beneficial only for the short horizon (h32), and Table 23 show BAM is not beneficial for all lengths.

  • Results in table 5 and 23 suggest that the BrainWM performs almost the same even when the stimulus is zeroed out. This suggests that the model recieves almost no information from the content of the stimulus itself, and the model may be relying only on information such as the initial state or the text description, instead of reading the ongoing stimulus. Interestingly, the no-BAM model actually loses performance when the stimulus is zeroed out; it may be due to how the stimulus is constructed for the no-BAM model, or it may suggest that the current way of supplying the information of the stimulus through BAM is unable to properly pass on the information, although I am not completely sure about this part. Nonetheless, the fact that a "world model for brain dynamics" does not care about the actual content of the stimulus is a huge red flag, and must be addressed for the proposed method to truely be a functioning world model.

Questions for authors
  • As a follow-up to weakness #1, BAM seems help in table 5 but not for table 23 in the true data h32 setting. Could the authors provide explanations for this difference?

  • As a follow-up to weakness #2, could the authors provide additional ablation to locate the source of forcasting performance when the stimulus content does not seem to be the core factor?

  • Could the authors compare BAM against direct conditioning for rest-to-task simulation, including correct, incorrect, and absent task descriptions? The forecasting ablations do not establish whether BAM improves conditional simulation.

  • Does the MONAI forecasting baseline receive the same per-frame stimuli and other information as BrainWM?

Reviewer_SFcWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes BrainWM for jointly modeling sMRI and fMRI and enabling forecasting and representation learning from these data. BrainWM uses a VAE to learn a joint latent representation of paired sMRI and fMRI. A diffusion transformer then learns to predict the next window of brain activity in the latent space learned by the VAE. The model also includes a component called BAM, which produces learned action tokens from task and stimulus information to condition the transformer layers. The authors evaluate the model on fMRI forecasting and generation. They also evaluate features from the VAE and dynamics model on downstream prediction tasks.

Critical strengths

The framework supports both fMRI forecasting and downstream prediction using latent representations learned from paired sMRI and fMRI. Bringing these capabilities into one framework is an interesting direction for modeling brain MRI data. Also evaluation of learned representations is performed on several standard fMRI tasks and compared with fMRI foundation model baselines, and the model is evaluated on forecasting and rest-to-task generation

Critical weaknesses
  1. The value of multimodal fusion and dynamics modeling is not sufficiently established. The current VAE is trained on both fMRI and sMRI data, and the results cannot distinguish a benefit from multimodal fusion from using structural MRI or fMRI alone. Can the authors do ablations using only structural MRI or fMRI to show whether combining these modalities helps? Paired sMRI and fMRI are also not always available, and the paper does not evaluate how the model works when either modality is missing, which is a limiting factor for applicability compared with other fMRI foundation models. More generally, BrainWM combines multimodal fusion, dynamics modeling during pretraining and action conditioning, but it remains unclear which components accounts for its downstream gains. For example, the VAE alone gives the highest AUC on ADNI and SRPBS for BrainWM variants. Although dynamics features help on some other tasks, the downstream results do not show a consistent benefit from dynamics modeling. The BAM ablations also do not show a consistent benefit from action modeling for long-horizon forecasting when stimuli are available. Can the authors clarify the contribution of each proposed component to their model?

  2. The practical value of the forecasting and simulation results are remaining unclear. The temporal correlations are low, falling to around 0.02 in late forecast blocks. Relatedly, for rest-to-task generation, BrainWM does not outperform the average response of other subjects performing the same task, and its absolute temporal correlations are also low. In this example, it’s unclear whether the model improves prediction of an individual’s response to the task beyond learning the shared task pattern across subjects. The low absolute values for both forecasting and individual rest-to-task prediction therefore need to be better contextualized in terms of practical scientific utility.

Questions for authors
  1. The related work does not discuss relevant spatiotemporal and dynamical models such as PLRNN-BOLD-SSM (Koppe et al. 2019), Deep Markov Factor Analysis (Farnoosh & Ostadabbas, 2021), SwiFT (Kim et al., 2023), SBIND (Hosseini & Shanechi, 2025), and ConvS5 (Smith et al., 2023). How does BrainWM differ from these approaches in its forecasting and dynamical modeling capabilities?

  2. Table 17 shows substantial and increasing generated-to-real variance inflation in both temporal and spatial generation modes. Do the forecasted signals have realistic statistics during rollout? Can the authors show true and forecasted time series across the rollout ?

References:

  • Koppe, G., et al. (2019). Identifying nonlinear dynamical systems via generative recurrent neural networks with applications to fMRI. PLOS Computational Biology,
  • Farnoosh, A., and Ostadabbas, S. (2021). Deep Markov Factor Analysis: Towards Concurrent Temporal and Spatial Analysis of fMRI Data. NeurIPS.
  • Kim, P. Y., et al. (2023). SwiFT: Swin 4D fMRI Transformer. NeurIPS.
  • Hosseini, S. M., and Shanechi, M. (2025). Dynamical Modeling of Behaviorally Relevant Spatiotemporal Patterns in Neural Imaging Data. ICML.
  • Smith, J., et al., (2023). Convolutional State Space Models for Long-Range Spatiotemporal Modeling. NeurIPS.
Reviewer_PwfPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BrainWM, a world model for whole-brain MRI that aims to support representation learning, forecasting and conditional simulation in the single model. Firstly, a multi-modal 3D VAE fuses sMRI and each fMRI frame into a 32 ×\times 12 ×\times 12 ×\times 12 latent. And then, a latent DiT is trained with a flow-matching objective to generate the next window. Finally, a Brain Action Model (BAM) learns per-frame fast and per-run slow latent action tokens through a prior–posterior scheme borrowed from latent-action world models. There are three training phases (VAE, within-run forecasting, cross-run post-training) on paired sMRI–fMRI runs from diverse public cohorts. Also, their experiments cover 3 capabilities. For representation, BrainWM is compared with 5 fMRI foundation models on 8 downstream tasks under both linear probing and fine-tuning. For forecasting, task- and rest-state rollouts are compared against a MONAI-based latent-diffusion baseline and simple training-free predictors. For simulation, the model generates the first 32-frame window of a task-state scan from a resting-state scan and a task description (HCP/CHCP), and a follow-up structural scan from a baseline scan and the elapsed interval (ADNI/PPMI). The authors claimed that a single generative model matches or exceeds dedicated representation models while also forecasting and simulating brain states.

Critical strengths

An interesting formulation of brain-state transitions as two-timescale latent actions: The authors model brain dynamics as a state plus latent actions on two timescales, (1) fast actions for frame-level changes within a run, and (2) slow actions for transitions across runs such as task switches or aging. Viewing brain activity as a state evolving under intrinsic dynamics and exogenous inputs is well established in neuroscience [1,2], and concurrent works have also begun to frame it as a brain world model [3,4]. Especially, I think this paper’s formulation is quite interesting because it goes a step further. The actions are not given as observed stimuli but inferred as latent variables from observed transitions, and they are separated by timescale, so that the effects of slower drivers, such as task switches or aging, can in principle be represented alongside frame-level changes. Ultimately, I believe brain foundation models should be able to model and simulate the drivers of brain-state change at multiple timescales, and this paper formulates the problem in this way. It seems that the current implementation still treats the fast and slow components somewhat separately, but I consider the formulation itself a meaningful step toward a unified treatment of brain dynamics across timescales.


 [1] Friston et al., Dynamic causal modelling, 2003.
 [2] Breakspear, Dynamic models of large-scale brain activity, 2017.
 [3] Z. Dong et al., NeuroWorld: A Latent Brain World Model for Stimulus-Conditioned Human Brain Dynamics, 2026.
 [4] J. Xia et al., BrainWorld: A Structural-Prior-Conditioned Generative Model for Whole-Brain 4D fMRI Dynamics, 2026.

Critical weaknesses
  1. Methodological novelty : Most components are adopted from prior work, including the latent-action mechanism itself, which follows existing latent-action models [1,2] and the prior–posterior scheme of [3]. What is new is its adaptation to brain imaging, namely defining actions as the drivers of brain-state change and separating them into fast and slow timescales. Of course, as noted in the critical strengths, separating the driving actions of brain-state change by timescale is itself an interesting and plausible formulation. However, in my opinion, the experiments provided in the paper do not explicitly demonstrate the joint contribution of fast and slow actions to the same brain-state transition. Although the fast and slow actions are architecturally concatenated, I think no experiments clarify the joint contributions of these two actions to the brain-state transition. In my view, therefore, the current experiments do not sufficiently verify whether this captures a meaningful two-timescale decomposition of brain dynamics, rather than simply serving as task-specific conditioning mechanisms. Additionally, the authors claim that unifying structure and function in a single latent state is a novel approach that enables structure to be generated rather than used merely as a condition. In practice, however, this distinction matters for the longitudinal experiment, the only setting in which structure actually changes. In forecasting and rest-to-task simulation, the source and target share the same T1, so anatomy effectively remains a fixed condition. In my opinion, this therefore appears to be a design choice made to enable longitudinal simulation rather than a methodological novelty, and a single experiment with limited results is not sufficient to support it as a core contribution. To summarize, while the formulation is interesting and plausible, including the separation of brain-state drivers into fast and slow timescales and the framing of tasks such as predicting future anatomy through longitudinal simulation, the methodological contribution appears limited. Since I think the paper’s novelty rests largely on these two ideas, I consider this the most consequential weakness.

  2. Missing ablations for some key design choices : The ablations focus on BAM in the forecasting setting, where they show a small benefit when stimuli are available. In contrast, several other components that the paper presents as contributions or motivates as design choices are not fully ablated, including the slow actions used for simulation, and the fused sMRI-fMRI latent compared with an fMRI-only latent. Without detailed comparisons about these, it is unclear which of the paper’s own design choices are responsible for its results. These could be addressed with additional experiments, but because they concern the paper’s key contributions, their absence can be a limitation of the current submission.


 [1] Schmidt and Jiang, Learning to Act without Actions, 2023.
 [2] Bruce et al., Genie: Generative Interactive Environments, 2024.
 [3] Luo et al., Being-H0.7: A Latent World-Action Model from Egocentric Videos, 2026.

Questions for authors
  1. Justification of the methodological novelty : As discussed in the weakness section, I think that the latent-action mechanism largely follows existing work, and the current experiments do not clearly clarify the joint contributions of fast and slow actions to brain-state transitions. And it seems that making anatomy generable mainly serves longitudinal simulation only. This is my main concern, and I am open to discussing it further. Could the authors clarify what they consider the core methodological contribution beyond these adaptations? Also, is there evidence, such as action-specific ablations or an analysis of the learned actions, that the two-timescale decomposition captures distinct and complementary drivers of brain-state change? A convincing answer to this question would most likely change my assessment and I believe that addressing this point would further reinforce the paper's core contributions.

  2. Contribution of the slow actions : The idea of modeling slow changes such as aging with a dedicated slow action, distinct from the fast actions for within-run dynamics, is appealing. However, it seems that the no-BAM ablation is reported only for forecasting in Tables 5 and 23. For rest-to-task and longitudinal simulation, how do the results compare with a variant that conditions the generator directly on the target description and elapsed interval, without training slow actions? Also, could the authors clarify whether both fast and slow actions are active during longitudinal experiments?

  3. Further use cases of generable anatomy : The ability to generate anatomical changes is currently evaluated only through longitudinal simulation, while anatomy remains constant in forecasting and rest-to-task simulation. Could the authors show other scenarios in which generating structure jointly with function is useful, for example simulating how anatomical change such as atrophy alters functional dynamics, or modeling structural development in longitudinal pediatric or adolescent cohorts? Such examples would help establish the fused state as a general contribution rather than a choice specific to one task.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.