acceptodds
Withdrawn from ICLR 2027

Natural Image Autoencoder-Based fMRI Representations for Trait and State Prediction

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Foundation models pre-trained on large-scale fMRI datasets have shown strong downstream performance, but at substantial data and computation cost. To investigate how much fMRI-specific pre-training is actually needed for such performance, we introduce FReD, which derives fMRI representations from a frozen Deep Compression AutoEncoder (DCAE) pre-trained exclusively on natural images and pairs them with a task specific readout. For trait prediction, FReD summarizes frame-wise representations by their temporal mean and log-standard deviation and applies linear probing, with late fusion across two normalization schemes. For state prediction, it represents each frame as a single token and models temporal dependencies with a shallow Transformer. Across four resting-state datasets spanning six trait-prediction targets, linear probes on frozen DCAE features generally outperform those on fMRI foundation model representations and remain competitive with fully fine-tuned fMRI foundation models. On three task-fMRI state-prediction tasks, a temporal readout on DCAE features performs comparably to the strongest foundation models evaluated. A Gaussian injection analysis further shows that localized signal changes are recovered more accurately from the frozen DCAE features than from the evaluated foundation-model representations. Together, these results show that strong performance on current fMRI benchmarks is possible without fMRI-specific representation pre-training, making frozen natural-image features as a useful baseline for assessing its added value.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI, natural image autoencoder

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_PyD4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In this study, fMRI representations were derived from DCAE pretrained exclusively on natural images and evaluated on downstream prediction tasks including trait prediction and brain state prediction. Experimental results demonstrated the competitive performance of the derived representations, compared with that from fMRI-specific pre-trained models.

Critical strengths

The DCAE derived representations were comprehensively evaluated across different tasks at both global level and frame level.

Critical weaknesses

Though competitive performance, the interpretability of the derived representation remains unclear, which is important for neuroimaging studies to explore the brain-wide associations with traits/behaviors.

Questions for authors
  1. Regarding the data normalization, it is not clear why the FReD-State uses the coordinate-frame normalization but the FReD-Trait uses combined global and coordinate-frame normalization.

  2. For the evaluation, the baselines should also include conventional functional connectivity (FC) based models (e.g., KRR, SVM), which demonstrated competitive performance compared with foundation models, as demonstrated in this study: Marraffini, Giovanni, et al. "The Variance Brain Foundation Models Forgot: Third-Order Statistics Predict Cognition Where Billion-Parameter Models Fail." arXiv preprint arXiv:2606.04010 (2026).

  3. It is not clear how to interpret the derived fMRI representation. Also, how to identify the brain regions that significantly contribute to the trait or brain state prediction? This is important for neuroimaging-based brain-wide association studies.

  4. For the fMRI variation captured by the principal directions of DCAE representations learned from nature images, what does high or low fraction of variance explained indicate from the perspective of neuroscience?

Reviewer_5PByOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FReD, which extracts fMRI features using a frozen DCAE pretrained on natural images. Trait prediction uses temporal means, log-standard deviations, and linear readouts, while state prediction uses frame-level tokens and a Transformer. The paper evaluates whether competitive fMRI prediction can be achieved without fMRI-specific representation pretraining.

Critical strengths
  1. The research question is useful, and the evaluation is broad. The paper covers six trait-prediction targets and three state-prediction tasks. Tables 1–2 show that frozen DCAE features with task-specific readouts are competitive with the evaluated fMRI foundation models, providing a useful baseline for assessing the value of fMRI-specific pretraining.

  2. The evaluation includes several informative controls. The paper examines normalization, temporal statistics, spatial grouping, and localized perturbations, alongside dimensionality reduction, partial fine-tuning, and comparisons on matched splits. Subject-disjoint splits and held-out hyperparameter selection further strengthen the credibility of the results.

Critical weaknesses
  1. The contributions of the representation and downstream processing are not fully separated. Although single-normalization, variance-feature, and dimensionality controls are included, FReD-Trait and the foundation models still use different aggregation pipelines (§3.2 and Appendix B.2). The results establish the competitiveness of the complete pipeline, but provide less direct evidence about the advantage of the DCAE representation itself.

  2. The independent role of temporal order remains unclear. FReD-State differs from FReD-Trait in readout capacity and nonlinear processing, and also in spatial grouping for volumetric inputs. Section 4.3 and Table 4 support task-specific readout choices, but do not isolate how much of the state-prediction improvement comes from temporal order rather than a more expressive readout.

Questions for authors
  1. Which existing results separate the contribution of frozen DCAE features from differences in statistical aggregation and readout?

  2. Which existing results distinguish the benefit of temporal order from that of a more expressive, potentially order-insensitive readout?

Reviewer_mUDyOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces FReD, which encodes each fMRI frame with a frozen autoencoder (DCAE) pre-trained only on ImageNet and trains a readout on top. For traits, the mean and log-standard deviation of the latents go into a linear probe. For states, each frame becomes one token for a small Transformer. It is compared with eight fMRI foundation models and TABLeT on six trait and three state tasks.

Critical strengths
  • Checking whether a frozen off-the-shelf encoder matches fMRI foundation models is a useful question.
  • Broad set of baselines, with linear probing and fine-tuning.
  • FReD-State is simpler than TABLeT and does better on HCP-Task and NSD.
Critical weaknesses
  • The method is a readout over a frozen pretrained model (Fig. 1). The same frozen DCAE was already used for fMRI by TABLeT (Kim et al., CVPR 2026), and frozen 2D image models for 3D volume slices by Raptor (An et al., ICML 2025). That simple models can match complex fMRI models was also shown before (Popov et al., NeuroImage 2024).
  • In linear probing (Fig. 2), gains over the baselines are small except on HCP-A-Intel, where FReD is clearly better. Against fine-tuned models (Tab. 1) it is better on some results and worse on others, mostly within one standard deviation.
  • BrainLM is not included, and it should be cited and compared.
  • The effect of the log-standard deviation feature is small (Fig. 5a), under 0.01 on about half of the tasks and at most 0.03, so it is unclear whether this component is needed.
  • It is not clear whether the properties of the encoder matter, or another one would do just as well. ImageNet pre-training has been shown to add little for medical imaging (Raghu et al., NeurIPS 2019).
  • The motivation for the 32× compression analysis (Sec. 5.2) is unclear. If it is meant to explain why the ImageNet model is useful, this should be stated.
  • Fig. 3 is hard to read, and the writing in general could be clearer.
Questions for authors
  • Could you test other frozen encoders, including a randomly initialized DCAE, with the same readouts?
  • Can you report significance tests for Tab. 1?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.