- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a self-supervised approach for learning representations of stimulus-evoked fMRI. The authors adapt a Joint-Embedding Predictive Architecture (JEPA) to train one encoder per NSD participant, predicting the latent representation of masked cortical parcels from a different repetition of the same stimulus. Crucially, the encoder never sees the stimulus images or any stimulus-derived representation. The frozen encoder is then evaluated with two generative decoding pipelines—MindEye2 (image reconstruction) and CorText (brain captioning)—and with an unsupervised cross-subject alignment procedure (per-token orthogonal maps fit from each subject's own embeddings, no shared stimuli). The main claims are: (i) brain-only latent prediction yields representations competitive with fully supervised end-to-end pipelines, and (ii) independently trained per-subject encoders can be aligned without paired data, so that a decoding head trained on one subject transfers to another without fine-tuning.
Critical strengths
Strengths
-
Clean and well-motivated objective. Pairing the context and target branches with different repetitions of the same stimulus is an elegant way to make stimulus-locked structure the only predictable signal, discouraging the encoder from representing scanner/physiological noise. The ablations (Table 3, 4) convincingly isolate the contribution of cross-trial pairing and the context loss.
-
Strong empirical results for a brain-only objective. Despite using no image or language supervision during encoder training, the frozen embeddings reach captioning performance on par with the fully supervised CorText and reconstruction close to MindEye2. This is a non-trivial and interesting finding.
-
Extends alignability to decoding-grade representations. Prior work (Marcos-Manchón et al., 2026) showed compact retrieval embeddings can be aligned without paired stimuli; showing this survives in representations rich enough to drive generative decoders, and that a head transfers unchanged across subjects, is a meaningful increment.
-
Careful controls. Trial-permutation, zeroed-betas, and no-alignment controls are included, and the anatomically defined visual-parcel subset avoids selection on neural data. The writing is clear and the method is well described.
Critical weaknesses
Weaknesses
-
Motivation–contribution mismatch (major). The introduction motivates the work by the per-participant data-scarcity problem and the promise of pooling data across participants. However, the method as evaluated does not pool any data: each encoder is trained solely on a single subject's own data, and the authors deliberately select the four subjects with the most data (all 40 sessions, 9,000 unique stimuli). Pooling is relegated to future work with no empirical evidence. The motivating problem is therefore not addressed by the contribution demonstrated here.
-
The method does not enable generalization to new subjects (major). The "cross-subject decoding without fine-tuning" claim concerns reuse of the decoding head only. Onboarding a new subject still requires (a) training a full per-subject JEPA encoder on that subject's repeated-stimulus data, and (b) fitting the per-token alignment maps from that subject's own training embeddings. There is no training-free or low-data path to an unseen subject. The authors acknowledge this indirectly in future work, but the framing ("a step towards pooling fMRI data across participants") currently overstates what the evidence supports. Given that per-subject data cost is the stated motivation, this is a substantive gap rather than a minor caveat.
-
Baseline comparability. Reconstruction trails MindEye2, which the authors attribute to MindEye2's head being pre-trained on seven additional subjects. This confound makes the head-to-head comparison hard to interpret; a matched-data comparison would strengthen the claim of competitiveness.
-
Limited scope. Four subjects, a single dataset (NSD), and reliance on repeated stimulus presentations. The objective's dependence on repetitions is a real constraint for datasets without repeats, acknowledged but not explored.
5,the exploited signal is narrow, limiting real-world cross-subject applicability.
The self-supervised objective relies on repeated, stimulus-evoked activity, and both the alignment and head-transfer are anchored in the stimulus-locked structure shared across subjects (here, visual-cortex responses to repeated NSD images). This imposes a stronger precondition on any "new" subject than the modality-agnostic framing in the conclusion suggests. In a realistic cross-subject/cross-dataset scenario, a target subject may have abundant data from other paradigms (e.g., motor, language, resting-state) but only a small amount of repeated visual-stimulus data. In that case:
The encoder cannot be meaningfully trained, because the objective has no stimulus repetitions to exploit in the non-visual data. The unsupervised alignment, which matches internal geometry between subjects, has no shared stimulus-locked structure to key on, so there is no reason the target subject's space should correspond to a visual-decoding source subject's space. Consequently, the approach does not relax the data requirement in the way the motivation implies: a subject must still possess substantial repeated, task-matched (here visual) stimulus-evoked data to participate at all. The vision of pooling across heterogeneous, differently-acquired datasets is therefore not supported—the method requires the very kind of data (repeated, stimulus-evoked, task-matched) that is scarce and expensive, which is the problem the paper set out to address.
Questions for authors
Questions
-
Held-out subject. Can you demonstrate onboarding a subject not seen during pre-training—e.g., few-shot encoder adaptation, or fitting only the alignment map while reusing a head—and report how decoding degrades relative to the full-data regime? This directly tests the cross-subject value proposition.
-
Data scaling. How does performance (encoder quality and alignment) degrade as the target subject's training stimuli are reduced from 9,000? This would clarify whether the approach helps precisely in the low-data regime that motivates the paper.
-
Pooling. Do you have any preliminary evidence that pooling multiple subjects' data for representation learning (vs. single-subject training) improves the learned representations, as the introduction suggests?
-
Fair reconstruction baseline. Can you report MindEye2 and your method under matched pre-training data (e.g., without the 7-subject head pre-training) to isolate the effect of the encoder? 5, For a target subject whose repeated visual-stimulus data is limited but who has large amounts of other-task or resting-state data, can the method leverage that auxiliary data at all? If not, please temper the "pooling across datasets not acquired for joint decoding" claim, and clarify the minimum amount of repeated stimulus-evoked data a new subject needs for the encoder and alignment to hold.
-
Cost comparison. How does the total cost of your pipeline for a new subject (per-subject encoder training + unsupervised alignment) compare to classical hyperalignment that uses shared stimuli? The practical advantage of removing shared stimuli hinges on this.
Assessment
The core technical idea is sound and the within-subject results are a genuinely interesting demonstration that brain-only latent prediction captures information competitive with stimulus-supervised pipelines. My main reservation is that the paper's headline framing—reducing per-subject data demands and pooling across subjects—is not supported by the experiments, and the "cross-subject" capability is narrower (head reuse) than the narrative implies. The held-out-subject and data-scaling experiments above would substantially change my evaluation if addressed.
Author rebuttal
Dear reviewer,
Thank you for your comments and constructive feedback. The proposed additional experiments could not be completed within the discussion period; below, we clarify the scope of the paper and address the concerns as far as the current results allow.
Pooling fMRI data across participants is the long-term goal that motivates this work, but it requires that independently learned representations can first be brought into a common space (without paired measurements). This paper addresses that prerequisite. Each encoder is trained on a single participant, and we show that the resulting representations can be aligned without using shared stimuli or stimulus content, such that decoding heads can be shared. We have nuanced our motivation and claims in the introduction based on your comments, and marked newly added content in blue.
We also agree with the limitations that were pointed out. Onboarding a new participant still requires training an encoder and fitting an alignment for that participant, and we do not explicitly test how either degrades with fewer training stimuli; the held-out-subject and data-scaling experiments you propose would directly test this, and we leave them for future work. Our alignment method further depends on repeated stimulus-evoked responses, and the alignment relies on stimulus-locked structure shared across participants. Integrating a participant using resting-state or other-task data is therefore not covered, however, cross-domain unpaired functional alignment is a larger research challenge that, to our knowledge, is currently unaddressed in the field. Hence, we do not consider this within the scope of our contribution. The field of visual neuroscience has recorded several datasets that fulfill the mentioned requirements of “repeated, stimulus-evoked, task-matched” data, of which the Natural Scenes Datasets is currently the largest of high quality (7T). Other examples of such datasets include the recently released Laion-fMRI [1], DeepRecon [2] and the Natural Objects Dataset [3]. To our knowledge, the proposed dataset mentioned containing “motor, language, resting-state [...] but only a small amount of repeated visual-stimulus data” would indeed be more naturalistic, but would be hard to record using neuroimaging methods. We are not aware of such datasets, hence we would be grateful if you could point us to any existing datasets, to explore in future experiments.
Contrary to hyperalignment, our method does not require paired stimulus trials, but only stimulus repeats. Hyperalignment and other functional alignment methods are limited by the number of paired trials between subjects, or require stimulus information in unpaired alignment settings. Most datasets only contain a small amount of paired trials, and even fewer exist between datasets. MED-VAE [4], a study on NSD, reports cross-subject retrieval for methods with each of these requirements, which we compare to ours below:
MED-VAE avoids shared stimuli, but it requires image features for every training stimulus. Our alignment needs neither shared stimuli nor stimulus content, and reaches higher top-1 accuracy among eight times as many candidates. ROIs, latent dimensionality, and test images differ between the two studies, so this comparison is indicative only, but our larger candidate set works against our method.
[1] LAION-fMRI: A densely sampled 7T-fMRI dataset providing broad coverage of natural image diversity. Zerbe et al.
[2] Deep image reconstruction from human brain activity. Shen et al.
[3] A large-scale fmri dataset for the visual processing of naturalistic scenes. Gong et al.
[4] Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE. Papathanasiou et al.