acceptodds
Under review as a conference paper at ICLR 2027

Latent Prediction Yields Cross-Subject Alignable Representations in Sensory-Evoked fMRI

Abstract

Decoding sensory-evoked functional magnetic resonance imaging (fMRI) data is a central analytical tool in cognitive neuroscience. Here we ask whether self-supervised learning (SSL), which has been scarcely explored in this domain, can yield representations that support strong downstream performance. To do so, we adapt a joint embedding predictive architecture (JEPA) to train participant-specific encoders on brain data alone. The encoders never see the stimulus images or any representation derived from them. Applied to the Natural Scenes Dataset (NSD), we evaluate our learned brain embeddings on the downstream tasks of brain captioning and image reconstruction, as well as cross-participant generalization. Our self-supervised approach achieves competitive performance with parameter-rich supervised pipelines that are trained end-to-end with access to image and language data. We further find that encoders trained independently for different participants can be aligned fully unsupervised, using neither images nor stimuli shared between them. After alignment, a neural decoding head trained on one participant can be applied to another without fine-tuning and retains most of its within-subject performance on image reconstruction and brain captioning. By removing the need for images and shared stimuli, our approach is a step towards pooling fMRI data across participants and, eventually, across datasets.

Primary area:
applications to neuroscience & cognitive science
Keywords:
self-supervised learning, fMRI, neural decoding, cross-subject alignment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WXFgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes a self-supervised approach for learning representations of stimulus-evoked fMRI. The authors adapt a Joint-Embedding Predictive Architecture (JEPA) to train one encoder per NSD participant, predicting the latent representation of masked cortical parcels from a different repetition of the same stimulus. Crucially, the encoder never sees the stimulus images or any stimulus-derived representation. The frozen encoder is then evaluated with two generative decoding pipelines—MindEye2 (image reconstruction) and CorText (brain captioning)—and with an unsupervised cross-subject alignment procedure (per-token orthogonal maps fit from each subject's own embeddings, no shared stimuli). The main claims are: (i) brain-only latent prediction yields representations competitive with fully supervised end-to-end pipelines, and (ii) independently trained per-subject encoders can be aligned without paired data, so that a decoding head trained on one subject transfers to another without fine-tuning.

Critical strengths

Strengths

  1. Clean and well-motivated objective. Pairing the context and target branches with different repetitions of the same stimulus is an elegant way to make stimulus-locked structure the only predictable signal, discouraging the encoder from representing scanner/physiological noise. The ablations (Table 3, 4) convincingly isolate the contribution of cross-trial pairing and the context loss.

  2. Strong empirical results for a brain-only objective. Despite using no image or language supervision during encoder training, the frozen embeddings reach captioning performance on par with the fully supervised CorText and reconstruction close to MindEye2. This is a non-trivial and interesting finding.

  3. Extends alignability to decoding-grade representations. Prior work (Marcos-Manchón et al., 2026) showed compact retrieval embeddings can be aligned without paired stimuli; showing this survives in representations rich enough to drive generative decoders, and that a head transfers unchanged across subjects, is a meaningful increment.

  4. Careful controls. Trial-permutation, zeroed-betas, and no-alignment controls are included, and the anatomically defined visual-parcel subset avoids selection on neural data. The writing is clear and the method is well described.

Critical weaknesses

Weaknesses

  1. Motivation–contribution mismatch (major). The introduction motivates the work by the per-participant data-scarcity problem and the promise of pooling data across participants. However, the method as evaluated does not pool any data: each encoder is trained solely on a single subject's own data, and the authors deliberately select the four subjects with the most data (all 40 sessions, 9,000 unique stimuli). Pooling is relegated to future work with no empirical evidence. The motivating problem is therefore not addressed by the contribution demonstrated here.

  2. The method does not enable generalization to new subjects (major). The "cross-subject decoding without fine-tuning" claim concerns reuse of the decoding head only. Onboarding a new subject still requires (a) training a full per-subject JEPA encoder on that subject's repeated-stimulus data, and (b) fitting the per-token alignment maps from that subject's own training embeddings. There is no training-free or low-data path to an unseen subject. The authors acknowledge this indirectly in future work, but the framing ("a step towards pooling fMRI data across participants") currently overstates what the evidence supports. Given that per-subject data cost is the stated motivation, this is a substantive gap rather than a minor caveat.

  3. Baseline comparability. Reconstruction trails MindEye2, which the authors attribute to MindEye2's head being pre-trained on seven additional subjects. This confound makes the head-to-head comparison hard to interpret; a matched-data comparison would strengthen the claim of competitiveness.

  4. Limited scope. Four subjects, a single dataset (NSD), and reliance on repeated stimulus presentations. The objective's dependence on repetitions is a real constraint for datasets without repeats, acknowledged but not explored.

5,the exploited signal is narrow, limiting real-world cross-subject applicability.

The self-supervised objective relies on repeated, stimulus-evoked activity, and both the alignment and head-transfer are anchored in the stimulus-locked structure shared across subjects (here, visual-cortex responses to repeated NSD images). This imposes a stronger precondition on any "new" subject than the modality-agnostic framing in the conclusion suggests. In a realistic cross-subject/cross-dataset scenario, a target subject may have abundant data from other paradigms (e.g., motor, language, resting-state) but only a small amount of repeated visual-stimulus data. In that case:

The encoder cannot be meaningfully trained, because the objective has no stimulus repetitions to exploit in the non-visual data. The unsupervised alignment, which matches internal geometry between subjects, has no shared stimulus-locked structure to key on, so there is no reason the target subject's space should correspond to a visual-decoding source subject's space. Consequently, the approach does not relax the data requirement in the way the motivation implies: a subject must still possess substantial repeated, task-matched (here visual) stimulus-evoked data to participate at all. The vision of pooling across heterogeneous, differently-acquired datasets is therefore not supported—the method requires the very kind of data (repeated, stimulus-evoked, task-matched) that is scarce and expensive, which is the problem the paper set out to address.

Questions for authors

Questions

  1. Held-out subject. Can you demonstrate onboarding a subject not seen during pre-training—e.g., few-shot encoder adaptation, or fitting only the alignment map while reusing a head—and report how decoding degrades relative to the full-data regime? This directly tests the cross-subject value proposition.

  2. Data scaling. How does performance (encoder quality and alignment) degrade as the target subject's training stimuli are reduced from 9,000? This would clarify whether the approach helps precisely in the low-data regime that motivates the paper.

  3. Pooling. Do you have any preliminary evidence that pooling multiple subjects' data for representation learning (vs. single-subject training) improves the learned representations, as the introduction suggests?

  4. Fair reconstruction baseline. Can you report MindEye2 and your method under matched pre-training data (e.g., without the 7-subject head pre-training) to isolate the effect of the encoder? 5, For a target subject whose repeated visual-stimulus data is limited but who has large amounts of other-task or resting-state data, can the method leverage that auxiliary data at all? If not, please temper the "pooling across datasets not acquired for joint decoding" claim, and clarify the minimum amount of repeated stimulus-evoked data a new subject needs for the encoder and alignment to hold.

  5. Cost comparison. How does the total cost of your pipeline for a new subject (per-subject encoder training + unsupervised alignment) compare to classical hyperalignment that uses shared stimuli? The practical advantage of removing shared stimuli hinges on this.

Assessment

The core technical idea is sound and the within-subject results are a genuinely interesting demonstration that brain-only latent prediction captures information competitive with stimulus-supervised pipelines. My main reservation is that the paper's headline framing—reducing per-subject data demands and pooling across subjects—is not supported by the experiments, and the "cross-subject" capability is narrower (head reuse) than the narrative implies. The held-out-subject and data-scaling experiments above would substantially change my evaluation if addressed.

1 comment
AuthorsOpenReview ↗

Author rebuttal

Dear reviewer,

Thank you for your comments and constructive feedback. The proposed additional experiments could not be completed within the discussion period; below, we clarify the scope of the paper and address the concerns as far as the current results allow.

Pooling fMRI data across participants is the long-term goal that motivates this work, but it requires that independently learned representations can first be brought into a common space (without paired measurements). This paper addresses that prerequisite. Each encoder is trained on a single participant, and we show that the resulting representations can be aligned without using shared stimuli or stimulus content, such that decoding heads can be shared. We have nuanced our motivation and claims in the introduction based on your comments, and marked newly added content in blue.

We also agree with the limitations that were pointed out. Onboarding a new participant still requires training an encoder and fitting an alignment for that participant, and we do not explicitly test how either degrades with fewer training stimuli; the held-out-subject and data-scaling experiments you propose would directly test this, and we leave them for future work. Our alignment method further depends on repeated stimulus-evoked responses, and the alignment relies on stimulus-locked structure shared across participants. Integrating a participant using resting-state or other-task data is therefore not covered, however, cross-domain unpaired functional alignment is a larger research challenge that, to our knowledge, is currently unaddressed in the field. Hence, we do not consider this within the scope of our contribution. The field of visual neuroscience has recorded several datasets that fulfill the mentioned requirements of “repeated, stimulus-evoked, task-matched” data, of which the Natural Scenes Datasets is currently the largest of high quality (7T). Other examples of such datasets include the recently released Laion-fMRI [1], DeepRecon [2] and the Natural Objects Dataset [3]. To our knowledge, the proposed dataset mentioned containing “motor, language, resting-state [...] but only a small amount of repeated visual-stimulus data” would indeed be more naturalistic, but would be hard to record using neuroimaging methods. We are not aware of such datasets, hence we would be grateful if you could point us to any existing datasets, to explore in future experiments.

Contrary to hyperalignment, our method does not require paired stimulus trials, but only stimulus repeats. Hyperalignment and other functional alignment methods are limited by the number of paired trials between subjects, or require stimulus information in unpaired alignment settings. Most datasets only contain a small amount of paired trials, and even fewer exist between datasets. MED-VAE [4], a study on NSD, reports cross-subject retrieval for methods with each of these requirements, which we compare to ours below:

MethodShared stimuliStimulus contentCandidatesTop-1
Procrustes hyperalignment872no12818.4%
SRM872no12821.8%
MED-VAEnoneResNet-50 features of every training image12846.2%
Oursnoneno100059.0%

MED-VAE avoids shared stimuli, but it requires image features for every training stimulus. Our alignment needs neither shared stimuli nor stimulus content, and reaches higher top-1 accuracy among eight times as many candidates. ROIs, latent dimensionality, and test images differ between the two studies, so this comparison is indicative only, but our larger candidate set works against our method.

[1] LAION-fMRI: A densely sampled 7T-fMRI dataset providing broad coverage of natural image diversity. Zerbe et al.

[2] Deep image reconstruction from human brain activity. Shen et al.

[3] A large-scale fmri dataset for the visual processing of naturalistic scenes. Gong et al.

[4] Task-guided cross-subject latent alignment: a multi-encoder-decoder VAE. Papathanasiou et al.

Reviewer_MboJOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper learns participant-specific representations of stimulus-evoked fMRI using a joint embedding predictive architecture (JEPA). A context encoder predicts cortical-parcel embeddings from a different repetition of the same stimulus, with an additional context-prediction loss and vertex-reliability weighting. Pre-training uses brain responses and repetition identity, but no stimulus images, image embeddings, or captions. On four Natural Scenes Dataset participants, frozen encoders support supervised image-reconstruction and captioning heads with performance close to MindEye2 and CorText. Independently learned representations are aligned using distribution matching and parcel-specific orthogonal transformations fitted to disjoint participant-specific training stimuli. The aligned representations support transfer of a decoder between participants without retraining that decoder. Evaluation includes within- and cross-participant retrieval, two generative readouts, component ablations, input-permutation and zero-input controls, and all 12 ordered participant pairs.

Critical strengths
  1. Unpaired alignment is demonstrated through useful decoder transfer. The strongest result goes beyond matching representational geometry: a captioning or reconstruction head trained on one participant operates on another participant's aligned embeddings without decoder retraining (Sections 3.4 and 4; Tables 1, 2, and 4). The training stimulus sets are participant-specific, while the 1,000 shared stimuli are reserved for evaluation (Section 3.1). The strong difference from the no-alignment control and cross-participant Recall@1 of 0.59 over 1,000 candidates make this a consequential demonstration that brain-only representations can retain fine-grained information in an alignable form.

  2. The representation is evaluated with complementary readouts and informative controls. Frozen-encoder captioning reaches CIDEr 0.673-0.674 versus 0.679 for CorText, and reconstruction remains competitive with MindEye2 across several metrics (Tables 1-2), despite the different supervision and pre-training arrangements being acknowledged. Tables 3-4 show that cross-trial pairing and particularly the context loss substantially improve retrieval. Trial permutation, zeroed inputs, and failed transfer without alignment help establish that the results depend on the brain signal and learned correspondence. Reporting every participant and ordered pair in Appendix B also makes the heterogeneity visible.

Critical weaknesses
  1. The comparisons do not isolate the advantage of this particular representation-learning approach. The component ablations in Tables 3-4 are useful, but the alternatives are raw/PCA-reduced betas and successive variants of JEPA. There is no matched cross-repetition denoising or contrastive encoder, nor a generative-readout comparison using the compact brain-only representations on which the adopted alignment procedure was originally demonstrated. Downstream ablations of the encoder are also absent. Consequently, the paper establishes that the proposed combination works, but leaves uncertain how much of its utility and alignability specifically requires latent prediction with this architecture. A small, carefully matched comparison would materially strengthen the methodological contribution without requiring a new dataset.

  2. The robustness and selection protocol need clearer reporting to interpret the quantitative claims. Section 3.1 specifies the train/test stimulus split, but Section 3.2 does not explicitly state which trials determine vertex-reliability weights, and Appendix A does not describe a validation set or the criterion for choosing checkpoints, hyperparameters, or alignment restarts. Table 8 lists five random seeds but the results provide no corresponding variability. The per-participant and per-pair tables are valuable, yet four participants and their 12 dependent ordered pairs do not establish run-to-run robustness. This matters particularly for claims of near-equivalence and retention of within-participant quality: visual-readout CIDEr decreases from 0.674 to 0.453, while full-sequence image retrieval decreases from 97.2% to 69.1% (Tables 1-2). The result remains useful, but uncertainty estimates and an explicit selection protocol are needed to calibrate its strength.

Questions for authors
  1. Clarification: Were vertex-reliability weights and all learned preprocessing quantities estimated exclusively from the 9,000 training stimuli? How were checkpoints, hyperparameters, and alignment restarts/seeds selected without using the shared test-stimulus correspondences? Please also explain the evaluation-time information used by the separately fitted test-session standardization in Section 3.1.
  2. Clarification / evidence that would affect the assessment: Do results from a matched cross-repetition contrastive or denoising encoder, or the cited compact brain-only embedding pipeline with the same decoder heads, exist? If so, how do within-participant decoding and unpaired transfer compare under the same data and parameter budget?
  3. Clarification: What variability is observed across the five alignment seeds in Table 8 and across independently trained encoders? Were those seeds averaged, or was one selected, and by which criterion? Please distinguish uncertainty over stimuli, runs, and participants rather than treating the 12 ordered pairs as independent participants.
  4. Clarification: Are the CorText and MindEye2 numbers reproduced under exactly the stated 1,000-image test split, repetition averaging, and reconstruction/refinement settings, or taken from their original reports? Figures 2-3 explicitly use refined outputs whereas Table 2 uses unrefined outputs; confirming the baseline setting would make the comparison interpretable.
4 comments
AuthorsOpenReview ↗

Author rebuttal (part 1)

Dear reviewer,

Thank you for your comments and constructive feedback. We apologize for the delayed response; the additional experiments took longer than anticipated. We address each of your questions below and have marked new content in the paper in blue.

Were vertex-reliability weights and all learned preprocessing quantities estimated exclusively from the 9,000 training stimuli?

This is correct; vertex-reliability weights were computed from the 9,000 training samples only.

How were checkpoints, hyperparameters, and alignment restarts/seeds selected without using the shared test-stimulus correspondences?

Pre-training uses the I-JEPA optimization recipe unchanged (AdamW, peak learning rate 1e-3 with a 40-epoch warmup and cosine decay, weight decay 0.04→0.4, EMA momentum 0.996→1.0). To prevent overfitting on the core subjects (1, 2, 5, 7), we tuned the network depth and width using the test results from the remaining subjects (3, 4, 6, 8). We always use the final checkpoint after 300 epochs; i.e., we do not use any mechanism to select checkpoints. For alignment, we selected the best run and hyperparameter values using the mutual nearest neighbour agreement of the mapped training embeddings, as implemented by Marcos-Manchón et al. [1], a criterion that never uses the shared test stimuli. For this scoring, we take 4,000 training embeddings from each subject, from images unique to that subject, so no correspondences exist or are used. After mapping the source embeddings into the target space, we pair each mapped source point with its nearest target point (cosine similarity), and each target point with its nearest mapped source point. A pair is a mutual nearest neighbour if the two choices agree.

Taken together, we are confident that our measures prevent information leakage from the test dataset. We have clarified these methodological considerations in Appendix A.

We also introduced an alignment analysis in Appendix G using a supervised, hyperalignment-style baseline: per-parcel orthogonal Procrustes fit on 500 paired shared images and evaluated on the remaining 500 shared images. We added this evaluation to exclude the possibility that the alignment parameters suit the JEPA encoders better, since this supervised upper bound does not use any hyperparameters. The supervised results follow the same trend across pre-training components as the unsupervised alignment results (Table 4) and confirm that the JEPA encoders with reliability weighting and context loss generate embeddings that can be better aligned than the raw betas.

MethodMean rank ↓Recall@1 ↑
Controls w/o encoder
Concatenated betas + PCA35.660.24
Parcellated betas42.020.20
JEPA encoders
JEPA default (same-trial, Lpred\mathcal{L}_\text{pred} only)103.620.03
+ cross-trial pairing33.690.17
+ context loss Lctx\mathcal{L}_\text{ctx}5.470.63
+ reliability weighting4.510.67
+ whitening (post-training)3.030.74

Following a request from Reviewer ZJb3, we added a further control to Table 4, "parcellated betas", in which an orthogonal map is fitted per parcel directly on the raw betas. It improves on chance but remains far below our learned representations (Recall@1 0.08 vs. 0.59). This accounts for the newly added row.

Please also explain the evaluation-time information used by the separately fitted test-session standardization in Section 3.1.

Our pipeline standardizes each NSD subject's betas vertex-wise within each scan session, before trials are separated into train and test splits. Because NSD interleaves shared and subject-unique stimuli within sessions, these statistics are computed over all trials of a session. The normalization is therefore not fitted on training data alone, and we have corrected the description in Section 3.1, which did not state this precisely.

MindEye2 states that it uses training-set statistics only, which is the source of the one asymmetry in our results: the MindEye2 values in Table 2 are taken from the original paper and were computed under a different normalization. The CorText comparison in Table 1 was re-run under our own pipeline and is therefore like-for-like, and all rows of our ablations (Tables 3 and 4) share this preprocessing, so the relative effects we report are unaffected.

We have made the procedure explicit in Section 3.1 and recorded the difference from MindEye2's preprocessing in the caption of Table 2. Computing the session statistics from training trials only is the stricter alternative, which we note for future work.

AuthorsOpenReview ↗

Author rebuttal (part 2)

Do results from a matched cross-repetition contrastive or denoising encoder, or the cited compact brain-only embedding pipeline with the same decoder heads, exist? If so, how do within-participant decoding and unpaired transfer compare under the same data and parameter budget?

To address this question, we now report the performance of training the same encoder architecture with contrastive learning. We pick the InfoNCE objective with a temperature of 0.07, as this is commonly used as the default temperature [2, 3]. We kept all training parameters as reported in our paper and only tuned the number of epochs and learning rate on a held-out development subject. We report results for 50 epochs with a learning rate of 0.01 and added these results in Appendix H with a summary in the fourth paragraph in Section 4.

Within-subject retrieval results on visual readout, averaged over subjects 1, 2, 5, and 7:

MethodMean rank ↓Recall @ 1 ↑
JEPA16.530.63
InfoNCE21.930.56

Cross-subject retrieval results on visual readout, averaged over subjects 1, 2, 5, and 7:

MethodMean rank ↓Recall @ 1 ↑
JEPA7.450.59
InfoNCE345.540.05

While the InfoNCE within-subject recall is only 7 percentage points off the JEPA performance, the alignment completely deteriorates when switching to contrastive learning. To exclude the possibility that this is due to the alignment parameters being tuned specifically for the JEPA encoders, we again run a supervised, hyperalignment-style baseline:

MethodMean rank ↓Recall @ 1 ↑
JEPA (ours)3.030.74
InfoNCE10.460.40

The JEPA baseline still outperforms the contrastive objective in this supervised Procrustes setup, indicating that, for our pipeline, JEPA extracts stimulus-driven information better (as shown by the within-subject results above) while also aligning the learned representations better.

Finally, we also test the decoding performance of the InfoNCE encoders using the CorText pipeline. The results show that JEPA also outperforms the contrastive objective in downstream tasks. We do not evaluate on image reconstruction as it takes significantly longer to train.

ModelBLEU-1BLEU-2BLEU-3BLEU-4METEORROUGE-LCIDErCLIP-SRefCLIP-S
JEPA0.6230.4390.3040.2120.2000.4630.6740.6330.703
InfoNCE0.5630.3700.2410.1610.1690.4190.4550.5710.646

[2] Momentum Contrast for Unsupervised Visual Representation Learning. He et al.

[3] Unsupervised Feature Learning via Non-Parametric Instance Discrimination. Wu et al.

AuthorsOpenReview ↗

Author rebuttal (part 3)

What variability is observed across the five alignment seeds in Table 8 and across independently trained encoders? Were those seeds averaged, or was one selected, and by which criterion? Please distinguish uncertainty over stimuli, runs, and participants rather than treating the 12 ordered pairs as independent participants.

The table below shows the alignment performance for each of the 5 seeds and for each of the directions. Importantly, the table shows the individual alignment transformations, while in Table 4 of our paper we report the scores of the symmetrized transformations following Marcos-Manchón et al. [1] (averaging the transformation matrices of B→A and A→B), which perform slightly better on average (0.590.59 versus a directional mean of 0.580.58). We select the best seed using the mutual nearest neighbour agreement that is computed as explained in the answer to the second question above. The results show that the alignment is stable across seeds.

DirectionSeed 0Seed 1Seed 2Seed 3Seed 4
1→20.9670.9660.9660.9660.967
1→50.5500.5500.5480.5510.550
1→70.5740.5810.5770.5810.582
2→10.9490.9490.9490.9490.949
2→50.5470.5440.5450.5450.545
2→70.5250.5180.5230.5230.523
5→10.4590.4620.4600.4610.470
5→20.4450.4420.4460.4440.446
5→70.3910.3910.3880.3900.393
7→10.5700.5700.5710.5640.566
7→20.5370.5350.5340.5340.533
7→50.4840.4800.4840.4820.479

The sources of uncertainty in this evaluation are of very different magnitude. Across seeds, the mean over directions varies by 0.0010.001, and no single direction varies by more than 0.0110.011, so the alignment procedure is essentially deterministic with respect to its random initialization. Across subject pairs, by contrast, Recall@1 ranges from 0.3910.391 to 0.9670.967. Alignment quality is thus a property of the participant pair, not of the run, and seed variability is negligible by comparison.

The pair-level spread is itself structured by participant rather than distributed evenly. The table below shows Recall@1 averaged over seeds and over the three directions in which each subject acts as source or as target: subjects1 and2 align well with every other subject and exceptionally well with each other (0.9670.967 and 0.9490.949), whereas subject~5 is the weakest source (0.430.43) while remaining an average target (0.530.53).

SubjectAs sourceAs target
10.6980.660
20.6720.649
50.4330.526
70.5280.497

We have added this analysis in Appendix F.

Are the CorText and MindEye2 numbers reproduced under exactly the stated 1,000-image test split, repetition averaging, and reconstruction/refinement settings, or taken from their original reports? Figures 2-3 explicitly use refined outputs whereas Table 2 uses unrefined outputs; confirming the baseline setting would make the comparison interpretable.

The MindEye2 numbers in Table 2 are sourced from the original paper results. Our reason for reporting the unrefined setting is that refinement blends the unCLIP reconstruction with the output of MindEye2’s low-level module. A refined comparison therefore measures the high-level and low-level pathways in combination, whereas the claim of this paper concerns what the frozen brain representation supports on its own. The unrefined setting isolates that. Figures 2–3 depict refined outputs as they provide a qualitative illustration of what our encoders support in the best case; all quantitative comparisons (Table 2) use the unrefined setting.

As for CorText, we indeed reproduce the numbers with the same settings as our results. However, both the CorText numbers and our results use the 515 test set that is shared by all of the 8 subjects as this is the convention that CorText uses in their paper.

We have clarified these points in Section 3.3 and in the captions of Tables 1 and 2.

Thank you for your helpful questions and feedback, which have helped us strengthen our submission. We hope these results address your concerns.

Reviewer_MboJOpenReview ↗

Training-only normalization and the contrastive comparison

Thank you for clarifying how you estimated the reliability weights and selected the alignment run using training images. The pair-level results make it much easier to see where alignment works well.

My main remaining question is how much the preprocessing affects retrieval. Could you report the change in within- and cross-subject Recall@1 when each session’s mean and variance are estimated from training trials alone? The current estimates include held-out trials, and the published MindEye2 numbers use training-only statistics. There is also a separate issue with architecture selection. Subjects 3, 4, 6 and 8 were held apart from the four evaluation subjects, but their test images overlap with the evaluation images. Please make that overlap explicit when describing how depth and width were chosen.

The five alignment seeds give very similar results with the encoders fixed. How much does Recall@1 vary when the encoders themselves are trained again from scratch? Even results for a representative subject pair would help interpret the reported precision.

I appreciate the comparison with InfoNCE using the same encoder architecture. Under supervised Procrustes, Recall@1 is 0.74 for the full method and 0.40 for InfoNCE. The fixed temperature and shorter training schedule for InfoNCE (50 epochs versus 300 for JEPA) leave some uncertainty about the size of that gap. In the same Procrustes ablation, adding the context loss raises Recall@1 from 0.17 to 0.63. I would bring that result into the main discussion, since it identifies which part of the objective contributes most to alignment.

Reviewer_ZJb3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors present a method to learn and align subject-specific latent representations of brain activity patterns. First a subject-specific denoised latent space is learned. Then the latent spaces are aligned to enable downstream decoders to be applied to each subject without having to retrain the decoders. The latent space can form the basis for the reconstruction of the images that elicited the brain activity and for the generation of verbal descriptions of perceptual content.

The authors use the Natural Scenes Data set, a 7-Tesla functional MRI data set, and the purely anatomy-based Schaefer cortex parcellation (restricted to 66 visual parcels). The encoders are transformers trained with a Joint-Embedding-Predictive-Architecture (JEPA) objective. The objective favors co-embedding of activity patterns elicited by the same image presented on separate trials, thus attenuating measurement noise and other trial-to-trial variation that is not stimulus driven. Masking a portion of the parcels (cortical regions) forces the model to extract the image information coherently reflected in multiple cortical regions. Importantly, the encoder is not trained on the downstream tasks, but task agnostic and thus presumably more generalizable.

Initially, a separate encoder is trained for each subject. The resulting subject-specific latent spaces are then aligned, separately for each anatomically defined cortical parcel, such that the distributions of latent patterns match, without requiring shared image sets in different subjects. The alignment assumes that the image distributions, but not the image samples match between subjects. The alignment consists in translation, whitening and an orthogonal transformation optimized by iterative closest-point matching. The approach can be used to generalize decoders to new subjects. However, substantial data will be needed in each subject to learn the encoding, and the stimulus distribution will need to be matched to that used for the original subjects.

The encoder essentially learns to distinguish stimulus-driven signal from all other variation (e.g. thermal noise, physiological noise, head motion, fixation instability across presentations of the same image).

Critical strengths

No supervision needed to learn denoising latent spaces.

No shared stimuli or functional localizer experiments needed to align each parcel across subjects because the approach leverages distributional alignment.

Critical weaknesses

The authors start with a JEPA objective using parcel masking, but find that JEPA alone is not sufficient. They then show that cross-trial prediction (which pulls together representations of brain activity elicited by a stimulus on different trials) is essential. This makes sense because it enables the encoder to suppress non-stimulus-driven variation (noise). They then add prediction of unmasked parcels, which brings a large additional improvement. This raises the question if a non-JEPA, simpler joint-embedding approach can perform equally well or better. It would more simply pull together representations of different trials for the same stimulus and prevent collapse, e.g., using contrastive learning or VICReg.

Another important open question is to what extent the parcel embeddings represent information about the brain activity in each parcel or brain activity across all parcels. To what extent can a single parcel’s token embedding and the average of all parcels support the decoding tasks? Using the same whitening transform appears to imply that the distributions to be whitened are all the same. However, the alignment transforms are fitted separately for each parcel. Would the same alignment also work across parcels?

The approach is related to hyperalignment, which remains rooted in cortical-surface brain-activity pattern space, uses orthogonal alignment, and requires shared stimuli. One could also consider using distributional alignment (as used here) of a latent space rooted in the brain: each parcel is a 2D cortical activity map and aligned through an orthogonal transform that matches the distributions. The current controls don’t rule out this approach.

Questions for authors

Can a non-JEPA, simpler joint-embedding approach perform equally well or better?

To what extent can a single parcel’s token embedding and the average of all parcels support the decoding tasks? Would the same cross-subject alignment (or a set of alignments regularized to be similar) also work across parcels?

2 comments
AuthorsOpenReview ↗

Author rebuttal (part 1)

Dear reviewer,

Thank you for your comments and constructive feedback. We apologize for the delayed response; the additional experiments took longer than anticipated. We address each of your questions below and have marked new content in the paper in blue.

One could also consider using distributional alignment (as used here) of a latent space rooted in the brain: each parcel is a 2D cortical activity map and aligned through an orthogonal transform that matches the distributions. The current controls don’t rule out this approach.

This is a great suggestion and something we had not considered. We have added this control to Table 4 and present it in the third paragraph of Section 4. The new control on parcellated betas is better than chance; however, alignment is still poor compared to the JEPA embeddings.

MethodMean rank ↓Recall@1 ↑
Controls w/o encoder
Concatenated betas + PCA408.990.00
Parcellated betas285.250.08
JEPA encoders
JEPA default (same-trial, Lpred\mathcal{L}_\text{pred} only)452.400.00
+ cross-trial pairing367.170.05
+ context loss Lctx\mathcal{L}_\text{ctx}15.000.45
+ reliability weighting13.110.49
+ whitening (post-training)7.450.59

Furthermore, we also added Appendix G, in which we run the alignment for all rows using a supervised, hyperalignment-style baseline: per-parcel orthogonal Procrustes fit on 500 paired shared images and evaluated on the remaining 500 shared images. We added this evaluation to exclude the possibility that the alignment parameters suit the JEPA encoders better, since this supervised upper bound does not use any hyperparameters. The supervised results follow the same trend across pre-training components and confirm that the JEPA encoders with reliability weighting and context loss generate embeddings that can be better aligned than the raw betas.

Supervised upper bound for cross-subject retrieval:

MethodMean rank ↓Recall@1 ↑
Controls w/o encoder
Concatenated betas + PCA35.660.24
Parcellated betas42.020.20
JEPA encoders
JEPA default (same-trial, Lpred\mathcal{L}_\text{pred} only)103.620.03
+ cross-trial pairing33.690.17
+ context loss Lctx\mathcal{L}_\text{ctx}5.470.63
+ reliability weighting4.510.67
+ whitening (post-training)3.030.74
AuthorsOpenReview ↗

Author rebuttal (part 2)

Can a non-JEPA, simpler joint-embedding approach perform equally well or better?

To address this question, we now report the performance of training the same encoder architecture with contrastive learning. We pick the InfoNCE objective with a temperature of 0.07, as this is commonly used as the default temperature [1, 2]. We kept all training parameters as reported in our paper and only tuned the number of epochs and learning rate on a held-out development subject. We report results for 50 epochs with a learning rate of 0.01 and added these results in Appendix H with a summary in the fourth paragraph in Section 4.

Within-subject retrieval results on visual readout, averaged over subjects 1, 2, 5, and 7:

MethodMean rank ↓Recall @ 1 ↑
JEPA16.530.63
InfoNCE21.930.56

Cross-subject retrieval results on visual readout, averaged over subjects 1, 2, 5, and 7:

MethodMean rank ↓Recall @ 1 ↑
JEPA7.450.59
InfoNCE345.540.05

While the InfoNCE within-subject recall is only 7 percentage points off the JEPA performance, the alignment completely deteriorates when switching to contrastive learning. To exclude the possibility that this is due to the alignment parameters being tuned specifically for the JEPA encoders, we again run a supervised, hyperalignment-style baseline:

MethodMean rank ↓Recall @ 1 ↑
JEPA (ours)3.030.74
InfoNCE10.460.40

The JEPA baseline still outperforms the contrastive objective in this supervised Procrustes setup, indicating that, for our pipeline, JEPA extracts stimulus-driven information better (as shown by the within-subject results above) while also aligning the learned representations better.

Finally, we also test the decoding performance of the InfoNCE encoders using the CorText pipeline. The results show that JEPA also outperforms the contrastive objective in downstream tasks. We do not evaluate on image reconstruction as it takes significantly longer to train.

ModelBLEU-1BLEU-2BLEU-3BLEU-4METEORROUGE-LCIDErCLIP-SRefCLIP-S
JEPA0.6230.4390.3040.2120.2000.4630.6740.6330.703
InfoNCE0.5630.3700.2410.1610.1690.4190.4550.5710.646

[1] Momentum Contrast for Unsupervised Visual Representation Learning. He et al.

[2] Unsupervised Feature Learning via Non-Parametric Instance Discrimination. Wu et al.

To what extent can a single parcel’s token embedding and the average of all parcels support the decoding tasks?

To address this question, we re-trained the downstream captioning task using the CorText pipeline on the averaged parcel tokens. We did not evaluate image reconstruction as it takes significantly longer to train. We observe a decrease in decoding performance when using a single averaged parcel token. Although performance remains better than controls, this drop in performance indicates that the parcel tokens carry information necessary for our observed decoding performance. We did not test the performance for each individual parcel in the sequence; however, we observe that the model decoding from the visual cortex token subset (66 tokens) leads to comparable performance to when using all 500 parcels. This indicates that the vertex reliability within a parcel is likely a good predictor of contribution to decoding performance.

ReadoutBLEU-1BLEU-2BLEU-3BLEU-4METEORROUGE-LCIDErCLIPScoreRefCLIPScore
66 visual tokens0.6230.4390.3040.2120.2000.4630.6740.6330.703
Mean of 66 visual tokens0.6070.4170.2790.1860.1890.4490.5810.6140.686
500 parcel tokens0.6250.4380.3030.2120.2010.4640.6730.6350.705
Mean of 500 tokens0.5960.4050.2690.1810.1820.4420.5480.5960.671

We have added these results in Appendix E and discuss them in the first paragraph of Section 4.

Would the same cross-subject alignment (or a set of alignments regularized to be similar) also work across parcels?

Fitting a single alignment transformation that is shared by all parcel tokens leads to slightly worse performance than learning a unique transformation per parcel. We added this result to Appendix B and refer to it in the second paragraph of Section 3.4.

MethodMean rank ↓Recall @ 1 ↑
Per parcel7.450.594
Shared across parcels8.290.549

Thank you for your helpful questions and feedback, which have helped us strengthen our submission. We hope these results address your concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.