- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether paired image–fMRI data can be used to improve pretrained visual representations without using category labels or behavioral supervision during adaptation.
Brain-VisA has two stages. First, the authors train an fMRI encoder with a self-supervised objective that treats repeated fMRI measurements of the same image as positive views. The intuition is that repeated measurements share stimulus-related information but differ in trial-specific and measurement noise. Second, the resulting brain representation is aligned with a pretrained visual representation using relational knowledge distillation. The visual backbone is frozen, while a small residual head adapts its output representation.
The experiments cover eight visual backbones from SimCLR, CLIP, and DINOv2. The reported gains on high-level abstract classification are large, and the improvements also extend to held-out adaptation categories, semantic OOD tasks, and THINGS odd-one-out judgments. At the same time, the adapted models mostly retain their original ImageNet category-recognition performance.
I like the basic idea. Repeated neural measurements are a natural source of self-supervision, and using relational alignment rather than direct feature matching makes sense.
My main concern is more specific: I am not yet convinced that the experiments isolate the contribution of the brain signal content itself. The paper shows that the full training procedure works very well, but the strongest interpretation is that useful semantic structure from human neural responses is what causes the improvement. I think that claim needs a stronger control.
Critical strengths
The first strength is the basic use of repeated measurements. In fMRI, repeated trials are often treated as redundant or mainly useful for averaging. Here they become a source of self-supervision: the shared image acts as the common factor, while trial-specific variation is treated as nuisance. That is a clean and well-motivated idea.
The second strength is the breadth of the empirical result. The method improves multiple SimCLR, CLIP, and DINOv2 backbones rather than one carefully chosen model. The paper also includes sensible comparisons against autoencoding, denoising autoencoding, contrastive brain SSL, pointwise cross-modal InfoNCE, and different two-stage schedules.
I also think the ImageNet retention result is important. The adaptation substantially changes high-level semantic performance without simply destroying the original visual representation.
Critical weaknesses
The paper wants to argue that useful representational structure in human brain responses is transferred into the vision model. But the current controls all continue to use the real image–fMRI correspondence. What I did not find is a control that keeps the same repeated-group structure and optimization procedure while destroying the meaningful content of the brain signal.
For example, I would like to see entire repeated-fMRI groups randomly permuted across images. That would preserve the number of images, the five repeated measurements per image, Stage-1 SSL, the brain encoder, the visual residual head, and the Stage-2 relational loss. The only thing removed would be the correct relationship between a given image and its neural response.
This matters because both spaces are allowed to adapt during Stage 2: the fMRI encoder and the visual residual head receive gradients from relational alignment. Stage 1 encourages repeated measurements of the same image to agree and constrains the representation distribution, but that alone does not guarantee that the resulting geometry is semantically meaningful. Without a shuffled-pairing or comparable content-destruction control, I cannot fully rule out a more generic explanation in which same-instance grouping plus co-adaptation acts as a strong regularizer.
To be clear, I am not claiming that this is what is happening. The size and consistency of the gains make the result interesting. But this alternative explanation goes directly to the paper's main story, so I think it matters more than adding another backbone or another benchmark.
A second, smaller point is the language around “held-out” and “OOD” generalization. The held-out categories are held out from Brain-VisA adaptation, not necessarily from the original pretraining data of CLIP, DINOv2, or SimCLR. The appendix explicitly makes this distinction.
So I think the right interpretation is that Brain-VisA improves transfer to categories that were not used in the brain-guided adaptation stage. That is still a meaningful result. I would just avoid language that could be read as generalization to categories fundamentally novel to the pretrained backbone.
Questions for authors
Have you run a control where entire repeated-fMRI groups are randomly reassigned to images before Stage 2? In my view, this is the cleanest way to test whether the improvement really depends on the content of the neural response rather than on grouped-instance regularization and joint co-adaptation. This result would materially affect my score.
If that experiment is not available, do you have any related control using random group embeddings, shuffled brain representations, or frozen random brain geometry? I do not think all of these are necessary, but I do think at least one content-destruction control is important for the main interpretation.
The main DIR experiments use a single optimization seed; the 100 evaluation trials come from resampling the few-shot support set rather than retraining the adaptation model. Have you checked training-seed stability for at least one representative backbone and participant? I would not ask for full multi-seed replication across all eight models, but a small check would be useful given the size of the gains.
Please clarify the intended meaning of “OOD” in the main text. My reading is that these categories are outside the Brain-VisA adaptation split, but not necessarily outside the backbone pretraining distribution.