acceptodds
Withdrawn from ICLR 2027

Brain-VisA: Brain-Vision Alignment via Self-Supervised Brain Representation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Human brain activity contains rich perceptual and semantic information that could complement pretrained models, yet exploiting this information is challenging. We investigate whether brain–vision alignment can improve the representations and generalization of pretrained vision models using paired image–fMRI data alone, without any behavioral supervision or category information. To enable vision models to benefit from brain–vision alignment, we leverage self-supervised learning to enhance the conceptual and semantic structure of brain representations. Specifically, we propose a two-stage **Brain-Vis**ion **A**lignment (**Brain-VisA**) framework. First, a joint-embedding predictive architecture (JEPA) learns structured brain representations through self-supervision, using repeated fMRI responses to the same image as natural positive views. Second, relational alignment jointly adapts brain and visual representations while retaining brain self-supervision. We find that the learned brain representations exhibit semantic concept structure across eight pretrained vision models from SimCLR, CLIP, and DINOv2, despite the absence of semantic supervision. Specifically, Brain-VisA improves one-shot classification accuracy for high-level semantic concepts by an average of 13.70% over the unadapted models. It further exceeds the category-informed brain–vision alignment baseline by an average of 10.14% on high-level abstract semantic classification, while requiring less training data and time. The aligned models also improve on generalization tasks including out-of-distribution recognition and the odd-one-out task. These results suggest that self-supervised learning can extract useful representational structure from limited, noisy brain signals, and that brain–vision alignment can improve the semantic representation structure and generalization of pretrained vision models without relying on any behavioral or category information.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Self-supervised learning, Brain–vision alignment, Semantic generalization, Relational alignment, Variance-Invariance-Sketching Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HTpfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether paired image–fMRI data can be used to improve pretrained visual representations without using category labels or behavioral supervision during adaptation. Brain-VisA has two stages. First, the authors train an fMRI encoder with a self-supervised objective that treats repeated fMRI measurements of the same image as positive views. The intuition is that repeated measurements share stimulus-related information but differ in trial-specific and measurement noise. Second, the resulting brain representation is aligned with a pretrained visual representation using relational knowledge distillation. The visual backbone is frozen, while a small residual head adapts its output representation.
The experiments cover eight visual backbones from SimCLR, CLIP, and DINOv2. The reported gains on high-level abstract classification are large, and the improvements also extend to held-out adaptation categories, semantic OOD tasks, and THINGS odd-one-out judgments. At the same time, the adapted models mostly retain their original ImageNet category-recognition performance.
I like the basic idea. Repeated neural measurements are a natural source of self-supervision, and using relational alignment rather than direct feature matching makes sense. My main concern is more specific: I am not yet convinced that the experiments isolate the contribution of the brain signal content itself. The paper shows that the full training procedure works very well, but the strongest interpretation is that useful semantic structure from human neural responses is what causes the improvement. I think that claim needs a stronger control.

Critical strengths

The first strength is the basic use of repeated measurements. In fMRI, repeated trials are often treated as redundant or mainly useful for averaging. Here they become a source of self-supervision: the shared image acts as the common factor, while trial-specific variation is treated as nuisance. That is a clean and well-motivated idea. The second strength is the breadth of the empirical result. The method improves multiple SimCLR, CLIP, and DINOv2 backbones rather than one carefully chosen model. The paper also includes sensible comparisons against autoencoding, denoising autoencoding, contrastive brain SSL, pointwise cross-modal InfoNCE, and different two-stage schedules.
I also think the ImageNet retention result is important. The adaptation substantially changes high-level semantic performance without simply destroying the original visual representation.

Critical weaknesses

The paper wants to argue that useful representational structure in human brain responses is transferred into the vision model. But the current controls all continue to use the real image–fMRI correspondence. What I did not find is a control that keeps the same repeated-group structure and optimization procedure while destroying the meaningful content of the brain signal. For example, I would like to see entire repeated-fMRI groups randomly permuted across images. That would preserve the number of images, the five repeated measurements per image, Stage-1 SSL, the brain encoder, the visual residual head, and the Stage-2 relational loss. The only thing removed would be the correct relationship between a given image and its neural response. This matters because both spaces are allowed to adapt during Stage 2: the fMRI encoder and the visual residual head receive gradients from relational alignment. Stage 1 encourages repeated measurements of the same image to agree and constrains the representation distribution, but that alone does not guarantee that the resulting geometry is semantically meaningful. Without a shuffled-pairing or comparable content-destruction control, I cannot fully rule out a more generic explanation in which same-instance grouping plus co-adaptation acts as a strong regularizer. To be clear, I am not claiming that this is what is happening. The size and consistency of the gains make the result interesting. But this alternative explanation goes directly to the paper's main story, so I think it matters more than adding another backbone or another benchmark. A second, smaller point is the language around “held-out” and “OOD” generalization. The held-out categories are held out from Brain-VisA adaptation, not necessarily from the original pretraining data of CLIP, DINOv2, or SimCLR. The appendix explicitly makes this distinction.
So I think the right interpretation is that Brain-VisA improves transfer to categories that were not used in the brain-guided adaptation stage. That is still a meaningful result. I would just avoid language that could be read as generalization to categories fundamentally novel to the pretrained backbone.

Questions for authors

Have you run a control where entire repeated-fMRI groups are randomly reassigned to images before Stage 2? In my view, this is the cleanest way to test whether the improvement really depends on the content of the neural response rather than on grouped-instance regularization and joint co-adaptation. This result would materially affect my score.

If that experiment is not available, do you have any related control using random group embeddings, shuffled brain representations, or frozen random brain geometry? I do not think all of these are necessary, but I do think at least one content-destruction control is important for the main interpretation.

The main DIR experiments use a single optimization seed; the 100 evaluation trials come from resampling the few-shot support set rather than retraining the adaptation model. Have you checked training-seed stability for at least one representative backbone and participant? I would not ask for full multi-seed replication across all eight models, but a small check would be useful given the size of the gains.

Please clarify the intended meaning of “OOD” in the main text. My reading is that these categories are outside the Brain-VisA adaptation split, but not necessarily outside the backbone pretraining distribution.

Reviewer_iUFDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Brain-VisA, a two-stage framework. First, repeated fMRI responses elicited by presenting the same image multiple times are treated as positive samples. A model based on a joint-embedding predictive architecture is then trained with self-supervised objectives to learn stable brain representations. Second, the relational structure between the brain representation space and the representation space of pretrained vision models is aligned, allowing brain signals to guide visual representation learning. The experiments show that, even without semantic supervision, the learned brain representations can improve the semantic structure of representations in vision models.

Critical strengths
  1. The problem is interesting and the motivation is clear. The paper addresses a valuable question: can neural signals, rather than labels or behavioral information, be directly used to improve vision models?
  2. The experiments cover multiple models and tasks. Rather than demonstrating the method on a single backbone, the authors evaluate three model families comprising eight models. They also test tasks including abstract semantic classification and category generalization.
Critical weaknesses
  1. The method is not described clearly enough, which makes it difficult to understand. Voxel-level fMRI data are typically four-dimensional, consisting of three spatial dimensions and a temporal dimension. And temporal dynamics are a key characteristic of fMRI signals. However, the paper refers to the input as a “voxel-response vector”. It is unclear how this vector is derived from the original fMRI data and whether the temporal dimension is retained or discarded.
  2. The methodological novelty appears limited. I did not identify a clear methodological contribution beyond combining existing techniques. For example, to prevent representation collapse when only positive samples are available, the authors introduce the existing VISReg to constrain the representations. They also directly apply RKD to align the pairwise relationships among images in the two representation spaces. In addition, several key modules—including the fMRI encoder, projection head, and residual head—are implemented as MLP. Can these tasks be reliably accomplished with such simple MLP? The authors should clarify the specific contribution of the proposed framework and provide stronger justification for these design choices.
  3. Important hyperparameter experiments are missing. The method section states that a view variable (a) is restricted to ({0,1,2}), but the reason for this choice is unclear. The authors should explain why these three values are imposed and investigate the sensitivity of the method to this hyperparameter. In particular, is using two masked views consistently optimal across different settings?
  4. Important control experiments are missing. The paper attributes the performance gains to representational structure contained in the brain signals, but it does not include experiments with shuffled brain-image pairings or with random fMRI signals replacing the real signals. Compared with the purely visual baseline, the proposed model introduces additional trainable parameters through the residual head, as well as the RKD-based regularization. Both factors could independently affect the final performance. These control experiments are therefore necessary to establish that the gains specifically arise from meaningful brain signals.
Questions for authors

Could the authors clarify how the voxel-response vector is constructed from the original fMRI data and whether temporal information is preserved? I would also appreciate further clarification on the methodological novelty and the motivation behind the current design choices. Additionally, could the authors provide more analysis of the view variable and conduct control experiments to verify that the improvements indeed come from meaningful brain signals rather than additional parameters or regularization? If the authors can clarify the novelty of their approach beyond simply combining existing techniques, and provide additional experimental evidence demonstrating the effectiveness of the proposed modules, I would be willing to reconsider my score.

Reviewer_VcyUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether paired image–fMRI data alone—without behavioral supervision or category labels—can improve the semantic representations and generalization of pretrained vision models. The authors propose Brain-VisA, a two-stage framework: (1) a JEPA-style self-supervised objective that treats repeated fMRI responses to the same image as natural positive views, combining cross-measurement agreement with isotropic-Gaussian distributional regularization (VISReg); and (2) relational co-adaptation (RKD) that aligns the relational geometry of brain and visual representation spaces via a lightweight, zero-initialized residual head on a frozen backbone, while retaining the brain SSL loss. At inference only the image encoder is used. Across eight pretrained models (SimCLR, CLIP, DINOv2), Brain-VisA improves one-shot high-level abstract classification by 13.70% over unadapted models and 10.14% over the category-informed MRS baseline, with additional gains on held-out categories, Coarse-OOD, and THINGS odd-one-out agreement.

Critical strengths

1,Well-motivated and clean problem formulation. Isolating the contribution of brain signals by removing category supervision (the key confound in MRS) is a meaningful and well-articulated research question. 2,Sensible architectural choices. Using repeated measurements as positive views is a natural fit for noisy fMRI; the negative-free JEPA + distributional regularization design is a principled way to suppress trial-specific noise without the semantic-collapse risk of contrastive negatives. The frozen-backbone + residual-head (LoRA-like) adaptation is lightweight and preserves fine-grained recognition. 3,Reasonable ablation coverage. Comparisons against AE/DAE/CL, single- vs. cross-measurement positive groups, two-stage schedule variants (Without/Before/With), RKD vs. InfoNCE, and a data-scale study collectively support most design decisions. 4,Practical inference setup. Requiring only images at test time makes the brain-derived supervision broadly reusable.

Critical weaknesses

W1. Checkpoint selection is unspecified and may introduce optimistic bias. Main results use the Stage-2 epoch-100 checkpoint for Brain-VisA but epoch-20 for MRS, while the authors also state that DIR uses all adaptation images "without an adaptation validation split." Without a held-out validation set, the criterion for selecting these checkpoints is unclear. If selection was guided by downstream evaluation performance, it constitutes tuning on the test objective and would inflate results. The epoch asymmetry between Brain-VisA and MRS also needs justification.

W2. Questionable fairness of the MRS baseline. MRS collapses on concrete-category recognition (ImageNet-150: 49.02% vs. 73.43% for Vanilla; ImageNet-1K: 35.56% vs. 56.65%). Combined with its much shorter training (epoch 20 vs. 100), this large degradation raises concerns that MRS may be under-tuned or evaluated under unfavorable conditions. Since the headline claim of exceeding MRS by 10.14 points depends on this baseline, its configuration and fairness must be demonstrated.

W3. The headline improvement is measured in-sample. The most prominent number (+13.70% over Vanilla) is computed on train-abstract-2/3/4, i.e., on the very adaptation images whose fMRI responses shaped the representation. Although the abstract labels do not supervise adaptation, this remains an in-distribution measurement of semantic organization and is weaker evidence than the held-out results. The more convincing test-abstract gains (+10.14 / +12.74) are obtained on only 50 held-out categories, which should be foregrounded instead.

W4. Limited scale and external validity. Each participant provides only 1,200 images / 6,000 measurements across 200 categories, with just three DIR participants (plus one NSD participant). All models are participant-specific, and no cross-participant generalization is demonstrated (acknowledged by the authors). The gains are also uneven across backbones (e.g., substantially smaller for DINOv2 than CLIP), which is not analyzed.

W5. Metric-definition choices may inflate the behavioral result. The THINGS odd-one-out evaluation counts ties as correct and uses a single designated image per concept. Given that the reported improvement is small (45.05% → 47.32%, +2.27 points), such scoring choices are non-trivial and warrant an ablation (e.g., ties excluded or counted as incorrect) to confirm the effect is real.

W6. Mechanistic claims rest largely on qualitative evidence. The central narrative—that cross-measurement agreement extracts stimulus-consistent semantic structure while suppressing trial-specific noise—is supported primarily by t-SNE visualizations (Figures 3f, 4, 7). t-SNE is sensitive to hyperparameters and susceptible to selective presentation; a quantitative measure of semantic structure (e.g., silhouette scores, kNN consistency, or representational-similarity analysis against a semantic hierarchy) would substantially strengthen these claims.

Questions for authors
  1. What happens under a permuted image–fMRI pairing control? Do the gains disappear, as the central hypothesis would predict?
  2. How were MRS hyperparameters selected, and is the ImageNet-1K drop reproducible / expected from the original MRS setup?
  3. How sensitive are results to the WordNet abstraction grouping? Do alternative groupings yield consistent conclusions?
  4. Any preliminary evidence on cross-subject generalization, even negative results?
  5. How robust are the gains to the choice of λalign\lambda_{\text{align}} and the masking ratio?

Suggestions to strengthen the paper: report multiple training seeds with standard deviations and significance tests; specify and equalize the checkpoint-selection protocol across all methods, ideally with a proper validation split; justify or re-tune the MRS baseline; present held-out (test-abstract) results as the headline rather than in-sample metrics; and complement t-SNE with quantitative structure metrics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.