acceptodds
Withdrawn from ICLR 2027

Rethinking Brain Decoding with CLIP: The Role of Adversarial Robustness

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Brain decoding aims to uncover neural mechanisms by inferring stimulus-related representations from brain signals. In fMRI studies, this is typically achieved by mapping fMRI responses to the latent representations of computational models. Recently, CLIP has become a popular choice for brain decoding due to its rich vision–language embedding space. However, aligning fMRI signals with CLIP representations remains challenging. As CLIP is not explicitly optimized for neural alignment, its representations may capture statistically predictive cues that are only partially reflected in brain activity, limiting decoding performance. In this paper, we investigate whether adversarially robust representations improve neural decoding with CLIP. Adversarial training suppresses reliance on non-robust features and can promote more stable, perceptually structured representations, which may better align with brain activity. We evaluate this by fixing the architecture of fMRI decoders and varying only the target representation (standard CLIP vs. robust variants) on fMRI-image retrieval and zero-shot classification tasks across NSD and GOD datasets. Empirical results show that this simple change consistently improves task performance and yields stronger alignment across multiple metrics. Attribution analysis further reveals consistently low agreement between standard CLIP and its robust variants, suggesting that adversarial robustness reorganizes feature importance in the visual representation. These findings suggest that the choice of target representation influences neural decoding performance and that adversarial robustness may serve as a useful criterion for brain decoding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
brain decoding, fMRI, CLIP, vision-language models, representation learning, adversarial robustness, neural alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2choOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This manuscript investigates whether adversarially robust CLIP representations provide better targets for visual brain decoding than standard CLIP representations. Using linear decoders on NSD and GOD, the authors evaluate retrieval performance, representational alignment, and image attribution patterns. The central motivation is reasonable: representations optimized for visual recognition or image–text alignment may not be optimal targets for decoding brain activity.

Critical strengths
  • The paper examines whether adversarially robust CLIP representations provide more suitable targets for decoding visual information from fMRI.
  • Evaluating retrieval without a downstream image generator reduces confounding from generative priors.
  • The reported results show substantial improvements in several settings across NSD and GOD.
Critical weaknesses
  1. The contribution is too incremental, and the manuscript does not establish sufficient methodological or conceptual novelty. The core intervention is to substitute existing adversarially fine-tuned CLIP checkpoints into the existing linear decoding framework of [2]. The paper introduces neither a new decoding architecture nor a new training objective or robustness method. Its methodological contribution therefore amounts primarily to an evaluation of an existing component substitution. Such an evaluation could still constitute a meaningful scientific contribution if it established a previously unresolved principle about neural decoding. However, the current experiments do not establish why adversarial robustness, specifically, improves decoding, which representational changes are responsible, or under what conditions the effect should generalize. The effective-dimensionality results leave a substantial alternative explanation unresolved, while the attribution analysis does not isolate a brain-signal-specific effect. Moreover, adversarial fine-tuning of CLIP has already been studied as a means of improving visual interpretability [1]; observing different or more concentrated attribution patterns in the present setting does not by itself establish a new neuroscientific finding. The specific evaluation of robust CLIP targets for fMRI decoding is a distinct experimental setting, but that distinction alone is insufficient to establish a substantial conceptual advance. As presented, the paper demonstrates that selected existing checkpoints improve a selected existing pipeline, without establishing either a methodological advance, a sufficiently supported explanatory insight, or a demonstrated practical advantage over representative strong retrieval methods. This is a fundamental limitation of the contribution, rather than a positioning issue that can be resolved by rewriting the related-work discussion.

  2. The current comparisons do not adequately represent state-of-the-art fMRI-to-image retrieval methods. Table 3 primarily compares target encoders within the same linear contrastive or ridge decoding pipeline. Although the method in [2] provides a recent and reasonable controlled baseline, this does not amount to a comparison against the strongest established retrieval systems. Representative methods such as MindEye2 [3] and Lite-Mind [4] are not directly benchmarked; Lite-Mind is particularly relevant because it evaluates both NSD image retrieval and GOD zero-shot classification. BrainCLIP [5] is cited but is not evaluated as a complete competing method. More recent work, including TROI [6], further motivates a clearer assessment of the competitiveness of the selected baseline. Consequently, the reported improvements establish an advantage over the chosen baseline, but do not establish competitiveness with state-of-the-art fMRI retrieval methods. Published accuracy values should not be compared directly across different candidate-pool sizes, subject selections, repetition averaging, feature representations, or training-data budgets. I recommend adding at least one representative strong retrieval baseline under a matched evaluation protocol and testing whether robust target representations also improve that pipeline. Achieving a new state of the art is not necessary for a convincing empirical contribution, but establishing performance relative to strong existing methods and demonstrating benefits beyond the selected linear pipeline would substantially strengthen the practical significance of the findings.

  3. The experiments do not adequately distinguish the effect of adversarial robustness from changes in target dimensionality and geometry. The TeCoA-0 comparison is valuable, but Appendix D reveals a strong alternative explanation: effective rank decreases from 235.50 for CLIP to 89.64 for TeCoA-4, and its correlation with ridge retrieval performance reaches −0.9653. Lower-dimensional targets may simply be easier to estimate from limited, noisy fMRI data using a linear decoder. Improvements in RSA and CKA do not independently rule out this explanation, because these measures also depend on representational geometry. A particularly informative control would compare robust representations with standard CLIP representations compressed to comparable effective dimensionality or matched spectral structure. The authors should also measure robustness directly rather than treating the adversarial training budget as a sufficient proxy.

  4. The subject selection and statistical reporting appear inconsistent across the manuscript. The main text specifies NSD subjects 1, 2, 5, and 7, whereas Table 4 includes eight subjects. Based on the rounded values in that table, the CLIP/FARE-4 means for the four specified subjects are approximately 0.23/0.37, while averaging all eight yields approximately 0.20/0.33, matching Table 3. Table 5 also reports Wilcoxon p-values of 0.008 for some metrics and 0.125 for RSA. For four nonzero paired observations, the minimum two-sided exact Wilcoxon p-value is 0.125; with eight observations, it can reach approximately 0.0078. These results may therefore reflect different analysis populations rather than incorrect calculations, but the current reporting does not make this clear. Each figure and table should identify the included subjects, the statistical unit, and the handling of repeated trials. Effect sizes, uncertainty estimates, and the approach to multiple comparisons should also be reported consistently.

  5. The attribution analysis does not establish that robust targets lead to more faithful use of brain information. The attribution score measures changes in compatibility between decoded fMRI representations and image embeddings after image occlusion. Because the image encoder changes across conditions, differences in attribution maps may reflect the visual encoder’s own response to occlusion rather than improved extraction of stimulus information from fMRI. Controls using shuffled fMRI–image pairings, mismatched trials, or mean brain responses would help determine whether the reported patterns depend on the correct brain signal. An image-only attribution baseline would further separate visual-encoder effects from decoding-specific effects. Low overlap between attribution maps establishes that they differ, but does not establish which map is more faithful. Similarly, increased spatial concentration is not sufficient evidence of improved neuroscientific interpretability.

[1] S. Gong et al. “Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning.” ICLR, 2025.

[2] C. Efird, A. Murphy, J. Zylberberg, and A. Fyshe. “Finding Shared Decodable Concepts and their Negations in the Brain.” ICLR, 2025.

[3] P. S. Scotti et al. “MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data.” ICML, 2024.

[4] Z. Gong et al. “Lite-Mind: Towards Efficient and Robust Brain Representation Network.” ACM Multimedia, 2024.

[5] Y. Ma et al. “BrainCLIP: Brain Representation via CLIP for Generic Natural Visual Stimulus Decoding.” IEEE Transactions on Medical Imaging, 44(10):3962–3972, 2025.

[6] Z. Wang et al. “TROI: Cross-Subject Pretraining with Sparse Voxel Selection for Enhanced fMRI Visual Decoding.” ICASSP, 2025.

Questions for authors

See Weaknesses.

Reviewer_8wwPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Considering the popularity of CLIP in brain encoding & decoding tasks, the paper re-investigates CLIP's feature space and explores how its adversarially robust variants (TeCoA and FARE) could affect the fMRI-to-image retrieval and fMRI-text classification on NSD and GOD datasets. A consistent improvement was observed and was tested to be statistically significant.

Critical strengths
  1. The motivation and research goal of the paper are clear: using robust CLIP features for better brain decoding performance.

  2. The paper is written in good English and is easy to follow.

  3. Extensive experiments show that robust variants of CLIP features help improve brain decoding performance in terms of fMRI-image retrieval and fMRI-text classification on the NSD and GOD datasets.

Critical weaknesses
  1. The novelty is limited. The key technical contribution is replacing the standard CLIP feature with adversarially fine-tuned variants TeCoA and FARE, which are publicly available, as the ground-truth training target of the linear decoder. No new model, objective, or analysis method is introduced.

  2. In the introduction, the paper hypothesizes that standard CLIP encodes non-robust or shortcut features that are not captured by the brain during fMRI acquisition, and that adversarial training removes them. These are never tested, and the only relevant evidence is that robust targets decode better. That is to say, no new insights can be obtained from the paper, especially regarding why the robust targets can improve brain alignment.

  3. The linear decoder from fMRI to the target CLIP feature is a reasonable approach for alignment analysis. However, the claims concern brain decoding broadly. The paper proposes adversarial robustness "as a criterion for selecting target representations in brain decoding" and concludes that target choice is a critical factor in decoding pipelines. However, SOTA CLIP-based decoding systems rely heavily on nonlinear adapters, diffusion priors, and generative models. Plus, their main evaluation is neither retrieval nor text classification. The paper provides no evidence that the effect transfers to them in terms of technical advancement and the task gap.

  4. The paper's organization and presentation need more work. Examples are:

  • Too much content is referred to in the appendix, but the paper itself should stand on its own, with key details included, such as the decoder objective.
  • A methodology section is necessary, although the only trained part is a linear regression.
  • Certain figures require formatting adjustments. Figure 2 leaves excessive whitespace on the sides, and Figure 6 would benefit from a log scale on the x-axis to improve visualization.
Questions for authors
  1. Could the author provide some evidence on the robust feature work on the non-linear decoder?

  2. How is same-image repetition handled in the case of NSD dataset? Do you train and evaluate on individual repetitions or on averaged repetitions?

Reviewer_6VdxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether using adversarially fine-tuned CLIP variants (TeCoA, FARE, at two perturbation budgets, on a shared ViT-L/14 backbone) instead of standard CLIP as the target space improves linear fMRI decoding, with data, decoder architecture, and training held fixed. On NSD and GOD, robust targets give higher fMRI-image retrieval, higher fMRI-text zero-shot accuracy on GOD, and higher Pearson/cosine/RSA/CKA alignment. Occlusion-based attribution shows robust-target decoders use different image regions than the CLIP decoder and show stronger scale-dependent structure. The authors propose adversarial robustness as a criterion for choosing target representations.

Critical strengths
  1. A well-motivated question: adversarially robust models are already known to predict neural responses better on the encoding side, and testing whether the benefit carries over to decoding targets is a natural and useful complement.
  2. Controlled design: one factor varied; shared ViT-L/14 backbone; one linear decoder under two training objectives (contrastive, ridge); two datasets, one with disjoint train/test semantics.
  3. Under contrastive decoding, effects are in a consistent direction across all eight NSD subjects and across both decoder-mediated and decoder-free (RSA/CKA) metrics.
  4. Sensible controls in the appendix: TeCoA-0 (ImageNet fine-tune without adversarial training), a third robust variant (QT-AFT), and an effective-rank analysis.
  5. Clearly written, with limitations stated, and the full pipeline is provided in the supplementary.
Critical weaknesses
  1. The paper's own effective-rank analysis shows that the robust models have much lower effective rank than CLIP and that lower rank tracks better decoding across the five models. A lower-dimensional target is easier to predict from noisy fMRI and has a simpler RDM, so retrieval and RSA/CKA could improve for any low-rank target, robust or not; the two counterarguments in the appendix are indirect. Since the thesis is that robustness specifically is the criterion to select targets by, this alternative needs a direct test, and a cheap one exists: project the CLIP embeddings onto their leading principal components (fitted on training images) so that the effective rank matches each robust model's, retrain the same decoder, and report Top-1 and RSA/CKA. If rank-matched CLIP reproduces most of the gain, the criterion is dimensionality rather than robustness; if not, the claim is much stronger.

  2. All three robust variants are fine-tuned on ImageNet, and the metadata file with the GOD dataset lists 20 of the 50 test images as ILSVRC-2012 training images, with 18 of the 50 test categories among the ImageNet-1k classes. TeCoA additionally learns to map such images to class-name text embeddings, the same format as the fMRI-text classification, and it shows by far the largest GOD gain. The TeCoA-0 control helps, since ImageNet fine-tuning without adversarial perturbation does not produce the gain on its own, and the NSD results are untouched. Still, this is the one point in this review that bears on the validity of a stated claim rather than on interpretation, and the check needs no new training: report GOD retrieval and fMRI-text accuracy separately for the 20 overlapping and the 30 non-overlapping test images. If the robust advantage holds on the non-overlapping images, the zero-shot claim stands; if it concentrates in the overlapping ones, the GOD section can be reframed.

Questions for authors
  1. Take CLIP's image embeddings, project them onto their leading principal components (PCA fitted on the training images), keeping just enough components to match each robust model's effective rank, and retrain the same decoder on this low-rank CLIP target. Does it improve retrieval and RSA/CKA over full CLIP by as much as the robust targets do?

  2. Does the robust advantage on GOD hold on the 30 test images that are not ILSVRC-2012 training images? Split the 50 GOD test images, using the dataset's own metadata file, into the 20 that are ILSVRC-2012 training images and the 30 that are not. Is the advantage of the robust-target decoders over the CLIP decoder, in retrieval and in fMRI-text accuracy, as large on the 30 non-overlapping images as on the 20 overlapping ones?

  3. Do the attribution differences come from the decoders or from the encoders? In the occlusion score only the image embedding changes while the decoded embedding is a fixed query, and the works the authors cite report more concentrated, object-focused attribution for adversarially trained encoders with no brain data involved (Chalasani et al., 2020; Gong et al., 2025). Could the authors recompute the maps for a subset of images with the image's own embedding as the query in place of the decoded one, and report the same IoU, Spearman and entropy comparisons? If the CLIP-vs-robust differences persist, the section describes the encoders; if they shrink, it is a genuine decoding result.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.