- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This manuscript investigates whether adversarially robust CLIP representations provide better targets for visual brain decoding than standard CLIP representations. Using linear decoders on NSD and GOD, the authors evaluate retrieval performance, representational alignment, and image attribution patterns. The central motivation is reasonable: representations optimized for visual recognition or image–text alignment may not be optimal targets for decoding brain activity.
Critical strengths
- The paper examines whether adversarially robust CLIP representations provide more suitable targets for decoding visual information from fMRI.
- Evaluating retrieval without a downstream image generator reduces confounding from generative priors.
- The reported results show substantial improvements in several settings across NSD and GOD.
Critical weaknesses
-
The contribution is too incremental, and the manuscript does not establish sufficient methodological or conceptual novelty. The core intervention is to substitute existing adversarially fine-tuned CLIP checkpoints into the existing linear decoding framework of [2]. The paper introduces neither a new decoding architecture nor a new training objective or robustness method. Its methodological contribution therefore amounts primarily to an evaluation of an existing component substitution. Such an evaluation could still constitute a meaningful scientific contribution if it established a previously unresolved principle about neural decoding. However, the current experiments do not establish why adversarial robustness, specifically, improves decoding, which representational changes are responsible, or under what conditions the effect should generalize. The effective-dimensionality results leave a substantial alternative explanation unresolved, while the attribution analysis does not isolate a brain-signal-specific effect. Moreover, adversarial fine-tuning of CLIP has already been studied as a means of improving visual interpretability [1]; observing different or more concentrated attribution patterns in the present setting does not by itself establish a new neuroscientific finding. The specific evaluation of robust CLIP targets for fMRI decoding is a distinct experimental setting, but that distinction alone is insufficient to establish a substantial conceptual advance. As presented, the paper demonstrates that selected existing checkpoints improve a selected existing pipeline, without establishing either a methodological advance, a sufficiently supported explanatory insight, or a demonstrated practical advantage over representative strong retrieval methods. This is a fundamental limitation of the contribution, rather than a positioning issue that can be resolved by rewriting the related-work discussion.
-
The current comparisons do not adequately represent state-of-the-art fMRI-to-image retrieval methods. Table 3 primarily compares target encoders within the same linear contrastive or ridge decoding pipeline. Although the method in [2] provides a recent and reasonable controlled baseline, this does not amount to a comparison against the strongest established retrieval systems. Representative methods such as MindEye2 [3] and Lite-Mind [4] are not directly benchmarked; Lite-Mind is particularly relevant because it evaluates both NSD image retrieval and GOD zero-shot classification. BrainCLIP [5] is cited but is not evaluated as a complete competing method. More recent work, including TROI [6], further motivates a clearer assessment of the competitiveness of the selected baseline. Consequently, the reported improvements establish an advantage over the chosen baseline, but do not establish competitiveness with state-of-the-art fMRI retrieval methods. Published accuracy values should not be compared directly across different candidate-pool sizes, subject selections, repetition averaging, feature representations, or training-data budgets. I recommend adding at least one representative strong retrieval baseline under a matched evaluation protocol and testing whether robust target representations also improve that pipeline. Achieving a new state of the art is not necessary for a convincing empirical contribution, but establishing performance relative to strong existing methods and demonstrating benefits beyond the selected linear pipeline would substantially strengthen the practical significance of the findings.
-
The experiments do not adequately distinguish the effect of adversarial robustness from changes in target dimensionality and geometry. The TeCoA-0 comparison is valuable, but Appendix D reveals a strong alternative explanation: effective rank decreases from 235.50 for CLIP to 89.64 for TeCoA-4, and its correlation with ridge retrieval performance reaches −0.9653. Lower-dimensional targets may simply be easier to estimate from limited, noisy fMRI data using a linear decoder. Improvements in RSA and CKA do not independently rule out this explanation, because these measures also depend on representational geometry. A particularly informative control would compare robust representations with standard CLIP representations compressed to comparable effective dimensionality or matched spectral structure. The authors should also measure robustness directly rather than treating the adversarial training budget as a sufficient proxy.
-
The subject selection and statistical reporting appear inconsistent across the manuscript. The main text specifies NSD subjects 1, 2, 5, and 7, whereas Table 4 includes eight subjects. Based on the rounded values in that table, the CLIP/FARE-4 means for the four specified subjects are approximately 0.23/0.37, while averaging all eight yields approximately 0.20/0.33, matching Table 3. Table 5 also reports Wilcoxon p-values of 0.008 for some metrics and 0.125 for RSA. For four nonzero paired observations, the minimum two-sided exact Wilcoxon p-value is 0.125; with eight observations, it can reach approximately 0.0078. These results may therefore reflect different analysis populations rather than incorrect calculations, but the current reporting does not make this clear. Each figure and table should identify the included subjects, the statistical unit, and the handling of repeated trials. Effect sizes, uncertainty estimates, and the approach to multiple comparisons should also be reported consistently.
-
The attribution analysis does not establish that robust targets lead to more faithful use of brain information. The attribution score measures changes in compatibility between decoded fMRI representations and image embeddings after image occlusion. Because the image encoder changes across conditions, differences in attribution maps may reflect the visual encoder’s own response to occlusion rather than improved extraction of stimulus information from fMRI. Controls using shuffled fMRI–image pairings, mismatched trials, or mean brain responses would help determine whether the reported patterns depend on the correct brain signal. An image-only attribution baseline would further separate visual-encoder effects from decoding-specific effects. Low overlap between attribution maps establishes that they differ, but does not establish which map is more faithful. Similarly, increased spatial concentration is not sufficient evidence of improved neuroscientific interpretability.
[1] S. Gong et al. “Boosting the Visual Interpretability of CLIP via Adversarial Fine-Tuning.” ICLR, 2025.
[2] C. Efird, A. Murphy, J. Zylberberg, and A. Fyshe. “Finding Shared Decodable Concepts and their Negations in the Brain.” ICLR, 2025.
[3] P. S. Scotti et al. “MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data.” ICML, 2024.
[4] Z. Gong et al. “Lite-Mind: Towards Efficient and Robust Brain Representation Network.” ACM Multimedia, 2024.
[5] Y. Ma et al. “BrainCLIP: Brain Representation via CLIP for Generic Natural Visual Stimulus Decoding.” IEEE Transactions on Medical Imaging, 44(10):3962–3972, 2025.
[6] Z. Wang et al. “TROI: Cross-Subject Pretraining with Sparse Voxel Selection for Enhanced fMRI Visual Decoding.” ICASSP, 2025.
Questions for authors
See Weaknesses.