- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes Neural Geometry Contrastive Learning (NeuroGeo), a brain-guided method for transferring semantic structure from fMRI representations to pretrained image models. Besides pair-level contrastive learning, the proposed method learns subject-specific virtual anchors to separate subject variation from shared semantic structure. The authors evaluate on DIR fMRI data over held-out ImageNet-21K concepts and on NSD data by integrating NeuroGeo-enhanced representations into existing decoders. Experimental results demonstrate the proposed method improves few-shot classification across various abstraction levels when combined with MindGPT and Yo'Mind.
Critical strengths
- The paper targets a practical obstacle in brain-guided model training, namely, brain and model embeddings are heterogeneous. The pair-level learning objective is easy to motivate and does not require the two modalities to share the same coordinate system.
- This work explicitly considers inter-subject variability and proposes trainable subject-specific virtual anchors. The ablation in Table 2 indicates that anchors improve the reported classification and decoding metrics over the corresponding no-anchor geometric-relation variant.
- The experiments cover CLIP and DINOv2 variants, one-shot classification at multiple abstraction levels, and brain-to-text decoding with two existing decoder pipelines. Ablation studies also compare geometric-relation versus embedding alignment and anchor versus no-anchor settings. The evaluations and conclusions are convincing.
Critical weaknesses
- The additive subject-variation model is a simplifying assumption, and the evidence does not yet establish the anchor interpretation. The paper lacks evidence that learned anchors recover subject offsets or that a model transfers to an unseen subject. The cross-subject claim should match these experiments.
- Eq. 6 specifies a brain-relation query against image-relation candidates. It is not written as a symmetric image-to-brain loss. The text says the same unified InfoNCE loss also covers anchor-to-concept relations, but the notation does not make explicit how the two anchor relations per sampled brain pair enter the batch denominator or how relation direction is handled. Please confirm that the one-way form is intentional and document the exact batch construction, normalization, and weighting.
- The paper provides both plots and quantitative variance-explained/reconstruction-error comparisons in Fig. 5. However, these measures establish better PCA compressibility or concept separation, not by themselves that concepts are more interpretable or that the manifold is more semantically useful. The authors may distinguish those supported observations from the stronger interpretability language, or define a held-out quantitative measure for that claim.
Questions for authors
I expect the authors to address my concerns raised in the weaknesses section, including more evidence about the effectiveness of virtual anchoring, the motivation of brain-to-image one-way contrastive learning, and the interpretability of the method.
2 comments
Thanks for the response. The latter two concerns have been addressed. But the interpretability of the proposed method still does not seem as strong as expected. Thus, I would like to keep my original score.
Thank you very much for your valuable feedback and suggestions. We address each comment in detail below.
About the anchor mechanism: Thank you for this valuable comment. First, we would like to clarify that the purpose of the proposed anchor mechanism is not to enable generalization to unseen subjects, but to address inter-subject variability and facilitate joint representation learning across multiple subjects. Second, to further validate the effectiveness of the anchor mechanism, we have conducted additional controlled experiments comparing fixed, shared, and subject-specific anchors under identical training configurations (See Line 507, Page 10 and Tab. A.3, Page 15). New results show that subject-specific learnable anchors achieve the best performance, suggesting that they can effectively capture inter-subject variability and facilitate joint representation learning across multiple subjects.
Regarding the details of the training procedure: Thank you for this valuable comment. We confirm that the InfoNCE formulation without an additional reverse image-to-brain loss, as NeuroGeo aims to transfer relational structures learned from neural activity to visual representations, rather than enforcing symmetric alignment between the two modalities. For each sampled fMRI pair, we construct three types of relations: one anchor-to-concept relation, one concept-to-anchor relation, and one inter-concept relation, as illustrated in Fig. 2(b). Thus, each batch contains all three types of relations, with each type accounting for one-third of the total relation pairs. The three relation types contribute equally to the loss, without additional weighting coefficients, and follow a consistent source-to-target direction. We have revised the relevant equations and descriptions in the manuscript to clarify these implementation details (See Page 5).
Independent neural validation about the learned semantic structure: Thank you for this insightful comment. To evaluate whether the reshaped image space better captures human semantic structures and exhibits greater interpretability, we have conducted two additional evaluations on unseen concepts.
The results demonstrate that NeuroGeo-enhanced representations exhibit improved alignment with measured neural geometry and human behavioral judgments, providing complementary neural and behavioral evidence for the effectiveness of the reshaped image space.
We have revised the manuscript in accordance with your comments, and all changes have been highlighted in the revised manuscript. Hopefully these will address most of your concerns, and can be taken into consideration when deciding the final score of the paper. Many thanks.