acceptodds
Under review as a conference paper at ICLR 2027

Neural Geometry Contrastive Learning: Towards Brain-Machine Intelligence Enhancement

Abstract

Modern artificial neural networks (ANNs) have exhibited remarkable capabilities in learning representations from large-scale data, yet improving their complex cognitive abilities (e.g., understanding abstract concepts) remains a major challenge. One potential reason behind this limitation is that there are still many discrepancies between the representational structure formed by ANNs and that of the human brain. Therefore, we propose neural geometry contrastive learning (NeuroGeo), a novel method that learns representations from human brain activity and transfers their representational geometric structures to ANNs. Unlike most existing methods, NeuroGeo explicitly aligns geometric structures by preserving how representations vary relative to one another, thereby avoiding strict alignment constraints across heterogeneous modality spaces. This formulation yields rich combinatorial supervision from limited brain activity. Moreover, by introducing learnable virtual anchors and representing each embedding as an anchor-relative displacement, NeuroGeo establishes adaptive reference origins for different modalities and subjects, unifying cross-subject representation learning and relational structure alignment within a contrastive learning framework. We find that NeuroGeo effectively transfers human representational structures to ANNs, significantly enhancing their ability to represent abstract concepts. Results highlight that NeuroGeo yields highly interpretable concept representations, thus leading to substantial performance gains in downstream tasks such as few-shot learning and neural semantic decoding. Code will be released upon acceptance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural decoding, Representation alignment, Representation learning, Brain-guided learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_dETwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Neural Geometry Contrastive Learning (NeuroGeo), a brain-guided method for transferring semantic structure from fMRI representations to pretrained image models. Besides pair-level contrastive learning, the proposed method learns subject-specific virtual anchors to separate subject variation from shared semantic structure. The authors evaluate on DIR fMRI data over held-out ImageNet-21K concepts and on NSD data by integrating NeuroGeo-enhanced representations into existing decoders. Experimental results demonstrate the proposed method improves few-shot classification across various abstraction levels when combined with MindGPT and Yo'Mind.

Critical strengths
  1. The paper targets a practical obstacle in brain-guided model training, namely, brain and model embeddings are heterogeneous. The pair-level learning objective is easy to motivate and does not require the two modalities to share the same coordinate system.
  2. This work explicitly considers inter-subject variability and proposes trainable subject-specific virtual anchors. The ablation in Table 2 indicates that anchors improve the reported classification and decoding metrics over the corresponding no-anchor geometric-relation variant.
  3. The experiments cover CLIP and DINOv2 variants, one-shot classification at multiple abstraction levels, and brain-to-text decoding with two existing decoder pipelines. Ablation studies also compare geometric-relation versus embedding alignment and anchor versus no-anchor settings. The evaluations and conclusions are convincing.
Critical weaknesses
  1. The additive subject-variation model is a simplifying assumption, and the evidence does not yet establish the anchor interpretation. The paper lacks evidence that learned anchors recover subject offsets or that a model transfers to an unseen subject. The cross-subject claim should match these experiments.
  2. Eq. 6 specifies a brain-relation query against image-relation candidates. It is not written as a symmetric image-to-brain loss. The text says the same unified InfoNCE loss also covers anchor-to-concept relations, but the notation does not make explicit how the two anchor relations per sampled brain pair enter the batch denominator or how relation direction is handled. Please confirm that the one-way form is intentional and document the exact batch construction, normalization, and weighting.
  3. The paper provides both plots and quantitative variance-explained/reconstruction-error comparisons in Fig. 5. However, these measures establish better PCA compressibility or concept separation, not by themselves that concepts are more interpretable or that the manifold is more semantically useful. The authors may distinguish those supported observations from the stronger interpretability language, or define a held-out quantitative measure for that claim.
Questions for authors

I expect the authors to address my concerns raised in the weaknesses section, including more evidence about the effectiveness of virtual anchoring, the motivation of brain-to-image one-way contrastive learning, and the interpretability of the method.

2 comments

Thank you very much for your valuable feedback and suggestions. We address each comment in detail below.


About the anchor mechanism: Thank you for this valuable comment. First, we would like to clarify that the purpose of the proposed anchor mechanism is not to enable generalization to unseen subjects, but to address inter-subject variability and facilitate joint representation learning across multiple subjects. Second, to further validate the effectiveness of the anchor mechanism, we have conducted additional controlled experiments comparing fixed, shared, and subject-specific anchors under identical training configurations (See Line 507, Page 10 and Tab. A.3, Page 15). New results show that subject-specific learnable anchors achieve the best performance, suggesting that they can effectively capture inter-subject variability and facilitate joint representation learning across multiple subjects.


Regarding the details of the training procedure: Thank you for this valuable comment. We confirm that the InfoNCE formulation without an additional reverse image-to-brain loss, as NeuroGeo aims to transfer relational structures learned from neural activity to visual representations, rather than enforcing symmetric alignment between the two modalities. For each sampled fMRI pair, we construct three types of relations: one anchor-to-concept relation, one concept-to-anchor relation, and one inter-concept relation, as illustrated in Fig. 2(b). Thus, each batch contains all three types of relations, with each type accounting for one-third of the total relation pairs. The three relation types contribute equally to the loss, without additional weighting coefficients, and follow a consistent source-to-target direction. We have revised the relevant equations and descriptions in the manuscript to clarify these implementation details (See Page 5).


Independent neural validation about the learned semantic structure: Thank you for this insightful comment. To evaluate whether the reshaped image space better captures human semantic structures and exhibits greater interpretability, we have conducted two additional evaluations on unseen concepts.

  • First, we compare the representational similarity matrices derived from the original and NeuroGeo-enhanced image representations with those derived from fMRI responses. Results show that the Spearman correlation increases from 0.2317 to 0.5087 after NeuroGeo enhancement (See Figure 6, Page 9).
  • Second, we evaluate the behavioral alignment of both the original and NeuroGeo-enhanced image representations with human similarity judgments from the THINGS dataset. Results show that the triplet odd-one-out accuracy improves from 43.7% to 49.4% after NeuroGeo enhancement (See Figure 6, Page 9).

The results demonstrate that NeuroGeo-enhanced representations exhibit improved alignment with measured neural geometry and human behavioral judgments, providing complementary neural and behavioral evidence for the effectiveness of the reshaped image space.


We have revised the manuscript in accordance with your comments, and all changes have been highlighted in the revised manuscript. Hopefully these will address most of your concerns, and can be taken into consideration when deciding the final score of the paper. Many thanks.

Thanks for the response. The latter two concerns have been addressed. But the interpretability of the proposed method still does not seem as strong as expected. Thus, I would like to keep my original score.

Reviewer_MrZLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NeuroGeo transfers relational geometry from fMRI to ANNs by aligning pairwise concept relation vectors with an InfoNCE loss, using learnable virtual anchors to absorb subject-specific variation. It reshapes CLIP and DINOv2 latent spaces and improves one-shot classification and brain-to-text decoding on DIR and NSD; PCA analyses suggest more human-like low-dimensional organization.

Critical strengths

Aligning relative concept geometry rather than point-wise embeddings is conceptually appealing and naturally suited to cross-subject variability; the virtual-anchor formulation is simple and integrates cleanly into contrastive training.

Empirically, gains are consistent across CLIP and DINOv2, multiple abstraction levels, and two downstream tasks, and ablations support both relational alignment and anchors. These results make the practical value of brain-guided representation reshaping credible.

Critical weaknesses

The novelty is incremental: AligNet (Nature, 2025) already transfers human-aligned structure by matching relative similarity distributions rather than absolute embeddings; NeuroGeo's advance is a specific contrastive relational-vector formulation plus anchors, not a new paradigm.

The pairing and sampling protocol is not clarified. Training uses 206,848 images but only 6,000 fMRI signals, with unclear pairing/sampling and no category-only or image-only control, so category structure may explain the gains. The ablation is insufficient: Table 2 changes the loss when adding anchors, so it cannot isolate the anchor mechanism from extra relational supervision.

Questions for authors

(1) The training set contains 206,848 ImageNet-21K images but only 6,000 fMRI signals over the same 150 categories. Please clarify the pairing and sampling protocol: how are images matched to fMRI signals; are relational pairs constructed at the category level or the sample level; and how do multiple images correspond to a single or few fMRI responses?

(2) Because category structure alone may explain the gains, can you add a category-only control and an image-only/random-brain control under matched training budgets?

(3) The ablation does not isolate the anchor mechanism: enabling anchors adds anchor-to-concept relations and changes the loss composition. Can you compare fixed, shared, and subject-specific anchors under identical losses, relation-pair counts, and training budgets?

(4) Figure 5 uses PCA concentration and reconstruction error to infer human-like geometry. Can the authors report agreement with held-out measured fMRI distances or existing human similarity judgments?

2 comments

Thank you very much for your valuable feedback and suggestions. We address each comment in detail below.


Regarding the details of training data construction: Thank you for pointing out this lack of clarity. Following [1, 2], NeuroGeo employs category-level semantic pairing, i.e., an fMRI recording can be paired with any image belonging to its corresponding stimulus category. This strategy allows us to leverage a substantially larger image dataset (206,848 images) alongside the limited fMRI recordings (6,000 signals), without requiring a one-to-one correspondence between individual images and fMRI responses. This strategy requires both modalities to cover the same categories. During training, we randomly sample two fMRI recordings corresponding to different stimulus categories and then randomly select one image from the corresponding category for each recording to construct the paired geometric relations. The details have been added to the Appendix (See Page 15).

[1] Mind artist: Creating artistic snapshots with human thought, CVPR 2024.

[2] Bridging the semantic latent space between brain and machine: Similarity is all you need, AAAI 2024.


Independent neural validation (human similarity judgments): Thank you for this insightful comment. To evaluate whether the reshaped image space better reflects the human semantic geometry,

  • we evaluate the behavioral alignment of both the original and NeuroGeo-enhanced image representations with human similarity judgments from the THINGS dataset. Results show that the triplet odd-one-out accuracy improves from 43.7% to 49.4% after NeuroGeo enhancement (See Figure 6, Page 9);
  • we compare the representational similarity matrices derived from the original and NeuroGeo-enhanced image representations with those derived from fMRI responses. Results show that the Spearman correlation increases from 0.2317 to 0.5087 after NeuroGeo enhancement (See Figure 6, Page 9).

The results demonstrate that NeuroGeo-enhanced representations exhibit improved alignment with human semantic geometry, providing complementary neural and behavioral evidence for the effectiveness of NeuroGeo.


Regarding the proposed paradigm: Thank you for this comment. We agree that AligNet has demonstrated the effectiveness of transferring human-like relational structures to ANNs. However, there are fundamental differences between AligNet and NeuroGeo. AligNet relies on human similarity judgments to construct supervision, which primarily provide discrete similarity judgments between concepts. In contrast, NeuroGeo learns relational structures directly from neural activity (fMRI), potentially capturing richer semantic information such as continuous geometric relationships between concepts. Furthermore, a major challenge in learning from neural activity is inter-subject variability, and NeuroGeo introduces learnable virtual anchors to accommodate inter-subject variability, enabling representation alignment across multiple subjects within a unified framework. Therefore, our work provides a new neural activity-guided learning framework that directly transfers human knowledge from neural signals to ANNs while naturally supporting cross-subject representation learning.


New control experiments: Thank you for this valuable suggestion. We have added a category-only control and an image-only/random-brain control experiments under matched training budgets (See Line 517, Page 10 and Tab. A.4, Page 15). The results show that i) category-only supervision cannot improve classification performance across different abstraction level. For example, supervision based on the 3 supercategories improves 3-class classification accuracy but significantly degrades performance on the 8-class and 50-class tasks; ii) randomly pairing fMRI recordings with images lead to performance degradation across all classification tasks. These findings suggest that fMRI signals provide richer hierarchical semantic information beyond category-level supervision, enabling NeuroGeo to improve generalization of ANNs across different levels of semantic abstraction.


New ablation experiments on the anchor mechanism: Thank you for this valuable comment. We have conducted additional experiments comparing fixed, shared, and subject-specific anchors while keeping the loss function, number of relation pairs, and training budgets identical (See Line 507, Page 10 and Tab. A.3, Page 15). The results show that subject-specific learnable anchors consistently outperform fixed and shared anchors, supporting their effectiveness in handling inter-subject variability during joint representation learning across multiple subjects.


We have revised the manuscript in accordance with your comments, and all changes have been highlighted in the revised manuscript. Hopefully these will address most of your concerns, and can be taken into consideration when deciding the final score of the paper. Many thanks.

Thank you for the detailed feedbacks. The clarifications and the additional experiments address most of my concerns, while the concern regarding the incremental novelty remains. So I keep the score unchanged.

Reviewer_zJ3gOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper introduces Neural Geometry Contrastive Learning for transferring human brain's representational geometric structures to ANNs. NeuroGeo jointly trains an fMRI encoder and a projection head on frozen vision-model features. Its contrastive objective aligns differences between embeddings, supplemented by representations relative to learned subject anchors. The authors evaluate one-shot classification at three levels of a single ImageNet-derived hierarchy, brain-to-text decoding on NSD, and representation structure through PCA. The results show promising evidence that the learned representations can be useful.

Critical strengths

I identify the following as the strengths of this paper:

  1. The problem is interesting and the methodology is intuitive to follow: Extracting useful supervision from limited neural recordings is a worthwhile alternative to simply scaling visual training data. Representing both pairwise differences and anchor-relative embeddings within one contrastive framework is an interesting design.
  2. The experiments provide meaningful evidence for NeuroGeo: Figure 3 shows improvements over Chen et al. across CLIP and DINOv2 backbones and all three abstraction levels, supporting the approach’s applicability across model families. Table 1 shows equivalent or improved brain-to-text decoding with both MindGPT and Yo’Mind, demonstrating that NeuroGeo’s representations benefit multiple existing decoding frameworks. Figure 5 shows clearer semantic grouping and greater variance captured by the leading principal components, suggesting a more compact representation with meaningful conceptual organization.
  3. Ablations supports the usefulness of the proposed components: Table 2 shows geometric alignment and anchors are more useful compared to plain embedding alignment in the classification task.
Critical weaknesses

The following are the critical weaknesses that I found:

  1. Joint optimization of the fmri decoder and image projector: Optimizing them jointly doesn't isolate the affect of what measured brain activity contributes using a frozen fmri encoder and agreement between their outputs therefore does not itself establish preservation of the original neural geometry.
  2. The experimental evaluation remains limited: Classification uses three labelings of a single ImageNet-derived collection, while Figure 3 lacks exact scores and suggests some fine-grained performance degradation for DINOv2. Decoding gains over Yo’Mind are modest, and their significance is difficult to assess without per-subject results and matched alignment baselines.
  3. Data used for experimentation is not clearly explained: The authors report that training data consists of 206,848 images but only 6,000 fMRI signals, covering the same categories. The formulation initially assumes paired observations, but the manuscript does not adequately explain how this much larger image collection enters that pairing.
Questions for authors
  1. Training data construction: How are 206,848 images paired with 6,000 fMRI signals? Which category labels or augmentation procedures enter this process?
  2. Classification results: Can the authors provide all Figure 3 values, uncertainty definitions, and discuss the apparent DINOv2 fine-classification degradation? Does the benefit transfer beyond the existing ImageNet hierarchy?
  3. Independent neural validation: Does the enhanced image representation better match held-out measured-fMRI geometry, beyond agreement with the jointly trained encoder?
  4. Decoding attribution: What gains remain over a fully matched embedding-alignment baseline, with identical decoder, captions, data, and evaluation? Are those gains consistent across subjects and training seeds?
2 comments

Thank you very much for your valuable feedback and suggestions. We address each comment in detail below.


Regarding the details of training data construction: Thank you for pointing out this lack of clarity. Following [1, 2], NeuroGeo employs category-level semantic pairing, i.e., an fMRI recording can be paired with any image belonging to its corresponding stimulus category. This strategy allows us to leverage a substantially larger image dataset (206,848 images) alongside the limited fMRI recordings (6,000 signals), without requiring a one-to-one correspondence between individual images and fMRI responses (this strategy requires both modalities to cover the same categories). During training, we randomly sample two fMRI recordings corresponding to different stimulus categories and then randomly select one image from the corresponding category for each recording to construct the paired geometric relations. The details have been added to the Appendix (See Page 15).

[1] Mind artist: Creating artistic snapshots with human thought, CVPR 2024.

[2] Bridging the semantic latent space between brain and machine: Similarity is all you need, AAAI 2024.


Independent neural validation: Thank you for this insightful comment. To evaluate whether the reshaped image space better reflects the original neural geometry, we conduct two additional evaluations on unseen concepts.

  • First, we compare the representational similarity matrices derived from the original and NeuroGeo-enhanced image representations with those derived from fMRI responses. Results show that the Spearman correlation increases from 0.2317 to 0.5087 after NeuroGeo enhancement (See Figure 6, Page 9).
  • Second, we further evaluate the behavioral alignment of both the original and NeuroGeo-enhanced image representations with human similarity judgments from the THINGS dataset. Results show that the triplet odd-one-out accuracy improves from 43.7% to 49.4% after NeuroGeo enhancement (See Figure 6, Page 9).

The results demonstrate that NeuroGeo-enhanced representations exhibit improved alignment with measured neural geometry and human behavioral judgments, providing complementary neural and behavioral evidence for the effectiveness of NeuroGeo.


About the classification results: Thank you for this suggestion. We have provided the numerical results and uncertainty definitions of the Fig. 3 (See Tab. A.1, Page 15). One possible explanation for the performance degradation of NeuroGeo-enhanced DINOv2 on fine-grained classification tasks is that its original representational geometry differs substantially from that of the human brain [1], requiring more extensive reshaping of its representation space to enhance high-level semantic organization, which may compromise its original fine-grained discriminative capabilities. Incorporating a multi-task learning loss that balances neural geometry alignment with the preservation of the model's original capabilities may help mitigate this limitation. We have added the above discussion to the revised manuscript (See Tab. A.1, Page 15).

[1] Human-like cognitive generalization for large models via mental representation-guided supervision, Nature Communications, 2026.


Decoding gain and consistency verification: Thank you for this valuable comment. we have included a comparison with the embedding-alignment baseline in Tab. 2, where both methods use the same decoder, captions, data, and evaluation. As shown in Tab. 2, NeuroGeo consistently outperforms the embedding-alignment baseline across all evaluation metrics. For example, it achieves relative improvements of 27.47% in BLEU-4, 5.23% in METEOR, and 9.45% in SPICE. To further assess the robustness of these improvements, we conduct additional experiments using different training subjects and random seeds, under the same experimental settings. Results show that the decoding gains are consistent when using different training subjects and random seeds (See Tab. A.2, Page 15).


We have revised the manuscript in accordance with your comments, and all changes have been highlighted in the revised manuscript. Hopefully these will address most of your concerns, and can be taken into consideration when deciding the final score of the paper. Many thanks.

Reviewer_zJ3gOpenReview ↗

Reply to author's response

Thank you for the response. The updated manuscript addresses most of my concerns, but the methodological novelty and improvements over recent work (Chen et. al., 2026) are relatively modest (as apparent from Tab. A.1). Therefore, I maintain my original scores.

Comments on the paper

1 comment
AuthorsOpenReview ↗

A brief overview of the major revisions

We thank the reviewers for their careful reading of our manuscript and for their constructive suggestions. Below, we provide a brief overview of the major revisions, including newly added results and substantial improvements to the presentation of the work.


  • We have conducted two additional evaluations on unseen concepts to provide independent neural validation of the learned manifold structure (See Figure 6, Page 9).
  • We have conducted additional controlled experiments comparing fixed, shared, and subject-specific anchors (See Tab. A.3, Page 15).
  • We have added a category-only control and an image-only/random-brain control experiments (See Tab. A.4, Page 15).
  • We have conducted additional experiments using different training subjects and random seeds (See Tab. A.2, Page 15).
  • We have provided the numerical results, uncertainty definitions and a discussion of the Fig. 3 (See Tab. A.1, Page 15).
  • Details of training data construction have been added to the Appendix (See Page 15).
  • We have revised the relevant descriptions of the training procedure to clarify the implementation details (See Page 5).

We have revised the manuscript in accordance with all the reviewers' comments, and all changes have been highlighted in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.