acceptodds
Under review as a conference paper at ICLR 2027

Mind-BIRTH: Learning Hierarchical Canonical Cortical Representations for Zero-Shot Cross-Subject Visual Decoding

Abstract

Reconstructing perceived images from human fMRI offers a way to study neural representations and visual processing, but most existing approaches remain strongly subject-specific. Cross-subject decoding is particularly challenging because individuals differ in voxel coverage, spatial organization, and response distributions, making it difficult to learn representations that transfer reliably to unseen subjects. We propose Mind-BIRTH, a framework for learning hierarchical canonical cortical representations for cross-subject visual decoding. Mind-BIRTH organizes visual cortical responses into Early, Middle, and Late stages, converts heterogeneous ROI responses into coordinate-anchored tokens, and maps them into stage-wise canonical spaces using visual-semantic alignment. The learned representations are then decoded through hierarchy-matched diffusion conditioning, preserving the correspondence between cortical hierarchy and generator depth. This design supports zero-shot reconstruction of held-out subjects without target-subject training and lightweight few-shot personalization when limited target data are available. Experiments on NSD show that Mind-BIRTH achieves strong semantic reconstruction performance among cross-subject methods under both zero-shot and few-shot settings while remaining competitive in low-level fidelity. Controlled ablations further validate the importance of stage-wise cortical representations, canonical alignment, and hierarchy-matched routing. These results suggest that learning transferable, hierarchy-aware cortical representations provides an effective inductive bias for cross-subject visual decoding.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Visual Decoding, Zero-shot Transfer

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FsdCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper converts early, mid, and late visual brain region responses into tags with two-dimensional location encodings, enabling staged visual semantic supervised learning of shared representations. Then, the three-stage conditions are injected into different depths of the diffusion model to achieve zero-sample cross-subject reconstruction and few-sample adaptation.

Critical strengths
  1. The protocol for simultaneously isolating the subject from the evaluation stimulus is clearly defined. Section 4.1 excludes shared1000 evaluation images and their corresponding brain responses from all source subjects, and few-sample fitting also excludes the target test image.

  2. The magnitude of semantic improvement from zero-sample testing warrants further validation.

  3. A relatively complete preliminary ablation is provided. This includes stage combination, a Flat model with parameter matching, positional encoding, alignment targets, and routing perturbations, and reports its own subject-by-subject results and cross-subject standard deviation.

Critical weaknesses
  1. The Flat model in Section 4.3 does indeed match the input stage, module, and number of trainable parameters, and is trained independently. However, its approach involves first resizing the three-stage conditions and then directly averaging them, before feeding the same fusion condition into the entire depth; the main model retains three distinguishable information streams. Converting three signals with stage identities into a single mean is a many-to-one information mixing. Same parameters do not necessarily mean the same usable information. The increase in CLIP from 77.2% to 79.5% in Table 3 only demonstrates that "preserving branches and specifying routes" is superior to "averaging first and then broadcasting," but it cannot independently prove the correctness of the authors' chosen biological hierarchical correspondence. The truly crucial control should retain stage identities, canceling the preset hierarchy through splicing, independent projection, or learnable fully connected routing. The appendix probe map is also insufficient to compensate.

  2. Equations 5-7 pass the frozen CLIP global image features through a trainable stage projection head and cosine-align it with the brain label mean. Target projection and brain mapping are trained simultaneously, and the three stages use the same global CLIP feature, without explicit cross-subject correspondence constraints. Considering the cosine term alone, ensuring that the target projections and mean brain labels of all stimuli converge to the same vector yields minimal loss. Constraining only the label mean does not guarantee that local labels have the same meaning across different subjects. Removing alignment from Table 4 leads to performance degradation, demonstrating the effectiveness of correct visual supervision, but it fails to distinguish between cross-subject normalization and ordinary auxiliary semantic regression. "Canonical" is a title-level concept, requiring at least direct evidence of cross-subject canonical retrieval, representational correspondence, and information retention.

  3. The main text and Appendix A.1 perform maximum projection, smoothing, scaling, and pooling of brain region responses in each subject's native volume along the zz axis into scalars: 128 for early, 64 for mid-stage, and 32 for late. The coordinate anchor refers to the regular position of the final two-dimensional canvas, not the cortical surface, visual field, pRF, or common anatomical coordinates. The same two-dimensional index from different subjects need not have the same functional meaning; maximum projection along depth may also mix different cortical fragments. According to the mask-outside-zero re-maximization projection formula given in the appendix, as long as there are out-of-mask positions in a column, valid responses with all negative values ​​may still be covered by zeros. The performance degradation after removing position encoding or using random coordinates only proves the usefulness of fixed-position indexes, not that these indexes constitute cross-subject functional anchors.

  4. Table 1 lists the results for zero-sample, 250-sample, and full-fit tests. This method explicitly excludes shared1000-source training stimuli, which is an advantage; however, it is unclear whether this rule is equally applied to all comparison subjects. The inclusion of "250 images" does not automatically imply the same brain signal budget, as the number of trials and image selections may differ. Figure 3 compares this zero-sample method with the one-hour fit MindEye2 (the caption is disclosed), but this presentation cannot replace a quantitative comparison with a uniform budget.

  5. The appendix does indeed report SD1.5, frozen modules, training steps, DDIM steps, and guidance intensity. However, the system includes a frozen text encoder and is guided by a classifier-free approach, but it doesn't specify whether training and inference use empty text, fixed cues, or text from other sources, nor does it fully define what information was removed from the conditional and unconditional branches. There is no evidence that the authors used test images for description, so this missing information cannot be escalated to a leak accusation. However, textual conditions are crucial inputs determining generative capacity and information budget and must be explicitly stated. Other important gaps include the complete structure of conditional branches, optimization of learning rate and scheduling, few-shot adaptation steps, and which parameters are updated in full adaptation.

  6. The appendix reports the standard deviations of four participants, but cross-participant differences do not equate to training randomness or the selection variance of 250 adaptation images. The few-shot experiments primarily rely on a single non-zero budget LoRA design (K=250) to support parameter savings, but this alone cannot demonstrate practical personalized efficiency. The authors acknowledge that the evaluation was conducted only on four NSD participants with the same acquisition protocol. Under the strong claim of "transferable canonical cortical representations," the existing single-dataset results are insufficient to distinguish robust brain representation learning.

Questions for authors
  1. How does the method, which preserves the splicing and fusion of three-stage identities, learnable routes, and six fixed permutations, perform after independent retraining? Is the so-called correct hierarchical correspondence still stable and optimal?

  2. Are the representations of the same stimulus across subjects more similar, while still distinguishing different stimuli? Are ordinary CLIP-assisted regression or fixed target projection sufficient?

  3. How sensitive is the method to projection axes, volume orientation, negative response handling, and the number of labels? Are signed mean, dual-channel (positive and negative), three-dimensional, or surface representations more robust?

  4. Where are all textual conditions obtained, and how is CFG specifically implemented? Please provide the complete conditional network, optimization settings, adaptation configuration, and validation selection process.

  5. Do hierarchical control and few-sample results hold true on multiple fit subsets? What are the learning curves and actual adaptation costs for different KK values?

2 comments
AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer FsdC,

As the discussion deadline is approaching, we kindly remind that we have answered all your concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards,

The authors of paper 12590

Thank you for the additional experiments. They strengthen the evidence for the proposed design and partially address my concerns. However, the contribution still appears primarily to be an architectural combination of CLIP-supervised stage representations and fixed stage-to-depth conditioning. The distinctive contribution of “canonical alignment” beyond shared semantic supervision remains unclear. I therefore remain unconvinced that the methodological advance is sufficiently substantial and maintain my original rating.

Reviewer_a26qOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents Mind-BIRTH for zero-shot cross-subject fMRI-to-image reconstruction and few-shot personalization. The method divides visual ROIs into Early, Middle, and Late groups, converts their responses into coordinate-anchored tokens, aligns pooled stage representations to projected CLIP features, and injects stage-specific conditions into selected diffusion-model depths. Evaluation on four NSD target subjects reports improved semantic reconstruction relative to the listed zero-shot baselines, together with stage-wise, routing, and component ablations. The problem is important and the reported semantic improvements are promising. However, the contribution is insufficiently distinguished from closely related hierarchical decoders, the comparability of the main baseline results is unclear, and the experiments do not yet establish the claimed canonical alignment and hierarchy-specific mechanisms.

Critical strengths
  1. The paper aims to address a practically relevant limitation of subject-specific visual decoding and evaluates both zero-shot transfer and limited-data adaptation.

  2. The stated protocol excludes evaluation image identities and their associated responses from every source subject, an important safeguard against stimulus-level leakage.

Critical weaknesses
  1. The novelty relative to existing hierarchical decoding methods is not sufficiently established. VCFlow [1] already combines functional cortical grouping, hierarchical visual alignment, and cross-subject decoding. MindHier [2] investigates correspondence between brain representations, visual features, and generative scales, while Hi-DREAM [3] incorporates Early/Middle/Late ROIs and depth-matched diffusion conditioning. Given these closely related studies, the authors should clarify what is methodologically novel in the proposed approach, which limitations of prior methods it addresses, and provide controlled comparisons to demonstrate the value of the claimed innovations.

  2. The central claim of canonical alignment lacks direct evidence. The method uses shared MLP mappings to align the mean token representation of each stage with targets derived from the same global CLIP feature through separate projection heads. This provides visual supervision, but does not directly establish consistent correspondence across subjects or ensure that the three stages preserve distinct low-, mid-, and high-level information. The existing ablations that remove or shuffle the alignment targets primarily demonstrate the usefulness of meaningful visual supervision. Cross-subject agreement for the same stimulus, cross-subject retrieval, and analyses of stage-specific representations are needed to substantiate the claimed shared hierarchical space.

  3. The ablations do not sufficiently isolate the contribution of hierarchy matching itself. Although the Flat baseline matches the parameter count, it averages the three conditioning streams before supplying the fused condition to every depth. This simultaneously removes stage-to-depth correspondence and compresses distinct conditioning information. The full model's advantage may therefore partly reflect avoiding information loss from averaging. Moreover, the three stages differ in token count and injection resolution, confounding hierarchy matching with representational capacity and spatial resolution. A fusion control that preserves all three streams, together with controls for token count and a clear account of whether random and reversed routing variants are retrained, would provide stronger evidence for the proposed hierarchical correspondence.

Questions for authors

See Weaknesses.

1 comment

I thank the authors for their responses and additional experiments. My biggest concern remains the novelty: the contribution still appears incremental relative to existing hierarchical decoding methods. I therefore maintain my original rating.

Reviewer_vsMBOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a method for reconstructing images presented to people during fMRI from patterns of hemodynamic response in visual cortical areas. It tackles the challenge of building a decoder that generalizes to new human subjects. The key idea is to use the human visual cortical stages to condition levels in a diffusion decoder. They use three stages of the cortical hierarchy: early, middle, and late. The 7-Tesla 1.8 mm isotropic voxel fMRI patterns in 3D are converted to 2D patterns using a maximum projection along the z direction before tokenization. The diffusion decoder uses the individually mapped regions of interest to decode from new subjects without any stimulus-response pairs from those subjects (zero-shot) or after finetuning with a small number of stimulus-response pairs. The decoder might exploit some coarse regularities in response patterns that hold across subjects. For example, there may be coarse gradients of selectivity within each region that can be exploited, even if detailed voxel tuning is idiosyncratic to each subject. Reconstructions are evaluated with a conventional range of measures and are quite good in terms of semantic content.

Critical strengths

The paper introduces a novel method for zero-shot cross-subject visual-reconstruction decoding: reconstructing images viewed by people during fMRI without training on the subject whose brain activity is being used.

The approach uses prior information about the visual cortical hierarchy, separating early, middle, and late visual representations and using them to condition different depths of the diffusion generator.

Semantic reconstruction performance is good, improving over some recent cross-subject methods on semantic similarity metrics. Ablation studies examine the contributions of the three cortical stages, spatial information, representational alignment, and hierarchical routing.

Critical weaknesses

(1) The method exploits rough anatomical correspondence, but the projection into 2D is arbitrary and fails to respect the structure of the cortical sheet. This is an ad-hoc approach that fails to leverage cortex-based methods that have been used for decades in cognitive neuroscience. The model must be exploiting coarse-grained structure in neural activity patterns, rather than the full fine-grained voxel-pattern information.

Why perform a maximum projection along z before tokenization? The maximum projection may remove useful information. Why project along z rather than x or y? This seems poorly motivated. It would seem preferable to either tokenize the 3D activity directly or use an individual cortical sheet reconstruction and project orthogonally onto the cortical sheet.

(2) The “zero-shot” approach requires definition of ROIs for V1, V2, and other functional regions in the new subject. This requires fMRI data for the individual subject. A truly zero-shot approach would leverage only anatomical information.

(3) The paper fails to build on, engage, or even cite highly relevant prior work. Andrew Luo and colleages (Yu et al. 2025; Nan et al. CVPR 2026) introduced an in-context method for cross-subject-generalizing fMRI image reconstruction. Feng et al. (NeuroAdapter, ICLR 2026) use learned flexible attentional routing for fMRI image-reconstruction. Another example is MindHier's hierarchy-matched image generation.

(4) Low-level improvements are inconsistent, and there is no human evaluation.

(5) The example reconstructions look compelling, but the paper does not explain how the examples were selected or show a randomly sampled, representative set of reconstructions and failures.

Questions for authors

How does performance compare to recent cross-subject generalizing decoders of Andrew Luo’s group?

1 comment
AuthorsOpenReview ↗

Sincerely looking forward to the further discussions

Dear Reviewer vsMB,

As the discussion deadline is approaching, we kindly remind that we have answered all your concerns carefully with additional experiments and clarification.

Could you please check our response and consider improving the rating if we successfully addressed you concerns?

Any feedback will be greatly appreciated and we are glad to provide any additional clarifications that you may need to bring a fruitful discussion.

Warm regards,

The authors of paper 12590

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.