acceptodds
Under review as a conference paper at ICLR 2027

MindStrata: Learning Robust Multi-Subject Cortical Visual Representations via Stratified Layout-Semantics Decoding

Abstract

Decoding cortical visual representations that generalize across human subjects remains challenging due to (i) limited modeling of compositional semantics and structured layout in multi-object scenes, (ii) brain functional differences across subjects, and (iii) evaluation protocols dominated by feature-based scores that are often misaligned with human perception. To address these challenges, we propose __MindStrata__, a neuro-cognitively inspired framework for multi-subject brain-to-scene decoding via stratified modeling. __MindStrata__ maps fMRI into structured brain-scene representations using two complementary branches: a ventral-like branch decodes multi-level semantics (objectinstancesscene) in a CLIP-aligned space, while a dorsal-like branch decodes multi-scale spatial organization (targetsurroundingglobal) within latent layout space. A Brain-Scene Routing Controller integrates these stratified semantic and spatial cues for structure-guided reconstruction. To mitigate multi-subject variability, we further introduce Consistency-based Functional Alignment, combining subject-specific normalization with a relational consistency loss. We also propose the __Semantic Faithfulness Evaluation Protocol__ that evaluates description-, category-, count-, and position-level correctness and aligns more closely with human preferences Experiments demonstrate fine-grained layout and semantic decoding across single-subject, multi-subject, and data-limited settings, reveal ROI-specific spatial localization, and show robust reconstruction performance under progressively reduced fMRI spatial resolution. Code will be publicly released.

Primary area:
applications to neuroscience & cognitive science
Keywords:
AI for science, brain decoding, computational neuroscience, medical signal processing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qdf6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces MindStrata, a method for decoding fMRI activity evoked by seen images into visual representations at multiple levels, which guide image reconstruction. It uses separate branches to predict semantic content and spatial layout, then combines their outputs for reconstruction. The model integrates information across subjects and representation levels. Alongside the model, the authors introduce an evaluation protocol that checks whether reconstructions preserve object categories, counts, and positions, as well as scene descriptions. They evaluate the approach on NSD using both existing and proposed metrics and compare it with prior methods.

Critical strengths

Visual decoding from brain activity is an important and interesting task. Separately modeling different levels of semantic content and spatial layout is a potentially useful direction for improving reconstruction of complex scenes.

Critical weaknesses
  1. Poor clarity and presentation make the motivation and method difficult to assess.
  • The introduction does not clearly establish the specific limitations of existing methods or explain why the proposed representation levels are needed. It relies heavily on Figure 2, but the brief explanations of its panels do not provide enough context to understand the motivation or contribution.

  • Figures 2 and 3 are crowded, with small text and insufficient explanations in their captions. Figure 3 introduces many technical details at once, without a clear separation between the overall pipeline and individual components. I found it difficult to understand these figures even when reading the accompanying text.

  • The method section lacks an accessible overview and intuitive explanations before introducing the technical details. Many components are presented without sufficient context about existing approaches, their limitations, or why the proposed additions are necessary. Frequent references to the appendix further interrupt the explanation. A simpler overview figure would substantially help.

  • The discussion of prior multi-subject integration methods is insufficient, and the distinction from recent decoding approaches is not clearly established. The neuroscience-inspired motivations are also stated at a high level without clearly explaining how they justify the specific design choices.

Overall, the paper’s poor writing, together with the specific issues described above, substantially limits ability to understand the paper contributions, method and results.


  1. The contribution and empirical validation are insufficiently established.
  • Incomplete baseline comparisons. The main comparison omits relevant recent methods, including Brain-IT, MindTuner, and NeuroVLA. As far as I can tell from the results reported in their papers, these methods outperform MindStrata on most/all existing metrics. Brain-IT and MindTuner are cited, but their omission from the main evaluation is not adequately explained. These methods are also not evaluated on the proposed metrics, making it difficult to establish the claimed improvement over existing approaches.

  • Limited analysis of what contributes to the results. Although the paper includes multiple metrics and ablations, it does not provide a sufficiently clear explanation of which components address which limitations, why they help, or how their effects relate to the claimed contribution. The results often read as a collection of tables and figures rather than a coherent analysis.

  • Unclear model-selection procedure. The framework combines many components and design choices, but the paper does not clearly describe an independent validation procedure for selecting them. Appendix D.2 reports a hyperparameter sweep without identifying the evaluation set used for selection. It therefore remains unclear whether the final configuration was selected independently of the reported NSD test results. Evaluation on another dataset would also strengthen the generalization claim.

  • Unclear organization and interpretation of experiments. Reconstruction results, ROI analyses, and interpretability experiments are mixed together without clearly establishing their respective purposes. The results section begins with ROI-specific analysis before establishing the main reconstruction findings, and the explanations often do not make the experimental question or takeaway clear.

Together, these issues leave the necessity of the proposed components, the improvement over strong baselines, and the reliability of the reported gains insufficiently established.


Overall Assessment. I consider this paper well below the acceptance threshold in its current form. The motivation, clarity, presentation, comparison with prior work, and validation require substantial improvement. Addressing these concerns would require more than additional explanations during rebuttal; the paper needs a clearer account of its methodological contribution and stronger evidence supporting it.

Questions for authors
  1. Motivation and contribution. What specific failure of existing decoding methods requires the proposed representation levels? What evidence shows that these levels address that failure, and what is the main methodological contribution beyond combining existing techniques?

  2. Baseline comparisons. Why are Brain-IT, MindTuner, and NeuroVLA omitted from the main evaluation?

  3. Model selection. Which architectural alternatives and hyperparameter settings were evaluated? What validation data were used to select them, including the consistency-loss weight in Appendix D.2? Were all choices fixed before evaluating on the reported test set?

  4. Limitations and failure cases. Beyond the brief discussion of pretrained supervision, what are the main failure modes of MindStrata? In which scenes or settings does the proposed hierarchy fail to improve reconstruction?

  5. Characterization of prior methods. The method section states that previous decoding methods typically rely on pretrained ControlNet. This does not describe many prominent decoding approaches. Which class of methods is this statement intended to cover, and how does the proposed controller differ from methods outside that class?

10 comments

Dear authors,

I have not yet received a response to my review. I will be unavailable starting in two hours until the end of the discussion period. If you would like an opportunity for further discussion, please post your response within the next hour. If it arrives later, I will still read it carefully, but only after the discussion period ends.

AuthorsOpenReview ↗

Author Response to Reviewer qdf6 Part 1

Thank you for your follow-up, and we apologize for the delay. We are carefully addressing all of your questions, and some additional experiments are still in progress. In the meantime, we would like to share the responses and results that are ready, and we will provide the remaining updates as soon as possible. Thank you for your patience and understanding.

W1.1 Clarification on introduction

Our goal is to recover objects and their arrangement in multi-object scenes from fMRI across subjects. In Fig2(a), existing decoders recover the outdoor setting yet omit the vehicle beside the person. Fig2(b) shows that even with features from the reference image, scene-level guidance can yield the wrong object count or location. These observations motivate the three representation levels introduced in §1 (“Our solutions”). The local feature describes the target, the regional feature captures it with nearby objects, and the global feature supplies scene context.

The table below links each panel to §1 and explains how it motivates a design choice or evaluates reconstruction.

PanelWhat Figure 2 shows and why it mattersLocation in the submitted §1 Introduction
(a) What scene-level decoding missesIn the first row, MindEye2 and MindEye recover an outdoor gathering but omit the vehicle beside the person. In the second, a kitchen remains recognizable despite the altered counter layout. These examples motivate preserving the target, its relations to nearby objects and the overall layout at distinct levels.The opening paragraph cites (a) at line 51. The next paragraph distinguishes individual objects, their contextual relations and the whole scene.
(b) Observation 1 · GenerationEven with reference-image features, image-only and image–text guidance both miscount the three oranges. With the caption, the missing skier reappears in the wrong place. This observation motivates giving the target and its regional context their own spatial guidance while retaining a global feature for the scene.“Key observations” (i), line 92. “Our solutions” (lines 100–107) introduces the three levels and the controller.
(c) Observation 2 · AlignmentIn the UMAP plot, fMRI features vary across subjects, and their distribution differs from that of the image features. This observation motivates consistency-based functional alignment (CFA), which adapts the shared encoder to each subject and aligns relationships among examples across modalities.“Key observations” (ii), line 93. Lines 108–111 introduce CFA.
(d) Observation 3 · EvaluationAmong 1,000 MindEye outputs, a reconstruction missing a zebra ranks 36th under conventional metrics, while one preserving the bird category ranks 810th. This observation motivates the Semantic Faithfulness Evaluation Protocol (SFEP), which measures category, count and position alongside scene description.“Key observations” (iii), line 96. Lines 112–113 introduce the four SFEP measures.
(e) What the comparison revealsThe plots compare methods under single-subject, multi-subject and data-limited decoding. For example, in the refined multi-subject SDXL setting, MindEye2 scores higher on pixel correlation, while MindStrata scores higher on object position. Reporting both shows which aspect of the reconstruction improves.The experiment overview cites (e) at line 116. Tables 1 and 7 give the numerical results.

We appreciate your feedback, and we revised the wording below:

§1, opening paragraph (last sentence)

Existing decoders can recover the overall setting of a complex scene while omitting individual objects or changing their spatial arrangement (Fig. 2a).

§1, “Our solutions” (second sentence)

The count and placement errors under global guidance (Fig. 2b) motivate two spatially guided representations, one for the target and one for the target together with its neighbors, alongside a global representation of the scene.

Revised caption:

Figure 2. Motivation for stratified visual decoding and evaluation of MindStrata. (a) Reconstructions can preserve the outdoor setting while omitting the vehicle, or retain the appearance of a kitchen while changing its layout. (b) Even with reference-image features, global guidance miscounts the oranges or misplaces the skier, motivating separate features for the target, the target in context and the whole scene, with spatial guidance for the first two. (c) The UMAP projection illustrates differences across subjects and between fMRI and image features, motivating consistency-based functional alignment. (d) Conventional metrics rank a reconstruction missing a zebra near the top of 1,000 MindEye outputs, motivating the Semantic Faithfulness Evaluation Protocol (SFEP). (e) Conventional metrics and SFEP compare reconstruction performance in single-subject, multi-subject and data-limited settings.

AuthorsOpenReview ↗

Author Response to Reviewer qdf6 Part 2

W1.2 Readability of Figures 2 and 3

Thank you for pointing out where the figures were difficult to follow.

Figure 2. The revised caption explains what each panel shows and why it is included. In W1.1, we also identify the passage in §1 that discusses each panel. We will enlarge the panel labels and axis text to improve legibility at the page size.

Figure 3. We redesigned Figure 3 and rewrote its caption so that readers can follow the reconstruction pipeline before examining the training details. The revised figure in the submission PDF can be read as follows.

  • Overall pipeline. The top strip introduces the two branches and three representation levels. From the fMRI input, follow the blue layout branch (b) and the purple semantic branch (c) into the reconstruction panel (e).
  • Training details. Panel (a) shows how the training images supply targets for the two branches. Panel (d) explains alignment across subjects and modalities. Each training loss is labeled beside the component it supervises.
  • How the branches interact. The caption explains how to read the three paths into (e). The global layout initializes generation, the global semantic feature conditions the diffusion model, and the green controller applies the local and regional features at locations specified by their predicted masks.

W1.3 Clarification on Method section

  1. Thank you for this suggestion. We have added a short overview immediately after the heading of §3 (“Method”) and before §3.1 (“Decoupling multi-granularity layouts and semantics”). It walks the reader through the fMRI-to-image pipeline in the revised Figure 3 before §3.1 introduces the training targets. We reproduce the paragraph here.

    Method overview (Figure 3). MindStrata reconstructs a viewed image from a participant’s fMRI response. During training, each image provides targets at three levels, illustrated in (a) by one baseball player, the group of players, and the full scene. Two branches decode the same fMRI response in parallel. The layout branch (b) predicts a coarse scene layout and two masks locating the target and its surroundings. The semantic branch (c) predicts visual features for the target, the target in context, and the whole scene. To share the semantic encoder across participants, we use subject-specific normalization and train it to preserve similarities among examples in the image and text feature spaces (d). At inference, the coarse layout initializes diffusion and the global semantic feature guides the whole image (e). The routing controller uses the target mask to apply the local feature and the surrounding mask to apply the regional feature as the image is generated.

  2. Why these components are needed. Prior work such as DecoFuse already separates semantic and spatial information (§2.1). A further question is where each decoded feature should guide generation. In Figure 2(b), adding a caption recovers the skier but places the skier in the wrong location. This motivates using separate fMRI-predicted masks to place target and contextual features during generation (§3.5). In Figure 2(c), fMRI features vary across participants, and their distribution differs from that of image features. This motivates adapting normalization to each participant and aligning pairwise similarities across modalities during training (§3.4).

  3. Reading the method without turning to the appendix. As described in W1.2, the redesigned Figure 3 lets readers follow the two branches (b, c) directly into reconstruction (e). The overview above explains what each branch predicts and how its outputs are used. Readers can therefore follow inference in the main text and consult Appendices B.1–B.3 for implementation details such as mask construction and feature projection.

Reviewer_ovirOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors propose a new decoding architecture for image brain decoding from fMRI. As I understand it, their objective is to make visual brain decoding more faithful, especially when decoded scenes comprise multiple objects and/or a complex layout. The architecture has a ventral and a dorsal arm, and diffusion is guided by a Brain-Scene Routing Controller which combines the two signals. It works cross-subject via subject-specific layer normalization layers. They also propose a new evaluation protocol (SFEP) and evaluate it with a human preference study to show that SFEP aligns best with human metrics with respect to typical metrics. Their model is better than MindEye2, especially on SFEP metrics, and it handles much better the degradation of input fMRI data.

Critical strengths

I agree that a more sound evaluation of visual brain decoding results is needed and long overdue in the community. I really appreciate the analysis on the misalignment between semantic faithfulness and conventional metrics, as well as the authors' proposal of SFEP. For me this is the main contribution of the work and the main point for acceptance.

Critical weaknesses

The authors take care to reduce model-level circularity by using evaluator backbones outside the families used to build the training supervision, and acknowledge residual influence in the discussion. However, I still have a concern on circularity, which comes naturally when a work is proposing both a new metric and a new model, together. Mindstrata is explicitly supervised on caption-, entity- and grounding-derived targets, and SFEP measures precisely these properties, which the baselines were never trained to capture. Additional independent evidence would make the claims stroger. For example showing direct human preference rates over methods from the existing user study.

Questions for authors
  1. Could the authors please make the overview figure, Figure 3, less complex? As much as it is informative, it can be visually overwhelming. Removing some repetitions, restricting the color palette, or separating this figure in two separate figures would help.
  2. Would the authors consider the addition of direct human preference rates as part of the human evaluation results?
4 comments
AuthorsOpenReview ↗

Author Response to Reviewer ovir Part 1/2

W1. Potential circularity between MindStrata and SFEP

Thank you for highlighting the importance of independent evidence for evaluating both SFEP and MindStrata.

1. Training and evaluation targeting the same properties is standard practice.

For example, a classifier learns from category labels and is evaluated by classification accuracy. Likewise, MindStrata learns to preserve object identity, count, and location, and SFEP evaluates whether reconstructions preserve these task-relevant properties. This alignment between training and evaluation does not, by itself, constitute circular reasoning.

2. External human evidence for both the metric and the model.

A circular justification would be to argue that “SFEP is reliable because it gives MindStrata high scores” and that “MindStrata is effective because it receives high SFEP scores,” using those scores as the sole evidence for both claims. The key empirical question is whether SFEP’s assessments and MindStrata’s reconstruction gains are also supported by human judgments. We assess both contributions against this external reference.

  • For SFEP, we measure whether its scores agree with human rankings (Figure 5 and Appendix C.4). The additional analysis in point 3 extends this validation to comparisons among other reconstruction methods, excluding all comparisons involving MindStrata.
  • For MindStrata, the additional analysis in point 4 reports direct human preference rates against each baseline, computed from anonymized rankings without using any SFEP scores.

Separately, using GIT and Grounded-SAM to construct training supervision and Emu3 and RT-DETR for evaluation reduces the risk of shared model-specific biases (§3.1 and §4.1).

3. SFEP validation on other reconstruction methods.

We reanalyzed the existing human rankings for comparisons among MindEye, MindBridge, and NeuroPictor, excluding all comparisons involving MindStrata. Conventional references are included to contextualize the agreement rates.

SFEP measureSFEP–human agreement (%)Reference–human agreement (%)
Category68.61EffNet-B, 64.44
Count67.57EffNet-B, 64.44
Description60.94EffNet-B, 63.57
Position63.97AlexNet(5), 60.84

Category, Count, and Description are evaluated against human semantic rankings, and Position against perceptual rankings. Each reference had the highest conventional-metric agreement for its criterion in Figure 5.

Within each row, SFEP and its reference are evaluated on the same pairs. Category-score ties are excluded from the Category analysis but are not used to filter the Description analysis, resulting in different reference agreement rates.

  • Agreement extends to other reconstruction methods. Across the four measures, observed agreement with human rankings ranges from 60.94% to 68.61% on the evaluated baseline pairs. These results provide a human-based assessment of SFEP on reconstructions from methods other than MindStrata.
  • The evidence for SFEP is external to MindStrata’s performance. Here, people’s rankings of other methods provide the evaluation reference. SFEP is assessed through its correspondence with those judgments, independently of the scores or comparative ranking of MindStrata.

4. Direct human preferences, as requested.

Participants preferred MindStrata over MindEye and MindBridge under both semantic and perceptual similarity, with all four reported 95% confidence intervals above 50% (see details in Q2). Preferences against NeuroPictor were approximately 57%, with intervals including or touching 50%. These comparisons are calculated directly from anonymized human rankings and provide evidence for MindStrata’s reconstruction gains over MindEye and MindBridge independently of SFEP scores.

AuthorsOpenReview ↗

Author Response to Reviewer ovir Part 2/2

Q1. Simplifying Figure 3

Thank you for this suggestion. We carefully considered your comments and devoted substantial time and effort to redesigning the layout of Figure 3. Please see Figure 3 and its updated caption in the revised submission PDF. Our aim was to make the fMRI-to-image reconstruction pipeline easier to follow while keeping the key methodological components identifiable.

  • Present the design logic first. The top strip now presents two complementary design principles: brain-inspired dual-stream decoding, expressed through semantic-guided localization and structure-guided inference, and multi-granularity scene decomposition into global, regional, and local representations.
  • Organize the overview by function. Clearly titled panels distinguish stratified training targets (a), layout decoding (b), semantic decoding (c), Consistency-based Functional Alignment (d), and reconstruction (e). This separates the roles of the components within a single overview.
  • Make the reconstruction paths explicit. The revised connections distinguish global-layout initialization, global semantic conditioning through pretrained cross-attention, and fine-grained guidance through the routing controller. They also show how regional and local semantic conditions are combined with spatial masks before the Stratified Fuser provides control to the diffusion backbone.
  • Use consistent visual cues and supervision labels. We use blue for the layout branch, purple for the semantic branch, and green for controller-related guidance. The displayed training losses are labeled near the predictions or components they supervise, while the revised caption explains the roles of training targets and inference.

Figure 3: Overview of MindStrata. The design combines brain-inspired dual-stream decoding with multi-granularity scene decomposition.

  • (a) Training images provide semantic and spatial targets at global, regional, and local granularities.
  • (b) A dorsal-like stratified decoder learns layout latents via multi-scale value alignment and global-scale relational alignment.
  • (c) A ventral-like stratified decoder progressively integrates local details and context into scene semantics. Verbal-visual alignment provides complementary semantic supervision, while diffusion priors predict CLIP image embeddings to condition reconstruction.
  • (d) Consistency-based Functional Alignment adapts a shared semantic encoder to individual subjects to mitigate multi-subject heterogeneity, and aligns inter-sample relationships between fMRI-derived and CLIP representations to reduce cross-modal discrepancies.
  • (e) The Brain-Scene Routing Controller links fine-grained semantics to spatial regions for semantic-guided localization and uses decoded layouts to constrain spatial interactions for structure-guided inference. The Stratified Fuser integrates these features to guide diffusion reconstruction.

Q2. Direct human preference rates

Yes. We report direct human preference rates below, computed from the existing study in Appendix C.4. Each of the 51 participants ranked four anonymized reconstructions for each of 50 NSD reference images, separately for semantic and perceptual similarity. Semantic judgments considered object categories, counts, and scene description; perceptual judgments considered color, structure, spatial layout, and overall appearance.

Direct human preference for MindStrata. For each baseline and criterion, n/N counts how often participants ranked MindStrata above that baseline (n) out of all valid comparisons (N). Each row contains 2,550 comparisons (51 participants × 50 images). A preference rate above 50% favors MindStrata.

Ranking criterionBaselineMindStrata ranked higher (n/N)Preference (%)95% CI (%)
Semantic similarityMindEye1676/255065.73[57.14, 73.80]
Semantic similarityMindBridge1976/255077.49[70.12, 84.27]
Semantic similarityNeuroPictor1444/255056.63[49.06, 64.12]
Perceptual similarityMindEye1680/255065.88[59.18, 72.39]
Perceptual similarityMindBridge2060/255080.78[73.92, 86.82]
Perceptual similarityNeuroPictor1448/255056.78[50.00, 63.33]
  • Against MindEye and MindBridge. MindStrata was preferred under both criteria, with all four reported 95% confidence intervals above 50%.
  • Against NeuroPictor. Observed preference rates were approximately 57% under both criteria, with confidence intervals including or reaching 50% at the reported precision.

These preferences were computed directly from anonymized human rankings, without using SFEP scores.


Thank you again for your support and positive assessment of the paper’s contributions and soundness. We hope that the additional evidence, clarifications, and revisions address your remaining concerns and further strengthen your confidence in our work.

I thank the authors for their efforts in this rebuttal. I am satisfied by the answers provided. I checked the revised manuscript and I am also satisfied by the changes they made.

I will maintain my rating.

Thanks again.

Thank you for reviewing our responses and the revised manuscript. Your constructive feedback has helped us improve the clarity and presentation of our work.

We appreciate your continued support and thank you for reaffirming your recommendation of clear acceptance.

Reviewer_2vHgOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a new method to decode images from a passive fMRI viewing task. The method accomplishes this via a global, surrounding, and target object supervision, combined further with image & text CLIP embedding supervision.

This multi-level supervision is combined further with a RSA-like loss (representational similarity alignment) in eq 5 and attention biasing.

When evaluated on their SFEP metric, their method obtains better results than comparable methods.

Note I consider SFEP the primary contribution of this work, not the decoding.

Critical strengths
  1. The SFEP evaluation is generally quite a bit more detailed than traditional measurements which rely on feature similarity or low-level image metrics

  2. The human study is very informative, I think it is a good result to have. The authors also include significance tests, which is quite rare in a conference submission.

  3. The figure quality I think is quite good, they are generally clear enough to easily understand the point the authors are trying to make.

Critical weaknesses
  1. It is unclear to me why decoding images from fMRI is a scientifically interesting task. Especially when combined with a powerful inductive bias such as a diffusion model, it is difficult to characterize the relative contributions of the brain signal and the diffusion model. In this setup, I consider the "better image decoding" to be an un-interesting task from a scientific perspective. It tells us nothing new about the brain, and it tells us nothing scientifically about the decoder. Rather it is an exercise in engineering towards an unknown purpose. The system is extremely complex, but what do we learn now scientifically?

  2. The authors are missing quite a bit of classical related work, including the work for representational similarity analysis and the early work (around 2010) for stimulus decoding from fMRI.

Questions for authors
  1. Line 104, there is a double dash before the "scene representation"
  2. Can you clarify if you are anatomically aligning the volumes across subjects? Are you using volume alignment (MNI) or surface alignment (fsaverage)? In general there is a lot of missing details about the fMRI pre-processing. It is not even clear how the authors are feeding in volumes with a different number of voxels into the same network.
  3. For Figure 5, could you highlight the specific metrics that are SFEP metrics?
  4. For Figure 7, could you clarify how the voxel-wise blurring is done? Is this done in cortical space using the surface connectivity? Or did you just perform a blur in volume space?
8 comments
AuthorsOpenReview ↗

Author Response to Reviewer 2vHg Part 1/3

W1. Scientific focus of MindStrata

Thank you for raising this question. We investigate which fine-grained semantic and spatial properties can be recovered from fMRI, and how the representation and generative decoder affect their recovery.

Image decoding from fMRI generally involves two stages. During alignment, an encoder maps fMRI voxels to features aligned with target image or caption representations from pretrained encoders. During generation, these fMRI-derived features condition diffusion denoising.

Quantitive retrieval evaluation of the alignment stage. We conduct four retrieval experiments using the aligned representations, without passing them through any diffusion model. We report top-1 retrieval accuracy for all four directions below.

MethodViTMulti-subjectfMRI-to-imageImage-to-fMRIfMRI-to-textText-to-fMRI
MindEyeViT-L/14No97.01%94.77%5.12%4.43%
MindStrataViT-L/14No96.88%95.36%37.26%32.81%
MindStrataViT-L/14Yes97.94%96.17%36.94%31.69%
MindEye2ViT-bigG/14No99.93%99.71%4.62%3.89%
MindEye2ViT-bigG/14Yes99.56%99.42%4.38%3.61%
MindStrataViT-bigG/14No99.87%99.68%30.48%28.23%
MindStrataViT-bigG/14Yes99.99%99.97%30.93%27.74%

Under matched backbone and subject settings, the large text-retrieval gains indicate stronger semantic alignment, consistent with the additional text supervision. Image retrieval remains comparable or improves; in the multi-subject ViT-bigG/14 setting, MindStrata outperforms MindEye2 in all four directions. These gains are already present in the learned fMRI representations before generation and cannot be explained solely by better diffusion conditioning or stronger image backbones.

Limitation identified of the generation stage. As shown in Figure 2(b), even with representations extracted directly from the ground-truth image and additional caption guidance, Versatile Diffusion (VD) can omit or misplace objects and produce incorrect counts. These errors persist when fMRI alignment is bypassed, showing that global conditioning can be insufficient to preserve fine-grained scene content and motivating explicit semantic and spatial constraints.

MindStrata implements these constraints through hierarchical layout–semantics modeling, aligning fMRI with image/text features at global, regional, and local levels and with spatial-mask latents. Both semantic and spatial conditions are predicted from fMRI at inference. Two further experiments evaluate the spatial readout and resulting reconstructions:

  • Direct spatial layout decoding from fMRI. The layout branch predicts mask latents from fMRI and reconstructs masks with a pretrained VAE decoder, without image diffusion denoising (Section 3.2). The quantitative results in Appendix D.3 (original Table 6), which will be moved to Section 4.2, evaluate these predicted masks against ground-truth masks: single-subject target segmentation mIoU reaches 35.53% for Dorsal and 37.50% for Nsdgeneral, compared with 6.98% for the random-input baseline; target bounding-box mIoU reaches 40.62% and 35.48%, respectively. This provides an explicit readout of object location and extent from fMRI, independent of the final generated images.

  • Semantic and spatial faithfulness in generated images. Under the same VD backbone, adding the local and regional streams improves SFEP Category from 0.462 to 0.489, Count from 0.418 to 0.435, and Position from 0.201 to 0.236 (original Table 2). These gains show that finer fMRI-derived conditions improve the recovery of object identity, number, and location.

Fine-grained brain concept localization. We further explore decoding as a tool for studying brain concept representations in Appendix D.9. MindSimulator [1] synthesizes fMRI responses to visual stimuli and uses them to localize concept-selective brain regions. We extend this direction by encoding segmented objects into synthetic fMRI and comparing reconstructions across MindStrata's streams. The examples in original Figure 20 show that the global stream tends to restore scene context, while the regional and local streams better preserve the isolated object, providing a model-based probe of contextual bias in concept decoding.

We will replace the ROI-ranking plot in Section 4.2 (original Figure 6) with the full quantitative layout-decoding table from Appendix D.3 (original Table 6). We will also revise Section 4.2 to clarify the comparisons across ROIs, mask types, target/surrounding regions, and subject settings.

AuthorsOpenReview ↗

Author Response to Reviewer 2vHg Part 2/3

W2. Related work discussion

Thank you for highlighting these omissions. We will expand the historical context.

Representational similarity analysis: RSA [2] compares representational geometry through pairwise similarities. Eq. (5) uses a related principle as a training objective, applying an L1 penalty between inner-product similarity matrices of fMRI-derived embeddings and image/text targets across subjects and semantic levels. We will acknowledge this connection in Section 3.4.

Early work discussion: Early stimulus decoding established complementary routes from fMRI to visual content: category decoding [3], identification of unseen natural images through voxel-wise receptive-field models [4], and reconstruction of binary patterns with multiscale local decoders [5]. Naselaris et al. [6] extended reconstruction to natural images within a Bayesian framework. Later methods decoded hierarchical DNN features and optimized images to match them [7], while recent diffusion-based methods map fMRI into learned latent and conditioning representations for high-resolution reconstruction [8]. We will describe this development along five dimensions:

  • Task complexity. Tasks have expanded from category classification and simple-pattern reconstruction [3][5] to detailed reconstruction of complex natural scenes [8], including multiple objects and their context.

  • Stimulus paradigm. Miyawaki et al. used 6-s random-pattern blocks [5]; NSD uses rapid event-related natural-scene presentations with a continuous-recognition task [9].

  • Data scale. Kay et al.'s dataset comprised 1,870 distinct images and two subjects [4]; NSD contains 213K fMRI trials across eight subjects, covering 70K distinct natural images [9].

  • Cross-subject learning. Early decoders were fitted separately per subject [4][5]; recent approaches learn shared representations across subjects and adapt them to new subjects [10].

  • Evaluation granularity. Early studies measured identification accuracy [4], pixel correlation [5], and semantic-category accuracy [6]; current evaluation also examines perceptual/feature similarity and finer semantic errors, including descriptions, object categories, and counts in SFEP.

We will incorporate this historical context into Sections 1.

Q1. Double dash in line 104

Thank you for catching this. We will change brain-–scene representation to brain–scene representation in line 104.

Q2. Anatomical alignment and preprocessing

We use fMRI beta volumes in each subject's native functional space (NSD's func1pt8mm, at 1.8-mm isotropic resolution). We do not apply additional anatomical registration across subjects: neither MNI volume alignment nor fsaverage surface alignment is used for the model inputs. Different voxel counts are accommodated by a separate learned linear input layer for each subject, which maps their voxel vector to 4,096 features before the residual MLP encoder.

For preprocessing, we start from the beta estimates released by NSD. The upstream NSD pipeline corrects slice timing, head motion within and across sessions, EPI distortion, and gradient nonlinearities, producing functional data in each subject's native space [9]. We use the betas_fithrf_GLMdenoise_RR single-trial estimates, incorporating voxel-specific HRF fitting, GLMdenoise, and ridge regularization, as described for GLMsingle [11]. Our subsequent processing applies each subject's nsdgeneral mask and extracts the selected voxels in a fixed order. Each voxel is then z-scored independently across the 750 trials of its session. The three responses to each training image are retained as separate samples; for each test image, we average the three normalized responses before passing them to the network. This yields 27,000 training samples and 1,000 test samples per subject, as summarized in Appendix A.2.

The masked input dimensions are 15,724, 14,278, 13,039, and 12,682 for subjects 1, 2, 5, and 7, respectively. Specifically, for subject ss, the input vector xs∈RVsx_s\in\mathbb{R}^{V_s} is mapped to hs=Wsxs+bs∈R4096h_s=W_sx_s+b_s\in\mathbb{R}^{4096}, with a separately learned Ws∈R4096×VsW_s\in\mathbb{R}^{4096\times V_s} and bs∈R4096b_s\in\mathbb{R}^{4096}. These common-dimensional features enter the residual MLP blocks in each semantic stream. Within each stream, the MLP linear weights are shared across subjects, while the LayerNorm scale and bias parameters remain subject-specific.

We will expand Appendix A.2 to specify the native functional space, preprocessing steps, voxel counts, and parameter-sharing scheme, and revise Section 3.3 to make the subject-specific input layer explicit.

AuthorsOpenReview ↗

Author Response to Reviewer 2vHg Part 3/3

Q3. SFEP metrics highlight

Thank you for the suggestion. In Figure 5, we will highlight the x-axis labels of the four SFEP metrics in dark red: Position in the left panel and Category, Count, and Description in the right panel.

Q4. Voxel-wise blurring

Thank you for requesting this clarification. Following the spatial-smoothing approach of Tang et al. [12], we perform the blurring in volume space by applying an isotropic 3D Gaussian filter independently to each single-trial beta volume in the subject's native functional space, with 1.8-mm isotropic voxels. Neighborhoods are defined on the three-dimensional voxel grid, without using cortical surface connectivity or averaging across trials.

Smoothing is applied to the full beta volume before extracting the nsdgeneral ROI, so neighboring voxels outside the ROI can contribute to values inside it. After ROI extraction, each voxel is z-scored within its session using the data at the corresponding smoothing level. Training repetitions remain separate samples, while the three normalized responses to each test image are averaged before decoding.

We use σ∈{0,1,2,3,4,5}\sigma\in\{0,1,2,3,4,5\} in voxel units, with σ=0\sigma=0 denoting no added smoothing, and use the same σ\sigma during training and testing. For σ=1,…,5\sigma=1,\ldots,5, the Gaussian kernels have FWHM values of approximately 4.2, 8.5, 12.7, 17.0, and 21.2 mm on our 1.8-mm voxel grid. These values describe the added smoothing kernels; the voxel grid and ROI voxel count remain unchanged, so they should not be interpreted as equivalent acquisition resolutions.

We will clarify the volumetric smoothing procedure and its source in the caption of original Figure 7 and Appendix D.6, including the order of smoothing and ROI extraction, the kernel units, and the matching training and testing conditions.

References

[1] Bao et al. MindSimulator: Exploring brain concept localization via synthetic fMRI. ICLR, 2025.

[2] Kriegeskorte et al. Representational similarity analysis—connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2008.

[3] Haxby et al. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science, 2001.

[4] Kay et al. Identifying natural images from human brain activity. Nature, 2008.

[5] Miyawaki et al. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron, 2008.

[6] Naselaris et al. Bayesian reconstruction of natural images from human brain activity. Neuron, 2009.

[7] Shen et al. Deep image reconstruction from human brain activity. PLOS Computational Biology, 2019.

[8] Takagi and Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. CVPR, 2023.

[9] Allen et al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 2022.

[10] Wang et al. MindBridge: A cross-subject brain decoding framework. CVPR, 2024.

[11] Prince et al. Improving the accuracy of single-trial fMRI response estimates using GLMsingle. eLife, 2022.

[12] Tang et al. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 2023.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.