- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces MindStrata, a method for decoding fMRI activity evoked by seen images into visual representations at multiple levels, which guide image reconstruction. It uses separate branches to predict semantic content and spatial layout, then combines their outputs for reconstruction. The model integrates information across subjects and representation levels. Alongside the model, the authors introduce an evaluation protocol that checks whether reconstructions preserve object categories, counts, and positions, as well as scene descriptions. They evaluate the approach on NSD using both existing and proposed metrics and compare it with prior methods.
Critical strengths
Visual decoding from brain activity is an important and interesting task. Separately modeling different levels of semantic content and spatial layout is a potentially useful direction for improving reconstruction of complex scenes.
Critical weaknesses
- Poor clarity and presentation make the motivation and method difficult to assess.
-
The introduction does not clearly establish the specific limitations of existing methods or explain why the proposed representation levels are needed. It relies heavily on Figure 2, but the brief explanations of its panels do not provide enough context to understand the motivation or contribution.
-
Figures 2 and 3 are crowded, with small text and insufficient explanations in their captions. Figure 3 introduces many technical details at once, without a clear separation between the overall pipeline and individual components. I found it difficult to understand these figures even when reading the accompanying text.
-
The method section lacks an accessible overview and intuitive explanations before introducing the technical details. Many components are presented without sufficient context about existing approaches, their limitations, or why the proposed additions are necessary. Frequent references to the appendix further interrupt the explanation. A simpler overview figure would substantially help.
-
The discussion of prior multi-subject integration methods is insufficient, and the distinction from recent decoding approaches is not clearly established. The neuroscience-inspired motivations are also stated at a high level without clearly explaining how they justify the specific design choices.
Overall, the paper’s poor writing, together with the specific issues described above, substantially limits ability to understand the paper contributions, method and results.
- The contribution and empirical validation are insufficiently established.
-
Incomplete baseline comparisons. The main comparison omits relevant recent methods, including Brain-IT, MindTuner, and NeuroVLA. As far as I can tell from the results reported in their papers, these methods outperform MindStrata on most/all existing metrics. Brain-IT and MindTuner are cited, but their omission from the main evaluation is not adequately explained. These methods are also not evaluated on the proposed metrics, making it difficult to establish the claimed improvement over existing approaches.
-
Limited analysis of what contributes to the results. Although the paper includes multiple metrics and ablations, it does not provide a sufficiently clear explanation of which components address which limitations, why they help, or how their effects relate to the claimed contribution. The results often read as a collection of tables and figures rather than a coherent analysis.
-
Unclear model-selection procedure. The framework combines many components and design choices, but the paper does not clearly describe an independent validation procedure for selecting them. Appendix D.2 reports a hyperparameter sweep without identifying the evaluation set used for selection. It therefore remains unclear whether the final configuration was selected independently of the reported NSD test results. Evaluation on another dataset would also strengthen the generalization claim.
-
Unclear organization and interpretation of experiments. Reconstruction results, ROI analyses, and interpretability experiments are mixed together without clearly establishing their respective purposes. The results section begins with ROI-specific analysis before establishing the main reconstruction findings, and the explanations often do not make the experimental question or takeaway clear.
Together, these issues leave the necessity of the proposed components, the improvement over strong baselines, and the reliability of the reported gains insufficiently established.
Overall Assessment. I consider this paper well below the acceptance threshold in its current form. The motivation, clarity, presentation, comparison with prior work, and validation require substantial improvement. Addressing these concerns would require more than additional explanations during rebuttal; the paper needs a clearer account of its methodological contribution and stronger evidence supporting it.
Questions for authors
-
Motivation and contribution. What specific failure of existing decoding methods requires the proposed representation levels? What evidence shows that these levels address that failure, and what is the main methodological contribution beyond combining existing techniques?
-
Baseline comparisons. Why are Brain-IT, MindTuner, and NeuroVLA omitted from the main evaluation?
-
Model selection. Which architectural alternatives and hyperparameter settings were evaluated? What validation data were used to select them, including the consistency-loss weight in Appendix D.2? Were all choices fixed before evaluating on the reported test set?
-
Limitations and failure cases. Beyond the brief discussion of pretrained supervision, what are the main failure modes of MindStrata? In which scenes or settings does the proposed hierarchy fail to improve reconstruction?
-
Characterization of prior methods. The method section states that previous decoding methods typically rely on pretrained ControlNet. This does not describe many prominent decoding approaches. Which class of methods is this statement intended to cover, and how does the proposed controller differ from methods outside that class?
10 comments
Author Response to Reviewer qdf6 Part 1
Thank you for your follow-up, and we apologize for the delay. We are carefully addressing all of your questions, and some additional experiments are still in progress. In the meantime, we would like to share the responses and results that are ready, and we will provide the remaining updates as soon as possible. Thank you for your patience and understanding.
W1.1 Clarification on introduction
Our goal is to recover objects and their arrangement in multi-object scenes from fMRI across subjects. In Fig2(a), existing decoders recover the outdoor setting yet omit the vehicle beside the person. Fig2(b) shows that even with features from the reference image, scene-level guidance can yield the wrong object count or location. These observations motivate the three representation levels introduced in §1 (“Our solutions”). The local feature describes the target, the regional feature captures it with nearby objects, and the global feature supplies scene context.
The table below links each panel to §1 and explains how it motivates a design choice or evaluates reconstruction.
| Panel | What Figure 2 shows and why it matters | Location in the submitted §1 Introduction |
|---|---|---|
| (a) What scene-level decoding misses | In the first row, MindEye2 and MindEye recover an outdoor gathering but omit the vehicle beside the person. In the second, a kitchen remains recognizable despite the altered counter layout. These examples motivate preserving the target, its relations to nearby objects and the overall layout at distinct levels. | The opening paragraph cites (a) at line 51. The next paragraph distinguishes individual objects, their contextual relations and the whole scene. |
| (b) Observation 1 · Generation | Even with reference-image features, image-only and image–text guidance both miscount the three oranges. With the caption, the missing skier reappears in the wrong place. This observation motivates giving the target and its regional context their own spatial guidance while retaining a global feature for the scene. | “Key observations” (i), line 92. “Our solutions” (lines 100–107) introduces the three levels and the controller. |
| (c) Observation 2 · Alignment | In the UMAP plot, fMRI features vary across subjects, and their distribution differs from that of the image features. This observation motivates consistency-based functional alignment (CFA), which adapts the shared encoder to each subject and aligns relationships among examples across modalities. | “Key observations” (ii), line 93. Lines 108–111 introduce CFA. |
| (d) Observation 3 · Evaluation | Among 1,000 MindEye outputs, a reconstruction missing a zebra ranks 36th under conventional metrics, while one preserving the bird category ranks 810th. This observation motivates the Semantic Faithfulness Evaluation Protocol (SFEP), which measures category, count and position alongside scene description. | “Key observations” (iii), line 96. Lines 112–113 introduce the four SFEP measures. |
| (e) What the comparison reveals | The plots compare methods under single-subject, multi-subject and data-limited decoding. For example, in the refined multi-subject SDXL setting, MindEye2 scores higher on pixel correlation, while MindStrata scores higher on object position. Reporting both shows which aspect of the reconstruction improves. | The experiment overview cites (e) at line 116. Tables 1 and 7 give the numerical results. |
We appreciate your feedback, and we revised the wording below:
§1, opening paragraph (last sentence)
Existing decoders can recover the overall setting of a complex scene while omitting individual objects or changing their spatial arrangement (Fig. 2a).
§1, “Our solutions” (second sentence)
The count and placement errors under global guidance (Fig. 2b) motivate two spatially guided representations, one for the target and one for the target together with its neighbors, alongside a global representation of the scene.
Revised caption:
Figure 2. Motivation for stratified visual decoding and evaluation of MindStrata. (a) Reconstructions can preserve the outdoor setting while omitting the vehicle, or retain the appearance of a kitchen while changing its layout. (b) Even with reference-image features, global guidance miscounts the oranges or misplaces the skier, motivating separate features for the target, the target in context and the whole scene, with spatial guidance for the first two. (c) The UMAP projection illustrates differences across subjects and between fMRI and image features, motivating consistency-based functional alignment. (d) Conventional metrics rank a reconstruction missing a zebra near the top of 1,000 MindEye outputs, motivating the Semantic Faithfulness Evaluation Protocol (SFEP). (e) Conventional metrics and SFEP compare reconstruction performance in single-subject, multi-subject and data-limited settings.
Author Response to Reviewer qdf6 Part 2
W1.2 Readability of Figures 2 and 3
Thank you for pointing out where the figures were difficult to follow.
Figure 2. The revised caption explains what each panel shows and why it is included. In W1.1, we also identify the passage in §1 that discusses each panel. We will enlarge the panel labels and axis text to improve legibility at the page size.
Figure 3. We redesigned Figure 3 and rewrote its caption so that readers can follow the reconstruction pipeline before examining the training details. The revised figure in the submission PDF can be read as follows.
- Overall pipeline. The top strip introduces the two branches and three representation levels. From the fMRI input, follow the blue layout branch (b) and the purple semantic branch (c) into the reconstruction panel (e).
- Training details. Panel (a) shows how the training images supply targets for the two branches. Panel (d) explains alignment across subjects and modalities. Each training loss is labeled beside the component it supervises.
- How the branches interact. The caption explains how to read the three paths into (e). The global layout initializes generation, the global semantic feature conditions the diffusion model, and the green controller applies the local and regional features at locations specified by their predicted masks.
W1.3 Clarification on Method section
-
Thank you for this suggestion. We have added a short overview immediately after the heading of §3 (“Method”) and before §3.1 (“Decoupling multi-granularity layouts and semantics”). It walks the reader through the fMRI-to-image pipeline in the revised Figure 3 before §3.1 introduces the training targets. We reproduce the paragraph here.
Method overview (Figure 3). MindStrata reconstructs a viewed image from a participant’s fMRI response. During training, each image provides targets at three levels, illustrated in (a) by one baseball player, the group of players, and the full scene. Two branches decode the same fMRI response in parallel. The layout branch (b) predicts a coarse scene layout and two masks locating the target and its surroundings. The semantic branch (c) predicts visual features for the target, the target in context, and the whole scene. To share the semantic encoder across participants, we use subject-specific normalization and train it to preserve similarities among examples in the image and text feature spaces (d). At inference, the coarse layout initializes diffusion and the global semantic feature guides the whole image (e). The routing controller uses the target mask to apply the local feature and the surrounding mask to apply the regional feature as the image is generated.
-
Why these components are needed. Prior work such as DecoFuse already separates semantic and spatial information (§2.1). A further question is where each decoded feature should guide generation. In Figure 2(b), adding a caption recovers the skier but places the skier in the wrong location. This motivates using separate fMRI-predicted masks to place target and contextual features during generation (§3.5). In Figure 2(c), fMRI features vary across participants, and their distribution differs from that of image features. This motivates adapting normalization to each participant and aligning pairwise similarities across modalities during training (§3.4).
-
Reading the method without turning to the appendix. As described in W1.2, the redesigned Figure 3 lets readers follow the two branches (b, c) directly into reconstruction (e). The overview above explains what each branch predicts and how its outputs are used. Readers can therefore follow inference in the main text and consult Appendices B.1–B.3 for implementation details such as mask construction and feature projection.
Dear authors,
I have not yet received a response to my review. I will be unavailable starting in two hours until the end of the discussion period. If you would like an opportunity for further discussion, please post your response within the next hour. If it arrives later, I will still read it carefully, but only after the discussion period ends.