acceptodds
Under review as a conference paper at ICLR 2027

BrainSeg: A Framework for Probing Category-Specific Neural Representations through Dense Semantic Decoding

Abstract

The human visual system transforms sensory inputs into structured representations of semantic content and spatial organization. Recent deep-learning methods can reconstruct visually compelling images from fMRI responses, but it remains unclear whether the semantic content of these reconstructions is supported by measured brain responses or supplied by generative priors. We introduce BrainSeg, a framework that reformulates fMRI-based visual decoding as dense semantic prediction. Rather than reconstructing image appearance, BrainSeg directly predicts which semantic categories are present and where they occur. BrainSeg maps fMRI responses into complementary semantic queries and multiscale spatial features and progressively couples these two representations to produce a dense semantic map over 133 categories. Training targets are generated offline using a frozen MaskFormer, whereas inference relies on fMRI alone. We evaluate BrainSeg on held-out stimuli from the Natural Scenes Dataset using metrics of category presence, spatial localization, and dense segmentation. We further assess whether its predictions depend on stimulus-specific neural signals using shuffled fMRI–stimulus pairings and category-conditioned layout controls. BrainSeg also enables post-hoc category-conditioned attribution, relating individual decoded components to voxel-level and ROI-level relevance patterns. By making "what was where" an explicit decoding target, BrainSeg provides an interpretable framework for probing the organization of category-specific neural representations in human visual cortex.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural Representations

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_hCCXOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

BrainSeg formulates fMRI-based visual decoding as direct prediction of dense semantic segmentation maps rather than reconstruction of an RGB image. It projects each fMRI response into semantic queries and spatial features, then uses a query-guided decoder to predict segmentation masks for 133 COCO categories. A frozen MaskFormer is used to produce the supervised (fMRI,SegMap) training pairs (from NSD dataset). The BrainSeg model trains to receive an input fMRI and output the segmentation map of the corresponding viewed image.

The authors evaluate separately trained models for four NSD participants on 1,000 shared COCO images, reporting mean per-image category-presence F1, macro F1 and mIoU against the MaskFormer targets. BrainSeg outperforms the simple baselines trained by the authors, like linear-probe, MLP, and shuffled-fMRI controls. These suggest that the predictions contain stimulus-specific information. The paper also reports evaluation against human COCO annotations and category-conditioned voxel attribution analyses.

Critical strengths

The paper asks useful and clearly motivated questions: Which semantic categories can be decoded from fMRI? Where are they represented in the brain and where are they located inside the image? These are explored without allowing a generative image prior to fill in details. Predicting the semantic map directly makes category presence and spatial localization separately measurable and is a useful complement to image reconstruction from fMRI.

The evaluation includes several controls, including shuffled fMRI–stimulus pairings and non-neural priors, cross-subject consistency and category-conditioned attribution, making the work more informative.

Critical weaknesses

Poor Evaluation Methodology:

  1. Improper training procedure: The most major criticism is that the 1,000 test images are used during training for learning-rate scheduling and checkpoint selection. The final reported shared-image scores are validation results, not independent test results, which is not standard practice. I would like to see model selection performed on a separate subset of the training samples, with the 1,000 shared images reserved for final evaluation, as in prior decoding works.

  2. Poor baseline comparison: The main comparisons are kNN retrieval (in fMRI space), a linear probe, and an MLP. Although the ablations suggest that the query-mask head improves performance over these baselines, these baselines are too simplistic, and do not establish an advantage over representations used by recent fMRI decoding methods. I would expect a comparison with a strong fMRI-to-image reconstruction method. The reconstructed RGB images could be passed through the same frozen MaskFormer used in this paper, and the resulting segmentation maps evaluated using the same metrics and stimuli. This would test whether direct semantic decoding from fMRI actually outperforms segmenting reconstructed images. The argument that generative models may introduce priors does not remove the need for this comparison; rather, it motivates investigating the trade-off between segmentation performance and reliance on reconstructed image content. Finally, using an existing fMRI-to-text or VQA on fMRI method could provide a complementary baseline for category-presence prediction (F1 score).

  3. 3. Questionable results: The evaluation does not yet support a strong claim of broad 133-class decoding. The gap between per-image F1 (0.5415) and macro F1 (0.2681) suggests that performance varies substantially across categories and may be dominated by class frequent or by spatially extensive classes. Although the authors acknowledge this limitation, they do not sufficiently quantify the class-dependent performance. A frequency-stratified analysis, including performance across frequent, medium-frequency, and rare classes, together with results for "thing" versus "stuff" categories, would help establish how broadly the model decodes the 133 categories. It would also be useful to report how many classes significantly outperform a class-frequency baseline. Finally, the 64×64 output is visibly limiting the edges of small objects. A higher-resolution or multiscale evaluation would help establish how much of the result reflects semantic decoding versus coarse spatial layout.

Questions for authors

Can the authors report performance stratified by category frequency and area, include a COCO things and stuff breakdown? These analyses would help determine whether the reported performance generalizes across categories or is concentrated in frequent and spatially extensive classes.

Reviewer_8fLNOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors decode a dense semantic map directly from fMRI (Natural Scenes Dataset), avoiding image reconstruction so the output reflects brain signal rather than generative priors. A MaskFormer-style decoder turns fMRI into class-labeled masks, trained on pseudo-labels from a frozen MaskFormer.

Critical strengths
  1. The paper addresses fMRI-based visual decoding from an angle that is usually underexplored in the literature; instead of focusing on increasing performance, which I believe is now mostly saturated in image brain decoding from fMRI, I appreciate that they insist on learning semantic layouts directly from brain activity.

  2. I also like that their design is simple and gives cortical interpretability essentially for free.

Critical weaknesses

The only critical weakness I can identify is that the central motivation is not tested directly. The authors are worried that decoding models involving frozen diffusion models/decoders may owe their very good performances to generative priors, rather than actual information that can be accessed from fMRI alone. I find it would be a very nice addition to directly test the semantic maps of existing methods (e.g. MindEye or BrainDiffuser) against the same MaskFormer teacher. This alone is no reason to support rejection; I just think it's a weakness that could be improved with direct comparison to the cited models. I expect BrainSeg to produce more aligned maps with respect to BrainDiffuser or MindEye and I am very curious to see the results.

Questions for authors

How does BrainSeg compare with segmenting the outputs of existing visual brain decoding from fMRI methods using the same MaskFormer teacher?

Two small things. One, mIoU is never spelled out completely or defined; although it's commonly know, it's best to avoid giving this for granted. Two, the legend of Figure 4 in the Appendix is either completely uninformative or extremely confusing. Please provide an improved version.

Reviewer_zXoROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents BrainSeg, a framework for decoding dense semantic layouts directly from fMRI responses. The method uses a frozen MaskFormer to generate segmentation targets and learns to map fMRI signals into semantic queries and multiscale spatial features, which are subsequently combined to predict category-specific masks. Experiments on four NSD subjects show improvements over linear, MLP, and retrieval-based baselines.

Critical strengths
  • Directly decoding semantic layouts provides a useful alternative to image reconstruction pipelines.
Critical weaknesses
  • The methodological novelty is limited. The proposed architecture largely follows established query-based semantic segmentation designs, with separate projections of fMRI responses into semantic queries and spatial tokens. The multiscale cross-attention and query-mask prediction mechanisms are closely related to existing segmentation architectures. While applying this formulation to fMRI decoding is useful, the paper does not establish a substantial advance beyond adapting these components to a different input modality.

  • The proposed semantic and spatial branches are not explicitly disentangled. Both are derived from the same fMRI response, and there is no direct evidence that they learn complementary semantic and spatial information. The current ablations mainly demonstrate the benefit of the query-mask prediction head, leaving the contributions of the dual-branch design and multiscale cross-attention insufficiently characterized.

  • The paper discusses approaches such as UMBRAE and MindDETR, yet no direct comparison is provided. Even when the output representations differ, category recognition and coarse spatial localization could be evaluated under a shared protocol.

Questions for authors
  • What evidence supports the claim that the semantic-query and spatial-feature branches learn complementary information? Could the authors provide branch-specific ablations, including single-branch variants and removal of multiscale cross-attention?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.