BrainSeg: A Framework for Probing Category-Specific Neural Representations through Dense Semantic Decoding

The human visual system transforms sensory inputs into structured representations of semantic content and spatial organization. Recent deep-learning methods can reconstruct visually compelling images from fMRI responses, but it remains unclear whether the semantic content of these reconstructions is supported by measured brain responses or supplied by generative priors. We introduce BrainSeg, a framework that reformulates fMRI-based visual decoding as dense semantic prediction. Rather than reconstructing image appearance, BrainSeg directly predicts which semantic categories are present and where they occur. BrainSeg maps fMRI responses into complementary semantic queries and multiscale spatial features and progressively couples these two representations to produce a dense semantic map over 133 categories. Training targets are generated offline using a frozen MaskFormer, whereas inference relies on fMRI alone. We evaluate BrainSeg on held-out stimuli from the Natural Scenes Dataset using metrics of category presence, spatial localization, and dense segmentation. We further assess whether its predictions depend on stimulus-specific neural signals using shuffled fMRI–stimulus pairings and category-conditioned layout controls. BrainSeg also enables post-hoc category-conditioned attribution, relating individual decoded components to voxel-level and ROI-level relevance patterns. By making "what was where" an explicit decoding target, BrainSeg provides an interpretable framework for probing the organization of category-specific neural representations in human visual cortex.