- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
BrainSeg formulates fMRI-based visual decoding as direct prediction of dense semantic segmentation maps rather than reconstruction of an RGB image. It projects each fMRI response into semantic queries and spatial features, then uses a query-guided decoder to predict segmentation masks for 133 COCO categories. A frozen MaskFormer is used to produce the supervised (fMRI,SegMap) training pairs (from NSD dataset). The BrainSeg model trains to receive an input fMRI and output the segmentation map of the corresponding viewed image.
The authors evaluate separately trained models for four NSD participants on 1,000 shared COCO images, reporting mean per-image category-presence F1, macro F1 and mIoU against the MaskFormer targets. BrainSeg outperforms the simple baselines trained by the authors, like linear-probe, MLP, and shuffled-fMRI controls. These suggest that the predictions contain stimulus-specific information. The paper also reports evaluation against human COCO annotations and category-conditioned voxel attribution analyses.
Critical strengths
The paper asks useful and clearly motivated questions: Which semantic categories can be decoded from fMRI? Where are they represented in the brain and where are they located inside the image? These are explored without allowing a generative image prior to fill in details. Predicting the semantic map directly makes category presence and spatial localization separately measurable and is a useful complement to image reconstruction from fMRI.
The evaluation includes several controls, including shuffled fMRI–stimulus pairings and non-neural priors, cross-subject consistency and category-conditioned attribution, making the work more informative.
Critical weaknesses
Poor Evaluation Methodology:
-
Improper training procedure: The most major criticism is that the 1,000 test images are used during training for learning-rate scheduling and checkpoint selection. The final reported shared-image scores are validation results, not independent test results, which is not standard practice. I would like to see model selection performed on a separate subset of the training samples, with the 1,000 shared images reserved for final evaluation, as in prior decoding works.
-
Poor baseline comparison: The main comparisons are kNN retrieval (in fMRI space), a linear probe, and an MLP. Although the ablations suggest that the query-mask head improves performance over these baselines, these baselines are too simplistic, and do not establish an advantage over representations used by recent fMRI decoding methods. I would expect a comparison with a strong fMRI-to-image reconstruction method. The reconstructed RGB images could be passed through the same frozen MaskFormer used in this paper, and the resulting segmentation maps evaluated using the same metrics and stimuli. This would test whether direct semantic decoding from fMRI actually outperforms segmenting reconstructed images. The argument that generative models may introduce priors does not remove the need for this comparison; rather, it motivates investigating the trade-off between segmentation performance and reliance on reconstructed image content. Finally, using an existing fMRI-to-text or VQA on fMRI method could provide a complementary baseline for category-presence prediction (F1 score).
-
3. Questionable results: The evaluation does not yet support a strong claim of broad 133-class decoding. The gap between per-image F1 (0.5415) and macro F1 (0.2681) suggests that performance varies substantially across categories and may be dominated by class frequent or by spatially extensive classes. Although the authors acknowledge this limitation, they do not sufficiently quantify the class-dependent performance. A frequency-stratified analysis, including performance across frequent, medium-frequency, and rare classes, together with results for "thing" versus "stuff" categories, would help establish how broadly the model decodes the 133 categories. It would also be useful to report how many classes significantly outperform a class-frequency baseline. Finally, the 64×64 output is visibly limiting the edges of small objects. A higher-resolution or multiscale evaluation would help establish how much of the result reflects semantic decoding versus coarse spatial layout.
Questions for authors
Can the authors report performance stratified by category frequency and area, include a COCO things and stuff breakdown? These analyses would help determine whether the reported performance generalizes across categories or is concentrated in frequent and spatially extensive classes.