- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The manuscript proposes InsideOut, which maps fMRI responses to a shared visual representation and separately decodes object categories and bounding boxes. A pretrained GLIGEN model then generates images by combining the category from one response with the location from another. The mapping is trained using MSE, SoftCLIP, and a modified EFDM loss. Experiments assess constituent decoding and cross-response recombination, reporting higher joint preservation at the read-out stage than with voxel-level manipulation or latent averaging.
Critical strengths
The manuscript addresses an interesting question in compositional fMRI-to-image generation, with the potential to improve reconstruction quality and provide insights into how the brain represents visual information.
Critical weaknesses
-
The evaluation primarily measures whether generated images preserve the source category and location, providing limited evidence about perceptual quality and realism. Standard reconstruction metrics, such as SSIM, LPIPS, and CLIP-based identification, would be useful for within-response reconstructions. Refer to [1] for a standard evaluation protocol for fMRI-to-image generation.
-
Although the paper includes a representation-level comparison with several baselines, many important baselines are missing. These comparisons are important for establishing whether the proposed framework offers advantages over stronger brain representations and existing structured decoding approaches.
-
The framework largely integrates existing representation-learning methods, task-specific read-outs, and pretrained generative components. Although the cross-response recombination task is interesting, the methodological contribution appears incremental. The paper should clarify the novelty of its proposed EFDM variant and isolate the benefits of its methodological changes through targeted ablations.
-
The experiments are restricted to the category and two-dimensional bounding box of a single salient object. Evaluations involving attributes, relations, and multiple objects would provide stronger support for the broader claim of compositional brain decoding.
-
In Section 5.3 (Table 2(a)), recombining decoded categories and boxes relies on GLIGEN’s existing conditioning capabilities. It is unclear what this demonstrates beyond constituent decoding accuracy and generator controllability.
[1] Scotti, Paul S., et al. "Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data." arXiv preprint arXiv:2403.11207 (2024). [2] Lan, Yu-Ting, et al. "Seeing through the brain: image reconstruction of visual perception from human brain signals." arXiv preprint arXiv:2308.02510 (2023).
Questions for authors
How does the framework handle visual stimuli with multiple objects?
8 comments
W3. Incremental components and EFDM
The framework largely integrates existing representation-learning methods, task-specific read-outs, and pretrained generative components.
Response: We agree that InsideOut is built from existing parts, namely MindEye-style training [4], a frozen LLaVA with adapters, and a fixed GLIGEN. However, we believe that integrating them, each designed for different purposes, into one workflow for a new scientific question is a meaningful contribution. We put them together to read a category and a box from each brain response without giving the box read-out the category, which the recombination test needs. Three of our design choices still matter in experiments. First, combining after the read-outs works better than combining inside the latent (Table 2(a)). Second, finding the box without the category name works better than the approach of prior decoders, which are given the name, by 15% (+0.052 box IoU) for VINDEX-S1 [15] and 10% (+0.042) for UMBRAE [3]. Third, when the same read-outs are trained on different single-subject representations, ours gives the highest joint preservation, 15% (+0.074), 33% (+0.145) and 47% (+0.186) above VINDEX-S1, MindDiffuser [14] and MindEye1 [4]. We will present these as design choices, separate from our main scientific findings.
The paper should clarify the novelty of its proposed EFDM variant. Isolate the benefits of its methodological changes through targeted ablations.
Response: Our EFDM-based loss adapts the original EFDM to our setting. The original EFDM is a feature transformation for style transfer that keeps the rank of each content value and replaces the value with the sorted values of the style feature. We use it as a training loss instead, pulling the visual tokens predicted from the brain toward a target that keeps their ranks but takes the sorted values of the ground-truth image tokens, so that only their value distribution is aligned. As this adapts an existing principle to our setting rather than introducing a new technique, we will call it EFDM-based loss instead of improved EFDM and revise Section 4.4 accordingly. In a controlled ablation with the same training schedule, removing the loss lowers joint preservation by 5% ().
W4 / Question. Multiple objects, attributes, and relations
The experiments are restricted to the category and two-dimensional bounding box of a single salient object.
Response: We agree that our evaluation covers only the category and box of one object per image. The cross-response test combines the category from one image with the location from another, so each image must provide one category and one location that belong to the same object. When an image contains many objects, it becomes unclear which object the category and the location refer to. This difficulty comes from the images themselves and appears as soon as a second object is present. Even a vision-language model (LLaVA-1.5) that sees the image directly, without any brain data, locates the main object less accurately as objects are added (box IoU 0.479, 0.429, 0.379 and 0.219 for one, two, three or four, and five or more objects). We therefore evaluate one object per image, so that the test focuses on recombination itself rather than on deciding which object is meant. We acknowledge this limitation, will state it clearly in the paper, and leave multiple objects, attributes and relations to future work.
How does the framework handle visual stimuli with multiple objects?
Response: InsideOut decodes one main object per image, even when the image contains several objects (76% of our test images). As shown above, the more objects an image contains, the less clear it is which object a category and a location refer to, and even a vision-language model that sees the image locates the main object less accurately (box IoU 0.479 for one object vs. 0.219 for five or more). We therefore choose one main object per image, so that each image gives one category and one location of the same object. This choice does not limit what the latent encodes. A simple linear classifier on the same latent names all object categories in an image with an mAP of 0.417 over 80 COCO categories (mean of four subjects), 35% higher than iMIND [6] with fMRI only (0.310) and 62% higher than CLIP-MUSED [8] (0.258), although the protocols differ. Handling multiple objects in recombination is a natural extension, which we leave to future work.
W5. What Table 2(a) demonstrates beyond GLIGEN controllability
Recombining decoded categories and boxes relies on GLIGEN's existing conditioning capabilities. It is unclear what this demonstrates beyond constituent decoding accuracy and generator controllability.
Response: GLIGEN is held fixed across all comparisons, with identical prompts, seed rules, and detectors, and the floor and ceiling evaluated under the same procedure. Thus, Table 2(a) evaluates how the level of combining brain responses affects the independent reuse of what and where, rather than generator controllability. Voxel-, encoder-output-, and latent-level combinations preserve both constituents less often than read-out-level recombination.
One may ask whether the latent-level edits score lower only because they are read by our read-outs and drawn by GLIGEN. We therefore removed both. We turned each edited latent directly into an image with a different generator (Versatile Diffusion) and checked where the object appears. For any latent edit, the object appears at the target location in at most 44.7% of cases, against 64.1% for read-out recombination, so the result does not depend on GLIGEN or our read-outs. We will clarify this in the manuscript.
References
[1] Yu Takagi and Shinji Nishimoto. High-resolution Image Reconstruction with Latent Diffusion Models from Human Brain Activity. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[2] Furkan Ozcelik and Rufin VanRullen. Natural Scene Reconstruction from fMRI Signals Using Generative Latent Diffusion. Scientific Reports, 2023.
[3] Weihao Xia, Raoul de Charette, Cengiz Öztireli, and Jing-Hao Xue. UMBRAE: Unified Multimodal Brain Decoding. European Conference on Computer Vision (ECCV), 2024.
[4] Paul S. Scotti et al. Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. Advances in Neural Information Processing Systems (NeurIPS), 2023.
[5] Paul S. Scotti et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. International Conference on Machine Learning (ICML), 2024.
[6] Zixiang Yin, Jiarui Li, and Zhengming Ding. iMIND: Insightful Multi-subject Invariant Neural Decoding. Advances in Neural Information Processing Systems (NeurIPS), 2025.
[7] Chong Li, Jingyang Huo, Weikang Gong, Yanwei Fu, Xiangyang Xue, and Jianfeng Feng. DecoFuse: Decomposing and Fusing the "What", "Where", and "How" for Brain-Inspired fMRI-to-Video Decoding. arXiv:2504.00432, 2025.
[8] Qiongyi Zhou, Changde Du, Shengpei Wang, and Huiguang He. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding. International Conference on Learning Representations (ICLR), 2024.
[9] Zheng Huang et al. Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI (PRISM). International Conference on Learning Representations (ICLR), 2026.
[10] Weikang Qiu, Zheng Huang, Haoyu Hu, Aosong Feng, Yujun Yan, and Rex Ying. MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding. International Conference on Machine Learning (ICML), 2025.
[11] Haonan Wang, Jingyu Lu, Hongrui Li, and Xiaomeng Li. ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding. Advances in Neural Information Processing Systems (NeurIPS), 2025.
[12] Xuan-Bac Nguyen, Manuel Serna-Aguilera, Arabinda Kumar Choudhary, Pawan Sinha, Xin Li, and Khoa Luu. COBRA: A Continual Learning Approach to Vision-Brain Understanding. arXiv:2411.17475, 2024.
[13] Weihao Xia and Cengiz Oztireli. Multigranular Evaluation for Brain Visual Decoding (BASIC). AAAI Conference on Artificial Intelligence (AAAI), 2026.
[14] Yizhuo Lu, Changde Du, Qiongyi Zhou, Dianpeng Wang, and Huiguang He. MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion. ACM International Conference on Multimedia (ACM MM), 2023.
[15] Weihao Xia and Cengiz Oztireli. Exploring the Visual Feature Space for Multimodal Neural Decoding (VINDEX). IEEE/CVF International Conference on Computer Vision (ICCV), 2025.
We sincerely thank the reviewer for the careful assessment. We agree that reconstruction fidelity, baseline coverage, and limits of the single-object setting should be clearer. Our central objective, however, is to test the independent reuse of brain-decoded visual constituents, not to introduce a new image generator. We respond below with additional measurements and a more precise scope of claims.
W1. Reconstruction and perceptual metrics
Response: We agree that constituent preservation does not measure perceptual quality or realism, and we do not claim realism for composed images. We couldn't evaluate realism for composed images because such composed images have no ground-truth images. Since its rendering is conditioned only on the decoded category and box, we use the rendering only to examine whether what and where can be independently recombined across brain responses.
Response: Reconstruction is not the goal of the paper, so we report the standard image reconstruction metrics only as a reference (1,000 test images, four subjects, baseline values from the original papers).
Our reconstructions are below MindEye and MindEye2, and our LPIPS is 0.662. We will state this scope in Section 5.3 and add this table to the appendix.
W2. Missing baselines
Response: We selected the baselines based on task relevance and types of information they decode. But we agree that broader baseline comparisons are valuable. iMIND [6], DecoFuse [7], CLIP-MUSED [8], PRISM [9], MindLLM [10], ZEBRA [11], COBRA [12], and BASIC [13] differ from our category–box recombination task in their research objectives, decoding outputs, or datasets. We will clarify these distinctions in Related Work. We additionally evaluated MindDiffuser [14], whose decoding outputs are relatively comparable to ours after UMBRAE [3], by pairing its semantic feature from one response with its detail feature, a VQ-VAE latent, from another. Its location preservation was 0.348, near the random floor of 0.319 and below 0.641 for ours. Even without recombination, its reconstructions preserve location in 0.404 of cells, against 0.693 for our within-response renderings, suggesting limited object-level spatial information in its features.
To compare stronger brain representations, we retrained our two read-outs on existing decoders' representations using identical settings and evaluated the same 1,000 cells. Ours gives the highest joint preservation among single-subject representations (joint and location differences p≤0.006), and representations trained with other subjects match or exceed ours, so the cross-response recombination test applies to other representations and benefits from stronger ones.
Bold marks the best value within each group, and underline marks the best among representations trained only on subject 1. All rows use our two read-outs trained on each representation. Category and box values therefore differ from Table 1, which reports each method's own output (for example, box IoU 0.337 for VINDEX-S1 and 0.400 for UMBRAE).