acceptodds
Under review as a conference paper at ICLR 2027

Compositional Brain Decoding via Visual Constituent Recombination

Abstract

Recent advances in fMRI visual decoding recover increasingly rich semantic and perceptual information, yet it remains unclear whether broadly overlapping object and spatial information can be recovered as independently usable constituents rather than only within their original associations. We propose InsideOut, a framework for compositional brain decoding that operationalizes object category and spatial location as what and where constituents. InsideOut explicitly decodes these constituents and recombines them across responses elicited by different images to test whether each preserves its source-specific information after its original association is broken. In the subject-1 composition evaluation, 58.1% of cross-response compositions preserve both category and location, and read-out-level recombination outperforms voxel- and latent-level alternatives. Together, these findings provide a compositional view of brain decoding in which distinct visual constituents can be separately recovered from brain responses and remain reusable beyond the associations in which they were originally observed.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain Decoding, fMRI, Compositionality, Representation Learning, Neuroimaging, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uFkyOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript proposes InsideOut, which maps fMRI responses to a shared visual representation and separately decodes object categories and bounding boxes. A pretrained GLIGEN model then generates images by combining the category from one response with the location from another. The mapping is trained using MSE, SoftCLIP, and a modified EFDM loss. Experiments assess constituent decoding and cross-response recombination, reporting higher joint preservation at the read-out stage than with voxel-level manipulation or latent averaging.

Critical strengths

The manuscript addresses an interesting question in compositional fMRI-to-image generation, with the potential to improve reconstruction quality and provide insights into how the brain represents visual information.

Critical weaknesses
  1. The evaluation primarily measures whether generated images preserve the source category and location, providing limited evidence about perceptual quality and realism. Standard reconstruction metrics, such as SSIM, LPIPS, and CLIP-based identification, would be useful for within-response reconstructions. Refer to [1] for a standard evaluation protocol for fMRI-to-image generation.

  2. Although the paper includes a representation-level comparison with several baselines, many important baselines are missing. These comparisons are important for establishing whether the proposed framework offers advantages over stronger brain representations and existing structured decoding approaches.

  3. The framework largely integrates existing representation-learning methods, task-specific read-outs, and pretrained generative components. Although the cross-response recombination task is interesting, the methodological contribution appears incremental. The paper should clarify the novelty of its proposed EFDM variant and isolate the benefits of its methodological changes through targeted ablations.

  4. The experiments are restricted to the category and two-dimensional bounding box of a single salient object. Evaluations involving attributes, relations, and multiple objects would provide stronger support for the broader claim of compositional brain decoding.

  5. In Section 5.3 (Table 2(a)), recombining decoded categories and boxes relies on GLIGEN’s existing conditioning capabilities. It is unclear what this demonstrates beyond constituent decoding accuracy and generator controllability.

[1] Scotti, Paul S., et al. "Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data." arXiv preprint arXiv:2403.11207 (2024). [2] Lan, Yu-Ting, et al. "Seeing through the brain: image reconstruction of visual perception from human brain signals." arXiv preprint arXiv:2308.02510 (2023).

Questions for authors

How does the framework handle visual stimuli with multiple objects?

8 comments

We sincerely thank the reviewer for the careful assessment. We agree that reconstruction fidelity, baseline coverage, and limits of the single-object setting should be clearer. Our central objective, however, is to test the independent reuse of brain-decoded visual constituents, not to introduce a new image generator. We respond below with additional measurements and a more precise scope of claims.


W1. Reconstruction and perceptual metrics

The evaluation primarily measures whether generated images preserve the source category and location, providing limited evidence about perceptual quality and realism.

Response: We agree that constituent preservation does not measure perceptual quality or realism, and we do not claim realism for composed images. We couldn't evaluate realism for composed images because such composed images have no ground-truth images. Since its rendering is conditioned only on the decoded category and box, we use the rendering only to examine whether what and where can be independently recombined across brain responses.

Standard reconstruction metrics, such as SSIM, LPIPS, and CLIP-based identification, would be useful for within-response reconstructions.

Response: Reconstruction is not the goal of the paper, so we report the standard image reconstruction metrics only as a reference (1,000 test images, four subjects, baseline values from the original papers).

MethodPixCorrSSIMAlex(2)Alex(5)IncepCLIPEffNet-B ↓SwAV ↓
Takagi and Nishimoto [1]0.2460.41078.985.683.882.10.8110.504
Brain-Diffuser [2]0.2730.36594.496.691.390.90.7280.421
UMBRAE [3]0.2830.34195.597.091.793.50.7000.393
InsideOut (mean of four subjects)0.3000.28290.695.590.892.10.7100.412
MindEye [4]0.3190.36092.896.994.693.30.6480.377
MindEye2 [5]0.3220.43196.198.695.493.00.6190.344

Our reconstructions are below MindEye and MindEye2, and our LPIPS is 0.662. We will state this scope in Section 5.3 and add this table to the appendix.


W2. Missing baselines

Many important baselines are missing.

Response: We selected the baselines based on task relevance and types of information they decode. But we agree that broader baseline comparisons are valuable. iMIND [6], DecoFuse [7], CLIP-MUSED [8], PRISM [9], MindLLM [10], ZEBRA [11], COBRA [12], and BASIC [13] differ from our category–box recombination task in their research objectives, decoding outputs, or datasets. We will clarify these distinctions in Related Work. We additionally evaluated MindDiffuser [14], whose decoding outputs are relatively comparable to ours after UMBRAE [3], by pairing its semantic feature from one response with its detail feature, a VQ-VAE latent, from another. Its location preservation was 0.348, near the random floor of 0.319 and below 0.641 for ours. Even without recombination, its reconstructions preserve location in 0.404 of cells, against 0.693 for our within-response renderings, suggesting limited object-level spatial information in its features.

To compare stronger brain representations, we retrained our two read-outs on existing decoders' representations using identical settings and evaluated the same 1,000 cells. Ours gives the highest joint preservation among single-subject representations (joint and location differences p≤0.006p \leq 0.006), and representations trained with other subjects match or exceed ours, so the cross-response recombination test applies to other representations and benefits from stronger ones.

RepresentationTraining dataCategory top-1Box IoULocationJoint
InsideOut (Ours)subject 10.6120̲.̲4̲4̲2̲0̲.̲6̲4̲1̲0̲.̲5̲8̲1̲
MindEye1subject 10.5510.3220.4520.395
MindDiffusersubject 10.4720.3490.5090.436
VINDEX-S1subject 10̲.̲6̲2̲0̲0.3880.5550.507
UMBRAEfour subjects0.6210.4420.6440.584
MindEye2eight subjects0.6690.4820.6870.635

Bold marks the best value within each group, and underline marks the best among representations trained only on subject 1. All rows use our two read-outs trained on each representation. Category and box values therefore differ from Table 1, which reports each method's own output (for example, box IoU 0.337 for VINDEX-S1 and 0.400 for UMBRAE).


W3. Incremental components and EFDM

The framework largely integrates existing representation-learning methods, task-specific read-outs, and pretrained generative components.

Response: We agree that InsideOut is built from existing parts, namely MindEye-style training [4], a frozen LLaVA with adapters, and a fixed GLIGEN. However, we believe that integrating them, each designed for different purposes, into one workflow for a new scientific question is a meaningful contribution. We put them together to read a category and a box from each brain response without giving the box read-out the category, which the recombination test needs. Three of our design choices still matter in experiments. First, combining after the read-outs works better than combining inside the latent (Table 2(a)). Second, finding the box without the category name works better than the approach of prior decoders, which are given the name, by 15% (+0.052 box IoU) for VINDEX-S1 [15] and 10% (+0.042) for UMBRAE [3]. Third, when the same read-outs are trained on different single-subject representations, ours gives the highest joint preservation, 15% (+0.074), 33% (+0.145) and 47% (+0.186) above VINDEX-S1, MindDiffuser [14] and MindEye1 [4]. We will present these as design choices, separate from our main scientific findings.

The paper should clarify the novelty of its proposed EFDM variant. Isolate the benefits of its methodological changes through targeted ablations.

Response: Our EFDM-based loss adapts the original EFDM to our setting. The original EFDM is a feature transformation for style transfer that keeps the rank of each content value and replaces the value with the sorted values of the style feature. We use it as a training loss instead, pulling the visual tokens predicted from the brain toward a target that keeps their ranks but takes the sorted values of the ground-truth image tokens, so that only their value distribution is aligned. As this adapts an existing principle to our setting rather than introducing a new technique, we will call it EFDM-based loss instead of improved EFDM and revise Section 4.4 accordingly. In a controlled ablation with the same training schedule, removing the loss lowers joint preservation by 5% (0.557→0.5280.557 \rightarrow 0.528).


W4 / Question. Multiple objects, attributes, and relations

The experiments are restricted to the category and two-dimensional bounding box of a single salient object.

Response: We agree that our evaluation covers only the category and box of one object per image. The cross-response test combines the category from one image with the location from another, so each image must provide one category and one location that belong to the same object. When an image contains many objects, it becomes unclear which object the category and the location refer to. This difficulty comes from the images themselves and appears as soon as a second object is present. Even a vision-language model (LLaVA-1.5) that sees the image directly, without any brain data, locates the main object less accurately as objects are added (box IoU 0.479, 0.429, 0.379 and 0.219 for one, two, three or four, and five or more objects). We therefore evaluate one object per image, so that the test focuses on recombination itself rather than on deciding which object is meant. We acknowledge this limitation, will state it clearly in the paper, and leave multiple objects, attributes and relations to future work.

How does the framework handle visual stimuli with multiple objects?

Response: InsideOut decodes one main object per image, even when the image contains several objects (76% of our test images). As shown above, the more objects an image contains, the less clear it is which object a category and a location refer to, and even a vision-language model that sees the image locates the main object less accurately (box IoU 0.479 for one object vs. 0.219 for five or more). We therefore choose one main object per image, so that each image gives one category and one location of the same object. This choice does not limit what the latent encodes. A simple linear classifier on the same latent names all object categories in an image with an mAP of 0.417 over 80 COCO categories (mean of four subjects), 35% higher than iMIND [6] with fMRI only (0.310) and 62% higher than CLIP-MUSED [8] (0.258), although the protocols differ. Handling multiple objects in recombination is a natural extension, which we leave to future work.


W5. What Table 2(a) demonstrates beyond GLIGEN controllability

Recombining decoded categories and boxes relies on GLIGEN's existing conditioning capabilities. It is unclear what this demonstrates beyond constituent decoding accuracy and generator controllability.

Response: GLIGEN is held fixed across all comparisons, with identical prompts, seed rules, and detectors, and the floor and ceiling evaluated under the same procedure. Thus, Table 2(a) evaluates how the level of combining brain responses affects the independent reuse of what and where, rather than generator controllability. Voxel-, encoder-output-, and latent-level combinations preserve both constituents less often than read-out-level recombination.

One may ask whether the latent-level edits score lower only because they are read by our read-outs and drawn by GLIGEN. We therefore removed both. We turned each edited latent directly into an image with a different generator (Versatile Diffusion) and checked where the object appears. For any latent edit, the object appears at the target location in at most 44.7% of cases, against 64.1% for read-out recombination, so the result does not depend on GLIGEN or our read-outs. We will clarify this in the manuscript.


References

[1] Yu Takagi and Shinji Nishimoto. High-resolution Image Reconstruction with Latent Diffusion Models from Human Brain Activity. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.

[2] Furkan Ozcelik and Rufin VanRullen. Natural Scene Reconstruction from fMRI Signals Using Generative Latent Diffusion. Scientific Reports, 2023.

[3] Weihao Xia, Raoul de Charette, Cengiz Öztireli, and Jing-Hao Xue. UMBRAE: Unified Multimodal Brain Decoding. European Conference on Computer Vision (ECCV), 2024.

[4] Paul S. Scotti et al. Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors. Advances in Neural Information Processing Systems (NeurIPS), 2023.

[5] Paul S. Scotti et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. International Conference on Machine Learning (ICML), 2024.

[6] Zixiang Yin, Jiarui Li, and Zhengming Ding. iMIND: Insightful Multi-subject Invariant Neural Decoding. Advances in Neural Information Processing Systems (NeurIPS), 2025.

[7] Chong Li, Jingyang Huo, Weikang Gong, Yanwei Fu, Xiangyang Xue, and Jianfeng Feng. DecoFuse: Decomposing and Fusing the "What", "Where", and "How" for Brain-Inspired fMRI-to-Video Decoding. arXiv:2504.00432, 2025.

[8] Qiongyi Zhou, Changde Du, Shengpei Wang, and Huiguang He. CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding. International Conference on Learning Representations (ICLR), 2024.

[9] Zheng Huang et al. Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI (PRISM). International Conference on Learning Representations (ICLR), 2026.

[10] Weikang Qiu, Zheng Huang, Haoyu Hu, Aosong Feng, Yujun Yan, and Rex Ying. MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-Text Decoding. International Conference on Machine Learning (ICML), 2025.

[11] Haonan Wang, Jingyu Lu, Hongrui Li, and Xiaomeng Li. ZEBRA: Towards Zero-Shot Cross-Subject Generalization for Universal Brain Visual Decoding. Advances in Neural Information Processing Systems (NeurIPS), 2025.

[12] Xuan-Bac Nguyen, Manuel Serna-Aguilera, Arabinda Kumar Choudhary, Pawan Sinha, Xin Li, and Khoa Luu. COBRA: A Continual Learning Approach to Vision-Brain Understanding. arXiv:2411.17475, 2024.

[13] Weihao Xia and Cengiz Oztireli. Multigranular Evaluation for Brain Visual Decoding (BASIC). AAAI Conference on Artificial Intelligence (AAAI), 2026.

[14] Yizhuo Lu, Changde Du, Qiongyi Zhou, Dianpeng Wang, and Huiguang He. MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion. ACM International Conference on Multimedia (ACM MM), 2023.

[15] Weihao Xia and Cengiz Oztireli. Exploring the Visual Feature Space for Multimodal Neural Decoding (VINDEX). IEEE/CVF International Conference on Computer Vision (ICCV), 2025.

Reviewer_yPoFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work asks whether object category ("what") and spatial location ("where") decoded from fMRI stay usable once their original pairing is broken; in other words, the authors want to verify the compositionality of what and where in brain responses by separately decoding them, and then recombining in various ways. They use the Natural Scenes Dataset (NSD); a MindEye-style encoder plus diffusion prior maps NSD responses to GIT image tokens, trained with an added EFDM distribution-matching loss. This representation is then frozen, and two separate read-outs operate on it: an MLP classifies the salient object's COCO category, while a LLaVA adapter outputs its bounding box. For composition, the category decoded from response A is paired with the box decoded from response B and rendered with GLIGEN.

Critical strengths
  1. The research question is important, well defined, and elegantly addressed by the architecture and the later analyses. Framing decoding in terms of compositionality, and asking where along the pipeline separable constituents first appear, is a useful perspective for both the neuroscience and ML communities.
  2. The comparison of recombination at different levels (voxels, latents, read-outs) points in the right direction for locating where compositionality emerges and the cross-response recombination is an elegant test.
  3. The paper is well written and easy to follow.
Critical weaknesses
  1. The statement in the discussion that early visual cortex "carries more of the box, without being assigned to it, fits this picture" is unclear to me. I think in general the paper would benefit from a richer discussion. In this case, it is not explained which picture is meant, nor how a cortical ablation result supports the preceding claim that the remaining location gap lies in the where read-out. The supporting evidence is also limited: box IoU is nearly identical when keeping only early visual or only face/body regions, the region groups are not size-matched, and box decoding from face/body regions may reflect category-typical box priors rather than spatial information.
  2. The "latent" blend is scored as both cells of a pair, so its identity and joint scores cannot exceed 0.5. In my opinion, a fairer latent-level comparison should produce a separate latent for each cross-response cell, e.g., by transplanting patch tokens within decoded boxes or by swapping a category-predictive linear subspace (z_AB = P_cat z_A + (I − P_cat) z_B). Furthermore, the comparison with Brain Algebra is a single instantiation. Their claim that read-out-level recombination outperforms voxel- and latent-level alternatives should be grounded on better latent-level experiments.
Questions for authors
  1. Could the authors clarify which picture the early-visual-cortex result supports, and how it relates to the claim that the remaining location gap lies in the where read-out? Would the results hold with size-matched region groups and against a baseline that decodes a category-mean-box?

  2. Could the authors add a latent-level baseline that produces a separate latent for each cross-response cell, and report Brain Algebra across a range of alpha? This would show whether read-out-level recombination is genuinely better.

  3. Could the authors report the within-response combinations (c_AA, c_BB), defined in equation 2, under the same identity/location/joint scoring of table 2? This would help quantify how costly it is to break the original what–where pairs.

4 comments

We sincerely thank the reviewer for recognizing the value of the research question and the cross-response test, and for suggesting concrete improvements. We agree that the original cortical interpretation and latent comparison required more careful controls. The following additional analyses directly address those points.


W1 / Q1. Cortical analyses and alternative explanations

The statement in the discussion that early visual cortex "carries more of the box, without being assigned to it, fits this picture" is unclear to me.

Response: We agree that the sentence was unclear and that the ablation alone cannot support it. It linked the early-visual result to the preceding claim that the remaining location gap lies mostly in the where read-out, suggesting that the brain response itself carries location well. A cortical ablation, however, shows where decodable information lies in the brain response, not where the downstream error arises, so it cannot support that claim. That claim rests on a different comparison. Location preservation is 0.641 from brain responses, already 91% of the 0.705 obtained when the same read-outs receive the image itself instead of brain data, and 0.941 with true labels. Most of the gap to 0.941 therefore remains even without brain data, but it may come from the read-out, the image representation, the generator or the detector, so we will not attribute it to the where read-out alone. We will state this claim separately from the cortical results and revise the attribution of this gap in Section 5.3 accordingly.

Box IoU is nearly identical when keeping only early visual or only face/body regions, the region groups are not size-matched.

Response: This is a valid confound. We therefore repeated the comparisons of Figure 5 with the same number of subsampled voxels in each group (2,252 voxels, the size of the scene group, three random draws, 981 test images). Category accuracy remains highest in the face and body group (0.400), about four times that of early visual cortex (0.102) and about 2.5 times that of the scene group (0.162). By contrast, box IoU differs by only 2% between the face and body group and early visual cortex (0.349 and 0.356) and is about 20% lower for the scene group (0.286). Compared with removing the same number of random voxels outside the three groups, removing early visual cortex lowers box IoU by 13% (0.437→0.3810.437 \rightarrow 0.381), and removing the face and body group lowers category accuracy by 25% (0.571→0.4270.571 \rightarrow 0.427). These controlled results support graded regional differences for category while confirming broadly distributed spatial decodability. We will report the sampling protocol and resulting scores.

Box decoding from face/body regions may reflect category-typical box priors rather than spatial information.

Response: We agree, and we tested this with a baseline that knows only the category. For each category, it outputs the average box of that category in the training images, so it uses no location information. This baseline reaches a box IoU of 0.334 with the true category (0.327 with the decoded category). Face and body voxels alone reach 0.348, only 4% above this baseline. Their box also overlaps the average box of the decoded category (IoU 0.377) more than the true box (0.348), so it largely reflects where objects of that category usually appear, as the reviewer suggested. The whole visual-cortex input reaches 0.447, 34% above the baseline, and its box overlaps the true box more than the average box (0.447 against 0.333), so it carries location information beyond the category. In recombination, replacing our box with the average box of the decoded category also lowers location preservation from 0.641 to 0.492. We will revise the face and body interpretation accordingly.

Could the authors clarify which picture the early-visual-cortex result supports, and how it relates to the claim that the remaining location gap lies in the where read-out?

Response: As noted above, the picture was the claim that the remaining location gap lies mostly in the where read-out, and the cortical result cannot support it. We will revise the sentence to clearly separate the cortical results from the composition results.


W2 / Q2. Fair latent-level comparisons

The "latent" blend is scored as both cells of a pair, so its identity and joint scores cannot exceed 0.5.

Response: We agree that scoring a single latent blend as both cross-response cells imposes a 0.5 ceiling on identity and joint preservation, and we will replace it with per-cell baselines. With a separate latent for each cell (details in next response), read-out recombination still gives the highest joint preservation (0.581), 13% above the category subspace swap (0.513) and 7% above the best latent edit in the table below (0.541).

A fairer latent-level comparison should produce a separate latent for each cross-response cell, e.g., by transplanting patch tokens within decoded boxes or by swapping a category-predictive linear subspace.

Response: We followed this suggestion and found that category-subspace swapping performed best among linear latent edits, transferring category more effectively than location. Location-subspace swapping preserved category in 99.1% of cells but showed limited location transfer. Spatial token manipulation and nonlinear editing improved location transfer but remained below read-out-level recombination, and these results reveal an asymmetry between category and spatial manipulation (Subject 1, 1,000 cells).

LevelMethodIdentityLocationJoint
VoxelBrain Algebra, best α\alpha (α=3\alpha = 3)0.7460.5240.451
LatentCategory subspace swap0.7720.5820.513
LatentLocation subspace swap0.8370.2410.226
LatentObject tokens moved to the target box0.8180.3970.360
LatentCategory component warped into the target box0.7580.6130.535
LatentLearned nonlinear what and where codes0.7640.6100.541
Read-outInsideOut0.8360.6410.581

One may ask whether the latent-level edits score lower only because they are read by our read-outs and drawn by GLIGEN. We therefore removed both. We turned each edited latent directly into an image with a different generator (Versatile Diffusion) and checked where the object appears. For any latent edit, the object appears at the target location in at most 44.7% of cases, against 64.1% for read-out recombination.

The comparison with Brain Algebra is a single instantiation.

Response: We evaluated Brain Algebra at α\alpha = 0, 1, 2, 3, 4 and 6, which gives joint preservation of 0.088, 0.176, 0.383, 0.451, 0.448 and 0.434. Its best value (0.451 at α\alpha = 3) is below the category subspace swap (0.513) and read-out recombination (0.581). Even when Brain Algebra is given the true category name, which our method never receives, it reaches 0.533 at α\alpha = 4, still below read-out recombination (p=0.05p = 0.05). We will report the full α\alpha curve and limit our conclusions to the evaluated settings.


Q3. Within-response vs. cross-response combinations

Could the authors report the within-response combinations (c_AA, c_BB) [...] under the same identity/location/joint scoring of Table 2?

Response: Following the reviewer's suggestion, we scored within-response combinations (cAAc_{AA}, cBBc_{BB}) on the same 500 Subject 1 pairs. Within-response combinations preserve identity, location and both in 0.835, 0.693 and 0.632 of cells, against 0.836, 0.641 and 0.581 for cross-response combinations. Breaking the original pairs therefore keeps identity and lowers joint preservation by 8% (0.632→0.5810.632 \rightarrow 0.581). Most of this cost comes from the generator and detector rather than from brain decoding, because true labels also lose 4% when recombined (0.966→0.9280.966 \rightarrow 0.928), as unusual category–location pairs are harder to render and detect. The extra loss for our decoded constituents beyond that of true labels is small and not significant (joint $-$0.013, p=0.44p = 0.44), so decoded constituents can be recombined almost as freely as true labels.

I appreciate the efforts that the authors have put into this rebuttal and I acknowledge their answers. I will keep my assessment.

We sincerely thank the reviewer for their time and valuable feedback. We will carefully incorporate the feedback and clarifications from our rebuttal into the revised manuscript.

Reviewer_EE7zOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new brain decoding task called Visual constituent recombination and proposes InsideOut as a solution by decoding both what and where information from the fMRI signals. The experimental results show that the proposed method achieves strong decoding performance on the NSD dataset and is capable of recombining an object from one image with the location from another.

Critical strengths
  1. The idea of remixing an object from one image with the location from another is interesting and new.

  2. The paper is written in good English with good flow and readability.

  3. The proposed InsideOut successfully achieves the purpose of recombination as designed and turns out to be strong on neural decoding tasks on the NSD dataset.

Critical weaknesses
  1. The scientific novelty is limited. Decoding semantics (object category) and spatial information from fMRI have each been studied a lot, which is acknowledged by the author in Section 2. Importantly, decoding two kinds of information from fMRI representations is also not new. iMIND [1] splits fMRI representation into subject-specific and object-specific components, one biometric and one semantic. Although the kind of information differs (subject/object rather than what/where), the dual-decoding design is the same. Plus, DecoFuse [2] for fMRI-to-video decoding splits the stimulus into "what", "where," and "how", which is mentioned only in related works without a clear differentiation. The remaining novelty, cross-response recombination, adds no new scientific finding. Both readouts in InsideOut are computed independently from each response, so any decoded label can be paired with any decoded box by construction. Moreover, it is unclear why this task is needed and what it reveals beyond single-response decoding accuracy.

  2. The technical contribution is limited as well. The majority of components in InsideOut are from prior work or are frozen pretrained checkpoints. The brain encoder training loss and diffusion prior follow MindEye. The localization and caption read-outs are based on a LLaVA-1.5-7B with an adapter. UMBRAE's [3] brain grounding did almost the same thing.

  3. The evaluation does not focus on the tasks for which the model is built. InsideOut is trained for object classification and box grounding. Yet ten of the twelve columns in Table 1 are caption metrics, and Figure 4 and Figure 7 show image reconstructions. A direct evaluation of both tasks is needed. If I understand correctly, InsideOut's what read-out is a single-label, 79-way classifier for one salient object, whereas NSD object decoding is usually evaluated as multi-label classification (mAP, AUC, etc.), by iMIND [1] and CLIP-MUSED [4]. The authors should report this protocol or justify reducing each scene to one object, given that 76% of test images contain more than one object.

  4. The paper claims that a prompt naming no category "allows the where read-out to operate independently of what" in Section 4.2. However, it is a point that was never validated. The prompt asks LLaVA to "locate the most salient object", which plausibly requires recognizing what the object is before localizing it. More importantly, both read-outs share the same latent zz. There is no mechanism to control "where" independently of "what" during decoding.

  5. The last minor point is that only 6 of the 33 references are from 2025 or later (2 from 2026). In a fast-moving field, this is not good enough.

[1] Yin, Zixiang, Jiarui Li, and Zhengming Ding. "$ i $ MIND: Insightful Multi-subject Invariant Neural Decoding." Advances in Neural Information Processing Systems 38 (2026): 148752-148794.

[2] Li, Chong, et al. "DecoFuse: Decomposing and Fusing the" What"," Where", and" How" for Brain-Inspired fMRI-to-Video Decoding." arXiv preprint arXiv:2504.00432 (2025).

[3] Xia, Weihao, et al. "Umbrae: Unified multimodal brain decoding." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024.

[4] Zhou, Qiongyi, et al. "CLIP-MUSED: CLIP-guided multi-subject visual neural information semantic decoding." International Conference on Learning Representations. Vol. 2024. 2024.

Questions for authors

Could the author clarify why we need to introduce the recombination of the object from one image with the location from another as a new brain decoding task? I don't see the necessity or a clear motivation.

7 comments

We thank the reviewer for the careful review and for finding the recombination idea interesting and new. Below we clarify our scientific question, narrow our claims where needed, and report additional experiments.


W1 / Question. Scientific novelty and motivation

Decoding semantics (object category) and spatial information from fMRI have each been studied a lot.

Response: We agree that decoding category or location alone is not new. Our question is different. As stated in the abstract, we ask whether object and spatial information, which overlap in visual cortex, can be decoded as separate pieces that remain usable when combined in new ways. Answering it also shows at which stage of decoding the two pieces become reusable. Single-response accuracy cannot answer this question, because it does not change with the stage at which two responses are combined. This distinction highlights the need to evaluate whether constituent attributes can be independently decoded from brain signal, as successful reconstruction of the original stimulus alone does not guarantee this. To our knowledge, no prior work has tested whether an object decoded from one brain response and a location decoded from another are both preserved when the two are combined into one image. The closest recombination work, Brain Algebra, adds concept patterns to brain responses in voxel space, and we compare with it in Table 2(a).

Decoding two kinds of information from fMRI representations is also not new. iMIND [...] splits fMRI representation into subject-specific and object-specific components [...]. DecoFuse [...] splits the stimulus into "what," "where," and "how", which is mentioned only in related works without a clear differentiation.

Response: We agree that decoding two kinds of information is not new, and we do not claim the two-branch design as our contribution. Prior methods split information for other goals. iMIND [1] separates subject from object information for subject identification and multi-label classification. DecoFuse [2] separates semantic, spatial, and motion features to reconstruct videos, and its spatial part is a feature map, not an object location. MindDiffuser [3] separates semantic and structural features to reconstruct images. None of them combines a piece decoded from one response with a piece decoded from another. We tried this with MindDiffuser, whose decoding outputs are relatively comparable to ours after UMBRAE [4], pairing the semantic feature of one response with the detail feature of another in its first stage, without its second-stage structural alignment. The location of the second response was kept in only 0.348 of cells, close to the random floor of 0.319 and well below 0.641 for ours. Combining its features directly therefore carries little object location across responses, and even within a response its reconstructions keep location in only 0.404 of cells. We will make this distinction clear in Related Work.

MethodWhat is separatedGoalObject locationCross-response
iMIND [1]subject / objectsubject ID, classification✗✗
DecoFuse [2]semantic / spatial / motionvideo reconstruction✗ (feature map)✗
MindDiffuser [3]semantic / structuralimage reconstruction✗✗
UMBRAE [4]no splitmultimodal decoding✓ (given the name)✗
Brain Algebrano splittesting compositionality✗✓ (voxel space)
InsideOut (Ours)category / boxtesting compositionality✓ (without the name)✓

The table shows how InsideOut differs, and the difference matters because existing scores cannot tell whether a decoder has really separated the two pieces. They compare each output with its own image, where the category and the location come together, so a decoder can score well by recognizing the category and guessing where such objects usually appear. Recombination pairs pieces from different images, so the drawn category no longer hints at where the object should be. It therefore reveals what reconstruction scores cannot.

For example, MindEye1 reconstructs images better than ours, yet with our read-outs on its representation, its recombined images show both the right category and the right location 32% less often (0.395 against 0.581), mostly because the location is wrong (0.452 against 0.641). Reconstruction scores miss this.

The remaining novelty, cross-response recombination, adds no new scientific finding. Both readouts in InsideOut are computed independently from each response, so any decoded label can be paired with any decoded box by construction.

Response: We agree that pairing itself is trivial, since any decoded category can be drawn at any decoded box. Pairing is not our finding. Recombination tests two things that the design does not guarantee.

First, are the two read-outs really independent? If the box read-out secretly used the category, changing the category would also move the box. We replaced the category information inside one response's latent with that of another response. The decoded category changed, but the box stayed in place (IoU 0.720 with its own box, against 0.269 with the other response's box).

Second, where should two responses be combined? Inside the latent, swapping the part that carries the category changes the drawn object (identity 0.772), but swapping the part that carries the location does not move the object (location 0.241, below the random floor of 0.319). None of the latent edits in our response to Reviewer yPoF keeps both pieces as often as combining after the read-outs (joint at most 0.541 against 0.581). The category can therefore be moved inside the latent, but the location is best combined after the read-outs. Accuracy on single responses cannot show either result.

Could the author clarify why we need to introduce the recombination of the object from one image with the location from another as a new brain decoding task?

Response: Recombination is not a new task for its own sake. It tests whether a decoder reads both the object and its location from the brain response, rather than reading only the object and guessing where it is. Standard scores such as reconstruction and captioning cannot show this. They compare each output with the image the subject saw, where the object and its location always come together. A decoder can therefore score well by recognizing only the object and letting the generator draw it at a typical place [5]. Recombination asks for a pair that the subject never saw, the object of one image at the location of another. To pass, a decoder must read each piece from its own brain response, because the object no longer hints at where it should be. Such a decoder shows what the brain response contains rather than what the generator guesses. Therefore, cross-response recombination, which enables novel combinations such as placing an object at an unusual location, demonstrates that decoded object and location information can be functionally independent. In neuroscience, compositionality is also tested through the ability to form novel combinations [8], consistent with its broader definition [6, 7]. New combinations test also applies to other decoders. We trained our read-outs on six brain representations, and the rate at which recombined images keep both the object and the location differs across them. Among representations trained on subject 1 alone, it is 0.581 (ours), 0.507 (VINDEX-S1), 0.436 (MindDiffuser) and 0.395 (MindEye1). Representations trained on more subjects reach 0.584 (UMBRAE, four subjects) and 0.635 (MindEye2, eight subjects).

RepresentationTraining dataCategory top-1Box IoULocationJoint
InsideOut (Ours)subject 10.6120̲.̲4̲4̲2̲0̲.̲6̲4̲1̲0̲.̲5̲8̲1̲
MindEye1subject 10.5510.3220.4520.395
MindDiffusersubject 10.4720.3490.5090.436
VINDEX-S1subject 10̲.̲6̲2̲0̲0.3880.5550.507
UMBRAEfour subjects0.6210.4420.6440.584
MindEye2eight subjects0.6690.4820.6870.635

Bold marks the best value within each group, and underline marks the best among representations trained only on subject 1. All rows use our two read-outs trained on each representation. Category and box values therefore differ from Table 1, which reports each method's own output (for example, box IoU 0.337 for VINDEX-S1 and 0.400 for UMBRAE).

W2. Technical contribution and prior baselines

The majority of components in InsideOut are from prior work or are frozen pretrained checkpoints.

Response: We agree that InsideOut is built from existing parts, namely MindEye-style training, a frozen LLaVA with adapters, and a fixed GLIGEN. However, we believe that integrating them, each designed for different purposes, into one workflow for a new scientific question is a meaningful contribution. We put them together to read a category and a box from each brain response without giving the box read-out the category, which the recombination test needs. Three of our design choices still matter in experiments. First, combining after the read-outs works better than combining inside the latent (Table 2(a)). Second, finding the box without the category name works better than the approach of prior decoders, which are given the name, by 15% (+0.052 box IoU) for VINDEX-S1 and 10% (+0.042) for UMBRAE. Third, when the same read-outs are trained on different single-subject representations, ours gives the highest joint preservation, 15% (+0.074), 33% (+0.145) and 47% (+0.186) above VINDEX-S1, MindDiffuser and MindEye1. We will present these as design choices, separate from our main scientific findings.

UMBRAE's brain grounding did almost the same thing.

Response: UMBRAE [4] is the closest prior work, but it solves an easier problem. Every box it is scored on is requested with the true category name, for example by asking for the box of a dog. Our box read-out is only asked to find the most salient object, without any name. Even so, ours is better in two direct comparisons. First, on UMBRAE's own brain representation, our box read-out reaches a box IoU of 0.442, 10% above UMBRAE's grounding (0.400). Second, in recombination, when both boxes are rendered with the same true category so that only the box differs, ours puts the object at the target location 15% more often (0.695 against 0.603 on 874 cells, p=0.002p = 0.002).

W3. Evaluation and multiple objects

The evaluation does not focus on the tasks for which the model is built.

Response: Table 1 already evaluates both tasks directly in its last two columns, and InsideOut is best on both. On subject 1, its category top-1 accuracy is 0.612, 15% above the strongest single-subject decoder (UMBRAE-S1, 0.533), and its box IoU is 0.442, 31% above VINDEX-S1 (0.337), even though VINDEX-S1 is given the true category name and our box read-out is not. The ten caption columns serve another purpose. Prior decoders output captions rather than categories, so the captions compare them on their own output and also check that our latent keeps scene content beyond the single object, and InsideOut leads on eight of the ten caption metrics. We agree that the category and box results should be more prominent. We will move these two columns to the front of Table 1, add box accuracy at IoU 0.5 (0.468), and move the reconstructions of Figure 4(a) to the appendix.

The authors should report this [multi-label] protocol or justify reducing each scene to one object, given that 76% of test images contain more than one object.

Response: Yes, our what read-out decodes one main object per image. The cross-response test combines the category from one image with the location from another, so each image must provide one category and one location that belong to the same object. When an image contains many objects, it becomes unclear which object the category and the location refer to, which is known as the binding problem [9]. This difficulty comes from the images themselves and appears as soon as a second object is present. Even a vision-language model (LLaVA-1.5) that sees the image directly, without any brain data, locates the main object less accurately as objects are added (box IoU 0.479, 0.429, 0.379 and 0.219 for one, two, three or four, and five or more objects). We therefore evaluate one object per image, so that the test focuses on recombination itself rather than on deciding which object is meant. Restricting the read-out to one object does not limit what the latent encodes.

On the standard multi-label evaluation, a simple linear classifier trained on our latent to name every object category in the image reaches an mAP of 0.417, an AUC of 0.921 and a Hamming loss of 0.026 over 80 COCO categories (mean of four subjects). Its mAP is 35% higher than that of iMIND [1] with fMRI only (0.310, 0.913, 0.027) and 62% higher than that of CLIP-MUSED [10] (0.258, 0.877, 0.030), the two methods the reviewer cites, although the protocols differ. We acknowledge this limitation, will state it clearly in the paper, and leave multi-object grounding and recombination to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.