Compositional Brain Decoding via Visual Constituent Recombination

Recent advances in fMRI visual decoding recover increasingly rich semantic and perceptual information, yet it remains unclear whether broadly overlapping object and spatial information can be recovered as independently usable constituents rather than only within their original associations. We propose InsideOut, a framework for compositional brain decoding that operationalizes object category and spatial location as what and where constituents. InsideOut explicitly decodes these constituents and recombines them across responses elicited by different images to test whether each preserves its source-specific information after its original association is broken. In the subject-1 composition evaluation, 58.1% of cross-response compositions preserve both category and location, and read-out-level recombination outperforms voxel- and latent-level alternatives. Together, these findings provide a compositional view of brain decoding in which distinct visual constituents can be separately recovered from brain responses and remain reusable beyond the associations in which they were originally observed.