HemiPatchGate: Probing Content- and Location-Dependent Hemispheric Differences with fMRI-Based Visual Decoding

How visual information from natural scenes is represented across the cerebral hemispheres is a key question about the organization of the human visual system. A fundamental feature of this organization is contralateral preference: each hemisphere preferentially represents the opposite visual field. Beyond this contralateral preference, do visual representations show systematic hemispheric differences associated with scene content and spatial location? To address this question, we introduce HemiPatchGate, which separately encodes left- and right-hemisphere functional magnetic resonance imaging (fMRI) responses and uses gates to assign relative weights to their representations at each image-patch location for visual feature prediction. Across four subjects in the Natural Scenes Dataset, the gating weights recover a contralateral pattern that is consistent across natural images without explicit supervision of the visual-field–hemisphere relationship. Beyond this consistent contralateral pattern, greater image coverage by people and animals is associated with whole-image gate shifts toward the right-hemisphere branch. At the local level, increasing foreground coverage is associated with a greater tendency for weighting to shift toward the ipsilateral branch near the image's vertical midline than elsewhere. Changes in prediction error after zeroing either hemispheric input further support the contralateral pattern and the more pronounced foreground-related tendency toward ipsilateral shifts near the midline.