acceptodds
Under review as a conference paper at ICLR 2027

HemiPatchGate: Probing Content- and Location-Dependent Hemispheric Differences with fMRI-Based Visual Decoding

Abstract

How visual information from natural scenes is represented across the cerebral hemispheres is a key question about the organization of the human visual system. A fundamental feature of this organization is contralateral preference: each hemisphere preferentially represents the opposite visual field. Beyond this contralateral preference, do visual representations show systematic hemispheric differences associated with scene content and spatial location? To address this question, we introduce HemiPatchGate, which separately encodes left- and right-hemisphere functional magnetic resonance imaging (fMRI) responses and uses gates to assign relative weights to their representations at each image-patch location for visual feature prediction. Across four subjects in the Natural Scenes Dataset, the gating weights recover a contralateral pattern that is consistent across natural images without explicit supervision of the visual-field–hemisphere relationship. Beyond this consistent contralateral pattern, greater image coverage by people and animals is associated with whole-image gate shifts toward the right-hemisphere branch. At the local level, increasing foreground coverage is associated with a greater tendency for weighting to shift toward the ipsilateral branch near the image's vertical midline than elsewhere. Changes in prediction error after zeroing either hemispheric input further support the contralateral pattern and the more pronounced foreground-related tendency toward ipsilateral shifts near the midline.

Primary area:
applications to neuroscience & cognitive science
Keywords:
fMRI-based visual decoding, hemispheric differences, visual neuroscience, natural scenes, patchwise gating

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_kEvfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies differences in visual representations between the hemispheres using fMRI responses to natural images. The authors introduce HemiPatchGate, which separately encodes responses from each hemisphere and uses gating weights to combine their representations for predicting CLIP image features at different patch locations. They evaluate the method on four NSD subjects and use hemisphere input zeroing and retraining controls to examine the gating readout. They recover the known contralateral pattern and report that greater coverage by people and animals is associated with a shift toward the right hemisphere branch. They also find that local foreground coverage is associated with a stronger shift toward ipsilateral weighting near the vertical midline than elsewhere.

Critical strengths
  1. Interesting motivation. Developing interpretable models to investigate differences in visual representations between the hemispheres is an interesting research direction.

  2. Clear presentation. The paper is well written and clearly motivates its research question.

Critical weaknesses
  1. The methodological contribution is not sufficiently justified.

There are significant flaws in the motivation and ablations of the proposed methods. The justification for the specific architecture over simpler alternatives is lacking, making the claim that it constitutes a contribution unconvincing. Training a separate decoder for each hemisphere is a simple baseline that, in my opinion, is cleaner and allows better disentanglement for this research question (see Question 1). It is unclear why the proposed complex method is needed. Moreover, the current method might entangle multiple factors influencing the presented results. The interpretation of gating weights also appears stronger than the evidence supports. These weights do not necessarily quantify the relative strength of information available from each hemisphere and could be affected by differences in signal quality or optimization (Questions 2). Residual prediction experiments would provide a much stronger and more informative analysis, explicitly showing what each hemisphere contributes beyond the other (Questions 3).


  1. Unclear and limited scientific contribution.

Contralateral organization is well established, and it is unclear why so much space is devoted to recovering this known pattern. The animacy finding needs to be distinguished from the established right hemisphere bias for faces, which are included in the animate content analyzed here. Likewise, the foreground related ipsilateral shift near the vertical midline could reflect known receptive field overlap and greater contributions from object responsive areas with broader receptive fields (Question 5). The paper examines only a small number of possible effects and presents limited quantitative and qualitative results characterizing them. Overall, the paper does not clearly establish what new understanding these findings provide beyond existing knowledge.


Overall Assessment. I consider this paper well below the acceptance threshold for the conference. Addressing these critical weaknesses would require significant changes to the paper that are beyond the scope of rebuttal in my opinion.

Questions for authors
  1. Comparison with separate decoders. Why use the proposed architecture rather than train a separate decoder for each hemisphere? This seems a simpler and potentially stronger baseline. What additional insight does gating provide?

  2. Interpretation of gating magnitude. How do gate weights quantify the relative strength of hemispheric information? Could small differences in signal quality or optimization lead one branch to receive most of the weight, without a correspondingly large difference in available information?

  3. Residual prediction. Why not train a decoder on one hemisphere and then measure what the other hemisphere can additionally predict about its residual errors? This could clarify the additional predictive information contributed by each hemisphere.

  4. Signal quality and dataset biases. Can you analyze differences in signal to noise ratio between hemispheres and across image categories, including animate versus other images? How do you distinguish their influence on decoder performance and gating from the representational differences of interest?

  5. Relation to established findings. How do the reported effects extend known right hemisphere face selectivity (Kanwisher et al., 1997) and ipsilateral receptive field coverage near the vertical meridian and in higher visual areas (Tootell et al., 1998; Dumoulin and Wandell, 2008)? Analyses separating face content from other animate content, and accounting for regions with different receptive field properties, are needed to clarify the additional scientific insight.

Reviewer_8atVOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether visual representations across the cerebral hemispheres exhibit content- and location-dependent differences beyond well-established contralateral spatial organization. Standard fMRI decoders merge inputs from both hemispheres prior to feature prediction, precluding fine-grained comparison of each hemisphere's local contribution to a given visual prediction. This is done by encoding the left- and right-hemisphere fMRI responses from the Natural Scenes Dataset (NSD) through independent Transformer branches. At each position in a 16×1616 \times 16 spatial grid, a learned, channel-wise sigmoid gate weights the two hemispheric representations before concatenation and prediction of OpenCLIP ViT-bigG patch tokens. The resulting gate values serve as a spatially resolved readout of relative hemispheric weighting.

Across all four NSD subjects, HemiPatchGate achieves feature retrieval performance on par with an ungated baseline (Top-1 54.0% vs. 53.3%) while providing a spatially resolved readout of hemispheric weighting. Without explicit spatial supervision, the model spontaneously recovers the classic contralateral visual-field organization across natural images, favoring the left branch for the right image half and the right branch for the left image half. Beyond this baseline spatial arrangement, scene content and spatial location dynamically modulate gating: images with higher animate coverage (people and animals) induce a whole-image shift toward the right-hemisphere branch.

Critical strengths
  • Combining separate hemispheric branches with patch-wise gating yields an intuitive spatial readout. The architecture recovers contralateral organization in every subject without explicit spatial supervision and without sacrificing retrieval accuracy.
Critical weaknesses
  • Gating is unnecessary for feature prediction. An ungated baseline achieves equivalent Top-1 retrieval (53.3% vs. 54.0%) and reaches lower validation loss across all four subjects (Table 6). The spatial asymmetry is also small.

  • A major component of the gating map is independent of fMRI inputs. Bilateral input zeroing leaves 27.3%–68.0% of the contralateral gating pattern intact (Table 10). Swapping left and right fMRI inputs reduces the pattern below 10% but leaves Δ‾L\overline{\Delta}^L positive rather than reversing it. This indicates that much of the readout reflects an input-independent spatial prior learned during training rather than dynamic brain decoding.

  • One missing critical baseline is having decoders trained exclusively on left-hemisphere or right-hemisphere fMRI inputs (using the same architecture or ridge regression). Comparing patch-wise prediction accuracy across image halves, near the vertical midline, and across animate coverage levels would directly verify whether informational differences exist between hemispheres without gating confounds.

  • All subject models are trained using a single random seed (seed 42), making it impossible to determine whether gate maps are stable across network initializations or even to establish a statistical test.

Minor Issues:

  • Using five overlapping gating metrics (gating difference, gate shift, gating-change magnitude, ipsilateral gate shift, zeroing contrast) makes the paper difficult to follow.

  • In table 2, zeroing contrasts are reported to six decimal places without accompanying confidence intervals or error bounds

Questions for authors
  • NSD provides subject-specific pRF mapping and eye-tracking data, could that be incorporated in the pipeline to establish personalized spatial predictions or control for eye movements?

  • Do decoders trained on only one hemisphere reproduce the contralateral, animate, and midline patterns in patch-wise feature prediction accuracy?

  • How many vertex groups per ROI category are assigned to each hemisphere? Does the animate effect persist when allocating equal numbers of vertex groups to both hemispheric branches?

Reviewer_Bf95OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study trains a decoder with separate left- and right-hemisphere branches and a patch-wise gate between them, and uses the gate weights to study hemispheric differences in NSD.

Critical strengths

The question of how content and location shape hemispheric differences in natural scene processing is interesting, and the paper includes the controls.

Critical weaknesses

The questions asked here (contralateral organization, content-related lateralization, effects near the vertical midline) can be and have been addressed much more directly with voxel-wise encoding models that include or learn spatial receptive fields. Those models make claims about the voxels themselves, and not about the internal weights of a decoder that are influences by other factors. The gate weights depend on the architecture, the losses and the CLIP targets, and it is not clear what they reveal about the brain that an encoding analysis would not.

Overall it is not well moivated why this is the right approach.

Questions for authors

see above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.