acceptodds
Under review as a conference paper at ICLR 2027

Bridging the Hierarchies: Neuro-Inspired Hierarchy-Aware Alignment for Visual Neural Decoding

Abstract

Visual neural decoding aims to retrieve perceived visual stimuli from neural signals and is an important research direction in brain–computer interfaces. Although EEG- and MEG-based methods have made promising progress, they insufficiently account for the neuroscientific priors that visual processing is hierarchically organized and information propagates progressively along cortical pathways. Two key limitations remain: I) incomplete inter-hierarchy correspondence, as existing methods associate neural and visual representations at a selected semantic level, leaving correspondence between the two representational hierarchies as a whole insufficiently captured; and II) insufficient modeling of intra-hierarchy propagation, as existing alignment strategies typically treat semantic levels in isolation, without modeling progressive information propagation across successive levels within each hierarchy. We propose Neuro-Inspired Hierarchy-Aware Alignment (NIA) to jointly address these limitations. NIA integrates a Hierarchy-Partitioned Channel Transformer (HPCT), Hierarchy-Matched Contrastive Learning (HMCL), and a Parallel Dual-Hierarchy Cascade (PDHC) to organize neural representations into a hierarchy, establish structured correspondence between the neural and visual representational hierarchies, and model progressive shallow-to-deep propagation within both hierarchies. Experiments on THINGS-EEG2 and THINGS-MEG demonstrate state-of-the-art zero-shot brain-to-image retrieval. On THINGS-EEG2, NIA improves Top-1/Top-5 accuracy by 9.5/1.3 and 9.9/10.0 percentage points under the intra- and inter-subject settings, respectively, validating the effectiveness of inter-hierarchy correspondence and intra-hierarchy progressive information propagation.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Visual Neural Decoding, Brain-Computer Interface, EEG, MEG, Hierarchical Alignment, Contrastive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4mtNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This research proposes a new algorithm to align neural and visual representations. Specifically, the authors organize neural representations into three levels using the Hierarchy-Partitioned Channel Transformer (HPCT) and use the Hierarchy-Matched Contrastive Learning (HMCL) loss to align them with representations from three depths of a visual encoder. To model relationships within each hierarchy, the authors propose the Parallel Dual-Hierarchy Cascade (PDHC) module to improve information flow across the three levels. Experiments were conducted on representative datasets in this field, and the average scores were higher than those of recent baselines.

Critical strengths
  1. This paper reports strong retrieval results on two representative datasets, covering both EEG and MEG as well as intra-subject and inter-subject settings.

  2. The authors provide extensive experiments and supplementary information, including component ablations, subject-wise results and evaluations with different neural and visual encoders. These experiments provide useful evidence for the empirical effectiveness and robustness of the proposed method.

Critical weaknesses
  1. Some comparisons with the strongest related methods require clarification. Table 2 omits BrainHIVE’s reported inter-subject MEG result, even though its Top-5 accuracy is higher than that of the proposed method. This result presentation make it difficult to assess the exact performance improvement.

  2. The fairness of the comparisons is still limited. Some results in the main tables are taken from different papers that use different visual encoders, signal preprocessing procedures, and training configurations. This paper uses the strong InternViT-6B visual encoder, while the controlled reproductions in the appendix are mainly limited to the EEG intra-subject setting. Therefore, it remains difficult to determine how much of the improvement comes from the proposed hierarchical alignment method and how much comes from stronger visual representations or different experimental settings.

  3. This three representation levels design appears somewhat heuristic. The shallow, intermediate, and deep visual features are selected from three approximately evenly spaced layers, but the paper provides limited justification for why these layers should correspond to meaningful levels of neural processing. The improvement seems to lack sufficient theoretical support or representational analysis.

  4. Existing ablations do not fully isolate the contribution of PDHC. In particular, the paper does not report results when PDHC is applied only to the neural hierarchy or only to the visual hierarchy. It remains unclear whether propagation on both sides is necessary or whether most of the improvement comes from only one of the two cascades.

Questions for authors
  1. To assess PDHC’s independent contribution, the authors could add more ablation studies, such as controls that apply PDHC only to the neural part or only to the visual part. The current ablations do not establish whether propagation in both hierarchies is necessary. 2.Table 2 reports both intra-subject and inter-subject results but omits BrainHIVE’s inter-subject MEG results. BrainHIVE reports Top-1/Top-5 accuracies of 5.4%/15.2%, compared with 6.2%/14.2% for this paper. Therefore, this paper’s claim of achieving the highest average accuracy on THINGS-MEG does not seem entirely reasonable. Is there any reason why this baseline was omitted.
  2. This is a neuroscience-inspired model, but the current analyses are mainly empirical. The authors could add more representational analyses or further examine whether the shallow, intermediate, and deep branches correspond to different time windows or levels of visual information.
Reviewer_amxtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NIA, a hierarchical alignment framework for zero-shot retrieval of images from brain signals. By modeling the hierarchical correspondences between neural representations and visual representations, the framework enhances retrieval performance. Achieving state-of-the-art (SOTA) results, this approach suggests that hierarchical alignment and cross-layer information interaction may facilitate the decoding of brain signals. However, the connection between the neuroscientific motivation and the actual network design remains insufficiently established; features from different layers of a neural network cannot be directly equated to distinct stages of visual processing in the human brain, nor do the artificial hierarchies within the model necessarily correspond to the actual hierarchical structure of neural representations.

Critical strengths

For the task of brain-to-visual reconstruction, the authors address both cross-layer correspondences and information propagation within individual layers. They achieved state-of-the-art results in both intra-subject and inter-subject experiments.

Critical weaknesses

The authors draw upon the hierarchical structure of visual encoders rather than the actual visual processing mechanisms of the human brain. Furthermore, the use of hierarchical structures in visual encoders is an approach that has already been explored in existing literature.

Questions for authors

Please refer to the major weakness

Reviewer_J589OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a hierarchical alignment framework (NIA) for EEG/MEG-based visual neural decoding, which achieves hierarchical alignment between neural signals and visual representations through attention-head grouping, multi-level contrastive learning, and gated feature propagation. However, the innovation of this approach is limited, as its core concepts overlap significantly with existing methods for hierarchical representation learning and cross-modal alignment. Furthermore, the interpretation of attention-head grouping as a hierarchy of neural representations lacks a robust neuroscientific basis; additionally, the experimental results fail to effectively distinguish the performance gains attributed to the proposed hierarchical propagation mechanism from those resulting from conventional feature fusion or multi-level supervision.

Critical strengths

The experiment achieved good zero-shot image retrieval performance on the THINGS-EEG2 and THINGS-MEG data sets.

Critical weaknesses

1, the research lacks sufficient innovation to justify the authors' strong claims regarding two identified "research gaps." The authors argue that existing literature has not adequately addressed "neural–visual hierarchical correspondence" and "progressive information propagation"; however, both concepts have already been explored in prior research. 2, the partitioning of attention heads does not constitute proof that the model has learned a genuine "neural hierarchy." Since these sets of attention heads receive identical input tokens and operate within the same attention mechanism—without being structured across varying network depths to represent shallow, intermediate, or deep layers—they do not inherently correspond to distinct brain regions or specific temporal stages of neural response. Consequently, such grouping does not naturally yield a hierarchical structure that aligns with the brain’s visual processing mechanisms. 3, the ablation studies fail to definitively rule out performance gains attributable to "multi-layer supervision" and "ensemble learning." The observed improvements may stem from multi-task supervision, representation diversity, or ensemble effects—suggesting that predictions across multiple layers are complementary. This could simply be interpreted as an ensemble benefit derived from averaging retrieval scores across different feature spaces, which is not equivalent to the "hierarchical correspondence" mechanism claimed by the authors. 4, the reported "neural decoding" capability is heavily reliant on extensive "trial averaging." The 200-way retrieval accuracy presented in the paper is not derived from a single EEG/MEG measurement; rather, it relies on averaging across 80 trials—a methodology that is neither compelling nor practical for real-world applications.

Questions for authors

1, the research lacks sufficient innovation to justify the authors' strong claims regarding two identified "research gaps." The authors argue that existing literature has not adequately addressed "neural–visual hierarchical correspondence" and "progressive information propagation"; however, both concepts have already been explored in prior research. 2, the partitioning of attention heads does not constitute proof that the model has learned a genuine "neural hierarchy." Since these sets of attention heads receive identical input tokens and operate within the same attention mechanism—without being structured across varying network depths to represent shallow, intermediate, or deep layers—they do not inherently correspond to distinct brain regions or specific temporal stages of neural response. Consequently, such grouping does not naturally yield a hierarchical structure that aligns with the brain’s visual processing mechanisms. 3, the ablation studies fail to definitively rule out performance gains attributable to "multi-layer supervision" and "ensemble learning." The observed improvements may stem from multi-task supervision, representation diversity, or ensemble effects—suggesting that predictions across multiple layers are complementary. This could simply be interpreted as an ensemble benefit derived from averaging retrieval scores across different feature spaces, which is not equivalent to the "hierarchical correspondence" mechanism claimed by the authors. 4, the reported "neural decoding" capability is heavily reliant on extensive "trial averaging." The 200-way retrieval accuracy presented in the paper is not derived from a single EEG/MEG measurement; rather, it relies on averaging across 80 trials—a methodology that is neither compelling nor practical for real-world applications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.