- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This research proposes a new algorithm to align neural and visual representations. Specifically, the authors organize neural representations into three levels using the Hierarchy-Partitioned Channel Transformer (HPCT) and use the Hierarchy-Matched Contrastive Learning (HMCL) loss to align them with representations from three depths of a visual encoder. To model relationships within each hierarchy, the authors propose the Parallel Dual-Hierarchy Cascade (PDHC) module to improve information flow across the three levels. Experiments were conducted on representative datasets in this field, and the average scores were higher than those of recent baselines.
Critical strengths
-
This paper reports strong retrieval results on two representative datasets, covering both EEG and MEG as well as intra-subject and inter-subject settings.
-
The authors provide extensive experiments and supplementary information, including component ablations, subject-wise results and evaluations with different neural and visual encoders. These experiments provide useful evidence for the empirical effectiveness and robustness of the proposed method.
Critical weaknesses
-
Some comparisons with the strongest related methods require clarification. Table 2 omits BrainHIVE’s reported inter-subject MEG result, even though its Top-5 accuracy is higher than that of the proposed method. This result presentation make it difficult to assess the exact performance improvement.
-
The fairness of the comparisons is still limited. Some results in the main tables are taken from different papers that use different visual encoders, signal preprocessing procedures, and training configurations. This paper uses the strong InternViT-6B visual encoder, while the controlled reproductions in the appendix are mainly limited to the EEG intra-subject setting. Therefore, it remains difficult to determine how much of the improvement comes from the proposed hierarchical alignment method and how much comes from stronger visual representations or different experimental settings.
-
This three representation levels design appears somewhat heuristic. The shallow, intermediate, and deep visual features are selected from three approximately evenly spaced layers, but the paper provides limited justification for why these layers should correspond to meaningful levels of neural processing. The improvement seems to lack sufficient theoretical support or representational analysis.
-
Existing ablations do not fully isolate the contribution of PDHC. In particular, the paper does not report results when PDHC is applied only to the neural hierarchy or only to the visual hierarchy. It remains unclear whether propagation on both sides is necessary or whether most of the improvement comes from only one of the two cascades.
Questions for authors
- To assess PDHC’s independent contribution, the authors could add more ablation studies, such as controls that apply PDHC only to the neural part or only to the visual part. The current ablations do not establish whether propagation in both hierarchies is necessary. 2.Table 2 reports both intra-subject and inter-subject results but omits BrainHIVE’s inter-subject MEG results. BrainHIVE reports Top-1/Top-5 accuracies of 5.4%/15.2%, compared with 6.2%/14.2% for this paper. Therefore, this paper’s claim of achieving the highest average accuracy on THINGS-MEG does not seem entirely reasonable. Is there any reason why this baseline was omitted.
- This is a neuroscience-inspired model, but the current analyses are mainly empirical. The authors could add more representational analyses or further examine whether the shallow, intermediate, and deep branches correspond to different time windows or levels of visual information.