acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Vision-Brain Alignment via Layer-Sensitive Temporal Dynamics

Abstract

Continuous neural recordings provide rich temporal dynamics for visual decoding, yet conventional cross-modal alignment methods typically aggregate time-series neural signals into static feature vectors. Such static temporal aggregation neglects the temporal progression inherent in visual cognitive processing, thereby limiting the explicit separation between low-level sensory details and high-level semantics. To address this limitation, we propose Layer-Sensitive Temporal Alignment (LSTA), a progressive framework designed to model continuous temporal dynamics without static feature compression. By structuring neural recordings into a holistic representation alongside early stimulus-evoked and late residual-semantic branches, the framework effectively decouples global context from time-varying perceptual signals. LSTA couples neural response latency with visual hierarchy depth, systematically aligning early perceptual dynamics with shallow visual layers and late cognitive responses with deeper representations. Extensive experiments demonstrate the effectiveness of LSTA, achieving state-of-the-art performance in zero-shot vision-brain retrieval tasks, with Top-1 and Top-5 accuracies reaching 91.4% and 98.6% on the THINGS-EEG dataset, and 42.7% and 67.4% on the THINGS-MEG dataset, respectively.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain-Computer Interface, Cross-Modal Contrastive Learning, Visual Neural Decoding, EEG Signal Processing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NRYcOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

LSTA addresses zero-shot brain–visual retrieval by encoding holistic, stimulus-period, and post-stimulus neural responses into separate embeddings and aligning them with multiple layers of a pretrained visual backbone. The neural encoders combine graph-based channel processing with an inverted Transformer, and a Gaussian branch-to-layer prior weights the symmetric contrastive losses and retrieval scores. The manuscript reports within-subject Top-1 accuracies of 91.4% on THINGS-EEG and 42.7% on THINGS-MEG, with additional LOSO results and branch, encoder, visual-backbone, and weighting-strategy analyses.

This assessment is based on the 17-page manuscript and the nine supplied source/configuration/documentation files. Static inspection and limited synthetic function checks were performed, not an end-to-end reproduction. The README identifies this as a partial core-code release; the findings below distinguish behavior of the documented entry point from the unverified provenance of the manuscript's numerical results.

Critical strengths

1. A concrete, task-motivated alignment design. Organizing neural responses into temporal views and connecting those views to visual depth is an intelligible inductive bias. The supplied implementation clarifies the independent branch encoders, per-layer visual projections, and weighted pairwise symmetric contrastive objective. The latter is consistent with Eqs. (5)–(6), rather than an unspecified new loss.

2. Relevant empirical coverage and alternative controls. The manuscript reports EEG and MEG experiments, within-subject and LOSO settings, and per-subject results averaged over five random-seed runs. Appendix C includes uniform and learned-attention alternatives, not just a reversed prior, and Figure 5 tests alternative neural and visual backbones. These are useful design checks. However, their evidential value depends on resolving the implementation and evaluation issues below.

Critical weaknesses

1. The documented evaluation selects checkpoints on the designated test partition and computes the reverse retrieval direction, so it does not establish the claimed independently evaluated EEG-to-image performance.

Test-set-driven model selection. config.py specifies separate training.pt and test.pt EEG paths, corresponding training/test image-feature caches, 200 epochs, and evaluation frequency pk=1 (lines 16–25). train.py constructs test_loader directly from these test paths (lines 109–139), evaluates it after every configured epoch, and saves the checkpoint whenever that test Top-1 improves (lines 217–228). It then returns and writes the selected test scores to CSV and averages these per-subject maxima (lines 237–281). No independent validation split or subsequent untouched-test evaluation is present in this entry point. This is test-partition feedback for model selection, not merely test monitoring. Withholding test concepts from gradient updates does not make a repeatedly test-selected score an independent estimate. The inspection does not show test samples in gradient training, and the size of any selection optimism is unknown; nevertheless, the supplied workflow cannot substantiate the stated evaluation without a validation-selected or prospectively fixed selection protocol and an independent final test.

Retrieval direction. In train.py:evaluate, the actual operations are:

weighted_eeg = torch.einsum("nid, ij -> njd", eeg_feats_norm, w_matrix)
sim_matrix = torch.einsum("mjd, njd -> mn", proj_img_feats_norm, weighted_eeg)

Here m indexes images and n indexes EEG samples. Applying topk(..., dim=-1) therefore retrieves EEG candidates for each image, not image candidates for each EEG query (lines 49–68). The logger is named TopK_I2E, and the final printout explicitly calls the result Best Image->EEG (lines 82–84 and 237–239). Conversely, Eq. (3) describes an EEG-by-image matrix, and Figure 10 explicitly describes EEG-feature queries for image retrieval. The two retrieval directions need not have the same Top-1/Top-5, even under a symmetric training loss. Computing EEG-to-image retrieval requires ranking image candidates in the transposed orientation of this supplied score matrix. I cannot conclude that 91.4% is necessarily the reverse-direction value, because the exact result-producing snapshot and outputs were not supplied. The discrepancy must be reconciled rather than assumed harmless or assumed to explain a particular amount of inflation.

Result provenance and scope. The loader pairs EEG rows with sorted image-feature keys and checks only sample counts (Datasets.py, lines 55–63 and 82–95); this may be correct for appropriately ordered caches, but does not verify image/concept identity or split disjointness. Explicit IDs and cache-generation manifests are needed. The supplied driver covers per-subject EEG training with 17 named EEG channels and one configured seed; it does not implement the reported MEG/LOSO/five-run orchestration. The README's explicit partial-release statement should be respected: this is missing verification coverage, not proof that those experiments were never performed. There are also baseline-reporting inconsistencies in Table 3: its displayed ATM-S subject values average to 27.10%/58.08%, not 28.5%/60.4%; HCF Top-1 averages to 84.08%, not 84.6%; and HVF Top-5 averages to 94.91%, not 94.6%. LSTA's own displayed subject averages are compatible with its reported means. These baseline discrepancies require source/configuration reconciliation and do not establish fabrication.

2. The supplied model/configuration does not implement several central descriptions, and the distinctive contribution remains insufficiently separated from generic multi-branch/multi-layer learning.

The default branch–prior mapping contradicts Eq. (4). model.py stacks features in the order [f_hol, f_stim, f_res] (lines 259–271). The configuration supplies mu=[3,5,8], and the weight builder and loss preserve that row order (config.py, lines 52–53; model.py, lines 287–300; loss.py, lines 46–57). Thus the supplied configuration assigns zero-based centers mu_hol=3, mu_stim=5, and mu_res=8, whereas the manuscript requires mu_stim < mu_hol < mu_res. The first two branch assignments are inconsistent with the stated ordering. The weights are also rounded to two decimals and left unnormalized in the implementation; their exact construction and scaling should be controlled in weighting ablations. A different experimental configuration may resolve the ordering issue, but the authors must identify the configuration and checkpoints that generated each principal table.

The implementation produces static branch summaries, not an explicit continuous temporal alignment. iTransformer.value_embedding maps each channel's complete time vector to a feature vector, the Transformer attends over the channel-token axis, and EEG_Encoder.forward averages that axis with x.mean(dim=1) (model.py, lines 145–185). Three independent encoders and projection heads then yield a tensor of shape [B,3,1024] under the supplied configuration. This can encode temporal patterns, but there is no time-indexed representation at the alignment interface or learned trajectory-to-depth correspondence supporting the Abstract/Introduction's claim of preserving continuous trajectories without static compression. Likewise, learnable_adj is a global parameter and the graph support is fixed and fully connected; GAT attention may still be input-dependent, so this does not imply that all channel interactions are static. It does mean that the claimed dynamic functional connectivity is not established by the global adjacency parameter alone (model.py, lines 70–100 and 124–141). The defensible formulation is structured temporal-view alignment unless an additional, currently undescribed mechanism is provided.

Contribution attribution and paper–code correspondence. Table 6 reports 90.1% Top-1/98.9% Top-5 for uniform weights, versus 91.4%/98.6% for the normal Gaussian prior. Figure 5 reports 90.4% Top-1 with an ATM encoder, versus 91.4% for the proposed encoder. These differences may be meaningful, but need paired uncertainty and matched capacity/optimization rather than being treated as proof that the specific mechanisms explain the overall high accuracy. The code confirms that the three encoders are independent, strengthening the need for a capacity-matched multi-head full-window control. The reversed-prior experiment is also confounded: Appendix C moves the holistic center to the visual-depth extremes as well as swapping stimulus and residual centers. Even the holistic-only model drops from 87.4% to 69.2% under that change, despite having no early/late branch relation to reverse. A control that leaves the holistic prior fixed is necessary to isolate the proposed temporal-depth correspondence.

The training script additionally enables random repeat-subset averaging, Gaussian noise, time shifts, channel dropout, smoothing, and frequency masking (train.py, lines 95–101; eeg_augmentation.py; config.py, lines 27–38). These are not specified in the manuscript's training/preprocessing account, and their use in matched baselines and ablations is unclear. The configured time_splite is not forwarded to the model constructor, which retains its default 0.2 (train.py, lines 143–154; model.py, lines 201–208); the actual MEG implementation of the stated 500 ms boundary therefore also needs to be supplied. These gaps prevent an unambiguous attribution of gains to the highlighted architecture and prior. More broadly, fixed stimulus-offset segmentation and branch-fusion accuracy gains do not independently establish perceptual–semantic disentanglement or biological inter-region cooperation.

Overall assessment. I recommend rejection in the current form. The decisive issues are the test-selected evaluation, the retrieval-direction discrepancy, and the mismatch between the specified model/prior and the submitted implementation—not the absolute magnitude of the reported accuracy or the mere reuse of standard modules. These are technical soundness and evidential concerns, not an allegation of intentional misconduct. A result-matched implementation and corrected, independently evaluated comparisons could materially change the assessment.

Questions for authors

1. Which retrieval direction generated the main tables? Please provide the exact evaluation function, checkpoint IDs, and image/EEG IDs for Tables 1, 3, and 4. For the same frozen checkpoints, report both EEG-to-image and image-to-EEG Top-1/Top-5, clearly defining the query and candidate axes. If the submitted evaluate() is not the result-producing code, please identify the corresponding version.

2. How was the final test kept independent of model selection? The supplied entry point selects and reports the maximum test score over 200 epochs. Was a separate validation protocol used for the manuscript? Please provide its split manifest and selection logs, or recompute the comparisons with validation-selected or prospectively fixed checkpoints and an independent final evaluation. Selecting a new maximum on the same test set after correcting the retrieval direction would not address this concern.

3. Which branch order and Gaussian centers were actually used? Please reconcile the code order [hol, stim, res] and centers [3,5,8] with mu_stim < mu_hol < mu_res. Identify the numerical weights, rounding/normalization, and saved configurations for the main and reversed-prior experiments. What operation, beyond static branch embeddings, supports the claimed continuous trajectory-preserving alignment?

4. Do the specific mechanisms outperform matched alternatives? Please provide paired subject- and run-level changes for normal Gaussian versus uniform/learned weights, GAT versus no GAT, and temporal branches versus a capacity-matched full-window multi-head model. Keep augmentation, visual features, training budget, and loss scaling matched. A stimulus/residual swap that leaves the holistic prior unchanged would directly test the claimed correspondence.

5. Can the reported experiments be traced to complete configurations and outputs? Please identify the image-feature extraction and EEG preprocessing steps, image/concept ordering checks, MEG and LOSO drivers, the passed stimulus-offset parameters, and five-seed result aggregation. Reconcile the baseline averages in Table 3 and distinguish quoted values from reruns. These requests target the reported experiments, not a new dataset or clinical study.

1 comment

We thank the reviewer for identifying inconsistencies between the submitted code and the manuscript, and for highlighting experimental details that require clarification. We have corrected the relevant materials and respond to each question below.

1. Retrieval direction and consistency between the evaluation code and manuscript

We apologize for the retrieval-direction error introduced while organizing the code for submission. The experiments reported in the paper use EEG-to-image retrieval, with EEG representations as queries and images as candidates, consistent with the manuscript. We have corrected the corresponding evaluation implementation in the submitted code.

2. Test-set independence and checkpoint selection

The checkpoint selection in the original code was intended to maintain a consistent reporting protocol with the comparison methods. To examine whether the performance advantage depends on checkpoint selection, Table 7 additionally reports evaluation using only the last-epoch checkpoint of each subject-specific model, without selecting the best epoch based on test accuracy. We repeat this experiment with five random seeds and report the mean and standard deviation. Under this setting, the model still outperforms the listed comparison methods, indicating that its advantage is not solely attributable to best-checkpoint selection.

3. Branch order and Gaussian prior centers

The branch order used in the experiments is [HOL, STIM, RES], and the corresponding Gaussian prior centers are [5, 3, 8]. Thus, the STIM center is shallower than the HOL center, which is in turn shallower than the RES center, as specified in the manuscript. The submitted configuration incorrectly listed [3, 5, 8] due to an error during code organization. This has now been corrected. We apologize for the confusion caused by this inconsistency.

4. Controlled comparisons of weighting strategies, GAT, and temporal–hierarchical alignment

We have provided the following comparisons to examine the contributions of the components and alignment strategies:

  • Table 8 compares uniform weighting, learnable weighting, and the normal Gaussian prior.
  • Table 9 includes the performance obtained after removing GAT, quantifying its contribution.
  • Table 8 adds the Constrained Reversed Gaussian Prior, which keeps the HOL prior fixed and swaps only the STIM and RES prior centers. This comparison more directly examines the correspondence between the temporal branches and visual hierarchy.

5. Experimental traceability, data pairing, MEG/LOSO settings, and numerical corrections

We have supplemented Appendix A with the image-feature extraction procedure and the preprocessing steps for EEG, MEG, and the other data.

For data pairing, image-feature keys are sorted lexicographically, and EEG samples are paired with the corresponding image features by row index. This procedure assumes that the EEG samples were prepared in the same order. The code checks sample-count consistency. The image keys are used only for data pairing and are not supplied to the model as inputs or as additional supervision.

The MEG and LOSO experiments share the existing LSTA implementation. MEG uses the corresponding data and configuration. In LOSO, one subject is held out for testing and the training data of the remaining subjects are used for model training. No additional optimization or adaptation is performed on the held-out subject. Subject IDs are used only for data partitioning and are not provided as model inputs.

Table 7 reports results across five random seeds together with their standard deviations. We have also checked and corrected the identified numerical inconsistencies, including the transcription error for HCF Subject 9 and the inconsistencies in the reported baseline averages.

We thank the reviewer again for the feedback, which helped us improve the consistency between the manuscript and the submitted materials and clarify the experimental procedures.

Reviewer_ZnMBOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a Layer-Sensitive Temporal Alignment (LSTA) framework to address the insufficient exploitation of neural temporal dynamics in existing visual decoding methods. By introducing holistic, stimulus-evoked, and post-stimulus neural response branches, together with a structured alignment prior between neural response latency and visual feature hierarchy, the proposed framework jointly models neural information across multiple temporal stages and hierarchical visual representations. The research problem is meaningful, and the methodological motivation is clear. The proposed approach demonstrates strong performance in both intra-subject and inter-subject retrieval tasks on EEG data.

However, the main methodological contribution primarily lies in the structural integration of existing neural encoders, multi-layer visual features, and prior-weighted contrastive learning, rather than introducing a fundamentally new mechanism for modeling continuous neural dynamics. Moreover, related questions concerning the temporal progression of visual representations have already been investigated in previous studies, such as A Large and Rich EEG Dataset for Modeling Human Visual Object Recognition. The ablation results also indicate that the holistic branch, which utilizes the complete temporal sequence, already accounts for most of the retrieval performance.

More importantly, the proposed framework explicitly imposes a temporal-hierarchical alignment prior during training and subsequently interprets the resulting layer-wise performance differences as evidence of the underlying neural processing hierarchy. Such an analysis is insufficient to independently establish the proposed neurobiological correspondence between early neural responses and low-level visual features, or between late neural responses and high-level semantic representations.

Overall, this work demonstrates competitive benchmark performance and solid engineering implementation. Nevertheless, there remains considerable room for improvement in terms of methodological novelty and the strength of the evidence supporting its neuroscientific interpretations. I encourage the authors to conduct more rigorous temporal-window control experiments, independent neural representation analyses, and carefully controlled ablation studies to better isolate and substantiate the actual contributions of the proposed method.

Critical strengths

The article achieves the joint modeling of multi-time-scale neural information and visual hierarchical features by constructing global, stimulation period and post-stimulation neural response branches. It performs exceptionally well in within-subject and between-subject retrieval on EEG data.If the argument can be clearly presented and the source of performance can be demonstrated, it will be a very influential article. The rest can be seen in the summary.

Critical weaknesses

Seen as points 1, 2 and 4 of "Questions For Authors"

Questions for authors

If the authors can solve the following problems, I will increase my score. 1.The proposed framework explicitly encourages early neural responses (stim) to align with shallow visual features and late neural responses (res) to align with deep visual features during training. Consequently, the layer-wise performance patterns observed in Fig. 2 may be induced by the predefined alignment objective rather than reflecting an intrinsic neurobiological relationship.

Although the authors provide an ablation study using a Reversed Gaussian Prior, this experiment only reports the overall retrieval accuracy and does not examine whether the layer-wise alignment patterns themselves change under reversed training.

I suggest that the authors train a counterpart model from scratch using an explicitly reversed alignment objective, where shallow visual features are aligned with the res branch and deep visual features are aligned with the stim branch, while keeping the architecture and other training configurations unchanged. More importantly, the authors should repeat the layer-wise retrieval analysis in Fig. 2 using this reversely trained model to investigate whether the relative performance trends of the stim and res branches are also reversed.

If the observed layer-wise correspondence changes according to the imposed training objective, this would suggest that the reported temporal-hierarchical relationship may be primarily driven by the training strategy rather than independently reflecting the underlying neural processing hierarchy. Such an experiment would help distinguish learned alignment biases from intrinsic neural representational properties.

2.The authors divide the neural recordings into stim and res periods and associate these two stages with distinct functional roles, assuming that the stim period predominantly reflects low-level visual processing, whereas the res period primarily captures high-level visual semantics. However, visual semantic processing is generally not considered to begin only after stimulus presentation has ended. Rather, different levels of visual processing may overlap and interact throughout the neural response.

I therefore suggest that the authors investigate a broader range of temporal window configurations, including different window boundaries and combinations, to examine whether the proposed functional correspondence remains consistent. Such experiments would provide stronger evidence than the current binary division into stim and res periods.

3.Several figures contain unclear annotations and insufficiently explained abbreviations. For example, in Fig. 3, the meanings of P_STIM, P_RES, and other related metrics are not clearly defined, making the results difficult to interpret. Similar presentation issues appear elsewhere in the manuscript. I recommend that the authors carefully review all figures and provide explicit definitions for abbreviations and evaluation metrics.

In addition, in the framework illustration (Fig. 1), should the holistic (hol) branch be connected to all visual layers? The current illustration may lead to ambiguity regarding the actual alignment strategy used for this branch. The authors should clarify this aspect to ensure consistency between the methodological description and the framework diagram.

4.In THINGS-EEG experiments, previous studies commonly employ a 1000 ms EEG recording window for visual decoding. Interestingly, the ablation results show that the holistic branch alone, which utilizes the complete temporal sequence, already outperforms all the compared baseline methods in both Top-1 and Top-5 retrieval accuracy.

This observation raises an important question regarding the actual source of the reported performance improvements. In addition to the proposed multi-stage temporal information fusion, the gains may also arise from differences in model architecture, feature extraction capacity, or other implementation choices. I therefore encourage the authors to provide more carefully controlled comparisons that explicitly isolate the contribution of the proposed temporal alignment mechanism from improvements attributable to the underlying model architecture and other components.

4 comments

We thank the reviewer for recognizing the motivation and empirical results of our work. In response to the concerns, we clarify the experimental design and provide additional controlled experiments and analyses below.

1. Whether the layer-wise correspondence is induced by the predefined alignment objective

We clarify that each data point in Figure 2 and the related layer-wise analyses is obtained by independently training a model that aligns a single temporal branch with a single visual layer. These results are not extracted from the full LSTA model. The experiments do not use a Gaussian hierarchical prior or impose a predefined correspondence between STIM and shallow layers or between RES and deep layers. Therefore, the observed relative performance trends are not directly induced by the prescribed temporal–hierarchical alignment objective. We apologize that the original description did not make this distinction sufficiently clear.

We have also added the Constrained Reversed Gaussian Prior control in Table 8. This setting keeps the HOL prior unchanged and swaps only the prior centers of STIM and RES, removing changes to the HOL prior as a confounding factor in the reversed-alignment comparison.

In addition, Figure 13 reports the correspondence between the STIM and RES representations and a text-based semantic representational structure. Text representations are used only for post hoc analysis and do not participate in model training. RES exhibits higher semantic correspondence, providing complementary evidence of representational differences between the two temporal branches.

2. Sensitivity to temporal window selection and overlapping perceptual–semantic processing

We agree that visual perception and semantic processing do not have a strict, mutually exclusive temporal boundary. To examine the influence of window selection, we have added experiments that vary the boundary between STIM and RES while keeping the total sampling duration, the HOL branch, and the layer-weight settings unchanged. The results are reported in Table 10.

The names STIM and RES indicate relative information preferences rather than exclusive functional roles. Likewise, the Gaussian weighting strategy does not completely separate the two types of visual information: both branches can use intermediate visual layers that contain perceptual and semantic information.

3. Definitions of figure annotations and clarification of HOL visual layer alignment

We thank the reviewer for pointing out the unclear definitions. We have added the definitions and calculations of the metrics in the text accompanying Figure 3, and reviewed the other figures and tables to clarify the relevant abbreviations and annotations.

Regarding HOL alignment, we use visual layers after excluding those at the shallow and deep boundaries of the hierarchy. Because HOL covers the complete temporal window and contains information from both STIM and RES, this choice aims to reduce potential mismatches with visual representations that disproportionately emphasize one type of information and to encourage HOL to focus on information shared by the two branches. Table 9 reports the performance change associated with this visual layer selection.

4. Separating the contribution of temporal alignment from encoder and implementation improvements

To examine the sources of the performance improvement, we have added the progressive ablation study in Table 9. It separates the gains associated with visual layer selection, encoder replacement, and the three-branch structure, and includes a comparison with and without GAT. With the encoder architecture and HOL visual layer selection fixed, adding STIM and RES further improves performance, supporting the contribution of the multi-stage branch design beyond the gain from a stronger encoder.

We thank the reviewer again for the suggestions, which helped us clarify the experimental design, the scope of our interpretations, and the contributions of the individual design choices.

Thank you for your additional experiments and response, which have to some extent clarified my doubts regarding points one and four. I will discuss with the other reviewers before deciding whether to change my rating.

Reviewer_ZnMBOpenReview ↗

Final comment

Thanks for your reply, I've already increased my rating.

Thank you for recognizing our work and for your constructive feedback, which has helped us improve the paper.

Reviewer_biTpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Layer-Sensitive Temporal Alignment (LSTA), a progressive framework for dynamic vision-brain alignment. The paper aims to better preserve the progressive evolution of neural responses over time. The authors first decompose the neural dynamics into an overarching holistic representation and two functional temporal stages. Then they adopt an Inverted Transformer to extract temporal features, with a Graph Attention Network introduced as a residual branch before the backbone to capture inter-channel spatial relationships. They further use Gaussian prior masks to align early perceptual dynamics with shallow visual layers and late semantic dynamics with deeper representations. Experiments demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The motivation of the paper is meaningful. The authors aim to better preserve the progressive evolution of neural responses over time.
  2. The reported experimental improvement is relatively strong, including EEG- and MEG-based methods.
  3. The ablation studies are relatively comprehensive, covering key components, alignment strategies, and backbone choices.
Critical weaknesses
  1. The evaluation is limited to two THINGS datasets. Testing on more diverse and challenging datasets would provide stronger evidence of generalization.
  2. The paper lacks a comparison of computational costs. The authors should report runtime and GPU memory usage against existing methods.
  3. The justification for the proposed alignment is mainly empirical. A theoretical analysis explaining why matching temporal branches to visual layers improves on existing alignment methods would strengthen the paper.
  4. The similarity matrix in Figure 10(a) provides limited information. The paper lacks convincing visualizations that clearly demonstrate the benefits of the proposed method.
Questions for authors

Please see weaknesses. If the authors can adequately address these concerns, I would consider raising my score.

1 comment

We thank the reviewer for recognizing the motivation, empirical performance, and ablation studies of our work. We have revised the manuscript to address the four concerns as follows.

1. Evaluation on more diverse datasets

We agree that evaluation on more diverse datasets would strengthen the evidence for generalization. We have therefore extended our experiments to Alljoined-1.6M, an EEG dataset released in 2025, and compared our method with ProCA (2026). The results are reported in Table 5 of Appendix B and provide additional evidence of the effectiveness of our method beyond the two THINGS datasets.

2. Comparison of runtime and GPU memory usage

We agree that computational efficiency is an important aspect of evaluating the method. We have added a comparison with UBP, HVF, and HCF under identical hardware conditions. Table 11 in Appendix E reports additional one-time preprocessing time, training time per epoch, and GPU memory usage. The accompanying analysis discusses how the encoder choice affects the computational cost of LSTA.

3. Theoretical justification for temporal–hierarchical alignment

We have added a theoretical analysis in Appendix F. Under a simplified fixed-target, squared-error formulation, the analysis characterizes conditions under which matching temporal branches with corresponding visual depths reduces alignment error relative to reversed and uniform supervision. This provides a conditional theoretical explanation for the proposed alignment strategy.

4. More informative visualizations demonstrating the benefits of LSTA

We have expanded the qualitative analysis to illustrate the benefits of the method more directly. Figure 6 presents retrieval examples where three-branch fusion corrects errors made by individual branches, illustrating their complementary contributions. Figure 12 visualizes the distribution of fused EEG representations using t-SNE and highlights category-related groupings. Semantic category labels were not used during training and are introduced only for visualization. Additionally, Figure 13 provides a post hoc representational similarity analysis of the semantic correspondence of the STIM and RES branches.

We thank the reviewer again for the constructive suggestions, which helped us strengthen the evaluation, theoretical analysis, and presentation of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.