- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
LSTA addresses zero-shot brain–visual retrieval by encoding holistic, stimulus-period, and post-stimulus neural responses into separate embeddings and aligning them with multiple layers of a pretrained visual backbone. The neural encoders combine graph-based channel processing with an inverted Transformer, and a Gaussian branch-to-layer prior weights the symmetric contrastive losses and retrieval scores. The manuscript reports within-subject Top-1 accuracies of 91.4% on THINGS-EEG and 42.7% on THINGS-MEG, with additional LOSO results and branch, encoder, visual-backbone, and weighting-strategy analyses.
This assessment is based on the 17-page manuscript and the nine supplied source/configuration/documentation files. Static inspection and limited synthetic function checks were performed, not an end-to-end reproduction. The README identifies this as a partial core-code release; the findings below distinguish behavior of the documented entry point from the unverified provenance of the manuscript's numerical results.
Critical strengths
1. A concrete, task-motivated alignment design. Organizing neural responses into temporal views and connecting those views to visual depth is an intelligible inductive bias. The supplied implementation clarifies the independent branch encoders, per-layer visual projections, and weighted pairwise symmetric contrastive objective. The latter is consistent with Eqs. (5)–(6), rather than an unspecified new loss.
2. Relevant empirical coverage and alternative controls. The manuscript reports EEG and MEG experiments, within-subject and LOSO settings, and per-subject results averaged over five random-seed runs. Appendix C includes uniform and learned-attention alternatives, not just a reversed prior, and Figure 5 tests alternative neural and visual backbones. These are useful design checks. However, their evidential value depends on resolving the implementation and evaluation issues below.
Critical weaknesses
1. The documented evaluation selects checkpoints on the designated test partition and computes the reverse retrieval direction, so it does not establish the claimed independently evaluated EEG-to-image performance.
Test-set-driven model selection. config.py specifies separate training.pt and test.pt EEG paths, corresponding training/test image-feature caches, 200 epochs, and evaluation frequency pk=1 (lines 16–25). train.py constructs test_loader directly from these test paths (lines 109–139), evaluates it after every configured epoch, and saves the checkpoint whenever that test Top-1 improves (lines 217–228). It then returns and writes the selected test scores to CSV and averages these per-subject maxima (lines 237–281). No independent validation split or subsequent untouched-test evaluation is present in this entry point. This is test-partition feedback for model selection, not merely test monitoring. Withholding test concepts from gradient updates does not make a repeatedly test-selected score an independent estimate. The inspection does not show test samples in gradient training, and the size of any selection optimism is unknown; nevertheless, the supplied workflow cannot substantiate the stated evaluation without a validation-selected or prospectively fixed selection protocol and an independent final test.
Retrieval direction. In train.py:evaluate, the actual operations are:
weighted_eeg = torch.einsum("nid, ij -> njd", eeg_feats_norm, w_matrix)
sim_matrix = torch.einsum("mjd, njd -> mn", proj_img_feats_norm, weighted_eeg)
Here m indexes images and n indexes EEG samples. Applying topk(..., dim=-1) therefore retrieves EEG candidates for each image, not image candidates for each EEG query (lines 49–68). The logger is named TopK_I2E, and the final printout explicitly calls the result Best Image->EEG (lines 82–84 and 237–239). Conversely, Eq. (3) describes an EEG-by-image matrix, and Figure 10 explicitly describes EEG-feature queries for image retrieval. The two retrieval directions need not have the same Top-1/Top-5, even under a symmetric training loss. Computing EEG-to-image retrieval requires ranking image candidates in the transposed orientation of this supplied score matrix. I cannot conclude that 91.4% is necessarily the reverse-direction value, because the exact result-producing snapshot and outputs were not supplied. The discrepancy must be reconciled rather than assumed harmless or assumed to explain a particular amount of inflation.
Result provenance and scope. The loader pairs EEG rows with sorted image-feature keys and checks only sample counts (Datasets.py, lines 55–63 and 82–95); this may be correct for appropriately ordered caches, but does not verify image/concept identity or split disjointness. Explicit IDs and cache-generation manifests are needed. The supplied driver covers per-subject EEG training with 17 named EEG channels and one configured seed; it does not implement the reported MEG/LOSO/five-run orchestration. The README's explicit partial-release statement should be respected: this is missing verification coverage, not proof that those experiments were never performed. There are also baseline-reporting inconsistencies in Table 3: its displayed ATM-S subject values average to 27.10%/58.08%, not 28.5%/60.4%; HCF Top-1 averages to 84.08%, not 84.6%; and HVF Top-5 averages to 94.91%, not 94.6%. LSTA's own displayed subject averages are compatible with its reported means. These baseline discrepancies require source/configuration reconciliation and do not establish fabrication.
2. The supplied model/configuration does not implement several central descriptions, and the distinctive contribution remains insufficiently separated from generic multi-branch/multi-layer learning.
The default branch–prior mapping contradicts Eq. (4). model.py stacks features in the order [f_hol, f_stim, f_res] (lines 259–271). The configuration supplies mu=[3,5,8], and the weight builder and loss preserve that row order (config.py, lines 52–53; model.py, lines 287–300; loss.py, lines 46–57). Thus the supplied configuration assigns zero-based centers mu_hol=3, mu_stim=5, and mu_res=8, whereas the manuscript requires mu_stim < mu_hol < mu_res. The first two branch assignments are inconsistent with the stated ordering. The weights are also rounded to two decimals and left unnormalized in the implementation; their exact construction and scaling should be controlled in weighting ablations. A different experimental configuration may resolve the ordering issue, but the authors must identify the configuration and checkpoints that generated each principal table.
The implementation produces static branch summaries, not an explicit continuous temporal alignment. iTransformer.value_embedding maps each channel's complete time vector to a feature vector, the Transformer attends over the channel-token axis, and EEG_Encoder.forward averages that axis with x.mean(dim=1) (model.py, lines 145–185). Three independent encoders and projection heads then yield a tensor of shape [B,3,1024] under the supplied configuration. This can encode temporal patterns, but there is no time-indexed representation at the alignment interface or learned trajectory-to-depth correspondence supporting the Abstract/Introduction's claim of preserving continuous trajectories without static compression. Likewise, learnable_adj is a global parameter and the graph support is fixed and fully connected; GAT attention may still be input-dependent, so this does not imply that all channel interactions are static. It does mean that the claimed dynamic functional connectivity is not established by the global adjacency parameter alone (model.py, lines 70–100 and 124–141). The defensible formulation is structured temporal-view alignment unless an additional, currently undescribed mechanism is provided.
Contribution attribution and paper–code correspondence. Table 6 reports 90.1% Top-1/98.9% Top-5 for uniform weights, versus 91.4%/98.6% for the normal Gaussian prior. Figure 5 reports 90.4% Top-1 with an ATM encoder, versus 91.4% for the proposed encoder. These differences may be meaningful, but need paired uncertainty and matched capacity/optimization rather than being treated as proof that the specific mechanisms explain the overall high accuracy. The code confirms that the three encoders are independent, strengthening the need for a capacity-matched multi-head full-window control. The reversed-prior experiment is also confounded: Appendix C moves the holistic center to the visual-depth extremes as well as swapping stimulus and residual centers. Even the holistic-only model drops from 87.4% to 69.2% under that change, despite having no early/late branch relation to reverse. A control that leaves the holistic prior fixed is necessary to isolate the proposed temporal-depth correspondence.
The training script additionally enables random repeat-subset averaging, Gaussian noise, time shifts, channel dropout, smoothing, and frequency masking (train.py, lines 95–101; eeg_augmentation.py; config.py, lines 27–38). These are not specified in the manuscript's training/preprocessing account, and their use in matched baselines and ablations is unclear. The configured time_splite is not forwarded to the model constructor, which retains its default 0.2 (train.py, lines 143–154; model.py, lines 201–208); the actual MEG implementation of the stated 500 ms boundary therefore also needs to be supplied. These gaps prevent an unambiguous attribution of gains to the highlighted architecture and prior. More broadly, fixed stimulus-offset segmentation and branch-fusion accuracy gains do not independently establish perceptual–semantic disentanglement or biological inter-region cooperation.
Overall assessment. I recommend rejection in the current form. The decisive issues are the test-selected evaluation, the retrieval-direction discrepancy, and the mismatch between the specified model/prior and the submitted implementation—not the absolute magnitude of the reported accuracy or the mere reuse of standard modules. These are technical soundness and evidential concerns, not an allegation of intentional misconduct. A result-matched implementation and corrected, independently evaluated comparisons could materially change the assessment.
Questions for authors
1. Which retrieval direction generated the main tables? Please provide the exact evaluation function, checkpoint IDs, and image/EEG IDs for Tables 1, 3, and 4. For the same frozen checkpoints, report both EEG-to-image and image-to-EEG Top-1/Top-5, clearly defining the query and candidate axes. If the submitted evaluate() is not the result-producing code, please identify the corresponding version.
2. How was the final test kept independent of model selection? The supplied entry point selects and reports the maximum test score over 200 epochs. Was a separate validation protocol used for the manuscript? Please provide its split manifest and selection logs, or recompute the comparisons with validation-selected or prospectively fixed checkpoints and an independent final evaluation. Selecting a new maximum on the same test set after correcting the retrieval direction would not address this concern.
3. Which branch order and Gaussian centers were actually used? Please reconcile the code order [hol, stim, res] and centers [3,5,8] with mu_stim < mu_hol < mu_res. Identify the numerical weights, rounding/normalization, and saved configurations for the main and reversed-prior experiments. What operation, beyond static branch embeddings, supports the claimed continuous trajectory-preserving alignment?
4. Do the specific mechanisms outperform matched alternatives? Please provide paired subject- and run-level changes for normal Gaussian versus uniform/learned weights, GAT versus no GAT, and temporal branches versus a capacity-matched full-window multi-head model. Keep augmentation, visual features, training budget, and loss scaling matched. A stimulus/residual swap that leaves the holistic prior unchanged would directly test the claimed correspondence.
5. Can the reported experiments be traced to complete configurations and outputs? Please identify the image-feature extraction and EEG preprocessing steps, image/concept ordering checks, MEG and LOSO drivers, the passed stimulus-offset parameters, and five-seed result aggregation. Reconcile the baseline averages in Table 3 and distinguish quoted values from reruns. These requests target the reported experiments, not a new dataset or clinical study.
We thank the reviewer for identifying inconsistencies between the submitted code and the manuscript, and for highlighting experimental details that require clarification. We have corrected the relevant materials and respond to each question below.
1. Retrieval direction and consistency between the evaluation code and manuscript
We apologize for the retrieval-direction error introduced while organizing the code for submission. The experiments reported in the paper use EEG-to-image retrieval, with EEG representations as queries and images as candidates, consistent with the manuscript. We have corrected the corresponding evaluation implementation in the submitted code.
2. Test-set independence and checkpoint selection
The checkpoint selection in the original code was intended to maintain a consistent reporting protocol with the comparison methods. To examine whether the performance advantage depends on checkpoint selection, Table 7 additionally reports evaluation using only the last-epoch checkpoint of each subject-specific model, without selecting the best epoch based on test accuracy. We repeat this experiment with five random seeds and report the mean and standard deviation. Under this setting, the model still outperforms the listed comparison methods, indicating that its advantage is not solely attributable to best-checkpoint selection.
3. Branch order and Gaussian prior centers
The branch order used in the experiments is [HOL, STIM, RES], and the corresponding Gaussian prior centers are [5, 3, 8]. Thus, the STIM center is shallower than the HOL center, which is in turn shallower than the RES center, as specified in the manuscript. The submitted configuration incorrectly listed [3, 5, 8] due to an error during code organization. This has now been corrected. We apologize for the confusion caused by this inconsistency.
4. Controlled comparisons of weighting strategies, GAT, and temporal–hierarchical alignment
We have provided the following comparisons to examine the contributions of the components and alignment strategies:
5. Experimental traceability, data pairing, MEG/LOSO settings, and numerical corrections
We have supplemented Appendix A with the image-feature extraction procedure and the preprocessing steps for EEG, MEG, and the other data.
For data pairing, image-feature keys are sorted lexicographically, and EEG samples are paired with the corresponding image features by row index. This procedure assumes that the EEG samples were prepared in the same order. The code checks sample-count consistency. The image keys are used only for data pairing and are not supplied to the model as inputs or as additional supervision.
The MEG and LOSO experiments share the existing LSTA implementation. MEG uses the corresponding data and configuration. In LOSO, one subject is held out for testing and the training data of the remaining subjects are used for model training. No additional optimization or adaptation is performed on the held-out subject. Subject IDs are used only for data partitioning and are not provided as model inputs.
Table 7 reports results across five random seeds together with their standard deviations. We have also checked and corrected the identified numerical inconsistencies, including the transcription error for HCF Subject 9 and the inconsistencies in the reported baseline averages.
We thank the reviewer again for the feedback, which helped us improve the consistency between the manuscript and the submitted materials and clarify the experimental procedures.