acceptodds
Under review as a conference paper at ICLR 2027

Rethinking EEG-to-Image Decoding: From Fine-Grained Temporal Encoding to Hierarchy-Consistent Design

Abstract

Existing EEG-to-image methods commonly use specialized encoders, cross-modal alignment, and generative decoders, but it remains unclear what each stage actually requires or whether added model complexity helps. In this work, we systematically revisit each stage of this common pipeline and introduce TRACE (Temporal Resolution-Aware Complementary EEG Decoding) based on our analysis. At the encoder level, controlled comparisons reveal a counterintuitive trend where a minimal MLP outperforms Transformer-based and large pretrained baselines; our further analysis shows that decoding performance depends strongly on fine-grained temporal encoding, whereas increasing encoder capacity provides no systematic benefit. At the alignment and decoder levels, our analysis identifies unclear separation of level-specific evidence and hierarchical conditioning mismatches. We therefore propose Complementary Dual-Subspace Alignment to organize semantic and structural visual cues into complementary subspaces, alongside Hierarchy-Consistent Injection to route them through a coarse-to-fine visual autoregressive (VAR) generator. On THINGS-EEG, TRACE achieves state-of-the-art retrieval and reconstruction performance. Averaged across ten subjects, it achieves the highest Top-1/Top-5 retrieval accuracy (77.5%/96.2%) and improves six of seven reconstruction metrics over prior methods. Code will be released upon acceptance.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain-Computer Interface (BCI), EEG-to-image decoding, visual reconstruction, temporal representation learning, cross-modal alignment, visual autoregressive generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qw3AOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces TRACE (Temporal Resolution-Aware Complementary EEG Decoding), a pipeline to decode visual images from EEG signals. It revisits each stage of the common EEG-to-Image pipeline, including the encoder, the EEG-image alignment, and image generation module. It shows that the information in visual EEG responses is concentrated locally (models trained on specific time windows only generalize to closely temporal windows and not further ones). Interestingly, an MLP model applied to EEG timepoints outperforms previous convolutional, transformer, and large EEG models. The second contribution is on alignment between EEG and visual features. The paper shows aligning EEG features separately with high-level and low-level visual features can improve the decoding performance. Finally, instead of using a single-level conditioning signal for image reconstruction, it uses both high- and low-level features for conditioning: semantic features initialize the coarse generation stage, while structural features guide subsequent refinement.

Critical strengths
  1. The paper has a logical flow and is generally easy to follow. The design choices (such as the choice of encoder and the dual-space alignment) proposed are investigated for what makes them better than other options. Therefore, it can provide insight to other researchers on what should be optimized to improve the performance.
Critical weaknesses
  1. One of the weaknesses of the paper is that almost every metric value has been reported as a single number without any standard deviation values. A single number cannot tell whether the method outperforms the others significantly or is the difference only related to a lucky seed or a few outlier high-performing seeds.
  2. The second weakness is the limited data the paper uses. The only dataset everything is built on is THINGS-EEG2. Claims such as the ones presented in figures 4b and 4c can be specific to this dataset. If the authors claim for example localized temporal information in brain-responses to visual images, it should be verified with other datasets as well. If the authors' claims are limited to only THINGS-EEG2, my concern shifts towards applicability of the proposed method to other scenarios.
Questions for authors
  1. For the hierarchical visual targets, could the authors clarify why they selected layers 2 and 3 for low-level features? Is this optimized somehow? Also, works like Du et al. (2026) [Deep Models, Shallow Alignment: Uncovering the Granularity Mismatch in Neural Decoding] suggests extracting visual features from the intermediate layers result in better decoding performance. I'd like to know the authors opinion on whether fusion is better than optimizing the extracted layer depth.

  2. I'm not sure if I understood why the augmented visual features are summed to get the target embeddings and if the augmentation is needed at all. Did the authors perform any ablation on the augmentations used?

  3. Why did the authors only normalized visual features and not EEG features? This is mentioned in the paper but not supported.

  4. In the capacity sweep experiment, why did the authors only tested different widths and not depth, given that deeper networks usually learn better representations than wider ones?

  5. Could the authors elaborate on the two control experiments used to test the effect of temporal resolution on the EEG encoder (two TSConv-inspired forms of temporal coarsening)? Specifically could you provide the details of the network used for each test and compare it with the original TSConv network? A pytorch pseudocode or mathematical explanation would be helpful. Also, if the only goal is to coarsen the temporal resolution, why didn't you simply downsample or low-pass filtered the input signal?

  6. I'm not sure if I understood the motivation for "temporal alignment and shift sensitivity experiment". Could you please explain how it contributes to the claims and design choices of the paper?

2 comments

Thank you for the questions. Line numbers refer to the submitted version; changes are in blue, and Appendix B is new.

W1. Variability. TRACE retrieval results (Tables 2, 6) are five-seed means. The ten-subject mean is 77.5±0.45% Top-1 and 96.2±0.19% Top-5 (sample SD), and per-subject Top-1 SDs range from 0.74 to 2.26 points. S8 analyses report SDs (Tables 14/16/17/20/24). Table 5 now reports three Stage-2 seeds per routing row (e.g., CLIP 0.8445±0.0017), and the main new controls (B.1, B.3, B.4, B.7) use 3–5 seeds. Against Brain-HIVE's published scores, TRACE has the highest average retrieval (higher Top-1 on 6/10 subjects), and its reconstruction gains are large and consistent: six of seven metrics improve on all ten subjects (A.4).

W2. Datasets. We transferred TRACE to Alljoined-1.6M, an independently recorded dataset (consumer-grade 32-channel EEG, lower SNR), on S01 and S14 (fused Top-1, %, three seeds):

EncoderTemporal setupS01S14
NeuroBridge (reported by ProCA)—7.813.5
EEGProject (TRACE)none / 4 ms11.83±1.7626.50±3.46
EEGProject, coarsened204 ms / 4 ms10.00±1.3223.83±1.76
TSConv204 ms / 20 ms10.67±0.7624.17±1.53
TSConv, refined20 ms / 4 ms11.17±2.0229.67±0.58

TRACE nearly doubles NeuroBridge on S14 and on S01 beats every method in ProCA's Table II. More importantly for your concern, the temporal finding itself transfers: coarsening EEGProject with 204-ms input smoothing lowers Top-1 on both subjects, and the same TSConv refinement raises it on both (B.7, Table 34). Together with the S5/S9 replication (A.8), the claim is not specific to one dataset or subject.

Q1. Layer choice. Layers 2–3 are RN50's mid-level stages, and layer 4 feeds the H embedding, so the two targets cover different depths. To test whether a better single depth could replace fusion, we kept the full model and replaced only the L target with each single layer (S8, five seeds; new B.4, Table 30):

L targetL alone Top-1Fused with H, Top-1
GAP(layer1)44.40±3.0975.80±1.30
GAP(layer2)58.20±2.0878.60±1.34
GAP(layer3)73.90±3.6081.70±2.30
[GAP(layer2); GAP(layer3)] (ours, Table 20)77.60±1.5284.20±0.91
H only: final embedding (Table 20)71.20±2.05—

In line with Du et al., intermediate-layer targets decode well: layer 3 alone (73.9%) already exceeds the final embedding (71.2%). But the best single depth stays 10.3 pp below fusion (84.2%), and fusing H with any single layer improves on that layer alone. Optimizing depth therefore does not replace fusion. Among the tested targets, layer2+3 is the best both alone and fused. This is consistent with the capacity control in B.3: a capacity-matched L+L reaches 74.1%, versus 83.2% for a smaller H+L. Sec. 3.2 now discusses Du et al. and points to this control.

Q2. Augmentation pooling. Features of four level-specific augmentations are averaged with equal weights (Eq. 3). As described in Sec. 3.2 (L222–224), this keeps level-stable information. The relevant controls (S8, Top-1, %):

ConfigurationTop-1
TRACE (pooled targets at training and test)84.20
Same trained models, original-image test gallery (A.9)60.90
w/o pooling at training and test (Tables 3, 21)76.00
w/ additional transforms (Table 21)73.50

Removing pooling lowers Top-1 by 8.2 pp, and adding further transforms also hurts, so more augmentation is not simply better. The new test-only control fixes the five trained models and changes only the test gallery, isolating test-time pooling.

Q3. Normalization. Image-only normalization applies to the training loss; retrieval uses cosine scores, so both sides are normalized (Eq. 4 now shows the EEG denominator). With a fixed logit scale, the EEG norm acts as a per-sample confidence during training. At retrieval, normalizing EEG changes fusion Top-1 by only 0.1 point (A.2).

Q4. Depth. The capacity analysis (Sec. 4.2, A.6) asks whether encoder size explains the ranking, not which architecture is largest. Both ends of depth are covered. On the deeper side, Table 4 already includes deeper encoders, and all fall below EEGProject in fused Top-1: TSConv (79.3%), ATM with an iTransformer (81.2%), and LaBraM with a 12-layer Transformer (23.5%), versus 84.2%. On the shallow side, the Linear control (Table 14) removes the hidden residual layer and changes Top-1 by only 1.6 points, whereas input smoothing at an identical parameter count costs 12.9 points (now in Sec. 4.2). Together with the 110-fold width sweep (Fig. 4a), neither width nor depth explains the ranking.

Q5. Temporal controls. B.8 now describes the three controls; their code is in the updated supplementary material. Both requested operations are included: smoothing is a 204-ms low-pass filter before an unchanged EEGProject, and 40-ms weight sharing equals block-average downsampling to 25 Hz. TSConv's pooling window/stride goes from 204/20 to 20/4 ms (S8, five seeds):

EncoderSmoothingWeight gridTop-1
EEGProject (TRACE)none4 ms84.20±0.91
EEGProjectnone40 ms (weight sharing)72.30±2.17
EEGProject204 ms (input low-pass)4 ms71.30±3.05
TSConv (original)204 ms20 ms79.30±1.68
TSConv20 ms4 ms84.60±1.39

Full retrieval and reconstruction results are in Table 16, and the S5/S9 replication in Table 19.

Our question, however, concerns encoder design, not signal quality: as defined in Sec. 3.1 (L191–192), fine-grained temporal encoding is the encoder's ability to exploit fine temporal variation in a fixed input. All encoders get the same 17×250 input, and TSConv's own pooling already does a 204-ms average and 5× subsampling internally. The controls are therefore bidirectional on the same input: coarsening EEGProject's aggregation lowers Top-1, and refining TSConv's raises it. Input downsampling or filtering can only remove information and cannot produce the second direction: the bottleneck is how the encoder aggregates time, not what the signal contains.

Q6. Motivation for the shift experiment. Our main claim is that EEGProject performs best because it keeps fine, tens-of-ms temporal resolution. The temporal interventions in Sec. 4.2 test this on the input side: coarsening the input or the encoder lowers accuracy. The shift experiment tests it on the trained model. If EEGProject relies on precise timing, shifting the test input by a few ms should hurt it more than the other encoders. That is the result: EEGProject is the best encoder at zero shift and degrades fastest, and TSConv overtakes it on most reconstruction metrics at about 8–12 ms (A.12).

As noted in L405–411, this result has another possible explanation. EEGProject has a separate weight for each time point, so it would be shift-sensitive even if the EEG carried no fine-timing information. Jitter training separates the two explanations. If the sensitivity came only from the weights, the same model trained with jitter should become shift-tolerant at little cost. If decoding depends on precise timing, shift tolerance should cost clean accuracy. The second is what happens (S8, fused Top-1, %):

Training jitterClean Top-1 (5 seeds)Test shift 0 → 24 ms (1 run)
none85.6±1.1488.00 → 3.00
±8 ms83.7±0.8486.00 → 8.75
±20 ms79.4±1.7578.50 → 49.50
±40 ms67.1±2.9267.50 → 67.75
±80 ms56.3±2.6856.50 → 52.25

With ±40 ms jitter, accuracy is flat across test shifts, but clean Top-1 falls from 85.6 to 67.1% (Tables 24–25).

The experiment contributes to the paper in three ways. It supports the claim that fine timing carries decodable information with the architecture and parameter count held fixed. It explains why we do not add shift invariance to EEGProject, such as jitter augmentation or temporal pooling: invariance costs accuracy. And EEGProject stays ahead of TSConv at every jitter level (Table 23), so its lead does not come from position-specific weights.

We hope these answers resolve your concerns about variability, data scope, and the design choices you asked about. If anything remains unclear or you have further questions, we would be glad to discuss them at any time during the discussion period. If your concerns are resolved, we would be grateful if you would consider raising your score; your assessment matters a great deal to us.

Reviewer_4Z5SOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACE examines encoder design, visual alignment, and generator conditioning for EEG-to-image decoding. Two MLP-based EEG branches align with high- and low-level RN50 features and condition a VAR decoder through separate interfaces. On THINGS-EEG, it reports 77.5%/96.2% Top-1/Top-5 in 200-way within-subject retrieval across ten subjects and five seeds, using 80-repeat test averages. Reconstruction experiments and temporal interventions examine the benefits of fine-grained encoding and complementary conditioning.

Critical strengths
  1. The temporal interventions are the strongest part of the paper. Coarsening EEGProject reduces retrieval, while refining TSConv's pooling and stride improves it. The pattern is reproduced on Subjects 5 and 9 (Tables 16–19). The linear control and jitter-training experiments add useful evidence about the role of temporal processing.

  2. Retrieval covers ten subjects and five seeds, and reconstruction results are reported for all ten subjects. The two-high-level-branch, augmentation, and same-VAR routing controls are relevant to the proposed design. Appendix A.2 also specifies fixed Stage-1 checkpoints based on training-loss convergence rather than test accuracy.

Critical weaknesses
  1. The results support useful complementary branches, but the semantic/structural interpretation is less convincing. Retrieval uses augmentation-pooled image features, whereas EEG-CAM uses untransformed images, different feature layers, and different map normalizations (Appendices A.9–A.10). The maps therefore do not directly explain the deployed score. The two-high-branch control is helpful, but a matched two-low-branch control is especially relevant because the low branch is stronger individually. Unnormalized EEG embeddings also give the branches implicit, query-dependent weights. Table 5 supports the chosen conditioning interfaces, but AdaLN acts throughout VAR and SOS information can affect later scales; it does not isolate an early-semantic/late-structural division. The detailed routing comparison uses one Stage-2 run per configuration, primarily on Subject 8, leaving the reliability of smaller gains uncertain.

  2. Part of the encoder explanation goes beyond the evidence. Appendix A.6 measures attention uniformity after averaging over 200 examples and four heads. Selective individual attention patterns can average to a nearly uniform matrix, so this does not establish that attention is uninformative. The appendix appropriately treats the connection to MVNN as an interpretation, but the main-text explanation is stronger. The temporal interventions are valuable, although weight sharing and stride changes also alter free coefficients and the representation reaching the readout. Conclusions about capacity and pretrained encoders should therefore remain limited to the tested configurations. The 1.8-point retrieval lead over Brain-HIVE also compares against published scores rather than a matched rerun, so its statistical reliability remains unestablished.

These concerns mainly affect the explanation of why the proposed design works. The temporal results are promising, but the alignment and routing claims need more direct support before I would recommend acceptance.

Questions for authors
  1. Can the semantic/structural distinction be tested quantitatively using the augmentation-pooled features actually used for retrieval? Does it remain after Stage-2 encoder fine-tuning?

  2. How does high/low fusion compare with a capacity-matched two-low-branch ensemble? How much does normalizing the EEG branches change the fusion result?

  3. What do per-example and per-head attention statistics show? Does replacing learned attention with uniform weights affect performance?

  4. Are the routing gains in Table 5 consistent across training seeds or additional subjects? How many reconstructions are generated per EEG input, and how are they used to calculate the reported scores?

  5. Why is the five-seed jitter-free reference 85.6 ± 1.14% in Table 24 but 84.2 ± 0.91% in Tables 14, 16, and 20? Were different configurations or seed sets used?

3 comments

Thank you for finding the temporal interventions the strongest part of the paper. The revised PDF (changes in blue; new experiments in Appendix B) adds the direct support you asked for on alignment and routing. Where we tighten wording (branch names, MVNN, routing), no reported number changes, and our main finding, that fine-grained temporal encoding rather than capacity drives encoder performance, is unaffected.

W1 & Q1. Sensitivity with augmentation-pooled features. We passed 8×8 patch shuffling and blur (σ=4) through the actual pooled retrieval pipeline on S8, before and after Stage 2 (EEG Top-1, %):

PerturbationBefore, HBefore, LAfter, HAfter, L
8×8 patch shuffle17.535.516.034.0
Gaussian blur, σ=447.536.044.039.0

H retrieves better than L under blur, and L better than H under shuffling: the interaction (H_blur−H_shuffle)−(L_blur−L_shuffle) is 29.5 pp before fine-tuning and stays positive after it (23.0 and 12.5 pp for two Stage-2 seeds; B.5, Table 31). On frozen targets the contrast comes mainly from L; in the deployed score, H's blur augmentations add blur invariance. Target layer and augmentations thus jointly shape the branches: H leans toward global configuration and L toward local texture/edges. We accept that "semantic/structural" overstated this, and the revision uses high-/low-level with these definitions (Sec. 3.2). The conclusion is unchanged: the alignments are complementary, and fusion (84.2%) exceeds either branch (71.2/77.6%). EEG-CAM is now presented as a qualitative illustration, with B.5 as the quantitative evidence.

W1 & Q2. Capacity and EEG normalization. B.3 adds L+L ensembles of two independent low-level branches (S8, five seeds):

TargetsEEG widthsParamsFusion Top-1
H+L (original)1024 / 153616.922M84.20±0.91
H+L1024 / 102413.171M83.20±1.68
L+L, capacity-matched1271 / 127116.921M74.10±2.27
L+L1024 / 153617.184M74.90±2.04

The capacity-matched L+L is within 0.004% of the original parameters yet 10.1 pp lower, and even the smaller H+L beats it by 9.1 pp. Within L+L, fusion barely improves over its best branch, while H+L adds about 10 pp, so the gain comes from complementary targets rather than from a second branch or more parameters (Table 29).

Retrieval normalizes both sides and sums the cosines equally; Eq. 4 omitted the EEG denominator, now corrected. Image-only normalization gives 84.30 vs 84.20% Top-1 (A.2), so implicit norm weighting has little effect.

W2 & Q3. Per-example attention and uniform replacement. You are right that averaging can hide selective rows. B.2 computes TV(A,U)=½Σ_k|A_k−1/18| per example, head, and query before any averaging (S8; one row per training seed):

NormalizationEntropy (H / L)TV (H / L)Fusion Top-1: learned → uniform
MVNN, seed 10.987 / 0.9830.051 / 0.07578.5 → 79.0
MVNN, seed 20.994 / 0.9870.038 / 0.06980.0 → 81.0
z-score, seed 10.987 / 0.9920.049 / 0.04164.0 → 62.5
z-score, seed 20.988 / 0.9940.049 / 0.03558.5 → 58.0
z-score, seed 30.987 / 0.9930.049 / 0.03961.0 → 60.5

Electrode queries remain near uniform (entropy 1 = uniform). The exception is the subject-token query, where one or two H heads per seed are nearly one-hot (TV up to 0.94): selective attention exists, but on the subject token, not across electrodes. Replacing all attention probabilities with 1/18 at inference, including those heads, changes fused Top-1 by at most 1.5 pp, and the same holds when ATM is trained on z-score inputs (B.1–B.2, Table 28). The near-uniformity therefore comes from the attention design itself, not from MVNN; we removed the MVNN explanation from Sec. 4.2 and A.6. Because ATM shares EEGProject's readout head, this does not affect the encoder ranking.

W1 & Q4. Routing across subjects and seeds. Hierarchy-Consistent Injection is an analysis-informed design component. We claim its empirical benefit under controlled ablation, not a mechanistic early/late division. Table 5 now reports three Stage-2 seeds for the full model and every control (S8, mean±SD):

RoutingPixCorrSSIMAlex(2)Alex(5)Inc.CLIPSwAV↓
Ours: H→SOS, L→AdaLN.2861±.0049.3974±.0024.9401±.0037.9657±.0049.8689±.0141.8445±.0017.4584±.0060
H→SOS+AdaLN.2687±.0033.3928±.0015.9281±.0012.9625±.0040.8431±.0018.8063±.0019.4694±.0011
Swapped.2670±.0018.3943±.0025.9311±.0040.9653±.0018.8438±.0028.8166±.0031.4664±.0027
H+L→both.2572±.0022.4002±.0010.9363±.0010.9656±.0012.8453±.0024.8292±.0012.4639±.0004

The proposed routing leads on six of seven metrics, and SD is small relative to the gaps (e.g., CLIP .8445±.0017 vs at most .8292 for the controls; for SSIM, see the A.12 diagnostic). On S6 it also leads on six of seven metrics (B.6, Table 33). We agree that AdaLN acts at every scale, so Sec. 3.3 now describes the design as an assignment of conditioning interfaces rather than a strict early/late split. Each averaged test EEG yields one reconstruction (200 images; fixed sampling seed, CFG 1.5, top-k 900, top-p 0.95), with no best-of-N selection (A.2).

Q5. Table 24 vs. Tables 14/16/20. They are independent five-seed runs. The Table 24 jitter-free control was trained inside the jitter pipeline (zero time shift instead of the moving-average EEG augmentation), so it is a different configuration as well as a different run (now stated in A.13).

W2. Other points. Parameter count does not explain the interventions: input smoothing keeps every parameter yet lowers Top-1 from 84.2 to 71.3%, while removing the hidden layer (Linear) costs only 1.6 pp (Sec. 4.2). Our capacity and pretraining claims concern the tested configurations.

For Brain-HIVE, comparing against its published scores is the fairest choice: we also ran its public repository, and our run scored lower than the published numbers, so the published scores favor the baseline. Against them, TRACE has the highest ten-subject average retrieval (77.5/96.2% vs 75.7/94.9% Top-1/Top-5; higher Top-1 on 6/10 subjects), with five-seed SDs of only 0.45/0.19 pp. The larger gain is in reconstruction: the ten-subject average PixCorr rises from .195 to .237 and AlexNet(2) from .843 to .910, and six of seven metrics improve for every one of the ten subjects (A.4).

We hope the added evidence resolves your concerns about the alignment and routing claims. If anything remains unclear or you have further questions, we would be glad to discuss them at any time during the discussion period. If your concerns are resolved, we would be grateful if you would consider raising your score; your assessment matters a great deal to us.

Thank you again for your detailed review. As the discussion period is coming to an end, we would like to kindly check whether our responses have addressed your concerns. If any point remains unclear or you have further questions, we would be glad to address it before the deadline. If your concerns have been resolved, we would be grateful if you would consider updating your score.

Reviewer_5QYwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper revisits the standard EEG-to-image pipeline (encoder → alignment → decoder) and proposes TRACE. It claims that (i) a minimal MLP encoder with per-timepoint weights outperforms convolutional, Transformer, and pretrained encoders, explained by “fine-grained temporal encoding” rather than capacity; (ii) semantic and structural visual evidence should be aligned in two independent subspaces; and (iii) the two subspaces should be routed to different stages of a coarse-to-fine VAR decoder (high-level → SOS, low-level → AdaLN). On THINGS-EEG, TRACE reports 77.5% Top-1 / 96.2% Top-5 averaged over ten subjects and five seeds, and improves six of seven reconstruction metrics over the strongest prior results. Code is provided in the supplementary material.

Critical strengths

Strong and consistent reconstruction results across all ten subjects. On the ten-subject average, TRACE improves PixCorr (.237 vs .195), SSIM (.368 vs .366), Alex(2) (.910 vs .843), Alex(5) (.945 vs .905), Inception (.816 vs .756), and SwAV (.488 vs .554); the only loss is CLIP (.798 vs .808). The per-subject tables show that six metrics improve for all ten subjects relative to Brain-HIVE, and CLIP improves on three. This is the paper’s most robust contribution.

An unusually thorough, stage-by-stage analysis, including a clean within-architecture test of the temporal-granularity hypothesis. Coarsening EEGProject’s temporal resolution hurts (84.2 → 72.3 with 40-ms weight sharing; 84.2 → 71.3 with 204-ms input smoothing), and refining TSConv’s pooling window and stride improves it from 79.3 to 84.6, replicated on Subjects 5 and 9. The paper also provides controlled encoder comparisons, capacity sweeps, temporal generalization/occlusion, branch-wise retrieval, decoder injection ablations, and a detailed appendix. This is more systematic than most work in this area.

Critical weaknesses
  1. The encoder ranking is established under the tested MVNN-normalized input protocol, but its generality across preprocessing choices remains unclear. All four encoders use the same MVNN, 17 channels, and 1-s window, which is a controlled comparison. However, different architectures can respond differently to this preprocessing, and the proposed link between MVNN and ATM’s nearly uniform attention is an interpretation rather than a causal result (A.6: “an architectural interpretation, not a direct causal test”). The bidirectional interventions are the strongest evidence for the temporal-granularity hypothesis, but they do not establish that the cross-architecture ranking would survive a different normalization. LaBraM is substantially lower than the other encoders, and its loading, freeze status, and optimization configuration are not described. The appropriate request is a no-MVNN or z-score comparison as a test of generality, not a rejection of the result.
  2. The semantic/structural division of the two branches lacks direct validation. The paper provides useful quantitative evidence that the two targets carry complementary predictive information. The remaining issues are: (i) the two branches have different widths (DL=1536 vs DH=1024), so the low branch’s higher standalone accuracy cannot be attributed solely to visual hierarchy; (ii) the two-high-level control’s width, total parameters, and augmentation configuration are not stated; (iii) retrieval gains and CAM do not establish a clean semantic/structural division—GAP features can contain texture and shape information, so losing explicit spatial layout does not mean the low-level target is not a structural cue; the “structural” label is an interpretation that lacks direct validation. Direct evidence would include error-overlap analysis, branch performance on semantic vs texture tasks, CCA/probes, and capacity/augmentation-matched controls.
Questions for authors
  1. The four encoders use the same MVNN preprocessing, which is a controlled comparison. Please test the ranking under a different normalization (raw-scale or z-score), or report the ranking under both, to establish the scope of the claim. The MVNN–ATM link is an interpretation; a direct test would compare ATM’s attention entropy under both normalizations. For LaBraM, please report the loading log, freeze/unfreeze status, learning rate, and patch configuration; its result is substantially lower than the other encoders, and the cause is not diagnosed.
  2. The high/low/fusion/two-high numbers (71.2/77.6/84.2/72.0) support complementary predictive information. To support the semantic/structural interpretation, please report (i) the width, total parameters, and augmentation configuration of the two-high-level control; (ii) same-width branches (both 1024 or both 1536); (iii) direct evidence such as error overlap, semantic vs texture probes, or CCA/subspace analysis. GAP features can contain texture and shape information, so the structural label needs direct validation.
  3. Appendix A.2 and Eq. 4 specify image-only normalization, whereas the supplementary retrieval code sums cosine similarities. Which scoring rule and fusion weights produced the reported results? Please provide the evaluation configuration and explain the different jitter-free controls in Tables 20 and 24. Removing image augmentation pooling reduces Top-1 by 8.2 points (84.2 vs 76.0); the isolated contribution of test-time pooling is unclear because the ablation changes both training and test. Did the baselines use a matching augmentation protocol?
  4. The released inter-subject script selects checkpoints by test Top-1, while the published intra-subject results use fixed epochs. This does not by itself invalidate the reported results, but please report inter-subject results with fixed epochs or a validation split, and unify the one-click script, which still uses the test-selection path. In addition, generaliztion of the method is limited under single dataset, tests under more datasets and across individuals are needed. Stage 2 also uses one run per configuration (Sec. 3.4); please report 3–5 seeds for the main reconstruction comparison and the decoder injection ablation.
  5. The LaBraM implementation exists but is not exposed in the training scripts; the Linear control, the width variant, and the diffusion decoder also lack runnable scripts/configs in the supplementary code. Please provide the integration and reproduction configs. Finally, A.1 states a 0.1–100 Hz band-pass, but the released preprocessing does not apply one; resampling anti-aliasing is not equivalent. Was the upstream data already filtered?
4 comments

Thank you for recognizing the reconstruction gains and the stage-by-stage analysis. The revised PDF marks changes in blue; new experiments are in Appendix B.

W1 & Q1. Normalization and LaBraM. The ranking does not depend on MVNN. On S8 we replaced MVNN with channel-wise z-score and retrained all four encoders, keeping everything else as in Table 4 (three seeds):

EncoderH Top-1L Top-1Fusion Top-1Fusion Top-5MVNN Fusion Top-1 (Table 4)
EEGProject64.00±1.7336.17±10.7772.50±1.8093.50±0.8784.20
TSConv56.17±1.8950.33±3.7567.33±3.7593.67±1.4479.30
ATM48.33±0.7644.67±1.0461.17±2.7589.67±2.0281.20
LaBraM25.17±2.319.50±1.8021.50±0.8753.00±3.5023.50

EEGProject still has the best fused Top-1 and LaBraM the worst; only TSConv and ATM swap (B.1, Table 27). This complements our main evidence, the within-architecture interventions (Table 16), which need no cross-architecture ranking and replicate on S5/S9 (A.8) and a second dataset (B.7).

We agree the MVNN–ATM link was an interpretation. Under z-score, ATM's electrode attention is equally near uniform (normalized entropy 0.987–0.994, vs 0.983–0.994 under MVNN; 1 = uniform), and uniform replacement changes fused Top-1 by ≤1.5 pp (B.2, Table 28). The cause is the attention design itself, not MVNN; we removed the MVNN explanation (Sec. 4.2, A.6). Since ATM shares EEGProject's readout, the encoder conclusion is unchanged.

LaBraM is loaded from the pretrained checkpoint and fully fine-tuned (not frozen) at LR 1e-4, with 200-Hz input and one 200-sample patch per channel (A.6). Loading log, identical for all branches and seeds: all 219 backbone tensors load with matching shapes and no unexpected keys; only the new head (head.weight/bias) and fc_norm are newly initialized.

W2 & Q2. Two-level targets. The two-high-level control uses two 1024-wide encoders (10.81M; 12.91M with projectors) with different high-level augmentations (A.9). B.3 adds equal-width H+L at both requested widths and L+L ensembles of two independent low-level branches (S8, five seeds):

TargetsEEG widthsParamsFusion Top-1
H+L (original, Table 20)1024 / 153616.922M84.20±0.91
H+L1024 / 102413.171M83.20±1.68
H+L1536 / 153620.673M81.60±6.26
L+L, capacity-matched1271 / 127116.921M74.10±2.27
L+L1024 / 153617.184M74.90±2.04

H+L at either equal width beats both L+L controls by 6.7–9.1 pp, so width and parameter count do not explain the gain. At width 1024, H/L/fusion are 68.1/73.7/83.2%, so fusion still adds 9.5 pp (B.3, Table 29).

B.5 adds direct evidence. Per 200 queries, 22.0 are correct only for H and 34.4 only for L. Perturbations of the frozen targets show L is nearly shuffle-invariant but blur-sensitive, and H the reverse; through the deployed pooled pipeline, the contrast holds between branches, with L ahead under shuffling and H ahead under blur (Table 31). We agree "structural" was imprecise: H leans toward global configuration and L toward local texture/edges, and the revision defines the branches this way (Sec. 3.2). Probes on targets with an a-priori level agree (Table 32): L is higher on Sobel edges and Gram texture, and H on CSLB functional (.327 vs .297) and encyclopaedic features. The branches share much information, so this refines the interpretation, not the result: fusion (84.2%) exceeds either branch (71.2/77.6%).

Q3. Scoring, seeds, and pooling. Eq. 4 and A.2 omitted EEG normalization: retrieval sums the two cosines 1:1. Both are corrected; image-only normalization changes Top-1 by 0.1 pp (A.2). The Table 24 jitter-free control is an independent five-seed run in the jitter pipeline (zero shift instead of moving-average augmentation), a different configuration from Table 20 (A.13). With five trained models fixed, changing only the test gallery gives 84.20% (pooled) vs 60.90% (original images; A.9). The 84.2→76.0 ablation removed train and test pooling together; matching train/test features is part of the method, as in NeuroBridge. Baselines are reported complete methods; our controlled comparisons fix all other components.

Q4. Fixed epochs, stability, and generalization. All reported results come from our intra-subject training script, which has always used fixed epochs (H60/L100) without test-based selection. Test-best selection came only from the public codebase we built on: the inter-subject script, which we removed (we run no inter-subject experiments), and the default of the one-click script, which now uses the same fixed epochs. No reported result is affected.

Table 5 now reports three Stage-2 seeds for the full model and every control. Its full-model row is the S8 main reconstruction setting, so the main comparison is also three-seed (S8, mean±SD):

PixCorrSSIMAlex(2)Alex(5)Inc.CLIPSwAV↓
Best prior (Table 1).2190.3960.8760.9260.7880.8270.5310
Ours: H→SOS, L→AdaLN.2861±.0049.3974±.0024.9401±.0037.9657±.0049.8689±.0141.8445±.0017.4584±.0060
H→SOS+AdaLN.2687±.0033.3928±.0015.9281±.0012.9625±.0040.8431±.0018.8063±.0019.4694±.0011
Swapped.2670±.0018.3943±.0025.9311±.0040.9653±.0018.8438±.0028.8166±.0031.4664±.0027
H+L→both.2572±.0022.4002±.0010.9363±.0010.9656±.0012.8453±.0024.8292±.0012.4639±.0004

The three-seed mean exceeds the best prior result on all seven metrics, and the proposed routing leads the controls on six of seven (for SSIM, see the A.12 diagnostic). Routing also leads on 6/7 metrics on S6 (B.6, Table 33).

On the independently recorded Alljoined-1.6M (B.7, Table 34), TRACE reaches 11.83/26.50% Top-1 on S01/S14 vs 7.8/13.5% for NeuroBridge (reported by ProCA), and both temporal interventions reproduce. Held-out subjects are left to future work.

Q5. Reproducibility and filtering. We have updated the supplementary material: both stages support Linear, the width variant, and LaBraM, with scoring and seed aggregation, and the diffusion baseline has its own entry. THINGS-EEG2 was band-pass filtered online at acquisition; the 0.1–100 Hz band in A.1 is this upstream filter, and our own preprocessing does not re-filter.

None of this changes our main finding: fine-grained temporal encoding, not capacity, drives encoder performance.

We hope these answers resolve your questions on normalization, the two-level targets, and reproducibility. If anything remains unclear or you have further questions, we would be glad to discuss them at any time during the discussion period. If your concerns are resolved, we would be grateful if you would consider raising your score; your assessment matters a great deal to us.

Thanks for the rebuttal. I will maintain my score.

Thank you for reading our responses and for your positive assessment of our work. Your questions on preprocessing, encoder comparisons, and the H/L branches helped us strengthen the analysis, and we appreciate the time and care you put into the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.