acceptodds
Under review as a conference paper at ICLR 2027

How Much Alignment Capacity Does EEG-to-Visual Decoding Need? Retrieval Is Preserved with 42% Fewer Parameters

Abstract

EEG-to-visual decoding increasingly maps noisy neural responses into expressive pretrained visual spaces that support natural-image retrieval and generation, yet the amount of learned alignment capacity required to preserve useful visual information remains unclear. We isolate this question in ENIGMA by replacing its dense final residual transformation with rank-224 factors while holding the parent encoder, visual targets, training objective, participant protocol, and evaluation fixed. The intervention reduces trainable EEG-model parameters from 1.405M to 0.815M (41.97%). In a prospectively frozen confirmation across five development-independent participants, mean 200-way single-trial Top-1 is 8.783% for ENIGMA and 8.909% for the compact model, with both one-sided 95% lower bounds above a prespecified -percentage-point retention margin. Compression also lowers effective rank, preserves ENIGMA-compact representation geometry under centered kernel alignment (CKA) and local image neighborhoods, and slightly increases correct-image cosine alignment. At 80 repetitions, the compact representation satisfies a prespecified DINO-content retention criterion when conditioning a fixed SDXL-Turbo/IP-Adapter generator with shuffled-EEG controls. Thus, substantial residual capacity is removable while preserving confirmed retrieval and high-repeat downstream conditioning, making alignment capacity an explicit design variable for EEG-to-visual decoders.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG-to-visual decoding, neural decoding, representation learning, multimodal alignment, low-rank factorization, model compression, image retrieval, non-inferiority testing

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_21HgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how much learned alignment capacity is needed for EEG-to-visual decoding. Based on the existing ENIGMA architecture, the authors replace its final dense 1024 × 1024 residual transformation with a rank-224 factorization. This reduces the trainable EEG-model parameters by 41.97%, while keeping the remaining architecture and training protocol matched. The main experiment reports that single-trial 200-way retrieval performance is preserved across five development-independent participants and two training seeds. The paper further analyzes representation geometry, effective rank, local image neighborhoods, and downstream image generation under a fixed generator. The question of alignment capacity is meaningful and relevant to the optimization of EEG-to-visual systems. The paper also provides positive results from a reasonably careful experimental study. However, the necessity of the residual transformation and the additional scientific insight from the capacity intervention remain insufficiently established. These issues limit the strength and scope of the paper's main contribution.

Critical strengths
  1. A relevant optimization problem with potential scientific value. The paper focuses on how much learned capacity is needed to align EEG representations with a pretrained visual space, rather than only seeking higher retrieval accuracy. This is a useful direction for understanding the design trade-offs of EEG-to-visual decoders. Studying capacity under a fixed visual target and training protocol also provides a clear starting point for isolating this question. (Introduction; Sec. 2.3)

  2. A controlled intervention with prospective confirmation. The authors modify a specific residual transformation while keeping the other model components and training conditions matched. The rank-224 configuration and retention criterion are fixed before the confirmation on five development-independent participants, with two training seeds per participant. The reported participant-level and crossed participant/image lower bounds both pass the prespecified −0.5 percentage-point retention margin. This makes the main comparison more informative than a compression result based only on a single run or a post hoc choice of configuration. (Sec. 3.1–3.3; Table 1; Fig. 2; Appendix A.6)

  3. The analysis examines several aspects of the learned representation. The authors evaluate ENIGMA–Compact CKA, effective rank, correct-image cosine alignment, and local image-neighborhood overlap at different repetition counts. They also examine retrieval under two repeat-aggregation paths and test downstream generation with shuffled-EEG controls. These analyses provide complementary evidence about the behavior of the compact model beyond the primary Top-1 metric. (Sec. 4–6; Figs. 2–4; Appendices C–E)

  4. The paper provides useful diagnostic results and practical measurements. The development-only rank sweep helps contextualize the selected operating point, while the generation experiment reports the boundary between the high-repeat and single-trial settings. The authors also measure parameters, MACs, training time, and inference latency, and distinguish the controlled ENIGMA comparison from the NICE comparison under a different protocol. These details help readers understand both the observed result and its practical scope. (Sec. 6–8; Tables 2–3; Appendix B; Appendix C, Table 7; Appendix F)

Critical weaknesses
  1. The necessity of the learned residual transformation is not established. The main experiment compares a dense residual matrix with its rank-224 factorization, but does not include a baseline that removes this specific residual correction. The original ENIGMA paper motivates the skip connection as a way to stabilize training and refine the initial projection, but does not report a direct ablation of this residual correction. Appendix B also shows that rank 64 achieves 7.053% Top-1, close to the 7.067% of rank 224, on the development participants. A zero-residual baseline would help establish how much this component contributes to retrieval and whether the proposed compression preserves a necessary function or mainly simplifies a component that may be less important than assumed. (Sec. 3.1; Appendix B, Table 3)

  2. The scientific insight about alignment capacity remains limited. The paper establishes a successful operating point in one existing architecture, but does not identify a minimum sufficient rank or a clear performance boundary. The rank constraint applies only to the learned residual update, while the full 1024-dimensional identity pathway remains available. Theorem 1 establishes the mathematical rank constraint but does not explain why this capacity is sufficient for the retrieval task. A clearer account of what the results reveal about alignment capacity beyond successful compression would strengthen the main contribution. (Sec. 2.3; Sec. 3.1; Sec. 8.1–8.3; Appendix B)

  3. The representation analysis does not fully explain why compression works. The CKA, effective-rank, and neighborhood results show that several measured representation properties remain similar after compression. However, both architectures are trained from scratch, so the compact network may adapt to the rank constraint during optimization. The current analysis does not clearly distinguish this adaptation from redundancy in the original residual transformation. Comparing ENIGMA–Compact CKA with CKA between independently trained dense ENIGMA models would provide useful context. The relation between the upstream 184-dimensional representation and the capacity of the final residual correction also deserves further discussion. (Sec. 3.1–3.2; Sec. 5; Fig. 3; Appendix D, Table 8)

  4. The evidence supports a narrower conclusion than the broader capacity question suggests. The confirmation is limited to THINGS-EEG2, one visual-target setting, a fixed 200-image gallery, and separately trained participant-specific models. The rank sweep does not systematically test how the required rank changes with EEG repetition count. The generation experiment satisfies the prespecified retention criterion at 80 repetitions, while the crossed lower bound narrowly misses it at one repetition. These results support the tested setting, but the behavior of alignment capacity across other datasets, visual targets, and levels of neural evidence remains unclear. (Sec. 3.2–3.3; Sec. 6; Sec. 8.3; Appendices B, C, and E)

  5. The presentation, prior-work positioning, and reproducibility need clarification. The Introduction moves quickly from recent advances in EEG decoding to the alignment-capacity question, without fully establishing the connection between them. The phrase "visual information recoverable from EEG" is also broader than the retrieval-based evidence presented. The relatively short reference list leaves some concern about whether related low-complexity EEG-to-visual mappings and model-compression work are sufficiently distinguished. In addition, the submitted materials do not provide the implementation code for the main ENIGMA/Compact comparison. Finally, the 41.97% parameter reduction corresponds to only about a 4.6% reduction in reported MACs, with nearly unchanged training time and slightly higher measured batch-1 latency. (Introduction; Sec. 2; Reproducibility Statement; Appendix C, Table 7)

Questions for authors
  1. Could the authors provide a zero-residual baseline that removes the learned residual correction entirely? Given the similar development performance at ranks 64 and 224, how much does this component itself contribute to retrieval, and what evidence supports the current capacity choice?

  2. What exactly does "alignment capacity" mean in this work: residual rank, trainable parameter count, or effective dimensionality of the learned representation? Beyond showing that rank-constrained compression preserves retrieval, what additional scientific insight do the results provide, and can they identify a meaningful capacity boundary?

  3. Since both models are trained from scratch, how can the authors distinguish redundancy in the original residual transformation from adaptation or regularization effects introduced by the low-rank parameterization? Could a comparison with independently trained dense ENIGMA models clarify the representation results, and how does the upstream 184-dimensional representation affect the capacity requirement?

  4. How far should the findings generalize beyond the current ENIGMA/THINGS-EEG2 setting? Why is single-trial retrieval the primary endpoint, and is there evidence that the capacity behavior changes with EEG repetitions or visual targets? Could the authors also clarify the exact reference implementation and training objective, and provide the main comparison code for independent verification?

1 comment

Thanks for the efforts. Yet my main concerns remain unresolved. The response narrows the original claim without providing further evidence for the actual capacity requirement or the necessity of the residual correction. I therefore maintain my rating.

Reviewer_8f5wOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether ENIGMA's final residual transformation can be compressed without substantially reducing EEG-to-image retrieval performance. The authors replace the dense 1024 × 1024 residual matrix with a rank-224 factorization, train the EEG model from scratch, and evaluate it on five development-independent THINGS-EEG2 participants (S03–S07) with two seeds in a matched 20-fit design. The compact model has 41.97% fewer trainable EEG-model parameters and meets the paper's prespecified single-trial retrieval retention criterion. The paper also reports a development-only rank sweep, representation analyses, an independently reproduced NICE baseline under a different protocol, and a generation experiment using 32 of the 200 test identities. The results support parameter reduction with retained retrieval performance in this particular ENIGMA/THINGS-EEG2 setup; they do not establish a minimum sufficient rank or broad generalization across architectures, datasets, or visual targets.

Critical strengths
  1. The controlled ENIGMA-versus-compact comparison matches the target representation, training exposure, optimizer schedule, participant protocol, and retrieval evaluation. The rank and retention criterion are reported as frozen before confirmation, and the test includes five separately fitted participants across two seeds.
  2. The paper reports participant-level effects and uncertainty, tests multiple repetition counts, and includes shuffled-EEG controls in the generation experiment. It also reports that the single-trial generation condition narrowly misses the retention criterion on the crossed lower bound while the participant-t lower bound passes, which helps delimit the positive high-repetition finding.
Critical weaknesses
  1. The capacity intervention is tested on only one architecture, one EEG dataset, and one 1,024-dimensional target representation. The NICE reproduction uses a different target (CLIP-768) and training protocol and therefore does not test the capacity claim. Because the five confirmation participants are fitted separately, the results demonstrate participant-specific replication, not cross-subject transfer or generalization to another dataset or architecture.
  2. The experiments do not establish a lower bound on how much alignment capacity is needed. Only one low-rank point (rank 224) is confirmed on the confirmation participants. The paper acknowledges that its development-only sweep, which includes two participants, does not identify a minimum sufficient rank. That sweep reports nearly identical Top-1 for rank 64 and rank 224 (7.053% vs. 7.067%), but neither rank 64 nor a lower-rank failure boundary is prospectively confirmed.
  3. The interpretation that the original dense residual transformation contains redundant capacity is not isolated by the experiment. Both the ENIGMA reference and the compact model are trained from scratch, and the full 1,024-dimensional identity pathway remains available in both. The paper acknowledges that Theorem 1 restricts only the learned residual correction, but no ablation freezes the encoder or removes the identity path. The results therefore show that an ENIGMA variant with a low-rank residual update, trained from scratch, can retain retrieval performance, but they do not establish that the retained performance reflects redundancy in the dense residual rather than compensation in the retrained encoder or identity path.
  4. The 42% reduction is in trainable EEG-model parameters, not in executed computation. Appendix C.3, Table 7 reports a 4.6% MAC reduction, essentially unchanged training time and batch throughput, a roughly 10.4% increase in batch-one latency, and negligible peak-memory reduction. The paper acknowledges this in §8.2 and frames low-rank factorization as a capacity-identification intervention rather than an execution-cost optimization. The scope of the parameter-efficiency claim should nevertheless be made explicit, since it does not translate into a comparable practical efficiency gain.
  5. The generation validation is narrow. It uses 32 of 200 test identities, one generator, one diffusion-noise seed per stimulus, one encoder seed, and one DINO-based metric family (DINOv2-small CLS cosine, with Top-1/Top-5 derived from the same features). All five participants contribute and the crossed bootstrap accounts for participant and image sampling, but the fixed 32-identity subset and single generator/seed still limit generalization. The compact model meets the stated retention criterion at 80 repetitions, but the single-trial crossed lower bound narrowly fails it. The evidence therefore supports a limited high-repetition conditioning result, rather than a general claim about generation utility.
  6. The external comparison is limited. NICE is the only independently reproduced baseline, and it uses a different visual target and training protocol. Other recent methods are discussed but not evaluated under a shared target and protocol, limiting the paper's comparative context.
Questions for authors
  1. Appendix B.1 states that rank 224 was fixed by the development-stage parameter-budget design, not selected from a rank sweep, and Appendix B.2's sweep was run afterwards. Given that post hoc sweep, would the authors prospectively confirm rank 64, or a lower rank, to test a lower-capacity boundary? The paper already states that it does not identify a minimum sufficient rank; a rank-64 confirmation would begin to address that question rather than only contextualizing the rank-224 operating point.
  2. Since both models are trained from scratch and the full identity pathway remains, what evidence supports attributing the retained performance specifically to redundancy in the original dense residual transformation? Would the authors consider an ablation that freezes the encoder, removes the identity path, or compares against a parameter-matched dense baseline?
  3. The retention claim is a prespecified non-inferiority claim, so the absence of superiority (p = 0.1875) is not itself a weakness. Two questions remain. First, how was the 0.5-percentage-point margin justified in practical terms beyond being inherited from the development plan (Appendix A.6), and what does it imply for the smallest practically meaningful degradation? Second, with five participants and between-participant effects ranging from −0.084 to +0.537 pp, what is the power to exclude a margin violation, and how robust is the retention decision to leaving out S07, which drives the positive mean?
  4. The generation stage fixes encoder seed 42 and one diffusion-noise seed per stimulus; the second encoder seed is used only for retrieval. Are the generation results stable across seed 43 and multiple noise seeds? How much would evaluating more of the 200 test identities change the DINO-based retention result? What is the basis for the 0.02-cosine retention margin, and what does it correspond to in terms of visible content?
  5. At k = 80, the compact-minus-ENIGMA difference is +0.550 pp under representation-first averaging (Table 5) but −0.150 pp under waveform-first averaging (Table 6). The paper describes both aggregation paths as closely matched. How should readers interpret this sign flip, and was one path prespecified for the secondary high-repetition claim?
Reviewer_eDf6OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

EEG-based visual decoding faces problems with computational cost. This work proposes a compact head for EEG feature projection and retains decoding performance compared to a commonly used head. They also performed some analysis of participant-level effects, repeat-count gain, and a formal non-inferiority test.

Critical strengths
  1. The work gave an extensive comparison between the compact head proposed and ENIGMA.
  2. They show some evidence for EEG-based visual decoding.
Critical weaknesses
  1. There are many EEG encoders and feature extraction methods that need further comparison.
  2. I don't understand the significance of reducing the computational cost from 1.4 M to 0.8M.
Questions for authors

Please read some references and do more thinking when you use AI for research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.