acceptodds
Withdrawn from ICLR 2027

Sample-Adaptive Expert Routing for EEG Visual Decoding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

EEG visual decoding analyzes the relationship between EEG and visual stimuli by aligning EEG to image. Because cortical responses are biased toward coarse, low-spatial-frequency content, recent work aligns EEG to blurred images that carry more of the visual information reflected in the EEG, but fixes the blur level and representation depth to a single global setting shared across all samples. This ignores the variability of EEG responses across subjects, stimuli, and samples, and hence the ideal degree of blur and level of abstraction is inherently sample-dependent. In this paper, we propose a novel sample-adaptive visual decoding method that composes the alignment target by conditioning on each EEG embedding. Building on a mixture-of-experts formulation, dynamic routers select, per sample, the blur level, encoder layer, and backbone of the visual representation, while two auxiliary objectives ground each expert in its own visual content and supervise the routers with the similarity between the EEG and each expert, respectively. Evaluated by zero-shot retrieval on two benchmarks, the proposed method achieves state-of-the-art performance, and our ablations confirm that the conditional routing and the two auxiliary objectives each contribute to the gains.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Electroencephalogram, Retrieval, Sample-adaptive, Router

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XBLhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. This paper proposes Sample-AdaptiVe Expert Routing (SAVER) for EEG-based visual decoding. Unlike existing methods that employ globally fixed visual alignment targets, SAVER dynamically selects visual representations conditioned on individual EEG embeddings.
  2. The method integrates multiple visual backbones, encoder layers, and image blur levels into an expert pool. Two EEG-conditioned routers adaptively aggregate these representations, while expert alignment and router guidance objectives improve representation learning and routing.
  3. Experiments on THINGS-EEG2 and THINGS-MEG demonstrate competitive improvements in zero-shot image retrieval across intra-subject, inter-subject, and cross-modality settings. Ablation studies further examine the contributions of adaptive routing and auxiliary objectives.
Critical strengths
  1. Well-motivated sample-adaptive alignment. The paper identifies an important limitation of globally fixed EEG-visual alignment targets and introduces a coherent framework for selecting representations according to individual neural responses. This provides a meaningful methodological extension to existing EEG visual decoding approaches.
  2. Strong empirical performance across evaluation settings. SAVER consistently improves retrieval accuracy over competitive baselines, particularly in the challenging inter-subject setting. The ablation studies also provide evidence that dynamic routing and auxiliary supervision contribute to the observed improvements.
Critical weaknesses
  1. Insufficient isolation of the benefits of adaptive routing. Although the ablations compare different routing schemes, the substantial gains introduced by router guidance make it difficult to determine how much improvement originates from genuinely sample-specific representation selection rather than additional supervision. More carefully controlled experiments are needed to substantiate the proposed mechanism.
  2. Limited evaluation of robustness and computational efficiency. The framework relies on 720 visual experts constructed from five pretrained backbones, introducing considerable representational and computational complexity. However, computational costs are not compared with competing methods. Moreover, the experiments use repetition-averaged neural signals and report mean accuracy over only three random seeds without uncertainty estimates, limiting the evidence for practical robustness and statistical reliability.
Questions for authors
  1. Can the authors provide additional controlled comparisons that isolate the contribution of EEG-conditioned routing from router guidance and increased model capacity, ideally under matched supervision and computational budgets?
  2. What are the training and inference costs of SAVER compared with the strongest baselines, particularly considering the 720 visual experts and query-dependent routing?
  3. How does SAVER perform on single-trial neural recordings rather than repetition-averaged signals? Can the authors also report standard deviations or confidence intervals to establish the reliability of the improvements?
Reviewer_77uxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

SAVER uses EEG-conditioned routing to combine representations across visual backbones, network layers, and blur levels and align them with brain-signal features. While the reported retrieval performance is promising, I have the following concerns regarding novelty, experimental validation, and scientific insight.

Critical strengths

The method of this paper is simple and its performance is advanced.

Critical weaknesses
  1. Limited methodological novelty and insufficient scientific insight. The method primarily combines multi-representation fusion, MoE-style soft routing, and auxiliary alignment losses for visual brain decoding. The results demonstrate practical value, but do not sufficiently establish a methodological contribution beyond existing adaptive fusion ideas. In particular, variation in routing weights across samples and subjects does not directly establish that the model captures meaningful differences in neural representations. The paper should explain which visual information is better aligned with brain signals, why this is the case, and whether the resulting relationships are stable and interpretable.

  2. Insufficient baseline comparisons and decoding validation. Although the paper includes uniform and globally learned static weighting, it lacks comparisons against non-MoE alternatives such as concatenation followed by projection or standard MLP-based fusion under comparable parameter and training budgets. It therefore remains unclear how much of the improvement comes from routing itself rather than additional capacity, auxiliary supervision, or richer visual features. The dynamic model in Table 4 also uses router guidance; the dynamic-routing result without this supervision should be included directly to separate their contributions. Furthermore, all decoding evaluations are restricted to retrieval. The paper neither establishes whether the learned representations support image reconstruction nor systematically examines which visual attributes they preserve. Retrieval accuracy alone does not establish how much information about color, shape, spatial structure, or semantics has been recovered. Reconstruction experiments or targeted attribute-probing analyses would provide stronger evidence about the content and limitations of the learned representations.

  3. Figure readability, writing clarity, and mathematical specification need improvement. Figure 1 uses very small labels, while Figure 3 lacks a color bar and sufficient axis information, making quantitative interpretation of routing weights difficult. Figure 2 does not clearly show how cross-routing, the adapter, and the auxiliary losses fit into the framework. Some explanations also exceed the evidence presented: Section 5.6 attributes failures to routing toward more strongly blurred experts without showing the corresponding routing statistics for successful and failed samples. The mathematical presentation should use consistent vector-stacking and matrix-dimension conventions, explicitly indicate stop-gradient operations, and clarify EEG feature normalization and contrastive-logit scaling in Eqs. (3)–(4). These issues affect both clarity and reproducibility.

  4. The expert pool is based more on broad enumeration than hypothesis-driven selection. The paper combines multilayer features from five visual backbones with twelve blur levels to construct 720 visual experts, but does not adequately justify these specific choices or connect them to testable hypotheses about neural representations. Low-spatial-frequency processing and hierarchical vision provide a broad motivation, but do not establish why this particular expert pool is appropriate. The design currently resembles constructing a large candidate set and relying on data-driven routing to assign weights, rather than selecting complementary representations based on explicit hypotheses. Ablations of individual backbones, layer depth, blur levels, and progressively added expert groups would help identify necessary and redundant components. The authors should also test whether a smaller, interpretable expert set achieves comparable performance. Without such analysis, the main contribution remains an engineering improvement rather than new insight into the correspondence between brain and visual representations.

Questions for authors

See the weakness.

Reviewer_438MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies EEG-based visual decoding as zero-shot image retrieval. It argues that existing methods fix the alignment target—the blur level, encoder layer, and backbone used to represent the stimulus—globally across all samples, whereas the visual information reflected in EEG varies across subjects, stimuli, and trials. The authors propose SAVER, which treats each (backbone, layer, blur) combination as a visual expert (720 experts in total) and uses two EEG-conditioned routers to compose, per sample, both the visual alignment target and the fused EEG representation. Two auxiliary objectives are introduced: expert alignment, which aligns each of the 12 EEG blur experts with the corresponding blur-level visual target, and router guidance, which supervises the routers with the cosine similarity between the EEG and each expert. Experiments report 200-way zero-shot retrieval on THINGS-EEG2 (intra- and inter-subject) and THINGS-MEG (intra-subject), i.e., across two recording modalities. SAVER outperforms 11 baselines, reaching 90.7% Top-1 intra-subject on THINGS-EEG2, 28.8% inter-subject, and 48.8% on THINGS-MEG. Ablations compare uniform, static, and dynamic routing and the two auxiliary losses. Code is provided in the supplementary material.

Critical strengths

Strong empirical gains, evaluated across two recording modalities. SAVER reaches 90.7% Top-1 on THINGS-EEG2 intra-subject versus 84.6% for the strongest baseline (HCF), 28.8% versus 23.4% inter-subject, and 48.8% versus 44.0% on THINGS-MEG, with improvements of 11.7 and 11.3 points on Sub5 and Sub9 and a reduced cross-subject standard deviation (5.4 → 2.4). The MEG model is re-trained with the same visual expert pool and hyperparameters; only the spatial convolution is adapted to the channel count. This is a meaningful evaluation across two recording modalities, though not a direct transfer of an EEG-trained model.

A well-motivated problem framing and a concrete, reusable design space. The paper correctly identifies that prior work fixes the alignment target globally, and unifying backbone, layer, and blur into a single expert pool conditioned on the EEG is a simple and general recipe. The uniform/static/dynamic comparison is the right high-level design in principle, and the code is provided in the supplementary material.

Critical weaknesses
  1. The ablation does not separate sample conditioning from architectural differences and router supervision. Table 4 is described as varying “only the router,” but the released code shows that the three rows differ in several respects. The reported numbers show that the full system works, but they do not isolate per-sample conditioning. The authors acknowledge joint contribution in Sec. 5.3, so the issue is not that they ignore the auxiliary objectives; it is that the experiments do not separate the contributions or establish that per-sample conditioning is necessary. A static control with matched EEG fusion, image adapter, retrieval loss, and training schedule, using LEA and an appropriately defined batch-averaged LRG objective, is missing.
  2. Protocol and reproducibility gaps that affect the reported state-of-the-art claims. (a) The 5% validation split is a random split over images after repetition averaging, not a concept-disjoint split. This is a difference in generalization conditions between validation and test, not test leakage or a failure of the zero-shot protocol; however, it means checkpoint selection is performed on seen concepts, and the paper should justify this choice or provide a concept-disjoint sensitivity analysis. (b) The code computes test accuracy every epoch and saves a test-selected checkpoint in addition to the validation-selected one. The evaluation script defaults to the validation-selected checkpoint, so I do not conclude that the paper used the test-selected one; the authors should confirm that all SAVER and ablation results use it and report the gap. (c) Table 2 omits MBP, ATS, and CognitionCapturer, and Table 3 reports only five baselines; the paper should state the selection criterion. (d) The MEG Sub4 result drops by 12.8 points (22.5 vs. 35.3). This suggests individual differences; it does not establish that the routing is unstable, but the regression is not explained. (e) The paper reports no parameter count, inference cost, or storage cost, and there is no visual-budget-matched comparison against the strongest baselines. These should be separated into offline feature extraction, cache storage, and online query cost; the 720 experts are feature representations, not 720 networks re-run per query. (f) The paper claims three seeds but reports no variance; standard deviations, confidence intervals, or paired tests are needed before interpreting small differences.
Questions for authors
  1. Section 4.2 states that the authors select the checkpoint with the best validation Top-1. The code also saves a test-selected checkpoint; the evaluation script defaults to the validation-selected one. Please confirm that all SAVER and ablation results use the validation-selected checkpoint, and report the gap between the two. The 5% validation split is by image, not by concept; this is a difference in generalization conditions, not test leakage, but it means checkpoints are selected on seen concepts. Was this intentional, and would a concept-disjoint split change the selected checkpoint?
  2. The released code shows that the uniform branch omits the image adapter, the static branch uses per-dimension weights plus four learnable constant features, the dynamic branch uses the fused EEG representation for the main loss while static/uniform use the raw encoder output, and the loss direction differs. The reported ablations establish the benefit of the full system, but do not cleanly separate sample conditioning from architectural differences and router supervision. Could you provide a static control with matched EEG fusion, image adapter, retrieval loss, and training schedule, using LEA and an appropriately defined batch-averaged LRG objective?
  3. Single-backbone SAVER and visual budget. The 720 experts come from 60 backbone×layer feature combinations × 12 blur levels. The strongest baselines use a single backbone, while MBP also uses multiple visual encoders (Sec. 2.1), so I am not claiming that all baselines are single-backbone. Could you report SAVER with a single backbone (e.g., CLIP ViT-B-32 only) and a visual-budget-matched comparison? Without this, the comparison conflates the visual ensemble with the routing mechanism.
  4. Table 2 omits MBP, ATS, and CognitionCapturer, all of which appear in Table 1; Table 3 reports only five baselines. Please state the selection criterion and, if possible, report their inter-subject and MEG numbers under your protocol. SAVER also drops by 12.8 points on THINGS-MEG Sub4; the text only notes that it is best on three of four subjects. The number suggests individual differences rather than routing instability, but what causes the regression?
  5. Router regularization and efficiency. The paper does not report router entropy or expert utilization. What is the average entropy of p and w, and what fraction of the 720 experts receives non-trivial weight? Concentrated use of a few experts may be a meaningful choice rather than a failure; the code contains a load-balancing loss that is disabled by default, and enabling it would be a reasonable supplementary analysis. Please also report separately (i) the offline cost of extracting image features for five backbones × twelve blur levels, (ii) the cache storage of the 720 expert features, and (iii) the online query cost.

Comments on the paper

1 comment

Thank you for your response. Overall, I believe the experimental results of this paper are trustworthy. However, my concern is that, as this paper is already conducting exhaustive searches to find as many visual features as possible for bidirectional alignment with brain embeddings, technically, multi-model distillation or fusion itself can already achieve good alignment results. Therefore, MoE might be unnecessary. Switching to the same parameter quantity would still work, and this would weaken the technical contribution of this paper. By the way, the MLP baseline submitted by the author in the Rebuttal is too weak. I don't think it is a strong baseline, so it still cannot prove the capability of MoE. Additionally, I think the technical contribution of this paper is too weak. Even if the performance is good, this paper does not provide any new technical insights or scientific insights. MoE is not a new concept in the ML community. Even the method in this paper, due to blending multiple visual features together, may prevent the brain signals from reconstructing images, which also undermines the interpretability of this method. I welcome the author to further discuss with me.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.