acceptodds
Under review as a conference paper at ICLR 2027

NeuroSketch: A Practical Design Recipe for Neural Decoding

Abstract

Neural decoding is fundamental to brain-computer interfaces, with growing applications in healthcare. Previous research has focused on leveraging signal processing and deep learning methods to enhance neural decoding performance. However, systematic guidance on architectural design for neural decoding remains limited. In this study, we develop NeuroSketch, a practical design recipe for neural decoding, through a basic architecture study followed by macro- and micro-level optimization. Comparing nine basic architectures, we find that CNN-2D outperforms other architectures in neural decoding tasks and explore its effectiveness from temporal and spatial perspectives. Building on this backbone, we combine gradual feature-map expansion and early downsampling at the macro level with grouped convolutions at the micro level. These choices form the recipe, which we instantiate as NeuroSketch-Base (1.4M parameters) and NeuroSketch-Large (4.2M parameters). The recipe is developed and evaluated through over 4,500 experiments across eight tasks spanning visual, auditory, and speech modalities and EEG, SEEG, and ECoG signals. Against ten baselines, the two variants collectively achieve the best accuracy on each task. Our code is available at https://anonymous.4open.science/r/NeuroSketch-anonymous-A5E4/.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural decoding, Architectural design, Brain signal

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_Nir7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

NeuroSketch presents a practical neural-decoding recipe that combines two-dimensional convolutions, gradual feature expansion, early downsampling, and grouped convolutions across multiple recording modalities and speech, visual, and auditory tasks.

Critical strengths

The study is unusually comprehensive, covering eight tasks, six datasets, multiple recording modalities, and numerous architectural alternatives. The appendices provide substantial implementation details, including model sizes, preprocessing procedures, training settings, baseline configurations, and supplementary analyses. The resulting Base and Large models are compact, practical, and frequently competitive across heterogeneous neural-decoding scenarios.

Critical weaknesses

Most headline results rely on random within-subject trial splits, primarily demonstrating subject-specific decoding rather than generalization to unseen participants. Repeated stimuli, shared sessions, or temporally adjacent trials may lead to overly favorable splits and inflated performance estimates. Furthermore, the architectural recipe is developed and evaluated on the same eight tasks, without an independent benchmark held out from architecture selection. This raises concerns about recipe-level overfitting. Cross-subject evaluation is limited to two tasks and three models, with improvements over the strongest baseline of only 0.8 and 1.2 percentage points. Performance on SEED-DV also remains near chance level, weakening claims of robust generalization to unseen participants.

Questions for authors
  • Can the authors provide block-wise, session-wise, or temporally separated evaluations to assess whether random trial splits lead to overly optimistic performance estimates?

  • Given that the recipe was developed and evaluated on the same eight tasks, how do the authors address the risk of recipe-level overfitting?

Reviewer_zShxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces NeuroSketch with the aim of providing systematic guidance on architectural design for neural decoding. By comparing nine basic architectures, they identify a 2D CNN as a suitable backbone. They develop NeuroSketch with two model sizes and evaluate them on eight tasks using EEG, SEEG, and ECoG recordings, with additional analyses of cross-subject performance, training-data requirements, and learned spatiotemporal features. Overall, this paper is well structured and demonstrates practical value.

Critical strengths
  1. This paper addresses a practical problem, and may provide insights in the model design for neural decoding.
  2. The experiment includes multiple decoding tasks and established baselines.
  3. The proposed model achieves competitive performance.
Critical weaknesses
  1. This paper combines and validates established architectural components, while the methodological novelty and new insights specific to neural decoding are still limited.
  2. The same training settings may not favour all baselines, and the evidence for spatial and temporal locality remains indirect.
  3. Although the paper includes several representative baselines, the authors should compare with leading methods for the corresponding decoding tasks to establish its SOTA performance.
Questions for authors

see above.

Reviewer_KWEoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces the problem that there is no systematic architectural guidance for neural decoding that is shared across tasks and recording types. It then proposes a macro-to-micro roadmap (as in ConvNeXt): it starts with CNN-2D, then refines it progressively with step widening, plus early downsampling and grouped convolution. This procedure yields two main products, NeuroSketch-Base and Large, with 1.4M and 4.2M parameters respectively, trained from scratch per subject. This is validated using 8 within-subject classification tasks from 6 external datasets (EEG, SEEG, ECoG) against 10 baselines, including 2 pretrained foundation models. The protocol is 3-fold with a fixed test set. The paper ends with the finding that the two models together reach the best or tied-best accuracy on all 8 tasks.

Critical strengths

The main strength of this paper is the breadth of the evaluation it performs. It includes 8 tasks covering 3 modalities (speech, visual, auditory) and 3 recording types (EEG, SEEG, ECoG), which is especially relevant since not many architecture studies cover invasive and non-invasive recordings together. This breadth is precisely what makes the main research question of the paper (is there a general recipe?) testable at all. I also appreciate that the authors released the full per-run results, which made it possible for me to check several results independently.

The paper also makes a second useful contribution by showing that compact CNNs trained from scratch can be competitive in within-subject decoding. For example NeuroSketch-Large (4.2M parameters) reaches 65.2% on the 61-word SEEG task (Du-IN), which is in the range of the published SOTA (62.7%). Du-IN is also the task where the evaluation protocol doesn’t seem to have any error, so I consider it the most trustworthy result. Small models trained from scratch still matter for neural decoding, especially when compute and data are limited, and are a useful counterpoint to large pretrained foundation models.

Critical weaknesses

A major flaw in the paper resides in the evaluation protocol, which invalidates 2 of the 8 tasks and inflates a third. With the OpenMIIR split, each song trial is cut into short windows that are assigned randomly to train and test, so windows from the same recording end up on both sides, and a model can do well just by recognizing the recording instead of decoding the music. Additionally, unlike the official OpenMIIR code, the released code does not skip the cue period, so the imagery data contains the cue clicks, which are played at the tempo of the song. This would explain the ~98% accuracy in Table 4, far above what has been published on this dataset (around 28% for perception, and imagery not above chance). With SEED-DV, clips are shuffled across blocks instead of using the official held-out-block split, so neighboring clips of the same class fall on both sides. Even with this, NeuroSketch (4.7%) stays below the 6.2% reported for GLMNet under the official protocol, and GLMNet is not included as a baseline.

Because of this the claim of best performance on all 8 tasks rests partly on tasks that measure recording identity rather than decoding. I would recommend splitting OpenMIIR by trial and excluding the cue period, using the official block-wise split for SEED-DV, and reporting the split unit, chance level and published SOTA for every task (even if only as extra columns in Table 1).

A second major issue is that the baselines don't seem to be reliably trained. In the released per-run results, SPaRCNet predicts a single class in every run on Chisco, and SPaRCNet and CBraMod sit exactly at chance in most SEED-DV runs, even though the same SPaRCNet reaches 97% on another task. Other baselines are far below their published numbers (e.g., EEG-Conformer reaches 20.5% on Du-IN, while the Du-IN paper reports 45.8%), and CBraMod is fine-tuned with a 10× higher learning rate and 10× more epochs than its authors use. On top of that, several of the reported wins are ties or within one standard deviation, and the code for the baselines is not released.

As a result, the headline margins (e.g., +26 points on Du-IN) partly reflect failed baseline runs, and the SOTA claim is not established. I would recommend tuning each baseline on validation data or using its published recipe, reproducing at least one published number per dataset, adding dataset-specific SOTA models (e.g., GLMNet), and reporting ties as such.

Other minor points: the claim that the gain comes "rather than from parameter count alone" is only half-true, as Base and Large differ by 18 points on Du-IN. In Figure 1a, the unweighted average over tasks with very different chance levels (0.5% to 50%) lets the high-accuracy tasks dominate, so a mean rank or a chance-normalized score would be better. In Figure 1b, the axes seem to be rescaled independently, which makes small differences look much bigger than they are. In Table 4, an F1 score seems to be reported as accuracy (0.05).

Questions for authors
  1. [Clarification] OpenMIIR protocol. Could the authors confirm how OpenMIIR was split (by trial, window or block) and whether the cue period is included in the imagery epochs? Results with a trial-level split and without the cue would help a lot here. If NeuroSketch still leads under that protocol, I would raise my assessment.

  2. [Clarification] Baseline training. How were the baseline hyperparameters chosen? It would be good to have an explanation for the SPaRCNet runs that predict a single class and the CBraMod runs at chance on SEED-DV, and to check whether the published EEG-Conformer result on Du-IN (45.8%) can be reproduced in the authors’ pipeline.

  3. [Clarification] Selection of the recipe. Were the design decisions in Section 2 made on validation or on test accuracy? They seem to use the same 8 tasks and subjects as the final evaluation, from single runs without error bars, which would make the gains optimistic. How were the Base and Large widths and depths chosen? Testing the frozen recipe on a held-out dataset (e.g., BCI-IV-2a or a P300 dataset) would show if it really generalizes.

  4. [Clarification] GFLOPs. In Table 3, the GFLOPs are given with no unit or input size, and values like 1,287 seem implausibly large per sample. What exactly was counted? The ordering also contradicts the paper’s own formulas: a separable convolution should cost about 1/9 of a standard one and a 4-group convolution 1/4, yet separable is listed as more expensive (492 vs 482). Since the efficiency claims (67.8%, 41.1% and 35% fewer FLOPs) are computed from this column, I think this should be clarified.

  5. [Clarification] ε-rank and downsampling. The ε-rank ratio is normalized by min(s, d), and in the CNN s shrinks with each downsampling step, so the ratio grows towards 1 almost by construction and is not really comparable with MedFormer’s. How do the authors support the conclusion that CNN-2D learns more information-rich representations? Relatedly, NeuroSketch-Base and Large are said to differ only in width and depth, but they also differ in the number of downsampling steps. Could the authors clarify this?

  6. [Clarification] What the 3×3 kernel spans. The text says that a 3×3 kernel "spans three adjacent recording channels", but in the released code the adjacent rows are the 3 time-phases of the same channel. Which one is intended? If it is the latter, the spatial-locality explanation might need rephrasing.

  7. [Suggestion] Channel-permutation control. To test the spatial-locality explanation directly, the authors could retrain with shuffled channels. If accuracy doesn't change, then the advantage of CNN-2D would not come from local cross-channel patterns.

Comments on the paper

1 comment
Area_Chair_pngqOpenReview ↗

Please Engage in the Author–Reviewer Discussion

Dear Reviewers,

As the author–reviewer discussion period is approaching its deadline (November 18, AoE), I would like to kindly remind you to review the authors' rebuttals and actively engage in the discussion as soon as possible.

Please address any outstanding questions or concerns and update your assessments and ratings if appropriate.

Thank you very much for your time and efforts!

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.