acceptodds
Under review as a conference paper at ICLR 2027

RPA: Residual Patch-Token Adapter for Image Retrieval from EEG and MEG

Abstract

Visual decoding from non-invasive MEG and EEG (M/EEG) aims to identify viewed images through representation learning that aligns brain signals with corresponding images. Most existing MEG and EEG (M/EEG) visual decoding methods align brain signals with a single global embedding extracted from a pretrained visual encoder, leaving open whether intermediate patch representations, which preserve richer and more granular rich visual information, can improve representation learning. To address this question, we introduce the Residual Patch Adapter (RPA), a lightweight, modular adapter that leverages all patch tokens from an intermediate layer of a ViT visual encoder for alignment. Through extensive ablation analyses, we first show that pooling or masking patch tokens degrades the learned representation, demonstrating that retaining the full set of patch tokens is important for EEG alignment, while the CLS token provides little unique information. We then use a series of six quantitative feature analyses to show that both higher-level semantics and lower-level visual features, including color and texture, are essential for this EEG-to-image alignment. Under current protocols, our system achieves Top-1 accuracies of 95.4% within-subject and 35.5% cross-subject on THINGS-EEG2, and 65.2% and 6.7%, respectively, on THINGS-MEG, achieving state-of-the-art (SOTA) performance across both datasets. Evaluations with alternative brain encoders, including pretrained EEG foundation models, demonstrate that the approach extends beyond the projection-based EEG encoder. Furthermore, we provide a plug-and-play interface that allows RPA to be replaced by convolution, attention, or ConvNeXt alternatives. Together, these findings provide significant insight into M/EEG-to-image representation learning by establishing design principles for leveraging the latent space of visual encoders, and open new directions for brain–image alignment and non-invasive brain–computer interface (BCI).

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, MEG, EEG to Image

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K57uOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RPA is a small trainable adapter on the frozen patch tokens of OpenCLIP ViT-H/14 layer 11: a residual convolutional bottleneck followed by average pooling and a linear map, trained contrastively with the UBP EEGProjection encoder. It reports 95.4% within-subject and 35.5% leave-one-subject-out top-1 on THINGS-EEG2, and 65.2% and 6.7% on THINGS-MEG. Supporting experiments cover layer sweeps over seven visual encoders, token pooling and masking, linear attribute probes, other EEG encoders and a reconstruction pipeline.

Critical strengths
  1. The controlled comparison at one layer is clear and the effect is large: CLS token 69.5%, patch mean 77.0%, pointwise RPA 93.0%, local RPA 95.4%, and every other adapter in Table 9, several of them much larger, does worse. This is the result the method stands on.
  2. The layer sweep is useful beyond this paper. Every encoder peaks before its final layer for both EEG and MEG, and Table 14 shows that layer choice can flip encoder rankings, a caution for anyone comparing visual backbones here.
Critical weaknesses
  1. Configuration was selected on the test set. Appendix D states that layer 11 was chosen because it gave the highest test top-1 after a full depth sweep, the MEG projection size of 64 is the best of eight test-evaluated settings (Table 20), and the adapter and bottleneck comparisons are also test numbers. The 3% validation split only picks checkpoints. With a 200-image gallery and margins such as 6.7% against 5.4% for MEG cross-subject, the headline numbers are optimistic.
  2. The state-of-the-art table uses published numbers from pipelines with different channels, windows and protocols, so only Table 3 is controlled. There the CLS and patch-mean controls get a linear head while RPA gets a nonlinear one, so the claim that the CLS token adds little is not separated from adapter capacity. The text also needs a careful edit (typos, unnumbered appendix references).
Questions for authors
  1. Clarification: what do the headline numbers become if the layer, time window, adapter and MEG projection size are selected on a validation gallery drawn from training concepts?
  2. Experiment: a capacity-matched nonlinear head on the CLS token and on the patch mean.
  3. Clarification: please tabulate single-trial accuracy (Figure 5D) for RPA and the CLS baseline.
  4. Experiment: re-run the two strongest baselines inside your pipeline with the same channels and window.
4 comments
AuthorsOpenReview ↗

Response to Reviewer K57u

Thank you for recognizing the strength of our controlled adapter comparison and the usefulness of the visual-layer analysis. We appreciate your constructive suggestions. All additional experiments and clarifications have been integrated into the revised manuscript, with changes highlighted in blue.

Q1. Validation-based model selection

We clarified the original model-selection procedure and conducted expanded EEG and MEG evaluations, selecting configurations using validation rather than test accuracy.

We corrected the ambiguous wording in Appendices A and D. The original layer search evaluated a subset of candidate blocks, and the visual layer, temporal window, adapter configuration, and MEG projection width were selected using validation. For EEG, each run holds out 3% of training samples for validation. Checkpoints are selected by validation retrieval Top-1, and the expanded layer sweep selects the block with the highest mean validation Top-1.

EEG visual-layer selection. Appendix K (Table 22 and Figure 14) provides the validation-selected results and the corresponding validation/test accuracy curves across evaluated blocks.

Visual encoderProtocolBlockVal. Top-1 (%)Test Top-1 (%)
CLIP ViT-H/14Within1057.45 ± 0.3296.12 ± 0.30
CLIP ViT-H/14LOSO1115.80 ± 0.1935.47 ± 0.33
DINOv2-L/14Within1253.80 ± 0.2794.25 ± 0.21
DINOv2-L/14LOSO1413.65 ± 0.2934.97 ± 0.86

Figure 14 shows that validation and test performance both favor intermediate visual layers, although the exact maxima may differ. In particular, CLIP-H under LOSO selects block 11 by validation, even though block 12 achieves a slightly higher test accuracy (36.12% versus 35.47%). We report the validation-selected block 11 result, rather than the test maximum.

MEG projection-width selection. We additionally evaluated projection widths from 32 to 160, together with a direct 271-sensor control. Appendix O (Table 28) reports the complete validation and test results. The comparison below includes the original width and the widths selected in the additional sweep.

ProtocolWidthVal. Top-1 (%)Test Top-1 (%)
WithinK=6417.46 ± 0.3461.38 ± 1.22
WithinK=160 (selected)17.98 ± 0.4265.58 ± 0.50
LOSOK=647.21 ± 0.326.04 ± 0.06
LOSOK=96 (selected)7.40 ± 0.186.51 ± 0.39

The additional MEG sweep shows a broad validation plateau among evaluated widths K≥64. Validation selects K=160 within subject and K=96 under LOSO. The K=64 entries above come from this additional sweep and are distinct from the original K=64 benchmark results.

The original benchmarks remain reported separately (EEG: 95.4%/35.5%; MEG: 65.2%/6.7%, within-subject/LOSO). The additional validation-selected results retain comparable performance and support the intermediate-layer finding without selecting configurations by test accuracy.

Q2. Capacity-matched nonlinear controls

We added the requested nonlinear CLS and patch-mean baselines, together with a per-token MLP control and CLS fusion, under matched model capacity and training settings. The MLP controls match RPA's parameter count within 0.023%.

Visual adapterWithin Top-1 (%)LOSO Top-1 (%)
RPA95.50 ± 0.2535.47 ± 0.33
CLS → MLP70.32 ± 0.3121.97 ± 0.41
Patch mean → MLP74.48 ± 1.1624.88 ± 0.19
Per-token MLP → mean91.90 ± 0.4334.45 ± 0.33
RPA + CLS95.37 ± 0.2935.55 ± 0.19

Applying the same MLP before pooling rather than after pooling improves Top-1 by 17.42/9.57 points within-subject/LOSO. RPA provides a further 3.60/1.02-point improvement over the per-token MLP.

Adding CLS to RPA produces no consistent mean improvement across the two regimes. These results strengthen the original controlled adapter comparison (Appendix M, Tables 25–26).

Q3. Single-trial retrieval

We added the requested single-trial results for RPA and the capacity-matched nonlinear CLS control, alongside their 80-repetition results.

ProtocolVisual adapterSingle trial (%)80 repetitions (%)
WithinRPA22.45 ± 0.3695.50 ± 0.25
WithinCLS → MLP12.41 ± 0.0370.32 ± 0.31
LOSORPA6.09 ± 0.0735.47 ± 0.33
LOSOCLS → MLP4.51 ± 0.0021.97 ± 0.41

RPA outperforms the CLS control by 10.04 points within subject and 1.58 points under LOSO on individual trials. Its advantage therefore persists without test-time repetition averaging, although both methods benefit substantially from averaging. Training remains unchanged (Appendix N, Table 27).

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer K57u,

We sincerely appreciate your careful review and constructive suggestions. We hope our additional experiments and explanations have addressed your concerns, and we would welcome your feedback on our response. We would be glad to clarify any remaining points.

Regards, Authors

AuthorsOpenReview ↗

Response to Reviewer K57u

Q4. Matched baseline evaluation

We reran Shallow Alignment and BlurPerception using the same EEG channels, temporal windows, splits, repetition averaging, and validation-based checkpoint selection as RPA.

MethodWithin Top-1 (%)LOSO Top-1 (%)
Shallow Alignment84.32 ± 0.3823.32 ± 0.13
BlurPerception76.13 ± 0.9218.28 ± 0.48
RPA95.50 ± 0.2535.47 ± 0.33

RPA exceeds Shallow Alignment by 11.18/12.15 points within-subject/LOSO and also outperforms BlurPerception. Shallow Alignment uses the same CLIP-H visual backbone, while BlurPerception retains its original RN50 backbone. We could not locate a public HCF implementation for a comparable rerun.

We also quantified the channel-selection effect: using 17 posterior channels instead of all 63 improves RPA's within-subject/LOSO Top-1 from 87.58/32.53% to 95.50/35.47%. The baseline reruns and channel comparison appear in Appendix L (Tables 23–24).

Presentation

We corrected typos, notation inconsistencies, and appendix references throughout the revised manuscript.

Thank you again for your helpful comments. We hope the added experiments and clarifications address your main concerns, and we welcome any further questions.

AuthorsOpenReview ↗

Further Follow-Up on Our Rebuttal

Dear Reviewer K57u,

Thank you for your careful assessment and insightful suggestions. We have incorporated additional experiments and revised the manuscript to better address the issues you identified. We hope these updates provide a clearer picture of our contributions and would greatly value your perspective on the revised work.

We remain happy to provide further clarification or discuss any outstanding concerns.

Best regards,
The Authors

Reviewer_SnttOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper targets the problem of visual decoding from EEG/MEG signals. Prior work mainly align M/EEG embeddings with the final layer CLS token from pretrained vision encoders. This paper, instead, explores the question of whether using patch token embeddings from intermediate layers as alignment targets can improve the decoding performance and how the patch tokens should be aggregated and used. It introduces Residual Patch Adapter (RPA) a lightweight and modular adapter that leverages and integrates all patch tokens from intermediate layers of the vision model and provides a target visual embedding for contrastive alignment. The paper shows that visual information is distributed across multiple patch tokens and only using CLS discards much of this information. The proposed method is evaluated on both image retrieval and image reconstruction tasks. The paper shows an impressive top-1 (for 200 test samples) retrieval accuracy of 95.4% and 35.5% on THINGS-EEG2 for within- and across-subject decoding. Ablation studies of number of patch tokens used, the vision encoder layer they are extracted from, and a comparison with the CLS token show that the full set of patch tokens extracted from intermediate layers are better targets for visual decoding from M/EEG signals.

Critical strengths

The results of the paper show significant improvement over baselines only by changing the target embedding layer depth and how the tokens are aggregated. If this is not due to leakage of test information to training, it is an interesting finding that should be considered when designing M/EEG-to-image models.

Critical weaknesses
  1. My major concern is about leaking test data information to model training. The results are impressive and within-subject top-1 retrieval accuracy is almost saturated (95.4%) and top-5 accuracy is 100% for some subjects on THINGS-EEG2 (Table 5)! The authors themselves mentioned on Appendix D that "We selected OpenCLIP ViT-H/14 layer 11 because it achieved the highest EEG Top-1 test accuracy after a full depth sweep among ViT-H/14". This is really concerning given that there are probably also other parameters such as the number of patch-tokens or RPA hyper parameters tuned this way. I see that the authors mentioned elsewhere, such as in Algorithm 1, that they use a validation set to choose the best RPA and brain encoder checkpoint, but the hyperparameters such as the vision encoder’s layer depth are not decided there.

  2. The authors use the 17 channels corresponding to visual perception regions of the brain while some of the baselines such as ATM and NICE use all 63 channels. So, the comparison does not seem to be fair.

Questions for authors
  1. I think the most important thing that could affect the results is a leakage in the depth-selection experiment; Table 3 shows that if we take the CLS token from layer 11 instead of the final layer, it already results in ~19% of improvement (compared to UBP row in table 1), which is huge! So, I suggest the authors repeat the depth selection experiment, this time without the test set, and report the validation results. Then, they repeat the main experiments of the paper with the best hyperparameters from validation and not test, and report the new results.

  2. My second question is about image reconstruction pipeline (section I.1). The reconstruction pipeline in Li et al. (2024) or Scotti et al. (2023) uses a combination of Info-NCE and MSE losses between the CLIP image embeddings and brain embeddings or the predicted image embedding from the prior. On the other hand, equation 9 uses Info-NCE loss for both paths and the alignment is between the brain and image embedding for both. I appreciate if the authors:

    • Clarify the relationship between r and z.
    • Clarify how the prior is trained itself.
    • Explain why aligning the same EEG embedding, z_b, with two different target visual embedding should help and does not collapse. Related to this, I think Table 21 requires at least one more row for pooled-CLIP baseline.
  3. Table 17 shows the retrieval accuracy when the targets are isolated image features. The authors use this to show that EEG contains information about different features of images and thus it makes sense to use intermediate layers as targets. However, they don’t show how these features are predicted from different layers of the vision model itself.

  4. The authors mention in section 4.5 that "all selected layers reach their highest decoding accuracy ... with most peaking at 170 ms, which temporally overlaps with the N170, an occipitotemporal component associated with early face processing.", but, it's not clear how a face processing event-related potential motivates processing natural images.

  5. How were CBraMod, LaBraM and BIOT adapted to 17 channels at 250 Hz?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Sntt [1/2]

Thank you for recognizing the strength of our controlled adapter comparison and the usefulness of the visual-layer analysis. We appreciate your constructive suggestions. All additional experiments and clarifications have been integrated into the revised manuscript, with changes highlighted in blue.

Q1. Validation-based model selection

We clarified the original model-selection procedure and conducted expanded EEG and MEG evaluations, selecting configurations using validation rather than test accuracy.

We corrected the ambiguous wording in Appendices A and D. The original layer search evaluated a subset of candidate blocks, and the visual layer, temporal window, adapter configuration, and MEG projection width were selected using validation. For EEG, each run holds out 3% of training samples for validation. Checkpoints are selected by validation retrieval Top-1, and the expanded layer sweep selects the block with the highest mean validation Top-1.

EEG visual-layer selection. Appendix K (Table 22 and Figure 14) provides the validation-selected results and the corresponding validation/test accuracy curves across evaluated blocks.

Visual encoderProtocolBlockVal. Top-1 (%)Test Top-1 (%)
CLIP ViT-H/14Within1057.45 ± 0.3296.12 ± 0.30
CLIP ViT-H/14LOSO1115.80 ± 0.1935.47 ± 0.33
DINOv2-L/14Within1253.80 ± 0.2794.25 ± 0.21
DINOv2-L/14LOSO1413.65 ± 0.2934.97 ± 0.86

Figure 14 shows that validation and test performance both favor intermediate visual layers, although the exact maxima may differ. In particular, CLIP-H under LOSO selects block 11 by validation, even though block 12 achieves a slightly higher test accuracy (36.12% versus 35.47%).

MEG projection-width selection. We additionally evaluated projection widths from 32 to 160, together with a direct 271-sensor control. Appendix O (Table 28) reports the complete validation and test results. The comparison below includes the original width and the widths selected in the additional sweep.

ProtocolWidthVal. Top-1 (%)Test Top-1 (%)
WithinK=6417.46 ± 0.3461.38 ± 1.22
WithinK=160 (selected)17.98 ± 0.4265.58 ± 0.50
LOSOK=647.21 ± 0.326.04 ± 0.06
LOSOK=96 (selected)7.40 ± 0.186.51 ± 0.39

The additional MEG sweep shows a broad validation plateau among evaluated widths K≥64. Validation selects K=160 within subject and K=96 under LOSO. The K=64 entries above come from this additional sweep and are distinct from the original K=64 benchmark results.

The additional validation-selected results retain comparable performance and support the intermediate-layer finding without selecting configurations by test accuracy.

Q2. Capacity-matched nonlinear controls

We added the requested nonlinear CLS and patch-mean baselines, together with a per-token MLP control and CLS fusion, under matched model capacity and training settings. The MLP controls match RPA's parameter count within 0.023%.

Visual adapterWithin Top-1 (%)LOSO Top-1 (%)
RPA95.50 ± 0.2535.47 ± 0.33
CLS → MLP70.32 ± 0.3121.97 ± 0.41
Patch mean → MLP74.48 ± 1.1624.88 ± 0.19
Per-token MLP → mean91.90 ± 0.4334.45 ± 0.33
RPA + CLS95.37 ± 0.2935.55 ± 0.19

Applying the same MLP before pooling rather than after pooling improves Top-1 by 17.42/9.57 points within-subject/LOSO. RPA provides a further 3.60/1.02-point improvement over the per-token MLP. However, normalization and bottleneck width also differ, so this additional gain cannot be attributed solely to the 3×3 convolution.

Adding CLS to RPA produces no consistent mean improvement across the two regimes. These results strengthen the original controlled adapter comparison (Appendix M, Tables 25–26).

Q3. Single-trial retrieval

We added the requested single-trial results for RPA and the capacity-matched nonlinear CLS control, alongside their 80-repetition results.

ProtocolVisual adapterSingle trial (%)80 repetitions (%)
WithinRPA22.45 ± 0.3695.50 ± 0.25
WithinCLS → MLP12.41 ± 0.0370.32 ± 0.31
LOSORPA6.09 ± 0.0735.47 ± 0.33
LOSOCLS → MLP4.51 ± 0.0021.97 ± 0.41

RPA outperforms the CLS control by 10.04 points within subject and 1.58 points under LOSO on individual trials. Its advantage therefore persists without test-time repetition averaging, although both methods benefit substantially from averaging. Training remains unchanged (Appendix N, Table 27).

AuthorsOpenReview ↗

Response to Reviewer Sntt [2/2]

Q4. Matched baseline evaluation

We reran Shallow Alignment and BlurPerception using the same EEG channels, temporal windows, splits, repetition averaging, and validation-based checkpoint selection as RPA.

MethodWithin Top-1 (%)LOSO Top-1 (%)
Shallow Alignment84.32 ± 0.3823.32 ± 0.13
BlurPerception76.13 ± 0.9218.28 ± 0.48
RPA95.50 ± 0.2535.47 ± 0.33

RPA exceeds Shallow Alignment by 11.18/12.15 points within-subject/LOSO and also outperforms BlurPerception. Shallow Alignment uses the same CLIP-H visual backbone, while BlurPerception retains its original RN50 backbone. We could not locate a public HCF implementation for a comparable rerun.

We also quantified the channel-selection effect: using 17 posterior channels instead of all 63 improves RPA's within-subject/LOSO Top-1 from 87.58/32.53% to 95.50/35.47%. The baseline reruns and channel comparison appear in Appendix L (Tables 23–24).

Presentation

We corrected typos, notation inconsistencies, and appendix references throughout the revised manuscript.

Thank you again for your helpful comments. We hope the added experiments and clarifications address your main concerns, and we welcome any further questions.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer Sntt,

Thank you again for your valuable feedback. We would be grateful if you could review our response and let us know whether the additional experiments and clarifications have addressed your concerns. We welcome any further comments or questions.

Regards, Authors

Reviewer_mm6POpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies image retrieval from non-invasive EEG/MEG recordings, focusing on how visual representations should be constructed for alignment with brain signals. The method freezes a pretrained visual encoder, extracts the complete patch-token grid from an intermediate layer, discards the CLS token, and adapts the grid using a residual bottleneck comprising 1×1, 3×3, and 1×1 convolutions, GroupNorm, and GELU. Global average pooling and a linear projection then produce the visual embedding. The brain encoder and visual adapter are jointly trained using a symmetric contrastive loss. For MEG, the method additionally learns a sensor-channel projection.

Critical strengths
  1. The paper reports practically meaningful and relatively consistent improvements in retrieval performance across the evaluated settings.
  2. The RPA visual adapter has approximately 2.562M parameters, substantially fewer than some of the attention-based and ConvNeXt alternatives evaluated in the paper. EEG-Conformer and CBraMod paired with RPA also achieve results close to those obtained with the default encoder. This suggests that the observed strong performance is not confined to the EEGProjection configuration. The module is simple, and its evaluation extends beyond a single brain encoder.
Critical weaknesses
  1. Appendix D on page 19 explicitly states that OpenCLIP ViT-H/14 layer 11 was selected because it achieved the highest EEG Top-1 test accuracy in a full layer sweep. Table 14 on page 20 further compares the maximum of six test-set scores for each encoder, and the accompanying text states that the benchmark LOSO result of 35.5% comes from a full layer sweep. Thus, the concern is not the absence of a validation set: although the paper reports validation-based checkpoint selection, the visual layer remains an important hyperparameter selected using test performance. Performance varies substantially across layers, and layer choice can even change the relative ranking of encoders. Consequently, the selected best test result cannot be treated as an unbiased estimate of generalization performance, and the comparison does not adequately establish the claimed advantage over published baselines.
  2. In Table 3, moving from patch-mean pooling to pointwise RPA improves accuracy by approximately 15.95 percentage points, while adding local convolution yields a further improvement of approximately 2.42 percentage points. The contributions of nonlinear adaptation before pooling and local spatial mixing therefore need to be distinguished. A post-pooling nonlinear MLP baseline with comparable capacity is currently missing. Moreover, GroupNorm in pointwise RPA shares statistics across spatial positions, so this variant should not be interpreted as strictly independent per-token processing.
Questions for authors
  1. Does the statement on page 19 accurately describe the use of the highest test accuracy to select the visual layer? Which data were used to select the visual layer, temporal window, and MEG projection dimensionality, respectively? Can the authors provide final results and matched baseline comparisons under an evaluation protocol independent of the data used for model selection?
  2. With the same visual layer, brain encoder, and training budget, how do a capacity-matched “MLP after mean pooling” and “per-token nonlinear transformation followed by pooling” compare? Does adding the CLS token to RPA provide a consistent improvement?
  3. Would the authors consider revising the claim that color and texture are “essential” to the more limited conclusion that they are decodable using independent probes and are consistent with the proposed interpretation? If the necessity claim is retained, what direct evidence supports it specifically for RPA?
2 comments
AuthorsOpenReview ↗

Response to Reviewer mm6P

Thank you for recognizing the retrieval gains, compact design, and results across brain encoders. Your questions prompted clearer selection reporting and more informative controls.

Q1 / Critical Weakness 1: model selection and matched evaluation

We clarified that configurations and checkpoints were selected by validation, added expanded validation-selected EEG and MEG results, and reran baselines under matched EEG settings.

We corrected the ambiguous wording in Appendices A and D; the selection procedure covers the visual layer, EEG window, adapter/bottleneck settings, and MEG width. The original layer search sampled candidate blocks.

Each EEG run holds out 3% of training samples and reports test accuracy at the best-validation checkpoint. The expanded sweep selects blocks by mean validation Top-1, giving the following results (Appendix K, Table 22 and Figure 14):

BackboneWithin-subject block / test Top-1 (%)LOSO block / test Top-1 (%)
CLIP ViT-H/1410 / 96.12 ± 0.3011 / 35.47 ± 0.33
DINOv2-L/1412 / 94.25 ± 0.2114 / 34.97 ± 0.86

Both validation and test favor intermediate layers, although their exact maxima can differ. The six-point test maxima were descriptive summaries; Appendix K now provides the validation-selected comparison. These expanded results are reported separately from the original block-11 comparisons. For MEG, validation selects K=160 within subject and K=96 under LOSO, yielding 65.58 ± 0.50% and 6.51 ± 0.39% test Top-1; the original benchmark retains K=64 (Appendix O, Table 28).

We also reran Shallow Alignment and BlurPerception with matched EEG inputs and validation-based checkpoint selection (Appendix L, Table 24). RPA exceeds Shallow Alignment by 11.18/12.15 Top-1 points within subject/LOSO and also outperforms BlurPerception. Shallow Alignment uses the same CLIP-H encoder; BlurPerception retains its released RN50 encoder.

Q2 / Critical Weakness 2: nonlinear processing, pooling, and CLS fusion

We added capacity-matched nonlinear controls showing the benefit of processing patches before pooling, while adding CLS to RPA gives no consistent mean improvement across the two EEG regimes.

We compared the adapters using the same frozen CLIP-H block-11 features, EEG architecture, inputs, and training budget, without additional tuning. The MLP controls match RPA's parameter count within 0.023% (Appendix M, Tables 25–26).

Visual adapterWithin-subject Top-1 (%)LOSO Top-1 (%)
RPA95.50 ± 0.2535.47 ± 0.33
CLS → MLP70.32 ± 0.3121.97 ± 0.41
mean → MLP74.48 ± 1.1624.88 ± 0.19
per-token MLP → mean91.90 ± 0.4334.45 ± 0.33
RPA + CLS95.37 ± 0.2935.55 ± 0.19

Applying the same MLP before pooling improves Top-1 by 17.42/9.57 points within subject/LOSO. This control processes each token independently, with LayerNorm within each token, addressing the spatial coupling introduced by GroupNorm in our earlier pointwise control.

RPA gains a further 3.60/1.02 points over the per-token MLP. Because normalization and bottleneck width also differ, this remaining gain cannot be attributed entirely to the 3×3 convolution. Adding a learned scalar-weighted CLS token changes mean Top-1 by −0.13/+0.08 points.

Q3: necessity of color and texture

We revised the claim to state that independent probes can predict color and texture from EEG, consistent with our interpretation, without claiming necessity for RPA.

This change appears in the abstract, Section 4.4, and conclusion. The added image-side probes examine descriptor predictability across visual layers (Appendix P, Figure 15).

Thank you again for the helpful comments.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer mm6P,

Thank you again for your thoughtful review. We would appreciate it if you could take a look at our rebuttal and let us know whether the additional experiments and clarifications address your concerns. We would be happy to discuss any remaining questions.

Regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

Thank you to the reviewers for the careful assessment. We appreciate mm6P’s recognition of the retrieval gains and compact adapter, and K57u’s comments on the controlled comparisons and usefulness of the layer analysis. Sntt also noted the potential importance of the finding, conditional on resolving the model-selection concern.

The revision addresses these concerns with three main comparisons:

  • Validation-selected evaluation. We clarified that the original configurations and checkpoints used validation. The expanded EEG sweep selects intermediate CLIP-H blocks, yielding 96.12% within-subject and 35.47% LOSO test Top-1. The additional EEG and MEG sweeps are reported separately from the original configurations (Appendices K and O).
  • Capacity-matched nonlinear controls. Applying the same MLP before pooling improves Top-1 by 17.42/9.57 points within subject/LOSO compared with applying it after pooling. RPA adds a further 3.60/1.02 points, although normalization and bottleneck width also differ (Tables 25–26).
  • Matched EEG baseline comparisons. With the same EEG channels and windows, RPA exceeds Shallow Alignment by 11.18/12.15 points and also outperforms BlurPerception. We separately quantify the 17-versus-63-channel effect (Tables 23–24).

We also added single-trial retrieval, a pooled-only reconstruction control, and descriptor probes across visual layers. The revised text clarifies the reconstruction pipeline and early visual recognition discussion, and limits the color/texture conclusion to predictability consistent with our interpretation. Each response below identifies the relevant results and manuscript changes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.