acceptodds
Under review as a conference paper at ICLR 2027

A Patchwise Aggregation Encoder for EEG-to-Image Retrieval: An Analysis of What Image Features Matter

Abstract

What image features matter in EEG-to-image retrieval? Published approaches on the Things-EEG benchmark align EEG responses with features of a pre-trained visual encoder, but which image features drive this alignment remains unclear. We answer this question with the Patchwise Aggregation Encoder (PAE), a 1.57 M-parameter image encoder trained from scratch. The image is split into non-overlapping patches, each patch is processed independently by a shared stack of residual blocks, and the patch features are fused by a learned weighted average. On zero-shot within-subject retrieval over 200 unseen images, PAE reaches a mean top-1 of 96.9% and top-5 of 99.7% across the 10 subjects, a new state of the art on this benchmark. A purely statistical variant, S-PAE, replaces the pixel input with 15 low-level statistics of each patch and is even stronger, at 97.6% top-1 and 99.9% top-5. Ablations then identify the features this alignment relies on. The feature space is dominated by luminance contrast and edge statistics, and the learned pooling weights concentrate on the image center, consistent with foveal dominance in early visual processing. Retraining with only three statistics retains 95.2% top-1 accuracy, and brightness and contrast alone reach 90.3%. EEG-to-image retrieval thus rests on the low-level statistics of local patches, computed independently and combined by learned spatial weights, which makes decoding compact and interpretable by construction. These findings offer a new perspective on EEG-image alignment and motivate further research on EEG-based visual decoding.

Primary area:
applications to neuroscience & cognitive science
Keywords:
EEG, EEG-to-image retrieval, visual decoding, brain-computer interface, representation alignment, interpretability, Things-EEG

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_USNTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces PAE, an image encoder trained from scratch that processes patches independently and combines them through learned spatial pooling. S-PAE replaces pixels with 15 statistics per patch. Jointly trained with an EEG encoder, they achieve 96.9% and 97.6% Top-1 on 200-way THINGS-EEG retrieval using 80-repeat test averages. The paper analyzes which statistics, spatial weights, and architectural choices support these results.

Critical strengths

The paper asks a worthwhile question about the visual information needed for EEG retrieval. Training a compact image encoder from scratch and testing explicit statistical inputs makes this question more accessible than comparisons between frozen pretrained backbones. The reported performance of S-PAE is particularly interesting.

The main results cover ten subjects and three training seeds. Validation-based checkpoint selection, test-pair shuffling, and mismatched-pair retraining are useful controls. The additional MEG, LOSO, and repetition-count experiments help establish the scope of the results.

Critical weaknesses
  1. The feature analyses show that a restricted representation works well, but do not fully support the broader explanation of what EEG–image alignment relies on. Both branches are retrained for each statistic subset, and the two-/three-statistic findings are reported only for Subject 1. The two-statistic input also retains two 14-by-14 spatial maps, rather than two global scalars. The forward selection order uses test accuracy (Figure 4, Appendix F.3), while the split used for backward elimination is unclear. Independent confirmation is therefore important. Center-biased pooling may reflect image composition, and suppression of candidate images without new EEG recordings does not establish a foveal mechanism. Strict patch locality also needs qualification: some S-PAE statistics depend on the full-image mean or neighboring pixels (Listing 1). These limitations matter because interpretation, not just retrieval performance, is a central contribution.

  2. The architectural comparisons do not cleanly isolate cross-patch interaction. PAE uses 0.85 hidden dropout and learned spatial pooling, whereas the ViT control uses no dropout and a CLS readout. The staged comparison in Appendix K helps, but changes several components together. Meanwhile, MAE-PWE reaches 96.1% using contextualized Transformer features (Table 11). This does not invalidate PAE, but weakens the claim that independent patch computation explains its advantage. Matched readout, regularization, and tuning would make this conclusion more convincing. Paired uncertainty is also needed for the smaller differences among PAE, S-PAE, and MAE-PWE. These unresolved attribution issues lead me to weak rejection despite the strong reported results.

Questions for authors
  1. Were the backward-elimination decisions in Appendix F.3 based on validation or test accuracy? How do the fixed two- and three-statistic subsets perform on subjects not used to choose them?

  2. Does retrieval performance change when the S-PAE statistics use strictly within-patch references and boundary handling?

  3. How much of the PAE–ViT gap remains when the pooling head, dropout, and tuning budget are matched? Can you report paired differences across subjects and seeds?

  4. Is the PAE clean reference in Table 15 96.0% or 96.5%? What additional averaging, if any, produced the non-0.5-percentage-point PWE entries in Table 12?

4 comments
AuthorsOpenReview ↗

Response to Reviewer USNT (Part 1/2)

We thank the reviewer for the careful reading and the detailed comments.

Q1.1: Were the backward-elimination decisions in Appendix F.3 based on validation or test accuracy?

The per-round drop decisions of the backward elimination used the test top-1 averaged over three seeds. Within each run, the checkpoint is selected on a validation split, as everywhere in the paper. The figure and table captions state "test top-1", and we now also say so explicitly in the main text. A replay of the adjudication with the validation accuracy of the same runs shows that this choice does not change the selection. The leading single statistics (σ_L, E_∇, f_e) and the final pair of the backward elimination, {μ_L, σ_L}, are identical under both metrics.

Q1.2: How do the fixed two- and three-statistic subsets perform on subjects not used to choose them?

The two- and three-statistic subsets quoted in the abstract are further confirmed on held-out subjects. The revised subsets table adds a last column with the mean ± std of the test top-1 over the 10 subjects, 88.0 ± 6.8 for {μ_L, σ_L} and 91.9 ± 6.5 for {μ_L, σ_L, E_∇}.

Q2: Does retrieval performance change when the S-PAE statistics use strictly within-patch references and boundary handling?

Not significantly. A controlled comparison is added as a new appendix of the revision, Appendix O "Retrieval with strictly within-patch statistics".

Setup. We built a strictly patch-local variant of the S-PAE statistics, in which every statistic is computed within a single 16×16 patch. In the variant, the gradients and the Laplacian are computed within each patch, with one-sided differences and a zero Laplacian at the patch border. This is the operator convention that the baseline applies at the image border. The fine-frequency fraction f_hf is normalized by the variance of its own patch. We retrained the S-PAE on subject 1 with three seeds under the identical protocol, with an identically retrained baseline as the comparison arm. The six statistics that are already strictly local are identical between the two versions by construction.

Results. The strictly patch-local S-PAE reaches 97.5 ± 0.5 top-1, against 97.3 ± 1.6 for the identically retrained baseline. The per-seed paired differences range from −1.5 to +2.5 with a mean of +0.2%, and the baseline arm itself reproduces the paper's value for subject 1, 97.0 ± 1.3. The difference between the two versions is therefore not significant relative to the seed-to-seed variation. At the input level, the nine changed statistics differ by only 0.05–0.14 standard deviations on average, and f_hf, the definitional change, differs by 1.3 standard deviations, leaving the per-patch input profile essentially intact, with a mean Pearson correlation of 0.99. The reading that the statistics are local to each patch survives the stricter definition.

AuthorsOpenReview ↗

Response to Reviewer USNT (Part 2/2)

Q3: How much of the PAE–ViT gap remains when the pooling head, dropout, and tuning budget are matched? Can you report paired differences across subjects and seeds?

Both halves are answered by a new appendix of the revision, Appendix R "Inserting cross-patch attention into the PAE" (Tables 21 and 22).

Setup. The appendix inserts the one computation the PAE does not perform, attention across patches, into the PAE. The host is the default PAE with every parameter and every sublayer unchanged, and the only addition is one attention sublayer after each of the three residual blocks, so the pooling head, the dropout, and the alignment head are matched by construction. The tuning budget goes to the inserted attention alone. Its two hyperparameters, the number of heads and the width d_qk of the queries and keys, are tuned on subject 1 over the 12 combinations of 1, 2, 4, and 8 heads with the widths 64, 128, and 192 (Table 21).

Results. Every configuration falls below the PAE, and accuracy decreases monotonically in the number of heads at every width. Validation top-1 is the selection criterion, and the selected configuration, a single head at d_qk = 128, is then trained with three seeds on each of the ten subjects. The mean test top-1 is 1.3% below the PAE on the tuning subject, 1.7% below on the nine remaining subjects, and 1.7% below over all ten subjects. Table 22 reports the paired differences across subjects and seeds, each row being one subject with its three seeds averaged, and an exact Wilcoxon signed-rank test over the nine subject means gives p = 0.039.

Q4.1: Is the PAE clean reference in Table 15 96.0% or 96.5%?

Each model column of Table 15 (Table 16 in the revision) evaluates a single checkpoint of the main results (subject 1, seed 21). The relative accuracies in parentheses divide by the clean accuracy measured for that checkpoint, which is 96.5% for the PAE. The clean cell of the PAE column wrongly displayed 96.0%, the average over the three seeds of Table 2 (96.5/96.5/95.0). This was a transcription error and no conclusion is affected. In the revision we report 96.5 in the clean row and draw the clean anchor of the figure at 96.5.

Q4.2: What additional averaging, if any, produced the non-0.5-percentage-point PWE entries in Table 12?

This was a transcription error on our side, with the validation accuracy of the selected checkpoint entered in place of its test accuracy. We have corrected the PWE row of Table 12 (Table 13 in the revision). The corrected values have mean top-1 34.6 ± 8.0 and top-5 65.4.

We thank the reviewer again for the careful reading, and we are happy to clarify any remaining points during the discussion.

Reviewer_USNTOpenReview ↗

Response to authors

Thank you for the detailed response. I appreciate the additional experiments and analyses, which have addressed several concerns raised in my original review.

My remaining concern is primarily about the strength of the central feature-attribution claims. The new analysis in Appendix Q provides useful evidence of correspondence between the statistics used by S-PAE and the features learned by PAE. However, the reported correspondence remains moderate: the statistics explain approximately 32% of the variance of the PAE feature dimensions on average, and the cross-subject rank correlation is 0.61. This supports an association between the dominant information dimensions of the two encoders, but does not fully establish that the low-level statistics identified by S-PAE adequately characterize the information used by the original PAE. The subset experiments demonstrate the sufficiency of these statistics for a separately trained retrieval model, while their explanatory role in the full PAE remains less conclusive.

I acknowledge that the reply has appropriately narrowed some of its claims. Nevertheless, I remain somewhat unconvinced that the current feature-attribution analysis provides sufficient evidence for the central explanatory contribution of the paper. Since understanding which visual features drive EEG-to-image alignment is a major motivation of this work, I consider this limitation important to the overall contribution. My assessment has not changed sufficiently to support acceptance. I therefore maintain my rating.

We thank the reviewer for the reply and for the careful reading of the new analyses. We agree with the reading of Appendix Q, which is also the scope the appendix itself states. Its closing sentence states that the correspondence "establishes that the dominant information axes of the two encoders agree, not that the features of the PAE reduce to the fifteen statistics." The paper does not claim that the fifteen statistics characterize the information used by the pixel-based PAE.

What the two reported numbers measure. The 32% is the average explained variance over the 192 dimensions of the global feature of the PAE, and the per-dimension maximum is 82%. The 0.61 is the mean over the ten subjects of the rank correlation between each subject's own PAE profile, derived without a regression from the 192 dimensions of that subject's encoder, and that subject's own single-statistic accuracies (range 0.36 to 0.74, positive for all ten). On subject 1, on which the ridge fit of Appendix Q is computed, the rank correlation between the two rankings over the eleven statistics is 0.83 (p = 0.003) and the five leading statistics are identical on both sides. The appendix addresses which statistics matter to the PAE and the S-PAE, and these rank correlations are the evidence for it, while the average explained variance answers how much of the global feature of the PAE can be reconstructed from the fifteen statistics of each patch.

Where the feature-attribution claims of the main text rest. These are statements about the PAE and the S-PAE that we train, under the intra-subject condition. Contribution item 3 of the introduction states that "an attribution analysis of the intra-subject encoders shows that the feature space of the S-PAE is dominated by luminance contrast and edge statistics and that the learned pooling weights of the PAE concentrate on the image center." The first statement rests on the single-statistic and subset retraining experiments of the S-PAE, and the second on the measured pooling maps of the PAE. Neither statement rests on Appendix Q. Appendix Q answers the reviewer's first question, whether the S-PAE and the PAE, which are trained independently, rely on the same statistics, with the association quoted above. The Discussion statement "Within subject, the low-level statistics suffice" is likewise independent of Appendix Q.

We understand that the remaining assessment concerns the weight of this evidence for the explanatory contribution of the paper. The feature-attribution claims of the main text are made at the level of the PAE and the S-PAE that we train, under the intra-subject condition, and Appendix Q claims an agreement of the dominant information axes of the two encoders, not a characterization of the PAE.

We thank the reviewer again for the careful reading, and we are happy to clarify any remaining points during the discussion.

Reviewer_uYTkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PAE, an image encoder trained from scratch, using a patch aggregation pooling, through learned spatial pooling for contrastive alignment with EEG. Its statistical variant, S-PAE, replaces image pixels with fifteen handcrafted statistics per patch. Experiments on THINGS-EEG and THINGS-MEG evaluate retrieval performance, while feature-subset, architecture, and image-suppression analyses investigate which visual properties support alignment.

Critical strengths

The paper reports strong intra-subject retrieval performance, with S-PAE achieving 97.6% top-1 accuracy on THINGS-EEG without a pretrained image encoder.

The paper's main construction part, not using a pretrained image encoder, is not explored in the field of brain vision decoding.

Critical weaknesses

1. Weak cross-subject advantage and limited evidence of robustness. The large within-subject gains do not carry over to cross-subject evaluation. In Table 12, PAE (24.9%) and S-PAE (24.8%) essentially match the paper's own CPE baseline (25.0%). It i unclear how much of the within-subject gain reflects subject-specific fitting rather than a genuinely better representation.

2. Unclear source of the result gains and limited novel methodological contribution. Table 11 complicates the main result. PWE, which adds a learned patchwise aggregation head to frozen pretrained MAE features, reaches 96.1% top-1, versus 95.7% for default PAE and 96.9% for finer-patch PAE. On THINGS-MEG, PWE also slightly beats PAE (60.8% vs. 59.8%). This suggests much of the improvement may come from the aggregation and readout design, not from training the encoder from scratch or processing patches independently. Although the PAE is, to our extent, the first model to train image encoder from scratch, this does not show a gain in ablation study. The other components of the model, PWE, EEG decoder, remains well known in the field.

3. The S-PAE design relies heavily on heuristic feature engineering. The rationale for selecting this particular set of fifteen patch statistics is insufficiently established. Table 6 shows that only three statistics already achieve 95.17% accuracy, compared with 97.00% for the full set on subject 1, leaving the justification for the additional descriptors unclear. The formulation therefore appears to be an empirically assembled collection of handcrafted features, with limited evidence that the selection follows a generalizable principle or constitutes a substantial methodological advance.

Questions for authors
  1. What evidence connects the successful S-PAE statistics to features actually learned by the original pixel-based PAE? Can any analysis be given

  2. Can the model applied to other dataset besides thingseeg and meg, since both dataset are small in test set, only 200 images each?

  3. Do the feature rankings and suppression findings hold beyond subject 1 (section 3.3)?

  4. Can some experiments be run to show that what mechanism is working for the intra-subject pipeline? given both PWE and PAE are superior and why this cannot transfer to inter subject?

6 comments
AuthorsOpenReview ↗

Response to Reviewer uYTk (Part 1/3)

We thank the reviewer for the questions, which motivated several new experiments and analyses that are now part of the revision.

Clarifying three readings in the weaknesses. First, the PWE and the CPE are heads designed, trained, and named in the submitted version. The appendix itself states "We design two heads…", and the PWE's aggregation is the same learned Weighted Pooling as the PAE's. The two heads are our own exploration of how to aggregate patch features. The paper describes the design as "the patch features are fused by a learned weighted average whose weights depend on patch position". To our knowledge, at submission time, for EEG-to-image retrieval on Things-EEG, we are the first to propose both this learned Weighted Pooling for aggregating patch features and the PWE, now the F-PAE, which applies it to the patch-level features of a frozen MAE ViT-L. These designs differ clearly from the published methods we are aware of. Second, the contribution claim is anchored on the encoders trained from scratch against the published baselines (VED, UBP), not on outperforming our own appendix variants, and the revision now also surfaces the PWE in the main text as the F-PAE (Foundation Model Patchwise Aggregation Encoder) with the inter-subject retrieval results now in the main results section. Our reply to Q4 gives the numbers. Third, the fifteen statistics of the S-PAE are adopted because each is a classical low-level statistic of natural images, for example the mean gray level of a patch analyzed as the DC component in the monograph on natural image statistics by Hyvärinen et al. (2009), and the appendix gives the original source of every individual form. The set is adopted in full so that the ablation can identify which of these statistics the alignment relies on.

Q1: What evidence connects the successful S-PAE statistics to features actually learned by the original pixel-based PAE? Can any analysis be given

The PAE and the S-PAE are trained independently, one on pixels and one on the fifteen statistics. The suppression analysis of the paper already compares the two encoders at the level of feature types on subject 1 and finds that both rely primarily on the spatial layout of the patches, with patch shuffle collapsing retrieval for both (relative accuracy 16.2–35.2%). We have added a new appendix section to the revision, Appendix Q "Correspondence between the S-PAE statistics and the features learned by the PAE", which compares the two encoders at the level of the individual statistics. The section has two parts.

Decodability. For the default PAE of the main result table on subject 1, we fit one ridge regression per feature dimension, from the fifteen patch statistics to each of the 192 dimensions of the global feature, on 800,000 patch positions with an 80% fit and 20% evaluation split. On the evaluation patches the statistics explain on average 32% of the variance of a feature dimension and up to 82% for the strongest dimension, so the statistics are linearly readable out of the features the PAE learned. The statistics with the largest importance, the absolute regression coefficient averaged over the 192 dimensions, are contrast σ_L (0.38), gradient energy E_∇ (0.22), edge fraction f_e (0.18), and brightness μ_L (0.15).

Ranking correspondence. On the S-PAE side, the sufficiency of a statistic is its single-statistic top-1 accuracy, extended to all ten subjects in the last column of the elimination table. The three strongest are σ_L (60.2%), E_∇ (51.6%), and f_e (46.4%), exactly the three statistics most important to the PAE, and the five leading statistics are identical on both sides (σ_L, E_∇, f_e, μ_L, E_Δ). The Spearman rank correlation between the two rankings is ρ = 0.83 (p = 0.003, n = 11, 100,000 permutations). Appendix Q reports the analysis for every subject. The rank correlation is positive for all ten subjects, with a mean ρ of 0.61 (range 0.36–0.74).

We state the scope of the claim. The fit explains about one third of the variance of a feature dimension on average, so the analysis establishes that the dominant information axes of the two encoders agree, not that the features of the PAE reduce to the fifteen statistics.

AuthorsOpenReview ↗

Response to Reviewer uYTk (Part 2/3)

Q2: Can the model applied to other dataset besides thingseeg and meg, since both dataset are small in test set, only 200 images each?

We evaluated S-PAE together with PWE and VED on a second EEG–image dataset, Alljoined-1.6M (Xu et al., 2025), whose test set contains 500 images. Averaged over the 20 subjects and three training seeds, S-PAE reaches the highest top-1 and top-5 accuracy of the three models, 3.5% and 11.6%, versus 2.7% / 9.6% for VED and 2.4% / 8.8% for PWE. The S-PAE advantage over VED is significant under paired query-level bootstrap in all three seeds. The differences are 2.35%, 2.08%, and 1.53%, the 95% CIs are [1.59, 3.13], [1.34, 2.85], and [0.78, 2.28], and all p-values are below 1e-4. These results are included in the revision as Appendix P "Evaluation on a second EEG dataset".

Q3: Do the feature rankings and suppression findings hold beyond subject 1 (section 3.3)?

Both hold on all 10 subjects. We re-ran the two analyses of Section 3.3 on subjects 2–10 with the same protocol, using the per-subject EEG encoders and their single seed checkpoints, and added them to the revision. Every row of the subset table carries a 10-subject mean ± std column, and the suppression table (Table 17) reports all five models over the 10 subjects.

Feature rankings. The Section 3.3 conclusion holds unchanged over the ten subjects. The feature space remains dominated by luminance contrast and edge statistics. In the rightmost column of the subset table (Appendix F.3 "S-PAE input subsets"), σ_L is the strongest single statistic at 60.2 ± 10.4, the edge statistics E_∇ and f_e follow at 51.6 ± 6.7 and 46.4 ± 6.9, ⟨s⟩ is the weakest at 5.2 ± 1.1, and the subsets keep their ordering, with 97.7 ± 1.8 for the full set, 91.9 ± 6.5 for {μ_L, σ_L, E_∇}, and 88.0 ± 6.8 for {μ_L, σ_L}.

Suppression findings. In the suppression table (Table 17), patch shuffle collapses retrieval for both models at all three grid sizes (relative-accuracy means 18.4–31.2%), rotation stays mild (82.3–94.9%), and grayscale leaves both models nearly intact (90.2 / 91.3).

AuthorsOpenReview ↗

Response to Reviewer uYTk (Part 3/3)

Q4: Can some experiments be run to show that what mechanism is working for the intra-subject pipeline? given both PWE and PAE are superior and why this cannot transfer to inter subject?

Intra-subject mechanism. In the intra-subject task, the architecture ablations of Section 3.4 show that the learned Weighted Pooling is the structure that carries the gain. "Replacing the Weighted Pooling with a fixed uniform average reduces accuracy to 59.00%, and concat pooling, which concatenates all 196 patch features for a global linear head, reduces it to 52.00%." "Averaging with learned per-position weights is therefore substantially more accurate than direct averaging or feature concatenation, whereas changing the pooling position or inserting nonlinearity after the pooling yields no further gain over the default." Under from-scratch training, the same ablations show that adding interaction across patches brings no significant improvement. "Adding interaction between the patch tokens does not significantly improve retrieval accuracy in the tested configurations, which falls to 48.17% for token mixing and 44.67% for attention, compared with 96.00% for the PAE." The revision adds a controlled check of this finding, Appendix R "Inserting cross-patch attention into the PAE" (Tables 21 and 22). One attention sublayer is inserted after each of the three residual blocks of the otherwise unchanged PAE, and the two hyperparameters of the inserted attention are tuned on subject 1 over 12 combinations. Every configuration of the tuning grid falls below the PAE, and the selected configuration, retrained with three seeds on each of the ten subjects, is 1.7% below the PAE over all ten subjects (exact Wilcoxon p = 0.039).

Inter-subject results. In the leave-one-subject-out protocol, the F-PAE (the PWE of the appendix) reaches the highest mean top-1 at 34.6%, and the PAE and S-PAE reach 24.9% and 24.8%, above the reported VED (20.0%) and UBP (12.4%). Relative to VED the improvements are 14.6%, 4.9%, and 4.8%. First, measured as the fraction of the intra-subject gain that carries over to inter-subject retrieval, our progression matches the field's previous one. The carried-over fraction is 4.8% of the S-PAE's 17.6% intra-subject gain over VED (27%), close to the 7.6% of VED's 29.1% intra-subject gain over UBP (26%). The VED and UBP intra-subject numbers are as reported in the original papers. Second, the F-PAE's LOSO advantage comes from the proposed head. On the same frozen MAE backbone it reaches 34.6% where the CPE head reaches 25.0% (appendix Table 13), so the inter-subject gain also depends on the PAE-style aggregation. The proposed aggregation design scales from the PAE and the S-PAE, both trained from scratch, to the patch-level features of a pretrained model. The revision reports these results in the main results section with the corrected LOSO Table 13 and introduces the F-PAE in the methods.

We thank the reviewer again for the careful reading, and we are happy to clarify any remaining points during the discussion.

Reviewer_6pQLOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates which image features support EEG-to-image retrieval. The authors propose PAE, an image encoder trained from scratch. It divides images into non-overlapping patches, independently encodes each patch using shared residual MLP blocks, and aggregates the features using learned, dimension-specific spatial weights. The image encoder is jointly trained with the EEG encoder adopted from VED through contrastive learning.

Critical strengths
  1. The compact encoders achieve substantial retrieval improvements without visual pretraining. In Table 2, PAE with a patch size of 8 achieves 96.9% top-1 accuracy, while S-PAE achieves 97.6%, compared with the reported VED result of 80.0%. These results demonstrate the practical value of compact image representations for this task.
  2. Rather than relying solely on post hoc interpretations of pretrained features, the authors explicitly restrict the information available to the image encoder and retrain the model. Table 6 shows that brightness, contrast, and gradient energy alone achieve 95.17% accuracy on subject 1, while brightness and contrast alone achieve 90.33%. The random-pairing sanity checks, perturbation analyses, and repetition-averaging curves further strengthen the empirical evidence.
Critical weaknesses
  1. Sections 3.3 and 4 and the conclusion connect the findings to early visual processing, foveal dominance, and the importance of particular features. However, the key feature-subset and pooling-map analyses primarily concern subject 1. Retraining with restricted inputs establishes their sufficiency for this retrieval task. It does not establish that they fully explain the decodable information in EEG or exclude contributions from correlated semantic information. Moreover, perturbing only the candidate images while keeping EEG unchanged alters both stimulus correspondence and the image-input distribution. The resulting accuracy drops therefore cannot directly establish the brain’s causal dependence on particular feature types. Although the authors describe several findings as consistent with visual mechanisms, these interpretations remain suggestive rather than independently validated.
  2. The interaction-based models in Table 3 show large accuracy drops, but the ViT in Appendix J also uses CLS-token readout, positional embeddings, and different regularization settings. The E1-to-E2 transition in Appendix K likewise replaces the readout while removing several other components. These comparisons establish that the current PAE configuration outperforms the tested alternatives. However, they do not fully isolate cross-patch interaction as the cause of the performance degradation. An identical training recipe also does not ensure that different architectures are comparably optimized. The main text should make these comparison limitations explicit. Stronger attribution requires more controlled comparisons or more restrained conclusions.
Questions for authors
  1. How consistent are the feature-subset results and central pooling preferences across subjects and random seeds? What evidence distinguishes the proposed visual-mechanism interpretation from effects of the centered stimulus presentation and image-content distribution?
  2. Can the effect of cross-patch interaction be isolated while retaining the same pooling, alignment head, and comparable parameter and tuning budgets? If such a comparison is unavailable, could the conclusions be restricted to the tested architectural configurations?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 6pQL

We thank the reviewer for the careful reading and the constructive comments.

Q1.1: How consistent are the feature-subset results and central pooling preferences across subjects and random seeds?

Both are consistent over the ten subjects, and the revision reports each with objective statistics. For the feature-subset results, the last column of the subset table (Appendix F.3 "S-PAE input subsets") gives the mean ± std over the 10 subjects. σ_L is the strongest single statistic at 60.2 ± 10.4 and ⟨s⟩ the weakest at 5.2 ± 1.1, and the subsets keep their ordering, with 97.7 ± 1.8 for the full set, 91.9 ± 6.5 for {μ_L, σ_L, E_∇}, and 88.0 ± 6.8 for {μ_L, σ_L}. Every configuration is trained over three seeds. The subject 1 column of the same table reports the seed mean ± std, for example 97.0 ± 1.3 for the full set and 65.5 ± 4.0 for σ_L alone. For the central pooling preference, the Appendix N "Central pooling preference across subjects and seeds" reports Spearman correlations of 0.925 across three retraining seeds of subject 1 (range 0.913–0.936), 0.774 across the 45 subject pairs (range 0.511–0.915), and 0.890 between each subject and the mean map over the ten subjects (range 0.771–0.961). The centroid eccentricity of the map averaged over dimensions stays small for every subject, 0.26 ± 0.10, against 0.17 for the checkpoint of subject 1 analyzed in the main text, and a uniform map gives 0. The spread stays below the uniform value at 3.88 ± 0.03 vs. 4.03.

Q1.2: What evidence distinguishes the proposed visual-mechanism interpretation from effects of the centered stimulus presentation and image-content distribution?

We agree that the center weighting alone cannot identify its cause. Foveal processing, always-centered presentation, and object-centered image composition all favor central patches and co-occur in this paradigm, and we cannot separate them in this dataset. We have therefore removed the mechanism attribution from the manuscript. The phrase "consistent with foveal dominance" is replaced by the plain finding that the learned pooling maps concentrate on the image center, and the phrase "mirrors retinal photoreceptor density … cortical magnification" is downgraded to "compatible with". The physiological accounts remain candidates that a permanently centered presentation equally predicts and that the present data corroborate but do not isolate. We state this explicitly in the Discussion section of the main text.

Q2: Can the effect of cross-patch interaction be isolated while retaining the same pooling, alignment head, and comparable parameter and tuning budgets? If such a comparison is unavailable, could the conclusions be restricted to the tested architectural configurations?

We have additionally run a controlled comparison, now in the revision as Appendix R "Inserting cross-patch attention into the PAE" (Tables 21 and 22).

Setup. The appendix inserts the one computation the PAE does not perform, attention across patches, into the PAE. The host is the default PAE with every parameter and every sublayer unchanged, and the only addition is one attention sublayer after each of the three residual blocks, so the pooling and the alignment head are matched by construction. The tuning budget goes to the inserted attention alone. Its two hyperparameters, the number of heads and the width d_qk of the queries and keys, are tuned on subject 1 over the 12 combinations of 1, 2, 4, and 8 heads with the widths 64, 128, and 192 (Table 21). The only budget axis that differs is the parameter count, which the insertion raises, so the comparison favors the attention side.

Results. Every configuration falls below the PAE, and accuracy decreases monotonically in the number of heads at every width. Validation top-1 is the selection criterion, and the selected configuration, a single head at d_qk = 128, is then trained with three seeds on each of the ten subjects. The mean test top-1 is 1.3% below the PAE on the tuning subject, 1.7% below on the nine remaining subjects, and 1.7% below over all ten subjects. Table 22 reports the paired differences across subjects and seeds, each row being one subject with its three seeds averaged, and an exact Wilcoxon signed-rank test over the nine subject means gives p = 0.039.

Restricted conclusions. We have also restricted the conclusions to the tested configurations in the text. The conclusion, the contributions, and the ablation discussion now state that in the intra-subject task, adding interaction between the patch tokens does not significantly improve accuracy in the tested configurations, and no longer claim a general effect of cross-patch interaction.

We thank the reviewer again for the careful reading, and we are happy to clarify any remaining points during the discussion.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Rebuttal

We thank all three reviewers for the careful reading and the constructive comments. The reviews motivated controlled experiments, five new appendix sections, one table correction, and several presentation changes. The revised manuscript has been uploaded. Unless noted otherwise, table and appendix numbers refer to this revision.

A clarification on the PWE and the CPE. The two heads are designed, trained, and named in the submitted version. The appendix states "We design two heads…", and the PWE's aggregation is the same learned Weighted Pooling as the PAE's. The paper describes the design as "the patch features are fused by a learned weighted average whose weights depend on patch position". To our knowledge, at submission time, for EEG-to-image retrieval on Things-EEG, we are the first to propose both this learned Weighted Pooling for aggregating patch features and the PWE, now the F-PAE, which applies it to the patch-level features of a frozen MAE ViT-L. These designs differ clearly from the published methods we are aware of.

Main changes in the revision.

The PWE is promoted to the main text as the F-PAE (Reviewer uYTk). The variant is now introduced in the methods section as the Foundation Model Patchwise Aggregation Encoder (F-PAE), and its inter-subject results have moved into the main results section (Table 3). The PWE had been placed in the appendix as an extension attempt on the PAE. Its intra-subject accuracy does not significantly exceed the simple PAE (paired difference +0.61%, p = 0.46, subject-level pairing over seed-averaged accuracies), so we attributed the main gain to the aggregation structure itself and anchored the contribution claim on the encoders trained from scratch. The corrected inter-subject results below motivated this promotion. The F-PAE substantially improves inter-subject retrieval, and the Discussion concludes that what the EEG alignment exploits within a subject and what carries across subjects are different levels of image description.

One inter-subject result is corrected (Reviewer USNT, Q4). The PWE row of the leave-one-subject-out table had copied pooled validation accuracies (mean 11.8) instead of test accuracies. We thank the reviewer for surfacing this error. The corrected entry is 34.6 ± 8.0, which makes the F-PAE the strongest model compared under this protocol and, to our knowledge, a new state of the art in inter-subject retrieval. An ablation shows that the performance gain of this variant depends on the same aggregation head as the PAE.

Claims are qualified and overclaims removed (Reviewers 6pQL and USNT). The three interaction assertions, in the conclusion, the contribution list, and the ablation section, now state that adding interaction does not significantly improve accuracy in the intra-subject task. The mechanistic attribution of the central pooling preference is removed, the learned maps are described as concentrating on the image center, the foveal phrasing is downgraded to "compatible with", and the Discussion states explicitly that the present data corroborate but do not isolate the physiological accounts. The headline claims of the abstract and conclusion are scoped to the intra-subject task. The selection criterion of the elimination analysis is now stated explicitly in the main text (test top-1 averaged over three seeds, with the checkpoint of each run selected on a validation split).

New experiments and sections.

  • Appendix N, "Central pooling preference across subjects and seeds" (Reviewer 6pQL, Q1; Reviewer uYTk, Q3). Statistics of the central pooling preference, with a mean ± std over the ten subjects in the S-PAE subset table.

  • Appendix O, "Retrieval with strictly within-patch statistics" (Reviewer USNT, Q2). A variant of the S-PAE statistics with strictly within-patch references, retrained with three seeds against an identically retrained baseline. Retrieval accuracy is not significantly changed.

  • Appendix P, "Evaluation on a second EEG dataset" (Reviewer uYTk, Q2). Evaluated on Alljoined-1.6M (Xu et al., 2025), beyond the 200-image test sets of Things-EEG and Things-MEG.

  • Appendix Q, "Correspondence between the S-PAE statistics and the features learned by the PAE" (Reviewer uYTk, Q1).

  • Appendix R, "Inserting cross-patch attention into the PAE" (Reviewer 6pQL, Q2; Reviewer USNT, Q3). Cross-patch attention, the interaction the PAE does not include, is inserted into an otherwise unchanged PAE, with the tuning budget spent on the attention alone. All tested configurations fall below the PAE, and the extension to ten subjects and three seeds gives a significant decrease.

The suppression analysis is extended to all 10 subjects (Table 17), and paired differences across subjects and seeds are added to the appendix.

We thank all reviewers again for the time and care they gave the manuscript, and we are happy to clarify any remaining points during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.