- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces PAE, an image encoder trained from scratch that processes patches independently and combines them through learned spatial pooling. S-PAE replaces pixels with 15 statistics per patch. Jointly trained with an EEG encoder, they achieve 96.9% and 97.6% Top-1 on 200-way THINGS-EEG retrieval using 80-repeat test averages. The paper analyzes which statistics, spatial weights, and architectural choices support these results.
Critical strengths
The paper asks a worthwhile question about the visual information needed for EEG retrieval. Training a compact image encoder from scratch and testing explicit statistical inputs makes this question more accessible than comparisons between frozen pretrained backbones. The reported performance of S-PAE is particularly interesting.
The main results cover ten subjects and three training seeds. Validation-based checkpoint selection, test-pair shuffling, and mismatched-pair retraining are useful controls. The additional MEG, LOSO, and repetition-count experiments help establish the scope of the results.
Critical weaknesses
-
The feature analyses show that a restricted representation works well, but do not fully support the broader explanation of what EEG–image alignment relies on. Both branches are retrained for each statistic subset, and the two-/three-statistic findings are reported only for Subject 1. The two-statistic input also retains two 14-by-14 spatial maps, rather than two global scalars. The forward selection order uses test accuracy (Figure 4, Appendix F.3), while the split used for backward elimination is unclear. Independent confirmation is therefore important. Center-biased pooling may reflect image composition, and suppression of candidate images without new EEG recordings does not establish a foveal mechanism. Strict patch locality also needs qualification: some S-PAE statistics depend on the full-image mean or neighboring pixels (Listing 1). These limitations matter because interpretation, not just retrieval performance, is a central contribution.
-
The architectural comparisons do not cleanly isolate cross-patch interaction. PAE uses 0.85 hidden dropout and learned spatial pooling, whereas the ViT control uses no dropout and a CLS readout. The staged comparison in Appendix K helps, but changes several components together. Meanwhile, MAE-PWE reaches 96.1% using contextualized Transformer features (Table 11). This does not invalidate PAE, but weakens the claim that independent patch computation explains its advantage. Matched readout, regularization, and tuning would make this conclusion more convincing. Paired uncertainty is also needed for the smaller differences among PAE, S-PAE, and MAE-PWE. These unresolved attribution issues lead me to weak rejection despite the strong reported results.
Questions for authors
-
Were the backward-elimination decisions in Appendix F.3 based on validation or test accuracy? How do the fixed two- and three-statistic subsets perform on subjects not used to choose them?
-
Does retrieval performance change when the S-PAE statistics use strictly within-patch references and boundary handling?
-
How much of the PAE–ViT gap remains when the pooling head, dropout, and tuning budget are matched? Can you report paired differences across subjects and seeds?
-
Is the PAE clean reference in Table 15 96.0% or 96.5%? What additional averaging, if any, produced the non-0.5-percentage-point PWE entries in Table 12?
4 comments
Response to Reviewer USNT (Part 2/2)
Q3: How much of the PAE–ViT gap remains when the pooling head, dropout, and tuning budget are matched? Can you report paired differences across subjects and seeds?
Both halves are answered by a new appendix of the revision, Appendix R "Inserting cross-patch attention into the PAE" (Tables 21 and 22).
Setup. The appendix inserts the one computation the PAE does not perform, attention across patches, into the PAE. The host is the default PAE with every parameter and every sublayer unchanged, and the only addition is one attention sublayer after each of the three residual blocks, so the pooling head, the dropout, and the alignment head are matched by construction. The tuning budget goes to the inserted attention alone. Its two hyperparameters, the number of heads and the width d_qk of the queries and keys, are tuned on subject 1 over the 12 combinations of 1, 2, 4, and 8 heads with the widths 64, 128, and 192 (Table 21).
Results. Every configuration falls below the PAE, and accuracy decreases monotonically in the number of heads at every width. Validation top-1 is the selection criterion, and the selected configuration, a single head at d_qk = 128, is then trained with three seeds on each of the ten subjects. The mean test top-1 is 1.3% below the PAE on the tuning subject, 1.7% below on the nine remaining subjects, and 1.7% below over all ten subjects. Table 22 reports the paired differences across subjects and seeds, each row being one subject with its three seeds averaged, and an exact Wilcoxon signed-rank test over the nine subject means gives p = 0.039.
Q4.1: Is the PAE clean reference in Table 15 96.0% or 96.5%?
Each model column of Table 15 (Table 16 in the revision) evaluates a single checkpoint of the main results (subject 1, seed 21). The relative accuracies in parentheses divide by the clean accuracy measured for that checkpoint, which is 96.5% for the PAE. The clean cell of the PAE column wrongly displayed 96.0%, the average over the three seeds of Table 2 (96.5/96.5/95.0). This was a transcription error and no conclusion is affected. In the revision we report 96.5 in the clean row and draw the clean anchor of the figure at 96.5.
Q4.2: What additional averaging, if any, produced the non-0.5-percentage-point PWE entries in Table 12?
This was a transcription error on our side, with the validation accuracy of the selected checkpoint entered in place of its test accuracy. We have corrected the PWE row of Table 12 (Table 13 in the revision). The corrected values have mean top-1 34.6 ± 8.0 and top-5 65.4.
We thank the reviewer again for the careful reading, and we are happy to clarify any remaining points during the discussion.
Response to authors
Thank you for the detailed response. I appreciate the additional experiments and analyses, which have addressed several concerns raised in my original review.
My remaining concern is primarily about the strength of the central feature-attribution claims. The new analysis in Appendix Q provides useful evidence of correspondence between the statistics used by S-PAE and the features learned by PAE. However, the reported correspondence remains moderate: the statistics explain approximately 32% of the variance of the PAE feature dimensions on average, and the cross-subject rank correlation is 0.61. This supports an association between the dominant information dimensions of the two encoders, but does not fully establish that the low-level statistics identified by S-PAE adequately characterize the information used by the original PAE. The subset experiments demonstrate the sufficiency of these statistics for a separately trained retrieval model, while their explanatory role in the full PAE remains less conclusive.
I acknowledge that the reply has appropriately narrowed some of its claims. Nevertheless, I remain somewhat unconvinced that the current feature-attribution analysis provides sufficient evidence for the central explanatory contribution of the paper. Since understanding which visual features drive EEG-to-image alignment is a major motivation of this work, I consider this limitation important to the overall contribution. My assessment has not changed sufficiently to support acceptance. I therefore maintain my rating.
We thank the reviewer for the reply and for the careful reading of the new analyses. We agree with the reading of Appendix Q, which is also the scope the appendix itself states. Its closing sentence states that the correspondence "establishes that the dominant information axes of the two encoders agree, not that the features of the PAE reduce to the fifteen statistics." The paper does not claim that the fifteen statistics characterize the information used by the pixel-based PAE.
What the two reported numbers measure. The 32% is the average explained variance over the 192 dimensions of the global feature of the PAE, and the per-dimension maximum is 82%. The 0.61 is the mean over the ten subjects of the rank correlation between each subject's own PAE profile, derived without a regression from the 192 dimensions of that subject's encoder, and that subject's own single-statistic accuracies (range 0.36 to 0.74, positive for all ten). On subject 1, on which the ridge fit of Appendix Q is computed, the rank correlation between the two rankings over the eleven statistics is 0.83 (p = 0.003) and the five leading statistics are identical on both sides. The appendix addresses which statistics matter to the PAE and the S-PAE, and these rank correlations are the evidence for it, while the average explained variance answers how much of the global feature of the PAE can be reconstructed from the fifteen statistics of each patch.
Where the feature-attribution claims of the main text rest. These are statements about the PAE and the S-PAE that we train, under the intra-subject condition. Contribution item 3 of the introduction states that "an attribution analysis of the intra-subject encoders shows that the feature space of the S-PAE is dominated by luminance contrast and edge statistics and that the learned pooling weights of the PAE concentrate on the image center." The first statement rests on the single-statistic and subset retraining experiments of the S-PAE, and the second on the measured pooling maps of the PAE. Neither statement rests on Appendix Q. Appendix Q answers the reviewer's first question, whether the S-PAE and the PAE, which are trained independently, rely on the same statistics, with the association quoted above. The Discussion statement "Within subject, the low-level statistics suffice" is likewise independent of Appendix Q.
We understand that the remaining assessment concerns the weight of this evidence for the explanatory contribution of the paper. The feature-attribution claims of the main text are made at the level of the PAE and the S-PAE that we train, under the intra-subject condition, and Appendix Q claims an agreement of the dominant information axes of the two encoders, not a characterization of the PAE.
We thank the reviewer again for the careful reading, and we are happy to clarify any remaining points during the discussion.
Response to Reviewer USNT (Part 1/2)
We thank the reviewer for the careful reading and the detailed comments.
Q1.1: Were the backward-elimination decisions in Appendix F.3 based on validation or test accuracy?
The per-round drop decisions of the backward elimination used the test top-1 averaged over three seeds. Within each run, the checkpoint is selected on a validation split, as everywhere in the paper. The figure and table captions state "test top-1", and we now also say so explicitly in the main text. A replay of the adjudication with the validation accuracy of the same runs shows that this choice does not change the selection. The leading single statistics (σ_L, E_∇, f_e) and the final pair of the backward elimination, {μ_L, σ_L}, are identical under both metrics.
Q1.2: How do the fixed two- and three-statistic subsets perform on subjects not used to choose them?
The two- and three-statistic subsets quoted in the abstract are further confirmed on held-out subjects. The revised subsets table adds a last column with the mean ± std of the test top-1 over the 10 subjects, 88.0 ± 6.8 for {μ_L, σ_L} and 91.9 ± 6.5 for {μ_L, σ_L, E_∇}.
Q2: Does retrieval performance change when the S-PAE statistics use strictly within-patch references and boundary handling?
Not significantly. A controlled comparison is added as a new appendix of the revision, Appendix O "Retrieval with strictly within-patch statistics".
Setup. We built a strictly patch-local variant of the S-PAE statistics, in which every statistic is computed within a single 16×16 patch. In the variant, the gradients and the Laplacian are computed within each patch, with one-sided differences and a zero Laplacian at the patch border. This is the operator convention that the baseline applies at the image border. The fine-frequency fraction f_hf is normalized by the variance of its own patch. We retrained the S-PAE on subject 1 with three seeds under the identical protocol, with an identically retrained baseline as the comparison arm. The six statistics that are already strictly local are identical between the two versions by construction.
Results. The strictly patch-local S-PAE reaches 97.5 ± 0.5 top-1, against 97.3 ± 1.6 for the identically retrained baseline. The per-seed paired differences range from −1.5 to +2.5 with a mean of +0.2%, and the baseline arm itself reproduces the paper's value for subject 1, 97.0 ± 1.3. The difference between the two versions is therefore not significant relative to the seed-to-seed variation. At the input level, the nine changed statistics differ by only 0.05–0.14 standard deviations on average, and f_hf, the definitional change, differs by 1.3 standard deviations, leaving the per-patch input profile essentially intact, with a mean Pearson correlation of 0.99. The reading that the statistics are local to each patch survives the stricter definition.