acceptodds
Under review as a conference paper at ICLR 2027

Perceptual-Prior Alignment with Subject-Adaptive Temperature for EEG-to-Image Retrieval

Abstract

EEG-to-image retrieval aims to identify perceived visual stimuli from non-invasive neural recordings, but remains challenging because EEG is noisy, spatially coarse, and highly variable across trials and subjects. Existing contrastive approaches commonly align EEG with a single, fixed visual representation using one temperature shared across all subjects. This formulation overlooks both trial-level variation in the visual information represented by EEG and subject-level heterogeneity in neural separability. We propose Perceptual-Prior Alignment with Subject-Adaptive Temperature (PPA-SAT), a retrieval-oriented framework that calibrates cross-modal alignment at two complementary resolutions. At the trial level, Perceptual-Prior Alignment (PPA) retains a heterogeneous bank of raw, blurred, noisy, low-resolution, and mosaic visual candidates. It constructs an EEG-conditioned target through residual attention fusion. At the source-subject level, Subject-Adaptive Temperature (SAT) complements this alignment by learning lightweight subject-specific offsets to a shared contrastive temperature, enabling subject-aware calibration with negligible parameter overhead. Experiments on THINGS-EEG 200-way zero-shot retrieval demonstrate consistent improvements under both intra- and inter-subject protocols. PPA-SAT achieves 91.7% Top-1 and 99.0% Top-5 intra-subject accuracy, exceeding the strongest prior method by 9.1 and 1.3 percentage points, respectively. Under leave-one-subject-out evaluation, it achieves 45.8% Top-1 and 72.7% Top-5 accuracy, outperforming the strongest baseline by 21.8 and 17.1 points. These results demonstrate the complementary value of trial-adaptive visual alignment and subject-aware contrastive calibration for robust EEG-to-image retrieval.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
EEG decoding, brain-computer interfaces, contrastive learning, zero-shot retrieval, multimodal representation learning, cross-subject generalization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ccm6OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PPA-SAT, a two-level alignment framework for EEG-to-image retrieval. PPA uses EEG-conditioned attention to combine five perceptual visual representations for each EEG–image pair. SAT introduces subject-specific temperature offsets to calibrate contrastive learning across source subjects.The method is evaluated on THINGS-EEG under intra-subject and leave-one-subject-out (LOSO) settings. The paper reports 91.7% Top-1 intra-subject accuracy and 45.8% Top-1 LOSO accuracy, with large improvements over the listed baselines. Overall, PPA introduces a meaningful idea with some scientific novelty, and the reported experiments provide encouraging results. However, several serious concerns remain regarding implementation consistency, evaluation validity, and experimental evidence. These issues significantly weaken confidence in the reported improvements and the paper's main conclusions.

Critical strengths
  1. Clear and meaningful problem formulation. The paper identifies two relevant sources of heterogeneity in EEG–visual alignment: trial-level differences in visual information and subject-level differences in neural separability. The motivation is clear, and the two-level formulation gives a reasonable distinction between what visual information should be aligned and how the alignment should be optimized.

  2. A meaningful and well-motivated extension of perceptual-prior alignment. PPA has a thoughtful design beyond the fixed visual mean fusion used in NeuroBridge. In Eqs. 3–7, the attention weights depend jointly on the current EEG query and candidate image. This allows the same candidate image to have different effective representations for different EEG queries. More importantly, PPA changes the retrieval scoring function itself, rather than simply producing another fixed visual embedding. The residual formulation in Eq. 6 also provides a controlled way to combine the uniform visual prior with EEG-conditioned attention. This is a concrete and meaningful methodological contribution.

  3. A simple and reasonable design for source-subject calibration. SAT introduces a lightweight temperature parameterization with a shared global component and source-subject-specific offsets (Sec. 3.4, Eq. 9). This is a reasonable way to account for subject differences during joint contrastive training, without introducing separate subject-specific encoders. The distinction between source-subject training calibration and unseen-subject inference is also clearly defined (Sec. 3.4, lines 223–225).

  4. Relevant experimental settings and useful component comparisons. The paper reports results for all ten subjects under both intra-subject and LOSO protocols, including Top-1 and Top-5 accuracy (Tables 1–2). These two settings are important for evaluating EEG–visual alignment, especially cross-subject generalization. The experiments also compare intermediate representations, perceptual-prior fusion, SAT, and their combinations (Tables 3–5). In particular, the comparison between uniform fusion, residual fusion, and EEG-conditioned attention is directly relevant to the proposed PPA mechanism.

Critical weaknesses
  1. Inconsistency between the proposed PPA scoring function and the submitted implementation. Sec. 3.3 (Eqs. 3–7) defines PPA as a query-conditioned scorer, and the paper states that ground-truth identity is not used during inference (p. 4, lines 194–196). However, the submitted training and evaluation code constructs fused image representations using their correctly paired EEG, rather than the current EEG query for every candidate. During evaluation, this uses known test EEG–image pairings in candidate construction. The training contrastive scores also differ from the proposed pairwise formulation. The supplementary package includes another analysis implementation with the correct query-conditioned scorer. It is therefore important to clarify which implementation produced Tables 1–2 and the ablations. This difference could affect how the reported retrieval performance should be interpreted.

  2. The checkpoint selection procedure needs further clarification. The submitted training code evaluates the test set after each epoch and uses test performance for best-checkpoint selection and early stopping. The provided LOSO script also selects checkpoints based on test Top-1 accuracy. In this implementation, the held-out subject is excluded from gradient-based training, but its test performance still influences model selection. This may introduce selection bias if the reported results follow this procedure. The paper does not clearly describe how the final checkpoints for Tables 1–2 were selected (Secs. 4.1–4.4). Clarifying this procedure would help establish the reliability of the reported generalization results.

  3. SAT implementation and ablation need further clarification. The image-to-EEG temperature scaling in the submitted code differs from Eq. 13 (Sec. 3.5). When source subjects have different temperatures, the two formulations define different contrastive objectives. Also, the provided SAT ablation changes both the learnability of the global temperature and the introduction of subject-specific offsets. Thus, Table 3 does not clearly isolate the contribution of subject-specific calibration. A comparison with a learnable global temperature without subject offsets would help clarify the independent benefit of SAT.

  4. The ablation studies could provide stronger evidence for the proposed mechanisms. Table 3 presents a component ablation, but its baseline and intermediate-only results exactly match the reported NeuroBridge [1] and Shallow Alignment [2] results in Table 2. It would be helpful to clarify whether all configurations were evaluated under matched training and evaluation settings. There is also an unclear relationship between Tables 4 and 5. The "All five candidates" result in Table 4 (79.20% Top-1) matches "Residual Mean + Entropy" in Table 5, rather than the full PPA result (91.70%). This makes it unclear whether Table 4 evaluates prior removal from the full PPA model. Furthermore, Appendix Table 6 reports normalized attention entropy close to 1. This does not contradict PPA, but more direct evidence would help explain how query-dependent visual weighting contributes to the large improvement, beyond a general conditional matching effect.

  5. Experimental comparability and reporting could be more complete. Sec. 4.2 states an EEG window of 0–250 ms, while the provided code uses 250 samples with a default sampling rate of 250 Hz, corresponding to around 1 second. The actual experimental window should be clarified. The SoTA comparison could also be more complete. Table 1 uses the 82.6% single-layer result from Shallow Alignment [2], which also reports 91.3% Top-1 using a stronger multi-layer configuration. The closely related SAMGA [3] is not discussed. These methods may use different protocols, but discussing them would help clarify the scope of the reported improvements and the novelty of PPA-SAT. Finally, Table 1 reports 91.70%/99.00%, while Appendix Table 6 reports 91.35%/96.70% under the stated entropy coefficient. This difference needs explanation. Multi-seed statistics for the main results are also not clearly reported. Some presentation issues remain, including the unfinished baseline-verification statement in Sec. 4.3 and the placement of Table 6 within the References.

Questions for authors
  1. Could the authors clarify which scoring function was used for Tables 1–2 and the main ablations? Were candidate representations computed using the current EEG query as defined in Eqs. 3–7? Also, how does the training implementation relate to the proposed pairwise contrastive objective?

  2. How were the final checkpoints selected? Did the reported experiments use test-set performance for checkpoint selection or early stopping? If so, could the authors provide results with an independent validation and model-selection protocol?

  3. Could the authors clarify the difference between Eq. 13 and the submitted SAT implementation? Also, does SAT still provide additional gains over a learnable global temperature without subject-specific offsets?

  4. Could the authors clarify the experimental configurations used in Tables 3–5, especially the difference between the five-prior result in Table 4 and the full PPA result in Table 5? What direct evidence supports the contribution of query-conditioned visual weighting to the reported improvements?

1 comment

1. Corrected scorer

We thank the reviewer for identifying a consequential problem. Constructing a candidate using its paired EEG and then scoring other queries against it leaks privileged pairing information and is invalid for query-conditioned retrieval. The submitted implementation also did not implement the stated pairwise contrastive objective. The original headline results (91.70% intra-subject; 45.80% inter-subject) are superseded; we cannot reliably map old tables to specific scripts.

The revised query-conditioned alternatives use exact pairwise scoring: weights for query i and candidate j depend only on that pair. The cosine numerator is the weighted sum of query–view dot products; the denominator is sqrt(w_ij^T G_j w_ij), where G_j is the 5×5 Gram matrix of the candidate's projected views. The same score matrix is used in both directional losses. Uniform PPA (lambda=0) uses fixed gallery embeddings. The shared-projector experiment also encodes the gallery independently of EEG. Sections 3.3, 3.6 and Figure 1 are revised.

2. Checkpoint selection and independent validation

We acknowledge that the main tables use test-based checkpoint selection. The three-seed study selects by test Top-1 with early stopping (patience 15 intra-subject; 20 inter-subject); shared-projector scripts select lexicographically by (Top-1, Top-5). Test labels therefore affect inter-subject selection, and results may be optimistic.

Appendix A.1 reports a separate three-seed intra-subject experiment: 200 training concepts are held out only for validation and 1,454 are used for fitting. Single-view accuracy is 69.58/93.47% (Top-1/Top-5), versus 77.83/96.15% for uniform PPA, an 8.25-point Top-1 gain. Residual query-conditioned variants reach 76.05% (without entropy) and 75.67% (with entropy); attention-only reaches 68.30%. For source-validation inter-subject selection, only subject 8 is complete. Its four-view global/SAT Top-1 is 23.50/24.50% (reference scale) and 18.50/21.00% (reciprocal scale); these are single-fold observations, not a ten-fold result. Source-only preprocessing remains unaudited.

3. SAT formulation and global-temperature control

We agree that the directional SAT formulations differ. Let m_ij be EEG i–image j similarity and c_s the subject-specific logit scale. Transposing the scaled EEG-to-image matrix gives image-to-EEG logits c_{s_j}m_ji, whereas independently scaling rows after transposition gives c_{s_i}m_ji. These coincide for a global scale, but not generally for SAT. Sections 3.4–3.6 distinguish reciprocal-softplus temperature and reference logit-scale parameterizations and specify both directional objectives.

Table 4 compares learnable global temperature with SAT. For uniform PPA, results are 26.95±0.75/52.92±0.54% without SAT and 26.43±0.40/51.90±0.48% with SAT (Top-1/Top-5). In matched four-view TSConv runs, global/SAT results are 31.90/61.15% and 31.85/60.50%. Appendix A.2 finds no consistent advantage from a high-learning-rate offset. We therefore do not claim an independent SAT gain.

4. Controlled ablations and attention

The former 79.20% all-five-prior and 91.70% headline results are not interpretable controlled comparisons; their historical configuration mapping is unresolved and they are superseded. Table 3 reports uniform, residual and attention-only settings, entropy coefficients and three-seed variability. Uniform PPA reaches 88.65% Top-1, versus 83.67% for both residual variants and 77.93% for attention-only. Attention-only normalized entropy is about 0.99, and the residual gate remains near initialization. Evidence favors uniform aggregation.

Table 5 adds a capacity-matched shared-projector view-count study. Original/four/five-view Top-1 is 75.25/83.20/83.35% intra-subject and 28.20/31.90/31.95% inter-subject. Five views improve over the original by 8.10 and 3.75 points; the fifth view adds little over four transformed views. We do not infer a separate-projector advantage from different settings.

5. Protocol, related work and reporting

We corrected the EEG interval to 250 samples at 250 Hz (approximately 0–1,000 ms) and clarify the 17-channel EEGProject and 63-channel TSConv configurations, fixed 200-image gallery and query averaging. Tables 1–2 distinguish reproduced NeuroBridge and single-layer Shallow Alignment from published results and report all subjects. Stronger multi-layer Shallow Alignment and SAMGA are discussed, not presented as matched baselines.

The five-view inter-subject result is 31.95/61.75% (Top-1/Top-5), from a one-seed shared-projector TSConv configuration; 88.65/98.48% intra-subject is a three-seed separately projected result. We report settings and seed counts separately from the four-view global/SAT comparison. We corrected the entropy-table discrepancy, moved tables out of References, narrowed novelty and performance claims, and state remaining limitations.

Reviewer_MwdhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies EEG-to-image retrieval and proposes PPA-SAT. PPA extracts features from layer 28 of a frozen InternViT-6B encoder for five image views: the original image and versions with blur, noise, low-resolution degradation, and mosaic degradation. It then uses residual attention fusion conditioned on the EEG query and each candidate image to construct a query-conditioned visual representation. SAT introduces a subject-specific scalar offset to a shared contrastive temperature to calibrate training across source subjects. The paper evaluates the approach through component ablations, comparisons of fusion strategies, and qualitative examples.

Critical strengths
  1. EEG representations exhibit substantial variability. Allowing different queries to use different combinations of visual representations provides greater flexibility than aligning all queries with a fixed visual target. This is a reasonable approach to addressing variability in neural–visual alignment.
  2. SAT adds only one scalar parameter per source subject, introducing little additional parameter overhead. The paper also clearly identifies its role as source-subject calibration during training. The reported cross-subject improvements therefore suggest a potentially useful approach with a modest increase in model complexity.
Critical weaknesses
  1. Section 4.3 retains a statement that, before submission, the baselines should be checked for identical concept splits and candidate-gallery definitions, and that literature-reported results should be distinguished from independently reproduced results. However, the current manuscript does not document whether these checks were completed or identify the provenance of each baseline result. Consequently, I cannot establish whether the key comparisons in Tables 1 and 2 use consistent evaluation protocols. Since the paper’s main empirical claims rely on substantial performance improvements, this gap directly affects their credibility and is a primary reason for my current recommendation of rejection. The authors should clarify the sources of the reported results, identify any protocol differences, and provide comparisons with the key baselines under matched settings.
  2. The correspondence between the ablation configurations and the final model is unclear, weakening the attribution of the improvements to the proposed components. For example, Table 6 reports 91.35%/96.70% Top-1/Top-5 accuracy at β = 10⁻³, whereas Table 1 reports 91.70%/99.00%, without explaining the difference in settings. In addition, the “All five candidates” result in Table 4 matches “Residual Mean + Entropy” in Table 5 rather than the PPA result, and Table 5 does not specify whether SAT is enabled. These configuration differences need to be clarified before the results can establish the contributions of query-conditioned fusion and its combination with SAT.
Questions for authors
  1. Which results in Tables 1 and 2 were independently reproduced, and which were taken from prior publications? Were the key comparisons conducted with matched concept splits, candidate galleries, EEG trial-averaging procedures, and visual feature extractors? If not, how do the key baselines perform under matched settings?
  2. What configurations produced the results in Tables 1, 4, 5, and 6? Please specify the fusion mechanism, whether SAT was enabled, the entropy regularization coefficient, and any differences in training or evaluation. In particular, please explain the discrepancy between Tables 1 and 6 and the relationship between the configurations in Tables 4 and 5.
1 comment
  1. Provenance, protocol, and visual feature extractor We appreciate the reviewer’s concern that the original unfinished statement undermined the credibility of the comparisons. In revised Section 4.3, NeuroBridge and single-layer Shallow Alignment (layer 28) are identified as reproduced using their released code with common evaluation preprocessing, 80-repetition averaged test EEG, disjoint concept splits, 200-image galleries and test-based selection. ATM, Neural-MCRL, UBP, AVDE, HyFI and EEGiT are identified as published values. Matching the evaluation protocol does not make the methods’ visual extractors, EEG encoders or training architectures identical. All PPA visual targets in the controlled studies use frozen InternViT-6B layer-28 features. Our same-pipeline single-view controls are therefore the main evidence for the aggregation effect. Their three-seed Top-1 scores are 80.18% intra-subject and 22.67% inter-subject, versus 88.65% and 26.95% for uniform PPA. Tables 1–2 report all subjects and mark the source of each row. We have removed the unfinished text and refrain from claiming matched improvements over literature-only rows.
  2. Configuration inconsistencies and revised fusion experiments We agree that the old Tables 1, 4, 5 and 6 did not unambiguously document the configuration-to-result mapping. We cannot reliably reconstruct the exact cause of the 91.70/99.00 versus 91.35/96.70 discrepancy or establish that the old 79.20% entry represented the same model as the headline result. The new manuscript therefore supersedes those values rather than retrospectively assign them to particular settings. The revised exact pairwise scorer uses the actual EEG query for every candidate in training and retrieval. In the controlled three-seed intra-subject study, Table 3 specifies the following configurations, all with a learnable global temperature and no SAT: uniform (λ = 0, no entropy), 88.65/98.48; learned residual (λ ≈ 0.05, β_ent = 0), 83.67/97.67; learned residual (β_ent = 10⁻³), 83.67/97.72; attention-only (λ = 1), 77.93/95.73 (Top-1/Top-5 in percent). The original-only target is 80.18/96.82. Thus the corrected findings favor uniform aggregation, not query-conditioned fusion. Table 4 separately crosses visual targets with global temperature and SAT. Table 5 is a distinct capacity-matched view-count ablation, so it should not be compared as if it were the same training configuration.
  3. New matched results and temperature analysis In the shared-projector experiment, the original/four/five-view Top-1 results are 75.25/83.20/83.35% intra-subject and 28.20/31.90/31.95% inter-subject, with all ten subjects and one seed per setting. The five-view Top-5 scores are 96.90% and 61.75%, respectively. The shared-projector study holds projector capacity constant within each setting, so this comparison isolates view count more directly than the separate-projector versus single-view comparison. It does not isolate the effect of projection order across the two studies. For SAT, the uniform three-seed EEGProject global control (26.95/52.92) exceeds SAT (26.43/51.90), and the new matched five-view TSConv global control (31.90/61.15) slightly exceeds SAT (31.85/60.50).
  4. Model selection and scope of claims Section 4.1 now explicitly states the primary test-based checkpoint-selection convention and its optimistic nature. Appendix A.1 repeats the intra-subject study with independent validation concepts: single-view 69.58 ± 0.68% versus uniform PPA 77.83 ± 0.45% Top-1. Only one independent inter-subject fold is complete; we do not present it as a full benchmark. These revisions support the narrower conclusion that perceptual diversity improves retrieval in the matched settings, without claiming query-conditioned or temperature-adaptive superiority.
Reviewer_gZSjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

PPA-SAT combines EEG-conditioned fusion of five visual views with source-subject temperature offsets, reporting large gains on intra-subject and leave-one-subject-out THINGS-EEG retrieval.

Critical strengths

Conditioning visual fusion on both the EEG query and candidate is a sensible design. The reported gains are substantial and extend to both metrics for every LOSO subject against EEGiT.

Critical weaknesses
  1. The main comparisons are not yet verifiable. Section 4.3 retains an instruction to check baseline splits and galleries and distinguish published from reproduced results. Consequently, it is unclear whether the large gains over Shallow and EEGiT reflect matched evaluation conditions. This is a central evidential gap, not a minor presentation issue.

  2. The fusion ablations are underspecified. Table 4's all-five-priors result matches Residual Mean + Entropy rather than PPA in Table 5. Moreover, substituting uniform attention into Eqs. (5)–(6) collapses the residual interpolation to the same uniform mean, yet Uniform Mean and Residual Mean have different scores. Their actual implementations must differ in an unstated way.

  3. Table 3 never defines what changes between Baseline and Intermediate. Their matching averages with named methods in Table 2 do not establish a controlled component ablation.

  4. The smaller SAT effect deserves more caution: its Top-5 increment over PPA is only 0.45 points, without run-level uncertainty. Missing EEGProject and model-selection details further hinder interpretation.

Questions for authors

Which baselines were reproduced under the same preprocessing, query aggregation, split, and gallery as PPA-SAT?

2 comments
  1. Baseline provenance and matched comparison Thank you for highlighting the unfinished verification statement. We agree that the original table did not support a reliable comparison. Section 4.3 and Tables 1–2 now explicitly identify NeuroBridge and single-layer Shallow Alignment (layer 28) as reproduced rows using the released implementations under the same evaluation preprocessing, 80-repetition test-query averaging, disjoint concept split, 200-image gallery and test-based selection convention. ATM, Neural-MCRL, UBP, AVDE, HyFI and EEGiT are literature-reported rows; we do not claim that their protocols or visual backbones are matched. The strongest controlled comparison is our own original-only target under the same training pipeline. The three-seed intra-subject Top-1 changes from 80.18% to 88.65%, and inter-subject from 22.67% to 26.95%. The revised paper does not claim an overall state of the art.
  2. Fusion definitions and inconsistent original Tables 4–5 We agree with the algebraic observation. If every attention weight is 1/5, the residual weights w = (1 − λ)/5 + λα are exactly 1/5 for any λ. Consequently, uniform attention and a residual interpolation with uniform attention cannot be distinct scoring mechanisms. The earlier names and table entries did not reliably identify the underlying configurations. We have replaced these comparisons rather than assign a speculative explanation to the original 79.20% entry. Section 3.3 defines uniform aggregation (λ = 0), learned residual query-conditioned aggregation (λ ≈ 0.05), and attention-only aggregation (λ = 1), with the actual query and each candidate used for every pair. Table 3 reports three-seed Top-1/Top-5: uniform 88.65/98.48; residual without entropy 83.67/97.67; residual with β_ent = 10⁻³, 83.67/97.72; attention-only 77.93/95.73. All use the learnable global temperature and no SAT. The original 91.70% is superseded. These results do not support a benefit from query dependence.
  3. Controlled component and view-count ablations The revised Table 4 crosses the same visual target choices with a learnable global temperature or SAT under the EEGProject pipeline, rather than treating two external methods as controlled components. Table 5 independently varies only the shared-projector visual input: original, four transformed views, and all five. The intra-subject Top-1 values are 75.25, 83.20 and 83.35%; inter-subject they are 28.20, 31.90, and 31.95%. These one-seed runs use identical projector capacity within each setting. The results suggest that most of the gain comes from transformed views, not specifically from adding the fifth view.
  4. SAT uncertainty, encoder and selection We agree that the previously reported small increment was insufficient evidence. The three-seed inter-subject uniform PPA comparison gives 26.95 ± 0.75% Top-1 and 52.92 ± 0.54% Top-5 with a learnable global temperature, versus 26.43 ± 0.40% and 51.90 ± 0.48% with SAT. A matched TSConv four-view test-selected run gives 31.90/61.15 without SAT and 31.85/60.50 with SAT (one seed); the new five-view 31.95/61.75 result is not used as its SAT control. Sections 4.1–4.2 now specify EEGProject, 250 samples at 250 Hz (1 s), seed counts and checkpoint selection. We report independent intra-subject validation-selected results in Appendix A.1 and explicitly acknowledge the incomplete inter-subject validation study. The revised conclusion is that SAT has no consistent observed benefit.

The authors were honest in the rebuttal, but their new experiments show that both query-conditioned fusion and SAT do not actually help, and inter-subject validation is still incomplete. I keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.