acceptodds
Under review as a conference paper at ICLR 2027

Beyond Full-Context Accuracy: Context-Marginalized Evaluation of Test-Time Structure-Aware Inference in EEG-to-Image Retrieval

Abstract

Test-time structure-aware inference can improve cross-subject EEG-to-image retrieval by exploiting relations among unlabeled test queries, but it also makes predictions depend on query-set composition. Existing evaluations typically report accuracy under a single full-query context, leaving unclear whether gains and comparative conclusions remain reliable when the available context changes. We introduce Context-Marginalized Retrieval Evaluation (CMRE), which repeatedly evaluates each target query within sampled subsets of companion queries, applying each inference rule to the corresponding rows of a fixed query–gallery similarity matrix. CMRE distinguishes structural exploitability, quantified by gains over independent retrieval, from context robustness, measured by same-query Top-1 context consistency. Under leave-one-subject-out evaluation on THINGS-EEG2 across six frozen representations and six structure-aware inference rules, structure-aware inference improves full-context Top-1 accuracy by up to 20.40 percentage points over independent retrieval. However, the rule selected under full-context evaluation is no longer optimal in 17 of 24 representation–context-size conditions (70.8%). Selecting the full-context winner incurs an average loss of 2.70 percentage points and a maximum loss of 8.29 percentage points under reduced contexts. Representation rankings are likewise inference- and context-dependent: the ordering under independent retrieval is not consistently preserved under structure-aware inference. Moreover, larger retrieval gains do not necessarily imply greater context robustness, showing that retrieval effectiveness and prediction stability provide complementary views of structure-aware inference. Together, these results show that full-context accuracy represents only one operating point of context-dependent inference. CMRE complements conventional evaluation by characterizing structural gain, context consistency, and the context dependence of conclusions about inference rules and frozen EEG-to-image representations.

Primary area:
datasets and benchmarks
Keywords:
EEG-to-image retrieval, cross-subject neural decoding, test-time inference, structure-aware inference, structural exploitability, context robustness

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TwceOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Context-Marginalized Retrieval Evaluation (CMRE) to study context dependence in test-time EEG-to-image retrieval. By keeping similarity matrices fixed and sampling different query contexts, CMRE measures retrieval accuracy, structural gains, and prediction consistency. Experiments with six frozen representations and six inference rules show that the best rule can change with context, while larger retrieval gains do not always mean more stable predictions. Additional experiments on THINGS-MEG provide supporting evidence. The paper is clear and easy to follow, and the motivation, method, and results are presented well. Some further discussion of the scientific implications and the reliability of some ranking changes would be helpful.

Critical strengths
  1. The paper addresses a real and meaningful problem. Since structure-aware inference uses information from other test queries, full-context accuracy alone may not tell the whole story. CMRE provides a reasonable way to study this issue. Keeping the similarity matrices fixed also makes the comparison between inference rules more controlled.

  2. The experiments cover six representations and six inference rules under LOSO evaluation. The results show meaningful changes in method rankings, such as the reversal between Hungarian and Sinkhorn on BLUR (Section 4.3, Figure 3). Separating retrieval gains from prediction consistency is also useful. The analysis of stable correct and stable wrong predictions (Appendix E.2) helps with interpreting the results.

Critical weaknesses
  1. Limited incremental scientific insight: I can see novelty in CMRE, but the findings are still mostly descriptive. Previous studies on transductive evaluation [1, 2] have examined how query-set assumptions affect performance. SATTC [3, Section 4.3] also studied retrieval with smaller unlabeled calibration windows, including N=50 and N=200. The evaluation protocols may differ between these works, but I think the additional scientific insight of CMRE could be explained more clearly. CMRE shows that context matters, but gives limited explanation of what drives these changes or when full-context evaluation remains reliable. A deeper analysis of these factors would make the contribution more convincing.
  2. Uncertainty in the main ranking results: The paper reports 17/24 ranking changes and an average selection regret of 2.70 pp. Based on Tables 1 and 8, I obtain 16/24 changes and around 2.68 pp from the reported values. Some ranking reversals are also very small. For example, NICE at m=25 differs by only 0.06 pp, while ATM at m=50 and m=150 differs by 0.05 and 0.03 pp. Appendix D.2 tests the overall gains over independent retrieval, but not directly the reliability of these ranking reversals. Appendix F checks convergence using one shared sampling schedule, rather than different independent samplings. The authors should clarify these results, especially how stable the main ranking conclusions are.
  3. Limited evaluation scope and structural assumptions: The main experiments use randomly sampled contexts from a fixed 200-way retrieval setting, with one ground-truth image per query (Section 3). As context size changes, the query-to-gallery ratio also changes. This may be especially important for assignment-based methods like Hungarian. Based on Tables 1 and 8, around 87% of the total selection regret comes from the three representations where Hungarian is the full-context winner. Appendix G also shows that Hungarian's gains decrease when unmatched gallery candidates are added. Still, that analysis only uses BLUR and does not jointly examine different context sizes. I would also like to know if these findings hold for streaming queries, repeated targets, or imbalanced query batches.
Questions for authors
  1. Compared with earlier transductive evaluation studies [1, 2] and SATTC [3], what additional scientific insight does CMRE provide beyond the evaluation protocol? Can the observed context dependence be linked to specific properties of the query sets or inference rules?

  2. Could the authors clarify the reported 17/24 ranking changes and 2.70 pp average selection regret? Based on Tables 1 and 8, I obtain 16/24 and around 2.68 pp. Also, how stable are these ranking changes across held-out subjects and independently sampled contexts?

  3. How much of the selection regret is related to Hungarian's one-to-one matching constraints and the changing query-to-gallery ratio? Would the main observations still hold with streaming queries, repeated targets, or unmatched gallery candidates?

[1] Veilleux, O., Boudiaf, M., Piantanida, P., and Ben Ayed, I. Realistic Evaluation of Transductive Few-Shot Learning. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.

[2] Martin, S., Boudiaf, M., Chouzenoux, E., Pesquet, J.-C., and Ben Ayed, I. Towards Practical Few-Shot Query Sets: Transductive Minimum Description Length Inference. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.

[3] Huang, Q., and Zhu, W. SATTC: Structure-Aware Label-Free Test-Time Calibration for Cross-Subject EEG-to-Image Retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16887–16896, 2026.

1 comment

The rebuttal addresses most of my concerns, particularly regarding the ranking results, statistical reliability, and evaluation settings. I see the contribution mainly as an evaluation framework and empirical study, with some limitations in its generality and explanation of context dependence. Even so, CMRE provides useful insights. The additional analyses on Hungarian, gallery size, and streaming are helpful. Some of these results could also be included in the main paper. I will maintain my rating. Good luck.

Reviewer_Qy9jOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Context-Marginalized Retrieval Evaluation (CMRE) for examining how EEG-to-image retrieval predictions depend on the accompanying test queries. Using frozen similarity matrices, it compares independent retrieval with several set-level and assignment-based inference rules across different query-context sizes. Experiments cover six EEG representations, with additional MEG results. The main finding is that full-context accuracy can obscure changes in method rankings and prediction consistency.

Critical strengths
  1. Fixing the representations and sharing sampled contexts across inference rules provides a reasonably controlled comparison.

  2. The evaluation includes multiple representations and inference methods, additional MEG experiments, and explicit analyses of consistently incorrect predictions.

Critical weaknesses
  1. CMRE primarily combines query-subset resampling, averaged retrieval accuracy, and prediction agreement. The paper needs to establish more clearly what this protocol reveals beyond the query-set sensitivity already discussed in related transductive evaluation work.

  2. Some reversals involve negligible differences: for ATM at (m=150), Sinkhorn achieves 15.40% versus 15.37% for Partial OT. Counting such cases alongside substantial reversals makes the “17/24” result difficult to interpret without paired confidence intervals or a practical tie threshold.

  3. With fewer queries and a fixed gallery, uniform balanced transport still allocates mass to gallery items without corresponding queries. Controls using unbalanced or adaptive marginals are needed to separate context sensitivity from this change in the assignment problem.

  4. Each query receives 10 evaluations for the smaller contexts but 30 evaluations from overlapping complements at (m=150). A matched-budget experiment with independently sampled contexts would help establish whether consistency comparisons depend on this construction.

  5. Appendix F compares prefixes of one sampling schedule against its longer prefix. This measures convergence within that schedule, but not variation across independently generated schedules, which matters particularly for near-tied rankings.

  6. The gallery-expansion study uses one representation with the full query set, while the main contexts are randomly constructed. Joint changes in gallery size and query composition, repeated concepts, and nonuniform query streams remain insufficiently examined.

Questions for authors
  1. How many of the 17 ranking reversals remain after accounting for paired subject-level uncertainty and treating negligible differences as ties?

  2. Do the consistency results and stable-prediction decompositions persist when every context size uses the same number of independently sampled contexts?

  3. Do the main conclusions survive reasonable hyperparameter variations and controls using unbalanced transport or adaptive gallery marginals?

  4. Can the exact frozen similarity matrices, checkpoint identifiers, query-construction settings, and evaluation seeds be provided?

Reviewer_wp7QOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses an important and hard problem on cross-subject EEG-image retrieval tasks. This benchmarking paper specifically proposes Context-Marginalized Retrieval Evaluation(CMRE) as a new benchmark for evaluating test-time structural aware inference queries in cross-subject EEG-image retrieval contexts. CMRE repeatedly evaluates each target query within sampled subsets of companion queries, applying each inference rule to the corresponding rows of a fixed query–gallery similarity matrix. The paper evaluates a broad array of SOTA EEG-image retrieval algorithms across a broad array of contexts and target queries.

Critical strengths
  1. It would really good for ICLR to encourage more such benchmark papers in this area especially to provide an independent evaluation of several SOTA algorithms and determining new evaluation metrics to compare these algorithms.

  2. The evaluation methodology is quite strong across the range of EEG-image retrieval algorithms.

Critical weaknesses
  1. The paper is quite simple in its core methodology and is easy to understand. If novelty is a core requirement for a dataset and benchmarks paper, then this paper may be viewed as a bit weak on this front. That being said, there is technical merit to the CMRE benchmarking metric proposed in this paper.

  2. I find a lot of the analyses in EEG-image retrieval contexts very limiting especially in their use of the THINGSEEG2 dataset which is limited to 10 subjects. The overall set of statements on Leave one subject out (LOSO) as a fundamental evaluation contribution, is a bit too simplistic and obvious in retrospect. My larger worry is given the limited subject availability, are these evaluation numbers truly statistically meaningful?

  3. (Not a weakness per se): I am curious to see how the numbers reported in this paper for the various tasks compare with the numbers reported by the authors in the original papers over similar datasets. In a lot of the evaluations, the numbers look very poor for many SOTA algorithms which may be a contribution to the research community in its own right, if true. The onus is on the authors to demonstrate this for a benchmarks paper.

Questions for authors

Thank you for writing this paper and doing a detailed evaluation and coming up with a new benchmarking metric in the EEG-image retrieval problem context. Can you answer the three weakness queries restated below:

  1. The paper is quite simple in its core methodology and is easy to understand. If novelty is a core requirement for a dataset and benchmarks paper, then this paper may be viewed as a bit weak on this front. That being said, there is technical merit to the CMRE benchmarking metric proposed in this paper.

  2. I find a lot of the analyses in EEG-image retrieval contexts very limiting especially in their use of the THINGSEEG2 dataset which is limited to 10 subjects. The overall set of statements on Leave one subject out (LOSO) as a fundamental evaluation contribution, is a bit too simplistic and obvious in retrospect. My larger worry is given the limited subject availability, are these evaluation numbers truly statistically meaningful?

  3. (Not a weakness per se): I am curious to see how the numbers reported in this paper for the various tasks compare with the numbers reported by the authors in the original papers over similar datasets. In a lot of the evaluations, the numbers look very poor for many SOTA algorithms which may be a contribution to the research community in its own right, if true. The onus is on the authors to demonstrate this for a benchmarks paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.