- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Context-Marginalized Retrieval Evaluation (CMRE) to study context dependence in test-time EEG-to-image retrieval. By keeping similarity matrices fixed and sampling different query contexts, CMRE measures retrieval accuracy, structural gains, and prediction consistency. Experiments with six frozen representations and six inference rules show that the best rule can change with context, while larger retrieval gains do not always mean more stable predictions. Additional experiments on THINGS-MEG provide supporting evidence. The paper is clear and easy to follow, and the motivation, method, and results are presented well. Some further discussion of the scientific implications and the reliability of some ranking changes would be helpful.
Critical strengths
-
The paper addresses a real and meaningful problem. Since structure-aware inference uses information from other test queries, full-context accuracy alone may not tell the whole story. CMRE provides a reasonable way to study this issue. Keeping the similarity matrices fixed also makes the comparison between inference rules more controlled.
-
The experiments cover six representations and six inference rules under LOSO evaluation. The results show meaningful changes in method rankings, such as the reversal between Hungarian and Sinkhorn on BLUR (Section 4.3, Figure 3). Separating retrieval gains from prediction consistency is also useful. The analysis of stable correct and stable wrong predictions (Appendix E.2) helps with interpreting the results.
Critical weaknesses
- Limited incremental scientific insight: I can see novelty in CMRE, but the findings are still mostly descriptive. Previous studies on transductive evaluation [1, 2] have examined how query-set assumptions affect performance. SATTC [3, Section 4.3] also studied retrieval with smaller unlabeled calibration windows, including N=50 and N=200. The evaluation protocols may differ between these works, but I think the additional scientific insight of CMRE could be explained more clearly. CMRE shows that context matters, but gives limited explanation of what drives these changes or when full-context evaluation remains reliable. A deeper analysis of these factors would make the contribution more convincing.
- Uncertainty in the main ranking results: The paper reports 17/24 ranking changes and an average selection regret of 2.70 pp. Based on Tables 1 and 8, I obtain 16/24 changes and around 2.68 pp from the reported values. Some ranking reversals are also very small. For example, NICE at m=25 differs by only 0.06 pp, while ATM at m=50 and m=150 differs by 0.05 and 0.03 pp. Appendix D.2 tests the overall gains over independent retrieval, but not directly the reliability of these ranking reversals. Appendix F checks convergence using one shared sampling schedule, rather than different independent samplings. The authors should clarify these results, especially how stable the main ranking conclusions are.
- Limited evaluation scope and structural assumptions: The main experiments use randomly sampled contexts from a fixed 200-way retrieval setting, with one ground-truth image per query (Section 3). As context size changes, the query-to-gallery ratio also changes. This may be especially important for assignment-based methods like Hungarian. Based on Tables 1 and 8, around 87% of the total selection regret comes from the three representations where Hungarian is the full-context winner. Appendix G also shows that Hungarian's gains decrease when unmatched gallery candidates are added. Still, that analysis only uses BLUR and does not jointly examine different context sizes. I would also like to know if these findings hold for streaming queries, repeated targets, or imbalanced query batches.
Questions for authors
-
Compared with earlier transductive evaluation studies [1, 2] and SATTC [3], what additional scientific insight does CMRE provide beyond the evaluation protocol? Can the observed context dependence be linked to specific properties of the query sets or inference rules?
-
Could the authors clarify the reported 17/24 ranking changes and 2.70 pp average selection regret? Based on Tables 1 and 8, I obtain 16/24 and around 2.68 pp. Also, how stable are these ranking changes across held-out subjects and independently sampled contexts?
-
How much of the selection regret is related to Hungarian's one-to-one matching constraints and the changing query-to-gallery ratio? Would the main observations still hold with streaming queries, repeated targets, or unmatched gallery candidates?
[1] Veilleux, O., Boudiaf, M., Piantanida, P., and Ben Ayed, I. Realistic Evaluation of Transductive Few-Shot Learning. Advances in Neural Information Processing Systems (NeurIPS), 34, 2021.
[2] Martin, S., Boudiaf, M., Chouzenoux, E., Pesquet, J.-C., and Ben Ayed, I. Towards Practical Few-Shot Query Sets: Transductive Minimum Description Length Inference. Advances in Neural Information Processing Systems (NeurIPS), 35, 2022.
[3] Huang, Q., and Zhu, W. SATTC: Structure-Aware Label-Free Test-Time Calibration for Cross-Subject EEG-to-Image Retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16887–16896, 2026.
The rebuttal addresses most of my concerns, particularly regarding the ranking results, statistical reliability, and evaluation settings. I see the contribution mainly as an evaluation framework and empirical study, with some limitations in its generality and explanation of context dependence. Even so, CMRE provides useful insights. The additional analyses on Hungarian, gallery size, and streaming are helpful. Some of these results could also be included in the main paper. I will maintain my rating. Good luck.