- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper raises an important question about the standard cross-subject evaluation protocol on MEG-MASC: because every subject hears the same four stories, holding out a subject does not hold out the stimulus, which can reach the model both through the other subjects’ recordings in pretraining and through the held-out subject’s calibration segments. The authors propose a stimulus-disjoint (story-level) split that withholds the evaluation story from both pretraining and calibration, and use it to measure what the overlap is worth: withholding the story removes roughly half of the above-chance retrieval signal, and a low-budget advantage of one encoder over another that is resolved under the standard split becomes undetectable under the corrected one. They further provide a proposition showing that a subject-level split cannot identify the quantity researchers are actually interested in — the encoder difference on unheard stimuli — because it is confounded with an exposure term that only a stimulus-withheld split removes. The analysis is complemented by an audit of a published reference decoder, where a brain-free word-onset baseline outscores the MEG decoder and the cost of the corrected split is decomposed into training-volume and exposure components, and by released audit tools that flag such overlap.
Critical strengths
S1. The problem this paper raises is important and can reasonably be expected to affect future research based on the MEG-MASC dataset.
Critical weaknesses
W1. The writing appears to rely heavily on AI-generated text: many sentences use a mid-sentence colon (e.g., L13, L17, L21, L28, L30, L49, L86), and some sentences begin with a bare verb (L41). The manuscript needs further refinement by the human authors before it is acceptable, but I believe that a revision of the entire manuscript is not appropriate during the rebuttal period.
W2. While the main claim is important and a corrected split is needed for more accurate benchmarking of architectures, I am not convinced the contribution is sufficient for acceptance. Although the currently adopted split has the described issue, a subject- and stimulus-disjoint split is not new in itself; for example, in fMRI-based image decoding the widely used NSD dataset already provides a split in which test stimuli are disjoint from every subject’s training data.
W3. I am not fully convinced by the message of Proposition 1. Its assumption (Eq. 3) is not grounded in prior work or empirical findings and appears to be introduced mainly to support the message the authors want to deliver. For example, the additive decomposition rules out any interaction between stimulus difficulty and exposure, and any subject dependence of the exposure benefit, neither of which is justified.
W4. The results section depends heavily on the appendix. The manuscript should be self-contained; where results are discussed in detail, they should appear near the corresponding paragraph. For example, the results referred to in L260–265 are entirely in the appendix, and the same holds for L294–308.
W5. Although the authors acknowledge this in the limitations, the scope of the paper is very narrow: the causal claims rest on MEG-MASC alone (Narratives fMRI appears only as a corroboration without a pretraining stage). It would strengthen the paper to check whether similar findings hold on a different dataset.
W6. The evidence that the split changes the model comparison is fragile on two counts.
(a) The resolved k=500 split main effect is driven by the stimulus-guard/story-split cell, which reuses the SEEN arm of Table 1 (L271-L274) and therefore differs from the other three cells in pretraining (story 3 withheld, 54% of the segments), calibration stories and test story; with pretraining fixed (recording guard), the split simple effect is near zero and unresolved (App. C).
(b) Even taken at face value, both k=500 stars clear zero by less than the plausible undercoverage of a percentile bootstrap at n=10, the interaction would not survive Bonferroni correction, and subject counts were extended 5→10 after an interim look without sequential correction, so the architecture result effectively rests on a single cell (k=100, standard protocol).
I think that it would be better to include the results with all four cells pretrained on the same story set for (a), and confidence intervals should be reported with multiple-comparison correction or more reliable confidence intervals for the small number of n.
Questions for authors
Q1. The explanation of the encoder categorization (static vs. dynamic) is too limited, even though the two encoders are central to the method and the experimental results; a definition appears only in Appendix I. How does this categorization relate to existing work, and why should these particular encoder variants be considered?