acceptodds
Under review as a conference paper at ICLR 2027

Stimulus Overlap Changes Model Comparisons in Cross-Subject Brain Decoding

Abstract

Cross-subject brain decoding holds out the listener, not the story: on MEG-MASC every listener hears the same four stories, so the test story reaches the model twice, through the other listeners in pretraining and, under the standard split, through calibration. A subject-only split does not check this overlap, and the tested model comparison is sensitive to it. Under the standard protocol a dynamic encoder (no residual connections; its pooled feature adds the latent’s start-to-end displacement) beats a matched static convolutional one at low budgets (criterion pre-specified in ×chance at k=100; in rank percentile +0.043 and +0.014, n=10 paired). With story-disjoint calibration (subject split and nominal pretraining volume fixed) the advantage is undetectable; withholding the story from pretraining too, no narrative–budget combination resolves in its favour; at k=2000 the static encoder leads under the standard protocol and with the story withheld. This shows split-sensitivity at the tested sample size, not equal architectures. Transfer survives: withholding the story from pretraining at matched story count (not segment count) leaves about 54% of above-chance signal (k≥500; three narratives, three seeds). Returning 59% of a narrative’s training blocks to a reproduced reference pipeline at matched volume prices the exposure: five exposed runs separate completely from three unexposed (+1.01 to +4.59 top-10 points), and the full dose gives +3.00, same-signed on a second narrative (+2.00), each from a single full-dose run. Still open: whether split or adjacency guard does more, how much of the fully corrected cell (which also withholds a pretraining story) is the split, and how far unequal segment counts move the halved signal. Repeated stories are legitimate for some targets; for claims about unheard stimuli, hold the story out of pretraining and calibration and report a stimulus-derived baseline. We release two stdlib-only tools that flag this overlap; the main results rest on one MEG corpus.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain decoding, MEG, cross-subject transfer, evaluation protocol, data leakage, stimulus-disjoint split, model selection

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wATUOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper raises an important question about the standard cross-subject evaluation protocol on MEG-MASC: because every subject hears the same four stories, holding out a subject does not hold out the stimulus, which can reach the model both through the other subjects’ recordings in pretraining and through the held-out subject’s calibration segments. The authors propose a stimulus-disjoint (story-level) split that withholds the evaluation story from both pretraining and calibration, and use it to measure what the overlap is worth: withholding the story removes roughly half of the above-chance retrieval signal, and a low-budget advantage of one encoder over another that is resolved under the standard split becomes undetectable under the corrected one. They further provide a proposition showing that a subject-level split cannot identify the quantity researchers are actually interested in — the encoder difference on unheard stimuli — because it is confounded with an exposure term that only a stimulus-withheld split removes. The analysis is complemented by an audit of a published reference decoder, where a brain-free word-onset baseline outscores the MEG decoder and the cost of the corrected split is decomposed into training-volume and exposure components, and by released audit tools that flag such overlap.

Critical strengths

S1. The problem this paper raises is important and can reasonably be expected to affect future research based on the MEG-MASC dataset.

Critical weaknesses

W1. The writing appears to rely heavily on AI-generated text: many sentences use a mid-sentence colon (e.g., L13, L17, L21, L28, L30, L49, L86), and some sentences begin with a bare verb (L41). The manuscript needs further refinement by the human authors before it is acceptable, but I believe that a revision of the entire manuscript is not appropriate during the rebuttal period.

W2. While the main claim is important and a corrected split is needed for more accurate benchmarking of architectures, I am not convinced the contribution is sufficient for acceptance. Although the currently adopted split has the described issue, a subject- and stimulus-disjoint split is not new in itself; for example, in fMRI-based image decoding the widely used NSD dataset already provides a split in which test stimuli are disjoint from every subject’s training data.

W3. I am not fully convinced by the message of Proposition 1. Its assumption (Eq. 3) is not grounded in prior work or empirical findings and appears to be introduced mainly to support the message the authors want to deliver. For example, the additive decomposition rules out any interaction between stimulus difficulty and exposure, and any subject dependence of the exposure benefit, neither of which is justified.

W4. The results section depends heavily on the appendix. The manuscript should be self-contained; where results are discussed in detail, they should appear near the corresponding paragraph. For example, the results referred to in L260–265 are entirely in the appendix, and the same holds for L294–308.

W5. Although the authors acknowledge this in the limitations, the scope of the paper is very narrow: the causal claims rest on MEG-MASC alone (Narratives fMRI appears only as a corroboration without a pretraining stage). It would strengthen the paper to check whether similar findings hold on a different dataset.

W6. The evidence that the split changes the model comparison is fragile on two counts.

(a) The resolved k=500 split main effect is driven by the stimulus-guard/story-split cell, which reuses the SEEN arm of Table 1 (L271-L274) and therefore differs from the other three cells in pretraining (story 3 withheld, 54% of the segments), calibration stories and test story; with pretraining fixed (recording guard), the split simple effect is near zero and unresolved (App. C).

(b) Even taken at face value, both k=500 stars clear zero by less than the plausible undercoverage of a percentile bootstrap at n=10, the interaction would not survive Bonferroni correction, and subject counts were extended 5→10 after an interim look without sequential correction, so the architecture result effectively rests on a single cell (k=100, standard protocol).

I think that it would be better to include the results with all four cells pretrained on the same story set for (a), and confidence intervals should be reported with multiple-comparison correction or more reliable confidence intervals for the small number of n.

Questions for authors

Q1. The explanation of the encoder categorization (static vs. dynamic) is too limited, even though the two encoders are central to the method and the experimental results; a definition appears only in Appendix I. How does this categorization relate to existing work, and why should these particular encoder variants be considered?

Reviewer_ELvGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper points out that cross-subject decoding on MEG-MASC holds out the listener but not the story, because all listeners hear the same four stories. It studies how this overlap affects both the decoding scores and the ranking of two encoder architectures, and it audits a published MEG decoding pipeline, including comparing it with a brain-free baseline. The authors also release tools to detect such overlap in data splits.

Critical strengths

The most valuable part of the paper is its careful measurement of stimulus exposure. The volume-pinned dose sweep, the story-count- and composition-matched arms, and the A/B/C decomposition of the reference pipeline separate exposure from training volume more cleanly than prior split-sensitivity work. The released tools make the check cheap to adopt, and the pre-specified criteria and per-table receipts make the evidence easy to calibrate.

Critical weaknesses
  1. I think the most important problem with this article is that the writing is extremely poor. I can hardly make sense of the abstract. There is simply no clear explanation of the background or the problem presented. And after reading the entire article, I only knew what the article was exploring, but I had no idea why it was exploring this matter.

  2. The titular claim, that stimulus overlap changes model comparisons, is not statistically established. The only direct test, the architecture × split interaction, resolves only at k=500 and only before multiple-comparison correction (sign-flip p = 0.0625), and it is unresolved at k=100; the rest of the evidence is a contrast between a significant and a non-significant cell, which the authors themselves note is not a test. The encoder pair was built and selected after pilots, the subject count was extended after an interim look, and pretraining uses 9 subjects and 2.5M-parameter models, far from the foundation-model setting that motivates the paper. The result is therefore at most an existence proof for one toy pair.

  3. The proposed mechanism, template matching across brains, is also not isolated. The target head on the GPT-2 embeddings is trained jointly with the encoder during pretraining, so in SEEN arms it has already fit the evaluation story's text. Part of the exposure gain may come from the target space rather than from shared brain responses, and the clock result suggests the metric is largely winnable from the stimulus side. A frozen or story-excluded target head is needed before the gain can be attributed to cross-subject neural overlap.

Questions for authors
  1. Does the SEEN–UNSEEN gap persist when the target head is frozen, or trained without the evaluation story's text?

  2. Why are only 10 of MEG-MASC's 27 subjects used, and does a multiplicity-corrected interaction test hold when all of them are included?

  3. Does the split-sensitivity of the ranking replicate for a published architecture pair rather than the bespoke one?

Reviewer_LJ1wOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper audits stimulus overlap in cross-subject brain decoding using MEG-MASC. It distinguishes exposure to evaluation stories during pretraining from exposure during held-out-subject calibration and examines how protocol changes affect retrieval and a comparison between two encoders. Further experiments reproduce a reference decoding pipeline, introduce a brain-free word-onset baseline, and investigate exposure effects while attempting to control training volume and candidate-pool changes. The paper reports that useful transfer remains when evaluation stories are withheld, while some model-comparison conclusions are protocol-sensitive at the tested sample size. It also describes tools for auditing stimulus overlap in split manifests and corpus metadata.

Critical strengths
  1. The paper demonstrates exceptional methodological rigor, confound bracketing, and statistical transparency. Every secondary change induced by withholding a narrative, shifts in candidate pool size, pretraining segment volume, calibration diversity, and checkpoint selection is explicitly identified and bracketed through controlled experimental arms such as volume-pinned dose-response sweeps and thinned block splits. The authors' statistical reporting, which carefully distinguishes a loss of significance from a significant interaction, provides small-sample sensitivity checks, and documents pre-committed decision rules, sets a very high standard for empirical auditing.

  2. The paper distinguishes two generalization targets that are often conflated. Holding out a participant does not necessarily hold out the material that participant experienced, and shared-stimulus transfer is not equivalent to transfer to unheard narratives. The authors also acknowledge that repeated stimuli are legitimate for some tasks. This framing makes the audit more useful than a blanket claim that all stimulus overlap invalidates an experiment.

Critical weaknesses
  1. The model-comparison result is not cleanly isolated from other protocol changes. One cell of the guard-by-split design also changes pretraining data, calibration stories, and evaluation-story composition. Contrasts involving it therefore cannot identify a split effect alone. The architecture-by-split interaction is also sensitive to statistical treatment and is not resolved after the reported multiplicity correction or small-sample checks. The authors appropriately acknowledge that the evidence establishes protocol-sensitive conclusions at this sample size, not a confirmed change in the difference between encoders. Nevertheless, this limits the strength of a central contribution beyond the general overlap audit.

  2. While the underlying experimental design is meticulous, the main text is written in an overly compressed, telegraphic style that introduces internal arm labels, caveats, and numerical spans before the reader has a clear map of the experimental setup. Readers must frequently flip to the appendices to reconstruct which stories are in pretraining, calibration, and testing for each arm. Unpacking the Abstract into plain prose and adding a clear overview table of all experimental arms in the setup section would greatly improve accessibility.

Questions for authors
  1. Which conclusions about the relative costs of exposure and data removal remain supported when restricted to matched-volume, common-evaluation contrasts? Should A–B be labelled as a combined reduction in training volume and exposure, and how would that change the interpretation that full exposure is worth approximately half the volume cost?

  2. Given the modest compute footprint reported for the internal runs, why were only ten of the twenty-seven MEG-MASC subjects included in the factorial comparison? Running the remaining subjects could provide the statistical power needed to resolve the architecture-by-split interaction under multiple-comparison correction. In addition, since your codebase includes a control without the endpoint-difference branch, which of the two modifications (removing residual connections vs. adding endpoint displacement) drives the low-budget advantage under the overlapping protocol?

  3. Since brain-free timing and word-length baseline outperforms the MEG encoder in rank percentile, how much of the surviving unseen-story pretraining transfer is simply cross-subject transfer of acoustic or word-onset envelope tracking rather than lexical-semantic decoding? If linearly residualizing the target language-model embeddings with respect to word length and local timing features before evaluation, how much above-chance retrieval remains in the seen and unseen conditions?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.