- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces GLANCE, a model that picks out which sentence a person heard from sEEG recorded while they watched movies (Brain Treebank). Each electrode lead gets its own BiGRU, and a spatial Transformer then combines the leads at each time bin. From there a global path matches a whole-sentence brain vector to a lightly adapted frozen EmbeddingGemma sentence embedding. A local path matches every time bin to contextual word embeddings using log-mean-exp pooling, with no word timing given. Training uses a multi-positive InfoNCE loss plus a masked second view. The model is trained separately on each of 13 recordings from 4 subjects and reaches 26.2% subject-averaged R@10 in 100-way retrieval (chance is 10%). On one recording it beats adapted SSENSE, DEEPER, BPR, BrainMosaic and ridge baselines, and ablations on that recording support the local path and the masking.
Critical strengths
-
The task is set up as 100-way retrieval with a known chance level, which avoids the inflated metrics that have caused problems in EEG-to-text generation work.
-
Using recording leads as the spatial unit, with a GRU per lead and attention across leads, is a sensible way to handle sEEG montages that differ across patients.
-
The local path scores each candidate word against brain time bins without needing word onset labels, which is a reasonable idea for natural listening.
-
All baselines use the same splits and the same 30 fixed candidate banks, so the comparison numbers line up directly. The window truncation analysis looks only at cutoffs up to 1 s, where every query keeps the same amount of signal, which is a careful way to rule out a length cue at those cutoffs.
Critical weaknesses
-
The window runs from sentence onset to 1 s after offset, so the number of valid bins tells the model the sentence duration, and no duration-only or word-count-only baseline shows how much retrieval this cue alone explains.
-
Trials are split at random within one continuous movie, so test sentences sit between training sentences that are only seconds away, and slow neural drift or scene context could inflate the scores compared with a blocked split by time.
-
Because windows extend 1 s past offset, windows of back-to-back lines can overlap in raw sEEG across the train and test splits, and the paper does not say whether this is handled.
-
All baseline comparisons and ablations are run only on Subject 2 Spider-Man, which is also GLANCE's best recording, so it is unclear whether the margins hold elsewhere.
-
DEEPER and BPR use DistilBERT text features while GLANCE and SSENSE use EmbeddingGemma, so part of the gap may come from the text encoder and not the brain model.
-
The gain from the local path is small, about 1 pp on average (25.2 vs 26.2 R@10) and 1.2 pp on the main recording, which is within one seed SD, and no paired test across seeds or recordings backs the "11 of 13" claim.
Questions for authors
-
What R@10 do you get by ranking candidates using only the gap between neural window length and transcript duration or word count?
-
How do the results change if you split each movie into contiguous time blocks with a gap between train and test?
-
How often do neural windows from training trials overlap in time with windows from test trials, and how did you deal with this?
-
Can you run the baselines and the No Local and No Masking ablations on at least one recording from each of the other three subjects?
-
How does BPR or DEEPER do when you give it the same EmbeddingGemma targets as GLANCE?
-
How does the model do when the candidates are matched to the true sentence on duration and word count?
-
What were the "Other" exclusions in Table 4, for example 443 of 998 sentences for Coraline and 653 for Subject 6 Megamind?
7 comments
Response to Reviewer arh7 — Part 2
Continued W1 / Q1 and Q6.
Fixed-duration neural-input control. To complement this evaluation with a control that directly removes variation in neural-input length, we additionally retrained and evaluated GLANCE using only the first 1.37 seconds after sentence onset. Windows shorter than this duration were excluded, so every retained trial provided the same number of valid neural-input bins.
| Recording | Fixed-duration GLANCE R@10 |
|---|---|
| S2 Spider-Man | 17.6 ± 2.1 |
| S3 Cars 2 | 15.0 ± 1.2 |
R@10 (%), mean ± SD across three blocked runs, using validation-selected checkpoints; chance R@10 = 10%.
These findings suggest that GLANCE is capable of retaining retrieval information when variation in neural-input length is removed.
For the main analyses, we emphasize duration-matched retrieval because it preserves the available sentence-related neural activity, whereas a fixed 1.37-second cutoff discards later activity from longer sentences. We interpret these results together with the fixed-duration control, which more directly addresses whether retrieval can persist without differences in input-window length.
W4 / Q4. Baseline comparisons across all four participants
We have expanded baseline evaluation to include Spider-Man, Cars 2, Shrek, and Toy Story, covering all four participants. Within each recording, all compared models use the same seeds, blocked layouts, test examples, and candidate sets.
| Recording | Model | R@10 | Duration-matched R@10 |
|---|---|---|---|
| S2 Spider-Man | GLANCE | 19.0 ± 1.4 | 16.7 ± 0.4 |
| S2 Spider-Man | SSENSE | 18.4 ± 2.2 | 13.1 ± 1.6 |
| S2 Spider-Man | BPR | 17.3 ± 1.1 | 15.9 ± 1.3 |
| S2 Spider-Man | DEEPER | 14.6 ± 1.8 | 14.9 ± 0.8 |
| S2 Spider-Man | BrainMosaic | 15.4 ± 1.9 | 11.8 ± 2.2 |
| S3 Cars 2 | GLANCE | 15.9 ± 0.5 | 14.6 ± 0.7 |
| S3 Cars 2 | SSENSE | 19.4 ± 2.0 | 13.7 ± 0.6 |
| S3 Cars 2 | BPR | 15.1 ± 1.4 | 14.2 ± 1.6 |
| S3 Cars 2 | DEEPER | 15.0 ± 0.2 | 14.4 ± 0.9 |
| S3 Cars 2 | BrainMosaic | 15.2 ± 3.1 | 10.0 ± 1.4 |
| S4 Shrek | GLANCE | 12.2 ± 3.1 | 11.8 ± 1.7 |
| S4 Shrek | SSENSE | 16.0 ± 5.2 | 12.0 ± 3.5 |
| S4 Shrek | BPR | 10.1 ± 1.0 | 8.9 ± 0.9 |
| S4 Shrek | DEEPER | 10.4 ± 0.3 | 10.3 ± 0.8 |
| S4 Shrek | BrainMosaic | 11.2 ± 1.9 | 10.3 ± 0.8 |
| S6 Toy Story | GLANCE | 12.7 ± 1.0 | 11.9 ± 1.0 |
| S6 Toy Story | SSENSE | 14.3 ± 1.3 | 11.3 ± 1.3 |
| S6 Toy Story | BPR | 10.8 ± 1.8 | 8.7 ± 2.2 |
| S6 Toy Story | DEEPER | 7.9 ± 1.5 | 8.1 ± 0.9 |
| S6 Toy Story | BrainMosaic | 12.5 ± 2.0 | 10.9 ± 3.3 |
R@10 (%), mean ± SD across three seed runs per recording.
GLANCE achieves the highest mean duration-matched R@10 in three of the four recordings (Spider-Man, Cars 2, and Toy Story). On the standard candidate evaluation, SSENSE achieves higher R@10 on three recordings. This pattern might suggest that SSENSE's advantage under standard evaluation is more sensitive to candidate-duration composition, while GLANCE's retrieval performance is comparatively stable under duration-matched evaluation.
We also evaluated GLANCE, SSENSE, and BPR using fixed 1.37-second neural windows beginning at sentence onset, so that every trial provided the same duration of neural input. This comparison examines retrieval performance after removing variation in input-window duration.
| Recording | Model | Fixed-input R@10 (%) | MRR |
|---|---|---|---|
| S2 Spider-Man | GLANCE | 17.6 ± 2.1 | 0.082 ± 0.004 |
| S2 Spider-Man | SSENSE | 12.7 ± 1.1 | 0.061 ± 0.006 |
| S2 Spider-Man | BPR | 15.5 ± 1.3 | 0.075 ± 0.002 |
| S3 Cars 2 | GLANCE | 15.0 ± 1.2 | 0.072 ± 0.008 |
| S3 Cars 2 | SSENSE | 12.8 ± 2.7 | 0.062 ± 0.009 |
| S3 Cars 2 | BPR | 14.3 ± 3.7 | 0.072 ± 0.016 |
Values are mean ± SD across three split layouts under temporally blocked evaluation. This fixed-input comparison uses validation-selected checkpoints.
GLANCE achieves the highest mean R@10 among the three models on both recordings and retains mean performance above the 10% random-ranking reference. These results provide additional evidence that variation in neural-input duration is not necessary for GLANCE’s retrieval performance.
Response to Reviewer arh7 — Part 3
W5 / Q5. Baselines with the same EmbeddingGemma targets
To address the text-encoder concern directly, we retrained BPR and SSENSE using EmbeddingGemma representations. The following results use the same blocked split datasets.
| Recording | BPR, DistilBERT | BPR, EmbeddingGemma | SSENSE, CLIP | SSENSE, EmbeddingGemma |
|---|---|---|---|---|
| S2 Spider-Man | 17.3 ± 1.1 | 14.3 ± 1.6 | 18.4 ± 2.2 | 15.5 ± 1.1 |
| S3 Cars 2 | 15.1 ± 1.4 | 12.5 ± 1.6 | 19.4 ± 2.0 | 14.7 ± 2.3 |
Standard R@10 (%), mean ± SD.
Replacing the baselines’ text representations with EmbeddingGemma did not improve their retrieval performance in these experiments. This suggests that the comparative results cannot be explained simply by assuming that EmbeddingGemma provides a uniform advantage. It does not fully separate the effects of architecture, optimization, and text representation.
We will explicitly identify the text encoder used in each condition, including the distinction between the original SSENSE adaptation using Gemma targets and the SSENSE-CLIP implementation evaluated here.
W6 / Q4. Expanded ablations and evidence for the local pathway
We agree that a small average improvement, or a count of recordings favoring one condition, is insufficient on its own to establish a reliable contribution.
We therefore expanded the retrained No Local and No Masking comparisons to four recordings, one from each participant, using the same layout. Each ablation is compared with its corresponding full-model control.
| Recording | Model | R@10 | Duration-matched R@10 | MRR | Duration-matched MRR |
|---|---|---|---|---|---|
| S2 Spider-Man | Full GLANCE | 19.0 ± 1.4 | 16.7 ± 0.4 | 0.084 ± 0.004 | 0.078 ± 0.004 |
| S2 Spider-Man | No Local | 17.4 ± 1.3 | 15.6 ± 0.6 | 0.082 ± 0.013 | 0.073 ± 0.011 |
| S2 Spider-Man | No Masking | 17.7 ± 1.8 | 16.5 ± 1.4 | 0.082 ± 0.003 | 0.076 ± 0.004 |
| S3 Cars 2 | Full GLANCE | 15.9 ± 0.5 | 14.6 ± 0.7 | 0.079 ± 0.002 | 0.074 ± 0.001 |
| S3 Cars 2 | No Local | 15.8 ± 0.8 | 14.5 ± 0.9 | 0.075 ± 0.003 | 0.068 ± 0.004 |
| S3 Cars 2 | No Masking | 16.5 ± 2.4 | 13.7 ± 1.6 | 0.080 ± 0.010 | 0.072 ± 0.009 |
| S4 Shrek | Full GLANCE | 12.2 ± 3.1 | 11.8 ± 1.7 | 0.054 ± 0.005 | 0.052 ± 0.006 |
| S4 Shrek | No Local | 10.7 ± 2.0 | 10.5 ± 1.8 | 0.048 ± 0.006 | 0.049 ± 0.008 |
| S4 Shrek | No Masking | 11.0 ± 0.8 | 10.0 ± 1.3 | 0.051 ± 0.002 | 0.049 ± 0.004 |
| S6 Toy Story | Full GLANCE | 12.7 ± 1.0 | 11.9 ± 1.0 | 0.061 ± 0.009 | 0.060 ± 0.009 |
| S6 Toy Story | No Local | 11.8 ± 1.2 | 12.39 ± 1.2 | 0.057 ± 0.008 | 0.054 ± 0.007 |
| S6 Toy Story | No Masking | 11.1 ± 2.3 | 9.7 ± 1.5 | 0.054 ± 0.012 | 0.052 ± 0.011 |
Mean ± SD across runs from three seeds.
Full GLANCE achieves higher mean standard R@10 than No Local in all four recordings, with differences of 0.1–1.6 percentage points and an average difference of approximately 1.0 percentage point. Mean R@1, standard MRR, and duration-matched MRR also favor full GLANCE in all four recordings.
To assess these differences, we performed two-sided paired t-tests comparing full GLANCE with the retrained No Local ablation. Full GLANCE achieved higher R@1, MRR, and duration-matched MRR at the unadjusted 0.05 significance level. The differences in standard and duration-matched R@10 did not reach this threshold.
The masking ablation shows higher mean duration-matched R@10 for the full model in all four recordings, while the standard R@10 comparison reverses on Cars 2. Its differences therefore vary by recording and metric, and we will calibrate its interpretation accordingly.
These results suggest that contextual word-level matching can provide complementary information for sentence retrieval, with a contribution that varies across recordings and metrics.
Q7. Meaning of the “Other” exclusions
The “Other” exclusions primarily reflected missing neural recording coverage or unavailable synchronization triggers and timing information. For Coraline, 443 short sentences could not be assigned a validated movie-time-to-neural-sample alignment because the available trigger and timing segments did not cover those transcript times. For Subject 6 / Megamind, 634 of the 653 excluded sentences occurred after the end of the available subject-specific neural recording, so no neural samples were available even at sentence onset. We will clarify this breakdown in Table 4.
Response to Reviewer arh7
We thank the reviewer for the thoughtful assessment and for recognizing the controlled retrieval setting, lead-based architecture, and motivation for timing-free local matching. Your comments helped us identify several important evaluation improvements. We have expanded the experiments to address temporal separation, overlapping windows, duration-related cues, comparisons across participants, and the contribution of the local pathway.
W2–W3 / Q2–Q3. Temporally blocked evaluation and overlapping neural windows
We agree that random trial splitting within a continuous movie does not adequately separate training and test data in time. We have therefore adopted temporally blocked splits for the revised evaluation.
Each recording is divided into ten contiguous, equal-duration blocks, with seven assigned to training, two to testing, and one to validation. Windows within five seconds of boundaries separating different splits are excluded, leaving at least ten seconds between retained training and test windows. Transcript identities shared with training are excluded from the held-out sets, and normalization and text centering use training data only.
We also audited the original split. For the original Spider-Man test partition, 69.91% windows overlapped at least one training window when the one-second post-offset extension was included. These counts describe the original random split. We are therefore replacing the original random-split evaluation with the revised temporally blocked evaluation throughout the paper. The revised protocol introduces the temporal separation described above, preventing cross-split model-window overlap even with the one-second post-offset extension retained.
The revised results across all 13 recordings are shown below. Values are mean ± SD across runs from three seeds. Recall metrics are percentages; random-ranking references are 1% for R@1 and 10% for R@10.
Values are mean ± SD across three seeds. Recall metrics are percentages. R@10-M denotes duration-matched R@10. LOTR = Lord of the Rings.
Performance is lower under this more demanding evaluation than under the original random split. Nevertheless, mean R@10 remains above the 10% random-ranking reference in all 13 recordings, including with duration-matched candidates.
Also, a complementary time-only ridge control, using sentence position within the movie without neural features, achieves R@10 of 10.4%, 9.6%, 8.4%, and 10.7% across Subject 2’s four recordings. These results suggest that movie position alone does not reproduce GLANCE’s retrieval performance under temporal blocking.
W1 / Q1 and Q6. Duration-related cues and matched candidates
We agree that variable input duration can expose information about sentence length. Following the reviewer’s suggestion, we evaluated duration-only and word-count-only rankers. In the separate audit covering all 13 recordings, these achieved 98.9% and 27.1% standard R@10, respectively. The word-count ranker predicts duration from candidate word count using a mapping fitted on training examples only. These results confirm that length information is available in the variable-window evaluation.
Duration-matched candidate evaluation. These findings motivated us to construct duration-matched candidate sets within the held-out test partitions. For each query, the candidate set contains the correct transcript and the 99 held-out distractor occurrences nearest to it in spoken duration, excluding occurrences with the correct transcript identity. This approximately matches candidate durations, although it does not completely eliminate duration-related cues.
We report results under this protocol alongside standard retrieval throughout our revised evaluation, including the main GLANCE results, baseline comparisons, ablations, and cross-movie experiments. The Duration-matched R@10 column in the table above reports performance under this evaluation.