acceptodds
Under review as a conference paper at ICLR 2027

GLANCE: Global–Local Alignment of Neural and Contextual Embeddings for Brain–Language Sentence Retrieval

Abstract

Identifying heard language from neural activity requires models that align distributed, time-varying brain signals with linguistic representations. We introduce GLANCE (Global–Local Alignment of Neural and Contextual Embeddings), a framework for sentence retrieval from stereo-electroencephalography (sEEG) recorded during naturalistic movie viewing. GLANCE models within-lead temporal dynamics using recurrent encoders, combines information across leads with spatial attention, and integrates temporal context into a global sentence representation. A complementary local pathway compares time-resolved neural features with contextual word embeddings from each candidate transcript. Both pathways operate in embedding space, combining whole-sentence similarity with word- level evidence without requiring word-onset supervision or a fixed word-to-time alignment. GLANCE is trained with a multi-positive contrastive objective that accounts for repeated transcripts, lightweight adaptation of frozen text features, and paired full and partially masked neural views. We evaluate the framework on 13 recordings from four Brain Treebank participants, with separate models trained and tested within each participant–movie recording. Under 100-way retrieval, GLANCE achieves 26.2% subject-averaged Recall@10 and exceeds the 10% chance level in every recording. Combining global and local scores generally improves retrieval over global-only scoring across recordings. These findings demonstrate the value of joint sentence-level and word-level matching for retrieving heard sentences from naturalistic intracranial activity.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Brain–language modeling, neural language decoding, intracranial neural recordings, spatiotemporal representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_arh7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GLANCE, a model that picks out which sentence a person heard from sEEG recorded while they watched movies (Brain Treebank). Each electrode lead gets its own BiGRU, and a spatial Transformer then combines the leads at each time bin. From there a global path matches a whole-sentence brain vector to a lightly adapted frozen EmbeddingGemma sentence embedding. A local path matches every time bin to contextual word embeddings using log-mean-exp pooling, with no word timing given. Training uses a multi-positive InfoNCE loss plus a masked second view. The model is trained separately on each of 13 recordings from 4 subjects and reaches 26.2% subject-averaged R@10 in 100-way retrieval (chance is 10%). On one recording it beats adapted SSENSE, DEEPER, BPR, BrainMosaic and ridge baselines, and ablations on that recording support the local path and the masking.

Critical strengths
  1. The task is set up as 100-way retrieval with a known chance level, which avoids the inflated metrics that have caused problems in EEG-to-text generation work.

  2. Using recording leads as the spatial unit, with a GRU per lead and attention across leads, is a sensible way to handle sEEG montages that differ across patients.

  3. The local path scores each candidate word against brain time bins without needing word onset labels, which is a reasonable idea for natural listening.

  4. All baselines use the same splits and the same 30 fixed candidate banks, so the comparison numbers line up directly. The window truncation analysis looks only at cutoffs up to 1 s, where every query keeps the same amount of signal, which is a careful way to rule out a length cue at those cutoffs.

Critical weaknesses
  1. The window runs from sentence onset to 1 s after offset, so the number of valid bins tells the model the sentence duration, and no duration-only or word-count-only baseline shows how much retrieval this cue alone explains.

  2. Trials are split at random within one continuous movie, so test sentences sit between training sentences that are only seconds away, and slow neural drift or scene context could inflate the scores compared with a blocked split by time.

  3. Because windows extend 1 s past offset, windows of back-to-back lines can overlap in raw sEEG across the train and test splits, and the paper does not say whether this is handled.

  4. All baseline comparisons and ablations are run only on Subject 2 Spider-Man, which is also GLANCE's best recording, so it is unclear whether the margins hold elsewhere.

  5. DEEPER and BPR use DistilBERT text features while GLANCE and SSENSE use EmbeddingGemma, so part of the gap may come from the text encoder and not the brain model.

  6. The gain from the local path is small, about 1 pp on average (25.2 vs 26.2 R@10) and 1.2 pp on the main recording, which is within one seed SD, and no paired test across seeds or recordings backs the "11 of 13" claim.

Questions for authors
  1. What R@10 do you get by ranking candidates using only the gap between neural window length and transcript duration or word count?

  2. How do the results change if you split each movie into contiguous time blocks with a gap between train and test?

  3. How often do neural windows from training trials overlap in time with windows from test trials, and how did you deal with this?

  4. Can you run the baselines and the No Local and No Masking ablations on at least one recording from each of the other three subjects?

  5. How does BPR or DEEPER do when you give it the same EmbeddingGemma targets as GLANCE?

  6. How does the model do when the candidates are matched to the true sentence on duration and word count?

  7. What were the "Other" exclusions in Table 4, for example 443 of 998 sentences for Coraline and 653 for Subject 6 Megamind?

7 comments
AuthorsOpenReview ↗

Response to Reviewer arh7

We thank the reviewer for the thoughtful assessment and for recognizing the controlled retrieval setting, lead-based architecture, and motivation for timing-free local matching. Your comments helped us identify several important evaluation improvements. We have expanded the experiments to address temporal separation, overlapping windows, duration-related cues, comparisons across participants, and the contribution of the local pathway.

W2–W3 / Q2–Q3. Temporally blocked evaluation and overlapping neural windows

We agree that random trial splitting within a continuous movie does not adequately separate training and test data in time. We have therefore adopted temporally blocked splits for the revised evaluation.

Each recording is divided into ten contiguous, equal-duration blocks, with seven assigned to training, two to testing, and one to validation. Windows within five seconds of boundaries separating different splits are excluded, leaving at least ten seconds between retained training and test windows. Transcript identities shared with training are excluded from the held-out sets, and normalization and text centering use training data only.

We also audited the original split. For the original Spider-Man test partition, 69.91% windows overlapped at least one training window when the one-second post-offset extension was included. These counts describe the original random split. We are therefore replacing the original random-split evaluation with the revised temporally blocked evaluation throughout the paper. The revised protocol introduces the temporal separation described above, preventing cross-split model-window overlap even with the one-second post-offset extension retained.

The revised results across all 13 recordings are shown below. Values are mean ± SD across runs from three seeds. Recall metrics are percentages; random-ranking references are 1% for R@1 and 10% for R@10.

RecordingR@1R@10R@10-MMRR
S2 Spider-Man2.3 ± 0.519.0 ± 1.416.7 ± 0.40.084 ± 0.004
S2 Venom2.2 ± 0.714.4 ± 1.513.2 ± 1.70.075 ± 0.004
S2 Guardians 13.1 ± 1.018.6 ± 3.117.3 ± 3.90.090 ± 0.010
S2 Guardians 22.4 ± 0.819.2 ± 4.017.6 ± 3.30.086 ± 0.016
S3 Cars 22.3 ± 0.515.9 ± 0.514.6 ± 0.70.079 ± 0.002
S3 LOTR 12.3 ± 1.217.7 ± 2.616.9 ± 2.80.084 ± 0.011
S3 LOTR 22.2 ± 0.416.5 ± 1.016.0 ± 0.80.079 ± 0.007
S4 Shrek0.7 ± 0.012.2 ± 3.111.8 ± 1.70.054 ± 0.005
S4 The Incredibles1.3 ± 0.612.1 ± 1.212.2 ± 1.30.061 ± 0.005
S4 Megamind1.2 ± 0.613.87 ± 0.2512.1 ± 1.20.059 ± 0.002
S6 Toy Story1.5 ± 0.712.7 ± 1.011.9 ± 1.00.061 ± 0.009
S6 Megamind1.5 ± 0.513.8 ± 0.712.2 ± 1.10.061 ± 0.003
S6 Coraline0.9 ± 0.813.4 ± 2.214.1 ± 2.80.060 ± 0.006

Values are mean ± SD across three seeds. Recall metrics are percentages. R@10-M denotes duration-matched R@10. LOTR = Lord of the Rings.

Performance is lower under this more demanding evaluation than under the original random split. Nevertheless, mean R@10 remains above the 10% random-ranking reference in all 13 recordings, including with duration-matched candidates.

Also, a complementary time-only ridge control, using sentence position within the movie without neural features, achieves R@10 of 10.4%, 9.6%, 8.4%, and 10.7% across Subject 2’s four recordings. These results suggest that movie position alone does not reproduce GLANCE’s retrieval performance under temporal blocking.

W1 / Q1 and Q6. Duration-related cues and matched candidates

We agree that variable input duration can expose information about sentence length. Following the reviewer’s suggestion, we evaluated duration-only and word-count-only rankers. In the separate audit covering all 13 recordings, these achieved 98.9% and 27.1% standard R@10, respectively. The word-count ranker predicts duration from candidate word count using a mapping fitted on training examples only. These results confirm that length information is available in the variable-window evaluation.

Duration-matched candidate evaluation. These findings motivated us to construct duration-matched candidate sets within the held-out test partitions. For each query, the candidate set contains the correct transcript and the 99 held-out distractor occurrences nearest to it in spoken duration, excluding occurrences with the correct transcript identity. This approximately matches candidate durations, although it does not completely eliminate duration-related cues.

We report results under this protocol alongside standard retrieval throughout our revised evaluation, including the main GLANCE results, baseline comparisons, ablations, and cross-movie experiments. The Duration-matched R@10 column in the table above reports performance under this evaluation.

AuthorsOpenReview ↗

Response to Reviewer arh7 — Part 2

Continued W1 / Q1 and Q6.

Fixed-duration neural-input control. To complement this evaluation with a control that directly removes variation in neural-input length, we additionally retrained and evaluated GLANCE using only the first 1.37 seconds after sentence onset. Windows shorter than this duration were excluded, so every retained trial provided the same number of valid neural-input bins.

RecordingFixed-duration GLANCE R@10
S2 Spider-Man17.6 ± 2.1
S3 Cars 215.0 ± 1.2

R@10 (%), mean ± SD across three blocked runs, using validation-selected checkpoints; chance R@10 = 10%.

These findings suggest that GLANCE is capable of retaining retrieval information when variation in neural-input length is removed.

For the main analyses, we emphasize duration-matched retrieval because it preserves the available sentence-related neural activity, whereas a fixed 1.37-second cutoff discards later activity from longer sentences. We interpret these results together with the fixed-duration control, which more directly addresses whether retrieval can persist without differences in input-window length.

W4 / Q4. Baseline comparisons across all four participants

We have expanded baseline evaluation to include Spider-Man, Cars 2, Shrek, and Toy Story, covering all four participants. Within each recording, all compared models use the same seeds, blocked layouts, test examples, and candidate sets.

RecordingModelR@10Duration-matched R@10
S2 Spider-ManGLANCE19.0 ± 1.416.7 ± 0.4
S2 Spider-ManSSENSE18.4 ± 2.213.1 ± 1.6
S2 Spider-ManBPR17.3 ± 1.115.9 ± 1.3
S2 Spider-ManDEEPER14.6 ± 1.814.9 ± 0.8
S2 Spider-ManBrainMosaic15.4 ± 1.911.8 ± 2.2
S3 Cars 2GLANCE15.9 ± 0.514.6 ± 0.7
S3 Cars 2SSENSE19.4 ± 2.013.7 ± 0.6
S3 Cars 2BPR15.1 ± 1.414.2 ± 1.6
S3 Cars 2DEEPER15.0 ± 0.214.4 ± 0.9
S3 Cars 2BrainMosaic15.2 ± 3.110.0 ± 1.4
S4 ShrekGLANCE12.2 ± 3.111.8 ± 1.7
S4 ShrekSSENSE16.0 ± 5.212.0 ± 3.5
S4 ShrekBPR10.1 ± 1.08.9 ± 0.9
S4 ShrekDEEPER10.4 ± 0.310.3 ± 0.8
S4 ShrekBrainMosaic11.2 ± 1.910.3 ± 0.8
S6 Toy StoryGLANCE12.7 ± 1.011.9 ± 1.0
S6 Toy StorySSENSE14.3 ± 1.311.3 ± 1.3
S6 Toy StoryBPR10.8 ± 1.88.7 ± 2.2
S6 Toy StoryDEEPER7.9 ± 1.58.1 ± 0.9
S6 Toy StoryBrainMosaic12.5 ± 2.010.9 ± 3.3

R@10 (%), mean ± SD across three seed runs per recording.

GLANCE achieves the highest mean duration-matched R@10 in three of the four recordings (Spider-Man, Cars 2, and Toy Story). On the standard candidate evaluation, SSENSE achieves higher R@10 on three recordings. This pattern might suggest that SSENSE's advantage under standard evaluation is more sensitive to candidate-duration composition, while GLANCE's retrieval performance is comparatively stable under duration-matched evaluation.

We also evaluated GLANCE, SSENSE, and BPR using fixed 1.37-second neural windows beginning at sentence onset, so that every trial provided the same duration of neural input. This comparison examines retrieval performance after removing variation in input-window duration.

RecordingModelFixed-input R@10 (%)MRR
S2 Spider-ManGLANCE17.6 ± 2.10.082 ± 0.004
S2 Spider-ManSSENSE12.7 ± 1.10.061 ± 0.006
S2 Spider-ManBPR15.5 ± 1.30.075 ± 0.002
S3 Cars 2GLANCE15.0 ± 1.20.072 ± 0.008
S3 Cars 2SSENSE12.8 ± 2.70.062 ± 0.009
S3 Cars 2BPR14.3 ± 3.70.072 ± 0.016

Values are mean ± SD across three split layouts under temporally blocked evaluation. This fixed-input comparison uses validation-selected checkpoints.

GLANCE achieves the highest mean R@10 among the three models on both recordings and retains mean performance above the 10% random-ranking reference. These results provide additional evidence that variation in neural-input duration is not necessary for GLANCE’s retrieval performance.

AuthorsOpenReview ↗

Response to Reviewer arh7 — Part 3

W5 / Q5. Baselines with the same EmbeddingGemma targets

To address the text-encoder concern directly, we retrained BPR and SSENSE using EmbeddingGemma representations. The following results use the same blocked split datasets.

RecordingBPR, DistilBERTBPR, EmbeddingGemmaSSENSE, CLIPSSENSE, EmbeddingGemma
S2 Spider-Man17.3 ± 1.114.3 ± 1.618.4 ± 2.215.5 ± 1.1
S3 Cars 215.1 ± 1.412.5 ± 1.619.4 ± 2.014.7 ± 2.3

Standard R@10 (%), mean ± SD.

Replacing the baselines’ text representations with EmbeddingGemma did not improve their retrieval performance in these experiments. This suggests that the comparative results cannot be explained simply by assuming that EmbeddingGemma provides a uniform advantage. It does not fully separate the effects of architecture, optimization, and text representation.

We will explicitly identify the text encoder used in each condition, including the distinction between the original SSENSE adaptation using Gemma targets and the SSENSE-CLIP implementation evaluated here.

W6 / Q4. Expanded ablations and evidence for the local pathway

We agree that a small average improvement, or a count of recordings favoring one condition, is insufficient on its own to establish a reliable contribution.

We therefore expanded the retrained No Local and No Masking comparisons to four recordings, one from each participant, using the same layout. Each ablation is compared with its corresponding full-model control.

RecordingModelR@10Duration-matched R@10MRRDuration-matched MRR
S2 Spider-ManFull GLANCE19.0 ± 1.416.7 ± 0.40.084 ± 0.0040.078 ± 0.004
S2 Spider-ManNo Local17.4 ± 1.315.6 ± 0.60.082 ± 0.0130.073 ± 0.011
S2 Spider-ManNo Masking17.7 ± 1.816.5 ± 1.40.082 ± 0.0030.076 ± 0.004
S3 Cars 2Full GLANCE15.9 ± 0.514.6 ± 0.70.079 ± 0.0020.074 ± 0.001
S3 Cars 2No Local15.8 ± 0.814.5 ± 0.90.075 ± 0.0030.068 ± 0.004
S3 Cars 2No Masking16.5 ± 2.413.7 ± 1.60.080 ± 0.0100.072 ± 0.009
S4 ShrekFull GLANCE12.2 ± 3.111.8 ± 1.70.054 ± 0.0050.052 ± 0.006
S4 ShrekNo Local10.7 ± 2.010.5 ± 1.80.048 ± 0.0060.049 ± 0.008
S4 ShrekNo Masking11.0 ± 0.810.0 ± 1.30.051 ± 0.0020.049 ± 0.004
S6 Toy StoryFull GLANCE12.7 ± 1.011.9 ± 1.00.061 ± 0.0090.060 ± 0.009
S6 Toy StoryNo Local11.8 ± 1.212.39 ± 1.20.057 ± 0.0080.054 ± 0.007
S6 Toy StoryNo Masking11.1 ± 2.39.7 ± 1.50.054 ± 0.0120.052 ± 0.011

Mean ± SD across runs from three seeds.

Full GLANCE achieves higher mean standard R@10 than No Local in all four recordings, with differences of 0.1–1.6 percentage points and an average difference of approximately 1.0 percentage point. Mean R@1, standard MRR, and duration-matched MRR also favor full GLANCE in all four recordings.

To assess these differences, we performed two-sided paired t-tests comparing full GLANCE with the retrained No Local ablation. Full GLANCE achieved higher R@1, MRR, and duration-matched MRR at the unadjusted 0.05 significance level. The differences in standard and duration-matched R@10 did not reach this threshold.

The masking ablation shows higher mean duration-matched R@10 for the full model in all four recordings, while the standard R@10 comparison reverses on Cars 2. Its differences therefore vary by recording and metric, and we will calibrate its interpretation accordingly.

These results suggest that contextual word-level matching can provide complementary information for sentence retrieval, with a contribution that varies across recordings and metrics.

Q7. Meaning of the “Other” exclusions

The “Other” exclusions primarily reflected missing neural recording coverage or unavailable synchronization triggers and timing information. For Coraline, 443 short sentences could not be assigned a validated movie-time-to-neural-sample alignment because the available trigger and timing segments did not cover those transcript times. For Subject 6 / Megamind, 634 of the 653 excluded sentences occurred after the end of the available subject-specific neural recording, so no neural samples were available even at sentence onset. We will clarify this breakdown in Table 4.

Reviewer_uADwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a framework for sentence retrieval during naturalistic movie viewing from sEEG data using both local (contextual word-level) and global (sentence) embeddings. The authors build a lead-structured neural encoder capturing channel-frequency features using a bidirectional GRU, learned embeddings, and a spatial Transformer. This shared neural representation feeds into a global and a local pathway, to produce a normalized brain vector and time-resolved brain vectors, respectively. For each candidate transcript, a sentence embedding and contextual word embeddings are extracted using EmbeddingGemma. They compute a global and a local score using the brain vectors and corresponding embeddings. They test the framework by ranking each query against 100-candidate bank, use 30 banks per query and report Recall @ 1, Recall @ 10 and mean reciprocal rank. They evaluate 13 movies across 4 subjects from the Brain Treebank dataset. Their framework predicts above chance across subjects and movies. Comparisons with prior models and a linear regression baseline for 1 movie in 1 subject show that their framework outperforms those. Comparisons across participants and movies of the 2 pathways scored separately, as well as an ablation study on the 1 movie in 1 subject show that, while the local word-level information is less important to the framework’s performance, it still contributes. A retrieval error analysis shows that wrong predictions are still above chance in terms of semantic similarity, and that this is not only due to temporal context.

Critical strengths
  1. The framework is well designed, both on the neural encoding side and the text embeddings side, to combine word-level and sentence-level information. The comparison of different scoring systems and ablation studies adequately assess the contribution of different components.
  2. The framework scores above chance on all movies for all participants, and outperforms previous models and a baseline on the 1 movie it is tested against. This suggests that the model performs well on diverse stimuli and brains, and may suggest that it’s competitive.
Critical weaknesses
  1. It is unclear why the comparison with other models and baseline are only performed on 1 movie in 1 participant. This limits the generalibility of the findings.
  2. It is also unclear why a movie watching dataset is used to test sentence retrieval during naturalistic listening. The added visual context might add processes that are not accounted for in speech processing.
Questions for authors

Are the hyperparameters eta and lambda arbitrarily chosen? Did you try cross-participant decoding?

8 comments
AuthorsOpenReview ↗

Response to Reviewer uADw

We thank the reviewer for the constructive feedback and positive assessment of GLANCE’s architecture. Your comments motivated expanded baseline comparisons, additional transfer experiments, and empirical evaluation of the hyperparameter choices. We address each point below. We have also revised the evaluation protocol to address temporal dependence within continuous movie recordings. The new temporally blocked evaluation replaces the original random-split results throughout the paper. We provide the revised GLANCE results as well.

In new blocked protocol, we divided each movie into ten equal-duration temporal blocks, assigning seven to training, one to validation, and two to testing. We exclude trials within five seconds on either side of boundaries separating different partitions, providing at least ten seconds of temporal separation between retained examples from adjacent partitions. The revised protocol removes cross-split model-window overlap. We evaluate two candidate-selection protocols:

  • Standard 100-way retrieval: The correct transcript is ranked among 99 randomly sampled distractors from the held-out test set.
  • Duration-matched 100-way retrieval: The correct transcript is ranked among the 99 held-out distractor occurrences nearest to it in spoken duration, excluding the correct transcript identity.

Duration matching reduces differences in candidate duration but does not eliminate duration-related cues. We therefore also report a fixed-duration neural-input control below.

Revised GLANCE results across all 13 recordings

RecordingR@1R@10R@10-MMRRMRR-M
S2 Spider-Man2.3 ± 0.519.0 ± 1.416.7 ± 0.40.084 ± 0.0040.078 ± 0.004
S2 Venom2.2 ± 0.714.4 ± 1.513.2 ± 1.70.075 ± 0.0040.073 ± 0.005
S2 Guardians 13.1 ± 1.018.6 ± 3.117.3 ± 3.90.090 ± 0.0100.087 ± 0.011
S2 Guardians 22.4 ± 0.819.2 ± 4.017.6 ± 3.30.086 ± 0.0160.078 ± 0.008
S3 Cars 22.3 ± 0.515.9 ± 0.514.6 ± 0.70.079 ± 0.0020.074 ± 0.001
S3 Lord of the Rings 12.3 ± 1.217.7 ± 2.616.9 ± 2.80.084 ± 0.0110.082 ± 0.016
S3 Lord of the Rings 22.2 ± 0.416.5 ± 1.016.0 ± 0.80.079 ± 0.0070.075 ± 0.010
S4 Shrek0.7 ± 0.012.2 ± 3.111.8 ± 1.70.054 ± 0.0050.052 ± 0.006
S4 The Incredibles1.3 ± 0.612.1 ± 1.212.2 ± 1.30.061 ± 0.0050.058 ± 0.003
S4 Megamind1.2 ± 0.613.87 ± 0.2512.1 ± 1.20.059 ± 0.0020.058 ± 0.005
S6 Toy Story1.5 ± 0.712.7 ± 1.011.9 ± 1.00.061 ± 0.0090.060 ± 0.009
S6 Megamind1.5 ± 0.513.8 ± 0.712.2 ± 1.10.061 ± 0.0030.060 ± 0.004
S6 Coraline0.9 ± 0.813.4 ± 2.214.1 ± 2.80.060 ± 0.0060.059 ± 0.005

Mean ± SD across three seed runs. Recall values are percentages. R@10 uses the combined global–local score; M denotes duration-matched evaluation. Random-ranking R@10 is 10%.

Under this revised protocol, GLANCE achieves mean R@10 above the 10% random-ranking reference in all 13 recordings, although performance varies across participants and individual runs. The subject-averaged standard and duration-matched R@10 values are 15.13% and 14.20%, respectively, averaging first across each participant's recordings and then equally across participants. Mean R@10 exceeds the 10% random-ranking reference in all 13 recordings under both standard and duration-matched evaluation.

AuthorsOpenReview ↗

Response to Reviewer uADw -part2

W1. Comparisons beyond one movie and participant

We agree that comparisons on a single recording are insufficient to establish whether the relative performance extends across participants. The original comparison focused on Spider-Man because it contained the largest number of retained sentence trials. We have now expanded the baseline evaluation to four recordings, one from each participant: S2 Spider-Man, S3 Cars 2, S4 Shrek, and S6 Toy Story. Expanded baseline comparison

RecordingModelR@10R@10-MMRR
S2 Spider-ManGLANCE19.0 ± 1.416.7 ± 0.40.084 ± 0.004
S2 Spider-ManSSENSE18.4 ± 2.213.1 ± 1.60.086 ± 0.004
S2 Spider-ManBPR17.3 ± 1.115.9 ± 1.30.076 ± 0.002
S2 Spider-ManDEEPER14.6 ± 1.814.9 ± 0.80.073 ± 0.004
S2 Spider-ManBrainMosaic15.4 ± 1.911.8 ± 2.20.069 ± 0.010
S3 Cars 2GLANCE15.9 ± 0.514.6 ± 0.70.079 ± 0.002
S3 Cars 2SSENSE19.4 ± 2.013.7 ± 0.60.090 ± 0.005
S3 Cars 2BPR15.1 ± 1.414.2 ± 1.60.077 ± 0.003
S3 Cars 2DEEPER15.0 ± 0.214.4 ± 0.90.071 ± 0.006
S3 Cars 2BrainMosaic15.2 ± 3.110.0 ± 1.40.074 ± 0.004
S4 ShrekGLANCE12.2 ± 3.111.8 ± 1.70.054 ± 0.005
S4 ShrekSSENSE16.0 ± 5.212.0 ± 3.50.081 ± 0.016
S4 ShrekBPR10.1 ± 1.08.9 ± 0.90.055 ± 0.004
S4 ShrekDEEPER10.4 ± 0.310.3 ± 0.80.059 ± 0.010
S4 ShrekBrainMosaic11.2 ± 1.910.3 ± 0.80.063 ± 0.006
S6 Toy StoryGLANCE12.7 ± 1.011.9 ± 1.00.061 ± 0.009
S6 Toy StorySSENSE14.3 ± 1.311.3 ± 1.30.070 ± 0.009
S6 Toy StoryBPR10.8 ± 1.88.7 ± 2.20.051 ± 0.011
S6 Toy StoryDEEPER7.9 ± 1.58.1 ± 0.90.049 ± 0.003
S6 Toy StoryBrainMosaic12.5 ± 2.010.9 ± 3.30.061 ± 0.011

*Mean ± SD across three seed runs. R@10 is reported in percent; R@10-M denotes duration-matched R@10.

GLANCE achieves the highest mean duration-matched R@10 in three of the four recordings (Spider-Man, Cars 2, and Toy Story). On the standard candidate evaluation, SSENSE achieves higher R@10 on three recordings. This pattern might suggest that SSENSE's advantage under standard evaluation is more sensitive to candidate-duration composition, while GLANCE's retrieval performance is comparatively stable under duration-matched evaluation.

Comparisons with fixed-duration neural inputs

We additionally retrained and evaluated GLANCE, SSENSE, and BPR using fixed 1.37-second neural windows beginning at sentence onset. Windows shorter than this duration were excluded, so every retained trial provided the same duration of neural input. This removes variation in input-window length as a possible retrieval cue.

RecordingModelFixed-input R@10 (%)MRR
S2 Spider-ManGLANCE17.6 ± 2.10.082 ± 0.004
S2 Spider-ManSSENSE12.7 ± 1.10.061 ± 0.006
S2 Spider-ManBPR15.5 ± 1.30.075 ± 0.002
S3 Cars 2GLANCE15.0 ± 1.20.072 ± 0.008
S3 Cars 2SSENSE12.8 ± 2.70.062 ± 0.009
S3 Cars 2BPR14.3 ± 3.70.072 ± 0.016

Standard 100-way retrieval, mean ± SD across three blocked runs.

GLANCE achieves the highest mean R@10 among these three models on both recordings and retains mean performance above the 10% random-ranking reference. These results suggest that its retrieval performance is not explained solely by variation in neural-input duration. They remain descriptive comparisons and do not establish a statistically significant advantage over the baselines.

AuthorsOpenReview ↗

Response to Reviewer uADw -part3

W1 continued. Ablations across all four participants

We also expanded the retrained No Local and No Masking comparisons to the same four recordings. Each ablation is compared with its corresponding full-model control using the same three seeds and blocked evaluation.

Recording / modelR@10R@10-MMRRMRR-M
S2 Spider-Man — GLANCE (full)19.0 ± 1.416.7 ± 0.40.084 ± 0.0040.078 ± 0.004
S2 Spider-Man — No Local17.4 ± 1.315.6 ± 0.60.082 ± 0.0130.073 ± 0.011
S2 Spider-Man — No Masking17.7 ± 1.816.5 ± 1.40.082 ± 0.0030.076 ± 0.004
S3 Cars 2 — GLANCE (full)15.9 ± 0.514.6 ± 0.70.079 ± 0.0020.074 ± 0.001
S3 Cars 2 — No Local15.8 ± 0.814.5 ± 0.90.075 ± 0.0030.068 ± 0.004
S3 Cars 2 — No Masking16.5 ± 2.413.7 ± 1.60.080 ± 0.0100.072 ± 0.009
S4 Shrek — GLANCE (full)12.2 ± 3.111.8 ± 1.70.054 ± 0.0050.052 ± 0.006
S4 Shrek — No Local10.7 ± 2.010.5 ± 1.80.048 ± 0.0060.049 ± 0.008
S4 Shrek — No Masking11.0 ± 0.810.0 ± 1.30.051 ± 0.0020.049 ± 0.004
S6 Toy Story — GLANCE (full)12.7 ± 1.011.9 ± 1.00.061 ± 0.0090.060 ± 0.009
S6 Toy Story — No Local11.8 ± 1.212.4 ± 1.20.057 ± 0.0080.054 ± 0.007
S6 Toy Story — No Masking11.1 ± 2.39.7 ± 1.50.054 ± 0.0120.052 ± 0.011

Mean ± SD across three seed runs. Recall values are percentages; M denotes duration-matched evaluation.

Full GLANCE achieves higher mean standard R@10 than No Local in all four recordings, with differences of 0.1–1.6 percentage points and an average difference of approximately 1.0 percentage point. Mean R@1, standard MRR, and duration-matched MRR also favor the full model in these four recordings. On Toy Story, however, No Local achieves higher duration-matched R@10 (12.4% versus 11.9%).

To assess these differences, we performed two-sided paired t-tests across the four paired recording means (n = 4). Full GLANCE achieved significantly higher R@1 (p = 0.015), MRR (p = 0.009), and duration-matched MRR (p = 0.002) than the No Local ablation at the unadjusted 0.05 significance level.

The masking ablation shows higher mean duration-matched R@10 for the full model in all four recordings, while the standard R@10 comparison reverses on Cars 2. Its contribution therefore also varies by recording and metric.

These results suggest that contextual word-level matching can provide complementary information for sentence retrieval, while supporting a qualified interpretation of its contribution.

W2. Movie viewing versus naturalistic listening

We appreciate the reviewer raising this distinction. Human intracranial datasets combining extended naturalistic dialogue with carefully aligned transcripts are scarce. Brain Treebank provides a valuable opportunity to study sentence retrieval from high-resolution neural recordings across multiple movies and participants. Its original analyses also report word-related neural responses after accounting for measured acoustic and visual features, supporting its relevance to neural–language modeling (Wang et al., 2024).

Our additional controls address several related alternative explanations. A ridge control using only a sentence’s position within the movie achieves near-chance R@10 across Subject 2’s four movies (8.4–10.7%, compared with a 10% random-ranking reference). The fixed-input experiments above remove differences in neural-window duration, while the cross-movie experiments below assess retrieval on movies not used for paired neural–transcript training.

We agree that accompanying visual and contextual information may also contribute to retrieval. Although these controls strengthen the evaluation, they do not isolate linguistic information from all acoustic and visual correlates. We will therefore consistently describe the task as sentence retrieval during naturalistic audiovisual movie viewing and clarify this limitation in the abstract, experimental setting, and discussion.

Reviewer_bZ3rOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles retrieval of perceived sentences from intracranial recordings (sEEG) while participants watch feature films (Brain Treebank dataset). The task is 100-way retrieval. Authors take the neural window for a heard sentence (onset to offset + 1s), pick the true transcript from 99 distractors drawn from the same movie. The proposed model, GLANCE, converts each contact’s signal into time-frequency features and reads each electrode lead with its own BiGRU (interesting idea). Eventually, the model has a split head to predict at the sentence and word level. Trained with InfoNCE with a multi-positive loss, still contrastive.

GLANCE reaches 26.2% subject-averaged rec@10 (chance 10%) and is above chance in every recording, which clears the main initial bar for me. It’s worth keeping in mind, though, that top-1 is 6% (chance 1%), so “identifying heard language” is a stronger phrase than the numbers really support. It beats several re-implemented baselines (SSENSE, BPR, DEEPER, BrainMosaic, ridge), and ablations (good to see!) suggest both the local path and masking help. However, both the baselines and the ablations are only run on a single recording, which happens to be GLANCE’s best one (34.4%), with three seeds and overlapping error bars, so I’m not sure how far these comparisons can be pushed. Combining the local and global scores improves on global alone by only about 1 percentage point on average, and several per-recording gains look well within seed noise. I'd say that's within some sampling error, so not sure how much to take away from that.

My main concern is what the model is actually reading out. Trials are split at random within a single continuous movie, so most test sentences sit seconds away from training sentences in the same scene. Window length also scales with sentence duration, and there is no duration-only or acoustic baseline, so I can’t rule out that some of the performance comes from scene, timing or acoustic cues rather than language. I was also a little disappointed that the “word-level evidence” claim is never checked against the word onsets the dataset provides (the authors say they deliberately discard them), when that seems like the obvious validation. The authors are honest that cross-subject and cross-movie generalisation isn’t tested, which I appreciated, but a within-subject cross-movie test was available here and would have gone a long way to reassure me.

Overall, a solid contribution on a difficult task, but not sure it passes the high-threshold for acceptance (esp based on the other papers I've reviewed).

Critical strengths
  • Figures are quite nice, some complex and small but it is really care that this isn't the first draft and some serious thought has gone into the reader's perspective to get these right, and that visual aspect of understanding the paper goes a long way and is much appreciated

  • Anonymised code given, which is good for reproducibility and good practice

  • Good ablations in the sense of the analyses applied, to verify the results they did show were defensible against the main questions that NeuroAI-savvy readers might have (just a shame it's not done on more subjects / films).

Critical weaknesses
  • Nearly every test sentence is going to have training sentences before and after it, from the same scene, speakers and topic. That, combined with a top-10 analysis, really concerns me because we know there are all sorts of positional and adjacency information in neural signals that have to be considered. A viewer of the film could have had a thought or feeling lasting multiple seconds and then test sentences can be matched to associated time frames because those times are in the training data and just happen to be captured (a block split would also work, over scenes, perhaps). This seems like a rookie mistake to make given what we know about train-test splitting in NeuroAI work.
  • Cross-movie testing not done (participants watched multiple films and the training - test division could be done between them but was not done and I wonder if it had been done but failed to show an effect, which would undermine usefulness of current approach as this seems like the logical next step)
  • Selecting the best performing GLANCE model to serve as a baseline also seems like an unfair advantage to the baselines.
  • The paper is a bit dramatic in terms of its conclusions (”identifying heard language” and “retrieving heard sentences”) when we see 6% accuracy in a 1-out-100 possibility range at top1. I’d certainly expect to see better scores to warrant such claims.
  • No justification for choice of EmbeddingGemma model or other models, just given as stated and I can't tell if multiple were tried and this came out best or if there was an initial thought that I could reasonably accept, so this raises questions in my mind.
Questions for authors

1; The local pathway is described as supplying word-level evidence, but this is not validated against the word onsets that Brain Treebank provides. Prior intracranial work relating contextual embeddings to neural activity at word onsets during naturalistic listening (e.g. from Hasson, Nastase & friends) offers a natural point of comparison. Does each word’s soft temporal assignment peak near its true onset, relative to a shuffled-onset null?

  1. The local pathway is presented as providing word-level evidence through word-to-time matching, yet Appendix A.4 states that the pipeline “does not retain or use released word onset or offset timestamps”, even though Brain Treebank provides them and they would allow this claim to be tested directly. If you had decided to do this, then alignment between each word’s soft assignment and its true onset would have shown that the local pathway genuinely locates individual words in the neural signal, which would support the paper’s central word-level claim. Without it, I'm just not sure we can attribute word-level information to the local pathway at all. Why the choice to not follow up on this relatively-easy step to bolster your analyses?
8 comments
AuthorsOpenReview ↗

Response to Reviewer bZ3r

We sincerely thank the reviewer for the careful reading and constructive suggestions. We appreciate your positive assessment of our methodology. Your comments have been very valuable in guiding additional experiments addressing temporal dependence, cross-movie generalization, baseline comparisons, and the interpretation of the local pathway.

We have expanded our evaluation and will incorporate these analyses into the revised manuscript. Importantly, the new temporally blocked evaluation replaces the original randomly split results throughout the paper.

W1. Temporal dependence and evaluation methodology

We agree with the reviewer that random trial-level splitting within continuous movie recordings can introduce temporal dependencies between training and test examples, potentially affecting the interpretation of retrieval performance.

To address this concern, we revised the evaluation protocol and retrained GLANCE using temporally blocked splits for all recordings. Each movie is divided into ten equal-duration temporal blocks, with seven assigned to training, one to validation, and two to testing. We exclude trials within five seconds on either side of boundaries separating different partitions, providing at least ten seconds of temporal separation between retained examples from adjacent partitions. We also ensure that transcript identities shared with training are excluded from held-out evaluation, and that normalization and text centering are determined using training data only.

We additionally evaluate retrieval under two candidate-selection protocols:

  • Standard 100-way retrieval: The correct transcript is ranked among 99 randomly sampled distractors from the held-out test set.
  • Duration-matched 100-way retrieval: The 99 distractor occurrences nearest in spoken duration to the correct transcript are selected from the held-out test set, excluding the correct transcript identity.

The second protocol reduces differences in candidate duration, although not completely eliminating duration-related cues.

Revised GLANCE results across all 13 recordings.

All results below use the revised temporally blocked evaluation. Values are mean ± SD across three seed runs. Recall values are percentages; R@10 denotes the combined global–local score, and chance R@10 is 10%.

Recording  R@1  R@10  R@10-M  MRR  MRR-M
S2 Spider-Man  2.3 ± 0.5  19.0 ± 1.4  16.7 ± 0.4  0.084 ± 0.004  0.078 ± 0.004
S2 Venom  2.2 ± 0.7  14.4 ± 1.5  13.2 ± 1.7  0.075 ± 0.004  0.073 ± 0.005
S2 Guardians 1  3.1 ± 1.0  18.6 ± 3.1  17.3 ± 3.9  0.090 ± 0.010  0.087 ± 0.011
S2 Guardians 2  2.4 ± 0.8  19.2 ± 4.0  17.6 ± 3.3  0.086 ± 0.016  0.078 ± 0.008
S3 Cars 2  2.3 ± 0.5  15.9 ± 0.5  14.6 ± 0.7  0.079 ± 0.002  0.074 ± 0.001
S3 Lord of the Rings 1  2.3 ± 1.2  17.7 ± 2.6  16.9 ± 2.8  0.084 ± 0.011  0.082 ± 0.016
S3 Lord of the Rings 2  2.2 ± 0.4  16.5 ± 1.0  16.0 ± 0.8  0.079 ± 0.007  0.075 ± 0.010
S4 Shrek  0.7 ± 0.0  12.2 ± 3.1  11.8 ± 1.7  0.054 ± 0.005  0.052 ± 0.006
S4 The Incredibles  1.3 ± 0.6  12.1 ± 1.2  12.2 ± 1.3  0.061 ± 0.005  0.058 ± 0.003
S4 Megamind  1.2 ± 0.6  13.9 ± 0.2  12.1 ± 1.2  0.059 ± 0.002  0.058 ± 0.005
S6 Toy Story  1.5 ± 0.7  12.7 ± 1.0  11.9 ± 1.0  0.061 ± 0.009  0.060 ± 0.009
S6 Megamind  1.5 ± 0.5  13.8 ± 0.7  12.2 ± 1.1  0.061 ± 0.003  0.060 ± 0.004
S6 Coraline  0.9 ± 0.8  13.4 ± 2.2  14.1 ± 2.8  0.060 ± 0.006  0.059 ± 0.005

M denotes duration-matched evaluation.

Under this revised protocol, GLANCE achieves mean R@10 above the 10% random-ranking reference in all 13 recordings, although performance varies across subjects and individual runs. The subject-averaged standard and duration-matched R@10 values are 15.13% and 14.20%, respectively.

AuthorsOpenReview ↗

Response to Reviewer bZ3r — Part 2

W1 continued

Additional controls for temporal and duration-related confounds

We also examined whether retrieval could be explained by a sentence's position within the movie or by variation in the duration of the neural input. A ridge control using only movie position achieved near-chance R@10 across Subject 2's four movies (8.4–10.7%, compared with a 10% random-ranking reference), suggesting that movie position alone is insufficient to account for GLANCE's performance.

To test the role of input duration directly, we retrained GLANCE using fixed 1.37-second neural windows beginning at sentence onset, so that every trial provided the same duration of neural input. With this variation removed, GLANCE retained mean R@10 above the 10% random-ranking reference ( values are mean ± SD across three runs) :

RecordingFixed-input R@10
S2 Spider-Man17.6 ± 2.1%
S3 Cars 215.0 ± 1.2%

Together with the temporally blocked evaluation, these controls suggest that GLANCE's retrieval performance is not explained solely by movie position or variation in input-window duration. They do not isolate linguistic information from all acoustic and visual correlates present during movie viewing, and we will clarify this limitation in the revised manuscript.

W2. Within-participant cross-movie generalization

We appreciate the reviewer emphasizing cross-movie evaluation as an important next step. We have now conducted within-participant cross-movie experiments for all four participants—Subjects 2, 3, 4, and 6—covering all 13 recordings, directly investigating whether GLANCE can retrieve sentences from a movie without being trained on paired neural–transcript examples from that movie.

Zero-shot retrieval on held-out movies

For each target movie, we trained GLANCE using the participant's other movies and evaluated it on the held-out target movie. Exact transcript identities appearing in the target test set were excluded from source-movie training.

We compared this setting against GLANCE trained using only the target movie's training partition. Both conditions are evaluated on the same held-out target-movie test partitions. Within-participant zero-shot cross-movie retrieval

Target movieTarget-movie R@10Zero-shot R@10Target-movie R@10-MZero-shot R@10-M
S2 Spider-Man19.0 ± 1.417.64 ± 1.0116.7 ± 0.415.68 ± 0.26
S2 Venom14.4 ± 1.516.93 ± 3.3613.2 ± 1.716.44 ± 3.67
S2 Guardians 118.6 ± 3.120.94 ± 9.3117.3 ± 3.920.03 ± 9.54
S2 Guardians 219.2 ± 4.022.12 ± 7.1117.6 ± 3.320.58 ± 5.89
S3 Cars 215.9 ± 0.515.46 ± 1.4214.6 ± 0.714.64 ± 1.12
S3 Lord of the Rings 117.7 ± 2.617.86 ± 2.5816.9 ± 2.817.62 ± 3.87
S3 Lord of the Rings 216.5 ± 1.016.84 ± 0.6916.0 ± 0.816.02 ± 1.37
S4 Shrek12.2 ± 3.113.00 ± 1.4011.8 ± 1.711.85 ± 1.59
S4 The Incredibles12.1 ± 1.29.92 ± 2.0412.2 ± 1.39.92 ± 1.50
S4 Megamind13.87 ± 0.259.27 ± 0.6612.1 ± 1.28.47 ± 1.26
S6 Toy Story12.7 ± 1.011.63 ± 0.5411.9 ± 1.011.41 ± 2.21
S6 Megamind13.8 ± 0.710.70 ± 1.2712.2 ± 1.111.12 ± 2.17
S6 Coraline13.4 ± 2.29.30 ± 1.8314.1 ± 2.89.20 ± 2.40

R@10 (%), mean ± SD across runs from three seeds. Zero-shot training uses other movies from the same participant, with no paired target-movie training examples. Unlabeled target-recording neural data are used for recording-specific normalization. The target-movie-only columns repeat the main GLANCE results above.

For Subjects 2 and 3, zero-shot retrieval achieves standard R@10 ranging from 15.46% to 22.12%, with subject-level means of 19.4% for Subject 2 and 16.7% for Subject 3. Zero-shot retrieval produces higher mean standard R@10 than target-movie-only training in five of these seven recordings, although the conditions differ in the amount of available training data. These results suggest transfer across movies within these participants.

Zero-shot standard R@10 reaches 13.00% on S4 Shrek and 11.63% on S6 Toy Story, although results for Subjects 4 and 6 are mixed, indicating that this transfer is not consistent across the full cohort.

AuthorsOpenReview ↗

Response to Reviewer bZ3r — Part 3

W2 Continued

Additional cross-movie training strategies

We also investigated whether incorporating recordings from other movies could improve target-movie retrieval when target-movie training data are available.

We considered two additional strategies:

  • Pooled training: Combining the target movie's training partition with recordings from the participant's other movies.
  • Pretraining followed by fine-tuning: Pretraining on the participant's other movies and subsequently fine-tuning on the target movie's training partition.

Additional within-participant cross-movie training experiments: standard R@10

Target movie  Target only  Pooled  Pretrain → FT
S2 Spider-Man  19.0 ± 1.4  19.09 ± 1.74  17.20 ± 2.03
S2 Venom  14.4 ± 1.5  17.97 ± 3.31  18.17 ± 1.54
S2 Guardians 1  18.6 ± 3.1  23.83 ± 7.90  20.32 ± 6.82
S2 Guardians 2  19.2 ± 4.0  22.75 ± 3.08  24.72 ± 4.22
S3 Cars 2  15.9 ± 0.5  17.21 ± 1.87  17.28 ± 0.96
S3 LOTR 1  17.7 ± 2.6  19.95 ± 1.48  20.60 ± 1.94
S3 LOTR 2  16.5 ± 1.0  17.32 ± 1.51  19.60 ± 1.72
S4 Shrek  12.2 ± 3.1  12.25 ± 2.32  15.28 ± 5.40
S4 The Incredibles  12.1 ± 1.2  10.96 ± 0.61  11.46 ± 3.12
S4 Megamind  13.87 ± 0.25  12.08 ± 1.66  10.77 ± 0.75
S6 Toy Story  12.7 ± 1.0  10.93 ± 1.18  10.91 ± 0.66
S6 Megamind  13.8 ± 0.7  9.36 ± 3.48  11.05 ± 1.75
S6 Coraline  13.4 ± 2.2  10.43 ± 1.31  11.89 ± 2.43

100-way R@10 (%), mean ± SD across runs from three seeds. LOTR = Lord of the Rings.

Additional within-participant cross-movie training experiments: duration-matched R@10

Target movie  Target only  Pooled  Pretrain → FT
S2 Spider-Man  16.7 ± 0.4  17.42 ± 1.58  14.65 ± 1.53
S2 Venom  13.2 ± 1.7  17.08 ± 3.49  17.26 ± 2.39
S2 Guardians 1  17.3 ± 3.9  21.35 ± 9.00  18.69 ± 7.03
S2 Guardians 2  17.6 ± 3.3  20.14 ± 4.10  20.38 ± 4.01
S3 Cars 2  14.6 ± 0.7  15.82 ± 1.85  15.38 ± 2.07
S3 LOTR 1  16.9 ± 2.8  19.05 ± 1.85  18.76 ± 0.25
S3 LOTR 2  16.0 ± 0.8  15.39 ± 2.39  18.80 ± 1.38
S4 Shrek  11.8 ± 1.7  11.65 ± 3.38  14.10 ± 4.40
S4 The Incredibles  12.2 ± 1.3  10.04 ± 0.27  11.51 ± 2.98
S4 Megamind  12.1 ± 1.2  10.90 ± 2.45  10.08 ± 0.86
S6 Toy Story  11.9 ± 1.0  10.66 ± 2.04  10.07 ± 0.63
S6 Megamind  12.2 ± 1.1  9.36 ± 2.83  10.78 ± 2.66
S6 Coraline  14.1 ± 2.8  9.64 ± 0.32  11.75 ± 1.42

Duration-matched 100-way R@10 (%), mean ± SD across the same three runs.

Definitions apply to both tables:

  • Target only: Trained on the target movie's training partition.
  • Pooled: Trained jointly on the target training partition and other movies from the same participant.
  • Pretrain → FT: Pretrained on the participant's other movies, then fine-tuned on the target training partition.

For Subject 2, pooled training increases mean standard R@10 from 17.8% to 20.9%, while pretraining followed by fine-tuning achieves 20.1%. For Subject 3, pooled training increases R@10 from 16.7% to 18.2%, while pretraining and fine-tuning reaches 19.2%.

For Subject 4, the benefit depends on the target movie. On Shrek, pretraining followed by fine-tuning increases standard R@10 from 12.2% to 15.28% and matched R@10 from 11.8% to 14.10%. The corresponding standard R@10 improvements are not observed on Megamind or The Incredibles. For Subject 6, pooled training and pretraining followed by fine-tuning yield lower mean standard R@10 than target-movie-only training on all three recordings.

These additional experiments suggest that incorporating neural–language data from other movies of the same participant can benefit target-movie retrieval, particularly for Subjects 2 and 3, while the results for Subjects 4 and 6 limit a claim of a uniform benefit. Together with the zero-shot results, they provide a more comprehensive assessment of GLANCE's within-participant cross-movie generalization.

We will include these experiments and clarify the distinction between zero-shot transfer, pooled training, and pretraining followed by fine-tuning in the revised manuscript. These experiments assess within-participant transfer and do not establish cross-participant generalization.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Consolidated Summary and Reviewer Follow-Up-part1

Dear Area Chair,

As the discussion period closes, we would like to summarize the experiments we completed in response to the three reviews. The reviewers identified GLANCE’s lead-structured neural encoder and its combination of sentence-level and word-level matching as strengths. Their questions motivated us to test whether GLANCE can retrieve heard sentences when training and test examples are separated in time, when sentence duration is controlled, and when the model is evaluated on a movie held out from training.

We first retrained GLANCE on all 13 recordings from four participants using temporally blocked splits. The revised protocol leaves at least ten seconds between retained neural windows from different partitions and excludes from evaluation any transcript that also appears in training. Participant-averaged Recall@10 (R@10) was 15.13% with randomly sampled distractors and 14.20% with the 99 distractors closest in spoken duration to the target sentence. These blocked results will replace the original random-split results throughout the manuscript.

We then tested whether retrieval persists when every trial provides exactly the same amount of neural activity. We retrained GLANCE, SSENSE, and BPR using only the first 1.37 seconds after sentence onset for each retained trial. GLANCE achieved the highest mean R@10 of the three models on both recordings tested, reaching 17.6% on Spider-Man and 15.0% on Cars 2.

For the broader baseline comparisons, all models used the same splits, seeds, and candidate sets within each recording. GLANCE achieved the highest mean duration-matched R@10 on three of four recordings (Spider-Man, Cars 2, and Toy Story), while SSENSE led on three recordings under standard retrieval. To examine the role of the text encoder, we also retrained BPR and SSENSE with the same EmbeddingGemma targets used by GLANCE. On both recordings tested, each baseline achieved higher R@10 with its original text encoder.

The expanded ablations clarified how word-level matching contributes to sentence retrieval. The local pathway alone achieved mean R@10 of 14.8% across S2’s four movies and 13.8% across S3’s three movies. Retraining GLANCE without this pathway reduced mean standard R@10 on all four ablation recordings, by approximately one percentage point on average. Paired comparisons across the four recording means also favored full GLANCE on R@1, mean reciprocal rank (MRR), and duration-matched MRR (unadjusted p = 0.015, 0.009, and 0.002, respectively). A separate analysis found no reliable alignment with individual word onsets in the single-movie models. The contribution supported by the ablations is that matching contextual word embeddings to neural activity helps rank candidate sentences.

Finally, we evaluated within-participant transfer to held-out movies. For each target movie, we trained on paired neural–transcript examples from the participant’s other movies and used unlabeled target-recording data only for normalization. Transfer was clearest for S2 and S3. Mean R@10 exceeded chance on each of their seven held-out movies, averaging 19.4% for S2 and 16.7% for S3. Other movies also helped when target-movie training examples were available. Relative to target-movie-only training, pooling movies increased S2’s mean R@10 from 17.8% to 20.9%, and pretraining followed by fine-tuning increased S3’s from 16.7% to 19.2%. Transfer was more variable for S4 and S6.

AuthorsOpenReview ↗

Consolidated Summary and Reviewer Follow-Up-part2

Summary of the discussion. All three reviewers now recommend acceptance. Reviewers bZ3r and arh7 raised their ratings after reviewing the new experiments, and Reviewer uADw recommended acceptance from the begining.

We addressed Reviewer uADw’s four questions through the following experiments and clarifications.

  • Comparisons beyond one recording. We expanded the baseline comparisons and retrained ablations to four recordings, one from each participant.
  • Movie viewing versus listening. A control using only sentence position within the movie scored near chance across S2’s four recordings (8.4–10.7%), and GLANCE retained above-chance retrieval with fixed-length inputs on both recordings tested. We will explicitly frame the task as sentence retrieval during audiovisual movie viewing, where linguistic, acoustic, and visual cues can covary.
  • Choice of η and λ. Across 27 runs from nine recordings, selecting these parameters on validation data changed mean test R@10 by only 0.1 percentage points, supporting the use of shared settings.
  • Generalization. We explored initialization from another participant’s model followed by training on the target participant. We also evaluated within-participant cross-movie transfer across all 13 recordings, holding each target movie out of paired training. Questions from reviewers bZ3r and arh7 also led us to audit temporal overlap, introduce the blocked and duration-matched evaluations, compare models using the same text encoder, and test the local pathway’s temporal assignments against annotated word onsets.

GLANCE’s mean R@10 exceeded the 10% chance level in all 13 recordings, even when training and test windows were separated in time and candidate sentences were matched on spoken duration. GLANCE also achieved the highest mean duration-matched R@10 on three of four recordings and the highest mean fixed-input R@10 on both recordings tested. The paired ablations provide further evidence that the local pathway contributes to sentence ranking.

We will incorporate these results into the revised manuscript. We thank the reviewers for the questions that substantially strengthened the paper and the Area Chair for their time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.