acceptodds
Under review as a conference paper at ICLR 2027

Concept-Aligned Neuronal Representation Transfer Across Brain States

Abstract

Neuronal representations formed during a given experience (e.g. place cells) reinstate during subsequent consolidation of that experience despite changes in behavioral state (e.g. place cell replay during sleep). Pinpointing when consolidation of specific memories occurs, especially those formed during one-shot, naturalistic experiences, remains a grand challenge. Detecting such internally-generated reinstatement in humans is particularly challenging with current methods. These combined issues create a single-exposure transfer-and-evaluation problem, where representations learned from sparse neuronal recordings during initial experience poorly generalize for evaluation in states with substantially different distributions. Here, using intracranial microelectrode recordings comprising 452 curated single-neuron units from 13 neurosurgical participants across densely-labeled source (i.e. movie watching) and target states where labels remain sparse (i.e. vocalized memory recall) or unavailable (i.e. sleep), we address this problem using cross-state domain adaptation and held-out neuronal validation during sleep. We build participant-specific models using region-wise attention encoders trained with multi-label classification and contrastive alignment to learned concept embeddings. To improve adaptation across brain states, we introduce Recall-Adapted Sleep Alignment (RASA), which combines sleep-domain alignment with recall-based semantic regularization. Crucially, we leverage the rare presence of concept-selective neurons, known to reactivate during both perceptually driven and internally generated reinstatement of their given concept, to test our decoding framework. Each reference neuron’s recording wire is excluded from model input throughout training, adaptation, and sleep evaluation. These neurons allow evaluation of decoding success during sleep by testing bidirectional temporal alignment between model-derived concept scores and their concept-selective activity even when behavioral labels are unavailable. Tested across 62 neurons selective to concepts during movie watching, RASA produces sleep-time concept scores that are more strongly concentrated around concept-matched held-out neuronal events and more precisely aligned with their timing than those from the movie-trained baseline and alternative neural decoders, providing evidence of sleep-time concept-aligned structure consistent with concept-specific reinstatement. We provide a general framework for transferring concept-aligned representations under severe cross-state distribution shift.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Neural decoding, sleep, memory consolidation, concept-selective neurons, cross-state domain adaptation, representation transfer

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ibfQOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This study investigates a dataset collected from patients with epilepsy, comprising neural recordings during movie watching, memory recall, and sleep, corresponding to densely labeled, sparsely labeled, and unlabeled data, respectively. The authors combine unsupervised domain adaptation with knowledge distillation to transfer a model trained on movie-watching data to the sleep state, while leveraging recall data to preserve concept-related representational structure. Furthermore, the authors identify concept-selective neurons and propose a temporal alignment-based evaluation framework to assess model performance during sleep. Experimental results suggest that the proposed framework can successfully detect concept-related neural reactivation during sleep.

Critical strengths

1.A highly valuable dataset and an important research objective. The study leverages a rare and valuable neural recording dataset to address a highly significant research question. This work represents a major step toward decoding human dream content. 2.A well-matched algorithmic design. Unsupervised domain adaptation has traditionally been less suitable for conventional neural decoding tasks. However, it is particularly well suited to the authors' cross-state transfer setting. Although the methods employed are well established, their application is highly appropriate given the unique characteristics of the dataset. 3.A highly valuable evaluation framework. The use of held-out concept-selective neurons partially addresses the fundamental challenge of evaluating neural decoding during sleep, where ground-truth labels are unavailable. This evaluation strategy provides important insights and inspiration for future research in this field.

Critical weaknesses

1.A somewhat lengthy and unfocused introduction. The introduction covers a broad range of research areas and is slightly longer than necessary. 2.A less compelling title. The current title is not particularly engaging, especially since the term "brain states" is somewhat vague. Explicitly emphasizing the transfer from wakefulness to sleep would better communicate the significance and novelty of the study. 3.Potential information leakage in the evaluation framework. I have some concerns about whether using held-out concept-selective neurons as evaluation references could introduce information leakage. Although the corresponding recording wires are excluded from the model inputs, the activity of these neurons may still be highly correlated with signals recorded from other electrodes.

Questions for authors

My questions are the same as the concerns raised in the Weaknesses section

3 comments
AuthorsOpenReview ↗

Response to Weakness 1: focus and length of the Introduction

Thank you for this helpful suggestion. We have shortened and refocused the Introduction around the central challenge of transferring concept-related neuronal representations from movie viewing to sleep, where behavioral labels are unavailable. We have condensed the broader background discussion and integrated the supporting rodent-replay and human fMRI background into Appendix A, while preserving the central motivation and contributions. Please see Section 1 (Introduction) and Appendix A (Related Work) in the accompanying revised manuscript.

AuthorsOpenReview ↗

Response to Weakness 2: title clarity

Thank you for this thoughtful suggestion. We agree that explicitly naming the wakefulness-to-sleep transition makes the study’s focus clearer. We have revised the title to “Transferring Concept-Aligned Neuronal Representations from Wakefulness to Sleep,” as shown on the title page of the accompanying revised manuscript.

AuthorsOpenReview ↗

Response to Weakness 3: correlations and reference-wire exclusion

Thank you for raising this important concern. Concept-related correlations within the neuronal population are part of the signal our framework is intended to learn. Excluding the reference neuron’s entire recording wire throughout training, adaptation and evaluation prevents the model from directly using that wire’s signal and requires predictions to be derived from the remaining population. We agree that this does not eliminate correlations with retained electrodes. The question is whether those correlations support concept-related representations or allow nonspecific shared activity to account for the observed alignment.

To address this, we performed an additional exclusion sensitivity analysis based on correlations measured during movie viewing. We set the threshold at the 90th percentile of the pooled absolute same-bundle correlations (|r| > 0.1198). In addition to the reference wire, all same-bundle wires exceeding this threshold were excluded throughout training, adaptation and evaluation, without a limit on their number. We trained and evaluated both the movie-only baseline and RASA under this rule using the original BECA evaluator.

RASA’s performance remained broadly stable after exclusion: across the same 62 reference neurons, pooled median specificity changed from 82.48 to 83.43, and temporal alignment from 1.165 to 1.164. These results support the conclusion that the observed alignment is not primarily dependent on the highly correlated same-bundle inputs removed in this control (Appendix B.6, Table 10).

This result complements Table 7, which shows a pooled alignment advantage for correctly matched concept outputs over mismatched outputs. We also found that the pooled matched-concept advantage persisted after excluding events centered at high regional population activity (Appendix B.4, Table 8). Together, these controls support concept-related population correspondence and address the concern that alignment is driven simply by highly correlated local inputs or nonspecific shared activity.

Reviewer_mnJ1OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the transfer of concept-aligned neural representations across different brain states, with a particular focus on decoding concept-related activity during sleep using neural representations learned from stimulus-driven experiences. The authors propose a neural decoding framework that combines hierarchical attention-based neural encoding with multi-label classification and contrastive concept alignment. To address the substantial distribution shift between movie watching and sleep, they introduce Recall-Adapted Sleep Alignment (RASA), which combines CORAL-based distribution alignment with recall-domain knowledge distillation. The former reduces the representation gap between movie and sleep states, while the latter aims to preserve the concept-related representation structure learned during movie viewing. To evaluate sleep decoding in the absence of explicit semantic labels, the authors further propose Bidirectional Event-Conditioned Alignment (BECA). This framework measures the temporal correspondence between model-derived concept scores and the firing activity of independently held-out concept-selective neurons. Experiments are conducted on intracranial microwire recordings from 13 participants, covering movie watching, free recall, and NREM sleep. Using 62 held-out concept-selective neurons as independent references, the proposed method achieves improved BECA specificity and temporal alignment compared with a movie-trained baseline and several existing neural decoding approaches.

Critical strengths

The paper addresses an interesting and meaningful research problem with clear neuroscientific motivation. Transferring concept-related neural representations learned from labeled perceptual experiences to unlabeled sleep states is both challenging and potentially valuable. By combining transfer learning with concept-aligned representation learning and attempting to validate the decoded representations using independent concept-selective neurons, the authors explore a promising direction for cross-state neural decoding.

Critical weaknesses

Weakness 1: The relationship between BECA alignment and genuine concept decoding remains insufficiently established

My primary concern is the gap between the proposed evaluation metric and the interpretation of successful concept decoding during sleep. BECA measures temporal correspondence between model-derived concept scores and the firing activity of independently held-out concept-selective neurons. However, such correspondence does not necessarily imply that the model accurately identifies the semantic content of memory reactivation. For example, both model predictions and reference-neuron activity may be influenced by shared physiological processes, such as variations in neural excitability, local population activity, or sleep-related oscillations. Although the concept-identity permutation analysis provides evidence for concept specificity, it does not fully eliminate these potential confounds. Moreover, BECA specificity is a percentile-based statistic relative to a circular-shift surrogate distribution rather than a conventional classification accuracy measure. A high specificity score therefore should not be interpreted as an equivalent level of sleep concept-decoding accuracy. Since concept decoding during sleep is central to the paper's claims, stronger evidence that BECA captures concept-specific information beyond shared physiological dynamics would substantially improve the work.

Weakness 2: The semantic validity and necessity of recall-guided distillation are not directly demonstrated

An important component of RASA is the use of recall-domain knowledge distillation to preserve concept-related neural representations during sleep adaptation. However, the recall recordings are not used with dense, time-resolved semantic labels, and the teacher model is trained only on movie-viewing data. Therefore, the teacher's embeddings on recall activity are not necessarily semantically accurate. The proposed distillation objective encourages the student to reproduce the frozen teacher's representations for the same recall inputs. While this can limit representational drift, it does not directly establish that the preserved embeddings encode the correct recalled concepts. Similarly, the reported teacher-consistency and recall-geometry agreement metrics measure representational consistency rather than semantic decoding accuracy. Consequently, it remains unclear whether recall provides a unique memory-related semantic constraint or whether a comparable regularization effect could be obtained using other unlabeled neural recordings.

Weakness 3: Dataset provenance and the distinction from prior work require further clarification

The manuscript provides detailed recording and preprocessing descriptions and explicitly states that spike sorting, unit curation, concept annotation, and neuronal selectivity analyses follow the pipeline of Ding et al. (2025). However, the paper does not clearly establish whether the recordings analyzed here were newly collected for this study or reused from an existing clinical dataset. The degree of participant-level and recording-level overlap with Ding et al. (2025) is also not specified. Given that related prior work has already investigated concept decoding from human intracranial recordings during movie viewing and memory recall, clarifying the provenance of the dataset and the independent contributions of the present study is important for assessing novelty. In addition, several methodological components, including CORAL, contrastive learning, and teacher–student distillation, are established techniques. The manuscript would benefit from a clearer explanation of the substantive architectural and methodological advances over closely related previous neural decoders.

Questions for authors

Q1. How robust is BECA to shared sleep-related physiological activity? Could the authors control for local population activity, slow oscillations, sleep spindles, or other sleep-related physiological events when evaluating the temporal correspondence between model predictions and held-out concept-selective neuronal firing? It would be particularly informative to establish whether the reported concept-specific alignment remains significant after accounting for these potential confounds.

Q2. Does recall-guided distillation preserve semantically meaningful representations? The current recall-distillation objective encourages agreement between the student and the movie-trained teacher but does not directly establish the semantic correctness of the teacher's recall representations. Could the authors evaluate concept decoding on a subset of reliably annotated recall events? Additionally, would applying the same distillation objective to alternative unlabeled recordings, such as awake resting-state data, yield comparable improvements? These experiments would help determine whether recall contributes a unique semantic constraint rather than general representation regularization.

Q3. What is the relationship between the current dataset and Ding et al. (2025)? The manuscript explicitly follows the preprocessing and neuronal selectivity pipeline of Ding et al. (2025). Could the authors clarify the extent of participant-level and recording-level overlap between the two studies? Furthermore, what are the principal architectural differences between the present hierarchical neural encoder and the Transformer-based decoder used in the previous work? How would the earlier decoder perform under the same sleep adaptation and BECA evaluation protocol?

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Q1: concept specificity beyond shared physiological activity

Thank you for raising this concern. The central question is whether nonspecific shared activity can explain why each reference neuron aligns preferentially with its corresponding concept output. Our interpretation does not require shared physiological activity to be absent during sleep. Instead, we address this question through two complementary tests: concept-identity dependence and robustness to excluding events centered at high regional population activity.

Alignment depends on concept identity. Each reference neuron's preferred concept is established during movie viewing, independently of its sleep alignment, and its entire recording wire is excluded from model inputs. Our evaluation then asks whether the corresponding concept output aligns more strongly with that neuron than outputs for other concepts.

Table 7 directly tests this correspondence. It compares matched and mismatched concept outputs against the same reference neuron and permutes concept assignments within each movie while preserving the recorded signals and their timing. A common activity trace reproduced identically across outputs would not distinguish correct from incorrect concept assignments. The observed matched advantage therefore establishes an important distinction: the evidence concerns concept-dependent correspondence, beyond temporal co-activation alone. A nonspecific shared-activity explanation must account for this identity dependence, not simply for the presence of correlated firing.

New control for shared regional activity. To address the reviewer's concern more directly, we additionally tested whether this matched advantage persists after excluding events centered at high regional activity. Both the decoder inputs and the reference signal are spiking activity, so sleep-related events such as slow oscillations and spindles can influence their alignment only through shared changes in spiking. We therefore constructed a spike-based population-activity proxy, motivated by work on summed population firing (Okun et al., 2015) and its relationship to NREM dynamics (Vyazovskiy et al., 2009).

After excluding all reference wires, we summed spike counts within each recorded brain region in 250-ms bins. For each region, we marked bins whose spike count exceeded its 90th percentile during NREM sleep. We then counted how many regions were marked in each bin. A bin was classified as high regional activity when this count exceeded its own NREM 90th percentile and at least two regions were marked.

We excluded events centered at high regional activity, applying the same rule to surrogates. For each reference neuron, each evaluated concept output, and each of the four BECA readouts, we calculated the fraction of events remaining after exclusion. The median of these fractions was approximately 93%. We then repeated the concept-identity analyses across 62 neurons from 13 participants, reported in Appendix B.4 and Table 8 of the accompanying revised manuscript, with a summary in Section 3.2:

BECA measureConcept-permutation p (Holm-adjusted)Matched–mismatched contrast [95% participant-clustered CI]
Specificity0.0048026.48 [11.53, 34.60]
Temporal alignment0.006600.307 [0.143, 0.660]

Table note: The concept-identity test used 5,000 permutations; confidence intervals used 5,000 participant-clustered bootstrap resamples. Holm correction covers the two pooled permutation-test outcomes. In the readout-specific breakdown, the embedding-only temporal-alignment permutation test did not reach significance (raw p=0.11178).

Consistent with Table 7, the pooled matched-concept advantage remained significant for both BECA measures after exclusion. Thus, the concept-specific evidence is not confined to events centered at high recorded regional activity. Together, these two tests argue against a simple nonspecific shared-activity explanation.

Finally, we agree that BECA specificity should not be read as classification accuracy: it quantifies alignment relative to temporal surrogates. We have made this distinction explicit in Section 2.5 of the revised manuscript. Because independent concept labels are unavailable during sleep, Table 7's matched–mismatched and concept-permutation controls provide the additional test of concept identity. Our claim is therefore concept-aligned neural activity consistent with reinstatement, rather than directly verified sleep memory content.

AuthorsOpenReview ↗

Response to Weakness 2 and Q2: semantic relevance and the role of recall distillation (1/2)

Thank you for requesting a more direct evaluation against recalled content. Recall distillation is designed to safeguard recall-associated representations during adaptation to sleep. Figure 3 addresses representational anchoring; to address the distinct question of semantic relevance, we additionally evaluated concept-specific model outputs against annotated verbal mentions, rather than relying on teacher consistency alone. This evaluation shows that the movie-trained teacher carries concept-related information during recall.

Why use recall as an anchor? Recall is a principled choice of anchor: it concerns the same movie experience while engaging internally generated activity. Evidence of movie-responsive neuronal reactivation during free recall provides the semantic motivation (Gelbard-Sagiv et al., 2008). A recalled concept need not be verbalized, and speech onset need not coincide with internal recall onset. We therefore use recall without dense training labels and use verbal annotations for supplementary evaluation. This motivates recall as a content-related anchor, without requiring it to be the only possible source of regularization. Suitable alternative unlabeled recordings were unavailable for a substitution control, so their comparative effectiveness remains an open question.

AuthorsOpenReview ↗

Response to Weakness 2 and Q2: semantic relevance and the role of recall distillation (2/2)

Additional evaluation against recall annotations. We used existing timestamps for concept names and referring expressions during pre- and post-sleep free recall. Because verbal reports provide incomplete observations of internal recall, we focused on repeatedly mentioned concepts, allowing model–vocalization alignment to be assessed across multiple events rather than isolated mentions. Requiring at least ten eligible matched mentions with complete pre-vocalization windows per participant–movie–concept combination across pre- and post-sleep free-recall recordings yielded model evaluations corresponding to 24 reference neurons from nine participants.

We evaluated classifier and embedding outputs during free recall using 2-s windows relative to concept-mention onset. Recall specificity compared responses associated with matched mentions against 2,500 equally sized samples of transcript-control words from the same recording. Temporal alignment measured the increase in model activity relative to neighboring windows. In the accompanying revised manuscript, these results are presented in Section 3.3 (Recall-Guided Preservation During Sleep Adaptation) and Table 4, with evaluation details in Appendix B.3.

From the movie-only baseline to the full adaptation framework. We evaluated all four models on the same recall events. The comparisons follow the design of the framework: establish the movie-trained starting point, examine recall distillation and sleep alignment separately, and then examine their combination.

  1. Movie-only baseline (Movie). We begin with a model trained only on movie data and evaluate its responses during free recall. Its BECA scores establish the recall-associated alignment available before either adaptation component is introduced.

  2. Recall distillation only (Movie + Recall). We next examine recall distillation without sleep alignment. Recall-associated alignment is retained relative to the Movie-only baseline, with no reduction in any of the four aggregate readouts. This pattern is consistent with the anchoring objective of recall distillation. This comparison alone, however, does not test protection against changes introduced by sleep alignment.

  3. Sleep alignment only (sCoral) (Movie + Sleep). We therefore examine adaptation to sleep without recall distillation. All four recall readouts are lower than the Movie-only baseline. This identifies the change that the recall safeguard is intended to mitigate: aligning representations to sleep can be accompanied by reduced alignment with concept-related verbal mentions during recall.

  4. RASA (Movie + Recall + Sleep). Finally, we combine sleep alignment with recall distillation. Compared with Sleep alignment only (sCoral), RASA yields higher specificity and temporal alignment for both classifier and embedding outputs. The recovery across all four recall readouts is consistent with the intended safeguard during sleep adaptation.

The objective is to transfer representations to sleep while preserving recall-associated structure, rather than to maximize recall performance. RASA need not exceed the Movie-only baseline on every recall metric; the key comparison is whether recall-associated alignment is better retained than with sleep alignment alone. The exploratory results below show this pattern.

Vocalization-to-model BECA evaluation during free recall

ModelTraining dataClassifier specificityClassifier temporal alignmentEmbedding specificityEmbedding temporal alignment
Movie-only baselineMovie82.56 ± 4.641.203 ± 0.15382.80 ± 3.871.177 ± 0.135
Recall distillation onlyMovie + Recall87.90 ± 4.111.274 ± 0.13584.22 ± 3.621.316 ± 0.118
Sleep alignment only (sCoral)Movie + Sleep68.72 ± 4.950.536 ± 0.17576.64 ± 4.040.820 ± 0.183
RASAMovie + Recall + Sleep84.06 ± 2.791.059 ± 0.17882.20 ± 3.761.021 ± 0.136

Values are median ± SEM across 24 reference-neuron-associated model evaluations during free recall, not sleep. Specificity uses transcript-control sampling (null expectation: 50). SEM is calculated as SD/√24, following the manuscript’s reporting convention.

AuthorsOpenReview ↗

Response to Weakness 3 and Q3: dataset provenance and distinction from Ding et al. (2025)

Thank you for requesting this clarification. Our contribution is concept-aligned representation transfer and evaluation during NREM sleep, extending beyond the awake-recall decoding studied by Ding et al. (2026).

Dataset overlap and additional recordings. We refer here to the revised 2026 version of Ding et al., which reports an expanded cohort of 15 participants, and have updated the citation accordingly. We introduce sleep recordings from all 13 participants for neural content decoding, extending the dataset beyond the awake recordings analyzed by Ding et al. (2026). These 13 participants are a subset of Ding et al.'s 15-participant cohort, and their 24 movie-viewing and recall recordings are reused from that study. We additionally introduce House, Twilight, and The Mandalorian recordings from three of these participants. Neither the sleep recordings nor these additional movie recordings were analyzed for neural content decoding by Ding et al. In the accompanying revised manuscript, this participant- and recording-level overlap is documented in Section 2.2, with the encoder relationship clarified in Appendix C.3. Section 1 also explicitly acknowledges prior awake-recall decoding and distinguishes the extension to sleep.

Methodological contributions. Moving from awake recall, where verbal reports provide an external reference, to sleep without time-resolved content labels requires both cross-state adaptation and a different evaluation strategy. Our framework addresses these requirements through:

  • Dual-head concept-aligned representations. Whereas Ding et al. use a classification decoder, our model jointly learns a classification head and a concept-aligned embedding head through classification and contrastive objectives. The embedding head provides concept-similarity readouts and the representation space used for sleep alignment and recall distillation. Both encoders integrate local activity across microwire bundles, but our encoder uses a learned global CLS token rather than averaging outputs from region-specific cross-region queries.
  • Recall-guided sleep adaptation. RASA combines movie–sleep distribution alignment with recall distillation to anchor representations during adaptation.
  • Sleep evaluation. BECA uses independently held-out concept-selective neurons, supplemented by concept-identity controls, to assess concept-related alignment without behavioral sleep labels.

CORAL, contrastive learning and distillation are established techniques; the contribution lies in the representation, adaptation and evaluation framework that connects them to the sleep-transfer problem.

Baseline selection. To isolate the contribution of our representation design, we compared classifier-only and dual-head training using the same encoder. In Section 3.1 and Table 2 of the revised manuscript, joint training increases classifier specificity from 56.10 to 68.40 and temporal alignment from 0.94 to 1.08, before sleep adaptation. This controlled comparison directly tests the contribution of joint classification and concept-aligned embedding learning. Ding et al.’s decoder is a closely related classification-based model; incorporating it into the full RASA pipeline would additionally require introducing the concept-aligned embedding head used by our adaptation objectives. We therefore prioritized the controlled within-encoder ablation alongside representative external baselines.

Reviewer_L55BOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a framework to learn neuronal representations across varying brain states in humans from microwire recordings in the media temporal lobe and other distributed brain regions. It relies on the setup where densely-labeled, one-shot stimuli presentations (here: movie) are used to record brain activity, combined with follow-up sessions of active recall of concepts shown during the movie presentation and un-labeled recordings during sleep phases. Those three phases represent correlated recordings, which are subject to a severe cross-state distribution shift, making it hard to align neural activity. The presented model allows the learning of neuronal representations under this shift by training an attention-based encoder that uses two learning objectives simultaneously: multi-label concept classification and contrastive learning using pre-learned concept embeddings. The representations are then further improved by adaptation across brain states (recall-Adapted Sleep Alignment - RASA), which aligns the representation to the sleep domain combined with a recall-based semantic regularization. The semantic regularization is achieved through distillation - specifically distilling recall-domain embeddings against a frozen, movie-trained teacher model to protect the semantic geometry. The study is based on a rare dataset of 13 neurosurgical patients, ultimately utilizing exactly 62 concept-selective neurons (out of a total of 452 curated units) as the independent, held-out references for sleep evaluation. The alignment is evaluated using Bidirectional Event-Conditioned Alignment (BCEA), which measures the bidirectional temporal correlation between model-derived concept scores and their concept-selective activity in sleep phases where labels are unavailable. The authors show that the presented framework surpasses five comparable decoders (EEG Conformer, EEGNet, SPaRCNet, CEBRA and SIMPL).

Critical strengths

(1) The problem is meaningful and difficult. Transferring representations from a single naturalistic experience to sleep combines substantial distribution shift with an absence of behavioral labels. The recorded dataset is very valuable, and the introduction of the RASA component is an interesting contribution to the field.

(2) Validation design is thoughtful. The exclusion of a selective neuron’s entire recording wire throughout training and adaptation provides a useful independent measurement, and requires substantial effort in setting up the validation framework for the sleep phase. Additionally, testing alignment in both directions and including mismatched-concept controls make the evidence substantially stronger.

(3) The authors draw realistic conclusions from their results. They acknowledge that agreement with concept-selective neurons does not establish subjective sleep content or reconstruct specific memories. While the nature of the data inherently limits interpretation, the authors handle these constraints with careful scientific judgement.

Critical weaknesses

(1) The baseline comparison mixes representation quality with adaptation. RASA receives sleep alignment and recall-based regularization, whereas the external baselines appear to receive movie training alone. This demonstrates the strength of the complete pipeline, but provides weaker evidence that the proposed encoder or dual-head representation is superior. Applying comparable adaptation to suitable baselines would be more informative and strengthen the manuscript.

(2) The strength and consistency of the gains need closer scrutiny. Specificity improves substantially, but temporal alignment improves less uniformly: sleep alignment alone slightly reduces the pooled median relative to baseline. With 62 neurons nested within 13 participants and multiple readouts per neuron, participant-level uncertainty matters. The participant-clustered analyses are helpful, and similarly clear uncertainty estimates for the main adaptation comparisons would be beneficial.

(3) The recall safeguard is supported partly by metrics it directly optimizes. Distillation explicitly preserves agreement with a frozen teacher, so improved teacher consistency is expected. Geometry preservation adds evidence, but neither establishes that the preserved representation correctly captures recalled content. The authors acknowledge this; it would still be favourable if clearer evidence of source-domain decoding quality were provided and, where feasible, validation against recall annotations.

Questions for authors

Smaller typos and language feedback\textbf{Smaller typos and language feedback}

  • P.8 and p.18: EEG Conformer has different results: specificity 52.80 ± 3.69 versus 54.60 ± 3.71, and temporal alignment 0.61 ± 0.18 versus 0.10 ± 0.20. Clarify the difference?
  • P. 8 and p.20, Tables 2 and 7: RASA’s pooled temporal-alignment medians differ: classifier 1.29 versus 1.334, and embedding 1.03 versus 0.988. Both tables appear to summarize the corresponding paired BECA readouts; please provide clarification?
  • P.17-19: Temporal alignment is labeled a “z-score,” whereas Sections 2.5 and C.5 define it as a paired t-statistic. Standardize the labels and notation, or explain any transformation.
  • P.10: The AI statement is not complete, including brackets to fill in.
  • P. 22: “frontal, pars, and cingulate regions”: “pars” appears incomplete, specify the intended anatomical regions.

General comments and clarification points\textbf{General comments and clarification points} (1) Learned concept embeddings: Mentioned several times in the manuscript (p.3,4,5,24), but it is unclear how these learned concept embeddings were obtained. I cannot find any reference in the manuscript about it. Further clarifications here are recommended.

(2) While I understand that the total analysis and dataset presentation require substantial space to explain its various components, the length of the Introduction and Methods sections seems disproportionate to the very brief Results section. I recommend restructuring the manuscript and Supplementary materials to improve this balance, ensuring that the most critical results are visualized in the main text.

(3) The manuscript does not provide a clear overview of distributions across patients. It would be interestint to know how the selected neurons distribute across the patients. How many selective neurons are used for each patient? What is the anatomical distribution, i.e., in what subregions are those neurons? What is the final population trained after wires for selective neurons are remove? A table specifying those would be helpful.

(4) Is there any possibility to pool all neurons across patients to generate a pseudo population, as is often common for these kinds of datasets? I assume this is not feasible due to the experimental setup with sleep phases? Would it make sense to include a pre-training phase on the movie and recall phases using the pseudo population?

(5) Table 1: One participant-specific model is trained for each concept-selective neuron. The table is a bit misleading, providing total movie time, sleep duration or recall duration as summed values across all patients. For each model the data is considerably smaller, so to be transparent, I would include measures like average values here instead.

(6) Please clarify whether the continuous movie-viewing recordings were partitioned into training, validation, and test sets. If so, were partitions defined by individual windows or contiguous temporal blocks, and were temporal gaps used to reduce dependence between partitions? The manuscript lacks information on this. Given that continuous data is analyzed, a rigorous separation of train and test data is critical. As Gerken et al. (2025) demonstrated, temporal autocorrelation can artificially inflate decoding performance. If data splits in this study were not carefully curated to account for this, the movie baseline model likely suffers from similar data leakage. Please provide more details here, and provide detailed information on how temporal separation of train and test data were implemented.

(7) Figure 3 is referenced first, Figure 2 only later. I would recommend to change the order to align with how it is discussed throughout the text? Also placement of the paragraph about Figure 3 seems not very fitting, it cites methods that are only explained later. Consider shifting this paragraph.

(8) The weighting of those losses allows to optimize the training per patient and per concept, detailed information on those hyperparameters are missing and would be beneficial for reproducibility. Please provide more details.

(9) Figure 5 shows BCEA heatmap for all concept neurons. Is there a concept-related structure visible, i.e., some concepts stronger than others?

(10) Concept-selective neurons were identified based on their responses during movie viewing and subsequently used as reference signals for sleep evaluation. Did the authors assess whether these neurons retained their concept selectivity during free recall? Where recall annotations permit, testing whether their activity is preferentially associated with recall of the corresponding concept, allowing for temporal uncertainty relative to verbal reports, would provide a useful intermediate validation of their use as concept-specific references during sleep.

10 comments
AuthorsOpenReview ↗

Response to Weakness 1: distinguishing representation quality from adaptation

Thank you for highlighting the need to distinguish representation learning from adaptation. The manuscript already includes movie-only representation comparisons and within-model adaptation comparisons. The evidence for the representation design therefore does not rest solely on the full RASA pipeline. In the accompanying revised manuscript, we have consolidated the movie-only external-baseline and within-encoder ablation results in Section 3.1 and Table 2, retaining the RASA results and explicitly identifying the training conditions. Appendix B.2 provides the corresponding architectural rationale and ablation discussion. The consolidated results are shown below.

Movie-only comparison: no sleep alignment or recall distillation

ReadoutModelBECA specificityBECA temporal alignment
ClassifierEEG Conformer52.80 ± 3.690.61 ± 0.18
ClassifierEEGNet54.85 ± 3.370.77 ± 0.29
ClassifierSPaRCNet51.00 ± 3.590.45 ± 0.21
ClassifierOurs, classifier-head only56.10 ± 3.330.94 ± 0.11
ClassifierOurs, full dual-head (movie-only)68.40 ± 3.311.08 ± 0.13
EmbeddingCEBRA52.10 ± 3.15−0.22 ± 0.14
EmbeddingSIMPL48.90 ± 3.50−0.50 ± 0.22
EmbeddingOurs, embedding-head only53.45 ± 3.361.07 ± 0.11
EmbeddingOurs, full dual-head (movie-only)62.25 ± 3.370.86 ± 0.11

Sleep evaluation of movie-only models. Values are median ± SEM across pooled neuron–direction readouts within each output head.

Under movie-only training, our dual-head model achieves higher specificity and temporal alignment than the listed external baselines in both readout groups. These advantages are already present before adaptation and come from joint classification and contrastive training: with a single head, the same encoder reaches a specificity of 56.10 (classifier) and 53.45 (embedding), compared with 68.40 and 62.25 with both heads.

Separating representation quality from adaptation. We did not apply RASA to the external baselines. RASA's objectives act on a concept-aligned embedding space, so applying it would require modifying the original models, for example by adding an embedding head to the classifier baselines and changing their training objectives. Such a comparison would assess modified versions of the baselines within our adaptation framework, rather than the published methods themselves. We instead separate the two questions with controlled comparisons, and now distinguish them in Section 3.1 and Section 4 (Conclusion and Discussion): the consolidated movie-only comparison supports performance differences before adaptation, while Table 3 evaluates adaptation within our model. The complete-pipeline comparison is not used to establish encoder superiority under matched adaptation.

AuthorsOpenReview ↗

Response to Weakness 2: adaptation gains and participant-level uncertainty

Thank you for emphasizing participant-level uncertainty. We have now added a paired participant-clustered bootstrap for the main adaptation comparisons in Table 3, directly accounting for the dependence among neurons and readouts from the same participant.

Participant-clustered uncertainty. We performed a paired bootstrap on the same 62 neurons from 13 participants as Table 3. In each of 10,000 replicates, we resampled participants with replacement, retaining all their neurons and readouts and using the same sample for every model. The table reports differences between pooled medians and their 95% bootstrap CIs, directly corresponding to the Table 3 comparisons.

ComparisonSpecificity differenceTemporal-alignment difference
RASA − movie-only baseline17.725 [0.799, 35.350]0.174 [−0.109, 0.396]
RASA − +sCoral4.275 [−5.526, 12.200]0.209 [−0.171, 0.365]

The participant-clustered analysis supports RASA’s specificity gain over the movie-only baseline. Temporal-alignment point estimates favor RASA in both comparisons, although their pooled 95% CIs include zero; the specificity difference between RASA and sCoral also has a CI spanning zero. The small sCoral-only temporal change is −0.035 [−0.227, 0.240], so the analysis does not establish a decline.

Distribution alignment and recall anchoring serve complementary roles. The movie-only baseline provides the starting point. Adding sCoral introduces sleep distribution alignment without the recall safeguard: the pooled temporal-alignment median changes from 0.99 to 0.96. Adding recall distillation to form RASA raises this value to 1.17. Sleep CORAL acts as a distribution-alignment regularizer, not a direct objective for BECA temporal alignment; the recall constraint is intended to anchor representations during this adaptation. The observed ordering is consistent with these complementary roles.

Our vocalization-to-model evaluation during free recall in Section 3.3 and Table 4, described in our response to W3, shows a corresponding descriptive pattern: sleep alignment alone yields lower recall readouts than the movie-only baseline, while RASA yields higher values than sleep alignment alone on all four readouts. These recall results complement the sleep results in Table 3; they do not replace an uncertainty analysis of the sleep gains.

In the accompanying revised manuscript, the paired participant-clustered confidence intervals are included in Section 3.2 and Table 3, with the bootstrap procedure in Appendix B.1.

AuthorsOpenReview ↗

Response to Weakness 3: recall semantic validation and source-domain decoding quality (1/2)

Thank you for requesting evidence beyond teacher consistency. Recall distillation is designed to anchor recall-associated representations during adaptation to sleep. Figure 3 and the consistency measurements address this representational role; we additionally evaluated model outputs against annotated verbal mentions to examine their relationship to recalled content. This evaluation shows that the movie-trained teacher carries concept-related information during recall.

Source-domain decoding quality. As explained in the Introduction, our focus is cross-state transfer to sleep, while stimulus-driven decoding has been investigated in prior work (Gerken et al., 2025; Zhang et al., 2023). We use all available movie data to maximize coverage of the single recorded experience and have clarified this training-data usage in Section 2.3. Held-out movie decoding would assess source-domain generalization but would not by itself establish successful sleep transfer. The recall evaluation below provides complementary evidence using content annotations outside the movie-viewing state.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks by Authors

Dear Reviewers and Area Chairs,

We sincerely thank all reviewers and ACs for their insightful and constructive feedback throughout the review process.

Understanding how the human brain represents recent experience during sleep requires both transferring representations across brain states and evaluating them without behavioral content labels. Our paper addresses these challenges using rare intracranial recordings spanning movie watching, free recall, and overnight sleep within the same individuals. We learn concept-aligned representations from a single naturalistic movie experience and introduce Recall-Adapted Sleep Alignment (RASA), which uses recall to safeguard these representations during adaptation to sleep. Bidirectional Event-Conditioned Alignment (BECA) then evaluates their temporal correspondence with independently held-out concept-selective neurons.

All three reviewers recognized the importance of transferring concept-related neuronal representations from perceptual experience to sleep. L55B and ibfQ highlighted the rare dataset and the held-out-neuron evaluation framework. L55B identified RASA as an interesting contribution, while ibfQ emphasized that the adaptation approach is particularly well suited to this cross-state setting. mnJ1 described the approach as a promising direction for cross-state neural decoding. L55B also appreciated our careful distinction between concept-related neural alignment and subjective sleep content.

The reviewers asked for stronger evidence of content specificity and the recall safeguard, clearer separation of representation learning from adaptation, and clarification of the relationship to prior work. Our rebuttal and revisions address these concerns through the following evidence and clarifications:

  • Concept-specific alignment during sleep (mnJ1, ibfQ). Our concept-identity controls compare matched and mismatched concept outputs against the same reference neuron. In a new regional population-activity exclusion analysis, the pooled matched-concept advantage remained significant for both BECA measures. RASA’s specificity and temporal alignment also remained broadly stable after excluding highly correlated same-bundle inputs. Together, these controls strengthen the evidence for concept-specific correspondence beyond a simple nonspecific shared-activity explanation. We clarified that BECA specificity is a surrogate percentile, not classification accuracy. Details are provided in Section 3.2 and Appendices B.4 and B.6.

  • Recall as a safeguard for content-related representations (mnJ1, L55B). To test whether recall preserves content-related structure beyond agreement with a frozen teacher, we added an exploratory evaluation of model outputs against verbal concept mentions. Relative to the movie-only baseline, recall distillation alone retained recall-associated alignment, whereas sleep alignment alone reduced all four aggregate readouts. Adding recall distillation increased all four relative to sleep alignment alone, consistent with the intended safeguard. Reference neurons also showed elevated activity around matched recall mentions relative to control words, providing complementary evidence for their content-related responses (Section 3.3 and Appendix B.3).

  • Contributions of representation learning and adaptation (L55B). We consolidated existing movie-only external-baseline and within-encoder ablation results, showing that the dual-head model’s advantages over external baselines are present before adaptation. We separately assessed adaptation gains with participant-clustered uncertainty. Point estimates favor RASA across all four adaptation contrasts, with the participant-clustered 95% confidence interval excluding zero for the specificity gain over the movie-only baseline and spanning zero for the other three (Sections 3.1–3.2, Tables 2–3).

  • Distinct contributions and dataset provenance (mnJ1, L55B, ibfQ). We clarified the study’s distinct focus: learning concept-aligned representations that transfer to unlabeled sleep and evaluating them through independent neuronal references. This requires cross-state adaptation and a validation strategy beyond the awake-recall decoding studied by Ding et al. (2026). We separately documented participant- and recording-level overlap, distinguishing reused movie/recall data from the sleep analyses and additional movie recordings introduced here (Section 2.2). We also streamlined the Introduction, expanded the Results, revised the title, and corrected table entries and temporal-alignment terminology.

Together, the rare recordings and this framework make it possible to test whether representations learned from a specific waking experience retain concept-related structure during sleep. The additional analyses strengthen this foundation for studying human memory representations across states without dense behavioral labels.

Thank you for considering the revised manuscript and our responses.

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.