acceptodds
Under review as a conference paper at ICLR 2027

Bounded Integration Windows Gate Hybrid Continuous–Sparse Readouts in a Spike-Converted Transformer Aligned with Language Cortex

Abstract

Large language models and the human brain both turn streams of words into structured meaning. A growing body of work shows that the internal activations of Transformer language models can linearly predict brain responses recorded while people listen to natural speech. How far this convergence reflects shared computation between language models and human brain remains open. Two basic properties of language cortex have been hard to test against models. Cortical regions integrate information over bounded, region-specific timescales rather than indefinitely. Cortical activity also contains sparse, event-like structure, which current model–brain comparisons impose by post-hoc analysis rather than read out from the model. The obstacle is instrumental: standard Transformers expose only continuous activations and a fixed context window. We remove it with a same-weight artificial neural network (ANN)  spiking neural network (SNN) substrate. A pretrained Transformer is converted into a spiking network without retraining, keeping the original weights but unfolding computation over discrete timesteps. Continuous, sparse, and hybrid representations can then be read from a single forward pass, and the model-side integration window becomes a tunable scalar. Applied to naturalistic fMRI from 49 listeners, the substrate reveals three coupled phenomena. Brain alignment saturates inside a bounded integration plateau, where the model also shifts toward more compositional layers, a profile that replicates on a second backbone and conversion family. Within this plateau, the hybrid continuous–sparse readout predicts held-out responses better than a matched random control, while a redundant signed variant does not. The size of this advantage is non-monotonic in unfolding time and peaks earlier under faster leak across all twelve language parcels, while the relative hybrid lift descriptively tracks the cortical hierarchy of temporal receptive windows. Together, these results are consistent with a hybrid continuous–sparse code that is useful only inside a finite integration window, and they suggest that bounded temporal dynamics—long studied in spiking systems for biological reasons—may also act as a useful inductive bias for representations that align with the brain.

Primary area:
applications to neuroscience & cognitive science
Keywords:
brain encoding, fMRI, spiking neural networks, temporal integration, language models, language model interpretability

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BFsjOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper converts a pretrained Transformer into a spiking network without retraining it. From the same forward pass, it extracts continuous, sparse, and hybrid continuous–sparse representations. Using naturalistic fMRI data from 49 people who listened to the same audiobook, the authors study how the number of integration steps, the leak rate, and the representation format affect brain-signal prediction. The results suggest that the model aligns better with language-cortex activity within a limited integration window, and that the hybrid representation outperforms a random sparse control across 13 language-related brain regions. However, this result should be interpreted as evidence about the model’s temporal dynamics rather than as a direct measurement of the brain’s actual time scale.

Critical strengths

The experiments are carefully designed, with random sparse controls, a second conversion method, and multiple temporal settings.
The paper connects spiking neural networks, representation formats, and brain-signal prediction, making it interesting from an interdisciplinary perspective.

Critical weaknesses

The conclusions are sensitive to the evaluation metric. The Pearson R2R^2 analysis can lead to a different conclusion from the main metric. Also, the second conversion experiment changes both the backbone model and the conversion method, so their effects cannot be fully separated.
fMRI has relatively low temporal resolution, so the model’s integration window cannot be directly interpreted as the actual temporal scale of the human cortex.

Questions for authors

The conclusions are sensitive to the evaluation metric. The Pearson R2R^2 analysis can lead to a different conclusion from the main metric. Also, the second conversion experiment changes both the backbone model and the conversion method, so their effects cannot be fully separated.
fMRI has relatively low temporal resolution, so the model’s integration window cannot be directly interpreted as the actual temporal scale of the human cortex.

4 comments
AuthorsOpenReview ↗

Response to Reviewer BFsj (1/3): overview; W1, sensitivity to the metric

We thank the reviewer for recognizing the motivation and direction of this work, and for noting that the experiments are "carefully designed, with random sparse controls, a second conversion method, and multiple temporal settings". The review raises three points. Two of them, separating backbone from conversion and using time-resolved recordings, are steps the paper lists as future work; we have now carried them out. Replies 1–3 take the points in order, with results and the change in the paper; we will upload one revised PDF after replying to all reviewers.

Your pointWhat we didOutcomeIn the revision
W1: conclusions depend on the metricRise and plateau under both metric families; the hybrid contrast on new data with both metrics fixed in advanceRise and plateau hold under both; hybrid contrast significant under both in 3 of 4 intracranial tests§5.1–5.3, App. C.5 (B1)
W1: backbone and conversion change togetherPaper's backbone with the second conversion; four more checkpointsRise and plateau replicate when only the conversion changes, and when the backbone changes§5.1, App. C.3 (B2)
W2: fMRI has low temporal resolutionTwo millisecond-resolution intracranial datasets; a second fMRI datasetThe rise replicates in every participant; under the correlation metric the plateau starts at the same cells§5.1, App. C.4–C.5 (B3)

W1 (first part): sensitivity to the metric

The paper's first conclusion, the rise and plateau of alignment, holds under both metric families in every dataset. For the hybrid contrast the paper uses held-out R²; on independent intracranial data, with both metrics fixed in advance, the contrast holds under both in three of four tests.

1. Rise and plateau hold under both metrics (Table B1, top). With the paper's model, alignment rises from T=4 to T=13 in every participant of every dataset under the correlation metric, and in all but one listener under held-out R², and it levels off under both.

2. On independent intracranial data the hybrid contrast holds under both metrics in three of four tests (Table B1, bottom). For the two intracranial datasets, held-out R² and a correlation metric were named before any hybrid result existed. The contrast is significant under both at both peak cells on stereo-EEG and at T6/τ=1.60 on ECoG.

Table B1. Paper's model. Top: participants with a positive rise from T=4 to T=13 (τ=1.30; τ=1.60). Bottom: matched contrast after the simulation null, q over the two cells.

Correlation metricHeld-out R²
Rise, Little Prince fMRI49/49; 49/4949/49; 48/49
Rise, second fMRI dataset (LeBel et al.)8/8; 8/88/8; 8/8
Rise, ECoG9/9; 9/99/9; 9/9
Rise, stereo-EEG10/10; 10/1010/10; 10/10
Contrast, stereo-EEG, T10/τ=1.308/10, q = .0088/10, q = .005
Contrast, stereo-EEG, T6/τ=1.609/10, q = .00910/10, q = .002
Contrast, ECoG, T10/τ=1.307/9, q = .1078/9, q = .053
Contrast, ECoG, T6/τ=1.608/9, q = .0317/9, q = .031

3. Why held-out R² for the hybrid contrast. It scores the prediction error directly, the quantity the encoder is fitted to minimize. The paper reports the divergence from Pearson² itself (App. A.7), which Reviewer 23qG lists as a strength ("transparently reports cases where different metrics support different conclusions"). Because held-out R² was adopted after that divergence was seen, the appropriate check is the one in point 2: new data, metrics fixed beforehand.

In the revision. §5.1 reports the rise and plateau under both metrics; §5.2–5.3 state the fMRI hybrid result for held-out R² and name the metrics with every result; Table B1 goes into new Appendix C.5.

AuthorsOpenReview ↗

Response to Reviewer BFsj (2/3): W1, backbone versus conversion

This is right for the submission, and the paper lists the separation as a next step ("a factorial design that separates backbone from conversion", Future work). We have now run it as far as the conversions allow (point 3). The rise and plateau replicate when only the conversion changes, and when the backbone changes under a fixed conversion (Table B2).

1. Same backbone, second conversion. We converted the paper's backbone, MA-BERT, with the second conversion, LAS, without any training or calibration. Alignment gains +0.052 from T=4 to T=13 in 48 of 49 listeners. From T=13 to T=16 it is equivalent to zero within ±10% of that gain, a bound set before the test, and at T=16 the converted network differs from its parent ANN by −0.0002. The advantage of layer 8 over layer 4 also grows from T=11 to T=16 under the correlation metric (40/49 listeners).

2. Same conversion, other backbones. With LAS fixed, we replaced MA-BERT by standard BERT: the paper's fine-tuned LAS model and three checkpoints that were never fine-tuned, one of them Chinese and tested on the 35 Chinese listeners of the paper's corpus. The rise appears in every listener for all four.

Table B2. Rise of layer-8 alignment from T=4 to T=13 and change from T=13 to T=16 (Little Prince fMRI, correlation metric, final pipeline).

BackboneConversionRise [95% CI]Listeners positiveT=16 minus T=13 (share of rise)
MA-BERT (paper's model), τ=1.30SpikedAttention+0.067 [+0.059, +0.075]49/49+0.0004 (0.6%)
MA-BERTLAS+0.052 [+0.045, +0.059]48/49−0.00003 (0.05%)
BERT fine-tuned on SST-2 (paper's LAS model)LAS+0.062349/49+0.0034 (5.4%)
BERT-base-cased, not fine-tunedLAS+0.0646 [+0.0569, +0.0722]49/49+0.0020 (3.1%)
BERT-base-uncased, not fine-tunedLAS+0.065349/49+0.0025 (3.8%)
bert-base-chinese, not fine-tuned (Chinese cohort)LAS+0.0967 [+0.0878, +0.1058]35/35+0.0017 (1.7%)

3. Why these comparisons. A standard BERT is not directly compatible with SpikedAttention (§3.1), so the backbone can be varied only under LAS, and the conversion can be varied only on MA-BERT. Rows 1–2 change the conversion alone; rows 2–6 change the backbone under one conversion. The profile appears in all of them, so it belongs neither to one backbone nor to one conversion. This is the replication that Reviewer 23qG describes as strengthening the investigation, now with the two factors varied separately.

In the revision. §5.1 replaces the single second-conversion replication by Table B2 and states which factor each row changes; the Limitations paragraph on the second conversion is updated accordingly; per-listener values go into new Appendix C.3.

AuthorsOpenReview ↗

Response to Reviewer BFsj (3/3): W2, the temporal resolution of fMRI

This is the reading the paper intends, and the review's summary states it well: the result is "evidence about the model's temporal dynamics rather than as a direct measurement of the brain's actual time scale". The window is a property of the model. What the new data add is that it does not depend on the slow fMRI signal: it replicates on recordings with millisecond resolution, with the same onsets under the correlation metric.

1. The paper's position. T and τ "are model-side computational variables and are not a proxy for cortical temporal receptive windows", and "the bounded plateau is a property of the model's encoding performance and is not a tested cortical claim" (Limitations). The abstract and introduction list a correspondence with cortical temporal receptive windows among the findings; the revision deletes that sentence from both, and no result depends on it.

2. Time-resolved recordings. The paper names "ECoG or MEG with finer temporal resolution" as future work. We ran the same sweep of T and τ on two independent intracranial datasets with word-locked, millisecond-resolution responses: Podcast electrocorticography (ECoG, 9 participants) and Brain Treebank stereo-EEG (10 participants). Alignment rises from T=4 to T=13 at both leaks in every participant (Table B3). Under the correlation metric the plateau starts at T=10 (τ=1.30) and T=6 (τ=1.60), the same cells as on fMRI. At T=16 the converted network reaches its parent ANN's level (ECoG: r = 0.142 and 0.138, against 0.140).

3. A second fMRI dataset. On LeBel et al. (2023; 8 participants, a different stimulus) the rise appears in all eight participants and the plateau starts at the same two cells.

Table B3. Paper's model: rise of alignment from T=4 to T=13 (correlation metric; 95% CI where computed) and start of the plateau.

DataResolutionRise, τ=1.30Rise, τ=1.60PositivePlateau starts (τ=1.30; 1.60)
Little Prince fMRI (paper)2 s+0.067 [+0.059, +0.075]+0.042 [+0.037, +0.046]49/49; 49/49T=10; T=6
LeBel et al. fMRI2 s+0.124+0.0758/8; 8/8T=10; T=6
Podcast ECoGmilliseconds+0.137 [+0.110, +0.161]+0.093 [+0.070, +0.112]9/9; 9/9T=10; T=6
Brain Treebank stereo-EEGmilliseconds+0.082 [+0.068, +0.096]+0.063 [+0.054, +0.071]10/10; 10/10T=10; T=6

So the bounded window is a model-side property that is stable across recording methods, stimuli and participants. Whether it corresponds to a cortical timescale is a separate question that the paper does not claim to answer.

In the revision. §5.1 adds Table B3; the sentence on cortical temporal receptive windows leaves the abstract and introduction; §5.1 and the conclusion describe the window as a property of the converted network; per-participant values go into new Appendix C.4 (second fMRI dataset) and C.5 (intracranial datasets).

We thank the reviewer again. The three points led to new experiments, two of them steps the paper had listed as future work, and with them its claims are better separated: a rise and plateau that hold across metrics, conversions, backbones and recording methods, and a hybrid result stated with the metric under which it was measured, and significant under both in three of four independent intracranial tests.

AuthorsOpenReview ↗

Response to Reviewer BFsj

We sincerely appreciate your time and constructive feedback. With the discussion period approaching its end, we would like to gently follow up and see whether our rebuttal has sufficiently addressed your comments and concerns. Should you have any remaining questions or require further clarification, we would be more than happy to provide additional details before the discussion closes.

Reviewer_nbnzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper converts a pretrained BERT into a spiking network with the same weights. This allows continuous, sparse, and hybrid features to be read from one forward pass, with the timestep count T and leak τ as settings. On fMRI from 49 listeners, alignment plateaus at large T, and the hybrid readout beats a matched random control, with the gain peaking earlier under faster leak.

Critical strengths
  • The same-weight ANN/SNN pair keeps weights fixed and changes only the readout format. This is a clean idea.
  • The authors also show that brain score alone cannot tell real from random.
Critical weaknesses
  • The spike rate is just a 0/1 mask of where the trace is non-zero. Any simple nonlinear feature from the ANN might give the same gain, so spiking may not be needed.

  • The “bounded integration window” may just be the conversion converging to the original ANN. At low T, almost no units fire, and the plateau starts once most units fire. The link to cortical timescales is only descriptive.

  • The effects are very small (about 1e-4 in held-out R²). The result flips with Pearson r², so it depends on the metric. And the metric was chosen post hoc.

Questions for authors
  • The gain may come from any nonlinear feature. Can you add a control that takes the ANN activations and thresholds them into a 0/1 mask, without any spiking, and test a general (not fine-tuned) checkpoint on a second fMRI dataset?
  • The plateau may be a conversion artifact. Can you show it reflects something beyond the trace approaching the ANN values, for example by comparing to the ANN with matched noise or quantization?
  • The effects are tiny and depend on the metric. Can you report them as a fraction of explainable variance, and explain why held-out R² is the right metric when Pearson r² gives the opposite answer?
6 comments
AuthorsOpenReview ↗

Response to Reviewer nbnz (1/4): overview; W1 & Q1

We thank the reviewer for recognizing the motivation and direction of this work, and for noting that the same-weight pair "keeps weights fixed and changes only the readout format". We ran every control the review asks for. Replies 1–4 take them in order, with results and the change in the paper; we will upload one revised PDF after replying to all reviewers.

Your requestOutcomeIn the revision
Q1: ANN activations thresholded to a 0/1 maskAt the peak cells it carries at least as much as the spike mask; the effect follows the fraction of active units§5.2, App. C.1 (R1, R2)
Q1: non-fine-tuned checkpointRise (in every listener) and plateau replicate; no hybrid claim (reply 2)§5.1, App. C.3 (R3, R5)
Q1: second fMRI datasetRise and plateau replicate; hybrid contrast not significant in the first test, significant in a follow-up on new stories§5.1, App. C.4 (R3, R4)
Q2: ANN with matched noise or quantizationNeither reproduces the rise§5.1, App. C.2 (R6)
Q3: fraction of explainable varianceMask contrast 1.2% and 1.0% of the noise ceiling§5.3 (R7)
Q3: why held-out R²; post hoc metricR² scores prediction error; with both metrics fixed in advance, significant under both in 3 of 4 intracranial tests§5.2, App. A.7

The identity is correct and is the design. The gain does not require spikes; for a 0/1 mask it depends on the fraction of active units, which the SNN's integration variables set.

1. The identity is deliberate. The mask is a deterministic function of the trace and adds no information, so any gain from it is a gain of format with the computation held fixed: the sparse readout "records no event that the trace lacks, yet it presents those events in a format that a linear encoder cannot rebuild from the trace" (§3.2).

2. The proposed control. A 0/1 mask thresholded from the parent ANN's activations at the spike mask's density carries at least as much at the peak cells (Table R1); at T10/τ=1.30 the two are equivalent within a ±50% bound set before the test. So the gain does not require spiking, as the paper's framing allows: "the hybrid result concerns representational format rather than an additional source of information" (Limitations).

Table R1. Matched contrast, 49 listeners, language network; all 95% CIs above zero.

ConstructionCellSpike maskANN threshold mask
Paper'sT10/τ=1.30+4.67+5.62
Paper'sT6/τ=1.60+3.80+5.71
Word-free scans neutralizedT10/τ=1.30+3.65+4.09
Word-free scans neutralizedT6/τ=1.60+3.96+5.10

3. A mask's gain depends on density. It depends on how many units are active (Table R2); at density .99, nearly all active, a mask carries nothing detectable. The SNN located this dependence. In the paper, T and τ set the fraction of fired units, and both the plateau's start (§5.1) and the largest contrast (§5.3) fall where it passes one half. We therefore predicted, before the sweep, that an ANN mask would carry most at intermediate density; under held-out R² this holds in both constructions.

Table R2. Matched contrast of ANN threshold masks at fixed density.

Density.05.25.50.75.958.99
Word-free scans neutralized+2.44+4.59+6.06+3.91+2.37+0.27
Paper's construction+0.07+2.28+5.83+4.41−0.54−0.26

The spike mask's density is 0.71 and 0.69 at the peak cells and 0.96 at the anchor cell, hence the weaker anchor contrast (reply 4).

4. Why a same-weight SNN. The paper asks how integration time shapes alignment and readout format. An ANN has no integration time; the converted SNN has two variables for it, T and τ, at fixed weights. They gave the variable the format effect follows, the prediction the sweep confirmed, and the density at which the ANN control was matched; cross-layer dynamics contribute to the integration window itself (reply 3).

In the revision. The abstract and §5.2 state the result as one of readout format with this control; §5.3 adds the sweep; Tables R1–R2 go into new App. C.1.

AuthorsOpenReview ↗

Response to Reviewer nbnz (2/4): Q1, non-fine-tuned checkpoint and second fMRI dataset

We have run non-fine-tuned BERT-base-cased and bert-base-chinese, converted with LAS (standard BERT is not directly compatible with SpikedAttention, §3.1). A second fMRI dataset, LeBel et al. (2023), with the paper's own model. The 35 Chinese listeners of the paper's corpus, a second cohort. Two intracranial datasets, Podcast ECoG and Brain Treebank stereo-EEG.

1. Rise and plateau replicate everywhere (Table R3).

Table R3. Rise of layer-8 alignment from T=4 to T=13 (correlation metric).

DataModelRisePositive
Little Prince fMRI (paper)Paper's model, τ=1.30+0.06749/49
Little Prince fMRIBERT-base-cased, not fine-tuned+0.06549/49
Little Prince fMRI, Chinese cohortbert-base-chinese, not fine-tuned+0.09735/35
LeBel et al. fMRIPaper's model, τ=1.30; 1.60+0.124; +0.0758/8; 8/8
ECoGPaper's model, τ=1.30; 1.60+0.137; +0.0939/9; 9/9
Stereo-EEGPaper's model, τ=1.30; 1.60+0.082; +0.06310/10; 10/10

From T=13 to T=16 alignment changes by at most 4% of the rise in every row. On LeBel et al. the plateau starts at T=10 and T=6 in all eight participants, as in the paper. The shift of the preferred layer replicates on both non-fine-tuned checkpoints (49/49; 35/35).

2. With the paper's model: the hybrid contrast on new data (Table R4).

Table R4. Matched contrast at the peak cells.

DataT10/τ=1.30T6/τ=1.60
Stereo-EEG, 10 participants, after the simulation null+4.6, 8/10, q = .005+7.4, 10/10, q = .002
ECoG, 9 participants, after the simulation null+5.3, 8/9, q = .053+9.4, 7/9, q = .031
LeBel et al., first test: 8 participants, 26 stories+1.6, 4/8, q = .07+2.3, 7/8, q = .05
LeBel et al., follow-up: 3 of these participants, 58 new stories+14.9, +9.5, +18.9; each q < .01+15.8, +16.7, +17.2; each q < .001
  • Intracranial. Held-out R² and a correlation metric were both named before any intracranial hybrid result existed; the contrast is significant under both in three of four tests (ECoG at T10/τ=1.30 is the exception; correlation-metric values in App. C.5).
  • Second fMRI dataset. This was seen after the fact, so we did not treat it as a result. Instead, before downloading any new data, we fixed a new test: three of the eight participants (those with further recordings), their 58 stories unused by the first test, and 2,000 voxels per fold. On these new stories the contrast is significant at both cells in each participant, with 2,000 voxels and also with the original 10,000.

3. Non-fine-tuned checkpoints: no hybrid claim (Table R5). We tested T=11 (the first cell with over half the units firing) and T=13.

Table R5. Matched contrast, LAS (listeners positive; q over two cells).

CheckpointCellPaper's construction (fixed in advance)Word-free scans neutralized
BERT-base-cased (49)T=11+2.0, 28/49, q = .028+0.1, 19/49
T=13−0.2, 25/49+1.4, 30/49, q = .012
bert-base-chinese (35)T=11−1.1, 10/35+1.8, 21/35, q = .004
T=13−23.3, 6/35+4.2, 30/35, q < .001

Both cohorts' recordings end with 31 scans without words. In the paper's construction the standardized mask is extreme there, and for these checkpoints the contrast comes mostly from that one run. Since the outcome depends on this choice and our fixed endpoint was the paper's construction, we make no hybrid claim for non-fine-tuned checkpoints. For the paper's model the choice does not change the sign at the peak cells (Table R1).

In the revision. §5.1 gains Table R3; the Limitations state what holds for non-fine-tuned checkpoints; Tables R3–R5 with per-participant values go into new Appendix C.3–C.5.

AuthorsOpenReview ↗

Response to Reviewer nbnz (3/4): W2 & Q2, is the plateau conversion convergence?

Our answer is yes: it is convergence: the plateau is the converted network recovering its parent's alignment, as the paper states, and we scope its interpretation accordingly. The recovery is, however, not what matched quantization or matched noise gives.

1. The plateau is the parent ANN's level. "The converted SNN therefore needs about ten steps to recover the brain alignment of its parent model, and further steps bring no gain" (§5.1). The question is whether any limited-precision approximation of the ANN would rise the same way.

2. The two proposed controls do not produce the rise (Table R6). Quantized ANN: the conversion's own quantizer applied to each layer's ANN activations separately, at the same T and τ, so no error passes between layers. Noise-matched ANN: Gaussian noise added to the ANN activations with the variance of the SNN's conversion error, unit by unit.

Table R6. Rise of layer-8 alignment from T=4 to T=13 (49 listeners, correlation metric).

Modelτ=1.30τ=1.60
SNN+0.067 [+0.059, +0.075], 49/49+0.042 [+0.037, +0.046], 49/49
ANN quantized layer by layer+0.005+0.0007
ANN with matched noise+0.003+0.002

Layers 4 and 11 give the same picture; across layers 4, 8 and 11 and both leaks, the SNN's rise is 6 to 18 times that of the larger of the two controls.

3. Only the SNN recovers its layers in order. Just before the plateau starts, layer 8 aligns worse than layer 4 in the SNN (−0.008 at τ=1.30, T=8; −0.014 at τ=1.60, T=4). From the plateau on, layer 8 is better (+0.006 to +0.008; 42–46/49 listeners), as in the parent ANN. In both control ANNs layer 8 is above layer 4 at every T. A deep layer that lags just before the plateau and leads after it fits firing that must propagate through the layers; per-layer resolution or noise does not produce it.

So the rise is the converted network recovering its parent's alignment, at a pace to which propagation across layers contributes and which matched resolution or matched noise within each layer does not reproduce.

On the cortical link. The paper labels it descriptive: T and τ "are model-side computational variables and are not a proxy for cortical temporal receptive windows" and "the bounded plateau ... is not a tested cortical claim" (Limitations). The abstract and introduction list it among the findings; the revision deletes that sentence from both, and no result depends on it.

In the revision. §5.1 adds a panel with the SNN, both control ANNs and the parent ANN at layers 4, 8 and 11 for both leaks; Table R6 and the layer comparison go into new Appendix C.2; the cortical-window sentence leaves the abstract and introduction.

Reviewer_23qGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper uses a same-weight ANN-to-SNN conversion framework to study how continuous, sparse, and hybrid representations predict language-related brain activity. It varies internal unfolding duration and leak dynamics while holding the text input fixed. The results suggest that hybrid readouts provide predictive benefits within a restricted model-side regime, although their implications for cortical computation remain uncertain.

Critical strengths
  1. The same-weight framework provides a useful controlled comparison. Extracting different readouts from the same forward computation reduces confounds arising from differences in model architecture, training data, and learned weights.
  2. Random-discrete controls, signed-readout negative controls, and held-out variance analyses demonstrate attention to potential evaluation artifacts. The paper also transparently reports cases where different metrics support different conclusions.
  3. Temporal sweeps and replication using another backbone and conversion method strengthen the investigation. These experiments help assess whether the observed temporal profile depends entirely on one particular conversion implementation.
Critical weaknesses
  1. The sparse representation is a deterministic nonlinear transformation of the continuous trace. Consequently, its benefit to a linear encoder may reflect nonlinear feature expansion rather than evidence for a distinct continuous–sparse coding principle. Random-feature controls do not establish that spiking-derived features are uniquely useful.
  2. The central advantage is strongly dependent on the evaluation metric. Improvements in held-out variance are not consistently accompanied by improved signed correlation, leaving unclear whether the effect reflects better stimulus tracking or differences in prediction calibration. Selecting the principal metric after observing divergent results also makes independent confirmation important.
  3. The proposed correspondence with cortical temporal hierarchy is weaker than the model-level findings. It relies on descriptive relative lifts whose spatial ordering is not reproduced by the controlled contrast. Regional differences could therefore reflect baseline predictability or normalization effects rather than distinct integration timescales.
Questions for authors

Does the hybrid advantage persist against dimension-matched nonlinear transformations of the continuous trace and parent ANN representations?

Does the advantage survive prediction-calibration controls and evaluation on independent data using pre-specified complementary metrics?

Does the controlled hybrid advantage systematically track independently established cortical timescales after accounting for regional baseline performance and measurement reliability?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 23qG (1/3): overview; W1 & Q1 (nonlinear feature expansion?)

We thank the reviewer for recognizing the motivation and direction of this work, and for noting that the same-weight framework "provides a useful controlled comparison". We ran the tests that Q1 and Q2 name. Replies 1–3 give results and changes in the paper; we will upload one revised PDF after replying to all reviewers.

Your questionWhat we didOutcomeIn the revision
Q1: dimension-matched nonlinear transforms of trace and parent ANNThree readouts appended to the trace, two to the parent ANNAt the peak cells the other trace readouts carry at least as much as the spike mask; for a 0/1 mask the gain depends on the fraction of active units§5.2–5.3, App. C.1 (G1)
Q2: calibration; independent data; pre-specified metricsTwo intracranial datasets, a second fMRI dataset; explained variance and a simulation nullSignificant under both metrics in 3 of 4 intracranial tests; under held-out R², a follow-up on new stories significant in each participant§5.2, App. A.7, C.4–C.5 (G2, G3)
Q3: does the controlled advantage track cortical timescales?Replications of the model-level findingsNot claimed or tested; the descriptive correspondence leaves abstract and introductionAbstract, §1, §5.1 (G4)

W1 & Q1

The deterministic relation is correct and is the design. The gain does not require spikes: at the peak cells, dimension-matched readouts of the trace carry at least as much as the spike mask. For a 0/1 mask, the gain depends on the fraction of active units, set by the SNN's integration variables: steps T and leak τ.

1. The relation is deliberate. The mask adds no information to the trace, so any gain from it is a gain of format with the computation held fixed: the sparse readout "records no event that the trace lacks, yet it presents those events in a format that a linear encoder cannot rebuild from the trace" (§3.2). The paper lists capacity-only baselines as untested (Limitations); we have now run them.

2. Dimension-matched readouts of the trace (Table G1). We appended three readouts with the trace's dimension: a 0/1 mask thresholded from the parent ANN's activations at the spike mask's density, the magnitude |c|, and random ReLU features of c. Each carries a positive contrast at both peak cells, at least as large as the spike mask's, also in a second feature construction that neutralizes scans without words. At T10/τ=1.30 the ANN mask and the spike mask are equivalent within a ±50% bound set before the test. This fits the paper's framing: "the hybrid result concerns representational format rather than an additional source of information" (Limitations).

Table G1. Matched contrast before the simulation null, 49 listeners, language network, paper's construction; all 95% CIs above zero.

CellSpike maskANN threshold maskMagnitude of cRandom ReLU of c
T10/τ=1.30+4.67+5.62+7.01+5.64
T6/τ=1.60+3.80+5.71+6.28+5.74

3. Parent ANN. After the simulation null, the same holds without conversion: appended to the parent ANN's activations a, |a| and random ReLU features of a carry +6.2 and +6.8, and the same readouts of c carry +6.6 and +7.9 at T10/τ=1.30.

4. Why a same-weight SNN. The paper asks how integration time shapes alignment and readout format. An ANN has no integration time; the converted SNN has two variables for it, T and τ, at fixed weights. They set the fraction of fired units, and both the plateau's start (§5.1) and the largest contrast (§5.3) fall where it passes one half. We therefore predicted, before the test, that ANN masks would carry most at intermediate density. Under held-out R² they do: +0.07 at density .05, +5.83 at .50, −0.26 at .99, with the maximum also at .50 in the second construction. So the SNN identified the variable a mask's gain depends on and supplied the density at which the ANN control was matched.

In the revision. The abstract and §5.2 state the result as one of readout format with these controls; §5.3 adds the density sweep; these results replace the Limitations sentence on capacity-only baselines; Table G1 (both constructions) and the parent-ANN values go into new App. C.1.

AuthorsOpenReview ↗

Response to Reviewer 23qG (2/3): W2 & Q2 (metric, calibration, independent confirmation)

The reviewer points out that the advantage depends on the metric: gains in held-out variance are not always matched by gains in signed correlation, so the effect could reflect prediction calibration rather than stimulus tracking. Because the principal metric was chosen after the divergence was seen, the reviewer asks for independent confirmation with pre-specified complementary metrics. We ran that confirmation. On two independent intracranial datasets, with held-out R² and a correlation metric both fixed in advance, the contrast is significant under both in three of four tests. A correlation metric does not depend on the scale or offset of a prediction, so a contrast that exceeds the simulation null under this metric reflects tracking.

1. Independent data, two complementary metrics fixed in advance (Table G2). We used Podcast electrocorticography (ECoG, 9 participants) and Brain Treebank stereo-EEG (10 participants) with the paper's model at the peak cells. Held-out R² and the correlation metric were named before any hybrid result on these data existed. The contrast is significant under both metrics at both peak cells on stereo-EEG and at T6/τ=1.60 on ECoG, before and after the simulation null is subtracted; ECoG at T10/τ=1.30 is the exception.

Table G2. Matched contrast after the simulation null: mean [95% CI], participants positive, q over the two cells (exact sign-flip test).

Dataset, cellHeld-out R² (×10⁻⁴)Correlation metric (×10⁻³)
Stereo-EEG, T10/τ=1.30+4.6 [+2.2, +6.8], 8/10, q = .005+1.6 [+0.9, +2.3], 8/10, q = .008
Stereo-EEG, T6/τ=1.60+7.4 [+4.9, +10.4], 10/10, q = .002+1.8 [+0.6, +2.8], 9/10, q = .009
ECoG, T10/τ=1.30+5.3 [−0.7, +9.6], 8/9, q = .053+1.6 [−0.6, +3.7], 7/9, q = .107
ECoG, T6/τ=1.60+9.4 [+2.8, +15.0], 7/9, q = .031+1.9 [+0.5, +3.0], 8/9, q = .031

2. A second fMRI dataset (Table G3). On LeBel et al. (2023; a different stimulus), the first test used the paper's model with all choices fixed beforehand and was positive in 7/8 participants at T6/τ=1.60. Its contrast grew when fewer, better-predicted voxels were used; since that was seen after the fact, we did not report it as a result but fixed a new test on it before downloading new data: three of the eight participants (those with further recordings), their 58 stories unused by the first test, and 2,000 voxels per fold instead of 10,000. Under held-out R² it is significant at both cells in each participant, also with the original 10,000 voxels (+7.3 to +12.4), and under the correlation metric it is positive too, in all three participants at both cells.

Table G3. LeBel et al., matched contrast under held-out R² (participants positive or per-participant values; q over the tests in each row).

TestT10/τ=1.30T6/τ=1.60
First test: 8 participants, 26 stories+1.6, 4/8, q = .21+2.3, 7/8, q = .07
Follow-up: 3 of these participants, 58 new stories+14.9, +9.5, +18.9; each q < .01+15.8, +16.7, +17.2; each q < .001

3. Controls on the paper's fMRI data: offset and fitting.

  • Offset. Explained variance discards each voxel's mean prediction error. Under it the contrast is +4.62 and +3.75 at the two peak cells, against +4.67 and +3.80 under held-out R².
  • Fitting without content. We simulated responses that depend only linearly on c, so that a contrast can come only from how the appended mask changes the fit. This simulation null gives +1.14 and +0.60. The observed contrast exceeds it by +3.5 and +3.2 (95% CIs above zero, also in the second construction).

On these data we state the hybrid result under held-out R², as the paper does; the evidence under a correlation metric is that of points 1 and 2.

4. Why held-out R², and its selection. Held-out R² scores the prediction error directly, the quantity the encoder is fitted to minimize. The paper states that it was selected "only after observing an empirical inversion under a Pearson² proxy" (App. A.7); the review credits the paper's report of that divergence as transparent. A metric chosen after seeing a divergence calls for the check in point 1: new data, both metrics fixed beforehand.

In the revision. §5.2 adds the tests on independent data under both metrics; Table G2 goes into new App. C.5 and Table G3 into App. C.4; App. A.7 adds the two calibration controls; every hybrid result names its metric.

AuthorsOpenReview ↗

Response to Reviewer 23qG (3/3): W3 & Q3 (correspondence with the cortical temporal hierarchy)

The reviewer finds the correspondence with the cortical temporal hierarchy weaker than the model-level findings: it rests on descriptive relative lifts whose spatial ordering the controlled contrast does not reproduce, so regional differences could reflect baseline predictability or normalization. The question is whether the controlled advantage tracks independently established cortical timescales. The paper does not claim that it does. It reports the correspondence as a description of the uncontrolled lift, and the observation that the controlled contrast does not reproduce the ordering is the paper's own (App. A.10). The revision takes the correspondence out of the abstract and introduction. The model-level findings do not depend on it, and they now replicate well beyond the original data.

1. What the paper states. The map of the relative lift is "descriptively consistent" with the temporal-receptive-window hierarchy, and "we report it as a correspondence of the uncontrolled lift" (§5.3). "The parcel ranking of the controlled contrast does not follow the ranking of the relative lift in Fig. 5a (Spearman ρ = −0.49 across the 12 parcels ...), which is a further reason to read that map as a description of the uncontrolled lift" (App. A.10). T and τ "are model-side computational variables and are not a proxy for cortical temporal receptive windows" (Limitations).

2. The alternative reading. The relative lift is a ratio to each parcel's trace-only score, so baseline predictability can shape its map, as the review notes. The descriptive map cannot exclude this, and no result of the paper is built on that map. What the controlled contrast does show across parcels is the model-side timing: under held-out R² it peaks at a smaller T under the faster leak, τ=1.60, in all twelve parcels (App. A.10). A test of the kind Q3 describes, against independently measured timescales with baseline and reliability accounted for, has not been run; the revision lists it as future work.

3. The change. The abstract and the introduction each list the correspondence among the findings. The revision deletes it from both, and §5.3 and the Limitations keep their statement that the map is descriptive.

4. The model-level findings replicate more widely (Table G4). The review rates these as the stronger part and lists "replication using another backbone and conversion method" as a strength. The backbone and the conversion are now varied separately, and the rise and plateau are tested on a second cohort, a second fMRI dataset and two intracranial datasets with millisecond resolution.

Table G4. Rise of layer-8 alignment from T=4 to T=13 (correlation metric).

DataModelRisePositive
Little Prince fMRI (paper)Paper's model, τ=1.30+0.06749/49
Little Prince fMRIPaper's backbone, second conversion (LAS)+0.05248/49
Little Prince fMRIBERT-base-cased, not fine-tuned, LAS+0.06549/49
Little Prince fMRI, Chinese cohortbert-base-chinese, not fine-tuned, LAS+0.09735/35
LeBel et al. fMRIPaper's model, τ=1.30; 1.60+0.124; +0.0758/8; 8/8
Podcast ECoGPaper's model, τ=1.30; 1.60+0.137; +0.0939/9; 9/9
Brain Treebank stereo-EEGPaper's model, τ=1.30; 1.60+0.082; +0.06310/10; 10/10

From T=13 to T=16 alignment changes by at most 4% of the rise in every row. Quantized and noise-matched versions of the parent ANN do not reproduce the rise (+0.005 and +0.003 at τ=1.30, against +0.067).

In the revision. The sentence on cortical temporal receptive windows leaves the abstract and introduction; §5.1 gains Table G4, with per-participant values in new App. C.2–C.5; Future work names the test of Q3.

We thank the reviewer again. Answering the three questions separates the paper's claims more clearly: a model-level rise and plateau that replicate across conversions, backbones and recording methods; a hybrid result stated as one of readout format, now tested against the baselines the paper had listed as untested, and significant under two metrics fixed in advance in three of four independent intracranial tests; and a cortical correspondence that the paper describes and no longer lists among its findings. The revision also extends the table of symbols (App. A.1) with the terms of reply 1 and names the metric with every result, to make the paper easier to read.

AuthorsOpenReview ↗

Response to Reviewer 23qG

We sincerely appreciate your time and constructive feedback. With the discussion period approaching its end, we would like to gently follow up and see whether our rebuttal has sufficiently addressed your comments and concerns. Should you have any remaining questions or require further clarification, we would be more than happy to provide additional details before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of our replies and the revised PDF

We thank the three reviewers for their careful reviews, for recognizing the motivation of this work and the novelty of the same-weight ANN–SNN framework, and for their constructive and helpful comments. Their requests led to new controls and new data, and our replies give the results point by point. In brief:

  • Readout format. The hybrid advantage is a gain of readout format: a density-matched threshold mask of the parent ANN carries it too, and its size depends on the fraction of active units, which the integration variables T and τ set.
  • Bounded plateau. The plateau is the converted network recovering its parent's alignment. Quantized and noise-matched versions of the parent ANN do not reproduce the rise. The rise and the plateau replicate across conversions, backbones, a second fMRI dataset and two intracranial datasets.
  • Metrics. On independent intracranial data, with held-out R² and a correlation metric fixed in advance, the hybrid contrast is significant under both in three of four tests.
  • Cortical timescales. The correspondence with cortical temporal receptive windows is descriptive and is no longer listed among the findings.

We have uploaded a revised PDF with the changes promised in our replies. The "In the revision" paragraph at the end of each point in our replies lists them.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.