- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper describes a dataset, FILM, which provides neural features from chronic flexible-electrode recordings paired with cursor kinematics. The data comprises 178 sessions from two non-human primates, who were implanted with 512 acquisition channels. The data were recorded over 32 to 277 days. The study suggests that decodability remains relatively high within the same day (R^2 ~0.45 to ~0.95, selected to be >=0.3) but decreases drastically across days without recalibration. The authors also discuss how much recalibration helps. The paper is written well, and it is clear that time and effort has been put into creating it. The authors also used various methods to analyze the dataset they collected. Though it also has various weaknesses.
Critical strengths
- The paper is well written. There is attention to detail. The figures generally make sense.
- Good introduction overall (though see a few comments in Weaknesses).
- The dataset it presents appears valuable. It includes 111 sessions over 277 days, which is a long coverage for flexible electrodes in primates. The paper also includes both the spiking-band power and threshold-crossing rate in the standard NWB format, which makes it valuable and easy to use.
- The authors are careful to do a proper train-test split for the decoding.
- They also compare recalibration across sessions with and without historical weights.
- Their limitations section is sincere and useful.
Critical weaknesses
- The authors need to do a better job explaining exactly what advantages flexible electrodes have over rigid ones. They do have advantages in terms of scarring, single-neuron tracking, and high channel density at depth. But why is decoding stability with flexible electrodes a separate issue from that with rigid electrodes, such as Utah arrays? What kind of result would distinguish the two? For example, the single-neuron tracking is not relevant in this paper, as no spike sorting has been carried out on these data. The authors mention this in the limitations, but it still limits the value of the dataset.
- Though recording from different brain regions in MB was a good idea, two of the recording sites (outside PMd) were different between MA and MB. This makes interpreting those data more difficult. It would have been better to record from the same regions in MA and MB. The long-term claims rely only on MA, while MB was only recorded for 23 sessions over 32 days.
- Quite a bit of text is required to explain the dataset and how to interpret it, which does not work with a 9-page limit for an ICLR paper. That means that to understand the dataset and its value the reader needs to go through all 31 pages of text, with 22 of those 31 pages (71%) in appendices. This manuscript is therefore better suited for a journal article than a conference paper.
- The authors note that sessions were kept only when both WF and GRU had R^2>=0.3, within-session cross validated. However, the authors do not report statistics on how many sessions were discarded, where they took place in the lust of sessions, etc. And the session omission procedure is described only in an appendix (C.1). This is problematic for a dataset meant to characterize long-term recording stability. The excluded sessions should also be released, so that anyone using the dataset could inspect them.
- ICLR is an ML conference. The authors suggest their dataset as a benchmark for decoding and domain adaptation. But their only baselines are WF and GRU, with recalibration tested only on WF. The authors should do much more to make this database valuable for the ICLR audience. By way of example, the FALCON dataset they cite include many more baselines, from humans to primates to birds.
- The overall upshot of the paper is not clear. The authors do not have a specific description of their contributions, which is uncommon in ICLR papers. The authors propose it as a dataset with some conclusions about within- and across-day results. But those are neither very surprising nor seem different from results with rigid electrode. Perhaps a comparison with studies of rigid electrodes would make this one more valuable. Put differently, the main findings are well established for Utah arrays, including good within-day decoding, declining cross-day transfer, recalibration helping, and historical weights helping most with little data. The novelty in this piece therefore needs to come from the electrode type. But there is no comparison across electrodes. The paper cites LINK. Running the same frozen-transfer and recalibration protocol there would be valuable in testing whether flexible electrodes behave differently.
Some minor issues:
- Unless SBP was carried out only in PMd and TCR only in M1, it is confusing to use the same color for PMd and M1 distinction in Fig. 1A and between SBP and TCR in Fig. 1B.
- In Fig. 1C, Vy at trial end seems to be >200 px/s, while Vx=0 exactly, which is strange. If the figure ends before the trial does, indicate that.
Questions for authors
You can find my questions in the Weaknesses section.
9 comments
Response to Weakness 2
Thank you for this comment. We agree that the different recording sites in MA and MB make direct between-subject comparisons difficult, and that MB’s 23 sessions over 32 days do not support the same long-term conclusions as MA’s 111 sessions over 277 days.
Our long-term cross-day results are based on MA_CO, recorded from PMd, and do not rely on comparisons with MB. Within each subject and recording subset, we evaluate decoding across days using the same implanted array and channel identities. MB provides a shorter-term evaluation in a second subject, over 23 sessions spanning 32 days. Its recordings from PMd, PMv, and M1 also allow exploratory analyses across those regions, but differences in recording sites prevent us from interpreting these as controlled comparisons between MA and MB.
We will revise the manuscript to distinguish MA's long-term evidence from MB's shorter-term and exploratory contributions, and to present cross-subject transfer between MA and MB as a separate analysis rather than as evidence for the long-term findings.
Response to Weakness 3
Thank you for this comment. We agree that the main paper should communicate FILM’s contribution without requiring readers to work through a lengthy appendix. We will revise the main text to give a concise description of the dataset, explain the questions it enables, and present the key results that establish its value.
We will also review the appendices for material that can be shortened or consolidated, especially explanations that repeat the main text. The remaining appendices will provide the detail needed to assess and reproduce the work: dataset acquisition and processing methods, along with supplementary analyses and results. We will clarify their contents and point readers to the relevant sections from the main text. This should make the paper easier to follow within the ICLR page limit while retaining the supporting detail.
Response to Weakness 1
Thank you for this comment. We agree that our study does not assess single-neuron tracking because we did not perform spike sorting. We used unsorted channel-level activity because it can retain movement information relevant to motor BCIs. In chronic recordings, changes in spike waveforms can make individual units harder to isolate and track. Events that still cross the detection threshold contribute to the channel's threshold-crossing rate (TCR), even when they cannot be reliably assigned to a single neuron. Prior work has shown that unsorted threshold crossings can support online cursor control. Longitudinal recordings have also shown that threshold-based decoding can remain effective despite declines in spike amplitude.
A practical question for both flexible-electrode and Utah-array recordings is how well a fixed decoder can use these channel-level signals to predict cursor velocity on later days. A larger pool of task-related channels may help preserve decoding performance if some channels become less informative over time. To examine this possibility, we compared FILM and LINK in terms of the number of task-related channels, the stability of their tuning across days, and the performance retained by a fixed decoder (Fig. 4 in the revised manuscript).
FILM (MA_CO) has a larger task-related channel pool, with a median of 167 of 512 recorded channels (32.6%), compared with 16 of 96 (16.7%) in LINK (Fig. 4A). Within FILM, cross-day decoding performance improves as more task-related channels are included. At a fixed input size of 96 channels, selecting channels by task relevance also outperforms random selection (Fig. 4B). These results show that the additional task-related channels provide useful information for cross-day decoding.
To compare tuning stability at the same selected channel count, we examined the 15 most task-relevant channels in each dataset. Preferred direction and modulation depth were more stable in the selected FILM channels than in the selected LINK channels over extended intervals (Fig. 4C).
In the full-channel comparison, we trained a WF decoder on one day and applied it to later days without updating its weights. Using the same protocol and TCR features in both datasets, FILM achieved higher cross-day decoding R2 than LINK and retained a larger fraction of its training-day performance. This fraction also declined more gradually in FILM as the train–test interval increased (Fig. 4D).
For practical BCIs, these results motivate a concrete goal of extending the time over which a fixed decoder maintains useful performance without recalibration. FILM makes it possible to test whether selecting and combining task-related channels with stable movement tuning can extend that period. Its longitudinal recordings and evaluation protocol provide a basis for measuring progress toward long-term “plug-and-play” BCI decoding. We will revise the manuscript to make this connection explicit.