- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a Bayesian method that aligns neural spike trains across trials. The key idea is to assume that each neuron has a template describing its typical response over time, which is warped differently across trials. The warping function is shared across all neurons, but the authors also fit each neuron separately to allow different alignments. The template function has a Gaussian process (GP) prior followed by a softplus transformation, while the warping function is assumed to be monotonic and piecewise linear, with segment lengths generated by applying a softmax transformation to Gaussian variables.
The posterior distributions of the template and warping functions are inferred using Hamiltonian Monte Carlo (HMC) within Gibbs sampling. The conditional independence structure allows parallel sampling, and the authors also use parallel tempering to explore different possible solutions. Finally, the warping complexity is selected using Pareto-smoothed importance sampling leave-one-out cross-validation (PSIS-LOO).
On synthetic data, the authors introduce a measure called "warp swap dissimilarity" to cluster neurons with similar warping functions. On rat orbitofrontal cortex (OFC) data, they find that most individual neurons prefer simple shift warps, while the full population prefers more complex piecewise-linear warps.
Critical strengths
This paper addresses an important problem in neural data analysis. Differences in neural response timing may contain useful information about the underlying neural circuits. Estimating these differences can help align neural responses across trials and improve our understanding of neural dynamics.
The proposed method successfully recovers the underlying template and warping functions on synthetic data, as shown in Figure 2.
The proposed warp swap dissimilarity measure works well for identifying groups of neurons with similar warping functions on synthetic data (Figure 3). It achieves better clustering performance than directly comparing the posterior means of the warping function
Critical weaknesses
In the rat decision-making experiment, the authors use PSIS-LOO to estimate model performance when leaving out one trial at a time. However, the warping function is trial-specific. When a whole trial is left out, the remaining trials provide no direct information about its warping function. It is therefore unclear how the method predicts the neural activity on the held-out trial. It is possible that the prior distribution of warping function would carry cross trial information that can be used by the method, but that does not seem to be clear to me.
In the evaluation on the rat decision-making data, the authors state that they use the trial-averaged neural activity from even trials as a template and generate predictions for odd trials. However, it is unclear how the warping functions for the odd trials are obtained. Do the authors first fit the model using all trials, including the odd trials, to estimate their warping functions before evaluating the model?
The comparison between different models and fitting methods in Figure 3E is interesting. In particular, the improvement in clustering performance when using the Bayesian method instead of a point estimate is promising. However, I have a few concerns: 1)The comparison with Williams et al. (2020) needs more details. What template functions were used in the baseline model? Was regularization included, and how were the regularization parameters selected? 2)The posterior-mean warping function already achieves good clustering performance. Although the proposed "warp swap dissimilarity" further improves performance, it is unclear how important this improvement is in practice. It would be interesting to test more challenging scenarios where the posterior-mean method performs poorly but using the full posterior distribution provides a clear advantage. This would help demonstrate when modeling posterior uncertainty is necessary.
The model includes neuron-specific template functions and trial-specific warping functions, which introduce many free parameters. When applying such a flexible model to neural recordings, an important question is whether the available data are sufficient to reliably estimate these parameters. It would be helpful to investigate how the estimation accuracy depends on the number of recorded neurons and trials, as well as the number of spikes per neuron. For example, the authors could perform simulations with different numbers of neurons and trials to study parameter recovery and uncertainty. This analysis would also help us understand the bias-variance tradeoff of the model and identify the conditions under which the proposed method works reliably.
Questions for authors
1.How is PSIS-LOO used for single-neuron model selection in Figure 4? In particular, how are the trial-specific warping functions handled when a whole trial is held out?
2.In the evaluation shown in Figure 4, how are the neural activities from odd trials used during model fitting? Are the odd trials used to estimate their own warping functions before evaluating predictive performance?
-
Could the authors provide a more challenging example to demonstrate the advantage of using the full posterior distribution of warping functions over their posterior means, beyond the results in Figure 3E?
-
How much neural recording data is needed to reliably estimate the model parameters? In particular, how does estimation accuracy depend on the number of recorded neurons, trials, and spikes per neuron?
5 comments
Response to Reviewer gatx [2 / 2]
Could the authors provide a more challenging example to demonstrate the advantage of using the full posterior distribution of warping functions over their posterior means, beyond the results in Figure 3E?
This is a great question. If the posterior is unimodal, we envision that the posterior distribution can likely be well-summarized by the mean of the posterior samples. However, if the posterior exhibits multimodality, the mean can be a poor summarization of the posterior. As an example, Figure S3 demonstrates the substantial multimodality observed in the warping functions when the model is fit to the OFC data. In this setting, the posterior mean of the warping functions is very low probability under the posterior and resembles none of the drawn samples, which would be an incorrect characterization of the warping functions relevant to that trial. We revised Figure S3 to include a visualization of the posterior mean compared to the samples from the posterior to highlight this effect. This means that we expect clustering on the posterior means to be appropriate in cases with well-behaved, unimodal posteriors (which is why it can achieve a max ARI of 1.0 on some of the synthetic subpopulation datasets), but will struggle in multimodal posteriors. The degree of warping multimodality present in all experimental datasets is an empirical question out-of-scope for this work, but we present the warp swap dissimilarity as a possible clustering approach which is designed to be robust to multimodality by directly using samples from the posterior. Outside of clustering, we highlight that the full posterior allows for us to propagate alignment uncertainty into downstream analyses via variance-tagged rasters (Figure 3A, middle) or posterior-smoothed spike counts (Figure 3A, bottom). Future work will assess if incorporating this uncertainty provides advantages for downstream tasks (such as encoding/decoding analyses) relative to only using the posterior mean.
In particular, how does estimation accuracy depend on the number of recorded neurons, trials, and spikes per neuron?
We completely agree that characterizing the bias-variance tradeoff and reliability of flexible statistical models is very relevant for practical applicability to neuroscience. To our knowledge, we are the first to rigorously tackle this question for time warping models. We highlight Section 3.1 and Figure 2B-C, which emphasize coverage as the relevant quantity of interest when fitting a Bayesian model, rather than simply accurate estimation of the ground truth parameters. We show that, even when shrinking the number of trials by 10x, the template estimates still exhibit ideal coverage behavior by widening the posterior (Figure 2B-C, left). Similarly, even when shrinking the number of neurons by 10x, the warping estimates also exhibit ideal coverage behavior by widening the posterior (Figure 2B-C, right). Tracking the dependence on signal-to-noise ratio is a great suggestion also brought up by reviewer yM9r. Rather than firing rate, a more appropriate measure of signal-to-noise ratio for our model is given by the output scale of the Gaussian process. Intuitively, this quantity determines the dynamic range of a neuron's template - at lower output scales, there is less variance in the neuron's firing rate, which means there is less ability to distinguish different timepoints in the template. This naturally makes the templates and warping functions less identifiable. Qualitatively, we observe appropriate widening of the posterior and increased prior influence as the output scale decreases in synthetic data. We include a new figure tracking the inferred posterior over template and warping functions for an example neuron/trial as we vary the output scale of a synthetic dataset (Figure S4).
I appreciate the authors’ clarifications on how they perform cross-validation on single-trial level and how the odd-trial data is used. However, I feel the answers to the bias-variance question are not sufficient. Could the author vary systematically the number of neurons/trials included to see how their method based on full posterior distribution outperform the methods that are only based on posterior means?
Given the fact that it is probably not possible to use this method to predict how a new trial is warped. I wonder whether this method can shed any new light on the mechanism underlying the neuroscience process. For example, in the OFC data the author analyzed, when using this method to identify individual neurons’ warping, is it possible to use this heterogeneity to gain more insights on what are the task-relevant signals and how this brain region perform this task?
Response to Reviewer gatx [1 / 2]
Thank you for your thoughtful review of our work.
We apologize for an oversight in the original draft of the paper, which naively applied the same PSIS-LOO model selection procedure for the population model and the single-neuron model. As you correctly point out, leaving out (neuron, trial)-tuples is an ill-defined procedure in the single-neuron case because the model is unable to predict data on entirely unseen trials given only data from a single neuron. Therefore, to perform model selection at the single-neuron level, we instead leave out (timebin, trial)-tuples. This LOO quantity is slightly optimistic due to non-negligible autocorrelation between adjacent timebins, but it allows us to perform model selection at the single-neuron level. With this revised LOO procedure, there was a shift in the optimal complexity histogram towards more complex warps (Figure 4C). However, the majority of neurons still preferred a shift (31/48 neurons) and none of the chosen example neurons had their optimal complexity change under the new procedure. Therefore, the qualitative conclusions of Section 3.3 still hold. The new complexity distribution also makes the trial-split log-likelihood result more significant (new p-value of 0.00977). The controls did not have their optimal complexity histograms change with the new procedure (Figure S6). We have revised the text to include the single-neuron PSIS-LOO procedure, and we have explicitly noted the limitation regarding the autocorrelation of timebins.
We compute a split-trial Poisson log-likelihood as a metric to compare the raw data, the population fits, and the single-neuron fits. The procedure is as follows: we first fit the model to all trials, take the aligned trial average of the even trials as a pseudo-template, pass this pseudo-template through the warps learned for the odd trials, and compute the Poisson log-likelihood of the odd trial data under that model. As your questions suggest, this is not a held-out measure of predictive performance, as we use the odd trial data to learn the necessary warping functions. Rather, it quantifies how well the aligned (or raw) trial average explains single-trial activity. In other words, we are computing how similar the activity looks on single trials compared to the trial-average. We view this as an extension of the approach used in Figure 2F-H of Williams et al. (2020) to Poisson data along with extra robustness conferred by using the trial average of the even trials to explain the observed activity on odd trials. We have added further clarification of the approach to the manuscript on lines 478-480 and 1157-1159.
The cross-validation procedure for hyperparameter tuning introduced in Williams et al. (2020) is not feasible for per-neuron fitting, as it relies on holding out entire trials. In order to select reasonable hyperparameters for the affine warping model, we performed 3-fold cross-validation when fitting to the entire population and fix these hyperparameters for all single-neuron fits. We saw qualitative examples where the affine warping model was able to identify warping functions close to the data-generating warping function when fit to single neurons (Figure S5C). We also found examples where the affine warping model did not identify warping functions close to the data-generating warping function (Figure S5B). Notably, in these cases, the posterior distribution identified by Bayesian time warping often provided reasonable coverage of both the data-generating warping function as well as the MLE identified by affine warping (Figure S5B). Critically, affine warping could still support clustering if its errors were correlated within a subpopulation which shared a warping function. However, the quantitative (Figure 3E) and qualitative data (Figure S5A-B) suggested that its errors were not correlated within a subpopulation. We have expanded Appendix B.2 (lines 1105-1110) to provide more details regarding the affine warping fitting procedure and Section 3.2 (lines 410-415) to clarify the shortcomings of affine warping for clustering.