- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
They suggests a VAE for neural population recordings with a latent transition drift, named SyLVER, modelled with a sparse linear combination of polynomials. To do the inference part they used structured filtering with a per-bin MLP and a reverse-time GRU that produce Gaussian potentials. The posterior marginals for the inference come from a moment-matched predict step through the sparse drift. They evaluated the method on van der Pol, pendulum, and Lorenz system, and on the MC-Maze motor cortex dataset, where the authors report decodable velocity and probe the learned dynamics.
Critical strengths
Well-motivated problem, I agree with that the two-stage pipelines can inherit errors of the inference step. Placing the sparse drift inside the structured filter is a clear idea and can shapes the predictive distribution and receives gradient through the ELBO. This makes sense and clearly explained design. Also honest reporting of failure modes where in they mention that Lorenz recovery fails, this is informative even though, as noted below it undercuts the paper’s claims.
Critical weaknesses
1- Table 1 reports single runs with no error bars while the numbers are very close, and SyLVER does not beat SINDy-AE on forecasting. On MC-Maze, no standard neural latents benchmark metrics like co-bps/etc or competing methods are reported, even though the paper says the dataset was chosen for this kind of comparison and make sense to use those metrics too and report them, see table 1 of (Pei, 2021).
2- As presented here the main claims are not supported by the results. The abstract and discussion say SyLVER recovers equations but in the Lorenz experiment it misses most of the true terms (Table 2). The van der Pol and pendulum, recover the right coefficient and they use a decoder fixed to the true readout. That removes the core difficulty the paper is meant to solve, which is learning sparse dynamics through a learned observation model. The introduction and discussion claim sparse equations, but Section 4.2 and the table 2 says the learned equation is dense ( see van der Pol example) and clearly wrong for Lorenz (missing dynamics for the x), can you clarify what is happening here? as I understand here the key claim is contradicted by the results as as shown here.
3- Some important baselines are missing. The paper argues that joint learning beats a two-stage pipeline, but never compares against one of them (for example, XFADS latents followed by SINDy). It also does not compare against SVISE (you put this as one of the closest models to yours in the discussion but never compared against or being used as a baseline), which as I understand has the closest prior method. Also, none of the proposed components (block optimization or masked-bin training) is ablated.
Pei, F., Ye, J., Zoltowski, D.M., Wu, A., Chowdhury, R.H., Sohn, H., O'Doherty, J.E., Shenoy, K.V., Kaufman, M.T., Churchland, M.M., Jazayeri, M., Miller, L.E., Pillow, J.W., Park, I.M., Dyer, E.L., & Pandarinath, C. (2021). Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity. ArXiv, abs/2109.04463.
Questions for authors
1- The text reports 0.53 for ridge on spikes, while the Figure 3 caption reports 0.76. What does the latent model add If ridge results on raw spikes beats the MLP on latents (0.74)? can you clarify this
2- Table 3 says Lorenz uses Poisson observations, but Figure 2 and Table 1 say Gaussian. Which was used?
3- How SyLVER performs on van der Pol and pendulum with a learned decoder instead of the true readout if we use the same alignment as SINDy-AE does? I am asking this since without this the recovery results don’t test the method’s main claim.
4- How SyLVER compare with a two-stage baseline or with SVISE when the observation map is known?
5 comments
Response to Reviewer yj8Q (2/2)
4. Seed spreads and NLB metrics
All synthetic results, baselines and ablations will be reported over three seeds with mean ± standard deviation (see our response to Reviewer k6du, §3 and §5). Table 1 is intended as a parity baseline, and we agree that the spreads are needed to support even that. On MC-Maze, we will report the standard Neural Latents Benchmark metrics (co-bps, velocity , PSTH , forward prediction) against published results, following Table 1 of Pei et al. (2021).
5. Decoding numbers on MC-Maze (Q1)
We apologize for the inconsistency. The 0.53 and 0.76 values came from different decoding setups and were not reported consistently. We have since rerun MC-Maze with eight latent dimensions and a cubic dictionary, and evaluated decoding with the NLB protocol (ridge regression from inferred rates). Hand velocity decodes from inferred rates at , and from the rates of the learned-equation rollout started just after movement onset at . Setting drops the forecast to , so the forecast is carried by the learned equation. What the latent model adds over a decoder on the spikes is therefore a compact latent state with an explicit equation that forecasts the dynamics. We will report the NLB spike-smoothing baseline under the same protocol so that the comparison is direct (see our response to Reviewer msAG, §4, for the full updated results).
6. Observation model on Lorenz (Q2)
All synthetic results in Table 1, Table 2 and Figure 2 use Gaussian observations. Our data generator produces both Gaussian and Poisson variants of each system from the same latent trajectories, and the Poisson entry for Lorenz in Table 3 refers to that variant; it is an error in the table. We will correct it and state the observation model in every table and figure caption. We use Gaussian observations for the synthetic benchmarks because this is the setting in which the baselines are typically evaluated, and Poisson observations for spike counts.
To summarize, we will: consolidate the results so that every table and figure states its setting consistently; report the updated fixed-decoder Lorenz results; report all synthetic experiments with fixed and learned decoders, aligned as for SINDy-AE and through an exact affine transformation; add the two-stage, SVISE and matched SINDy-AE baselines and the three component ablations over three seeds; report NLB metrics on MC-Maze; and revise the claims to match the evidence. We are confident that this plan addresses the reviewer's concerns about soundness. We would welcome the reviewer's view on whether it does, whether there are other comparisons they would find convincing, and whether we should prioritize anything differently.
References
Champion, K., Lusch, B., Kutz, J. N., & Brunton, S. L. (2019). Data-driven discovery of coordinates and governing equations. Proceedings of the National Academy of Sciences, 116(45), 22445–22451.
Course, K., & Nair, P. B. (2023). State estimation of a physical system with unknown governing equations. Nature, 622(7982), 261–267.
Gao, L. M., & Kutz, J. N. (2022). Bayesian autoencoders for data-driven discovery of coordinates, governing equations and fundamental constants. arXiv preprint arXiv:2211.10575.
Pei, F., Ye, J., Zoltowski, D., Wu, A., Chowdhury, R. H., Sohn, H., O'Doherty, J. E., Shenoy, K. V., Kaufman, M. T., Churchland, M., et al. (2021). Neural Latents Benchmark '21: Evaluating latent variable models of neural population activity. arXiv preprint arXiv:2109.04463.
Thank you for the detailed rebuttal. Most of the evidence I requested (matched baselines, learned-decoder recovery, ablations, seed spreads and NLB metrics) is promised but not yet provided also my remaining concerns are unanswered:
-
The rebuttal concedes that no sparse equation will be claimed on neural data, and my question on decoding from latents versus raw spikes is still unanswered.
-
Clarifications here dont match up as for example which van der Pol equation has the spurious terms and the Lorenz observation model. The new Lorenz result still relies on an oracle decoder which has more spurious than true terms and comes from a single run.
The new comment didn't clarify the main claim and the planned revisions with promised experiment which I believed needs a fresh review. I therefore maintain my score.
Response to Reviewer yj8Q (1/2)
We thank the reviewer for a careful and detailed review, and for recognizing the motivation for placing the sparse drift inside the structured filter. The reviewer's points on result consistency, matched baselines and the learned-decoder setting are well taken, and we will address each of them; we outline our plan below. Several points overlap with our responses to Reviewers k6du and msAG, and we refer to those responses for the full details where noted.
1. Equation recovery on Lorenz and van der Pol
We agree that the submitted results did not support the claims as presented, and that the reporting across Table 1, Table 2 and Figure 2 was inconsistent. We are consolidating these results. Since submission we have rerun the synthetic systems with the decoder fixed to the true readout, the same setting as the van der Pol and pendulum rows. In this setting SyLVER recovers all seven true Lorenz terms with the correct sign, in the rescaled generating frame:
The y˙ and z˙ coefficients are close to their true values. The x˙ coefficients have the correct support and sign but inflated magnitudes, and there are eight small terms outside the true support. The submitted Lorenz row instead came from a run with a learned decoder, in which the equation is expressed in a different latent frame and cannot be compared term by term with the physical equation (see our response to Reviewer k6du, §4). Table 2 therefore mixed two settings, which we should have made explicit.
On van der Pol, we would like to gently clarify that the recovered equation is not dense. The dense equation discussed in Section 4.2 refers to MC-Maze. For van der Pol, y˙=−0.97x+1.54y−1.48x2y is recovered with no terms outside the true support. x˙ recovers the true term (0.95y) together with five small additional terms (Table 4), which we will report with seed spreads.
We will revise the abstract, introduction and discussion so that each claim states the setting in which it holds.
2. Learned decoder and alignment (Q3)
We agree that the fixed-decoder setting removes part of the difficulty the paper targets. In the revision, every synthetic experiment will be reported both with the decoder fixed to the true readout and with a learned decoder. With a learned decoder, the latent is identifiable only up to an affine transformation, so the support and coefficients of the equation depend on the frame (see our response to Reviewer k6du, §4, for a worked example).
SINDy-AE learns a nonlinear decoder, and its equations are compared with the true ones after a post-hoc change of variables: a permutation of the latents together with a per-coordinate scaling and shift (Champion et al., 2019; Gao & Kutz, 2022). We will apply this same alignment to SyLVER's learned-decoder runs on van der Pol and pendulum, as the reviewer suggests. In addition, we will fit a full affine map from the learned to the true latents and transform the recovered coefficients exactly into the physical frame, which is possible because the polynomial dictionary is closed under affine maps. We will apply both procedures to all methods (see our response to Reviewer msAG, §1). We would welcome the reviewer's view on whether this comparison addresses their concern, or whether there is another test of the learned-decoder equations they would find more convincing.
3. Baselines and ablations (Q4)
We are adding the following, as described in our response to Reviewer msAG (§1, §2):
(continued in next comment)