- Rating
- 4
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose a response calibration loss function and a training scheme suitable for learning the effects of neural perturbations. The scheme is evaluated empirically across several model families, including conditional flow matching. The evaluation is performed almost exclusively on synthetic data, including both idealized repeated-trial setups and a realistic continuous data acquisition strategy. The synthetic results confirm the utility of the modified loss function.
Critical strengths
-
Comprehensive set of evaluations including ablations.
The authors evaluate many distinct models, including ones that predict the effect of perturbation directly (ResponseNet, SVD shrinkage), statistical and generative models that forecast the activity of each neuron (GRU, Gaussian Transformer, Flow Matching), and mechanistic models matched to the data generator (Rate RNN). The training comparisons are also thorough, testing models with and without the response calibration term and across multiple objective ablations.
-
Realistic data acquisition setup.
Beyond the primary repeated-snapshot benchmark, the authors also test a training data acquisition strategy that closely matches a realistic biological protocol (Section 4.3 / Appendix B.2). In this setup, synthetic data is recorded continuously without state resets, using randomly interleaved stimulated and blank trials.
Critical weaknesses
-
The text is very dense and hard to parse.
It is difficult to follow the paper for a reader without direct expertise in this specific niche. The explanations in the main text are overly compressed and often not self-contained, forcing the reader to constantly jump to the appendices to understand how key baselines (such as ResponseNet) or experimental protocols actually work.
-
No positive confirmation on real biological data.
The evaluation providing positive evidence for the method is entirely synthetic, which only gives indirect evidence that the approach will work in practice. The authors report a negative result on a real biological recording (Appendix C), where calibration provides no improvement. The test only evaluates a linear ridge predictor rather than PC-Flow, and the lack of improvement is attributed to an unsuitable experimental protocol (missing matched blank trials) and limited recording coverage.
Questions for authors
-
The performance comparison indicates that PC-Flow is computationally demanding at both training and inference stages. While the timings are reported, the practical trade-off for real workflows is not fully clear. Do the authors expect this scaling to be a bottleneck in realistic setups? When the extra cost of PC-Flow justified over the much faster calibrated Gaussian Transformer or ResponseNet?
-
Did the authors try to locate other public biological datasets where the proposed method (and specifically PC-Flow) can be evaluated? Can the authors share their general thoughts on what experimental changes are needed to make their approach applicable to real recordings?
-
Conditioning the flow on the perturbation ( vs. ) looks structurally very similar to Classifier-Free Guidance (CFG). Did the authors consider applying CFG at inference time to the already trained uncalibrated CFM model? Specifically, one could integrate the guided velocity field with to steer the flow toward the perturbation (see https://sander.ai/2022/05/26/guidance.html). This simple experiment might clarify whether uncalibrated CFM merely underestimates the magnitude of the perturbation effect or learns an incorrect directional response pattern across neurons altogether.
-
Why does PC-Flow outperform ResponseNet on the mean contrast? Both PC-Flow and ResponseNet use the exact same Transformer backbone and supervise the same calibration target, yet PC-Flow achieves a significantly lower response NRMSE (0.120 vs. 0.183 in Table 2). Why does learning the full generative trajectory distribution improve the estimation of the mean contrast compared to predicting the contrast directly?
-
As shown in Table 2, response calibration dramatically improves the response NRMSE (from 0.317 down to 0.120 for PC-Flow) while leaving the forecast CRPS score virtually unchanged (0.128 vs. 0.128). Could the authors clarify why general forecasting metrics are so insensitive to perturbation calibration?
7 comments
Response to Reviewer WU7h (2/5)
W2 / Q2 (continued). [new] Results on the two datasets. We applied the paper's no-reset protocol. The calibration target is the inverse-probability-weighted contrast of each stimulated condition against blank trials [3]. Models are scaled to the data (two attention blocks, width 64). Because sessions contain only a few hundred trials, we used five-fold cross-fitting [4]: each fold's model is fitted on the other four folds and scored only against contrasts computed from the held-out fold, so predictions and references never share trials. The floor is the error that the true contrast itself would obtain against these noisy held-out references. As in the paper, the score excludes the targeted cells' own responses (Section 4.8, Table 4, Appendix C).
| Response NRMSE, non-targeted cells | Barrel cortex [1] | Motor cortex [2] |
|---|---|---|
| Pooled contrast | 0.921 | 0.948 |
| Gaussian transformer + cal. | 0.882 | 0.906 |
| CFM | 0.969 | 0.962 |
| PC-Flow | 0.871 [0.808, 0.938] | 0.897 [0.865, 0.931] |
| Floor | 0.781 | 0.846 |
Calibration improves both model families (barrel cortex: PC-Flow − CFM = −0.098 [−0.126, −0.069]; calibrated − uncalibrated Gaussian = −0.069 [−0.100, −0.046]), and the ordering matches the simulations. PC-Flow and the calibrated Gaussian are statistically indistinguishable at a few hundred trials per session (−0.011 [−0.040, 0.018]). Non-targeted responses are weak and mostly suppressive in barrel cortex [1], so all models lie near the floor; the condition stimulating all 200 targets, held out of training, separates them most clearly (floor 0.624, PC-Flow 0.762, calibrated Gaussian 0.776, CFM 0.903). The revised abstract states the benefit of calibration on real data and that PC-Flow's further benefits are so far shown only in simulation.
[new] C. elegans with PC-Flow. The reviewer noted that the submitted C. elegans test [5] used only ridge regression. Under the same date-held-out protocol we now also fitted small PC-Flow, calibrated Gaussian, CFM and response-network models (group RMSE 0.10163 for PC-Flow, against 0.10175 for predicting zero change). All stay within the six-date interval of predicting zero change, as expected for recordings without blank trials (Appendix C.4).
What experiments need. We added these requirements to the Discussion, in order of importance: (1) randomly interleaved blank trials with logged assignment probabilities, because the target is a randomized contrast rather than a pre/post difference; (2) trial selection and grouping based only on pre-stimulus information; (3) a fixed pulse duration and power, ideally with a few doses; (4) repeated stimulation of the same targets; (5) sampling faster than the response, with stimulation artifacts shorter than the response peak; (6) random subsets of a multi-target set for interaction tests; (7) stable cell identities across the session. The barrel-cortex protocol [1] meets (1)–(4) and most of (5).
References
[1] H. W. P. Dalgleish, L. E. Russell, A. M. Packer, A. Roth, O. M. Gauld, F. Greenstreet, E. J. Thompson, M. Häusser. How many neurons are sufficient for perception of cortical activity? eLife 9:e58889, 2020. doi:10.7554/eLife.58889. Data: doi:10.6084/m9.figshare.13128950, doi:10.6084/m9.figshare.13135505
[2] K. Daie, K. Svoboda, S. Druckmann. Targeted photostimulation uncovers circuit motifs supporting short-term memory. Nature Neuroscience 24(2):259–265, 2021. doi:10.1038/s41593-020-00776-3. Data: doi:10.25378/janelia.13546157
[3] A. Boruvka, D. Almirall, K. Witkiewitz, S. A. Murphy. Assessing time-varying causal effect moderation in mobile health. Journal of the American Statistical Association 113(523):1112–1121, 2018. doi:10.1080/01621459.2017.1305274
[4] V. Chernozhukov, D. Chetverikov, M. Demirer, E. Duflo, C. Hansen, W. Newey, J. Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal 21(1):C1–C68, 2018. doi:10.1111/ectj.12097
[5] F. Randi, A. K. Sharma, S. Dvali, A. M. Leifer. Neural signal propagation atlas of Caenorhabditis elegans. Nature 623(7986):406–414, 2023. doi:10.1038/s41586-023-06683-4. Data: doi:10.17605/OSF.IO/E2SYT
Response to Reviewer WU7h (3/5)
Q1. Is PC-Flow's cost a bottleneck, and when is it justified?
Where the cost is. On the profiled CPU (Table 5), a PC-Flow training update costs 18.9 s on average, against 0.91 s for the calibrated Gaussian and 0.86 s for the response network, because every fourth update integrates 16 Heun steps (a second-order ODE solver) for eight pulse/blank sample pairs. Reading out one history–source contrast with 32 pairs takes 21.8 s, against 0.027 s for the Gaussian's analytic mean and 0.014 s for the response network. The readout is the main bottleneck: 48 histories × 20 sources × 256 pairs amount to tens of CPU hours per circuit if run serially. Neuron attention also grows quadratically with the number of recorded cells. The first remedies are batching sources and sharing blank trajectories across sources, which removes about half of the trajectories; the revised Discussion states this.
[new] Reducing the cost. PC-Flow-lite, which calibrates on every 8th instead of every 4th update with 4 instead of 8 pairs and 8 instead of 16 integration steps, costs about 2.98 s per update, three times the calibrated Gaussian, and reaches response error 0.133 [0.121, 0.147], versus 0.120 for the default and 0.190 for the calibrated Gaussian; giving the calibrated Gaussian PC-Flow's training time leaves it at 0.188 (Fig. 6d). Reading out with 32 or 8 pairs instead of 256 cuts readout time 8× or 32× and gives 0.129 or 0.156 (Figure 7(d)), both still below the calibrated Gaussian.
When it is justified. For population-average single-source responses with ample repeats, the calibrated Gaussian or the response network is the better choice: their readout is orders of magnitude cheaper and, on the real data in part 2/5, equally accurate. PC-Flow is worth its cost when a question requires:
- state-dependent responses: the error on the state-dependent part of the response (each model's mean over states removed) is 0.127 for PC-Flow versus 0.203 for the calibrated Gaussian without resets;
- joint or tail-event predictions: at dose 4, PC-Flow's advantage in energy score, a proper score for joint predictive distributions [1], is about six times larger than at dose 0.4, and its Brier score for tail events is 0.071 versus 0.096;
- few repeats or partial observation, where the absolute gap is largest (Fig. 3b, Fig. 6a);
- predictions for combined pulses, where its error on the nonadditive part of eight-source responses is 0.058 versus 0.150.
References
[1] T. Gneiting, A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association 102(477):359–378, 2007. doi:10.1198/016214506000001437
Response to Reviewer WU7h (1/5)
We thank the reviewer for the positive assessment and for questions that each pointed to a real gap; the guidance and CRPS questions led to two new analyses that we think clarify the paper. Section, figure and table numbers refer to the revised PDF.
W1. Dense, not self-contained text.
We rewrote the paper. Section 3 now defines every baseline in the main text: the response network uses the same attention backbone with a deterministic head that predicts the stimulated-minus-blank response directly from history, stimulated source and dose; the Gaussian transformer uses the same backbone with a mean head and a low-rank-plus-diagonal covariance head; the rate RNN has the simulator's tanh state equation with learned weights and a learned initial-state encoder. Each experimental subsection states the protocol facts needed to read its table, a roadmap table at the start of Section 4 maps each question to its evidence and answer, Fig. 1(d) shows the model and its training.
W2 / Q2. Real data, other datasets, and required experimental changes.
Dataset search. The method needs trial-resolved recordings in which stimulated and blank trials are interleaved at random with known probabilities. Two public datasets meet this requirement, and we analyzed both (results in part 2/5):
Other candidates lack randomized blank trials or trial-resolved data: we found no public release of the recordings in [3]; [4] releases data for reproducing figures; the CA1 data of [5] compare stimulation sites rather than stimulated versus blank trials; and public in vitro multi-electrode array datasets record spontaneous activity or pharmacological, electrical or closed-loop stimulation [6–8]. Table 24 summarizes the search.
References
[1] H. W. P. Dalgleish, L. E. Russell, A. M. Packer, A. Roth, O. M. Gauld, F. Greenstreet, E. J. Thompson, M. Häusser. How many neurons are sufficient for perception of cortical activity? eLife 9:e58889, 2020. doi:10.7554/eLife.58889. Data: doi:10.6084/m9.figshare.13128950, doi:10.6084/m9.figshare.13135505
[2] K. Daie, K. Svoboda, S. Druckmann. Targeted photostimulation uncovers circuit motifs supporting short-term memory. Nature Neuroscience 24(2):259–265, 2021. doi:10.1038/s41593-020-00776-3. Data: doi:10.25378/janelia.13546157
[3] A. Wagenmaker, L. Mi, M. Rozsa, M. S. Bull, K. Svoboda, K. Daie, M. D. Golub, K. Jamieson. Active learning of neural population dynamics using two-photon holographic optogenetics. Advances in Neural Information Processing Systems 37:31659–31687, 2024. doi:10.52202/079017-0994
[4] I. A. Oldenburg, W. D. Hendricks, G. Handy, K. Shamardani, H. A. Bounds, B. Doiron, H. Adesnik. The logic of recurrent circuits in the primary visual cortex. Nature Neuroscience 27(1):137–147, 2024. doi:10.1038/s41593-023-01510-5
[5] M. A. Triplett, E. Bäumler, A. Prodan, R. Stonis, D. S. Peterka, M. Häusser, L. Paninski. Computational optimization of two-photon holographic stimulation sites in vivo. Journal of Neural Engineering 23(2):026015, 2026. doi:10.1088/1741-2552/ae4925
[6] Y. Bar El, S. Kanner, A. Barzilai, Y. Hanein. Calcium imaging, MEA recordings, and immunostaining images dataset of neuron-astrocyte networks in culture under the effect of norepinephrine. GigaScience 8(2):giy161, 2019. doi:10.1093/gigascience/giy161
[7] B. J. Kagan, A. C. Kitchen, N. T. Tran, F. Habibollahi, M. Khajehnejad, B. J. Parker, A. Bhat, B. Rollo, A. Razi, K. J. Friston. In vitro neurons learn and exhibit sentience when embodied in a simulated game-world. Neuron 110(23):3952–3969.e8, 2022. doi:10.1016/j.neuron.2022.09.001
[8] F. E. Kapucu, A. Vinogradov, T. Hyvärinen, L. Ylä-Outinen, S. Narkilahti. Comparative microelectrode array dataset of the functional development of hPSC-derived and rat neuronal networks, version 1.0 [dataset]. G-Node, 2021. doi:10.12751/g-node.wvr3jf