- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
authors present a dataset of 20 min live FUCCI imaging with 47k tracks and endpoint long read scRNA seq with 16k cells from BJ fibroblasts under MYODI mmRNA, siPRRX1 and their combination. Modalities are aligned by condition and the paper benchmarks 10 RNA embeddings and 6 perturbation predictions.
Critical strengths
the resource is completely novel. No public equivalent method combines time-resolved FUCCI tracking paired with long-read scRNA-seq under single and combinatorial TF perturbations.
authors include useful controls for example causal temporal features, a phase/progress-matched shuffle, with acknowledgment of the condition and library confound.
Critical weaknesses
Main issue is that inferring transcriptional response from imaging is not tested.
There is no within-condition, per-program, or leave one condition out evaluation.
RNA control is a separate unsynchronized library so control-subtracted metrics partly reflect batch.
Questions for authors
How many wells are there per condition?
Were the 16 features ranked on training wells only?
What is the condition classification accuracy?
Which imaging frames are matched to the 48h atlas?
Does performance fall to chance for frames before 24h when MYOD1 has not yet been added?
Please report per-program R2 and a within-condition evaluation. Also report the additive and no change baselines for the held out double, and say how overexpression and siRNA were encoded for GEARS and scGPT.
This is a good dataset and valuable resource in perturbation modelling. The paper is well written. If all my concerns are addressed i will raise score by 1 point. If the concerns arent addressed or not justified properly i will reduce the score by 1 point.
5 comments
Here, we clarify questions on replication, temporal performance and perturbation encoding
Q1: How many wells are there per condition?
The imaging data was captured within 3/3/2/2 wells respectively for Hybrid/Negative Controls/mMYOD1/siPRRX1 Each RNA-seq experimental condition was captured as a single biological sample.
Q2: Were the 16 features ranked on training wells only?
Yes, the 16 features were ranked on the training wells only. The selected features have been updated in the manuscript
Q3: What is the condition classification accuracy?
Table 5: Condition classification accuracy
| Model | Condition classification accuracy |
|---|---|
| Manual features | 0.437 |
| MorphEm | 0.471 |
| DINOv1 ViT-S/16 | 0.397 |
| DINOv2 ViT-S/14 | 0.332 |
| OpenPhenom | 0.327 |
| ChAda-ViT | 0.3 |
| SubCell | 0.338 |
Q4: Which imaging frames are matched to the 48h atlas?
No imaging frame is directly matched to a single RNA cell. All imaging observations that pass the quality metrics are evaluated against the 48-hour RNA atlas based on the condition, phase and trajectory.
Q5: Does performance fall to chance for frames before 24h when MYOD1 has not yet been added?
To determine the performance of MYOD1 frames before and after the 24h point, we analyzed the probability and accuracy on these as two separate batches. The pre 24h MYOD1 class recall of 0.455 increased to 0.733 after the induction mark, while the MYOD1 probability increased from 0.335 to 0.503. However, MYOD probability probability in control also increased from 0.29 to 0.502. We interpret this as well dependent drift rather than MYOD1 specific induction. (Rebuttal Table 6)
Table 6: Pre and post MYOD induction accuracy
| Program scoring | True condition | Accuracy, pre-24 h | Accuracy, post-24 h | Mean P(MYOD1), pre-24 h | Mean P(MYOD1), post-24 h |
|---|---|---|---|---|---|
| Standardized mean | Control | 0.495 | 0.065 | 0.290 | 0.502 |
| Standardized mean | MYOD1 | 0.455 | 0.733 | 0.335 | 0.503 |
| Standardized mean | siPRRX1 | 0.697 | 0.355 | 0.064 | 0.177 |
| Standardized mean | siPRRX1/MYOD1 | 0.232 | 0.297 | 0.327 | 0.400 |
| UCell | Control | 0.503 | 0.051 | 0.286 | 0.499 |
| UCell | MYOD1 | 0.447 | 0.750 | 0.329 | 0.503 |
| UCell | siPRRX1 | 0.689 | 0.368 | 0.067 | 0.178 |
| UCell | siPRRX1/MYOD1 | 0.233 | 0.269 | 0.349 | 0.417 |
Q6: Please report per-program R2 and a within-condition evaluation. Also report the additive and no change baselines for the held out double, and say how overexpression and siRNA were encoded for GEARS and scGPT
For the heldout double perturbation, the no change baseline produced (R^2=-0.017) and RMSE (=0.193), whereas the strict additive baseline produced (R^2=0.308) and RMSE (=0.159).
For GEARS and scGPT, the perturbation identity was encoded only as perturbed gene identities. The perturbation direction was not explicitly encoded for PRRX1 knockdown and MYOD1 overexpression. Instead the models learnt the target specific response associated with each perturbed gene identity from the observed single perturbation profiles.
All relevant tables and the figures have been updated in the manuscript supplement
Acknowledging rebuttal
Thank you authors for detailed rebuttal. these answered my questions and I appreciate the candid scoping to population-level recovery.
some concerns remain.
(1) within-condition evaluation is missing. please report per program R2 for each condition and gain over a phase matched baseline if possible. this is because pooled result is dominated by the cell cycle programs P08 & P09.
(2) Please state whether GEARS and scGPT beat the additive baseline on the same metric.
i will raise the score to 4 if these are addressed or justified in some way.
We clarify the benchmark's population level scope report within condition results and assess control batch confounding.
Dear Reviewer,
We thank you for your detailed and thoughtful comments and questions.
The imaging and sequencing data were measured from different cells and the dataset does not support single-cell expression inference. Our held out well transcriptional program inference benchmark provides a quantitative cross-modal benchmark at a population level only.
With the current model architecture, the model cannot generalize to unseen conditions and cannot perform direct RNA or LOCO(Leave-One-Condition-Out) prediction. We restrict our claims to population level state recovery and report program specific and within condition results in the held-out well state recovery task below.
Table 1: Transcriptional Program Prediction:
Table 2: Condition balanced accuracy:
Table 3: Per program R2
The control RNA-seq data was acquired as a separate library and is perfectly confounded by the control condition. We therefore retrained the imaging models with Control excluded from preprocessing, training and evaluation. Perturbation only R2 for all imaging representations, demonstrating that the perturbation association is independent of learning the control class. This analysis does not resolve the batch confounding in perturbation vs control comparisons.
Table 4: Model retraining results without control
Our next comment addresses the questions raised.