- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
Summary
- This paper investigates why existing perturbation prediction models often fail to outperform simple baselines, focusing on cross-context generalization and limitations of evaluation metrics.
- Causal Transportability: The authors theoretically characterize when perturbation effects can be transferred across biological contexts based on shared causal mechanisms.
- Causal DGP: They develop an OU process-based semi-synthetic simulator to systematically evaluate generalization under changes in causal regulatory networks () and basal expression ().
- Vendi Score: They adapt the Vendi Score with null calibration to detect diversity collapse, complementing PDS in evaluating perturbation-response diversity and discrimination.
- Experiments on synthetic and real Perturb-seq datasets demonstrate substantial cross-context generalization gaps and investigate their relationship with transcriptome-wide impact (TI) concordance.
Critical strengths
Strengths
- Purpose-driven causal simulation: Rather than attempting to model every biological factor, the authors construct a controlled benchmark specifically designed to investigate causal mechanism shifts. Extending OU-based dynamics to cross-context evaluation is an interesting and meaningful contribution.
- Theoretical analysis: The decomposition into , and provides a clear framework for understanding perturbation transportability. The derivation of perturbation-specific transportability conditions is particularly insightful.
- Noise aware evaluation: The null calibration of the Vendi Score using empirical control cells is a thoughtful approach to accounting for finite-sample variability.
Critical weaknesses
Weaknesses
- Simplified causal structure: Although the Causal DGP enables controlled evaluation, it does not fully capture biological complexities such as nonlinear regulation, context-dependent perturbation efficiency, and context-specific measurement effects.
- Limited novelty of the Vendi Score adaptation: Limitations of existing evaluation metrics and mode collapse in cellular perturbation prediction have already been investigated (Viñas Torné et al., 2025; Wu et al., 2025; Mejia et al., 2025). Moreover, the Vendi Score has previously been applied to diversity evaluation in single-cell transcriptomics (Xu et al., 2026; DenAdel et al., 2026). While the proposed null-calibrated pseudobulk Vendi Score is a useful adaptation, its additional benefits over existing diversity and discrimination metrics could be more clearly demonstrated.
- Causal mechanism identifiability: It remains unclear whether the models fail because they cannot learn causal mechanisms or because the available observations are insufficient to identify the target-context mechanisms. This distinction is important for interpreting the generalization failures (Pratapa et al., 2020).
- Limitations of TI concordance: TI summarizes the magnitude of perturbation effects but does not preserve gene-specific directions or regulatory relationships. Therefore, high TI concordance does not necessarily imply similar causal mechanisms, raising questions about its reliability as a proxy for mechanistic divergence.
Questions for authors
Questions
- Identifiability: Could the authors distinguish failures caused by insufficient causal learning from those caused by fundamental identifiability limitations, potentially using an oracle or mechanism-aware baseline?
- TI Concordance: Since TI does not preserve gene-specific effect directions, how reliable is it as a proxy for mechanistic divergence? Could it be directly compared with ground-truth mechanism divergence in the synthetic data?
- Theory vs. Observation: Proposition 1 establishes transportability conditions for latent mean perturbation effects, whereas the experiments evaluate observed expression after nonlinear emission. How does this difference affect the connection between the theoretical guarantees and empirical findings?
4 comments
Rebuttal Response by Authors
Dear Reviewer,
Thank you for your thoughtful review. As the discussion period closes on Wednesday, we wanted to check whether you have had a chance to read our rebuttal.
We have made a careful effort to address each of your concerns, and we would be grateful if you would consider updating your score if you feel they have been resolved. We are happy to clarify any remaining points before then.
Thank you for the detailed responses and additional analyses. The explanations on TI concordance and the Vendi score address some of my concerns. However, I think the distinction between learning failures and identifiability limitations, as well as the gap between theory and observed outcomes, remains somewhat unclear. I will therefore maintain my original rating.
Rebuttal Response by Authors
Thank you for engaging and for noting that our responses on TI concordance and the Vendi score addressed some of your concerns. We clarify the two remaining points below.
[W3/Q1] Learning failures vs. identifiability limits
causalDGP separates the two because mechanism shifts are known by construction. The matched in-context split serves as a model-specific oracle: test cells are identical, but in-context training also includes target-context cells of the held-out perturbations (Lines 357-360). The transfer gap (in-context minus cross-context performance) thus isolates each model's cost of transfer. Proposition 1 determines identifiability:
- shifts: target effects equal source effects and are identifiable from source data, so any transfer gap is a learning failure; scLDM shows no transfer gap here, confirming the information is available (Figure 5).
- shifts: target effects are exactly recoverable given , so the transfer gap reflects missing information about this map, not model capacity. Target-context controls and seen perturbations identify it only partially (Section 2.3; Pratapa et al., 2020), and no evaluated model closes this gap, which grows with rewiring (Figure 5; Table 11).
In real data, the type of mechanism shift is unknown, so learning failures and identifiability limits cannot be fully separated. Instead, low TI concordance flags mechanistic divergence, where identifiability limits are likely: Zhu25 has the lowest concordance and largest failures, Replogle22/Nadig25 moderate concordance and partial transport (Table 9; Figures 4 and 5). Zhu25 also has more control cells per context than Replogle22/Nadig25 (Table 1), so limited data is an unlikely cause.
[Q3] Theory vs. observation
The nonlinear emission changes observed effect magnitudes but does not alter the underlying latent-space identifiability conditions established by Proposition 1. Hence, the theoretical conditions and empirical findings remain consistent. The emission (softplus mean with negative binomial sampling) maps latent states to observed counts and is monotone and shared across contexts (Section 2.2, Lines 160-191), so latent effects remain identifiable from observed expression. Under shifts, the emission can alter observed effect sizes, especially for weakly expressed genes, but accounting for it can close this gap: scLDM shows no transfer gap under no shift or shifts (Figure 5). shifts change the latent effects themselves and require . Consistent with Proposition 1, transfer gaps are small under no shift and , larger under and combined shifts (Figure 5), and increase with rewiring (Table 11).
We hope this addresses the reviewer's remaining concerns, and we would be grateful if they would consider these points in their final assessment. We are happy to provide further clarification.
Rebuttal by Authors
We thank the reviewer for the insightful and detailed comments, and for recognizing our purpose-driven simulator, perturbation-specific transportability analysis, and null-calibrated Vendi score. Changes in the revised paper are in blue; we address weaknesses [W] and questions [Q] below.
We agree. This formulation provides a tractable setting in which the effects of mechanism shifts can be precisely characterized. Context-dependent efficiency could be added by letting Γq vary across contexts, measurement effects by varying the emission model, and nonlinear regulation by using a nonlinear drift. Each adds a context-specific mechanism, making transport harder, yet models already fail in the simplified setting (Figure 5). The real datasets likely include these complexities, and TI concordance tracks the gap in synthetic and real data (Table 9). We have added these extensions to Appendix B.2.1 (Lines 1104-1113); measurement-level shifts are also noted in Section 2.2 (Lines 160-191).
We agree that metric limitations and mode collapse have been studied (Lines 45-72), and now cite prior single-cell uses of Vendi, which measure cell-level diversity without null calibration (DenAdel et al., 2026; Xu et al., 2026; Lines 262-264). Our pseudobulk Vendi instead treats each perturbation response as a unit with a null-calibrated RBF kernel; its predicted-to-observed ratio distinguishes collapse from spurious diversity. Unlike uncalibrated effective rank, which stays high under weak effects, the calibrated Vendi score tracks effect strength from the null (Figure 3a). Paired with PDS-ℓ1, Vendi separates mode collapse from discrimination (Figures 3b and 11): across contexts, deep models lose discrimination but stay more diverse than simple baselines, a pattern neither metric reveals alone. At the dataset level, Vendi and split-half PDS-ℓ1 indicate response diversity and reproducibility, a proxy for noise (Table 5).
We agree this distinction matters. In causalDGP, it can be largely separated; in real data, only partially. Matched splits differ only in target data for held-out perturbations (Lines 357-360), so the gap isolates the cost of withholding them; with Proposition 1, they serve as a mechanism-aware reference. Under baseline-only shifts, target effects equal source effects, so remaining gaps reflect insufficient learning (Figure 5). Under regulatory shifts, transport requires an identifiable transport map AC′−1AC, whose recovery remains challenging (Pratapa et al., 2020; Section 2.3), so part of the gap may reflect a fundamental identifiability limit. In real data, failures are consistent across models and datasets, and low TI concordance indicates where responses diverge (Table 9). Notably, Zhu25 has more control cells per context than Replogle22/Nadig25 (Table 1), so its failures (Figures 4 and 5) are unlikely to reflect limited sample size.
We agree that TI summarizes effect magnitudes, so high concordance does not guarantee similar mechanisms. Its reliability is nonetheless supported where ground truth is known. In causalDGP, where rewiring is known by construction, TI concordance declines monotonically with rewiring (0.91, 0.48, 0.25, 0.18 at 0, 25, 50, and 75%; Table 11) and tracks the transfer failures (Figure 5). In real data, it is consistent with known cell-line differences (Replogle22/Nadig25; Figure 14) and coincides with cross-donor failures (Zhu25; Figure 5). Although magnitude-preserving direction changes are possible in principle, TI concordance distinguished every known mechanism shift in causalDGP. We now present low TI concordance as an empirical proxy for mechanistic divergence, not a guarantee of similarity (Section 4.3.3, Lines 462-464).
Proposition 1 concerns latent transport, and nonlinear emission can alter measured effects across baselines even when latent effects are equal (Section 2.3, Lines 203-209). The link to our experiments is identifiability: since the latent-to-mean map is monotone and shared across contexts (Section 2.2, Lines 160-191), target effects remain identifiable in principle, though information is attenuated for weakly expressed genes. Baseline-only shifts can still produce small observed gaps unless models account for the emission. Empirically, the gap follows the predicted ordering: small under no-shift and baseline-only settings, larger under regulatory and combined shifts (Figure 5).