- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses cellular intervention selection under limited experimental budgets by introducing VCDesign, a framework that formulates the task as budget-constrained ranking over candidate pools toward desired cellular states. For candidates lacking historical measurements, the authors propose VCDesign-CED, which predicts intervention effects by combining perturbation data with biological priors and integrates these predictions with direct candidate scoring. When candidate-specific responses are masked, VCDesign-CED achieves a median improvement of +0.0492 in aggregated paired BU@20 over base scorers. Conversely, direct retrieval performs better when historical responses exist, and routing solvers by data availability improves selection quality on mixed pools. Controlled experiments further characterize the impact of measurement scale and biological knowledge on retrospective directional ranking.
Critical strengths
Strengths
- The evaluation framework guides method selection across different information regimes. Under a unified intervention target and experimental budget, the paper compares ranking methods when candidate-specific responses are available versus unavailable, examining atlas scale and candidate coverage through controlled experiments. Direct retrieval outperforms CED when historical responses are accessible, and selecting methods based on response availability improves mixed-pool performance. These results clarify how existing measurements influence ranking gains and help practitioners select appropriate methods based on available data.
- Controlled experiments confirm practical gains from effect prediction in candidate ranking. In the pre-locked K562 benchmark, CED achieves a median paired BU improvement of +0.0319 over base scorers, with a 95% confidence interval of [+0.0282, +0.0378]. Cross-context gains persist even after excluding candidates used during base scorer training and development. These results support supplementing direct candidate scoring with perturbation history and biological priors, offering empirical justification for intervention screening when candidate-specific responses are unavailable.
Critical weaknesses
Weaknesses
-
The public implementation of ridge regression cross-validation fails to enforce the identity-level isolation claimed in the paper. In the multi-context training pool, a single gene can span multiple row entries. However, _cross_validate invokes identity_folds over row counts, which partitions row indices uniformly at random without grouping by gene identity. Consequently, records for the same intervention across different contexts leak simultaneously into both training and validation folds. Furthermore, the response SVD basis is fitted on the entire training pool prior to inner-loop splitting, thereby absorbing the inner validation targets. This departs from the claimed unseen-identity validation setting for tuning regularization strength. Grouping splits by gene identity and refitting the response basis strictly within each training fold are necessary to verify parameter choices and the robustness of reported gains.
-
Candidate pool normalization alters relative rankings of existing candidates, and the public pool-size experiment diverges from the methodological definition. Equation (5) standardizes both branches with respect to the current candidate pool, making their relative weighting dependent on pool composition. A reproducible numerical counterexample confirms that appending candidates inferior on both branches can invert the ranking between the original top two choices. In contrast, the public script poolsize.py computes fused scores over the entire candidate pool before subsetting for evaluation, omitting per-subset re-normalization. Because these two computation orders can yield different top selections, the scale experiment does not directly validate the pool-dependent scoring behavior defined in the paper, leaving the stability of candidate selection under the actual formulation unverified.
Questions for authors
Questions
-
Implementation verification and critical checks. Does the multi-context ridge regression code correspond to the reported results? How do the regularization strength selection and primary BU gains change after grouping the inner cross-validation by gene identity and fitting the response basis strictly on each training fold?
-
Do the performance gains of CED hold if the response normalization scheme is selected solely on the training or validation set and then evaluated on an independent test set?
4 comments
Response to Reviewer 8aDp (2/2)
Q1. "Does the multi-context ridge regression code correspond to the reported results? How do the regularization strength selection and primary BU gains change after grouping the inner cross-validation by gene identity and fitting the response basis strictly on each training fold?"
Yes, and neither changes. Rebuilding the held-out results from the code reproduces the published per-query rows with maximum error 0.0. After the fix, the penalty stays in all 36 fits and every BU@20 value is unchanged (+0.0492 [+0.0456, +0.0534]). Details of the fix are under W1 and in Appendix G.5.
Q2. "Do the performance gains of CED hold if the response normalization scheme is selected solely on the training or validation set and then evaluated on an independent test set?"
Yes. A selection that never reads the held-out data makes the published choice, so the gain stays +0.0492 [+0.0456, +0.0534]. With the other choice it would be +0.0454 [+0.0419, +0.0497], so the gain holds either way.
- What was selected: we read "response normalization scheme" as CED's training target, unit-normalized or raw responses, which the submission chose on the held-out pools.
- How: a preregistered nested selection. For each held-out context, the choice was made on the other two non-K562 contexts, each run under the same masked protocol. The held-out context's data were never loaded.
- Result: pooled over its two inner contexts, unit targets lead for every held-out context (RPE1 +0.0065 [+0.0048, +0.0079], HepG2 +0.0039 [+0.0023, +0.0054], Jurkat +0.0007 [-0.0010, +0.0026], no detectable difference; Table 36). Every selection therefore picks the published choice, and the per-context gains are the published ones (RPE1 +0.0526, HepG2 +0.0698, Jurkat +0.0390).
- If you meant the fusion normalization: it has no fitted parameter, and the alternative gives +0.0548 (W2).
- Scope: the margins are small, so this shows that the published choice is what a selection-clean procedure picks, not that unit targets are better in general. All K562 choices were made on the training and development splits.
Changes. Section 5.1 and Appendix G.5 (Table 36). Raw-target results are in Appendix E.1.
Thank you again for reading the code. Both corrections are now part of the released implementation. If they resolve your soundness concerns, we would be grateful if you would reconsider your assessment. If any other part of the code or the paper still looks inconsistent, please tell us and we will check it during the discussion period.
Thank you for the detailed response and for fixing both issues in the released code. The additional analyses address my concerns. I have raised my rating from 2 to 3.
Thank You for Your Constructive Feedback
Thank you very much for your thoughtful and constructive feedback. We sincerely appreciate your careful review and valuable suggestions, which have helped us improve our work. We are grateful for your positive reassessment.
Response to Reviewer 8aDp (1/2)
Thank you for checking our implementation this closely. Both weaknesses you found were real, and both are fixed. In summary:
The revised PDF and the code archive in the supplementary material contain all changes.
Numbers are median paired BU@20 gains of VCDesign-CED over the base scorer (higher is better), on candidates whose own responses are withheld from all training contexts, pooled over the three held-out cell lines. Brackets are 95% cluster-bootstrap intervals over query genes.
W1. "The public implementation of ridge regression cross-validation fails to enforce the identity-level isolation claimed in the paper."
Correct on both points. We fixed both, and no reported result changes.
Changes. Section 4 and Appendix G.5. In the code,
four_context_v1.predictors.fitnow usesselection="grouped"by default (revision_v1/grouped_fit.py, which raises an error if a gene falls on both sides of a fold).selection="frozen"keeps the submitted path so the submitted numbers reproduce exactly, andrevision_v1/test_grouped_fit.pytests that grouped folds never split a gene and that both paths give the same predictor at one penalty.W2. "Candidate pool normalization alters relative rankings of existing candidates, and the public pool-size experiment diverges from the methodological definition."
Correct. The gap was wider than
poolsize.py, and fixing it changes no conclusion.Changes. The text under Eq. 5 (Section 4) and Appendix G.5. In the code,
four_context_v1.evaluate.fuse(..., standardize_over=mask)standardizes over the given candidates only, andpoolsize.py --standardize subsetre-normalizes inside each subset. The defaults reproduce the published numbers.