- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces CGLCP, a method to create reliable prediction intervals for variables that aren't supposed to causally affect each other. Since our assumptions about these relationships are sometimes wrong, the method uses random spot-checks and statistics to estimate the errors and adjust the predictions accordingly.
Critical strengths
- It tackles a real-world problem where global estimates don't capture local errors.
- The math is solid and comes with strong theoretical guarantees.
- The experiments are thorough, especially because they honestly report when the method has to give "infinite" (or unhelpful) intervals just to play it safe.
Critical weaknesses
- An earlier paper https://openreview.net/forum?id=qdrkePJZCW did something very similar. The authors need to clearly explain why their new math is a major step forward compared to that older work.
- The mathematical assumptions are pretty strict (like assuming perfect spot-checks) and are hard to guarantee with real-world biology data.
- When tested on real data (K562), the method gives infinite intervals nearly 40% of the time, which makes you wonder how practical it actually is for daily use.
Questions for authors
- How exactly does your math differ from the previous Asiaee paper?
- What happens to the method's accuracy if the spot-checks are wrong (specifically, false negatives)?
- In practice, when is the extra effort and cost of running these checks actually worth it?
- Does the real-world data you tested actually fit your strict mathematical assumptions?
4 comments
W3, Q3 & Q4
Corresponding to Section 3.3, Section 6.2 (Results and sensitivity analyses), Tables 1 and 8, Appendices F.6–F.9, Appendix B, and Section 7 in the revised manuscript.
Table R3.2: Coverage and Finite Output Under Upstream Effect Screening
| Calibration setting | Pooled CP (label-free) | GraphLCP (label-free) | CGLCP (certified) | CGLCP finite output |
|---|---|---|---|---|
| Original pools | 90.95% | 91.09% | 97.68% | 61.61% |
| Effect screening | 88.97% | 88.97% | 96.72% | 61.70% |
We appreciate the reviewer’s practical perspective on assumption fit, infinite intervals, and cost-benefit trade-offs.
1. How K562 fits our assumptions (Q4): We distinguish design-enforced conditions from biological idealizations (Appendix F.9). Non-score pool construction (Assumption 1) and uniform score-independent sampling (Assumption 3's sampling clause) hold by construction via sealed fold splitting and randomized auditing. By contrast, assay-based equivalence intervals () serve as operational proxies rather than exact causal memberships—with unresolved pairs driving much of the conservative 38.7% abstention rate (optimistic handling raises finite output from 61.61% to 77.55%, Table R1.2)—and cell-state or scale differences can perturb exact safe-score exchangeability (addressed via tail diagnostics in Appendix F.8 and total-variation bounds in Appendix B).
2. When verification checks and abstentions are worth it (W3 & Q3): Infinite intervals in CGLCP act as explicit calibrated abstentions rather than silent failures: returning signals that local contamination is too high (or pool size too small) to certify a finite 90% interval at the current audit budget, prompting targeted follow-up or a larger global pool (which raises finite output to 68.6%–82.1%, Appendix F.6).
Verification is most valuable when upstream filtering removes obvious large responses, leaving subtle, small-residual contaminants where label-free methods lose their natural conservative cushion. We modeled this on K562 by retaining the bottom 80% of calibration candidates by absolute reference-fold effect magnitude (Table R3.2). Under this screening, label-free Pooled CP and GraphLCP drop below the 90% target in point estimate (to 88.97%; 95% CIs spanning 90%), whereas CGLCP sustains 96.72% coverage and 61.70% finite output using ~14 audited labels per query. In high-stakes perturbation or safety screens where false-safe assurance is costly and upstream QC filters out large effects, this targeted verification cost buys reliable protection and explicit abstention flags when local evidence is insufficient.
Thank you for addressing my questions. The clarification of the contribution beyond Asiaee et al. and the new analyses of verification errors and practical trade-offs strengthen the paper and exact verification and the high rate of infinite intervals remain concerns, I will maintain my rating since it justify the current state of the paper.
I again thank you for your time and good luck to you!!
Thank you for your time and valuable feedback on our work. Your comments and suggestions have provided strong help in further improving our paper and reflecting on its value (including the clarification beyond Asiaeet al., verification errors, and practical trade-offs). We appreciate the concerns you raised regarding exact verification and the high rate of infinite intervals, and we will continue to address them in our future revisions. Your input has been truly beneficial to our project. Thank you again for the time you invested in our work, and good luck with your future research!
We thank Reviewer pf4r for supporting our local-risk motivation and transparent reporting. We have addressed all questions regarding methodological novelty, verification noise, and practical applicability.
Below are our responses to each weakness and question.
W1 & Q1
Corresponding to Sections 1, 2, and 4 (explicit attribution), Section 5 and Algorithm 1 (local certificate and error budget), Appendix F.6, and Tables 1 and 5 in the revised manuscript.
We build upon the sharp oracle contamination inequality of Asiaee et al. (2026, Appendix C), which is explicitly credited in Sections 2 and 4. Our mathematical contribution lies in making this bound operational when safety labels are partially observed rather than fully known.
Specifically, Asiaee et al. require exact safe/unsafe labels for the selected calibration pool. Addressing the setting where labels are hidden and costly to verify, we construct an exact finite-population hypergeometric certificate from a limited random audit, couple it with a split error budget (ρ for certificate validity and β for conformal calibration) to derive the minimal certificate-valid rank (Proposition 1), and establish an exact oracle-relative efficiency decomposition separating certificate-validity cost from over-certification error (Eq. 12). For example, with n=60, certified unsafe count 12, α=0.10, and ρ=0.02, our certified rank is 57, whereas an additive certified rule exhausts the error budget and returns +∞ (0% finite output vs. 61.61% on K562 under matched information; Appendix F.6).
W2 & Q2
Corresponding to Section 3.3 (Remark 1), Section 6.1 (Verification errors and score placement), Appendix E.6, Table 3, Figure 13, and Section 7 in the revised manuscript (complementing Appendix E.5).
Table R3.1: Impact of Noisy Verification and Unsafe-Score Placement Settings: n=60, true unsafe B=24, nominal target 90%. "Low" places unsafe scores near zero (theoretical worst-case); "High" places them in the right tail (large-residual regime). Finite output relies solely on the audit count, independent of score placement.
With unsafe members treated as the positive class, verification false negatives (FN) and false positives (FP) affect the certificate asymmetrically, interacting with the score placement of contaminants (Appendix E.6):
False Positives (FP) are conservatively safe: FP errors inflate B regardless of score placement, widening fixed-pool intervals relative to exact verification. As shown in Table R3.1, this preserves nominal coverage under valid safe-score exchangeability (exceeding 98% empirically here) at the expense of finite availability (dropping to 14.74% at 60% FP).
False Negatives (FN) decouple certificate validity from empirical miscoverage: FN errors under-certify the unsafe count, yet empirical coverage degrades much more gradually than the certificate itself:
Together, Remark 1 (giving the degraded lower bound (1−pfail)(1−β)) and Appendix E.5 outline how an external upper bound on missed unsafe labels can be incorporated into a noise-aware union-bound certificate, while Table R3.1 clarifies that practical undercoverage requires both substantial FN rates and small contaminating residuals.