- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper argues that predefined gene sets, although poor standalone encoders of cell state in the evaluated settings, can help detect discrepancies between new data and a reference dataset. The method learns relationships between predefined program readings and a low-dimensional cell state using reference data, then applies split conformal calibration to establish a consistency threshold. On new cells or cell groups, it tests whether any state can jointly explain the observed program readings. When no state satisfies the calibrated constraints, it flags the unit and returns a small set of programs certifying the incompatibility.
Critical strengths
The paper presents an interesting and novel application of gene sets as consistency checks. The geometric analysis provides a compelling explanation for how redundant programs can support discrepancy detection despite offering limited precision for state estimation. Returning a small, verifiable witness also makes the detected inconsistency easier to inspect.
Critical weaknesses
The practical utility for practitioners remains insufficiently clear. In particular, it is unclear what actionable biological conclusions a witness supports beyond identifying an unusual combination of program readings. A concrete example showing how a practitioner would interpret a flag, investigate its witness, and gain insight beyond standard differential expression or pathway analysis would strengthen the paper.
Questions for authors
- How does the method scale computationally with the number of reference cells, programs, and state dimensions? What would be the runtime and memory requirements for reference atlases containing millions of cells?
- How should practitioners use a massive, heterogeneous atlas as a reference for a smaller newly collected dataset? In particular, how much matched calibration data is required, and how should differences in sequencing protocol, donor composition, and cell-type representation be handled to preserve meaningful error control?
How a practitioner reads a flag and its witness, cost on a million-cell reference, and how much matched calibration data is needed
Thank you for the constructive review. We added the requested practical evidence; changes are in blue in the revised PDF.
How a practitioner uses a flag and its witness (Section 7, Table 2).
What this adds beyond differential expression or pathway analysis. Differential expression reports which genes differ from a comparison group, not whether the difference is a change of state or of regulation. A ranked list of deviating pathways cannot tell either: on 941 flagged IFN-β groups, the nine most deviating programs were jointly satisfiable by some state in all but at most 3.2% of groups, but by a reference group's state in at most 1.9%, so the list cannot separate an unseen state from a regulatory change; the witness does. It is also reusable as a compact check: over 400 flagged groups, a witness's nine programs alone flag on average 68% of the other stimulated groups of its cell type (48% for CD4 T cells) and no control group (Table 33).
Scaling (Tables 12 and 36). Scoring cost does not depend on the reference size; it grows with the number of programs and the state dimension as in step 2. Fitting needs only sums over reference cells, so a reference can be streamed in chunks. We timed a streamed fit on references of 104, 105 and 106 cells resampled from Kang control cells: the pass over 1,000,000 cells took 2.0 minutes (about 8,400 cells per second) after a one-minute setup on a 50,000-cell subsample, on a laptop CPU. Peak memory was 7.5 GB at every size, set by the setup, not by the reference size. Calibration costs one linear program per calibration group.
Using a large heterogeneous atlas (Section 7, Tables 14, 21, 35). The atlas can supply the encoder and the view model, but calibration must match the new data.