acceptodds
Under review as a conference paper at ICLR 2027

AptamerBench: Auditing Generalization Across Patients and Cross-Modal Correspondence in Paired Single-Cell Profiling

Abstract

Accurate prediction in unseen patients does not by itself establish subtype discrimination within patients or the value of pairing measurements from the same cell. We introduce AptamerBench, a benchmark for cell typing from single-cell aptamer-binding (APT) profiles, with 361,792 paired cells across 40 participants, 293 aptamer features, and an operational RNA-derived hierarchy of five lineages and 27 subtypes. AptamerBench combines patient-disjoint evaluation, subject-balanced scoring, and controls retrained on shuffled APT profiles. Standard APT-only models achieve low subtype performance in new patients. With true-lineage decoding applied to both conditions, an MLP trained on real APT exceeds its matched control retrained on profiles shuffled within each patient and lineage by 0.160 in subject-balanced Macro-F1 (95% CI [0.132, 0.172]). This contrast remains positive under joint disease/acquisition-group holdout. High conditional scores can nevertheless coexist with poor discrimination within patients. Under the original patient-disjoint folds, real APT, shuffled APT, and context from other cells of the same patient each achieve approximately 0.87 subject-balanced Macro-F1 when distinguishing Memory CD8T-1 from NK cells, while the context model has zero minority-class recall in every patient. An RNA+APT gain over the RNA reference is not reproduced under group holdout, and the tested controls do not resolve an additional classification benefit from exact same-cell pairing. Label-independent retrieval nevertheless detects same-cell RNA–APT correspondence in held-out patients. An exploratory multicenter RNA/ADT analysis reproduces the same qualitative separation. AptamerBench provides reusable tests that distinguish generalization to new patients, conditional subtype association, within-patient discrimination, modality benefit, pairing-specific predictive value, and detectable cross-modal correspondence.

Primary area:
datasets and benchmarks
Keywords:
single-cell aptamer profiling, cross-patient cell typing, patient-disjoint evaluation, multimodal learning, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ALW2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces AptamerBench, a single-site cohort of 361,792 PBMCs from 40 participants (five different phenotype plus healthy). Each cell has paired scRNA-seq and 293 aptamer-binding (APT) count features. Targets are 27 RNA-cluster-derived subtypes, grouped into five "operational" parents (B, T, Myeloid, CD8T/NK, Other). All evaluation uses five disease-stratified, patient-level folds. The paper poses three questions:

Q1: APT-only prediction of parent/subtype in unseen patients, using LR, XGBoost, an MLP, and three custom Transformer variants.

Q2: Whether APT keeps subtype signal after controlling for patient and parent. The true parent is given at decoding, and real APT is compared with a model retrained on APT permuted within patient×parent strata.

Q3: Whether adding APT to RNA helps, and whether any gain depends on exact same-cell pairing (RNA vs RNA+APT vs grouped-shuffled APT vs patient-context controls). This is complemented by a label-free RNA↔APT retrieval audit.

Main findings: (i) APT alone does not predict cell types well in unseen patients.

(ii) APT nonetheless carries some cell-level subtype signal beyond patient and parent composition: when the true parent is given, real APT outperforms a control in which APT profiles are shuffled among cells of the same patient and parent.

(iii) Adding APT to RNA does not consistently improve cell-type prediction. Replacing each cell's own APT profile with that of another cell from the same patient gives the same performance, so pairing each cell's RNA with its own APT gives no measurable classification benefit.

(iv) A label-free retrieval test, which matches each cell's RNA profile to its APT profile among other cells of the same held-out patient, succeeds above chance. RNA and APT therefore share cell-level correspondence even though this correspondence does not improve cell-type prediction.

Critical strengths
  1. Patient-aware evaluation. The paper shows that cell-level performance pooled across patients can be misleading, and it adds subject-balanced and within-patient metrics to expose this. Single-cell annotation benchmarks rarely report within-patient metrics, so this practice is useful beyond this dataset.

  2. A new data modality: single-cell aptamer profiles paired with RNA in a clinical cohort. If released, this data would be a useful resource.

Critical weaknesses
  1. Limited value as a benchmark.

(a) The data are not available. The paired data are protected and would be deposited "after acceptance, pending final ethics and data-use approval", so nobody can currently use or check the benchmark.

(b) The scale is small and confounded: 40 participants from one site, 8 test patients per fold. The authors state that disease and control groups were collected in different acquisition windows, so disease is fully confounded with batch. This limits sample-level tasks and any claim about generalization.

(c) The measured modality is not described. The paper never says what the 293 aptamers bind, how the panel was chosen, or whether the targets are known.

(d) The baselines are limited. The three Transformer variants (Flat, hierarchical, Soft Cascade) are described only briefly (App. B.2), and no source is cited for them, so it is unclear whether they are established methods or new designs. Since Q1 is essentially tabular classification of protein-like count features, established classifiers for such data (e.g., cytometry/ADT-style classifiers or recent tabular models) would be natural baselines, but none is evaluated. The paper describes the Q1 models as "representative model families", but this claim needs either standard baselines or a justification for the chosen models.

  1. The paper claims to be "the first evaluation suite", but it is unclear what is first.

  2. The three questions also carry limited weight. Q2 only rejects a weak null (that APT carries no cell-level signal once the true parent is given), which reads as a secondary analysis of Q1 rather than a headline benchmark task.

  3. Q3 is partly circular: targets are RNA-cluster labels and RNA is an input, so APT is structurally disadvantaged. Most Q3 contrasts are left "unresolved" because the study is underpowered.

Questions for authors

Clarifications that could change my assessment:

  1. What do the 293 aptamers bind (known protein targets or SELEX-derived aptamers with unknown targets), and how was the panel chosen? APT-only parent-level performance is low even for five coarse classes such as B/T/Myeloid. Can you provide per-aptamer QC or show agreement with canonical lineage markers measured in RNA? And how did you preprocessing the raw data?

  2. Where do the three Transformer variants come from? Q1 is essentially tabular classification of protein-like count features, so why were these variants chosen over established tabular models (e.g., FT-Transformer, CatBoost, TabPFN) or cytometry/ADT-style deep learning models/classifiers?

  3. In Q3, targets are RNA-cluster labels and RNA is also an input, which structurally favors RNA. Could Q3 be evaluated with labels that do not come from RNA alone?

  4. What specifically is "first" about AptamerBench?

Suggestions:

  1. Introduce aptamers and APT measurement in the main text. The current text assumes this background.

  2. Consider sample-level tasks (e.g., patient-level cell-type composition), noting the disease/acquisition confound.

3 comments

Thanks for your constructive feedback.

Data correction. The cohort contains 40 specimens from 38 donors; two HCC donors contributed two specimens each. We regrouped by donor and rebuilt the affected APT comparisons; all specimens from one donor stay in the same split. The APT estimates below replace the earlier sample-ID-based ones.

W1(d) / Q2: Established baselines and Transformer provenance. We trained LR, XGBoost, CatBoost and the standard FT-Transformer for five-parent and 27-subtype prediction. Each model/task/fold used four prespecified configurations selected on donor-disjoint validation data (40 selections, 100 final refits; three seeds per stochastic model). Pooled donor-balanced SB-Macro-F1 [95% CI]:

ModelFive-parent27-subtype
LR0.29048 [0.27086, 0.30903]0.13352 [0.11451, 0.14595]
XGBoost0.31748 [0.29518, 0.33854]0.13571 [0.11984, 0.15051]
CatBoost0.31463 [0.29189, 0.33611]0.12430 [0.11032, 0.13545]
Standard FT-Transformer0.32323 [0.30011, 0.34452]0.13778 [0.12228, 0.15036]

Intervals use 5,000 donor/seed bootstrap draws, conditional on the selections. Fine-level FT-minus-XGBoost is +0.00207 [−0.00569, +0.00738]. The new models span fine-level scores of 0.124–0.138, supporting the Q1 finding that ordinary subtype prediction is difficult for the tested models.

The three earlier Transformers are implementation-specific references sharing an APT encoder: Flat uses separate parent/subtype cross-entropy heads; the hierarchical reference adapts hierarchical cross-entropy; Soft Cascade uses predicted-parent soft routing. Their objectives also differ, so their scores do not isolate routing or loss. We will define them explicitly and cite the adapted components. The newly added official FT-Transformer (Gorishniy et al., 2021; package 0.0.2) keeps its original tokenizer and implementation. TabPFN and cytometry/ADT-style deep models were not evaluated during the rebuttal period; we plan to add them in subsequent work.

W1(c) / Q1 / Suggestion 5: APT measurement and probe QC. We will introduce aptamers as nucleic-acid affinity reagents and explain barcode-linked APT binding counts in the main text.

Our experimental team curated the 293-aptamer panel from the literature. It combines published aptamers from different laboratories, with documented sequences and reported binding or selection targets (e.g., CD8, CD20, CTLA-4). The panel list includes names, sequences, reported binding objects, source references and reported affinities where available. Some entries were selected against cells or other non-protein targets, and we will distinguish these from aptamers with identified protein targets. We will add this description to the main text and release the manifest and references after acceptance. Reported targets describe the evidence in the source studies; they do not make every probe a validated protein-abundance marker in this PBMC assay.

We completed QC of all 293 probes across 361,792 cells. Median probe detection is 82.06% (range 3.96–99.91%). The new tables report counts, quantiles, detection/zero rates, specimen-level distributions and APT/RNA depth associations. Detection measures coverage, not cell-type specificity.

For the RNA-marker comparison, we fixed six modules before analysis (B, T, NK, Myeloid, Platelet, Plasma; three genes each). Correlations were computed within each donor, adjusted for total counts and detected-feature counts in both modalities, then averaged across donors. As an exploratory example, APT-106 has mean partial Spearman 0.2063 with the LST1/TYROBP/FCER1G module, positive in all 38 donors. This is an association with an RNA module, not agreement between a probe and its reported target. Target-matched correlations (e.g., CD8-binding aptamers with CD8A, CD20-binding aptamers with MS4A1) have not yet been evaluated, so we do not claim the panel is validated against canonical lineage targets. We will include all 293×6 comparisons with donor-bootstrap intervals.

Raw-data processing. APT reads were linked to RNA profiles by cell barcode. The laboratory QC description specifies retaining cells with more than 500 and fewer than 6,500 detected RNA genes and less than 10% mitochondrial transcripts. The benchmark contains 361,792 annotated cells with all 293 APT count features; no additional probe or APT-count filtering was applied. The laboratory description reports CLR normalization for its upstream APT analysis, whereas our models start from raw APT counts, apply log(1+x), and fit standardization on training cells only. RNA-marker comparisons use library-normalized log expression. We will separate the laboratory workflow from benchmark preprocessing in Methods and release the processing documentation with the data.

W4 / Q3: RNA-derived labels and pairing benefit. We agree that RNA-derived labels can favor RNA when assessing modality benefit. Q3 therefore measures prediction of the specified annotation, not performance against an independent biological reference. Corrected 27-label pooled fine SB-Macro-F1 differences:

ContrastDifference [95% CI]
Paired − RNA+0.01490 [+0.00714, +0.02471]
Paired − specimen shuffle−0.00656 [−0.01928, +0.00968]
Paired − specimen×parent shuffle+0.00480 [+0.00036, +0.00990]
Paired − specimen context+0.00762 [+0.00028, +0.01420]

The corrected results show positive differences against some controls; the specimen-shuffle interval includes zero. We will report each comparison separately and remove the broad statement that pairing has no measurable benefit. With 15-class probability aggregation, paired−RNA is +0.00482 [−0.00190, +0.01372] and paired−specimen×parent shuffle is +0.00409 [+0.00067, +0.00878]. This is re-scoring, not retraining or independent-label validation, and we do not interpret zero-crossing intervals as equivalence.

W2 / Q4 and W3: What is new, and the role of Q2. “First” refers to the joint evaluation defined in the paper: prediction in unseen donors (Q1), subtype association beyond specimen/parent composition (Q2), and adding APT to RNA versus preserving exact same-cell pairing (Q3), together with within-donor scoring and correspondence retrieval. Earlier Apt-seq (Delley et al., Sci. Rep. 8:2919, 2018) established paired aptamer/transcriptome measurement, and SPARK-seq (Luo et al., Science 391:eadv6127, 2026) studies aptamer–target discovery and kinetics. Our claim concerns the benchmark and evaluation questions, not these measurement technologies. We will replace the broad “first evaluation suite” wording with this description.

Q1 asks whether APT predicts subtype in a new donor. Q2 asks whether real APT beats a retrained specimen×parent shuffle (which preserves specimen and parent composition) when both use the same true parent at decoding. Q2 quantifies the size of this conditional gain rather than only testing against zero: real APT scores 0.38112 and the shuffle 0.21843, a gain of +0.16270 [0.13087, 0.17625], or 16.27 absolute percentage points. It does not imply the same gain in ordinary prediction (APT-only MLP: 0.13811 without true-parent decoding), and we will report both together. Models were independently selected and retrained under alternative label definitions (20 selections, 120 refits):

Q2 settingSubtypes / parentsReal − shuffle [95% CI]
H0: original labels27 / 5+0.16270 [+0.13087, +0.17625]
H1: alternative parents27 / 6+0.13844 [+0.11034, +0.15380]
L1: coarser labels15 / 6+0.12052 [+0.10325, +0.13588]

The contrast remains positive in all settings; each is a different task, so effect sizes are interpreted within their own setting. The labels remain RNA-derived, and we have not evaluated independently measured labels or independently reviewed/high-confidence subsets. Retrieval tests cross-modal correspondence separately and is not a substitute. We will keep Q2 and Q3 claims within these tested settings.

W1(a): Data access and protection. Public release of the paired RNA/APT dataset has been approved and will take place after acceptance, so the full dataset is not yet available for reviewer inspection. We will add the repository link or accession and data-use conditions to the manuscript and correct the earlier “pending final ethics and data-use approval” wording. We will release the dataset, panel manifest with sequences and source references, probe-level QC tables, code, corrected specimen-to-donor registry, folds, label mappings and scoring definitions, so others can reproduce the benchmark and inspect the panel.

W1(b) / Suggestion 6: Clinical resource scale and evaluation scope. AptamerBench contains 361,792 paired RNA/APT cell profiles (293 APT features) from 40 specimens and 38 donors across five cancer groups and healthy controls. Collecting it requires clinical recruitment, fresh blood processing, paired profiling and annotation; RNA/ADT datasets measure antibody-derived tags and cannot replace these paired aptamer observations.

The large cell count does not increase the number of independent donors (7–8 test donors per fold). Donor-disjoint folds test new donors within this cohort; disease/acquisition-group holdout changes both together. Neither separates biological from acquisition effects or establishes cross-site APT generalization; the multicenter RNA/ADT experiment tests the evaluation procedure in another assay and cohort. Composition-based disease prediction would inherit the same confounding, so we will describe sample-level tasks as extensions needing appropriate sampling and acquisition controls.

Thank you for the detailed response. I appreciate the added baselines, the probe-level QC, and the revised wording.

However, assay validity remains unresolved. Target-matched checks (e.g., CD8-binding aptamers vs. CD8A, CD20-binding aptamers vs. MS4A1) have not been evaluated, and the response does not explain why APT-only performance is low even for coarse lineages such as B/T/Myeloid. In this state, the benchmark cannot distinguish whether "prediction is difficult" reflects a modeling or generalization problem or simply assay noise.

I therefore keep my rating.

Reviewer_q7mJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces the AptamerBench, a benchmark for cell typing from single-cell aptamer-binding (APT) profiles. Using AptamerBench, the paper performs multiple set of studies, revealing that APT-only fine-subtype performance is low. Under true-parent decoding, real APT exceeds a retrained patient-by-parent shuffle. The primary RNA+APT gain over RNA is small and disappears under joint disease/acquisition-group holdout, while label-independent retrieval detects same-cell correspondence.

Critical strengths
  1. Separating cross-patient transfer, conditional subtype association, within-patient discrimination, modality gain, and retrieval is a real strength. A single pooled score would blur these results together.

  2. The real and shuffled APT models are trained separately, with shuffling within patient and parent groups. Patient-disjoint folds and patient-level resampling fit the structure of the data.

  3. The Memory CD8T-1 versus NK example makes the value of patient-level metrics concrete: patient context can achieve about 0.87 pooled Macro-F1 while minority-class recall is zero in every patient. Within-patient results reveal a failure that the pooled score misses.

  4. The presentation of the paper is professional and the code is available.

Critical weaknesses
  1. 40 participants from one hospital provide limited independent units. Disease groups were acquired in different windows, so the joint group holdout cannot separate biological from acquisition effects or establish cross-site transport.

  2. The subtype labels come from RNA rather than an independent biological reference. This can favor RNA in Q3 and makes the magnitude of conditional APT signal dependent on annotation quality and ontology choices.

  3. The paired-minus-shuffle confidence intervals include zero, but some still admit useful positive effects. It would help to say what size of pairing benefit these intervals can actually rule out.

  4. Under the original folds, patient context matches paired RNA+APT closely; under joint group holdout, context falls below RNA and paired RNA+APT exceeds it by 0.02848 [0.01387, 0.04533]. This limits the generality of the primary context explanation.

Questions for authors
  1. What is the largest pairing-specific Macro-F1 gain still compatible with the reported intervals? What size of gain would matter in practice?

  2. Do the Q2 and Q3 results hold with independently reviewed labels, coarser labels, or only high-confidence annotations?

  3. Why does patient context track paired RNA+APT under ordinary patient folds but degrade under joint disease/acquisition-group holdout? Is this driven by particular held-out groups or cell types?

1 comment

Thanks for your constructive feedback.

Data correction. We identified 40 specimens from 38 donors: two HCC donors each contributed two specimens. We rebuilt the affected APT analyses with all specimens from each donor kept together in fit, validation or test. These results replace the earlier sample-ID-based estimates.

W3 / Q1: Size of pairing benefit. Q3 uses 25 validation selections and 75 fits across five conditions and three seeds. The 95% CIs use 5,000 paired donor/seed bootstrap draws, conditional on the validation selections. Paired-minus-control differences in pooled fine SB-Macro-F1:

ControlDifference [95% CI]
Specimen shuffle−0.00656 [−0.01928, +0.00968]
Specimen×parent shuffle+0.00480 [+0.00036, +0.00990]
Specimen context+0.00762 [+0.00028, +0.01420]

The intervals allow gains up to 0.97, 0.99 and 1.42 percentage points, respectively; larger gains fall outside these intervals for the tested models and controls. Smaller gains may still matter, depending on the application and cell types. We did not prespecify a minimum useful gain. We will update the abstract and conclusions to reflect the positive parent-shuffle/context contrasts and unresolved specimen-shuffle contrast.

W2 / Q2: Label dependence. For Q3, we summed the 27-class models' probabilities within the 15-class L1 groups, without retraining or changing shuffle groups. Pooled fine SB-Macro-F1 differences:

Contrast27 labels [95% CI]15 labels [95% CI]
Paired − RNA+0.01490 [0.00714, 0.02471]+0.00482 [−0.00190, 0.01372]
Paired − specimen×parent shuffle+0.00480 [0.00036, 0.00990]+0.00409 [0.00067, 0.00878]

The gain over RNA has a smaller point estimate and includes zero at 15 labels; the parent-shuffle contrast stays positive.

For Q2, we independently selected and retrained real/shuffled models under H1 and L1 on the same donor folds (20 selections, 120 refits). H1 changes the parents; L1 merges subtypes within H1 parents. True-parent fine SB-Macro-F1 contrasts:

SettingSubtypes/parentsReal − shuffle [95% CI]
H027/5+0.16270 [0.13087, 0.17625]
H127/6+0.13844 [0.11034, 0.15380]
L115/6+0.12052 [0.10325, 0.13588]

Q2 stays positive across these settings, but each defines a different task. Matching RNA inputs in Q3 and true-parent decoding in Q2 helps interpret the controls; it does not remove RNA-label dependence. We did not evaluate independently reviewed or high-confidence cell subsets. We will state this limitation.

W4 / Q3: Context under group holdout. We rebuilt group holdout with donor-disjoint inner validation. Pooled context-minus-RNA is −0.04812 [−0.07628, −0.02861]:

GroupContext − RNA [95% CI]
Normal−0.09218 [−0.12324, −0.04951]
GC−0.03600 [−0.06263, −0.00801]
CRC−0.02942 [−0.04479, +0.00280]
BTC/PC/HCC+0.00854/+0.00342/−0.00239 (intervals include zero)

Removing any one group leaves the pooled point estimate negative (−0.06518 to −0.01384). Largest class-F1 drops are Erythrocyte (−0.762; 46 cells, 4 donors), Naïve B-2 (−0.249) and Memory CD8T-1 (−0.244) in Normal; Naïve CD4T-2 (−0.217) and Naïve CD8T (−0.139) in GC. These are descriptive three-seed averages, not class-level significance tests.

Context is close to paired under ordinary folds, with paired slightly better, but this pattern does not extend to group holdout. We will restrict the explanation accordingly and add the decompositions to the appendix.

With 27 labels, group-holdout paired−context is +0.03709 [0.02046, 0.05650], whereas paired−RNA is −0.01103 [−0.02536, 0.00191]. Paired exceeds context but its interval against RNA includes zero. The paired−context advantage persists with 15 labels: +0.03054 [0.01236, 0.05166].

W1: Cohort scope. The 38 donors come from one hospital. Disease and acquisition windows are confounded, so the data cannot identify biological versus acquisition effects. Ordinary folds test new donors within this cohort; group holdout changes disease and acquisition together. Neither establishes cross-site APT generalization. Composition-based explanations also remain unproven. We will state these limits explicitly. Cross-site validation requires another APT cohort; the multicenter RNA/ADT study tests the evaluation procedure on another assay.

Reviewer_pgy9OpenReview ↗
Rating
1
Confidence
3
Informativeness
3
Presentation
2
Soundness
1
Read review
Paper summary

AptamerBench presents an evaluation framework for paired single-cell RNA and aptamer-binding measurements from 361,792 cells across 40 participants, with 293 APT features and an RNA-derived hierarchy of five lineages and 27 subtypes. The benchmark separates held-out-subject APT-only prediction, conditional subtype signal under true-parent decoding, modality contribution, the value of exact RNA-APT pairing, and label-independent cross-modal retrieval. It combines patient-disjoint splits, subject-balanced metrics, matched shuffling controls, group-holdout sensitivity analyses, and exploratory validation on an external RNA-ADT cohort. The main empirical picture is that ordinary APT-only fine-subtype transfer to unseen patients is weak, whereas a positive conditional contrast appears when the true parent class is supplied. Exact-pairing gains for classification are not consistently established, although held-out-patient retrieval detects cross-modal correspondence.

Critical strengths
  1. The benchmark makes a valuable conceptual distinction among ordinary prediction, privileged conditional decoding, modality increments, pairing-specific effects, and label-independent retrieval. The patient-disjoint and subject-balanced design is well motivated and exposes conclusions that would be obscured by pooled cell-level evaluation.

  2. The empirical study includes extensive matched controls and sensitivity analyses, including paired bootstrap intervals, alternative label hierarchies, joint disease and acquisition-group holdout, and an external cohort. The manuscript is also unusually explicit about several limitations of its positive results.

Critical weaknesses
  1. The strongest positive findings do not establish a practically deployable prediction result or a biologically specific pairing effect. The Q2 contrast supplies the true parent class at decoding time, while ordinary unknown-parent fine-subtype prediction remains weak. The retrieval experiments demonstrate detectable correspondence under controlled candidate construction, but they cannot determine whether that signal is cell-specific biology rather than cell type, cell state, sample context, or shared technical variation. Moreover, the QC-matched candidate sets are centered on the true pair, making their difficulty and null behavior different from the original random-candidate experiment. These results support diagnostic association and cross-modal correspondence, but not the stronger practical or mechanistic interpretation needed for the paper's central contribution.

  2. The uncertainty reporting and cross-analysis synthesis are incomplete. Table 18 reports ordinary pairing gains of +0.015 and +0.007 without separate confidence intervals; the reported -0.009 interval concerns the difference between feature-budget effects and does not quantify either gain. The external Stephenson analysis also mixes endpoints with materially different conclusions: pooled intervals include zero, whereas the within-subject paired-versus-subject-shuffled contrast is +0.010 with a 95% confidence interval of [0.003, 0.017]. Describing this only as reproducing the same overall pattern obscures which estimand transfers and prevents a clean assessment of robustness.

Questions for authors
  1. Please state the exact deployable claim supported by Q2 when the true parent class is supplied, and report the directly corresponding ordinary unknown-parent result needed to evaluate practical subtype prediction.

  2. Please provide separate confidence intervals for the +0.015 and +0.007 effects in Table 18 and map every reported interval to its estimand. Please also explain why the external pooled and within-subject endpoints differ, explicitly including the +0.010 [0.003, 0.017] paired-versus-subject-shuffled result.

  3. Please distinguish the combinatorial 1/64 top-1 probability of a uniform random ranking from the empirical null performance of a model evaluated on truth-centered QC-matched candidate sets. If available, please report retrieval with same-subtype and same-sample distractors; otherwise, state clearly that the present analysis does not establish biological specificity.

1 comment

Thanks for your constructive feedback.

Data correction. The 40 specimens come from 38 donors (two HCC donors contributed two specimens each). We regrouped by donor and reran preprocessing, validation selection, fits and evaluations. The donor-corrected APT results below replace the earlier sample-ID-based estimates. The Q2 contrast is essentially unchanged (+0.160 [0.132, 0.172] before; +0.163 [0.131, 0.176] now).

W1 / Q1: Q2 claim and ordinary prediction. Q2 tests whether real APT carries subtype-associated information beyond a matched control that preserves specimen and parent composition. Fine SB-Macro-F1, equal weight per donor:

ConditionFine SB-Macro-F1
APT-only, parent unknown0.13811
True-parent decoding, specimen×parent-shuffled APT0.21843
True-parent decoding, real APT0.38112

Real APT exceeds the independently fitted shuffled control by +0.16270 [0.13087, 0.17625], both using the same true parent at decoding. So when the true parent is supplied, real APT carries more subtype-associated information than the matched shuffle. Ordinary prediction without the true parent scores 0.13811 and answers a different question. We will state this distinction in the abstract, introduction and conclusions, and will not interpret Q2 as evidence of improved ordinary prediction or a biological mechanism.

W2 / Q2: Intervals and external estimands. In the external RNA/ADT cohort (Table 18, unaffected by the donor correction), we compare paired RNA+ADT with RNA+ADT whose ADT is shuffled within donor×sample×parent strata, in subject-balanced Macro-F1:

RNA featuresDifference [95% CI]
2,000+0.01515 [0.00298, 0.02609]
5,000+0.00660 [−0.00676, 0.01840]

The 5k−2k difference (−0.00855 [−0.02319, 0.00302]) compares feature budgets and does not replace either interval. We will add both intervals to Table 18 and label all three estimands.

In Stephenson, paired RNA+ADT versus RNA+ADT with ADT shuffled within subjects:

EndpointDifference [95% CI]
Pooled subject-balanced Macro-F1+0.00356 [−0.00673, 0.01300]
Within-subject (support-aware) Macro-F1+0.01041 [0.00305, 0.01715]

These are different estimands. Pooled scoring computes F1 after combining patient-normalized confusion matrices; within-subject scoring computes supported-class F1 per subject, then averages. The pooled interval includes zero and the within-subject interval is positive. We will replace “same overall pattern” with this endpoint-specific account.

Q3 (W1): Retrieval with same-specimen, same-subtype distractors. We independently selected and refitted full-profile and QC-only models on the corrected donor folds (three training seeds; shuffled models cross these with three shuffle realizations). Model selection uses same-specimen validation candidates without subtype labels; subtype labels define only the additional evaluation strata.

At K=64, each query has one true partner and 63 distinct distractors from the same specimen and operational subtype (353,870/361,792 cells, 97.81%; 38 donor groups). Real and shuffle models use identical queries and candidate sets.

DirectionReal Recall@1Shuffle Recall@1Difference [95% CI]
RNA→APT0.042860.01551+0.02734 [0.02261, 0.03271]
APT→RNA0.043130.01553+0.02760 [0.02214, 0.03391]

Scores pool queries across each donor's specimens, then average donors equally; 95% CIs use 5,000 paired donor/seed bootstrap draws.

Under uniform random ranking, expected Recall@1 is 1/64 = 0.015625; the learned shuffle models reach about 0.0155 here. We distinguish these quantities and will describe the earlier truth-centered QC-matched analysis as a candidate-conditioned stress test.

QC-only real−shuffle contrasts are +0.00687 [0.00562, 0.00809] (RNA→APT) and +0.00781 [0.00639, 0.00936] (APT→RNA). The direct contrast (full-profile real−shuffle) − (QC-only real−shuffle) is +0.02047 [0.01552, 0.02591] (RNA→APT) and +0.01978 [0.01423, 0.02570] (APT→RNA), so the full-profile contrast exceeds the tested QC-only baseline in both directions.

Retrieval supports detectable correspondence conditional on subtype and specimen. It does not establish biological specificity; cell-state and technical contributions remain possible. Results for K=16/32/64/128 will go to the appendix.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.