- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces AptamerBench, a single-site cohort of 361,792 PBMCs from 40 participants (five different phenotype plus healthy). Each cell has paired scRNA-seq and 293 aptamer-binding (APT) count features. Targets are 27 RNA-cluster-derived subtypes, grouped into five "operational" parents (B, T, Myeloid, CD8T/NK, Other). All evaluation uses five disease-stratified, patient-level folds. The paper poses three questions:
Q1: APT-only prediction of parent/subtype in unseen patients, using LR, XGBoost, an MLP, and three custom Transformer variants.
Q2: Whether APT keeps subtype signal after controlling for patient and parent. The true parent is given at decoding, and real APT is compared with a model retrained on APT permuted within patient×parent strata.
Q3: Whether adding APT to RNA helps, and whether any gain depends on exact same-cell pairing (RNA vs RNA+APT vs grouped-shuffled APT vs patient-context controls). This is complemented by a label-free RNA↔APT retrieval audit.
Main findings: (i) APT alone does not predict cell types well in unseen patients.
(ii) APT nonetheless carries some cell-level subtype signal beyond patient and parent composition: when the true parent is given, real APT outperforms a control in which APT profiles are shuffled among cells of the same patient and parent.
(iii) Adding APT to RNA does not consistently improve cell-type prediction. Replacing each cell's own APT profile with that of another cell from the same patient gives the same performance, so pairing each cell's RNA with its own APT gives no measurable classification benefit.
(iv) A label-free retrieval test, which matches each cell's RNA profile to its APT profile among other cells of the same held-out patient, succeeds above chance. RNA and APT therefore share cell-level correspondence even though this correspondence does not improve cell-type prediction.
Critical strengths
-
Patient-aware evaluation. The paper shows that cell-level performance pooled across patients can be misleading, and it adds subject-balanced and within-patient metrics to expose this. Single-cell annotation benchmarks rarely report within-patient metrics, so this practice is useful beyond this dataset.
-
A new data modality: single-cell aptamer profiles paired with RNA in a clinical cohort. If released, this data would be a useful resource.
Critical weaknesses
- Limited value as a benchmark.
(a) The data are not available. The paired data are protected and would be deposited "after acceptance, pending final ethics and data-use approval", so nobody can currently use or check the benchmark.
(b) The scale is small and confounded: 40 participants from one site, 8 test patients per fold. The authors state that disease and control groups were collected in different acquisition windows, so disease is fully confounded with batch. This limits sample-level tasks and any claim about generalization.
(c) The measured modality is not described. The paper never says what the 293 aptamers bind, how the panel was chosen, or whether the targets are known.
(d) The baselines are limited. The three Transformer variants (Flat, hierarchical, Soft Cascade) are described only briefly (App. B.2), and no source is cited for them, so it is unclear whether they are established methods or new designs. Since Q1 is essentially tabular classification of protein-like count features, established classifiers for such data (e.g., cytometry/ADT-style classifiers or recent tabular models) would be natural baselines, but none is evaluated. The paper describes the Q1 models as "representative model families", but this claim needs either standard baselines or a justification for the chosen models.
-
The paper claims to be "the first evaluation suite", but it is unclear what is first.
-
The three questions also carry limited weight. Q2 only rejects a weak null (that APT carries no cell-level signal once the true parent is given), which reads as a secondary analysis of Q1 rather than a headline benchmark task.
-
Q3 is partly circular: targets are RNA-cluster labels and RNA is an input, so APT is structurally disadvantaged. Most Q3 contrasts are left "unresolved" because the study is underpowered.
Questions for authors
Clarifications that could change my assessment:
-
What do the 293 aptamers bind (known protein targets or SELEX-derived aptamers with unknown targets), and how was the panel chosen? APT-only parent-level performance is low even for five coarse classes such as B/T/Myeloid. Can you provide per-aptamer QC or show agreement with canonical lineage markers measured in RNA? And how did you preprocessing the raw data?
-
Where do the three Transformer variants come from? Q1 is essentially tabular classification of protein-like count features, so why were these variants chosen over established tabular models (e.g., FT-Transformer, CatBoost, TabPFN) or cytometry/ADT-style deep learning models/classifiers?
-
In Q3, targets are RNA-cluster labels and RNA is also an input, which structurally favors RNA. Could Q3 be evaluated with labels that do not come from RNA alone?
-
What specifically is "first" about AptamerBench?
Suggestions:
-
Introduce aptamers and APT measurement in the main text. The current text assumes this background.
-
Consider sample-level tasks (e.g., patient-level cell-type composition), noting the disease/acquisition confound.
3 comments
W4 / Q3: RNA-derived labels and pairing benefit. We agree that RNA-derived labels can favor RNA when assessing modality benefit. Q3 therefore measures prediction of the specified annotation, not performance against an independent biological reference. Corrected 27-label pooled fine SB-Macro-F1 differences:
| Contrast | Difference [95% CI] |
|---|---|
| Paired − RNA | +0.01490 [+0.00714, +0.02471] |
| Paired − specimen shuffle | −0.00656 [−0.01928, +0.00968] |
| Paired − specimen×parent shuffle | +0.00480 [+0.00036, +0.00990] |
| Paired − specimen context | +0.00762 [+0.00028, +0.01420] |
The corrected results show positive differences against some controls; the specimen-shuffle interval includes zero. We will report each comparison separately and remove the broad statement that pairing has no measurable benefit. With 15-class probability aggregation, paired−RNA is +0.00482 [−0.00190, +0.01372] and paired−specimen×parent shuffle is +0.00409 [+0.00067, +0.00878]. This is re-scoring, not retraining or independent-label validation, and we do not interpret zero-crossing intervals as equivalence.
W2 / Q4 and W3: What is new, and the role of Q2. “First” refers to the joint evaluation defined in the paper: prediction in unseen donors (Q1), subtype association beyond specimen/parent composition (Q2), and adding APT to RNA versus preserving exact same-cell pairing (Q3), together with within-donor scoring and correspondence retrieval. Earlier Apt-seq (Delley et al., Sci. Rep. 8:2919, 2018) established paired aptamer/transcriptome measurement, and SPARK-seq (Luo et al., Science 391:eadv6127, 2026) studies aptamer–target discovery and kinetics. Our claim concerns the benchmark and evaluation questions, not these measurement technologies. We will replace the broad “first evaluation suite” wording with this description.
Q1 asks whether APT predicts subtype in a new donor. Q2 asks whether real APT beats a retrained specimen×parent shuffle (which preserves specimen and parent composition) when both use the same true parent at decoding. Q2 quantifies the size of this conditional gain rather than only testing against zero: real APT scores 0.38112 and the shuffle 0.21843, a gain of +0.16270 [0.13087, 0.17625], or 16.27 absolute percentage points. It does not imply the same gain in ordinary prediction (APT-only MLP: 0.13811 without true-parent decoding), and we will report both together. Models were independently selected and retrained under alternative label definitions (20 selections, 120 refits):
| Q2 setting | Subtypes / parents | Real − shuffle [95% CI] |
|---|---|---|
| H0: original labels | 27 / 5 | +0.16270 [+0.13087, +0.17625] |
| H1: alternative parents | 27 / 6 | +0.13844 [+0.11034, +0.15380] |
| L1: coarser labels | 15 / 6 | +0.12052 [+0.10325, +0.13588] |
The contrast remains positive in all settings; each is a different task, so effect sizes are interpreted within their own setting. The labels remain RNA-derived, and we have not evaluated independently measured labels or independently reviewed/high-confidence subsets. Retrieval tests cross-modal correspondence separately and is not a substitute. We will keep Q2 and Q3 claims within these tested settings.
W1(a): Data access and protection. Public release of the paired RNA/APT dataset has been approved and will take place after acceptance, so the full dataset is not yet available for reviewer inspection. We will add the repository link or accession and data-use conditions to the manuscript and correct the earlier “pending final ethics and data-use approval” wording. We will release the dataset, panel manifest with sequences and source references, probe-level QC tables, code, corrected specimen-to-donor registry, folds, label mappings and scoring definitions, so others can reproduce the benchmark and inspect the panel.
W1(b) / Suggestion 6: Clinical resource scale and evaluation scope. AptamerBench contains 361,792 paired RNA/APT cell profiles (293 APT features) from 40 specimens and 38 donors across five cancer groups and healthy controls. Collecting it requires clinical recruitment, fresh blood processing, paired profiling and annotation; RNA/ADT datasets measure antibody-derived tags and cannot replace these paired aptamer observations.
The large cell count does not increase the number of independent donors (7–8 test donors per fold). Donor-disjoint folds test new donors within this cohort; disease/acquisition-group holdout changes both together. Neither separates biological from acquisition effects or establishes cross-site APT generalization; the multicenter RNA/ADT experiment tests the evaluation procedure in another assay and cohort. Composition-based disease prediction would inherit the same confounding, so we will describe sample-level tasks as extensions needing appropriate sampling and acquisition controls.
Thank you for the detailed response. I appreciate the added baselines, the probe-level QC, and the revised wording.
However, assay validity remains unresolved. Target-matched checks (e.g., CD8-binding aptamers vs. CD8A, CD20-binding aptamers vs. MS4A1) have not been evaluated, and the response does not explain why APT-only performance is low even for coarse lineages such as B/T/Myeloid. In this state, the benchmark cannot distinguish whether "prediction is difficult" reflects a modeling or generalization problem or simply assay noise.
I therefore keep my rating.
Thanks for your constructive feedback.
Data correction. The cohort contains 40 specimens from 38 donors; two HCC donors contributed two specimens each. We regrouped by donor and rebuilt the affected APT comparisons; all specimens from one donor stay in the same split. The APT estimates below replace the earlier sample-ID-based ones.
W1(d) / Q2: Established baselines and Transformer provenance. We trained LR, XGBoost, CatBoost and the standard FT-Transformer for five-parent and 27-subtype prediction. Each model/task/fold used four prespecified configurations selected on donor-disjoint validation data (40 selections, 100 final refits; three seeds per stochastic model). Pooled donor-balanced SB-Macro-F1 [95% CI]:
Intervals use 5,000 donor/seed bootstrap draws, conditional on the selections. Fine-level FT-minus-XGBoost is +0.00207 [−0.00569, +0.00738]. The new models span fine-level scores of 0.124–0.138, supporting the Q1 finding that ordinary subtype prediction is difficult for the tested models.
The three earlier Transformers are implementation-specific references sharing an APT encoder: Flat uses separate parent/subtype cross-entropy heads; the hierarchical reference adapts hierarchical cross-entropy; Soft Cascade uses predicted-parent soft routing. Their objectives also differ, so their scores do not isolate routing or loss. We will define them explicitly and cite the adapted components. The newly added official FT-Transformer (Gorishniy et al., 2021; package 0.0.2) keeps its original tokenizer and implementation. TabPFN and cytometry/ADT-style deep models were not evaluated during the rebuttal period; we plan to add them in subsequent work.
W1(c) / Q1 / Suggestion 5: APT measurement and probe QC. We will introduce aptamers as nucleic-acid affinity reagents and explain barcode-linked APT binding counts in the main text.
Our experimental team curated the 293-aptamer panel from the literature. It combines published aptamers from different laboratories, with documented sequences and reported binding or selection targets (e.g., CD8, CD20, CTLA-4). The panel list includes names, sequences, reported binding objects, source references and reported affinities where available. Some entries were selected against cells or other non-protein targets, and we will distinguish these from aptamers with identified protein targets. We will add this description to the main text and release the manifest and references after acceptance. Reported targets describe the evidence in the source studies; they do not make every probe a validated protein-abundance marker in this PBMC assay.
We completed QC of all 293 probes across 361,792 cells. Median probe detection is 82.06% (range 3.96–99.91%). The new tables report counts, quantiles, detection/zero rates, specimen-level distributions and APT/RNA depth associations. Detection measures coverage, not cell-type specificity.
For the RNA-marker comparison, we fixed six modules before analysis (B, T, NK, Myeloid, Platelet, Plasma; three genes each). Correlations were computed within each donor, adjusted for total counts and detected-feature counts in both modalities, then averaged across donors. As an exploratory example, APT-106 has mean partial Spearman 0.2063 with the LST1/TYROBP/FCER1G module, positive in all 38 donors. This is an association with an RNA module, not agreement between a probe and its reported target. Target-matched correlations (e.g., CD8-binding aptamers with CD8A, CD20-binding aptamers with MS4A1) have not yet been evaluated, so we do not claim the panel is validated against canonical lineage targets. We will include all 293×6 comparisons with donor-bootstrap intervals.
Raw-data processing. APT reads were linked to RNA profiles by cell barcode. The laboratory QC description specifies retaining cells with more than 500 and fewer than 6,500 detected RNA genes and less than 10% mitochondrial transcripts. The benchmark contains 361,792 annotated cells with all 293 APT count features; no additional probe or APT-count filtering was applied. The laboratory description reports CLR normalization for its upstream APT analysis, whereas our models start from raw APT counts, apply log(1+x), and fit standardization on training cells only. RNA-marker comparisons use library-normalized log expression. We will separate the laboratory workflow from benchmark preprocessing in Methods and release the processing documentation with the data.