- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce SubcellPert, a benchmark for predicting how RNA abundance and its spatial distribution within cells change across perturbations or conditions. They bring together 14 imaging datasets containing about 13.9 million cells and 3.65 billion RNA molecules. They test their models on held-out cellular contexts or perturbations. In the Oracle setting, models receive target RNA counts and predict locations; in the Joint setting, they predict both. The authors evaluate eight baselines, including five frozen foundation encoders with a shared prediction head. Simple transport methods are often competitive, and the results show why spatial error needs to be interpreted alongside abundance accuracy and gene coverage.
Critical strengths
-
The problem is well motivated: perturbations can affect both RNA abundance and where RNA is located within a cell. Bringing 14 imaging datasets into a common benchmark is a useful contribution.
-
The Oracle and Joint settings help separate errors in count prediction from errors in localization. The authors also clearly show how low spatial error can coexist with poor abundance prediction or missing genes.
Critical weaknesses
-
The spatial score can miss important errors within individual cells. Equation 15 pools each gene’s coordinates across cells and compares these distributions separately. For example, two genes could be together in every target cell but separated in every predicted cell, while their pooled distributions remain unchanged. The spatial score would not detect this error. As a result, low spatial error does not by itself establish that a model has recovered how RNA is organized within individual cells. This limits the conclusions that can be drawn from the current evaluation.
-
The neural comparison does not isolate the value of the pretrained encoders. All five models use frozen encoders, the same spatial prediction head, and five epochs of training. The head permits only limited affine changes to the source RNA coordinates (Equation 8; Appendix E.4). The common contraction reported in Section 4.2 could therefore reflect the shared prediction setup. Without an otherwise identical model that omits pretrained features, it remains unclear how much the encoders contribute. The authors appropriately restrict their claims to this protocol, but an additional control is needed to interpret these results.
Questions for authors
-
Evaluation validation: Can the authors provide the fixed-count spatial-sensitivity check mentioned in Appendix F, including a case where pooled gene-wise distributions are preserved but colocalization within cells changes? Which aspects of within-cell organization is the benchmark intended to validate?
-
Baseline validation: How does the shared adapter perform without pretrained encoder features? How much of the reported spatial improvement can be reproduced by simply contracting the source coordinates by a fixed factor?
-
Reproducibility clarification: Are the processed data, exact train/validation/test splits, evaluator, and baseline code available to reviewers? I could not locate a link in the PDF. Please also point to the supporting details for the contraction and checkpoint-selection audits reported in Section 4.2.
6 comments
Response to Reviewer x86q (2/4)
W2. The neural comparison does not isolate pretrained encoders from the shared prediction setup
We agree that contraction cannot be attributed to the frozen encoders alone. The shared head is initialized at identity and uses A = I + 0.1 tanh(R); diagonal entries approach 0.90 at their lower limit. Both this restricted parameterization and random-pair spatial regression can contribute to the observed behavior.
We have now retrained an adapter without pretrained features in both tasks, preserving the other inputs, architecture, optimization budget, and selection rules. It zeros both the 512 pretrained-state channels and the availability channel, retains the 513-channel interface, and does not load pretrained weights or vocabulary. Oracle results appear under Q2; the Joint abundance comparison is:
| Dataset | Model | Joint MSE ↓ | PCC-delta ↑ | Coverage ↑ |
|---|---|---|---|---|
| PF-Astro | scGPT | 0.006366 | 0.036872 | 0.998048 |
| PF-Astro | Adapter-only | 0.010896 | 0.040383 | 0.993343 |
| Adipose | scGPT | 0.009092 | 0.782610 | 0.896774 |
| Adipose | Adapter-only | 0.008131 | 0.780373 | 0.892473 |
| ATM-RT | scGPT | 0.089611 | 0.249681 | 0.986577 |
| ATM-RT | Adapter-only | 0.095901 | 0.222877 | 1.000000 |
The effects are mixed. PF-Astro's full scGPT and SpatialFormer models select the epoch-zero checkpoint, reproducing source-preserving resampling; adapter-only selects epoch 2. Thus, their lower PF-Astro MSE is not evidence of a learned pretrained-feature advantage. The control establishes the performance recoverable without pretrained inputs under this adapter protocol. It does not isolate pretraining from encoder architecture, alignment, or the availability indicator; no random-encoder control was run.
We separately replaced random-pair spatial regression with condition/context-grouped MMD, keeping the scGPT encoder, affine head, budget, and checkpoint-selection rule fixed. Oracle W1 changes from 0.249710 to 0.249465 on PF-Astro, 0.303159 to 0.297894 on Adipose, and 0.095904 to 0.085372 on ATM-RT. On ATM-RT, radial and pair errors also improve, but whole-cell W2² worsens from 0.000406587 to 0.000471035 and centroid RMSE from 0.071900 to 0.075034. Radial error worsens on PF-Astro and Adipose. This minibatch MMD uses a biased estimator including singleton groups, so it tests one distributional objective implementation. We have not tested a wider head or longer training, and these comparisons do not identify the cause of contraction.
Response to Reviewer x86q (3/4)
Q1. Fixed-count sensitivity validation, including preserved pooled marginals
We have executed both the fixed-count gene-location swap and the preserved-marginals counterexample with the evaluator used for this experiment wave. For these two-cell diagnostic constructions, the pair-support threshold is one cell; benchmark comparisons retain the five-cell threshold. Appendix F.4 and Table 10 now include the measured validation and both constructions.
| Executed synthetic check | Count MSE | Primary W1 | Whole-cell W2² | Mean pair W1 |
|---|---|---|---|---|
| Identity | 0 | 0 | 0 | 0 |
| Gene-location swap, counts fixed | 0 | 1.234976 | 0.350435 | 0 |
| Cross-gene arrangement changed, marginals fixed | 0 | 0 | 0.145822 | 1.333333 |
The gene-location swap preserves per-cell counts and the unmarked RNA cloud while changing gene marginals. Its pair-distance score remains zero because that construction preserves pair separations; it is a positive sensitivity check for marginal localization, not a universal check for all spatial diagnostics.
For the preserved-marginals construction, each cell contains one molecule of each gene, all at y = 0:
| Population | Cell 1 x coordinates: (A, B, C, D) | Cell 2 x coordinates: (A, B, C, D) |
|---|---|---|
| Target | (+1, +1, −1, −1) | (−1, −1, +1, +1) |
| Prediction | (+1, −1, +1, −1) | (−1, +1, −1, +1) |
Every cell has centroid zero and RMS radius one, and every gene's pooled coordinate distribution is preserved. The primary metric correctly returns zero marginal error, while A–B separation W1 is 2, mean separation W1 over all six pairs is 1.333333, and whole-cell feature W2² is 0.145822. Thus, the construction survives coordinate normalization and explicitly demonstrates the primary metric's blind spot. These are executed synthetic checks, not fitted-model results. The revised manuscript describes the validated scope as marginal localization, supplemented by finite-feature and gene-pair checks of dependence and heterogeneity.
Q2. Adapter without pretrained features and a fixed-contraction baseline
Both controls are complete. The adapter-only model was retrained, as described under W2. The fixed-contraction control applies r_pred = 0.90 × r_resampling to the same sealed Oracle count-matched resampling coordinates. Exact counts, source and coordinate draws, seeds, and evaluation frames are unchanged. Predictions are neither recentered nor renormalized after scaling. The factor 0.90 was fixed from the reported audit, not tuned on test outcomes; scaling by 1 reproduces the reference scores.
| Dataset | Resampling W1 ↓ | Fixed 0.90 W1 ↓ | scGPT W1 ↓ | SpatialFormer W1 ↓ | Adapter-only W1 ↓ |
|---|---|---|---|---|---|
| PF-Astro | 0.253889 | 0.248966 | 0.249710 | 0.249855 | 0.249966 |
| Adipose | 0.311341 | 0.295843 | 0.303159 | 0.301087 | 0.297296 |
| ATM-RT | 0.085586 | 0.096138 | 0.095904 | 0.095954 | 0.095982 |
Adapter-only changes Oracle W1 relative to scGPT by +0.10%, −1.93%, and +0.08%; there is no consistent pretrained-feature advantage here. For PF-Astro, fixed contraction achieves 117.8% and 122.1% of the scGPT and SpatialFormer W1 improvements over resampling; for Adipose, the corresponding ratios are 189.4% and 151.1%. We calculate these as (W1_resampling − W1_fixed) / (W1_resampling − W1_neural). Values above 100% mean the fixed control improves the metric more than the neural model; these are descriptive error-gap comparisons, not a causal decomposition. On ATM-RT both neural references and fixed contraction are worse than resampling, so a fraction of positive neural gain is undefined.
The additional error measures show why contraction is not uniformly beneficial:
| Dataset | Radial W1: resampling → fixed 0.90 ↓ | Centroid RMSE: resampling → fixed 0.90 ↓ |
|---|---|---|
| PF-Astro | 0.141238 → 0.161031 | 0.275876 → 0.262182 |
| Adipose | 0.251959 → 0.239098 | 0.340681 → 0.320979 |
| ATM-RT | 0.074034 → 0.100143 | 0.075154 → 0.071539 |
On ATM-RT, fixed contraction also worsens pair W1 from 0.154131 to 0.175531, despite improving whole-cell feature W2² from 0.000477174 to 0.000406257. The control can match the primary-metric improvement on some datasets, but neither a centroid improvement nor one favorable distributional score establishes recovery of intracellular organization.
Response to Reviewer x86q (1/4)
Thank you for the concrete requests for metric validation, model controls, and reproduction resources. We have completed the additional controls on PF-Astro, Adipose, and ATM-RT under their frozen seed-1 protocols. Neural models completed five epochs at batch size 256; PF-Astro/Adipose use the best validation checkpoint including initialization, and ATM-RT uses the final epoch under its original no-validation protocol. Results below are descriptive, use the all-gene panel, and weight responses equally within each dataset.
W1. The spatial score can miss important within-cell errors
We agree. Equation 15 measures gene-wise marginal localization accuracy after unit/cell-balanced pooling. Cell balancing addresses unequal contributions to that marginal measure; it does not preserve cross-gene dependence or fully describe cell-to-cell heterogeneity. We have narrowed the metric claims in Section 3.5 and the conclusion, distinguishing complete-cell generation from validation of complete intracellular organization.
We have added two complementary diagnostics to the saved predictions: sliced W2² between distributions of complete-cell spatial feature vectors, and W1 between distributions of within-cell gene-centroid separations. The first concatenates 29 moment/Fourier features per gene across the full vocabulary, excludes the explicit count feature, and uses native feature scales with 64 fixed normalized random directions (seed 42). The second uses 496 pairs among 32 genes selected by vocabulary-name hashing, retaining units with at least five coexpressing cells. Both compare unpaired populations, balancing units and cells; pair scores average supported pairs and leave unsupported pairs missing.
ATM-RT illustrates a ranking disagreement: scGPT has lower whole-cell feature W2² but higher marginal and pair errors. These diagnostics capture additional aspects of organization, not a complete validation of it. Oracle holds exact cell-by-gene counts and pair support fixed: supported pairs average 212/496 on PF-Astro (minimum 136), 78/496 on Adipose, and 476/496 on ATM-RT. Joint diagnostics also depend on predicted gene presence, because it affects absent-gene feature filling and coexpression support; we therefore use Oracle for this controlled spatial comparison.