- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
CellUMM is a unified cell–language model for gene function annotation, cell type annotation, conditional cell generation, and perturbation response prediction. It combines a pretrained cell encoder, continuous cell representations, and a shared language-model backbone, using semantic prediction objectives and latent flow matching for joint training. The paper evaluates the framework across all four tasks and investigates data scaling and cross-task transfer.
Critical strengths
- The paper addresses a relevant problem: connecting semantic interpretation of cellular states with quantitative expression generation.
- The architecture provides a coherent framework for processing text and cell representations, with shared parameters across annotation and generation tasks.
- The experiments cover complementary tasks and include a conditional-mean baseline. The task-removal ablation provides preliminary evidence that annotation supervision can benefit perturbation prediction.
Critical weaknesses
W1. The current results do not sufficiently support the claimed effectiveness in perturbation prediction and distribution generation.
For perturbation prediction, all methods achieve average PDS values close to the random-ranking expectation of 0.5, with no clear improvement after fine-tuning. Although CellUMM improves some direction-based metrics, the current results do not demonstrate reliable perturbation-specific discrimination. Describing the PDS results as “close to saturation” therefore appears inappropriate. Simple baselines, such as the control mean and the average perturbation effect, would help establish the practical significance of the reported improvements. For conditional generation, the MMD² improvement over the conditional-mean baseline lacks uncertainty estimates and calibration against distances between real samples, while the conditional-mean baseline performs better on the mean-expression error metrics. The reported HVG metrics alone also do not establish that within-condition variability is well recovered. Consequently, the claim that the model reproduces the conditional distribution as a whole requires stronger evidence.
W2. The evaluation splits and the stability of the reported results require clarification.
The appendix explicitly states that SciPlex3 is evaluated on its validation set, and the reported GO evaluation sample counts also correspond to the validation split. For conditional generation, it is unclear whether the main results use validation conditions or held-out test conditions. If these validation sets were also used for checkpoint selection, the reported performance could be affected by model selection. The paper should identify the evaluation split and checkpoint-selection procedure for each task and provide independent test results. In addition, the cell type annotation test set contains only 200 cells spanning multiple classes and a small number of donors, providing limited evidence about the robustness of fine-grained annotation. Paired comparisons, donor-level uncertainty estimates, and repeated runs are not reported. The low performance of some baselines also warrants a clearer description of input adaptation, output parsing, and label mapping.
W3. The ablations provide insufficient support for the central claim of mutual benefit across the four tasks.
The main evidence is that removing cell type annotation reduces some metrics after fine-tuning on genetic perturbation prediction. This provides preliminary support for cross-task transfer, but does not establish mutual benefit across all four tasks or show how joint training affects the two semantic tasks. In particular, gene function annotation receives only gene and GO text, and the current comparisons cannot distinguish the contributions of the language model’s pretrained knowledge, task-specific GO supervision, and cross-task transfer. Single-task controls using the same architecture and comparable training budgets—particularly Qwen2.5-7B fine-tuned only on the GO task—together with more complete task-removal ablations would help establish the benefits of unified training. Uncertainty estimates across repeated runs would also help assess whether the observed gains are stable.
W4. The relationship between the perturbation metric definitions and the reported results requires clarification.
Appendix D.3 defines a metric named yield, but does not explicitly establish its correspondence to directional fidelity in Table 5. If directional fidelity follows Equation (27), and it is aggregated over the same valid perturbations with the same weights as DE Jaccard, the set relationships in Equations (27) and (29) imply the bound . The values reported in Table 5 do not satisfy this constraint. If directional fidelity instead uses a different definition, the appendix does not fully specify this central metric. The authors should therefore clarify the implemented formulas, aggregation weights, empty-set conventions, and handling of undefined values, and provide per-perturbation results to make the perturbation experiments and associated ablations verifiable.
W5. Several methodological descriptions are inconsistent, and important training details are missing.
- Resampler training status. The main text describes the resampler as trainable, whereas Appendix E.9 places it in the frozen parameter group. Its training status at each stage and its pretraining procedure should be clarified.
- Dimensions and projection. The appendix states that cell programs share the hidden dimension of language tokens and require no additional projection. However, the backbone and cell programs have different hidden dimensions, and Equation (2) explicitly introduces a projection. Matching dimensions alone would also not establish that modality alignment is unnecessary.
- PubMedBERT results. Appendix C.1 states that results under the same protocol are unavailable, whereas Figure 2 reports them. These descriptions should be reconciled.
- Expression decoder training. The pretraining data, training configuration, and calibration procedure are insufficiently specified. Reconstruction performance on real cells is also missing, making it difficult to distinguish errors introduced by latent generation from those introduced by expression decoding.
- Expression-scale conversion. Decoder outputs and task evaluations use CPM or CP10K. The normalization target at each stage and the conversion operations should be explicitly described.
Questions for authors
Q1. How do the authors explain the average PDS values remaining close to the random-ranking expectation across methods, with no clear improvement after fine-tuning? Could the authors provide per-perturbation results with uncertainty estimates and include simple baselines, such as the control mean and the average perturbation effect, to assess whether the model learns perturbation-specific responses?
Q2. Dividing each method’s chemical perturbation scores by its corresponding genetic perturbation scores in Table 5 reveals relatively consistent ratios. This is particularly apparent for DE Jaccard, whose chemical-to-genetic ratios are approximately 0.93–0.94 across all three methods in both the zero-shot and fine-tuned settings. Given the different datasets, cellular backgrounds, and intervention types, could the authors explain this consistency and provide results by cell line and perturbation, together with the respective evaluation procedures?
Q3. Which main results are computed on validation sets, and which use held-out test sets? How are checkpoints selected, and do any of the reported evaluation sets participate in model selection? Could the authors provide held-out test results for conditional generation and SciPlex3, and clarify why the GO evaluation sample counts match the validation split?
Q4. Could the authors provide single-task controls using the same architecture and comparable training budgets, particularly Qwen2.5-7B with LoRA fine-tuning only on the GO task? How does joint training affect each of the two semantic tasks, and are the gains stable across random seeds?
Q5. Does directional fidelity in Table 5 correspond to yield in Equation (27)? Please specify its implemented formula and that of DE Jaccard, their aggregation weights, empty-set conventions, and handling of undefined values. Do both metrics use the same valid perturbation set? Could per-perturbation scores and the relevant set cardinalities be provided to reconcile the reported results with the appendix definitions?
Q6. Could the authors resolve the inconsistencies concerning resampler training, dimensional projection, and PubMedBERT results listed in W5, and provide the decoder pretraining, calibration, and expression-scale conversion procedures? What reconstruction performance does the frozen decoder achieve on real target cells, and how does this help characterize its limitations for final generation quality?
Q7. How sensitive are the GO annotation results to evidence types and negative-sample construction? Could the authors provide results stratified by evidence type, or a robustness analysis excluding IEA annotations, and explain how excluded known associations are prevented from being reassigned as negative examples?
11 comments
Respose to Reviewer TByD (2)
(2) PDS stratified by effect strength. If what is learned is an intervention-specific response, PDS should rise with the strength of the reference effect.
Table 3. PDS stratified by quartile of reference effect strength (VCC test50, fine-tuned, mean).
| Model | Q1 (weakest) | Q2 | Q3 | Q4 (strongest) |
|---|---|---|---|---|
| CellFlow | 0.4988 | 0.4972 | 0.4951 | 0.4933 |
| STATE | 0.4971 | 0.4940 | 0.4926 | 0.4935 |
| CellUMM | 0.4962 | 0.4988 | 0.5003 | 0.5111 |
CellUMM is the only one of the three that rises monotonically with effect strength (0.4962 → 0.5111), while both baselines stay flat and move in the opposite direction. This locates the identifiable regime: intervention identity becomes distinguishable only once the reference effect is large enough.
(3) Distribution-level validation of conditional generation. Three items are added: a real-cell reference matched to the sample size of the generation setting (8 reference cells drawn at random per condition and compared with the remaining reference cells, giving 0.0667 over 107 conditions); the per-gene standard deviation ratio between generated and real cells, together with the Pearson correlation of the standard deviations; and the standard deviation across 3 sampling seeds (≤ 0.005) together with the per-condition results.
Table 4. MMD² at different numbers of generated cells (114 test conditions, 3 sampling seeds, mean).
| Model | n = 8 | n = 32 | n = 128 | n = 512 |
|---|---|---|---|---|
| Real-cell reference (8 vs. 24) | 0.0667 | — | — | — |
| Organ/cell-type mean | 0.148 | 0.148 | 0.147 | 0.147 |
| SAVE | 0.157 | 0.149 | 0.144 | 0.141 |
| scVI | 0.162 | 0.153 | 0.147 | 0.143 |
| CFGen | 0.221 | 0.213 | 0.207 | 0.203 |
| InstructCell | 0.364 | 0.356 | 0.350 | 0.346 |
| CellUMM | 0.136 | 0.121 | 0.114 | 0.109 |
Table 5. HVG consistency, mean error, and within-condition dispersion at n = 512 (114 test conditions, mean).
| Model | MMD² ↓ | MMD² relative to reference ↓ | Mean-expr. MSE ↓ | HVG1000 J ↑ | Per-gene std ratio | Pearson r of per-gene std |
|---|---|---|---|---|---|---|
| Real-cell reference | 0.0667 | 1.00 | — | — | 1.00 | 1.00 |
| Organ/cell-type mean | 0.147 | 2.20 | 0.0251 | 0 | 0.00 | 0.00 |
| SAVE | 0.141 | 2.11 | 0.0293 | 0.065 | 0.27 | 0.38 |
| scVI | 0.143 | 2.14 | 0.0452 | 0.045 | 0.16 | 0.29 |
| CFGen | 0.203 | 3.04 | 0.0653 | 0.034 | 0.22 | 0.31 |
| InstructCell | 0.346 | 5.19 | 0.1054 | 0.047 | 0.19 | 0.24 |
| CellUMM | 0.109 | 1.63 | 0.0253 | 0.072 | 0.41 | 0.62 |
- Uncertainty. The standard deviation across the 3 sampling seeds is ≤ 0.005 for every entry, and the MMD² ranking is stable across seeds; per-condition distributions (including quartiles) are provided with the revision.
- Calibration against the real-cell distance. The real-cell reference of 0.0667 sets the absolute scale of MMD². CellUMM's 0.136 (n = 8) and 0.109 (n = 512) are 2.04 and 1.63 times the reference, the lowest of all methods (the others are 2.11–5.19).
- Within-condition variability. The mean baseline emits a single vector, which gives it a natural advantage on an MSE that measures only the conditional mean (0.0251), but its per-gene std ratio is 0.00, its HVG Jaccard is 0, and it produces no within-condition heterogeneity; CellUMM's mean error is comparable (0.0253) while its per-gene std ratio is 0.41 and its std correlation 0.62. Accordingly, the revision narrows the statements in Section 5.4 and Section 6 to: given that mean error and distributional distance are simultaneously comparable, CellUMM is a method that can express within-condition heterogeneity and whose heterogeneity partially corresponds to reality; the gap between 0.41 and 1 corresponds to the capacity ceiling of the decoder's 256-dimensional factor subspace (Appendix E.6).
Respose to Reviewer TByD (3)
Q2 (W2, Q3): Evaluation splits, checkpoint selection, and annotation test scale
A2. For every task the checkpoint is selected by validation loss (minimizing the mean L_val/L_C0), and the test set is evaluated once on the selected checkpoint only; no evaluation set participates in model selection. The per-task splits are given below.
Table 6. Training / validation / test splits for the four tasks, and the set reported in the main text.
| Task | Training | Validation (checkpoint selection only) | Test (evaluated once; reported in the main text) |
|---|---|---|---|
| Gene function annotation | 13,738 genes; 300,000 multi-label questions + 200,000 binary questions | 1,962 genes; 6,741 + 20,000 questions | 3,927 genes; 13,387 + 40,000 questions |
| Cell type annotation | 5,000,000 cells; 548 classes | 200 cells (validation split of the Fine split, disjoint from the test set) | 200 cells / 31 classes / 5 donors (expanded to 2,000) |
| Conditional cell generation | 346 conditions (4,946 flow-matching supervised cells) | 106 conditions (≤32 reference cells, 2,707 cells in total) | 114 conditions (81,782 cells) |
| Perturbation prediction (chemical) | Tahoe-100M: 48,122 conditions | 2,510 conditions (132 conditions monitored) | SciPlex3: 442 test conditions (drug-disjoint) |
| Perturbation prediction (genetic) | Replogle–Nadig: 7,316 conditions | 30 conditions (132 conditions monitored) | VCC H1 hESC: 50 interventions, 61,587 cells |
- Evaluation splits. Both parts are computed on test sets — VCC test50 for the genetic part and the 442 drug-disjoint test conditions of SciPlex3 for the chemical part (the 1,595 training conditions are used only for fine-tuning); none of them takes part in training, validation, or checkpoint selection. The split descriptions in Appendix B.6 and Figure 9 are made consistent with this. CellUMM is best on all four metrics in both parts (directional reach 0.5556 vs. 0.2920 / 0.2277; 0.5108 vs. 0.2634 / 0.2051).
- Evaluation sample size for GO. Figure 2 is computed on the test split (13,387 multi-label questions and 40,000 binary questions); the "6,741 and 20,000" in Appendix B.6 are the question counts of the validation split, and the revision corrects them to the test counts and notes that the validation split serves only for checkpoint selection with the sigmoid threshold fixed at 0.5. The corresponding numbers for the validation split are listed in the appendix as a consistency check.
(4) Annotation test scale. The test set is expanded to 2,000 cells; mean ± std over 3 seeds and the paired comparisons are given below.
Table 7. Cell type annotation on the expanded test set (2,000 cells / 31 classes / 5 donors, 3 seeds, mean ± std).
| Model | Accuracy (%) ↑ | Macro-P (%) ↑ | Macro-R (%) ↑ | Macro-F1 (%) ↑ |
|---|---|---|---|---|
| STATE SE | 50.85 ± 1.31 | 45.02 ± 1.44 | 51.94 ± 1.28 | 45.61 ± 1.36 |
| LangCell | 48.92 ± 1.42 | 47.31 ± 1.38 | 49.42 ± 1.33 | 43.28 ± 1.41 |
| GPT-OSS-120B | 45.31 ± 1.55 | 49.55 ± 1.47 | 45.32 ± 1.52 | 42.71 ± 1.49 |
| CellWhisperer | 43.88 ± 1.63 | 45.40 ± 1.51 | 44.11 ± 1.58 | 40.94 ± 1.54 |
| C2S-27B | 43.72 ± 1.58 | 46.22 ± 1.49 | 43.86 ± 1.56 | 40.51 ± 1.52 |
| Geneformer (frozen) | 37.36 ± 1.72 | 34.98 ± 1.61 | 37.12 ± 1.66 | 31.44 ± 1.63 |
| Geneformer (fully fine-tuned) | 59.62 ± 1.18 | 55.18 ± 1.24 | 58.91 ± 1.21 | 56.02 ± 1.22 |
| scGPT (fully fine-tuned) | 58.74 ± 1.22 | 54.31 ± 1.29 | 57.86 ± 1.25 | 55.13 ± 1.27 |
| CellUMM | 60.34 ± 1.09 | 57.02 ± 1.16 | 61.28 ± 1.12 | 57.61 ± 1.14 |
- Paired comparisons: versus fully fine-tuned Geneformer, accuracy +0.72 pp (95% CI [−1.86, +3.31]) and macro-F1 +1.59 pp (95% CI [+0.21, +2.97]); versus STATE SE, accuracy +9.49 pp (95% CI [+6.02, +12.94]); the difference from the 200-cell test set (60.50%) is within 1 percentage point.
Respose to Reviewer TByD (1)
Q1 (W1, Q1): Discriminability of PDS, reference baselines, and conditional-distribution evaluation
A1. PDS measures whether a predicted effect can identify its corresponding intervention among all interventions. On VCC H1 hESC, the effect vectors of different interventions under the same cell background share a strong common response direction, and the PDS of all three methods lies between 0.49 and 0.51; the metric therefore has limited discriminability in this setting. The revision replaces "close to saturation" with "limited discriminability" and no longer uses PDS as the primary evidence of perturbation-modeling capability. Directional metrics do provide discrimination: after fine-tuning, CellUMM reaches a directional reach of 0.5556 on the genetic part of Table 5 of the main text, versus 0.2920 and 0.2277 for CellFlow and STATE (Table 1). Both parts are computed on test sets — VCC test50, and the 442 drug-disjoint test conditions of SciPlex3 (Table 2); the split description in Appendix B.6 is made consistent with this.
(1) Reference predictors. We add control mean (which directly outputs the control cell distribution) and mean perturbation effect (which applies the training-set average effect to each intervention), with gene coordinates, DE calling, and aggregation identical to Section 5.5 of the main text. Tables 1 and 2 report the results of the two parts of Table 5 on the same test evaluation sets, together with the variation over 3 seeds.
Table 1. VCC test50: reference baselines for perturbation prediction (main training configuration, fine-tuned, 3 seeds, mean ± std).
Table 2. SciPlex3 test442 (drug-disjoint): reference baselines for perturbation prediction (main training configuration, fine-tuned, 3 seeds, mean ± std).