- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how cytometry foundation models represent immune cell composition. SAEs expose features in sample embeddings, but associations with population abundances do not isolate which composition changes affect those features. The paper proposes controlled in silico perturbations that resample a sample's cells to exchange a target population with the remainder of a specified scope, keeping cells outside that scope fixed. Consistent feature responses are converted into annotations naming the target, scope, and response direction.
Critical strengths
- The target–scope formulation makes composition interpretations explicit and testable. Specifying the displaced cells distinguishes whole-sample abundance changes from changes within an enclosing population. Resampling a sample's own measured cells provides a controlled way to implement these exchanges.
- The experiments provide substantial evidence of repeatable composition responses in the frozen models. More than 98% of annotations reproduce their slope direction with corrected significance on held-out samples, and MAESTRO decoder steering agrees with 93.2% across all six signed doses.
Critical weaknesses
- The paper shows that SAE features can be interpreted reliably, but it's unclear why these interpretations are useful in practice. The authors don't demonstrate what new insights or benefits they provide beyond simply using the foundation model or standard cytometry analysis.
- The added value over simpler interpretations is insufficiently established. Correlation-derived annotations and PCA or random Top-k features annotated by the same procedure are not evaluated. The existing random steering controls do not establish whether the proposed annotations provide more informative interpretations than these alternatives.
Questions for authors
- How do perturbation-based annotations compare with simpler correlation-based annotations under the same evaluation protocol?
- Are the results robust to donor-level clustering or using only one sample per donor?
- Do the shared-feature programs remain consistent when cases and controls come from the same study?
4 comments
W2 & Q1. Comparison with correlation, PCA and random annotations under the same protocol
We agree with this assessment. The paper did not compare against these simpler alternatives, and we have now done so. The comment raises two separate questions, whether the perturbation is needed and whether the SAE is needed. We address each below. Perturbation versus correlation (same SAE features). A simple correlation might give the same annotations, in which case the perturbation would not be needed. We find that the two are not interchangeable. Populations rise and fall together across samples, so a feature can correlate with a population that it does not respond to. Changing one population directly separates the two cases.
To test this, we annotated the same SAE features by correlation on the discovery samples. For each condition, we took the features with the largest absolute Spearman correlation between their activation and the target's share across samples, as many as the perturbation method annotated, with the sign of the correlation as the direction. Both kinds were then scored with the evaluations of the paper (Sections 5.1, 5.2, and 5.4).
| GPCT, perturbation | GPCT, correlation | MAESTRO, perturbation | MAESTRO, correlation | |
|---|---|---|---|---|
| Held-out Stage 2, perturbation response replicated (%) | 99.98 | 90.4 | 98.3 | 78.5 |
| Held-out Stage 3, firing difference in annotated direction (%) | 93.8 | 91.7 | 94.3 | 93.7 |
| Expert statements recovered (%) | 81.8 | 62.6 | 64.6 | 77.8 |
| Zero-shot AUROC, held-out samples | .847 | .792 | .770 | .806 |
When the composition of held-out samples is changed directly (Stage 2), about one in ten correlation annotations for GPCT and one in five for MAESTRO do not replicate, whereas nearly all perturbation annotations do. The two methods also pick different features, agreeing on the feature for a condition in only 19% (GPCT) and 23% (MAESTRO) of annotations. One example is GPCT feature 1251 (see W1), which correlates positively with myeloid dendritic cells yet falls when these cells are increased directly.
In expert-statement recovery, perturbation annotations are 19.2 pp higher for GPCT, whereas correlation annotations are higher for MAESTRO. Zero-shot AUROC follows the same pattern, and Stage 3 is close. A correlation annotation states in which samples a feature is high, whereas a perturbation annotation states how the model responds when one population changes, which is what our framework claims.
SAE versus PCA and random features (same annotation procedure). PCA or random features annotated by the perturbation procedure of the paper might be equally informative, in which case the SAE would not be needed. We find that SAE annotations are the most informative of the three. This procedure yields reproducible annotations whichever features it is applied to, so reproducibility does not distinguish them. What distinguishes them is agreement with expert knowledge, where the SAE leads in both models.
PCA uses the principal components of the same sample embedding. Random features are Gaussian projections of it with the dictionary size and Top-k firing (k = 64) of the SAE. Both were annotated exactly as the SAE features were.
| GPCT, SAE | GPCT, PCA | GPCT, random | MAESTRO, SAE | MAESTRO, PCA | MAESTRO, random | |
|---|---|---|---|---|---|---|
| Annotations | 12,266 | 939 | 16,334 | 7,440 | 531 | 14,509 |
| Held-out Stage 2, perturbation response replicated (%) | 99.98 | 99.8 | 99.97 | 98.3 | 100.0 | 99.9 |
| Held-out Stage 3, firing difference in annotated direction (%) | 93.8 | 94.1 | 95.4 | 94.3 | 97.2 | 96.3 |
| Expert statements recovered (%) | 81.8 | 55.6 | 70.5 | 64.6 | 40.4 | 39.8 |
| Zero-shot AUROC, held-out samples | .847 | .644 | .776 | .770 | .766 | .791 |
SAE annotations recover the most expert statements in both models, by 26.3 pp over PCA and 11.3 pp over random features for GPCT, and by 24.2 pp and 24.8 pp for MAESTRO. For GPCT, SAE annotations give higher zero-shot AUROC as well, while for MAESTRO the three are similar. Held-out replication is high for all three, and random features receive more annotations than SAE features, so neither separates them.
For random features, annotation counts, held-out replication and expert recovery are means over five seeds. We will add both comparisons to the appendix.
Q2. One sample per donor
Repeated samples from the same donor are a plausible source of bias in our results. One donor can have several samples in our data, and samples from the same donor resemble each other. The tests of the paper nevertheless count each one as a separate observation, so a result could look more significant than it is. To check this, we repeated the two statistical tests of the annotation procedure with one sample per donor. Our analysis suggests that the results are robust to repeated donors.
We used the samples with known donor identity, excluding batch controls. These are 1,214 samples from 583 donors, of which the perturbed samples are 448 from 309 donors for GPCT and 445 from 297 for MAESTRO. We drew one sample at random from each donor and ran both tests, the perturbation response (Stage 2) and the firing comparison (Stage 3), over all features and conditions with the multiple-testing corrections of the paper. One sample per donor leaves fewer samples, so for a fair comparison we also drew two independent random sets of the same number of samples without regard to donor and analysed them in the same way.
A feature and a condition that pass a test form a significant pair. We measured how many significant pairs of one analysis are also found by another, and we did this twice. The first measurement compares the two random sets with each other. The two random sets have the same number of samples and are drawn in the same way, so they differ only in which samples were drawn. Their agreement therefore shows how much the results change when only the drawn samples differ. We use this value as the reference. The second compares one sample per donor with a random set. If repeated donors mattered, the second value would be clearly lower than the first. All values are means over 100 repetitions, each with a new draw of one sample per donor and of the two random sets.
| Significant pairs also found in a random set of the same number of samples (%) | Random set (reference) | One sample per donor |
|---|---|---|
| Perturbation response (Stage 2), GPCT | 96.8 | 95.8 |
| Perturbation response (Stage 2), MAESTRO | 83.0 | 83.2 |
| Firing comparison (Stage 3), GPCT | 51.7 | 52.3 |
| Firing comparison (Stage 3), MAESTRO | 48.3 | 49.6 |
In every row, the two values differ by at most 1.3 percentage points. One sample per donor therefore agrees with a random set as closely as two random sets agree with each other. For the perturbation response of GPCT, about 96% of the pairs are found again in both columns.
Why both analyses use the same number of samples. One sample per donor retains 583 of the 1,214 samples (309 of 448 and 297 of 445 perturbed samples). Fewer samples alone reduce the number of significant results, so a decrease relative to the paper could reflect either the smaller sample or the removal of repeated donors, and the two could not be distinguished. We therefore fixed the number of samples in both analyses, so that they differ only in whether repeated donors are kept. A difference between the two columns can then be attributed to repeated donors. This design also indicates how much agreement can be expected at this sample size. With fewer samples, results near the significance threshold are less stable across random sets, and two random sets share about half of their significant pairs in the firing comparison. One sample per donor reaches the same level.
Removing repeated samples from the same donor therefore changes the findings no more than drawing a different random set of samples does. We will add this analysis to the appendix.
Q3. Same-study case–control comparisons
In several disease comparisons of the paper, cases and controls originate from different studies, so the programs might not remain consistent when both come from the same study. To assess this, we used acute COVID-19, for which several separate cohorts are available, and derived the programs independently within each cohort, using only its own cases and controls. Our analysis suggests that the programs remain consistent for this disease.
Five cohorts were analysed. Three are acquisition studies within our data (A, B and C), and two are external cohorts, one of hospitalized and one of intensive-care patients, which share no sample, donor or study with our data. The procedure of Section 6 was applied with GPCT to each cohort separately, and additionally to the three studies pooled.
In Section 6, conditions that share features are grouped into communities, and a program is one of these communities. We therefore compared the cohorts by the conditions (a population changed within a parent population) that each assigns to a community. The agreement between two cohorts is the Jaccard index of these conditions, that is, the number of conditions common to both cohorts divided by the number present in either. If the programs were not consistent across cohorts, few conditions would be shared and the index would approach zero.
| Cohort (case / control samples) | Study A | Study B | Study C | External, hospitalized | External, intensive care | Conditions in a community |
|---|---|---|---|---|---|---|
| Study A (47 / 6) | – | .66 | .06 | .59 | .48 | 50 |
| Study B (30 / 30) | .66 | – | .07 | .58 | .48 | 56 |
| Study C (12 / 41) | .06 | .07 | – | .06 | .06 | 6 |
| External, hospitalized (77 / 15) | .59 | .58 | .06 | – | .45 | 61 |
| External, intensive care (116 / 8) | .48 | .48 | .06 | .45 | – | 49 |
| Studies A + B + C pooled (89 / 77) | .66 | .59 | .07 | .65 | .51 | 73 |
In four of the five cohorts, the procedure identifies largely the same set of conditions. Studies A and B and the two external cohorts overlap with one another, with Jaccard indices between .45 and .66, and the direction of the effects is fully concordant, since all 266 case–control differences detected in both cohorts of a pair have the same sign. The pooled analysis yields 73 conditions, which overlap with both external cohorts (.65 and .51), and all 171 shared differences again have the same sign. These 73 conditions include all 9 conditions of the analysis reported in the paper.
Study C is the exception. It comprises 12 case samples from only 3 donors and yields 6 conditions, compared with 49 to 61 in the other cohorts. With so few conditions, its Jaccard index cannot exceed .12 even if all of them were shared, and 3 or 4 of the 6 are indeed present in each of the other cohorts. The small number of conditions most likely reflects the small number of cases. We therefore consider Study C inconclusive and do not count it as a replication.
Beyond acute COVID-19, re-estimating the case–control differences within the same study retains their direction for 93.9% (GPCT) and 100% (MAESTRO) of the differences, across the 8 and 5 disease comparisons that have same-study controls.
Programs derived within a single study are therefore in agreement with those derived in other studies, including two cohorts that are entirely independent of our data. We will add these analyses to the appendix.
We would like to thank the reviewer once more for the detailed review.
We hope the revision and these responses address the reviewer's concerns. If any point remains unclear, or if a further analysis would be helpful, we would be glad to provide it within the discussion period, and we would greatly value the reviewer's view of the revised paper.
W1. Practical value
The practical value of the interpretations was not highlighted sufficiently in the paper, and we clarify it here. Compared with using the foundation model directly, the benefit lies in identifying what the model responds to, with greater accuracy than a correlation-based analysis provides. The interpretations further reveal which populations the model treats as moving together in disease, an aspect that standard cytometry analysis does not capture.
Beyond using the foundation model directly. The model alone gives embeddings and predictions but not what drives them. The simplest way to find out is to correlate its features with population frequencies across samples, but a correlation can point to the wrong population. GPCT feature 1251 is an example. It is higher in samples rich in myeloid dendritic cells (ρ = 0.48), so a correlation would label it a dendritic cell feature. Yet increasing these cells directly lowers it (in 239 of 310 held-out samples), because the feature responds to monocytes, which are more abundant in the same samples (ρ = 0.64). Our perturbation identifies this because it changes one population within a sample and measures the response of the feature.
Feature 1251 is not an isolated case. For every feature and every population, we compared the Spearman correlation between the two across discovery samples with the response of the feature when the population is changed directly.
Of the pairs with a significant correlation, 20.1% (GPCT) and 5.4% (MAESTRO) respond in the opposite direction, as feature 1251 does, and a further 18.5% and 64.7% show no significant response. A correlation therefore agrees with the response of the model in only 61.4% and 29.9% of the pairs. The annotations also turn the disease signal of the model into statements about populations. Without using clinical labels, they recover 81.8% (GPCT) and 64.6% (MAESTRO) of 99 expert statements about which populations rise or fall in disease (Section 5.4).
Beyond standard cytometry analysis. Standard analysis compares populations one at a time and finds which of them differ between cases and controls. Shared-feature programs (Section 6) add which of these populations the model treats as moving together. For example, one MAESTRO program for ANCA vasculitis links CD4 T-cell subsets including Th1 and Th1Th17 cells. The programs remain consistent when cases and controls come from the same study (Q3).