- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper adds an aux task to the State transition model: besides predicting treated cells from controls plus a drug label, the same set transformer also predicts controls from treated cells with a separate linear label encoder and the same energy-MMD loss at weight 1. Evaluation uses the State few-shot split of State-Tahoe-Filtered in 2000-HVG space: training covers 45 cell lines plus part of the conditions in five evaluation lines, and the rest are held out (Sec 4.1). One GenInv run is compared with one forward-only run ("State-ST") at 200k steps. With shuffled inverse labels (100k steps) the Pearson-delta gain stays, but roughly half of the direction and LFC gains are lost (Tab. 2).
Critical strengths
- The idea could hardly be simpler! And the comparison is like for like: one extra pass through the same network, no change at inference (Sec 3.3), same architecture, split, optimizer and step budget (L245). The listed settings also match Arc's released HVG few-shot config, so State-ST isn't a strawman. If the gain is real, anybody training a State-style model gets it in a few lines which would be cool.
- The authors test their own story and report the result even where it hurts, e.g. the shuffled-label control keeps the whole Pearson-delta gain(L370). They also state single seed (L247), DE overcalling (L430).
Critical weaknesses
- My main concern is noise. The contribution is one loss term, and it's backed by one run per arm, all seed 42 (L247). The only scale for variation I can find is State-ST's own drift, where Pearson-delta drops from 0.9275 at 100k to 0.9220 at 200k (L313). That isn't seed variance but it's the size of the gains (0.0047, 0.0092) and the gap widens only bc State-ST falls faster than GenInv. The recall gain comes with more positive calls (Sec 4.5). To be fair, LFC Spearman looks more solid: +0.036 at 100k, +0.052 at 200k in all five lines both times, and GenInv itself improves too.
- The controls test the label but not the direction. Shuffled labels give the same Pearson-delta as correct ones (0.9323 vs 0.9322, L370), so as the authors say that benefit isn't treatment-specific. They mention a "regularizing effect" (L317), which a second pass in any direction might provide on a recipe without weight decay (L249, which I believe comes from the State release?). There's no State-ST run with standard regularization or with a treated-to-treated pass, which keeps the second pass and treated inputs and removes only the reverse direction, and Sec 4.6 isn't a matched ablation (L462). The label-specific part (LFC +0.020, L372) is again one run against one.
Overall a simple, honestly reported idea that may well work, but "establish" (L028), even "in this setting" (one backbone, one dataset), is more than one seed per arm and no control for direction support. I'm close to the border on this one tbh.
Questions for authors
- (experiment) What's the across-seed spread of Pearson-delta and DE LFC Spearman for both models? 100k steps would do. If the LFC gain clearly exceeds it over three seeds, I'd raise my score.
- (clarification) What does Tab. 1 look like with best-validation checkpoints for both models, as in the State release?
- (experiment) Does State-ST with weight decay, or with a treated-to-treated pass (disjoint samples), close the LFC gap? If not, that would move me up.
- (experiment) Tab. 3's baselines never see the five evaluation lines (Sec 4.5). How do your linear models score when refit on the few-shot data, or State's context-mean baseline? Pearson-delta against controls rewards shifts shared across perturbations (Viñas Torné et al., 2025). The label swap (L362) shows the label matters, but not how much of 0.92 is shared shift!
Thank you for proposing concrete tests of the gains in our tables.
Response to Q1:
We agree that the gain needs an across-seed comparison. At 100,000 steps, GenInv improved log-fold-change (LFC) correlation in each seed pair. Even the smallest gain exceeded either model's across-seed SD.
We resumed State-ST and GenInv from their 50,000-step checkpoints for seeds 43, 44 and 45, retaining optimizer state and batch size 64. We scored all six final checkpoints under the submission protocol, weighting five test lines equally. Values are seed means ± SD.
We will report the paired LFC gains of +.02848, +.02776 and +.01767, averaging +.02464. With validation selection, mean LFC and Pearson gains were +.01945 and +.00348. For both metrics at both endpoints, the exact two-sided sign-flip test gave p=.25, its minimum with three pairs.
Response to Q2:
Your question about State's checkpoint rule prompted us to recompute Table 1. When both models select the lowest forward-validation loss, GenInv scores higher on all four metrics.
Candidates included the best checkpoints retained from the 10,000- and 50,000-step stages and checkpoints every 2,000 steps from 52,000 to 100,000. In seed order, State-ST selected 96k/98k/92k and GenInv 72k/92k/92k. The table gives three-seed test means ± SD.
This will be our main table, with final checkpoints reported as a sensitivity check. Scoring the selected checkpoints with cell_eval2 also favored GenInv in Delta Pearson and average precision, although State-ST had higher LFC (.16901 versus .16561). In a separate 50,000-step analysis, switching from predicted to observed DMSO controls reversed the LFC ranking with samples, averaging and significant genes held fixed.
Response to Q3:
Thank you for suggesting these two alternatives. Neither closed the LFC gap. GenInv had higher LFC than both controls with validation selection and at the final checkpoint.
We trained both variants to 100,000 steps. One used AdamW with weight decay 5e-4 instead of zero. The other added a pass predicting one treated population from disjoint cells under the same treatment. Multi-seed values are means ± SD.
The final gaps to GenInv's three-seed mean were .01805 for AdamW and .05382 for treated-to-treated training. We will add both controls to the ablation table and document their different training histories and the checkpoints available for selection.
Response to Q4:
You are right that access to target-line data matters. All three Table 3 baselines improved after refitting, and both neural models still led in Pearson-delta and LFC. A separate context-mean comparison also showed that a shared shift alone fell short of State-ST's correlation.
For Additive, Ridge and Diagonal we scored the original fit and two adjustments (nine settings). Held-out treatments had no target-line training data, so their original coefficients stayed fixed. Using 348 training conditions per target line, we fitted gene-wise offsets, or scales and offsets, shared across treatments. We excluded validation and test treatments, applied scaling and offsets to population means before clipping, and retained the original DMSO predictions and test references. Entries give five-line mean Pearson-delta / LFC Spearman on the same test conditions.
Diagonal had the best linear Pearson-delta (.77218) and Ridge the best LFC (.54104). With common validation selection, State-ST reached .92881/.61689 and GenInv .93228/.63634 across three seeds.
The context mean predicts the target line's mean treated training expression for every perturbation. It scored .41345 in Pearson-delta versus State-ST's .87324 in a matched seed-43, 10,000-step comparison. We will add the refits and context mean to the appendix and clarify that neither this comparison nor the label swap assigns an additive fraction of .92 to the shared shift.