- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 3(+1)
Read review
Paper summary
The paper successfully demonstrates that the ‘profile routed residual memory’ method alleviated catastrophic forgetting during sequential training on four tasks.
The data are in the form (cells, gene activity) + intervention label and the model predicts activity under an intervention, the continual learning problem is training this model sequentially on new data.
I was not familiar with the virtual cell models used before reading this paper, much of the unique contribution seems to derive from creating a continual learning method specially designed for these models, trained on cell line data.
Critical strengths
The authors demonstrate that it does alleviate catastrophic forgetting for foundational virtual cell models. They carefully select which memory entries can be updated in their constructed PRM memory matrix, and this along with the sparse writes and protection of old writes, appears to be why it works.
The parts of the sparse memory mechanism are taken from elsewhere but much of it appears to be a new contribution.
Critical weaknesses
The method seems tremendously complex and my main concern is whether this complexity is justified.
PRM works by training sparse values in their memory matrix and then protecting them for future tasks. It took me a while to understand how this was injected into the transformer foundational models (equation 7).
This seems to be a classic continual learning problem using a pre-trained transformer, which is why a very complicated bespoke solution seems odd. Though they do compare against some classical baselines.
In the appendix table 4, it appears that performance on the pre-training data drops (if I've understood it correctly). This weakens the continual learning claim somewhat.
Questions for authors
One check I would like to see is if the careful selection of the memory entries are necessary or if you could just use a random sparse selection with the same number of writable entries. Is most of the gain coming from simply updating a small set of parameters and protecting earlier ones? I see some controls for a ‘frozen router’ but not this.
The drop in performance for X-Cell on the pre-training data seems to be a crucial result. Would you be able to see if this is the case for the other foundational models?
3 comments
Thank you again for highlighting pretraining retention. We are following up with additional completed experiments that test a remedy using PRM's existing protection and calibration mechanism.
Regarding Q2 and W4, Can PRM also protect pretraining performance:
Extending X-Cell PRM's ownership and anchoring to pretraining controls raises final Pearson-Δ from 0.1202 to 0.6220 on the same 860 held-out HCT116 groups, from an initial score of 0.6788. This provides direct evidence that the mechanism can protect more pretraining knowledge.
We then test the existing anchoring weight at a fixed starting checkpoint, memory size, gene panel and response-update budget. Each row is one adaptation campaign.
| Anchoring weight | Acquisition | Sequential forgetting (%) | Pretraining retention (Pearson-Δ) | K562 acquisition |
|---|---|---|---|---|
| 1 | 0.5748 | 21.22 | 0.6211 | 0.5494 |
| 2 | 0.5770 | 14.24 | 0.6295 | 0.5512 |
| 4 | 0.5677 | 6.52 | 0.6224 | 0.5126 |
| 8 | 0.5407 | 2.62 | 0.6327 | 0.4091 |
Pretraining retention is final Pearson-Δ on the same 860 held-out HCT116 groups, from an initial score of 0.6788.
Doubling the anchoring weight from 1 to 2 raises held-out Pearson-Δ from 0.6211 to 0.6295 and lowers sequential forgetting from 21.22% to 14.24%, while acquisition changes from 0.5748 to 0.5770. It improves both measured retention outcomes by tuning the existing objective, without adding a model component. The completed weight-8 evaluation gives held-out Pearson-Δ of 0.6327 and sequential forgetting of 2.62% in the same campaign, with lower acquisition of 0.5407.
The appendix subsection “Protection of pretraining knowledge” reports the pretraining gains, their cost to sequential retention, and the full sweep showing the acquisition trade-off at larger weights. First-round changes remain blue, and these second-round additions are red.
The new evidence addresses Q2 and W4 with an intervention that extends protection to pretraining knowledge and a calibration adjustment that improves retention. We would be grateful if you would reconsider your assessment and score in light of these additional results.
Thank you to the authors for the thorough response.
I now understand the protection of sparse updates a little better. However, it remains unclear to me whether the additional complexity of the procedure provides an advantage over simpler methods.
The substantial drop in performance on the pretraining data has been investigated more carefully, including experiments aimed at preserving pretraining performance. I consider this to have strengthened the paper’s soundness, and I have raised the soundness score to 3.
I still find the paper quite hard to understand, partly due to its complexity. The authors have attempted to address this and justify the need for the complexity, but it remains quite hard for me to assess. This is reflected in my low confidence score of 2.
Thank you for asking whether simpler protected sparse updates explain the gains and whether pretraining knowledge is retained. We added random-allocation and retention tests, and extended protection to pretraining controls in X-Cell. The corresponding manuscript changes are marked in blue.
Regarding Q1, Would random sparse allocation with the same number of protected writable entries work:
Matching the number of randomly reserved rows does not recover PRM's acquisition in this control. Across three campaigns, uniform sampling from all unprotected rows matches PRM's nominal allocation counts at every stage and table, but gives acquisition 0.1264 ± 0.0021 compared with PRM's 0.5752 ± 0.0032.
The router reached only 613 of the random arm's 260,869 reserved row instances during allocation probes, compared with all 260,869 for PRM. Equal nominal counts therefore left the random arm with far less capacity that the router actually used.
Whether ranking equally active rows adds a benefit remains open. The completed control supports PRM's use of routing information, because equal nominal sparsity did not provide equal access to trainable memory.
Regarding Q2 and W4, Does pretraining performance also decline for scDFM and scGPT:
Yes, the new scDFM evaluation also shows a substantial decline. Across 352 matched HCT116 training groups, perturbation-Δ Pearson falls from 0.4199 before sequential adaptation to 0.1219 afterward, an absolute drop of 0.2980. This single checkpoint pair uses identical group keys, the fixed 1,898-gene panel and the same sampling protocol on a training cohort.
A separate scGPT PRM rerun falls from 0.7121 to 0.0056 across 366 matched HCT116 training groups. This rerun uses a different checkpoint family from the submitted scGPT row, and the appendix labels it separately.
For X-Cell, extending protection to pretraining controls gives final cohort performance of 0.6170 ± 0.0052 across three seeds at the 90% route-mass target. Performance improves from 0.1463 for standard PRM to 0.6230, while sequential forgetting rises from 0.59% to 18.87%.
These results show that pretraining-cohort forgetting also occurs in the tested scDFM and scGPT campaigns. The X-Cell extension shows that adding protection for pretraining contexts can reduce this forgetting, albeit with a cost to sequential retention.
Regarding W1, Is the complexity justified:
The component experiments show why protection and calibration serve different roles within PRM. Ownership protects earlier memory values, while calibration restores access to rows that remain trainable.
Under full protection, K562 acquisition is 0.5310 with calibration versus 0.1143 with frozen routing in one run each, while permuted protected-risk information gives 0.1138 ± 0.0003 across three runs. The shared-checkpoint X-Cell controls also favor PRM over host fine-tuning and LoRA with 5% replay across three adaptation seeds, although progressive slots remain stronger on this benchmark.
These interventions justify calibration in the tested PRM design by linking it to recovered acquisition under protection.
Regarding W2, How does Equation (7) inject memory into the backbone:
Equation (7) adds a transformed memory read to the backbone hidden state through a residual connection. The current hidden state modulates how the retrieved information affects the prediction.
During response training, the host, router and base memory stay frozen while selected residual rows learn, and separate boundary calibration adjusts routing before it is frozen again. We now place this explanation before the equations. Equation (7) therefore lets learned memory change predictions without updating the backbone weights.
Regarding W3, Why use a specialized method for a classical continual-learning problem:
PRM addresses a general failure in sparse-memory adaptation, where new inputs retrieve protected entries they can no longer update. Its calibration rule uses current retrieval patterns and earlier ownership to restore writable access while anchoring earlier routes. We test this rule across three backbones, with gene identities and control profiles supplying the observable content and context.
We hypothesize that prompt embeddings or support-set summaries could provide context in other applications, provided earlier context representations remain available for anchoring. The methodological contribution is the coordination of routing and protected updates, with control cells providing a concrete biological implementation for this method.