Stabilizing Slide Representations for Continual Whole-Slide Image Learning: Frozen Attention and Prototype Rehearsal

Class-incremental whole-slide image (WSI) classification must learn newly arriving cancer types without catastrophic forgetting of previously learned classes and without access to data from earlier tasks. Compact class prototypes offer an exemplar-free way to rehearse earlier classes, but a prototype is valid only in the representation space that produced it. Freezing only the patch encoder does not stabilize slide-level representations: a trainable multiple-instance learning attention aggregator maps the same patch bag to different embeddings as its attention changes across tasks, making stored prototypes stale. We introduce PACE (Prototype-Anchored Continual Expansion), which fixes a pretrained gated-attention aggregation map so that identical patch features yield identical embeddings across tasks, and rehearses K class-conditional K-means centroids in this fixed space; only an expanding linear classifier adapts to the new cancer types. PACE retains no slides, patch bags, or pseudo-samples and needs no task identity at inference. Across seven public cohorts, 11 class-incremental curricula, and 21 continual baselines, PACE attains the highest mean end-of-stream average balanced accuracy (AACC) on eight curricula and ties on one; its margins over the strongest competitor reach 10.5 points, and slide-retaining methods lead on the other two curricula, where classes are smallest or most heterogeneous. A matched ablation shows that fixing the aggregation map, not its initialization, is what makes rehearsal effective, and that one prototype per class (K = 1) recovers 82% of the AACC gain achieved with K = 16. For continual prognosis, a prototype-free extension with frozen cohort-specific Cox heads also attains the highest macro-averaged concordance index among the evaluated continual methods under all five orderings of a six-cohort stream, though its margins over the strongest competitor (0.8–2.5 points) are smaller than in diagnosis and within the standard deviation across folds. Stabilizing the slide representation at the aggregation map thus makes compact, slide-free prototype rehearsal effective for continual WSI diagnosis and, without prototypes, extends to cohort-incremental prognosis.