Density-Based Online Autocuration for Pathology Image Model Pre-Training

Tile streams for pathology image model pre-training are curated at the slide level, or offline at the tile level with an oracle encoder. Morphologies are unevenly distributed within a slide, so random tile sampling carries each slide's imbalance into the training stream and common morphologies overshadow rare ones. This work curates the stream online, during DINOv2 pretraining, from a density estimate the model produces itself. A density cache keeps a fixed-size memory of $M$ stored tile signatures and counts how often new tiles land on each; the counts give the typicality of an incoming tile, and tiles enter the batch with probability inversely proportional to it. One hyperparameter, the tilt, sets how far the batch leans toward low-typicality tiles. ViT-B and ViT-L encoders trained with DINOv2 on 287M tiles were evaluated on twelve biomarker mutation-prediction tasks from H&E. Fold-paired against the uncurated baseline at the same checkpoint, every curated run improved mutation prediction on average, by $0.3$ to $1.0$ percentage points of AUROC over the twelve tasks and by $2$ to $4$ on THCA/NRAS at ViT-B, and the two thyroid tasks improved in every run at both scales. At ViT-L the improvement is on the tasks with the most consistent folds, and two of the three runs are significant when each task is weighted by the precision of its measurement, with a tilt of $1.0$ the best of the three. At ViT-B it is on the tasks with few positive slides per fold and is resolved in one run of seven. Across the eight $124$k runs whose caches did not starve, the gain rises with the spread of hit-rates over the cache, its bandwidth (Spearman $+0.75$). A $500$k-step pair at ViT-B, four times the schedule, is at parity on aggregate ($+0.3$ percentage points, 95% interval $[-0.4,+0.9]$), and the thyroid and colorectal tasks improve again under this third independent baseline. Mapped back onto a slide, the stored signatures separate morphologically coherent regions at tile resolution, so the cache also yields an unsupervised tissue segmentation.