- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Density-Based Online Autocuration, a method for curating the training stream during pathology image model pre-training. Within DINOv2 self-supervised pre-training, the method constructs tile signatures from representations produced by the evolving model and estimates local density using a fixed-capacity density cache. The estimated density is used to adjust the admission probability of candidate tiles, reducing the sampling frequency of tiles associated with common morphologies without changing the original training loss.
The authors conduct experiments with ViT-B and ViT-L encoders on a pathology image corpus containing approximately 287 million tiles, evaluating the frozen encoders on twelve biomarker mutation prediction tasks using H&E images. Under the 125k-step training schedule, the authors report aggregate mean AUROC improvements of approximately 0.3–1.0 percentage points across the twelve tasks for the curated configurations relative to their corresponding baselines. For ViT-L, the improvements are concentrated in a subset of tasks with more consistent evaluation results, whereas for ViT-B, the gains are more pronounced on tasks with fewer positive slides per evaluation fold. In the extended 500k-step ViT-B experiment, the curated model achieves near parity with its corresponding baseline in aggregate, with a reported mean difference of +0.3 percentage points and a 95% interval of [−0.4, +0.9].
The authors also demonstrate the potential of using the same density cache for unsupervised tissue segmentation. However, the segmentation results are not quantitatively evaluated against manually annotated reference masks. Overall, the work introduces a method for integrating online density estimation into the data stream of self-supervised pre-training and investigates its effects through downstream mutation prediction and qualitative tissue segmentation experiments.
Critical strengths
- A simple and interesting idea for online data curation.
I like the idea of using the model's own evolving representations to guide data curation during pre-training, rather than relying on a separate encoder to curate the data offline. The density cache provides a practical way to estimate how common a tile's representation is and adjust its sampling probability accordingly, without changing the original DINOv2 training objective. This is a natural and interesting direction for addressing morphological imbalance in pathology data.
- A reasonably broad experimental study with useful observations.
The authors evaluate the method with both ViT-B and ViT-L encoders and explore several curation configurations. I also appreciate that the paper reports less favorable results, including the near-parity between the curated model and its baseline under the extended 500k-step schedule, rather than focusing only on positive outcomes. These experiments provide useful insights into when the method may help and where its benefits remain limited. That said, more independent pre-training runs would help establish how consistently these observations hold.
Critical weaknesses
-
Downstream evaluation. The authors report the best AUROC from six evaluations on the held-out set without a separate validation set. The five shuffle splits also have overlapping held-out sets, and slides from the same patient can appear in both the downstream head-training and held-out evaluation partitions. The reported patient overlap is 8% on average and 21% for BLCA/FGFR3. Given the relatively small reported gains, patient-disjoint evaluation and a model selection protocol that does not rely on the held-out set would strengthen the results.
-
Computational cost and baselines. The proposed method requires additional forward passes to score candidate tiles. The authors report that step time increases by 1.65–2.40 times for some ViT-B configurations. However, the main comparisons are made at matched training steps rather than matched computational budgets. A comparison with standard pre-training and a simpler feature-space resampling baseline under matched compute would help clarify whether the proposed method offers sufficient benefits to justify its additional cost.
-
Reproducibility and training dynamics. Most curated runs resume from their corresponding baseline checkpoints, so the comparisons share the same pre-training history. While this helps isolate the effect of curation, it remains unclear whether the gains persist across independent training runs. The paper mainly reports downstream results at the final checkpoints, while the training dynamics analysis focuses on the density cache rather than downstream performance. Independent pre-training runs with different random seeds and downstream evaluations at intermediate checkpoints would help clarify the consistency of the reported gains.
-
Comparison with existing pathology foundation models. The supplementary material compares the proposed models with UNI, Virchow, Virchow2, Prov-GigaPath, and H-Optimus-0. However, substantial performance gaps remain on several tasks. For example, on COAD/BRAF, ViT-L (run 11) achieves an AUROC of 69.4, compared with 80.9 for Virchow2 and 80.5 for H-Optimus-0. Although the primary goal is to study online data curation rather than to develop a new state-of-the-art model, the authors should clarify the practical significance of these results and whether the proposed strategy can also benefit stronger pre-trained encoders.
Questions for authors
I’m still wondering how useful this sampling strategy is in practice. Since it requires extra forward passes to select tiles, could the authors clarify whether it actually provides a meaningful benefit over simpler sampling strategies or standard pre-training when the compute budget is the same?
4 comments
Response to 43bU Weakness 2 and Question 1
Weakness 2. Computational cost and baselines. The proposed method requires additional forward passes to score candidate tiles. The authors report that step time increases by 1.65–2.40 times for some ViT-B configurations. However, the main comparisons are made at matched training steps rather than matched computational budgets. A comparison with standard pre-training and a simpler feature-space resampling baseline under matched compute would help clarify whether the proposed method offers sufficient benefits to justify its additional cost.
Question 1. I’m still wondering how useful this sampling strategy is in practice. Since it requires extra forward passes to select tiles, could the authors clarify whether it actually provides a meaningful benefit over simpler sampling strategies or standard pre-training when the compute budget is the same?
We thank the reviewer for this question, which goes to the practical value of the method. We took the suggestion and trained a new pair of ViT-L runs for it. At matched compute, the curated run is level with standard pre-training. Its macro-mean AUROC over the twelve tasks is 75.5 at 200k steps, against 75.5 for the baseline at 240k steps and 75.2 for the baseline at 260k steps, where the curated run is +0.36 pp above and wins 36 of the 60 task-folds (Table 1).
The paper matches training steps because our claim is about which tiles the model learns from. A scout pass, the forward pass that scores a candidate tile, carries no gradient, so at matched steps the two runs differ only in the tiles of each batch. Selection methods and curated datasets are commonly tested this way (Mindermann et al., 2022, and Oquab et al., 2024). As for the cost, the step-time ratios of 1.65 to 2.40 that the reviewer cites include CPU time in the data loader. Counted as GPU work alone, the scout passes add 22% to 29% to a ViT-B run. At ViT-L, where the GPU sets the step time, they added 16% to run 11, our main run.
For the match itself, our cluster mixes SXM and PCIe variants of the H100 node, and the same run is slower on the PCIe nodes, so a match by measured GPU hours is unreliable in our work. We matched on the step times of the SXM nodes, where a curated step takes 1.27 times as long as a baseline step, and this sets the curated run at 200k steps equal to the baseline at 240k steps. A hardware-independent count of the scout passes gives 254k steps, so we also report the baseline at 260k steps.
A simpler feature-space resampling baseline would not avoid this cost, since it also needs a forward pass over every tile that it chooses from. Done offline, that is an oracle encoder and one pass over all 286.6M tiles of our corpus before the first training step, and again whenever slides are added. Our curator scores each tile in the step that draws it, with no oracle encoder and no stored embeddings, which is where it is most useful in practice. Training this baseline would take a further ten-day ViT-L run, so we give its cost here.
Table 1. The new ViT-L pair at matched compute. Both runs use a constant schedule, as in DINOv3 (Siméoni et al., 2025), so that every checkpoint is a usable end point. Each difference is the curated run at 200k steps minus its baseline, in pp of AUROC, as the mean of the fold-paired differences over the 60 task-folds, with a 95% interval from a bootstrap over tasks.
| Baseline at | Macro-mean AUROC, curated against baseline | Difference, all slides | Folds won of 60 | Difference, patient-disjoint folds | Difference, fixed epoch (120) |
|---|---|---|---|---|---|
| 240k steps | 75.5 against 75.5 | +0.01 (−0.83 to +0.87) | 31 | −0.19 (−1.02 to +0.77) | +0.24 (−0.53 to +1.10) |
| 260k steps | 75.5 against 75.2 | +0.36 (−0.41 to +1.21) | 36 | +0.25 (−0.72 to +1.30) | +0.82 (−0.29 to +1.93) |
In conclusion, we trained the comparison at matched compute that you asked for, and the curated run is level with standard pre-training (75.5 against 75.5). A feature-space sampler would need the same forward passes as our curator, or an oracle encoder. We will add Table 1, with its per-task results, to the appendix. If this answers your question, we would be grateful if you would revisit your score.
References. Mindermann S. et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. ICML 2022, PMLR 162:15630–15649. Oquab M. et al. DINOv2: Learning robust visual features without supervision. TMLR 2024. Siméoni O. et al. DINOv3. arXiv:2508.10104, 2025.
Response to 43bU Weakness 4
Weakness 4. Comparison with existing pathology foundation models. The supplementary material compares the proposed models with UNI, Virchow, Virchow2, Prov-GigaPath, and H-Optimus-0. However, substantial performance gaps remain on several tasks. For example, on COAD/BRAF, ViT-L (run 11) achieves an AUROC of 69.4, compared with 80.9 for Virchow2 and 80.5 for H-Optimus-0. Although the primary goal is to study online data curation rather than to develop a new state-of-the-art model, the authors should clarify the practical significance of these results and whether the proposed strategy can also benefit stronger pre-trained encoders.
We thank the reviewer for this question, and we are happy to clarify both points.
The gap to the published encoders reflects the scale of their training, i.e., the scale of their data, and the epochs covered during training. Our 125k-step ViT-L runs see each tile 1.34 times, whereas UNI saw each of its tiles 3.8 times (Section 4 of the paper), and the other four encoders were trained on 171,189 to 3.1M slides against our 69,790 (Table 8 of the paper). When we trained longer, the gap narrowed. A new ViT-L baseline that we trained for the rebuttal period uses a constant schedule (like DINOv3) and reaches a macro-mean AUROC of 75.45 at 200k steps, compared with 74.21 for run 9 at 124k steps, which brings the gap to UNI from 5.73 to 4.49 pp. We plan to extend this to 400k iterations to further close this gap, hopefully within the discussion period (we will provide the results table to you as an addendum to this response if possible). However, we cannot extend the curated counterpart to 400k steps within the discussion period.
The practical significance of our results lies in the controlled comparison. Each curated run is compared with a baseline that has the same corpus and the same number of steps, so its improvement (+0.91 pp for run 11, our main ViT-L run) is what curation adds at a fixed budget. Curation acts only on which tiles enter the batch and leaves the training objective as it is, so it can be added to the training of a stronger encoder as well, and our response to Weakness 2 gives the curated counterpart of this baseline at 200k steps.
In conclusion, the gap to the published encoders reflects the scale of their training, and it narrowed by 1.24 pp when we trained longer (from 74.21 to 75.45). Our results show what curation adds at a fixed budget (+0.91 pp for run 11). If this clarifies the practical significance of our results, I hope you would take our response into consideration when revisiting the score.
Response to 43bU Weakness 3
Weakness 3. Reproducibility and training dynamics. Most curated runs resume from their corresponding baseline checkpoints, so the comparisons share the same pre-training history. While this helps isolate the effect of curation, it remains unclear whether the gains persist across independent training runs. The paper mainly reports downstream results at the final checkpoints, while the training dynamics analysis focuses on the density cache rather than downstream performance. Independent pre-training runs with different random seeds and downstream evaluations at intermediate checkpoints would help clarify the consistency of the reported gains.
We thank the reviewer for these points. They are fair, and we would like to answer them plainly.
On independent runs, most curated runs start from a checkpoint of their baseline, as the reviewer notes. We did this so that a curated run and its baseline differ only in the tiles that they see afterwards. The paper still has three baselines that we trained separately, and Table 1 lists the improvement of every curated run over its baseline. The size of the improvement varies, and its sign does not. We did not repeat a run with only the seed changed. One ViT-L run takes more than a week on eight H100 GPUs, and we chose to spend our compute on more settings and a second model size. We agree that seed repeats would add to this evidence.
On intermediate checkpoints, the paper reports every encoder at the end of its schedule. Our learning rate, weight decay and teacher momentum follow cosine schedules over the full run (Table 3 of the paper). A checkpoint from the middle of a run therefore comes from a model whose training is still under way, and we do not think that it gives a fair comparison. Siméoni et al. (2025) point to a related limit of such schedules in DINOv3, where "employing cosine schedules implies knowing the optimization horizon a priori".
Table 1. Improvement of each curated run over its baseline, in pp of AUROC, as the mean over the 60 task-folds (Table 2 of the paper).
| Baseline, trained separately | Curated runs | Improvement of each curated run |
|---|---|---|
| Run 1, ViT-B | 2 to 8 | +0.80, +0.75, +0.74, +0.62, +0.87, +0.82, +1.04 |
| Run 9, ViT-L | 10 to 12 | +0.26, +0.91, +0.42 |
| Run 13, ViT-B with 500k steps | 14 | +0.28 |
We hope we have addressed your concern here as much as was possible in this discussion period.
References. Siméoni O. et al. DINOv3. arXiv:2508.10104, 2025.
Response to 43bU Weakness 1
We thank the reviewer for this careful reading of our evaluation. We ran both of the analyses that the reviewer proposes, and report that the improvements fortunately remain under each (Table 1).
For the patient-disjoint evaluation, we removed from each held-out fold every slide whose patient also has a slide in the training part of that split (the 8% and 21% that the reviewer cites), and we scored the same trained heads on the slides that were left. Run 11, our main ViT-L run, then improves on its baseline by +1.00 pp, with a 95% interval of +0.40 to +1.57 pp, compared with +0.91 pp on all slides. The improvement becomes larger in ten of the eleven comparisons.
For a model selection that does not rely on the held-out set, we fixed the last epoch (120) for every encoder, task and split, so that nothing is selected at all. Run 11 then improves by +1.23 pp, with a 95% interval of +0.12 to +2.51 pp, and by +1.21 pp on the patient-disjoint folds.
For the overlapping held-out sets (any two of them share 27% of their slides), we applied the correction of Nadeau and Bengio (2003), which widens the standard error of the precision-weighted mean by a factor of 1.87 (1.81 on the patient-disjoint folds). The improvement of run 11 remains significant in both cases (p=0.008 and p=0.004). A bootstrap over patients, which keeps all slides of a patient together across the five splits, gives +0.91 pp with a 95% interval of +0.06 to +1.79 pp.
Table 1. Curated run minus its baseline, in pp of AUROC, as the mean of the fold-paired differences over the 60 task-folds. Runs 2 to 8 are ViT-B, runs 10 to 12 are ViT-L, and run 14 is the ViT-B pair at 500k steps. "Best epoch" is the protocol of the paper, and "epoch 120" fixes the last epoch for every encoder, task and split. The first column of runs 10 and 12 is recomputed from the stored per-slide scores (+0.26 and +0.42 in the paper).
In conclusion, we ran both of the analyses that you proposed, and the improvements remain under each (+1.00 pp for run 11 on patient-disjoint folds and +1.23 pp at a fixed epoch). The improvement of run 11 also holds with the overlap of the held-out sets taken into account (p=0.008). We should add Table 1 to the appendix for our final revision, which we aim to do on the final day of discussions. We hope you will take this requested evidence into consideration.
References. Nadeau C., Bengio Y. Inference for the generalization error. Machine Learning 52:239–281, 2003.