- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes Patch-JEPA, a cross-scale latent predictive pretraining framework for whole-slide images. It uses aligned fine-scale patches as pretraining targets so that a coarse-scale encoder learns representations informed by fine morphology, while inference requires only coarse patches. The method avoids direct coarse–fine matching or fine-patch reconstruction by predicting fine-scale latent structure from coarse representations. Experiments on TCGA-RCC, TCGA-BRCA, TCGA-UCEC, and Camelyon16 show that Patch-JEPA is competitive with standard SSL, cross-scale SSL, multi-scale MIL, and slide-level foundation-model baselines, while reducing the need for fine-scale inference-time computation.
Critical strengths
-
The framework is interesting because it shifts cross-scale computation from inference to pretraining. At test time, Patch-JEPA only uses the coarse-scale encoder, avoiding fine patch encoding and explicit multi-scale fusion.
-
The empirical evaluation is relatively broad. The paper compares against single-scale MIL, cross-scale MIL, slide-level foundation models, standard SSL objectives, and cross-scale SSL objectives across four pathology benchmarks.
Critical weaknesses
-
The main weakness is that the performance gains are not uniformly strong across datasets and endpoints. Patch-JEPA is competitive, but it is not always clearly better than strong baselines. For example, in the main system-level table, GigaPath is stronger on BRCA AUROC, and several cross-scale MIL methods are as strong as or better on other metrics. The appendix also notes that survival and stage results are mixed, with the benefit clearest on RCC and more variable on BRCA/UCEC.
-
The method depends on accurate aligned coarse–fine patch extraction during pretraining. This may be nontrivial across scanners, tissue preparation pipelines, pyramid-level metadata, and institutions. The authors acknowledge that Patch-JEPA requires aligned coarse–fine regions and that coarse-only inference cannot recover fine cellular details absent from the coarse view.
-
Although the paper evaluates several benchmarks, the scope is still limited to a small set of public pathology cohorts and mostly classification-style downstream tasks. Larger multi-institutional pretraining corpora, stronger pathology foundation encoders, and more clinically diverse endpoints would make the method's generality more convincing.
-
The emphasis on inference efficiency is not fully convincing from a clinical perspective. In medical diagnosis, users may tolerate longer inference times if it leads to more accurate or reliable predictions.
Questions for authors
-
The gains appear strongest on some datasets and endpoints but more mixed on others. Can the authors provide a more detailed analysis of why Patch-JEPA works better on certain cohorts, such as RCC, but is less consistently beneficial on BRCA, UCEC, or Camelyon16?
-
Can the authors provide statistical significance tests or paired comparisons to support the claim that Patch-JEPA is consistently better or competitive across benchmarks?
-
How sensitive is the method to the choice of frozen fine-scale target encoder? Would using stronger pathology foundation encoders as the fine-scale target change the results?
3 comments
Follow-up on significance tests and pretraining data
Dear Reviewer N56K,
We would like to follow up on our response. On significance, we ran paired t-tests over ten downstream seeds on RCC with Bonferroni correction (Appendix C.2, Table R1). On the larger gain on RCC, we measured how much of the feature variance the slide label explains at coarse and fine scale. On pretraining data, we added transfer to Camelyon16, BRACS and PANDA (Table 5) and the data-scaling results (Table R5). We also revised the limitations and the efficiency wording in Section 5 and the Conclusion.
Please let us know whether these address your concerns. We can still answer follow-up questions before the discussion period ends.
Dear Reviewer N56K,
As a brief follow-up on your question about pretraining data: in our Nov 18 reply, scaling pretraining from 565 to 2,406 slides improved all 24 evaluated settings on Camelyon16, BRACS and PANDA, in addition to the significance tests (Appendix C.2, Table R1) reported earlier.
We would be grateful to hear whether these results address your concerns.
We sincerely thank Reviewer N56K for the thoughtful questions. Changes made in response to this review are highlighted in orange in the revised paper, and in purple where another reviewer asked for the same change.
Response to W1·Q2: Uneven gains, and significance tests.
Patch-JEPA reads only coarse patches and infers with a linear probe (1.5K downstream parameters), while the MIL baselines read fine patches with 0.4M–4.8M parameters. It is first or second in 9 of the 12 columns of Table 1, and where it is not first the gap is within its fold SD, which Section 3.2 now states. In Table 2, where only the objective changes, it has the highest ACC in all four cohorts. For significance, we fixed the encoders on RCC fold 0, repeated downstream training with 10 seeds and ran paired t-tests against four baselines (Appendix C.2, Table R1). Patch-JEPA is higher in every comparison, and all but MSPN AUROC remain significant after Bonferroni correction (threshold 0.05/12). The first row is the mean ± SD over the 10 seeds, and the other rows give the difference and the two-sided p.
Response to W2: Reliance on aligned coarse-fine extraction.
Alignment reads the same region once at each magnification and splits the fine image 4×4, so it needs only the pyramid-level metadata that fixed-magnification MIL pipelines already use (Appendix B.2). The multi-institution TCGA cohorts and Camelyon16, scanned at two institutions with different scanners, were all extracted this way. Patch-JEPA predicts fine representations in latent space and does not recover cellular detail, and the Discussion now says so.
Response to W3·Q3: Cohorts and tasks, and the choice of target encoder.
Beyond classification, the paper already evaluates survival and stage (Appendix Table 12), transfer to unseen cohorts (Table 5) and fine latent-field regression (Table 4). We do not use a pathology foundation encoder as the target on purpose: with a strong target, a gain could not be attributed to the objective, so we use an ImageNet-1K ViT-S/16 with the same architecture as the context encoder, and Table 6 shows a variant that updates it by EMA. Large multi-institution corpora are outside our scope, so we tested more pretraining data instead. Adding BRACS, breast tissue like BRCA, to UCEC raised all three BRCA metrics of Patch-JEPA, and Patch-JEPA is highest under every pretraining set (Appendix C.6, Table R5; five-fold mean ± SD on BRCA).
Response to W4: Efficiency from a clinical perspective.
We agree that accuracy comes first. Patch-JEPA has higher ACC than the MIL pipelines that read fine patches at inference in all four cohorts (Table 1) while reading 16 times fewer patches, so longer inference would not buy accuracy here. Fewer patches also mean faster inference (Figure 4) and less storage per slide, costs that accumulate as hospitals process and archive slides every day. The Abstract and Conclusion now describe efficiency this way.
Response to Q1: Why the gain is largest on RCC.
Patch-JEPA has the highest ACC in all four cohorts under the same linear probe (Table 2). To see why the gain is largest on RCC, we measured on sampled slides the share of site-feature variance explained by the slide label (η², frozen ImageNet ViT-S/16 features) at the coarse and fine views. From coarse to fine, this share rises by 1.6 percentage points on RCC and by 0.3 in the other cohorts (table below). This does not establish the cause, but since Patch-JEPA learns to predict fine-scale targets from the coarse view (Table 4), we think the gain may be largest on RCC because more of its label information lies at the fine scale.