acceptodds
Under review as a conference paper at ICLR 2027

Cross-Scale Latent Predictive Pretraining for Whole-Slide Image

Abstract

Whole-slide images (WSIs) contain diagnostic information across magnifications, from tissue architecture to cellular morphology. Multi-scale MIL methods often encode patches from different magnifications independently and combine their representations post hoc, so the coarse representation may fail to preserve cues predictive of fine-scale morphology. Using both scales also requires fine patch encoding at inference. Cross-scale self-supervised learning (SSL) introduces interaction during pretraining, but direct representation matching or fine-patch reconstruction does not account for cross-scale asymmetry because fine views contain details unavailable from coarse inputs. We introduce **Patch-JEPA**, a cross-scale latent predictive pretraining framework. A frozen fine-scale target encoder produces an ordered fine-scale latent field from aligned fine patches. Separate prediction branches map coarse representations to this field and an aggregated coarse-scale target, training the coarse-scale context encoder without direct cross-scale matching or fine-patch reconstruction. Fine-scale representations are used only as pretraining targets, and the retained context encoder uses only coarse patches for coarse-only inference. Patch-JEPA is competitive with standard SSL, cross-scale SSL, and post-hoc multi-scale WSI pipelines on four pathology benchmarks. On RCC, ridge probing yields \(R^2 = 0.450\) for aligned fine-scale latent-field prediction, compared with \(0.231\) for Cross-MAE. Model-side inference cost remains close to a single-scale baseline under a fixed-bag timing protocol.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computational Pathology, Whole-Slide Image Analysis, Self-Supervised Learning, Latent Predictive Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_N56KOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Patch-JEPA, a cross-scale latent predictive pretraining framework for whole-slide images. It uses aligned fine-scale patches as pretraining targets so that a coarse-scale encoder learns representations informed by fine morphology, while inference requires only coarse patches. The method avoids direct coarse–fine matching or fine-patch reconstruction by predicting fine-scale latent structure from coarse representations. Experiments on TCGA-RCC, TCGA-BRCA, TCGA-UCEC, and Camelyon16 show that Patch-JEPA is competitive with standard SSL, cross-scale SSL, multi-scale MIL, and slide-level foundation-model baselines, while reducing the need for fine-scale inference-time computation.

Critical strengths
  1. The framework is interesting because it shifts cross-scale computation from inference to pretraining. At test time, Patch-JEPA only uses the coarse-scale encoder, avoiding fine patch encoding and explicit multi-scale fusion.

  2. The empirical evaluation is relatively broad. The paper compares against single-scale MIL, cross-scale MIL, slide-level foundation models, standard SSL objectives, and cross-scale SSL objectives across four pathology benchmarks.

Critical weaknesses
  1. The main weakness is that the performance gains are not uniformly strong across datasets and endpoints. Patch-JEPA is competitive, but it is not always clearly better than strong baselines. For example, in the main system-level table, GigaPath is stronger on BRCA AUROC, and several cross-scale MIL methods are as strong as or better on other metrics. The appendix also notes that survival and stage results are mixed, with the benefit clearest on RCC and more variable on BRCA/UCEC.

  2. The method depends on accurate aligned coarse–fine patch extraction during pretraining. This may be nontrivial across scanners, tissue preparation pipelines, pyramid-level metadata, and institutions. The authors acknowledge that Patch-JEPA requires aligned coarse–fine regions and that coarse-only inference cannot recover fine cellular details absent from the coarse view.

  3. Although the paper evaluates several benchmarks, the scope is still limited to a small set of public pathology cohorts and mostly classification-style downstream tasks. Larger multi-institutional pretraining corpora, stronger pathology foundation encoders, and more clinically diverse endpoints would make the method's generality more convincing.

  4. The emphasis on inference efficiency is not fully convincing from a clinical perspective. In medical diagnosis, users may tolerate longer inference times if it leads to more accurate or reliable predictions.

Questions for authors
  1. The gains appear strongest on some datasets and endpoints but more mixed on others. Can the authors provide a more detailed analysis of why Patch-JEPA works better on certain cohorts, such as RCC, but is less consistently beneficial on BRCA, UCEC, or Camelyon16?

  2. Can the authors provide statistical significance tests or paired comparisons to support the claim that Patch-JEPA is consistently better or competitive across benchmarks?

  3. How sensitive is the method to the choice of frozen fine-scale target encoder? Would using stronger pathology foundation encoders as the fine-scale target change the results?

3 comments

We sincerely thank Reviewer N56K for the thoughtful questions. Changes made in response to this review are highlighted in orange in the revised paper, and in purple where another reviewer asked for the same change.


Response to W1·Q2: Uneven gains, and significance tests.

Patch-JEPA reads only coarse patches and infers with a linear probe (1.5K downstream parameters), while the MIL baselines read fine patches with 0.4M–4.8M parameters. It is first or second in 9 of the 12 columns of Table 1, and where it is not first the gap is within its fold SD, which Section 3.2 now states. In Table 2, where only the objective changes, it has the highest ACC in all four cohorts. For significance, we fixed the encoders on RCC fold 0, repeated downstream training with 10 seeds and ran paired t-tests against four baselines (Appendix C.2, Table R1). Patch-JEPA is higher in every comparison, and all but MSPN AUROC remain significant after Bonferroni correction (threshold 0.05/12). The first row is the mean ± SD over the 10 seeds, and the other rows give the difference and the two-sided p.

ACCAUROCF1
Patch-JEPA0.9011 ± 0.00450.9755 ± 0.00170.8854 ± 0.0079
vs ABMIL+0.0335 (7.1e-6)+0.0133 (6.2e-8)+0.0379 (1.7e-6)
vs MSPN+0.0293 (1.5e-5)+0.0052 (0.0218)+0.0359 (6.9e-5)
vs GigaPath+0.0262 (7.3e-6)+0.0024 (2.1e-4)+0.0437 (2.6e-6)
vs X-MAG+0.0553 (5.3e-8)+0.0304 (2.1e-5)+0.0588 (3.9e-7)

Response to W2: Reliance on aligned coarse-fine extraction.

Alignment reads the same region once at each magnification and splits the fine image 4×4, so it needs only the pyramid-level metadata that fixed-magnification MIL pipelines already use (Appendix B.2). The multi-institution TCGA cohorts and Camelyon16, scanned at two institutions with different scanners, were all extracted this way. Patch-JEPA predicts fine representations in latent space and does not recover cellular detail, and the Discussion now says so.


Response to W3·Q3: Cohorts and tasks, and the choice of target encoder.

Beyond classification, the paper already evaluates survival and stage (Appendix Table 12), transfer to unseen cohorts (Table 5) and fine latent-field regression (Table 4). We do not use a pathology foundation encoder as the target on purpose: with a strong target, a gain could not be attributed to the objective, so we use an ImageNet-1K ViT-S/16 with the same architecture as the context encoder, and Table 6 shows a variant that updates it by EMA. Large multi-institution corpora are outside our scope, so we tested more pretraining data instead. Adding BRACS, breast tissue like BRCA, to UCEC raised all three BRCA metrics of Patch-JEPA, and Patch-JEPA is highest under every pretraining set (Appendix C.6, Table R5; five-fold mean ± SD on BRCA).

MethodPretraining dataBRCA ACCBRCA AUROCBRCA F1
MPCSUCEC0.823 ± 0.0340.752 ± 0.0800.497 ± 0.059
+ BRACS0.836 ± 0.0220.705 ± 0.0890.568 ± 0.068
+ BRACS + Camelyon160.831 ± 0.0210.757 ± 0.0310.591 ± 0.028
X-MAGUCEC0.825 ± 0.0280.673 ± 0.0700.459 ± 0.023
+ BRACS0.821 ± 0.0260.661 ± 0.1000.457 ± 0.017
+ BRACS + Camelyon160.825 ± 0.0300.724 ± 0.0750.494 ± 0.056
Patch-JEPAUCEC0.841 ± 0.0400.807 ± 0.0610.598 ± 0.080
+ BRACS0.857 ± 0.0320.849 ± 0.0490.653 ± 0.065
+ BRACS + Camelyon160.854 ± 0.0250.836 ± 0.0520.652 ± 0.037

Response to W4: Efficiency from a clinical perspective.

We agree that accuracy comes first. Patch-JEPA has higher ACC than the MIL pipelines that read fine patches at inference in all four cohorts (Table 1) while reading 16 times fewer patches, so longer inference would not buy accuracy here. Fewer patches also mean faster inference (Figure 4) and less storage per slide, costs that accumulate as hospitals process and archive slides every day. The Abstract and Conclusion now describe efficiency this way.


Response to Q1: Why the gain is largest on RCC.

Patch-JEPA has the highest ACC in all four cohorts under the same linear probe (Table 2). To see why the gain is largest on RCC, we measured on sampled slides the share of site-feature variance explained by the slide label (η², frozen ImageNet ViT-S/16 features) at the coarse and fine views. From coarse to fine, this share rises by 1.6 percentage points on RCC and by 0.3 in the other cohorts (table below). This does not establish the cause, but since Patch-JEPA learns to predict fine-scale targets from the coarse view (Table 4), we think the gain may be largest on RCC because more of its label information lies at the fine scale.

CohortCoarseFine
RCC0.5%2.1%
BRCA0.2%0.5%
UCEC0.2%0.5%
Camelyon160.1%0.4%
AuthorsOpenReview ↗

Follow-up on significance tests and pretraining data

Dear Reviewer N56K,

We would like to follow up on our response. On significance, we ran paired t-tests over ten downstream seeds on RCC with Bonferroni correction (Appendix C.2, Table R1). On the larger gain on RCC, we measured how much of the feature variance the slide label explains at coarse and fine scale. On pretraining data, we added transfer to Camelyon16, BRACS and PANDA (Table 5) and the data-scaling results (Table R5). We also revised the limitations and the efficiency wording in Section 5 and the Conclusion.

Please let us know whether these address your concerns. We can still answer follow-up questions before the discussion period ends.

Dear Reviewer N56K,

As a brief follow-up on your question about pretraining data: in our Nov 18 reply, scaling pretraining from 565 to 2,406 slides improved all 24 evaluated settings on Camelyon16, BRACS and PANDA, in addition to the significance tests (Appendix C.2, Table R1) reported earlier.

We would be grateful to hear whether these results address your concerns.

Reviewer_6LrfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes Patch-JEPA, an SSL pretraining method for WSI analysis that uses both low and high mags. The key idea is to train an encoder on low-magnification patches so that it can predict the high-magnification features of the same region. Training uses two targets: one is a single pooled summary of the high-mag features, and the other is the full grid of high-mag features with their sequential order. At inference, only the low-mag encoder is used, so the high-mag branch is needed only during training. The authors test the method on four pathology datasets, mainly on slide-level classification, with some extra experiments on staging and survival prediction.

Critical strengths
  1. The paper adapts JEPA-style latent prediction for learning the correlation between different pathology magnifications. The ordered fine-scale targets and complementary prediction branches provide a novel approach to incorporating fine-scale supervision into coarse representations.

  2. Discarding the fine-scale branch after pretraining reduces inference computation and addresses a practical need for low-resolution pathology encoders. This direction is relevant to rapid slide analysis and diagnostic triage, where repeatedly processing high-resolution patches is very costly.

Critical weaknesses
  1. Section 3.1 and Appendix C.1 state that self-supervised pretraining uses all available unlabeled WSIs in each evaluation corpus, followed by five-fold downstream evaluation. Consequently, the pretrained encoder has already been pretrained using images from the downstream test folds. Withholding their labels does not ensure an independent test set: the five-fold split isolates downstream supervision but does not isolate representation learning. Under a standard pathology FM evaluation protocol, this constitutes test-image leakage and prevents the main results from establishing generalization to previously unseen patients. The authors should explicitly identify these results as transductive and provide evaluations with a patient-level split for pretraining and testing, either through training-fold-only pretraining or an independent pretraining corpus. Or just pretrain on TCGA and evaluate on other public datasets.

  2. As a patch-level pretraining framework, the downstream evaluation is predominantly on slide-level analysis. Although the paper includes latent-field prediction, retrieval, and image-derived measurements, these analyses do not establish that coarse representations preserve diagnostically meaningful local morphology. In particular, predicting representations closely related to the pretraining targets is not a substitute for evaluation against independent pathological annotations. Patch-level tissue or lesion classification would provide more direct evidence for the central claim that fine-scale supervision improves the morphological information retained at low resolution.

  3. Pretraining is conducted separately within four relatively small evaluation datasets. This setting primarily demonstrates adaptation to individual cohorts, rather than establishing a broadly transferable pathology pretraining strategy. The paper does not show how performance changes with increasing pretraining data or model scale. The two cross-domain experiments are useful, but stronger evidence would require an independent, diverse pretraining corpus, evaluation of a fixed pretrained encoder across unseen cohorts, and data-scaling experiments demonstrating that the proposed objective remains beneficial as pretraining grows.

  4. The appendix shows that UNI2-H outperforms Patch-JEPA on BRCA and UCEC. Virchow2 evaluated at its supported 5× magnification also achieves higher AUROC on these datasets: 0.916 and 0.836, compared with 0.906 and 0.798 for Patch-JEPA-B. Patch-JEPA performs better on other endpoints and uses a smaller encoder, but these mixed results leave its practical advantage unresolved. Direct comparisons of predictive performance, inference latency, and memory consumption against SOTA foundation models, particularly native-5× Virchow2, are needed to substantiate the proposed performance–efficiency tradeoff.

Questions for authors
  1. Can the authors confirm whether test patients were excluded from all representation learning? If they were not excluded, do the reported improvements persist when pretraining and testing are strictly split at the patient level?

  2. How does Patch-JEPA perform when pretrained once on all TCGA WSIs and subsequently evaluated on CPTAC and other public non-TCGA benchmarks commonly used in pathology foundation-model studies? Please evaluate the fixed pretrained encoder without additional self-supervised adaptation to the test cohorts and compare it with current SOTA foundation models under matched downstream protocols.

7 comments

We sincerely thank Reviewer 6Lrf for the thoughtful questions. Changes made in response to this review are highlighted in green in the revised paper, and in purple where another reviewer asked for the same change.


Response to W1 and Q1: Transductive pretraining.

Patch-JEPA and every SSL method in Table 2 were pretrained once per cohort and evaluated with five-fold cross-validation, so test-fold slides were included in self-supervised pretraining under the same condition for every method; Section 3.1 and Appendix C.1 now state that this setting is transductive. For an inductive check on UCEC, we kept the paper's patient-level split, re-pretrained every SSL method on training-patient slides only with three seeds, and evaluated test patients with the same linear probe. As in Table 2, Patch-JEPA has the highest ACC (0.800); the results are in Appendix C.3.3 (Table R2) and stated in the main text. The table gives the mean over the three seeds.

MethodACCAUROCF1
SimCLR0.7780.5760.503
DINOv20.7840.7520.615
MAE0.7810.4550.439
MPCS0.7840.6790.453
X-MAG0.7810.5580.439
Cross-MAE0.7810.4680.439
Patch-JEPA0.8000.7270.554

Response to W2: No patch-level evaluation.

To check against annotations unrelated to the pretraining target, we ran patch-level lesion classification. Using the Camelyon16 tumor annotations, we labeled coarse patches as tumor or normal, pretrained each SSL method inductively on the official training slides only, and evaluated on the test slides. Patch-JEPA has the highest ACC and F1 (Appendix C.5, Table R4).

EncoderACCAUROCF1
Patch-JEPA0.9420.9450.781
DINOv20.9110.9080.585
MPCS0.9340.9730.738
Cross-MAE0.9010.8670.510

Response to W3 and Q2: Per-cohort pretraining, fixed-encoder transfer and data scaling.

Patch-JEPA is an SSL objective, not a foundation model. Foundation models are trained on different data, which makes a fair comparison hard, and re-pretraining every compared objective on all of TCGA exceeds our single shared GPU (Appendix Table 33), so we compare objectives on the same data and backbone (Table 2). To still check the trend, we added (a) fixed-encoder transfer to unseen cohorts (revised Table 5) and (b) pretraining-data scaling (Appendix C.6, Table R5). Patch-JEPA with a larger ViT-B/16 encoder is in Appendix Table 14, and foundation-model patch encoders under the same downstream protocol are in Tables 24, 25 and R3.

(a) Transfer to unseen cohorts. With the UCEC-pretrained encoders frozen, Patch-JEPA has the highest ACC and AUROC on Camelyon16 and BRACS and is below MPCS and X-MAG on PANDA (five-fold mean ± SD).

CohortEncoderACCAUROCF1
Camelyon16Patch-JEPA0.619 ± 0.0350.654 ± 0.0330.522 ± 0.036
Cross-MAE0.601 ± 0.0070.533 ± 0.0530.392 ± 0.017
X-MAG0.594 ± 0.0060.492 ± 0.0120.373 ± 0.002
MPCS0.602 ± 0.0320.613 ± 0.0470.496 ± 0.036
BRACSPatch-JEPA0.386 ± 0.0520.674 ± 0.0400.199 ± 0.020
Cross-MAE0.295 ± 0.0450.570 ± 0.0510.103 ± 0.027
X-MAG0.318 ± 0.0490.637 ± 0.0470.115 ± 0.033
MPCS0.381 ± 0.0650.653 ± 0.0390.223 ± 0.042
PANDAPatch-JEPA0.439 ± 0.0060.755 ± 0.0050.359 ± 0.018
Cross-MAE0.356 ± 0.0090.681 ± 0.0060.219 ± 0.008
X-MAG0.454 ± 0.0120.767 ± 0.0040.380 ± 0.020
MPCS0.457 ± 0.0040.776 ± 0.0020.383 ± 0.013

(b) Pretraining-data scaling. We re-pretrained MPCS, X-MAG and Patch-JEPA on UCEC, + BRACS and + BRACS + Camelyon16 and evaluated on TCGA-BRCA; adding BRACS raises all three metrics of Patch-JEPA, and Patch-JEPA has the highest ACC, AUROC and F1 under every pretraining set (five-fold mean ± SD).

MethodPretraining dataBRCA ACCBRCA AUROCBRCA F1
MPCSUCEC0.823 ± 0.0340.752 ± 0.0800.497 ± 0.059
+ BRACS0.836 ± 0.0220.705 ± 0.0890.568 ± 0.068
+ BRACS + Camelyon160.831 ± 0.0210.757 ± 0.0310.591 ± 0.028
X-MAGUCEC0.825 ± 0.0280.673 ± 0.0700.459 ± 0.023
+ BRACS0.821 ± 0.0260.661 ± 0.1000.457 ± 0.017
+ BRACS + Camelyon160.825 ± 0.0300.724 ± 0.0750.494 ± 0.056
Patch-JEPAUCEC0.841 ± 0.0400.807 ± 0.0610.598 ± 0.080
+ BRACS0.857 ± 0.0320.849 ± 0.0490.653 ± 0.065
+ BRACS + Camelyon160.854 ± 0.0250.836 ± 0.0520.652 ± 0.037

Response to W4: Practical advantage over UNI2-H and Virchow2.

Predictive performance against both models is already in Appendix Table 24, and the revision adds the inference cost in Table R3. Patch-JEPA-B is within 0.04 AUROC of Virchow2 at the fine scale on BRCA and UCEC and higher on RCC and Camelyon16, with about 10 times fewer FLOPs per patch and 16 times fewer patches per slide. AUROC below is at the fine scale for the foundation models and at the coarse scale for Patch-JEPA.

EncoderParamsPretraining dataGFLOPs/patchLatency (ms/patch)Memory (GB)RCC AUROCBRCA AUROCUCEC AUROCC16 AUROC
Virchow2632M3.1M WSIs170.14.034.650.9650.9160.8360.673
UNI2-H681M>300k WSIs185.54.075.130.9700.9400.8730.770
Patch-JEPA (ViT-S/16)21.7M399–938 WSIs4.60.090.340.9830.8720.7750.743
Patch-JEPA (ViT-B/16)85.8M399–938 WSIs17.60.220.870.9880.9060.7980.773

Thank you for the follow-up. It appears under our response to Reviewer hg8V; we reply here.


We read your comments as three requests: (1) inductive pretraining and patch-level evaluation (W1, W2), (2) larger pretraining data and current MIL methods (this follow-up), and (3) pretraining at foundation-model scale with a direct comparison to foundation models (W3, W4, Q2).


(1) Inductive pretraining and patch-level evaluation. The results are in our earlier response: inductive pretraining on UCEC training patients in Table R2, and Camelyon16 patch-level lesion classification in Table R4.


(2) Larger pretraining data and current MIL. Results with larger pretraining data are in Table R5, and ABMIL and CLAM heads in Appendix Tables 21 and 22. We added two MIL methods published in 2026, ASMIL (ICLR 2026) and CAR-MIL (ECCV 2026), from their official code. With the encoders pretrained on the largest set of Table R5 (UCEC+BRACS+Camelyon16) frozen, and the inputs and five folds of Table R5 on TCGA-BRCA (mean ± SD, seed 0; CAR-MIL without its baseline warm start), Patch-JEPA is highest with both heads.

MILEncoderACCAUROCF1
ASMILMPCS0.819 ± 0.0260.759 ± 0.0340.627 ± 0.030
X-MAG0.818 ± 0.0280.756 ± 0.0490.523 ± 0.085
Cross-MAE0.823 ± 0.0270.648 ± 0.0740.451 ± 0.008
Patch-JEPA0.849 ± 0.0400.840 ± 0.0500.702 ± 0.036
CAR-MILMPCS0.806 ± 0.0310.802 ± 0.0560.602 ± 0.047
X-MAG0.824 ± 0.0460.783 ± 0.0470.616 ± 0.123
Cross-MAE0.816 ± 0.0390.697 ± 0.0700.502 ± 0.069
Patch-JEPA0.861 ± 0.0310.879 ± 0.0330.745 ± 0.042

Please let us know whether (1) and (2) address your concerns.


(3) Foundation-model scale. You asked for two things: a comparison using an encoder pretrained once on all of TCGA and kept fixed (Q2), and a direct comparison of performance, inference latency and memory with foundation models including Virchow2 (W4). Both are questions of foundation-model development and differ from the question of this paper. Foundation-model work uses an existing SSL objective, mostly DINOv2, and scales data and model size to obtain a general encoder (UNI2-H: more than 300,000 slides and 681M parameters; Virchow2: 3.1M slides and 632M parameters; Table R3). Within our resources, we provide the following.

(a) Large-scale data. Training and evaluating a ViT-Large/Huge model on the slide collections used by foundation models (more than 300,000 for UNI2-H, 3.1M for Virchow2) is not possible with our resources within the discussion period. Instead, to see how performance changes as data grows, we enlarged the pretraining data up to a combination of three cohorts and compared Patch-JEPA with other SSL methods.

(b) Efficiency. Tables 24 and R3 compare performance, inference latency and memory directly with Virchow2, UNI2-H and other foundation models. Per patch, Patch-JEPA takes 0.22 ms and 0.87 GB, Virchow2 4.03 ms and 4.65 GB, and UNI2-H 4.07 ms and 5.13 GB. The Limitations paragraph states that larger corpora and backbones are untested.

Your assessment has focused on large-scale expansion. We ask that the paper be evaluated again within its scope.

Reviewer_XV2iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

There is one consistent problem in WSI diagnosis that patch representations may fail to fully integrate complementary information across different magnification levels. To end this, this paper proposes a cross-scale latent predictive pretraining framework, which enables fine-grained predictive results using only coarse-grained patches. Compared with various pipelines, experiments on four benchmarks show the effectiveness of this proposed method.

Critical strengths
  • The ability to perform fine-grained analysis using only coarse-grained patches is highly significant.
Critical weaknesses
  • The limited scope of the training objectives and the downstream tasks used for validation raise questions as to whether the core motivation of the method has been sufficiently supported. I think it is neccessary to provide more validation.
Questions for authors
  • The paper writing is disorganized. In addition, some module names in the figures don't match those used in the main text.

  • How is it demonstrated that coarse-grained patches can directly capture fine-grained semantic information? I think pixel reconstruction is a good strategy. But the authors claimed that it was unsuitable for this context, and they didn't provide enough analysis.

  • The authors only evaluated the method on classification tasks. For a claimed generalizable method, the demonstration of its ability is not enough.

4 comments

We sincerely thank Reviewer XV2i for the thoughtful questions. Changes made in response to this review are highlighted in blue in the revised paper, and in purple where another reviewer asked for the same change.


Where the submission reports the requested results

The submission already reported the results requested in W, Q2 and Q3 at the locations below. Section and main-text table numbers are unchanged in the revision, and Appendix Tables 11 and 22 are Tables 12 and 23 there.

PointRequestMain text of the submissionAppendix of the submission
WBroader objectives and tasksSection 3.3, Table 2; Section 3.2, Table 1Tables 11, 22
Q2Analysis of pixel reconstructionSection 3.3, Table 2; Section 3.5, Table 4-
Q3Tasks other than classificationSection 3.6, Table 5; Section 3.5, Table 4Table 11

For Q1, we replaced Figure 2 with the version that uses the notation of the text and added a guide table at the start of the appendix.


Response to W1: The objectives and tasks are narrow.

We compare the three main SSL families, contrastive alignment, distillation and reconstruction, under the same data, backbone and evaluation (Table 2), and evaluate three slide-level tasks: subtype and metastasis classification in four cohorts, and survival and stage in three TCGA cohorts (Table 1; Appendix Table 11 of the submission, Table 12 in the revision).


Response to Q1: Writing, and module names in the figures.

We submitted an earlier version of Figure 2 by mistake: it used the superscripts ℓ/h instead of c/f, and Context Encoder and Target Encoder instead of coarse-scale context encoder and fine-scale target encoder. We apologize for the confusion. The revision uses the current Figure 2 with the notation of the text, and Figure 1(c) now uses the same module names. The appendix now opens with a guide table of where each task and RQ is reported, and the RQ1 results point to the survival and stage table.


Response to Q2: Evidence that coarse patches carry fine information, and pixel reconstruction.

We agree that pixel reconstruction is a good strategy, and the paper compares it as a baseline. Coarse and fine views show the same tissue at different magnifications, and the fine view holds details absent from the coarse view. Because of this asymmetry, reconstructing fine pixels also trains the model to fit low-level detail the coarse input cannot predict, such as stain-dependent color and texture. Patch-JEPA therefore defines the target in latent space and learns only the fine-scale structure that coarse context can predict (Section 2). Patch-JEPA is above Cross-MAE, which reconstructs the other magnification on aligned views, and MAE in all four cohorts (Table 2), and it is highest in predicting the aligned fine-scale latent field from the frozen coarse representation (R² 0.450 vs. 0.231 for Cross-MAE, Table 4).


Response to Q3: Only classification is evaluated.

Besides classification in four cohorts, the paper already evaluates survival (C-index) and stage in three TCGA cohorts (Appendix Table 11 of the submission, Table 12 in the revision), where Patch-JEPA is highest on RCC survival and stage and on UCEC survival. It also evaluates transfer to cohorts and organs not used in pretraining (Table 5) and regression of the aligned fine-scale latent field from the frozen coarse representation (Table 4).

AuthorsOpenReview ↗

Follow-up on figures and appendix guide

Dear Reviewer XV2i,

Thank you for your review. We renamed the modules of Figure 1(c) to match the text, opened the appendix with a guide to the results (Appendix Table 7), and point to the survival and stage results from Section 3.2 (Appendix Table 12).

We would be glad to hear whether this answers your questions, and we can still clarify anything that remains before the discussion period closes.

Dear Reviewer XV2i,

We would like to kindly follow up before the discussion closes. Beyond the clarity fixes in our earlier reply (Figure 1(c) naming, appendix guide in Table 7, survival/stage results in Table 12), we have since added experiments with pathology foundation models (UNI2-H, CONCH v1.5) as targets on unseen cohorts (posted Nov 18).

We would appreciate any feedback on whether your concerns have been addressed.

I thank authors for these response. However, I find that the author response generally lacks empirical experimental support and therefore lacks more direct evidence. In their response, the authors use R² (0.450/0.231) to determine that coarse patches can capture fine-grained feature, but I think R² = 0.450 is only moderate and still indirect. I believe that some other validations such as the ablation study to isolate the negative effect of pixel-level reconstruction would better support the conclusion. Therefore, I have to maintain my original score.

Reviewer_hg8VOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1(−1)
Read review
Paper summary

This paper presents Patch-JEPA, a cross-scale latent predictive pretraining framework for whole-slide image analysis. The main motivation is that coarse- and fine-magnification pathology views are spatially aligned but contain asymmetric information, making direct cross-scale matching or reconstruction suboptimal. Patch-JEPA uses a frozen fine-scale target encoder to construct ordered fine-scale latent fields and trains a coarse-scale context encoder to predict both the fine-scale latent structure and an aggregated coarse-scale target. Only the coarse encoder is retained for downstream inference, avoiding fine-scale encoding at test time.

Critical strengths
  1. The paper is motivated by a meaningful and practical problem in computational pathology. Instead of relying on expensive multi-scale processing at inference, the work focuses on learning stronger coarse-view representations with fine-scale supervision during pretraining. This is consequential because coarse-only inference can substantially reduce computation while still retaining information predictive of fine-scale morphology.

  2. The cross-scale asymmetry is clearly identified and addressed with an appropriate predictive formulation. Patch-JEPA predicts fine-scale latent structure from coarse representations. This is a well-motivated design choice and is supported by the objective comparisons and ablation studies.

Critical weaknesses
  1. The target encoder is not described clearly enough. The paper states that a pretrained Vision Transformer (ViT) with frozen weights is used as the fine-scale target encoder, but the main text does not clearly specify which pretrained model is adopted. The authors should explicitly state whether the frozen fine encoder is based on UNI, CONCH, DINOv2, or another checkpoint.

  2. The comparison with slide-level foundation models is not sufficiently comprehensive. The evaluation would be stronger if more recent state-of-the-art slide foundation models, such as TITAN and CARE, were included as baselines. [1] Ding et al. A multimodal whole-slide foundation model for pathology [2] Zhang et al. CARE: A Molecular-Guided Foundation Model with Adaptive Region Modeling for Whole Slide Image Analysis

  3. Several important implementation details are missing. In particular, the paper does not clearly describe how positional information is encoded, nor whether masked tokens are replaced with learnable mask tokens, set to zero, or removed from the input sequence. These details are important for reproducibility. More importantly, if the model does not explicitly preserve positional information while being trained to recover spatially organized fine-scale tokens, this raises concerns under the JEPA-style predictive learning setting.

  4. The patch encoder used for the MIL baselines may be relatively weak. The MIL methods use a ViT-S/16 patch encoder, which may underestimate the performance of modern MIL pipelines. It would be more convincing to evaluate these methods with stronger pathology foundation encoders, such as UNI2 or CONCH v1.5, to ensure a fairer and more representative comparison.

  5. The slide-level aggregation strategy is overly simplistic. The paper constructs WSI representations using mean pooling over patch features, which is unlikely to be the strongest choice for slide-level modeling. Even a basic attention-based MIL model such as ABMIL would provide a more competitive and standard aggregation strategy. If the authors intend to position this work as an MIL-based WSI method, I would expect an ABMIL-style aggregator to be trained separately for each downstream task while keeping the patch encoder and other settings fixed.

  6. I have concerns about the fairness and reliability of the reported comparisons. Under otherwise matched settings, Patch-JEPA uses only mean pooling yet reportedly outperforms several stronger MIL and multi-scale methods, which I find somewhat surprising and would like to see better justified. Since the code has not been released, it is currently difficult to independently verify the implementation and experimental protocol. This limits my confidence in some of the reported results and conclusions.

Questions for authors

Please respond to the Critical Weaknesses one by one.

8 comments

We sincerely thank Reviewer hg8V for the thoughtful questions. Changes made in response to this review are highlighted in red in the revised paper.


Where the submission reports the requested results

Appendix table numbers are those of the submission; each is one higher in the revision.

PointRequestLocation in the submission
W1Target-encoder checkpointAppendix C.1, Table 33
W5Aggregators beyond mean pooling (ABMIL, CLAM)Appendix Tables 20, 21
W6Other patch features under mean poolingAppendix Table 23

For W2 and W4, we explained that these comparisons fall outside our scope and clarified what Table 1 compares. For W3, we redrew Figure 2 and added position encoding and mask-token handling to Section 2.2 and Appendix D.


Response to W1: Which checkpoint is the frozen target encoder?

The fine-scale target encoder uses frozen ImageNet-1K ViT-S/16 weights (timm vit_small_patch16_224.augreg_in1k). The submission gave this checkpoint in Appendix C.1 and Table 33, and the revision also names it in a footnote of Section 2.1. We did not use UNI, CONCH or DINOv2 weights because they are pretrained on large pathology or curated image collections, so with one of them as the target a gain could come from following a strong target rather than from our objective.


Response to W2: Why are TITAN and CARE not compared?

The slide encoders in Table 1, HIPT and GigaPath, are pretrained on WSI images only, as Patch-JEPA is. TITAN is aligned with pathology text and CARE with RNA and protein profiles, so a comparison with them would mix the pretraining objective with an added modality. Both are also slide encoders built on top of patch features, while Patch-JEPA pretrains the patch encoder itself.


Response to W3: Position encoding and mask-token handling.

Figure 2 was redrawn to show both, and Section 2.2 and Appendix D give the details.

Coarse-scale context encoder. The mask m_i^c is applied to the coarse patch before the encoder (the Mask box in Figure 2). The input pixels of masked tokens are set to zero and no token is removed, so the encoder processes all 196 tokens and receives tissue only at visible tokens. A fixed 2D sine-cosine position embedding is added to all 196 tokens, so a masked token enters with its position and no tissue.

Coarse-scale predictor. It receives the encoder outputs at visible tokens and, at each masked position, one learnable mask token shared by all positions, drawn as [M] in Figure 2. It adds the same position embedding to both, which tells it the positions to predict.

Fine-scale projector. A transposed convolution (kernel 4, stride 4) turns each coarse token into the 4×4 block of fine tokens covering the same tissue, so each predicted fine token has a fixed position on the 56×56 grid and is compared with the target at the same index.

Fine-scale target encoder and losses. The frozen target encoder uses its own learned position embedding inside each fine patch and encodes the aligned fine patches, which the context encoder never receives. ℓ_c is computed only at the masked coarse tokens m_i^c, and ℓ_f only at the fine tokens under them, m_i^f (dashed boxes in Figure 2).


Response to W4: MIL baselines should use UNI2 or CONCH v1.5.

We did not evaluate the MIL baselines with UNI2 or CONCH v1.5 because Table 1 compares where magnification information is combined, and that comparison needs both sides to start from the same weights. Patch-JEPA uses only the ImageNet-1K checkpoint that the MIL baselines also use, so the two sides differ only in whether the scales are combined in slide modeling or in patch pretraining. Giving UNI2 or CONCH v1.5 to the baselines alone would add large-scale pathology pretraining to one side, and the result would reflect that pretraining data rather than the stage of integration. MIL on SSL encoders pretrained in each cohort is in Appendix Tables 14 and 15 of the submission (Tables 15 and 16 in the revision), and UNI2-H patch features under the same readout are in Appendix Table 23 of the submission (Table 24 in the revision).


Response to W5 and W6: Mean pooling, why it competes with MIL, and code.

Patch-JEPA is not a MIL method; its goal is to move the magnification integration that MIL performs at the slide level into the patch encoder. Mean pooling with a linear probe is therefore enough to show this contribution, since any gain must come from the patch features. With a strong patch encoder this readout alone already performs well: UNI2-H features with mean pooling and a linear probe exceed every MIL method of Table 1 in all four cohorts (Appendix Table 23 of the submission, Table 24 in the revision). Attention heads on the frozen Patch-JEPA encoder are already in the appendix: with ABMIL and CLAM trained per task, ACC is equal to or higher than with mean pooling in all four cohorts (Appendix Tables 20 and 21 of the submission, Tables 21 and 22 in the revision). Code will be released upon acceptance; Algorithm 1, Appendix C.1 and Table 34 of the revision give the procedure and settings.

This is why I think this paper is less interesting for the computational pathology community if you can't prove the advantages of your pretraining method/framework in the same settings (the capability to scale up and integrate with SOTA MIL methods, etc.) as others. Basically, you have only proved the efficiency of Patch-JEPA in a narrow setting which is not practical for pathology foundation model development. It's a good practice, but not a comprehensive research work. Sorry, I will keep my score.

Comments on the paper

2 comments
Area_Chair_wEJWOpenReview ↗

Reminder - Rebuttal posed, please engage

Dear Reviewers,

I'm writing to update you on the discussion: we have now passed the halfway mark, and the authors' rebuttal has been posted.

I would like to encourage all of you to take part in the discussion at this stage. Please review the authors' responses, verify whether your concerns and questions have been adequately addressed, and provide the authors with any additional feedback you consider necessary.

Your active involvement will be valuable in helping us reach a well-informed final decision. Thank you for your time and contribution.

Best regards,

Your AC

AuthorsOpenReview ↗

General Responses

Summary of the rebuttal

We thank all reviewers for their comments. The table below lists each concern, what we did, and where the change appears in the revised manuscript. Update (Nov 18): Following the follow-ups of 6Lrf and hg8V, we added foundation-model-scale experiments (UNI2-H and CONCH v1.5 as targets, comparison with TITAN and CARE, data/encoder scaling, and encoding cost); the "Comparison with foundation models" row is updated accordingly.

ConcernRaised byWhat we didRevised manuscript
Transductive pretraining and no patch-level evaluation6Lrf (W1, W2, Q1)We re-pretrained every SSL method of Table 2 on UCEC training-patient slides only, with three seeds, and Patch-JEPA has the highest ACC on test patients. We added Camelyon16 patch-level lesion classification with Patch-JEPA, DINOv2, MPCS and Cross-MAE pretrained on the official training slides only, and Patch-JEPA has the highest ACC and F1.Section 3.3, Appendix C.3.3 (Table R2), Appendix C.5 (Table R4)
Pretraining on one cohort, no transfer or data scaling6Lrf (W3, Q2, follow-up), N56K (W3)We transferred the frozen UCEC encoders to Camelyon16, BRACS and PANDA. We re-pretrained MPCS, X-MAG and Patch-JEPA on UCEC, + BRACS and + BRACS + Camelyon16, and Patch-JEPA has the highest BRCA ACC, AUROC and F1 under every pretraining set. On the largest set, Patch-JEPA is also highest with ASMIL (ICLR 2026) and CAR-MIL (ECCV 2026) heads. Scaling pretraining from 565 to 2,406 TCGA slides raises all 24 evaluated settings.Section 3.6 (Table 5), Appendix C.6 (Table R5), follow-up responses to 6Lrf
Comparison with foundation models6Lrf (W4, follow-up), hg8V (W2, W4, follow-up), N56K (Q3)We trained a Patch-JEPA ViT-B (85.8M) on 2,406 TCGA slides with UNI2-H and CONCH v1.5 as targets and evaluated on unseen cohorts with ABMIL and DTFD-MIL. On Camelyon16 it is close to UNI2-H and 0.08–0.11 AUROC above CONCH v1.5; on BRACS and PANDA it is up to 0.035 below, while using 2,406 vs. 100K–350K+ slides and 85.8M vs. 306–681M parameters. Pretrained on UCEC alone, it is above its target FM in all four UCEC settings. Under the TITAN linear-probe protocol it reaches 0.750/0.741 on Camelyon16 against 0.717/0.701 for TITAN/CARE, is close on PANDA and below on BRACS. It encodes a slide in about 1/7 of the time and 1/5 of the memory of UNI2-H. We also report parameters, pretraining data, FLOPs, latency and memory of each encoder.Appendix Tables 15, 16, 24 and R3; follow-up responses to hg8V and 6Lrf (Nov 18, Tables 1–5)
Method details and figureshg8V (W1, W3), XV2i (Q1)We name the target-encoder checkpoint, redrew Figure 2 with the notation of the text, describe the position embeddings, the mask token and where each loss is computed, rename the modules of Figure 1(c), and open the appendix with a guide to the results.Sections 2.1 and 2.2, Figures 1(c) and 2, Appendix D, Appendix Table 7
Readouts and tasks beyond the main tableshg8V (W5, W6), XV2i (W, Q3)We point to the ABMIL and CLAM readouts from the RQ2 setup and to the survival and stage results from the RQ1 results.Sections 3.2 and 3.3, Appendix Tables 12, 21 and 22
Uneven gains and significanceN56K (W1, Q1, Q2)We state that the gap is within the fold SD where Patch-JEPA is not first, ran paired t-tests over ten downstream seeds on RCC with Bonferroni correction, and measured how much of the feature variance the slide label explains at each scale to explain the larger gain on RCC.Section 3.2, Appendix C.2 (Table R1), response to N56K
Limitations and efficiency claimsN56K (W2, W4)We state that alignment needs only pyramid-level metadata and that Patch-JEPA predicts fine-scale latents without reconstructing cellular detail, and describe efficiency as 16 times fewer patches at inference.Abstract, Section 5, Conclusion

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.