- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper evaluates frozen pathology encoders for spatial gene-expression prediction. It shows that HEST’s fixed PCA-Ridge readout is poorly calibrated, with minimal effective regularization. The proposed CALIBRE pipeline tunes PCA dimension and Ridge penalty via cross-validation, optionally adding encoder ensembling and spatial smoothing. Across HEST and two external cohorts, calibrated readouts consistently improve frozen encoders and often match or outperform STFlow, highlighting the importance of training-selected readouts in fair model comparisons.
Critical strengths
-
The paper identifies a concrete issue in the HEST protocol: the prescribed Ridge penalty appears nearly inactive. The effective-degrees-of-freedom analysis provides a clear explanation for why readout calibration may improve performance.
-
The single-encoder experiments are carefully controlled: encoders, outer splits, targets, and metrics are fixed, while readout hyperparameters are selected within training folds. Results are also tested on external cohorts.
-
The paper usefully distinguishes standardized fixed-probe evaluation from per-encoder, training-tuned readouts, a distinction that is important for interpreting frozen-encoder benchmarks.
Critical weaknesses
-
The title and framing are somewhat misleading. “Fold-Calibrated Readouts for Frozen Pathology Encoders” suggests a broadly applicable technical design for adapting pathology encoders across diverse downstream tasks. In practice, the paper is primarily a correction to a specific evaluation choice in HEST: PCA-plus-Ridge prediction of spatial gene expression from frozen embeddings. The paper would benefit from stating this narrower contribution more explicitly.
-
The central evidence is narrow in scope. The paper convincingly shows that fixed readout parameters can affect results under the HEST-style frozen-encoder + PCA + Ridge protocol for spatial gene-expression regression. However, it remains unclear whether this issue generalizes to other pathology tasks, benchmarks, or probe families, such as classification, survival prediction, nonlinear heads, or different pooling strategies.
-
The role of the seven-encoder ensemble is unclear given the paper’s evaluation framing. Ensembling is reasonable for constructing a strong predictive system, but it is not an evaluation of an individual frozen encoder. Its inclusion blurs the distinction between auditing the HEST readout protocol and proposing a multi-model prediction pipeline, and it complicates comparisons against single-model trained baselines such as STFlow.
Questions for authors
See Weaknesses.
3 comments
Thanks for your response. The rebuttal clarifies that the contribution is primarily an audit and recalibration of a specific HEST-style PCA–Ridge protocol for spatial gene-expression prediction, rather than a general method for frozen pathology encoders.
While this analysis is useful, it reinforces my main concern: the scope is too narrow for ICLR, especially given the lack of evidence beyond this benchmark setting. I will therefore maintain my original score.
Thank you for following up and noting that the revised framing and component separation are clear. While our main claim is properly scoped to linear spatial-expression probing (in the 15 November revised PDF via the PDF icon beside the title, changes in blue), the paper's empirical evidence and evaluation implications extend well beyond a single HEST configuration in three concrete ways:
1. A general probe-scaling diagnosis, plus distinct tests of per-fold calibration and fixed-setting transfer.
- Diagnosing effective-regularization collapse (Appendix L.1): HEST's prescribed ridge penalty (
100/(dg) = 0.0078atd=256, g=50) mirrors a sample-averaged classification-probe constant applied to squared error summed over1,293to67,651training spots. That spot-count scaling alone accounts for 3.1 to 4.8 of the 6.1 orders-of-magnitude penalty gap and leaves 256.0 of 256 effective degrees of freedom (virtually zero shrinkage), exposing a general linear-probing pitfall whenever training-set sizes span orders of magnitude. - Broad backbone impact on HEST (Section 4, Appendix L): With embeddings, splits, targets, and metrics held fixed across the benchmark slides, per-fold inner-CV calibration improves all 7/7 pre-registered pathology backbones, and a single calibrated cell applied unchanged improves all 15/15 tested backbones and 132/135 backbone-task cells (+0.0368 mean gain, positive on all nine cancer tasks).
2. Multi-cohort and multi-benchmark evidence beyond HEST.
- External patient-held-out cohorts across institutions and platforms: On ST-Net, unsmoothed per-fold calibration lifts the primary encoder (
H-Optimus-1) from 0.0811 to 0.1033 across 19 held-out breast cancer patients with training-fold gene selection (, Table 2). Crucially, in Appendix Q's cohort-wide-panel check, applying the fixed HEST constant recovers only 19% of per-fold calibration's gain (+0.0045vs.+0.0235, modal optimal penalties ), showing directly why training-fold calibration is needed across cohorts rather than one universal constant. On an external 22-patient kidney cohort registered before download from a different institution, organ, and platform, per-fold calibration rises from 0.1880 to 0.1991, improving all 22/22 held-out patients (Appendix T; on 8-patientHER2STwith matched P-ES,0.2408to0.2697has , Appendix P). - Controlled probe sensitivity on
HESCAPEandVCBench: On reconstructed HESCAPE five-fold sample-grouped splits, cross-validating a HEST-style head improves both tested encoders across all 5/5 folds (+0.0174onUNI, ;+0.0151onProv-GigaPath, , Appendix C, Table C.1). Outside histology, on VCBench's released random 80/20 split for predicting protein-derived targets from RNA embeddings, probe calibration lifts frozenscGPTfrom 0.0638 to 0.1306 (+0.0668) in a four-encoder comparison while preserving the top ranking (Appendix J).
3. Changing how frozen foundation representations are judged against task-trained generative models.
Published HEST-style evaluations (compounded by test-fold epoch selection in released STFlow baselines) created the impression that frozen pathology foundation models substantially lag end-to-end generative architectures. Once training-fold selection, spatial smoothing, and all seven encoders are matched in our kidney follow-up (Appendix Y, p. 52), our calibrated frozen-encoder system and a seven-encoder STFlow ensemble differ by only +0.0019 (95% CI [−0.0098, 0.0142], ), with most of the multi-encoder margin over single-encoder STFlow driven by encoder averaging (while shared-smoother single-encoder gaps remain unresolved after Holm correction). Showing across 15 backbones, external held-out patient cohorts, and two additional benchmarks that readout miscalibration distorts foundation-model comparisons addresses a core representation-evaluation problem for ICLR, and we hope this supports reconsidering the score.
Author Response to Reviewer eFuV
Thank you for the review. The paper concerns PCA-ridge readouts for spatial gene-expression regression from frozen encoders, and the revised abstract, Section 6 (Limitations), and Section 7 (Conclusion) state this narrower scope explicitly. Those sections also report that a seven-encoder STFlow ensemble leaves the kidney margin unresolved. Our contribution is diagnosing an almost inactive benchmark penalty and measuring how correcting that readout changes frozen-encoder and baseline comparisons.
The evidence for that correction. HEST's prescribed ridge penalty (100/(dg) = 0.0078 at d=256, g=50) mirrors a sample-averaged classification-probe constant applied to squared error summed over 1,293 to 67,651 training spots (Appendix L.1), retaining 256.0 of 256 effective degrees of freedom at the reported precision. Correcting that readout raises the pre-registered H-Optimus-1 nine-task mean from its published 0.4229 to 0.4465 (Table A.2) with embeddings, outer splits, targets, and metrics fixed, without averaging or smoothing. The corresponding unsmoothed single-encoder changes are 0.0811 to 0.1033 in the fold-internal ST-Net analysis (Table 2) and 0.1880 to 0.1991 on kidney (Appendix T). On reconstructed HESCAPE folds grouped by released sample index, cross-validating the inherited HEST head gains +0.0174 on UNI and +0.0151 on GigaPath (Appendix C). Probe selection on VCBench's released random split raises scGPT by +0.0668 without changing the leader or model ranking (Appendix J). Neither check establishes patient- or site-held-out transfer. These results do not extend to classification, survival prediction, nonlinear heads, or other pooling strategies, and we do not claim a general adaptation method.
The seven-encoder recipe. This recipe evaluates a combined prediction pipeline rather than an individual representation, and the revision presents it explicitly as a system baseline. Table 1 and Appendix O separate encoder count, calibration, and access to smoothing. That separation changes the conclusion. With encoder and smoother matched, the ST-Net and kidney comparisons against released-rate STFlow remain unresolved after Holm correction (Table 2 and Appendix T), separating on ST-Net only when inner learning-rate tuning weakens STFlow (Appendix S). On HER2ST, even the full recipe differs from densely training-selected STFlow by only +0.0022 over eight patients, with a 95% paired-patient t interval of [-0.0370, 0.0415] (Appendix P). The full-system results therefore do not show that calibration alone generally outperforms trained architectures. In new Appendix Y, a post-hoc kidney experiment removes the encoder mismatch by giving STFlow all seven encoders, with one model per encoder (six new encoder arms, with stored H-Optimus-1 runs reused). With encoders and smoothing matched, the full recipe leads by only +0.0019 (95% paired bootstrap interval [-0.0098, 0.0142], 12/22 patients, p=0.92), whereas the seven-model STFlow ensemble exceeds smoothed single-encoder STFlow by +0.0180 on H-Optimus-1, one of its members, and by +0.0283 on UNI, which it does not contain. On this cohort, most of the margin over single-encoder STFlow comes from averaging encoders rather than from the readout.
Technical framing. Inner cross-validation, ridge regression, averaging, and spatial smoothing are established procedures. On HEST, a strong fixed setting recovers most of the calibration gain, which further limits any claim that elaborate per-task tuning is essential. We retain the submitted title during discussion. The final version will add "for Spatial Gene-Expression Prediction" to the title to make this scope explicit.