- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
STAMP proposes a source-only score for ranking models by their expected out-of-distribution performance. It compares output differences between semantically related pairs and random pairs, using longitudinal same-patient radiographs for medical models and same-class images for ImageNet models. The evaluation covers 44 medical models and 27 natural-image models, with strong reported correlations across several distribution shifts. The paper also presents a correlation-ratio interpretation, an OOD error bound, and a model-ranking guarantee.
Critical strengths
1.The proposed statistic is inexpensive and requires only model outputs and paired source images. This makes it potentially useful when target samples and model internals are unavailable.
2.The evaluation covers diverse architectures and multiple medical and natural-image shifts. Partial correlations, within-CE analysis, pair-sampling sensitivity, and failure cases provide useful checks beyond reporting a single pooled correlation.
Critical weaknesses
1.The main theoretical guarantee is not established. Proposition 1 moves from an upper bound and an approximate error expression to an ordering of actual model errors. Ordering upper bounds does not establish that the underlying errors have the same ordering, and the approximation residual is absent from the final ranking inequality. Lemma 3 does not justify replacing average projected variance-to-margin ratios with the global scatter ratio; its brief covariance argument is insufficient. Moreover, a positive Spearman correlation of 0.73 does not verify the non-decreasing margin condition in A5(ii). These affect the central theoretical contribution rather than peripheral details.
2.The experimental comparisons do not fully isolate the claimed effects. CE models are fully fine-tuned, whereas SSL/VLM models use frozen backbones with linear probes, so their separation cannot be attributed to pretraining objective alone—even with the backbone held approximately fixed. Table 2 also compares correlations over different model populations: temporal STAMP uses 44 models, while the strongest augmentation baseline uses 31. The ImageNet version requires source labels for both pairing and temperature fitting, so “label-free” needs qualification. Finally, excluding same-class random pairs differs from independent sampling from the source distribution, which is required for the stated identity (E[AV]=2S_T).
3.Several reporting contradictions undermine confidence in the conclusions. Table 9 defines its difference as Temporal minus Labeled, but its second column is headed “Synthetic STAMP.” For ResNet-18, the displayed values are 0.7764 and 0.9528, giving −0.1764, while the table reports +0.177 and the text claims temporal scores are higher throughout. These are per-model scores, yet Section 7 describes the differences as improvements in correlation. The paper also reports permutation (p<10^{-10}) from 10,000 permutations, which that procedure cannot resolve without an additional tail-estimation method. Section 9 says ATC and AoTL outperform STAMP on MIMIC, directly contradicting Table 2. Such inconsistencies require an audit of the underlying results.
Questions for authors
1.Can you provide a corrected ranking theorem that explicitly includes bound slack and approximation error? Please also justify Lemma 3 and give a concentration result that accounts for the AV denominator.
2.Can you rerun all baseline correlations on the same model subset and compare CE, SSL, and VLM initialization under matched downstream adaptation? Please distinguish evidence that STAMP predicts performance from evidence about what causes the differences between training regimes.
3.What exactly does Table 9 compare: temporal pairs, label-matched pairs, or synthetic augmentations? Please correct the signs, labels, and associated claims, and explain the source of the reported permutation p-values.
4.How is the alarm threshold in Appendix B.8 fitted? If the 35% calibration subset uses VinDr AUROC to identify failing models, that screening procedure requires target-label-derived information even though the STAMP score itself does not. Regardless of calibration, observing no false positives on a small held-out set does not justify “zero-risk model rejection.”
7 comments
RESPONSE TO REBUTTAL -1
Thank you for the detailed response. The clarifications substantially address my concerns about Table 9, the p-values, the Section 9 inconsistency, the use of source labels, and the alarm-threshold experiment. The same-subset results and the separate within-regime correlations are also useful and strengthen the empirical case for STAMP as a model-ranking heuristic.
My main remaining concern is the theoretical claim. The new theorem appears to guarantee an ordering only when the bound gap exceeds a slack term. Please clarify whether this slack is observable or can be bounded from the available source data. If it depends on unknown target errors or target-specific quantities, the condition does not by itself provide a usable source-only ranking guarantee. Similarly, the reported pairwise margin conditions hold for only 71–80% of model pairs, which supports an empirical tendency rather than a general ranking theorem. The revised paper should make this distinction explicit and avoid presenting the result as a universal guarantee.
For the concentration result, please ensure that the full proof simultaneously controls the numerator and the random AV denominator and states the required lower-bound assumptions on AV. The five-sample stability experiment is reassuring but does not replace this argument.
Finally, the frozen ImageNet-supervised control under the identical probing protocol would be valuable, but it is currently a promised experiment. Overall, the rebuttal strengthens the empirical contribution considerably and resolves the reporting issues. My remaining reservation concerns the scope and practical meaning of the revised theoretical guarantee.
Thank you for the follow-up and for acknowledging the empirical and reporting fixes. We agree with each remaining point, and we answer them directly.
1. Is the slack observable from source data? No. contains the target error, so it cannot be computed or bounded from source data without further assumptions. Theorem C is therefore not a source-only ranking guarantee, and the revision will not present it as one. What the theory does establish from source quantities is narrower:
- an upper bound on target error that is strictly decreasing in (for fixed and ), and
- the exact condition under which an ordering of bounds becomes an ordering of errors (bound gap ≥ slack).
On the source domain the slack is observable, because can be computed. We will report the measured source slack for every model, so readers can see how tight the bound is where it can be checked.
2. Scope of the claim. We agree: conditions that hold for 71–80% of model pairs support an empirical tendency, not a general theorem. The revision makes this distinction explicit:
- The section "Ranking guarantee" is renamed "Bound ordering", and the words "guarantee" and "theoretically grounded ranking" are removed from the abstract, contributions and Table 10.
- Proposition 1 and the "approximate rank consistency" proposition are replaced by Theorem C, followed by a remark stating that the slack is unobservable and that the pairwise conditions hold for most but not all model pairs.
- The contribution is restated as: a correlation-ratio statistic with a bound that explains the direction of its association with OOD error, and an empirical ranking validated on 71 models.
3. Concentration with simultaneous control of numerator and denominator. The full statement is as follows.
Assumptions. (a) The semantic-pair statistics are i.i.d. in with mean and variance ; the random-pair statistics are i.i.d. in with mean and variance . The two samples need not be independent of each other. (b) Non-degeneracy: . (c) Sample size: with .
Lemma D. Let and . With probability at least ,
Proof. By Bernstein's inequality, each of and holds with probability at least , so by the union bound both hold simultaneously with probability at least . Assumption (c) gives , so on this event the random denominator satisfies . Using the numerator is at most and the denominator is at least , which gives the bound.
The lower-bound requirement on AV is therefore explicit: together with the sample-size condition (c). The stabiliser in the implemented denominator changes the ratio by a relative amount below at the observed . We agree that the five-sample experiment does not replace this argument. In the revision we report, for each model, the empirical and and state whether condition (c) is met at ; where it is not, we say so and rely on the bootstrap interval only as an empirical measure.
4. Frozen ImageNet-supervised control. We agree it is valuable and that it is still a promised experiment. It uses the same backbones the fine-tuned ViT-B/16 and DeiT-B/16 started from, kept frozen, with the identical probe recipe used for the SSL/VLM probes. We will post the result here if it finishes before the discussion closes, and it will be in the revision in either case. Our claim does not depend on it: the paper now states the CE/SSL difference as a training-regime observation, and the within-regime correlations (.820/.785/.728 within CE models, .801/.807/.705 regime-stratified) carry the predictive claim.
In summary, the revised paper presents STAMP as an empirically validated model-ranking statistic with an explanatory bound, and it no longer claims a universal ranking guarantee.
Thank you for the rigorous review. The core finding stands: ρ = 0.855 / 0.851 / 0.844 on three external hospitals (n = 44).
W1/Q1: Ranking theorem, Lemma 3, concentration. We have strengthened the theory into exact inequalities with no "≈" step. Let e(f) be the nearest-source-centroid error in output space and γy=miny′=y∥μy−μy′∥.
W2/Q2: Matched comparisons; CE vs. SSL/VLM.
Same model subset (ρ with VinDr / CheXpert / MIMIC AUROC, identical models):
The rotation row is the same 1−SV/AV statistic with augmentation pairs, i.e. an instance of STAMP. STAMP is statistically equivalent to labeled source AUROC on both sets (paired bootstrap CIs include 0). No other label-free baseline is reliable on both: TTA-Multi works within CE models but collapses once probes are included, and ATC shows the reverse. AoTL, entropy, confidence, FD, FAS, MC-Dropout and embedding instability are all below .38 on both sets.
Prediction vs. cause. STAMP's predictive validity does not rely on the CE/SSL separation. Within CE models ρ = .820/.785/.728; within frozen probes ρ = .733/.867/.617; regime-stratified ρ = .801/.807/.705. The separation itself reflects the training regime (objective plus adaptation), and we will word it that way. A matched control (frozen ImageNet-supervised ViT-B/DeiT-B with the identical probe recipe) will be added.
"Label-free." Our claim is target-data-free and target-label-free, as the abstract states. Source labels are used for pairing and temperature fitting, and we will make this wording uniform.
Same-class exclusion. Let q=∑yπy2. For balanced classes 2ST=q⋅2SW+(1−q)E[AVdiff], so STAMP estimates 1−R with R=1−qrr(1−q), r=SW/ST. Since dR/dr>0, this is a strictly monotone transform of η2: every model ranking is exactly preserved, with ∣R−r∣≤2.5×10−4 on ImageNet. For the medical pairs q=1.6×10−4, and recomputing STAMP with fully unrestricted random pairs gives a rank agreement of 0.988 with ρ = .840/.858/.836.
W3/Q3: Table 9, p-values, Section 9.
Q4: Alarm threshold. You are right: the threshold was fit with VinDr AUROC on calibration models. We state this and remove "zero-risk." We add a rule with no target information: flag the bottom 30% of candidates by STAMP. It finds the bottom-30% models with precision 0.92 (Wilson 95% CI [0.67, 0.99]) and F1 0.92 on each of VinDr, CheXpert and MIMIC (always-alarm F1: 0.46), the same as ranking by labeled NIH AUROC.