acceptodds
Under review as a conference paper at ICLR 2027

STAMP: Predicting Out-of-Distribution Generalization without Target Data

Abstract

Predicting whether a trained model will generalize under distribution shift remains difficult, especially when target-domain data are unavailable. We introduce STAMP (Semantic Temporal Augmented Model Prediction), a source-only, target-label-free criterion that estimates out-of-distribution (OOD) performance from paired source-domain images. STAMP computes the output-space correlation ratio by contrasting semantically stable pairs with random pairs: higher indicates that model outputs vary with semantic identity rather than nuisance variation. On 44 chest X-ray models spanning CNNs, ViTs, MetaFormers, foundation models, and SSL/VLM probes, temporal STAMP attains Spearman correlations of – with macro AUROC on VinDr-CXR, CheXpert, and MIMIC-CXR; a class-matched variant improves single-class RSNA from to . STAMP attains the best average source-only medical ranking and outperforms the target-domain ATC and AoTL estimators without any target data. On 27 ImageNet models, temperature-scaled STAMPTS attains on ObjectNet and on four additional distribution shifts, with partial correlations of – after controlling for ImageNet accuracy. Requiring approximately 12s per model on one GPU, STAMP is a practical pre-deployment model-selection and auditing tool.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Out-of-distribution generalization, generalization prediction, Distribution shift, Model selection, Label-free model evaluation, Chest X-ray

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zbPNOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

STAMP proposes a source-only score for ranking models by their expected out-of-distribution performance. It compares output differences between semantically related pairs and random pairs, using longitudinal same-patient radiographs for medical models and same-class images for ImageNet models. The evaluation covers 44 medical models and 27 natural-image models, with strong reported correlations across several distribution shifts. The paper also presents a correlation-ratio interpretation, an OOD error bound, and a model-ranking guarantee.

Critical strengths

1.The proposed statistic is inexpensive and requires only model outputs and paired source images. This makes it potentially useful when target samples and model internals are unavailable.

2.The evaluation covers diverse architectures and multiple medical and natural-image shifts. Partial correlations, within-CE analysis, pair-sampling sensitivity, and failure cases provide useful checks beyond reporting a single pooled correlation.

Critical weaknesses

1.The main theoretical guarantee is not established. Proposition 1 moves from an upper bound and an approximate error expression to an ordering of actual model errors. Ordering upper bounds does not establish that the underlying errors have the same ordering, and the approximation residual is absent from the final ranking inequality. Lemma 3 does not justify replacing average projected variance-to-margin ratios with the global scatter ratio; its brief covariance argument is insufficient. Moreover, a positive Spearman correlation of 0.73 does not verify the non-decreasing margin condition in A5(ii). These affect the central theoretical contribution rather than peripheral details.

2.The experimental comparisons do not fully isolate the claimed effects. CE models are fully fine-tuned, whereas SSL/VLM models use frozen backbones with linear probes, so their separation cannot be attributed to pretraining objective alone—even with the backbone held approximately fixed. Table 2 also compares correlations over different model populations: temporal STAMP uses 44 models, while the strongest augmentation baseline uses 31. The ImageNet version requires source labels for both pairing and temperature fitting, so “label-free” needs qualification. Finally, excluding same-class random pairs differs from independent sampling from the source distribution, which is required for the stated identity (E[AV]=2S_T).

3.Several reporting contradictions undermine confidence in the conclusions. Table 9 defines its difference as Temporal minus Labeled, but its second column is headed “Synthetic STAMP.” For ResNet-18, the displayed values are 0.7764 and 0.9528, giving −0.1764, while the table reports +0.177 and the text claims temporal scores are higher throughout. These are per-model scores, yet Section 7 describes the differences as improvements in correlation. The paper also reports permutation (p<10^{-10}) from 10,000 permutations, which that procedure cannot resolve without an additional tail-estimation method. Section 9 says ATC and AoTL outperform STAMP on MIMIC, directly contradicting Table 2. Such inconsistencies require an audit of the underlying results.

Questions for authors

1.Can you provide a corrected ranking theorem that explicitly includes bound slack and approximation error? Please also justify Lemma 3 and give a concentration result that accounts for the AV denominator.

2.Can you rerun all baseline correlations on the same model subset and compare CE, SSL, and VLM initialization under matched downstream adaptation? Please distinguish evidence that STAMP predicts performance from evidence about what causes the differences between training regimes.

3.What exactly does Table 9 compare: temporal pairs, label-matched pairs, or synthetic augmentations? Please correct the signs, labels, and associated claims, and explain the source of the reported permutation p-values.

4.How is the alarm threshold in Appendix B.8 fitted? If the 35% calibration subset uses VinDr AUROC to identify failing models, that screening procedure requires target-label-derived information even though the STAMP score itself does not. Regardless of calibration, observing no false positives on a small held-out set does not justify “zero-risk model rejection.”

7 comments

Thank you for the rigorous review. The core finding stands: ρ = 0.855 / 0.851 / 0.844 on three external hospitals (n = 44).

W1/Q1: Ranking theorem, Lemma 3, concentration. We have strengthened the theory into exact inequalities with no "≈" step. Let e(f)e(f) be the nearest-source-centroid error in output space and γy=min⁡y′≠y∥μy−μy′∥\gamma_y=\min_{y'\neq y}\|\mu_y-\mu_{y'}\|.

  • Lemma A (replaces Lemma 3). A misclassified xx has ∥f(x)−μy∥≥γy/2\|f(x)-\mu_y\|\ge\gamma_y/2, so Markov's inequality gives Ps(err∣y)≤4 trΣy/γy2P_s(\text{err}\mid y)\le4\,\mathrm{tr}\Sigma_y/\gamma_y^2. With the measurable margin regularity κf=min⁡yγy2/SB\kappa_f=\min_y\gamma_y^2/S_B: es(f)≤4κf1−ηf2ηf2e_s(f)\le\frac{4}{\kappa_f}\frac{1-\eta_f^2}{\eta_f^2}. No covariance argument is needed.
  • Lemma B (target). With a coupling E∥Xs−Xt∥2≤d2\mathbb E\|X_s-X_t\|^2\le d^2 and LL-Lipschitz ff, Minkowski and Jensen give et(f)≤U(f):=4κf((1−ηf2)/ηf2+Ld/SB)2e_t(f)\le U(f):=\frac{4}{\kappa_f}\big(\sqrt{(1-\eta_f^2)/\eta_f^2}+Ld/\sqrt{S_B}\big)^2, strictly decreasing in η2\eta^2.
  • Theorem C (ranking with explicit slack). For any f1,f2f_1,f_2: et(f1)≤et(f2)−[U(f2)−U(f1)]+s(f2)e_t(f_1)\le e_t(f_2)-[U(f_2)-U(f_1)]+s(f_2), with slack s(f2)=U(f2)−et(f2)≥0s(f_2)=U(f_2)-e_t(f_2)\ge0. Error ordering holds whenever the bound gap exceeds the slack; this is the exact condition you asked for.
  • A5(ii) is replaced by measured pairwise conditions. Across 780 model pairs, the higher-STAMP model has the larger SBS_B in 80% and the larger margin regularity (10th-percentile κ\kappa) in 71%. They hold for most pairs but not all, so the ranking is stated as empirical.
  • Concentration with the AV denominator. By Bernstein's inequality, for εa<μa\varepsilon_a<\mu_a: ∣SV/AV−μs/μa∣≤(εs+μsμaεa)/(μa−εa)|SV/AV-\mu_s/\mu_a|\le(\varepsilon_s+\tfrac{\mu_s}{\mu_a}\varepsilon_a)/(\mu_a-\varepsilon_a). Empirically, over five independent pair samples the per-model std is 0.005 and ρ stays within 0.86–0.89 on all three hospitals.

W2/Q2: Matched comparisons; CE vs. SSL/VLM.

  • Same model subset (ρ with VinDr / CheXpert / MIMIC AUROC, identical models):

    Method40 models (CE + probes)31 CE models
    STAMP, longitudinal pairs.891 / .843 / .817.820 / .785 / .728
    STAMP, rotation pairs.927 / .866 / .884.860 / .758 / .790
    NIH AUROC (needs labels).918 / .872 / .879.866 / .736 / .773
    ECE.571 / .418 / .471.324 / .107 / .162
    ATC.521 / .538 / .510.083 / .099 / .060
    TTA-Multi.297 / .344 / .294.752 / .756 / .740

    The rotation row is the same 1−SV/AV1-SV/AV statistic with augmentation pairs, i.e. an instance of STAMP. STAMP is statistically equivalent to labeled source AUROC on both sets (paired bootstrap CIs include 0). No other label-free baseline is reliable on both: TTA-Multi works within CE models but collapses once probes are included, and ATC shows the reverse. AoTL, entropy, confidence, FD, FAS, MC-Dropout and embedding instability are all below .38 on both sets.

  • Prediction vs. cause. STAMP's predictive validity does not rely on the CE/SSL separation. Within CE models ρ = .820/.785/.728; within frozen probes ρ = .733/.867/.617; regime-stratified ρ = .801/.807/.705. The separation itself reflects the training regime (objective plus adaptation), and we will word it that way. A matched control (frozen ImageNet-supervised ViT-B/DeiT-B with the identical probe recipe) will be added.

  • "Label-free." Our claim is target-data-free and target-label-free, as the abstract states. Source labels are used for pairing and temperature fitting, and we will make this wording uniform.

  • Same-class exclusion. Let q=∑yπy2q=\sum_y\pi_y^2. For balanced classes 2ST=q⋅2SW+(1−q) E[AVdiff]2S_T=q\cdot2S_W+(1-q)\,\mathbb E[AV_{\text{diff}}], so STAMP estimates 1−R1-R with R=r(1−q)1−qrR=\frac{r(1-q)}{1-qr}, r=SW/STr=S_W/S_T. Since dR/dr>0dR/dr>0, this is a strictly monotone transform of η2\eta^2: every model ranking is exactly preserved, with ∣R−r∣≤2.5×10−4|R-r|\le2.5\times10^{-4} on ImageNet. For the medical pairs q=1.6×10−4q=1.6\times10^{-4}, and recomputing STAMP with fully unrestricted random pairs gives a rank agreement of 0.988 with ρ = .840/.858/.836.

W3/Q3: Table 9, p-values, Section 9.

  • Table 9: column 2 is augmentation-pair STAMP and Δ = column 2 − column 1. The header and sign are fixed and Δ is described as a per-model score difference. No headline result depends on it.
  • The p < 10⁻¹⁰ values are asymptotic Spearman p-values (e.g. 1.4 × 10⁻¹³ on VinDr). The permutation test independently gives p ≤ 10⁻⁴, its resolution limit. Both are now labeled.
  • The Section 9 sentence is a typo. Table 2 is correct and favors STAMP: 0.844 vs. ATC 0.402 and AoTL 0.357 on MIMIC.

Q4: Alarm threshold. You are right: the threshold was fit with VinDr AUROC on calibration models. We state this and remove "zero-risk." We add a rule with no target information: flag the bottom 30% of candidates by STAMP. It finds the bottom-30% models with precision 0.92 (Wilson 95% CI [0.67, 0.99]) and F1 0.92 on each of VinDr, CheXpert and MIMIC (always-alarm F1: 0.46), the same as ranking by labeled NIH AUROC.

Reviewer_zbPNOpenReview ↗

RESPONSE TO REBUTTAL -1

Thank you for the detailed response. The clarifications substantially address my concerns about Table 9, the p-values, the Section 9 inconsistency, the use of source labels, and the alarm-threshold experiment. The same-subset results and the separate within-regime correlations are also useful and strengthen the empirical case for STAMP as a model-ranking heuristic.

My main remaining concern is the theoretical claim. The new theorem appears to guarantee an ordering only when the bound gap exceeds a slack term. Please clarify whether this slack is observable or can be bounded from the available source data. If it depends on unknown target errors or target-specific quantities, the condition does not by itself provide a usable source-only ranking guarantee. Similarly, the reported pairwise margin conditions hold for only 71–80% of model pairs, which supports an empirical tendency rather than a general ranking theorem. The revised paper should make this distinction explicit and avoid presenting the result as a universal guarantee.

For the concentration result, please ensure that the full proof simultaneously controls the numerator and the random AV denominator and states the required lower-bound assumptions on AV. The five-sample stability experiment is reassuring but does not replace this argument.

Finally, the frozen ImageNet-supervised control under the identical probing protocol would be valuable, but it is currently a promised experiment. Overall, the rebuttal strengthens the empirical contribution considerably and resolves the reporting issues. My remaining reservation concerns the scope and practical meaning of the revised theoretical guarantee.

Thank you for the follow-up and for acknowledging the empirical and reporting fixes. We agree with each remaining point, and we answer them directly.

1. Is the slack observable from source data? No. s(f2)=U(f2)−et(f2)s(f_2)=U(f_2)-e_t(f_2) contains the target error, so it cannot be computed or bounded from source data without further assumptions. Theorem C is therefore not a source-only ranking guarantee, and the revision will not present it as one. What the theory does establish from source quantities is narrower:

  • an upper bound U(f)U(f) on target error that is strictly decreasing in η2\eta^2 (for fixed κf\kappa_f and Lfd/SB,fL_f d/\sqrt{S_{B,f}}), and
  • the exact condition under which an ordering of bounds becomes an ordering of errors (bound gap ≥ slack).

On the source domain the slack is observable, because es(f)e_s(f) can be computed. We will report the measured source slack for every model, so readers can see how tight the bound is where it can be checked.

2. Scope of the claim. We agree: conditions that hold for 71–80% of model pairs support an empirical tendency, not a general theorem. The revision makes this distinction explicit:

  • The section "Ranking guarantee" is renamed "Bound ordering", and the words "guarantee" and "theoretically grounded ranking" are removed from the abstract, contributions and Table 10.
  • Proposition 1 and the "approximate rank consistency" proposition are replaced by Theorem C, followed by a remark stating that the slack is unobservable and that the pairwise conditions hold for most but not all model pairs.
  • The contribution is restated as: a correlation-ratio statistic with a bound that explains the direction of its association with OOD error, and an empirical ranking validated on 71 models.

3. Concentration with simultaneous control of numerator and denominator. The full statement is as follows.

Assumptions. (a) The nn semantic-pair statistics are i.i.d. in [0,D][0,D] with mean μs\mu_s and variance vsv_s; the mm random-pair statistics are i.i.d. in [0,D][0,D] with mean μa\mu_a and variance vav_a. The two samples need not be independent of each other. (b) Non-degeneracy: μa>0\mu_a>0. (c) Sample size: m≥max⁡{32 vaΛ/μa2,  8DΛ/(3μa)}m\ge\max\{32\,v_a\Lambda/\mu_a^2,\;8D\Lambda/(3\mu_a)\} with Λ=log⁡(4/δ)\Lambda=\log(4/\delta).

Lemma D. Let εs=2vsΛ/n+2DΛ/(3n)\varepsilon_s=\sqrt{2v_s\Lambda/n}+2D\Lambda/(3n) and εa=2vaΛ/m+2DΛ/(3m)\varepsilon_a=\sqrt{2v_a\Lambda/m}+2D\Lambda/(3m). With probability at least 1−δ1-\delta, ∣SVAV−μsμa∣≤2μa(εs+μsμaεa).\Big|\frac{SV}{AV}-\frac{\mu_s}{\mu_a}\Big|\le\frac{2}{\mu_a}\Big(\varepsilon_s+\frac{\mu_s}{\mu_a}\varepsilon_a\Big).

Proof. By Bernstein's inequality, each of {∣SV−μs∣≤εs}\{|SV-\mu_s|\le\varepsilon_s\} and {∣AV−μa∣≤εa}\{|AV-\mu_a|\le\varepsilon_a\} holds with probability at least 1−δ/21-\delta/2, so by the union bound both hold simultaneously with probability at least 1−δ1-\delta. Assumption (c) gives εa≤μa/2\varepsilon_a\le\mu_a/2, so on this event the random denominator satisfies AV≥μa−εa≥μa/2>0AV\ge\mu_a-\varepsilon_a\ge\mu_a/2>0. Using SVAV−μsμa=(SV−μs) μa−μs (AV−μa)AV μa,\frac{SV}{AV}-\frac{\mu_s}{\mu_a}=\frac{(SV-\mu_s)\,\mu_a-\mu_s\,(AV-\mu_a)}{AV\,\mu_a}, the numerator is at most μaεs+μsεa\mu_a\varepsilon_s+\mu_s\varepsilon_a and the denominator is at least μa2/2\mu_a^2/2, which gives the bound. □\square

The lower-bound requirement on AV is therefore explicit: μa>0\mu_a>0 together with the sample-size condition (c). The stabiliser 10−810^{-8} in the implemented denominator changes the ratio by a relative amount below 10−610^{-6} at the observed AV≈0.02AV\approx0.02. We agree that the five-sample experiment does not replace this argument. In the revision we report, for each model, the empirical vav_a and μa\mu_a and state whether condition (c) is met at N=2,000N=2{,}000; where it is not, we say so and rely on the bootstrap interval only as an empirical measure.

4. Frozen ImageNet-supervised control. We agree it is valuable and that it is still a promised experiment. It uses the same backbones the fine-tuned ViT-B/16 and DeiT-B/16 started from, kept frozen, with the identical probe recipe used for the SSL/VLM probes. We will post the result here if it finishes before the discussion closes, and it will be in the revision in either case. Our claim does not depend on it: the paper now states the CE/SSL difference as a training-regime observation, and the within-regime correlations (.820/.785/.728 within CE models, .801/.807/.705 regime-stratified) carry the predictive claim.

In summary, the revised paper presents STAMP as an empirically validated model-ranking statistic with an explanatory bound, and it no longer claims a universal ranking guarantee.

Reviewer_RvojOpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a new scoring method for ranking trained classifiers by their expected out-of-distribution performance. The STAMP score compares prediction differences within semantic pairs against those within random pairs. The paper reports strong correlations with OOD performance across multiple medical and natural-image benchmarks as well as multiple pre-trained supervised/self-supervised/language-supervised models.

Critical strengths
  1. The evaluation covers a substantial collection of architectures and distribution shifts.
  2. The proposed method is source-only, which does not require labels from target domain.
  3. Algorithm 1 provides a clear illustration of the computation process of STAMP-TS.
Critical weaknesses
  1. The semantic assumption underlying temporal pairing needs stronger justification. Section 3.4 (L146–151) constructs pairs from consecutive radiographs of the same patient regardless of changes in recorded pathology labels. However, genuine disease progression or recovery should sometimes lead a clinically appropriate model to change its predictions. The score may therefore reward insensitivity to meaningful pathology changes, alongside robustness to acquisition variation. Although the paper acknowledges label noise and provides related analyses, these do not sufficiently distinguish the two explanations.

  2. The discussion contradicts the reported baseline results. Section 9 (L461–463) states that ATC and AoTL “gain a small edge on MIMIC-CXR.” However, Table 2 (L359–361) reports MIMIC-CXR correlations of 0.844 for STAMP, 0.402 for ATC, and 0.357 for AoTL, indicating the opposite ordering. Please clarify the discussion.

Questions for authors

As this submission falls outside my primary area of expertise, and the writing is not straightforward enough to understand. I do not have further questions / suggestions for the authors and my initial score should be interpreted with caution and given limited weight.

1 comment

Thank you for the review.

W1: Temporal pairing may reward insensitivity to pathology change. To clarify §3.4: all main results use consecutive same-patient radiographs whose recorded pathology labels are identical (9,927-pair pool). Label-agnostic pairs appear only in an ablation. Pairs with a recorded change in disease state are therefore not in the semantic set, and a model that correctly changes its prediction when pathology changes is not penalized. Two results also show that STAMP is not simply rewarding insensitivity. Stability alone (SV) is not predictive (ρ = −0.19); only the ratio against between-patient variation is (ρ = 0.89 on the same 40 models). And class-matched STAMP raises RSNA from 0.311 to 0.663, which ties the score to pathology classes. We also ran new stratified analyses that separate the two explanations directly (ρ with VinDr / CheXpert / MIMIC AUROC, n = 44):

Semantic pairsVinDrCheXpertMIMIC
Same patient, same labels (main).855.851.844
Same patient, labels changed only.036.229.178
Different patients, same labels, same view.769.687.673

When the pairs cross a recorded pathology change, the predictive signal vanishes. Insensitivity to genuine pathology change is therefore not what STAMP rewards or what predicts OOD performance. With pathology held fixed, the signal survives even across different patients.

W2: Discussion vs. Table 2. The Discussion sentence is a typo. Table 2 is correct and favors STAMP: on MIMIC-CXR, STAMP reaches 0.844 against 0.402 (ATC) and 0.357 (AoTL), without target data. The sentence is fixed.

Presentation. §3 will state exactly which pairs and labels each variant uses. The theory is condensed into one section with consistent notation and exact inequalities, and all baselines are reported in a single table on identical model sets.

Reviewer_trQmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes STAMP, a source-only proxy for ranking classifiers by their expected performance under distribution shift. Given semantically related and random source-image pairs, it measures the ratio of within-pair and random-pair discrepancies in the models’ output probabilities. The authors interpret the statistic using the correlation ratio and analyze its relationship to OOD error under additional assumptions. The medical instantiation uses longitudinal chest radiographs from the same patient; the natural-image instantiation uses same-class ImageNet pairs and temperature calibration. The evaluation covers 44 chest X-ray models and 27 ImageNet models across nine benchmarks. STAMP reports Spearman correlations of 0.844–0.855 on three major medical OOD datasets, while STAMP-TS reports 0.905–0.984 on five natural-image shifts. The paper also reports partial correlations, ablations, model-family comparisons, and failure analyses.

Critical strengths

1.STAMP addresses the practical problem of model selection without target-domain data. It is simple, inexpensive, and requires only black-box outputs, provided suitable source-domain pairs are available.

2.Results are encouraging across medical and natural-image datasets. The within-CE correlation (ρ = 0.820) and partial correlation controlling for source AUROC and model size (0.536) suggest predictive value beyond these factors.

Critical weaknesses

1.Longitudinal pairs may share patient-specific anatomy or acquisition cues even when pathology changes, so it remains unclear whether STAMP measures task-relevant stability. Its advantage over simpler alternatives is also uncertain: correlations are similar, and the model sets differ. Matched-model comparisons, selection regret, and a negative control would help clarify its value.

2.The theory does not fully support the ranking claim. Lemma 1 assumes independent class-conditional pairs, which differs from the longitudinal setting, while Proposition 1 relies on ordering error bounds without sufficiently controlling approximation errors. Table 9 also has inconsistent labels and difference signs, with a recalculated mean of 0.1523 rather than 0.161. Lemma 4 gives ε ≈ 0.265 rather than 0.067 under the stated formula and needs explicit denominator control. These issues need correction.

Questions for authors
  1. Does longitudinal pairing capture pathology-related stability or persistent patient and acquisition cues? Pathology-stratified pairs or a negative control could clarify this.

  2. What conditions allow Proposition 1 to establish actual OOD error ordering from bounds and approximations?

  3. Given RSNA’s origin in the NIH collection, could you report image- and patient-level overlap with the source training and pairing pools?

1 comment

Thank you for the constructive review and for recognizing STAMP's practical value.

W1/Q1: Pathology stability or patient/acquisition cues? Value over simpler alternatives.

  • The main pairs hold pathology fixed. To clarify §3.4: all main results use consecutive same-patient radiographs with identical recorded Finding Labels (9,927-pair pool). Label-agnostic pairs are an ablation. STAMP therefore measures stability to acquisition variation at constant recorded pathology, and it never rewards ignoring a recorded pathology change.

  • Evidence that the signal is task-relevant. (i) On the same 40 models, SV alone gives ρ = −0.19 and AV alone 0.64, while the ratio gives 0.89, so stability by itself does not predict OOD performance. (ii) STAMP ranks models correctly within regimes: ρ = .820/.785/.728 within 31 CE models, and .801/.807/.705 regime-stratified. (iii) Class-matched STAMP lifts RSNA from 0.311 to 0.663, which shows the score is tied to the pathology classes.

  • New pathology-stratified analyses and controls (ρ with VinDr / CheXpert / MIMIC AUROC, n = 44, 2,000 pairs each unless noted):

    Semantic pairsVinDrCheXpertMIMIC
    Same patient, same labels (main).855.851.844
    Different patients, same labels, same view.769.687.673
    Same patient, labels changed only.036.229.178
    Same patient, any labels.776.849.834
    Held-out NIH test patients only (1,192 pairs).673.768.726

    Three conclusions follow. (1) With no shared patient anatomy, most of the signal remains, so it is not driven by patient-specific cues. (2) When pairs cross a recorded pathology change, the signal vanishes: being insensitive to pathology change does not predict OOD performance, and the main pairs exclude such cases. (3) The signal holds on patients never seen in training.

  • Matched models. With every method recomputed on identical models, STAMP is statistically equivalent to labeled NIH AUROC without evaluation labels (40 models: .891/.843/.817 vs. .918/.872/.879; 31 CE models: .820/.785/.728 vs. .866/.736/.773; paired bootstrap CIs include 0). No other label-free baseline is reliable on both sets: TTA-Multi reaches ≈ .75 within CE models but ≈ .31 once probes are included, and ATC, ECE and AoTL are all lower. Augmentation-pair variants are the same 1−SV/AV1-SV/AV statistic with a different pair source.

  • Selection regret (best OOD AUROC minus that of STAMP's top-1 model, out of 44): VinDr 0.000 (STAMP selects the true best model), CheXpert 0.012, MIMIC 0.012. Random selection gives 0.052 / 0.043 / 0.038.

W2/Q2: Theory.

  • Lemma 1 and longitudinal pairs. For visits conditionally i.i.d. given the patient, E[SV]=2 Ep tr Var(f∣p)\mathbb E[SV]=2\,\mathbb E_p\,\mathrm{tr}\,\mathrm{Var}(f\mid p), so longitudinal STAMP estimates the correlation ratio with patient × finding state as the grouping variable. We add this estimand explicitly.
  • Conditions for OOD error ordering (Q2). With U(f)U(f) our upper bound and slack s(f)=U(f)−et(f)≥0s(f)=U(f)-e_t(f)\ge0: et(f1)≤et(f2)−[U(f2)−U(f1)]+s(f2)e_t(f_1)\le e_t(f_2)-[U(f_2)-U(f_1)]+s(f_2). The ordering holds whenever the bound gap exceeds the slack. UU is now an exact inequality, U(f)=4κf((1−η2)/η2+Ld/SB)2U(f)=\frac{4}{\kappa_f}\big(\sqrt{(1-\eta^2)/\eta^2}+Ld/\sqrt{S_B}\big)^2 with measurable margin regularity κf=min⁡yγy2/SB\kappa_f=\min_y\gamma_y^2/S_B, so no approximation error remains to control.
  • Table 9. Column 2 is augmentation-pair STAMP and Δ = column 2 − column 1 (mean 0.152). The header and sign are fixed. No headline result depends on it.
  • Lemma 4. The constant is corrected, and the lemma is replaced by a variance-adaptive Bernstein bound with explicit denominator control: ∣SV/AV−μs/μa∣≤(εs+μsμaεa)/(μa−εa)|SV/AV-\mu_s/\mu_a|\le(\varepsilon_s+\tfrac{\mu_s}{\mu_a}\varepsilon_a)/(\mu_a-\varepsilon_a) for εa<μa\varepsilon_a<\mu_a. Empirically the bootstrap CI half-width is ≈ 0.013.

Q3: RSNA overlap. RSNA is a re-annotated subset of NIH ChestX-ray14, so we treat it as a re-annotation (label-shift) stress test, and it is already reported as the failure case. No headline claim uses it: VinDr, CheXpert and MIMIC come from different hospitals with no NIH overlap. The revision will report exact image- and patient-level overlap with the training split and the pair pools, plus RSNA results restricted to held-out NIH test images.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.