acceptodds
Under review as a conference paper at ICLR 2027

Not All Gains Are Equal: Controlled Attribution in Medical Foundation Model Deployment

Abstract

Local adaptation can improve a medical foundation model's performance, but a lower post-adaptation error alone does not reveal whether the gain comes from correcting systematic output shift, using local supervision, or changing the reliability of downstream decisions. We introduce a controlled deployment attribution framework that separates calibration-recoverable improvement, the supervision-dependent value of released predictions, and the decision-level effect of predictive uncertainty. We instantiate the framework on PanEcho using 10,175 echocardiography sequences linked to 8,283 reports with seven continuous measurements. Before crediting a replacement head with the full improvement, we find that train-only affine calibration recovers 74.5% of the specified macro normalized-MAE gap between released predictions and a historical local MLP. Released predictions are most useful as an anchor under limited local supervision, while their advantage narrows as labels increase. Propagating patient-group prediction intervals through fixed measurement-derived rules changes the downstream risk–coverage trade-off: false assertions decrease from 10.52% at 100% assertion coverage to 2.47% at 49.40% coverage. In this controlled deployment setting, the results support attributing observed local gains to calibration, supervision, and decision-level effects rather than to a single adaptation gain.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Models, Model Adaptation, Model Evaluation, Calibration, Uncertainty Quantification, Medical AI

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_hrWLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Here, the authors study the local adaptation of PanEcho and especially disentangle the adaptation improvement in terms of output calibration, residual prediction, and uncertainty-based abstention.

Critical strengths
  • Studying what actually is happening when adapting these medical foundation models is important, especially when the finding is what the authors find: that in certain settings a simple recalibration would be enough to get the same outcome.
Critical weaknesses
  • W1. For me, the main weakness is the contribution of the paper. The evaluation covers one model, one center, and seven related measurement targets, and it is unclear what general insight readers should take from it. The calibration finding is interesting, but whether it generalizes across domains or even across models for the same domain remains open. The other findings, that retaining pretrained predictions helps with fewer labels and that abstention reduces errors among retained cases, are expected. For me, the paper offers a detailed local evaluation of PanEcho, but the contribution beyond that is too limited for ICLR.
  • W2. Another aspect is that recovering part of the error gap with an affine transformation does not explain what caused the gap or what the replacement head learned. The transformation could correct an offset, rescale predictions, or shrink them toward the local target distribution. So, for me, the interpretation as "attribution" is too strong for what the results actually show.
Questions for authors

See weaknesses.

Reviewer_oSdzOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors introduce a framework for deployment attribution. The authors experiment using a private dataset of echocardiograms using PanEcho. The paper is limited by its unmotivated definitions and desiderata, the use of a single dataset on a single domain that limits the generalizability of the method, the lack of sufficient comparator methods, and insufficient dataset details. As it stands, the work does not meet the technical bar for acceptance; consequently, I recommend rejection.

Critical strengths
  • Large dataset. 10,175 echocardiography sequences is an impressive amount of data and the annotation / label generation effort seems to be high.
Critical weaknesses
  • Unmotivated definitions. The authors do not motivate the different definitions for their targets. For example: there is an entire body of research into uncertainty quantification which the authors reference but does not motivate the method’s definition (see Section 2.3)
  • Single dataset, single domain. The authors use PanEcho weights on a private dataset. Evaluating their method on such a setting (particularly without comparator methods, see below) raises questions on the generalizability of their results or method.
  • Comparator methods. The authors report no comparator methods outside of zero-shot PanEcho results (Table 1) in their main text. This includes both cardiac foundation models and other attribution methods. While the appendices mention EchoPrime with some preliminary results, they are insufficient in supporting the claims raised in the main text.
  • Dataset details and preprocessing methods. It’s not clear to me whether the authors’ dataset uses B-mode echocardiograms (which would be a series of grayscale images). The authors select 16 RGB frames. If the images are B-scale, the decision to convert them to JPEG and RGB seems unnecessary (I suspect it’s because they are using PanEcho, but they should explicitly mention this). Regardless, without both sufficient information about the dataset it is difficult to assess whether the preprocessing methods are appropriate or not.
  • Paper is hard to read. I found the entire paper difficult to read: an illustrative example would be the last sentences of each Related Work subsections that begin with “Our work differs…”
  • Possible anonymity breach. Section 6.4 contains the name of the clinical center.
Questions for authors

Please see the weaknesses section above.

Reviewer_SofBOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper aims to characterize and enhance the performance loss of adapting medical foundation models in local contexts. They propose an empirical framework for correcting output shift via calibration, using local supervision, and changing the reliability based on uncertainty. They focus on an echocardiogram dataset.

Critical strengths
  • The problem is interesting and well motivated.
Critical weaknesses
  • The introduction lacks citations.
  • Some of the text is difficult to understand, such as : "Retaining the released prediction may also help when local labels are limited, even though this prediction is derived from the same frozen representation available to the local head."
  • Although this is an empirical study, the scope is very limited. It would be interesting to investigate this across multiple foundation models and datasets.
Questions for authors

See above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.