- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a framework to adapt pretrained clinical waveform models to new tasks with limited labeled target data. It also addresses reliability concerns in clinical setting. The papers extensively evaluate the framework on multiple clinical datasets and considers cross-hospital transfer, changes in clinical settings, and different waveform acquisition protocols.
Critical strengths
The paper addresses an important problem of adapting pretrained clinical models to new tasks with limited labeled data. The papers extensively evaluate the framework on multiple clinical datasets and considers cross-hospital transfer, changes in clinical settings, and different waveform acquisition protocols.
Critical weaknesses
The paper proposes a framework for automated model adaptation, but several key steps in the Adapter and Automator are not clearly explained or sufficiently evaluated, e.g. the selection of key parameters and the computational overhead introduced by training and evaluating multiple adaptation recipes need further discussion. These are important practical considerations for a framework intended for real-world clinical applications. Also, clinical deployment should consider not only predictive performance but also how to maximize clinical utility.
I also have concerns about the experimental evaluation, especially in clinical settings with highly imbalanced and limited data. The use of AUROC for recipe selection, the fairness of baseline comparisons, and the small number of positive patients raise questions about the reliability of the reported improvements.
Questions for authors
- Adaptor
Eq.3 defines a supervised recipe-fitting objective. a. Could the authors clarify whether the naive selector in Figure 3B uses the same trained recipe library as AutoAdapt? b. Could the authors provide additional ablation results to show the effectiveness of the components in Eq.3?
- Automator
The paper states that the selection scope of the Automator's structural choices is reported in Section 5. Several key components seems unclear, e.g. coefficients (\theta) \mathcal N and rho_t .
- Computational complexity
The combination of multiple checkpoints, fine-tuning strategies, data policies, and input routes may introduce substantial computational overhead and additional model-selection complexity.
- AUROC choice.
The Automator uses patient-level AUROC as the primary adaptation utility for recipe selection. However, several MIMIC-IV tasks contain only 1–6 positive adaptation patients, which may lead to highly unstable AUROC estimates. Many clinical prediction tasks are inherently class-imbalanced.
5.Evaluation sample size
Table 4 reports evaluation patient counts for MIMIC-IV, the sample sizes and positive event rates are not fully reported for the other datasets. Also, how the five-minute waveform segments are selected from the observation window?
- Fairness of baseline comparisons
Table 2 shows substantial improvements over full fine-tuning and other adaptation baselines. However, several MIMIC-IV tasks contain only a handful of positive adaptation patients. AutoAdapt benefits from multiple source checkpoints, adaptation strategies, and reliability-based model selection.
7.Adaptation cohort selection
The authors state that each ALOTT task uses 64 adaptation patients (32 positive and 32 negative). Could the authors clarify the sampling procedure?
- MIMIC-IV
The MIMIC-IV mortality adaptation cohort contains only 1–2 positive patients, while most target-adaptation strategies require at least four positive patients (Table 8).
2 comments
Response to Reviewer xYMM (2/2)
Q5: Sample sizes and waveform segments
We report evaluation patient counts and positive-patient counts for every MIMIC-IV task in the appendix, and the input is the final five minutes before the prediction time. The waveform is resampled to 125 Hz and mapped to the fixed channel vocabulary of the source model. Missing channels are zero-filled and accompanied by a presence mask.
Q6: Fairness of baseline comparisons
All methods use the same adaptation and evaluation patients in each run, and the fixed recipes and the naive selector use the same candidate recipe library as AutoAdapt. The naive selector differs from AutoAdapt only in selecting a single recipe rather than estimating a reliability-weighted combination. Figure 3C isolates the contribution of the combination on the same library, patients, and seeds: adding the weighted recipe combination to reliability routing raises mean AUROC from .704 to .846, and adding candidate-level prediction geometry raises it to .877.
Comparison with existing combination methods. LoraHub, AdaMerging, and HOSO also combine multiple candidates, but without patient-level reliability. On MIMIC-IV, they reach .561, .553, and .564 mean AUROC, against .877 for AutoAdapt (Table 2). On ALOTT, LoraHub is the strongest displayed baseline at .656, against .701 for AutoAdapt (Table 1).
Comparison with a selector that uses evaluation labels. Figure 3B compares AutoAdapt with a frozen-grid oracle that chooses the best of the 16 frozen checkpoint–head recipes for each task using evaluation labels. AutoAdapt exceeds this oracle on every task without using evaluation labels.
Q7: Sampling of the ALOTT adaptation cohort
Each ALOTT task uses the same 64 adaptation patients, consisting of 32 positive and 32 negative patients, for every method. The evaluation cohort is kept separate, and training uses at most eight waveform windows per patient so that recording length cannot substitute for the number of patients.
Q8: Mortality with one or two positive adaptation patients
With only one or two positive adaptation patients, target-fitting recipes are treated as unreliable under the minimum-support rule, and an undefined leave-one-positive comparison is not used as evidence for adaptation. Under Table 8, a recipe that fits target parameters requires at least four positive adaptation patients. When removing the only positive patient makes the leave-one-positive AUROC undefined, that diagnostic is not used. The system then retains a frozen source-based default rather than interpreting an undefined comparison as evidence for adaptation. Mortality also has only 6–7 positive evaluation patients, so we interpret it as the highest-variance task in Table 2 and state this in Section 6.2.
Response to Reviewer xYMM (1/2)
Thank you for your review and for recognizing the importance of adapting clinical models with limited labeled data and the breadth of our evaluation across cross-hospital transfer, care settings, and acquisition protocols. We address your concerns and questions below.
The naive selector uses the same candidate recipe library and the same adaptation patients as AutoAdapt. Its only difference is that it selects a single recipe rather than estimating a reliability-weighted combination. Figure 3A also shows that the choices within a recipe interact: the best head changes with the checkpoint, and mean AUROC ranges from .505 to .626 across the 16 checkpoint–head pairs. A single fixed choice is therefore not best for every task.
The coefficients θ, the combination map Nλ, and the global thresholds are shared across all tasks and fixed before the task-specific weights are estimated; ρt increases with the number of positive adaptation patients and is zero when the comparison with the default is not identifiable.
Coefficients θ. The scoring function of Eq. 6 can be written as at(r)=βt⊤ϕt(r) with βt=θ0+Θht and θ=[θ0;vec(Θ)]. Each recipe is scored from its features ϕt(r): its adaptation evidence, task similarity, distribution-shift descriptors, prediction geometry, stability, and update complexity. The feature weights βt depend on the evidence available for the task, ht, which contains the numbers of positive and negative adaptation patients and the adaptation prevalence.
Recipe-combination map Nλ. Nλ maps the recipe scores (at(r))r∈Rt to task-specific combination weights wt, with one regularization setting λ shared across all tasks.
Reliability weight ρt. The support component of ρt is
nt++κnt+,
where nt+ is the number of positive adaptation patients and κ is a positive-support smoothing constant. This component is combined with the bootstrap probability that the weighted combination improves over the default recipe, with leave-one-positive-patient stability, and with the minimum-support checks of Table 8. If the adaptation comparison is not identifiable, the system assigns zero reliability, and the final weights wt=ρtwt+(1−ρt)er0,t reduce to the default recipe.
Global parameters. AUROC is the primary selection score, with a paired safety margin of .02 AUROC, a secondary Brier margin of .01, and 5,000 patient bootstrap draws for recipe selection. The concrete recipe fits and combination weights are computed only from the declared adaptation patients. We have added these definitions to Section 4.2, and the reproducibility appendix reports the support threshold, the paired AUROC margin, the Brier margin, the bootstrap count, and the fallback rule.
Frozen recipes only require fitting a small target head, and recipe selection is performed once per task before final inference. Frozen recipes reuse cached encoder representations. Recipes that update the encoder change the following numbers of parameters:
At inference, only the recipes with nonzero combination weight are evaluated.
AUROC is one instantiation of the selection score St, and AutoAdapt does not rely on it alone: recipes are compared only when enough positive patients are available, and a comparison is accepted only when it remains stable. AUROC is threshold-free and does not depend on prevalence, which makes it suitable for comparing recipes across imbalanced tasks. Its instability with few positive patients is handled by four safeguards: a minimum number of positive adaptation patients (Table 8), stability under leaving out one positive patient, a paired bootstrap bound with a margin of .02 AUROC, and a secondary calibration check with a .01 Brier margin. When these checks fail, AutoAdapt keeps the default recipe. Any other prespecified score, such as AUPRC, can replace AUROC without changing the Automator.