acceptodds
Under review as a conference paper at ICLR 2027

AutoAdapt: Reliable Few-Shot Adaptation under Clinical Distribution Shifts

Abstract

Large pretrained clinical models provide a practical way to reuse learned prior knowledge across hospitals by adapting models to them. In practice, a target hospital may only have a small labeled patient cohort, a setting commonly referred to as few-shot adaptation. This requires making multiple decisions, such as which pretrained model to adapt, how much of the model to update, and which patients to use. Nevertheless, this process faces two primary challenges. First, the best adaptation strategy varies across clinical tasks. Second, evaluating and comparing candidate strategies becomes unreliable due to the small patient cohort. In this work, we introduce AUTOADAPT with two core designs to deal with these challenges. The Adapter defines an extensible space of adaptation recipes, and the Automator selects a recipe based on experimental evidence within the adaptation patients. We propose a reliability rule to ensure that only the most effective strategy on most available patients will be selected. These selected strategies then form a combination for effective few-shot adaptation. We conduct extensive experiments across critical care, emergency care, and diagnostic datasets, and the results show that AUTOADAPT consistently achieves state-of-the-art performance using only a few patients for adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Few-shot Adaptation, Out-of-Distribution Generalization, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_xYMMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a framework to adapt pretrained clinical waveform models to new tasks with limited labeled target data. It also addresses reliability concerns in clinical setting. The papers extensively evaluate the framework on multiple clinical datasets and considers cross-hospital transfer, changes in clinical settings, and different waveform acquisition protocols.

Critical strengths

The paper addresses an important problem of adapting pretrained clinical models to new tasks with limited labeled data. The papers extensively evaluate the framework on multiple clinical datasets and considers cross-hospital transfer, changes in clinical settings, and different waveform acquisition protocols.

Critical weaknesses

The paper proposes a framework for automated model adaptation, but several key steps in the Adapter and Automator are not clearly explained or sufficiently evaluated, e.g. the selection of key parameters and the computational overhead introduced by training and evaluating multiple adaptation recipes need further discussion. These are important practical considerations for a framework intended for real-world clinical applications. Also, clinical deployment should consider not only predictive performance but also how to maximize clinical utility.

I also have concerns about the experimental evaluation, especially in clinical settings with highly imbalanced and limited data. The use of AUROC for recipe selection, the fairness of baseline comparisons, and the small number of positive patients raise questions about the reliability of the reported improvements.

Questions for authors
  1. Adaptor

Eq.3 defines a supervised recipe-fitting objective. a. Could the authors clarify whether the naive selector in Figure 3B uses the same trained recipe library as AutoAdapt? b. Could the authors provide additional ablation results to show the effectiveness of the components in Eq.3?

  1. Automator

The paper states that the selection scope of the Automator's structural choices is reported in Section 5. Several key components seems unclear, e.g. coefficients (\theta) \mathcal N and rho_t .

  1. Computational complexity

The combination of multiple checkpoints, fine-tuning strategies, data policies, and input routes may introduce substantial computational overhead and additional model-selection complexity.

  1. AUROC choice.

The Automator uses patient-level AUROC as the primary adaptation utility for recipe selection. However, several MIMIC-IV tasks contain only 1–6 positive adaptation patients, which may lead to highly unstable AUROC estimates. Many clinical prediction tasks are inherently class-imbalanced.

5.Evaluation sample size

Table 4 reports evaluation patient counts for MIMIC-IV, the sample sizes and positive event rates are not fully reported for the other datasets. Also, how the five-minute waveform segments are selected from the observation window?

  1. Fairness of baseline comparisons

Table 2 shows substantial improvements over full fine-tuning and other adaptation baselines. However, several MIMIC-IV tasks contain only a handful of positive adaptation patients. AutoAdapt benefits from multiple source checkpoints, adaptation strategies, and reliability-based model selection.

7.Adaptation cohort selection

The authors state that each ALOTT task uses 64 adaptation patients (32 positive and 32 negative). Could the authors clarify the sampling procedure?

  1. MIMIC-IV

The MIMIC-IV mortality adaptation cohort contains only 1–2 positive patients, while most target-adaptation strategies require at least four positive patients (Table 8).

2 comments
AuthorsOpenReview ↗

Response to Reviewer xYMM (1/2)

Thank you for your review and for recognizing the importance of adapting clinical models with limited labeled data and the breadth of our evaluation across cross-hospital transfer, care settings, and acquisition protocols. We address your concerns and questions below.

Q1: The naive selector

The naive selector uses the same candidate recipe library and the same adaptation patients as AutoAdapt. Its only difference is that it selects a single recipe rather than estimating a reliability-weighted combination. Figure 3A also shows that the choices within a recipe interact: the best head changes with the checkpoint, and mean AUROC ranges from .505 to .626 across the 16 checkpoint–head pairs. A single fixed choice is therefore not best for every task.

Q2: Coefficients θ\theta, Nλ\mathcal N_\lambda, and ρt\rho_t

The coefficients θ\theta, the combination map Nλ\mathcal N_\lambda, and the global thresholds are shared across all tasks and fixed before the task-specific weights are estimated; ρt\rho_t increases with the number of positive adaptation patients and is zero when the comparison with the default is not identifiable.

Coefficients θ\theta. The scoring function of Eq. 6 can be written as at(r)=βt⊤ϕt(r)a_t(r)=\beta_t^\top\phi_t(r) with βt=θ0+Θ ht\beta_t=\theta_0+\Theta\,h_t and θ=[θ0;vec⁡(Θ)]\theta=[\theta_0;\operatorname{vec}(\Theta)]. Each recipe is scored from its features ϕt(r)\phi_t(r): its adaptation evidence, task similarity, distribution-shift descriptors, prediction geometry, stability, and update complexity. The feature weights βt\beta_t depend on the evidence available for the task, hth_t, which contains the numbers of positive and negative adaptation patients and the adaptation prevalence.

Recipe-combination map Nλ\mathcal N_\lambda. Nλ\mathcal N_\lambda maps the recipe scores (at(r))r∈Rt(a_t(r))_{r\in\mathcal R_t} to task-specific combination weights w~t\widetilde w_t, with one regularization setting λ\lambda shared across all tasks.

Reliability weight ρt\rho_t. The support component of ρt\rho_t is

nt+nt++κ,\frac{n_t^+}{n_t^+ + \kappa},

where nt+n_t^+ is the number of positive adaptation patients and κ\kappa is a positive-support smoothing constant. This component is combined with the bootstrap probability that the weighted combination improves over the default recipe, with leave-one-positive-patient stability, and with the minimum-support checks of Table 8. If the adaptation comparison is not identifiable, the system assigns zero reliability, and the final weights wt=ρtw~t+(1−ρt)er0,tw_t=\rho_t\widetilde w_t+(1-\rho_t)e_{r_{0,t}} reduce to the default recipe.

Global parameters. AUROC is the primary selection score, with a paired safety margin of .02 AUROC, a secondary Brier margin of .01, and 5,000 patient bootstrap draws for recipe selection. The concrete recipe fits and combination weights are computed only from the declared adaptation patients. We have added these definitions to Section 4.2, and the reproducibility appendix reports the support threshold, the paired AUROC margin, the Brier margin, the bootstrap count, and the fallback rule.

Q3: Computational cost

Frozen recipes only require fitting a small target head, and recipe selection is performed once per task before final inference. Frozen recipes reuse cached encoder representations. Recipes that update the encoder change the following numbers of parameters:

Adaptation routeUpdated parameters
LoRA197,121
Partial fine-tuning6,306,305
Full fine-tuning26.81M

At inference, only the recipes with nonzero combination weight are evaluated.

Q4: AUROC for recipe selection

AUROC is one instantiation of the selection score St\mathcal S_t, and AutoAdapt does not rely on it alone: recipes are compared only when enough positive patients are available, and a comparison is accepted only when it remains stable. AUROC is threshold-free and does not depend on prevalence, which makes it suitable for comparing recipes across imbalanced tasks. Its instability with few positive patients is handled by four safeguards: a minimum number of positive adaptation patients (Table 8), stability under leaving out one positive patient, a paired bootstrap bound with a margin of .02 AUROC, and a secondary calibration check with a .01 Brier margin. When these checks fail, AutoAdapt keeps the default recipe. Any other prespecified score, such as AUPRC, can replace AUROC without changing the Automator.

AuthorsOpenReview ↗

Response to Reviewer xYMM (2/2)

Q5: Sample sizes and waveform segments

We report evaluation patient counts and positive-patient counts for every MIMIC-IV task in the appendix, and the input is the final five minutes before the prediction time. The waveform is resampled to 125 Hz and mapped to the fixed channel vocabulary of the source model. Missing channels are zero-filled and accompanied by a presence mask.

Q6: Fairness of baseline comparisons

All methods use the same adaptation and evaluation patients in each run, and the fixed recipes and the naive selector use the same candidate recipe library as AutoAdapt. The naive selector differs from AutoAdapt only in selecting a single recipe rather than estimating a reliability-weighted combination. Figure 3C isolates the contribution of the combination on the same library, patients, and seeds: adding the weighted recipe combination to reliability routing raises mean AUROC from .704 to .846, and adding candidate-level prediction geometry raises it to .877.

Comparison with existing combination methods. LoraHub, AdaMerging, and HOSO also combine multiple candidates, but without patient-level reliability. On MIMIC-IV, they reach .561, .553, and .564 mean AUROC, against .877 for AutoAdapt (Table 2). On ALOTT, LoraHub is the strongest displayed baseline at .656, against .701 for AutoAdapt (Table 1).

Comparison with a selector that uses evaluation labels. Figure 3B compares AutoAdapt with a frozen-grid oracle that chooses the best of the 16 frozen checkpoint–head recipes for each task using evaluation labels. AutoAdapt exceeds this oracle on every task without using evaluation labels.

Q7: Sampling of the ALOTT adaptation cohort

Each ALOTT task uses the same 64 adaptation patients, consisting of 32 positive and 32 negative patients, for every method. The evaluation cohort is kept separate, and training uses at most eight waveform windows per patient so that recording length cannot substitute for the number of patients.

Q8: Mortality with one or two positive adaptation patients

With only one or two positive adaptation patients, target-fitting recipes are treated as unreliable under the minimum-support rule, and an undefined leave-one-positive comparison is not used as evidence for adaptation. Under Table 8, a recipe that fits target parameters requires at least four positive adaptation patients. When removing the only positive patient makes the leave-one-positive AUROC undefined, that diagnostic is not used. The system then retains a frozen source-based default rather than interpreting an undefined comparison as evidence for adaptation. Mortality also has only 6–7 positive evaluation patients, so we interpret it as the highest-variance task in Table 2 and state this in Section 6.2.

Reviewer_vTuzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AutoAdapt describes few shot adaptation of pretrained clinical physiological models under distribution shift. The adapter builds recipes combining source ckpt, update methods, patient/window policies and input modalities. The automator weights recipes using held out patient predictions, clinical-task similarity, input distribution compatibility and stability. Also, the reliability gate blends the combination with a predefined default when evidence is weak. Across six tasks each, paper reports mean AUROC of 0.701 versus 0.656 on ALOTT and 0.877 versus 0.585 on MIMIC-IV, against the strongest reported baselines. Both ALOTT and MIMIC-IV are development benchmarks.

Critical strengths
  1. The paper treats patients rather than correlated waveform windows as independent evidence. It uses patient level loss, held out predictions, and positive patient counts. This is appropriate for few shot clinical adaptation and makes label scarcity explicit especially for mortality.

  2. The paper also evaluates multiple transfer settings, repeated adaptation panels and naive selection using the same candidate library. The results suggest that checkpoint and prediction-head choices interact, supporting joint selection. On MC-MED AutoAdapt achieves 0.678 AUROC versus 0.672 for local EHR and 0.637 for fixed fusion. This supports retaining a reliable default though improvement over local EHR is marginal.

Critical weaknesses
  1. In Eq 6, it does not define N_lambda, and Eq 7 does not specify how rho_t is computed or how these relate to the eligibility rules in Table 8. The paper also does not explain how global parameters were selected.

  2. Further, table 4 reports only 1–2 positive adaptation patients, while Table 8 requires at least four for target-fitted methods. Table 5 also lists no aligned mortality source head. It is unclear which recipes are eligible, what default is used, and how leave one positive validation works when removing the only positive makes AUROC undefined.

  3. AutoAdapt combines multiple recipes while the naive selector chooses one. It is unclear whether other baselines have access to the same checkpoints, aggregation policies, and modalities. On ALOTT, AutoAdapt uses fusion in four of six tasks, but EHR only and fixed fusion comparisons shown would be helpful.

  4. Figure 3C does not clearly define the ablation stages or isolate contribution. Figure 2 also omits fourth recipe dimension. The reported 20 task ALOTT mean (0.686) also differs from the unweighted average of the displayed results (~0.681)

Questions for authors
  1. How were params : theta, lambda, weight constraints, gate thresholds, and defaults selected? Were evaluation results used? Please clarify the data splits, random seeds, and evaluation metrics in Figure 2.

  2. How are N_lambda and rho_t computed, and how do they relate to Table 8? For mortality with 1-2 positive patients, which recipes and defaults are valid? How is leave-one-positive validation handled when no positives remain?

  3. Do baselines have access to the same checkpoints, updates and aggregation policies? How does AutoAdapt compare with a matched regularized ensemble without task features or reliability gate? Please clarify the ablation stages in Figure 3C.

  4. Does bootstrapping repeat the recipe selection and weight estimation or only evaluate fixed weights? Also why does the reported 20 task ALOTT mean (0.686) differ from the displayed average (~0.681)?

It will be useful to explain further how much the proposed reliability mechanism improves over existing combination methods.

2 comments
AuthorsOpenReview ↗

Response to Reviewer vTuz (1/2)

We thank you for your detailed review and for recognizing that AutoAdapt treats patients, rather than correlated waveform windows, as the unit of evidence, and that it evaluates multiple transfer settings with repeated adaptation panels. We address your concerns and questions below.

W1/Q1/Q2: Nλ\mathcal N_\lambda, ρt\rho_t, and global parameters

The reliability weight ρt\rho_t increases with the number of independent positive adaptation patients and is set to zero when the comparison with the default is not identifiable. All global thresholds are shared across tasks and fixed before the task-specific combination weights are estimated.

Recipe-combination map Nλ\mathcal N_\lambda. Nλ\mathcal N_\lambda maps the recipe scores (at(r))r∈Rt(a_t(r))_{r\in\mathcal R_t} of Eq. 6 to task-specific combination weights w~t\widetilde w_t, with one regularization setting λ\lambda shared across all tasks. We have added its definition to Section 4.2.

Reliability weight ρt\rho_t. The support component of ρt\rho_t is

nt+nt++κ,\frac{n_t^+}{n_t^+ + \kappa},

where nt+n_t^+ is the number of positive adaptation patients and κ\kappa is a positive-support smoothing constant. This component is combined with the bootstrap probability that the weighted combination improves over the default recipe, with leave-one-positive-patient stability, and with the minimum-support checks of Table 8. If the adaptation comparison is not identifiable, the system assigns zero reliability, and the final weights wt=ρtw~t+(1−ρt)er0,tw_t=\rho_t\widetilde w_t+(1-\rho_t)e_{r_{0,t}} reduce to the default recipe.

Global parameters. The global thresholds are shared across tasks and fixed before the final task-specific combination weights are estimated. AUROC is the primary selection score, with a paired safety margin of .02 AUROC, a secondary Brier margin of .01, and 5,000 patient bootstrap draws for recipe selection. The concrete recipe fits and combination weights are computed only from the declared adaptation patients. The reproducibility appendix reports the support threshold, the paired AUROC margin, the Brier margin, the bootstrap count, and the fallback rule.

Splits and metrics. MIMIC-IV uses five fixed patient partitions. Each run uses an adaptation cohort and a disjoint evaluation cohort, and all methods are evaluated at the patient level after aggregating each patient's waveform windows. On ALOTT, every method uses the same 32 positive and 32 negative adaptation patients per task. AUROC is the primary metric, and the Brier score serves as a secondary calibration check.

W2/Q2: Mortality with one or two positive adaptation patients

With only one or two positive adaptation patients, target-fitting recipes are treated as unreliable under the minimum-support rule, and an undefined leave-one-positive comparison is not used as evidence for adaptation. Under Table 8, a recipe that fits target parameters requires at least four positive adaptation patients. When removing the only positive patient makes the leave-one-positive AUROC undefined, that diagnostic is not used. The system then retains a frozen source-based default rather than interpreting an undefined comparison as evidence for adaptation. Mortality also has only 6–7 positive evaluation patients, so we interpret it as the highest-variance task in Table 2 and state this in Section 6.2.

AuthorsOpenReview ↗

Response to Reviewer vTuz (2/2)

W3/Q3: Access of baselines to the same library, and comparison with existing combination methods

All methods use the same adaptation and evaluation patients in each run, and the naive selector and the fixed recipes in Figure 3 use the same candidate recipe library as AutoAdapt. The naive selector differs from AutoAdapt only in selecting a single recipe rather than estimating a reliability-weighted combination. Figure 3C isolates the contribution of the combination on the same library, patients, and seeds: adding the weighted recipe combination to reliability routing raises mean AUROC from .704 to .846, and adding candidate-level prediction geometry raises it to .877.

Comparison with existing combination methods. LoraHub, AdaMerging, and HOSO also combine multiple candidates, but without patient-level reliability. On MIMIC-IV, they reach .561, .553, and .564 mean AUROC, against .877 for AutoAdapt (Table 2). On ALOTT, LoraHub is the strongest displayed baseline at .656, against .701 for AutoAdapt (Table 1).

W4/Q3/Q4: Figure 3C, Figure 2, the 20-task ALOTT mean, and the bootstrap

Figure 3C stages. Each stage adds one component while keeping the patients and seeds fixed:

  1. Pretrained + head (.567): the general MIMIC-III pretrained encoder, frozen, with a target logistic head.
  2. Fixed recipe (.626): the strongest single checkpoint–head recipe in Figure 3A, the deterioration checkpoint with a cosine prototype.
  3. Reliability routing (.704): the Automator selects recipes using the task and shift prior, the held-out utility, and the reliability rule.
  4. Weighted portfolio (.846): the single selection is replaced by the weighted recipe combination of Eqs. 6–7.
  5. Candidate geometry (.877): recipe-level prediction geometry is added to the recipe features, so that redundant recipes are distinguished from complementary ones.

We have revised the caption and Section 6.6 to define these five stages.

Figure 2. We have added the input-route dimension to Figure 2, so that it shows all four recipe dimensions.

20-task ALOTT mean. Thank you for catching this. The mean over the 20 displayed tasks is .681, not .686, and we have corrected this value in the revision.

Bootstrap. The bootstrap resamples patients rather than waveform windows. Recipe-selection diagnostics use 5,000 draws, and the main shift analyses use 2,000 draws. The patient-level split and the prediction protocol remain fixed within each analysis.

Reviewer_tcUGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes AutoAdapt, a framework for few-shot adaptation of pretrained clinical models to new hospitals under distribution shifts. It constructs a manually specified library of adaptation recipes combining different pretrained checkpoints, fine-tuning strategies, data policies, and input modalities, and uses a reliability-aware mechanism to select and ensemble recipes based on limited labeled patient data. Experiments on physiological waveform and clinical datasets report improvements over existing adaptation methods, including a substantial increase in mean AUROC from 0.585 to 0.877 on six MIMIC-IV prediction tasks with only 1–6 positive adaptation patients.

Critical strengths
  1. The paper addresses a practically important problem: adapting pretrained clinical models to new hospitals with very limited labeled data.
  2. The proposed framework is modular and conceptually straightforward, combining established adaptation techniques without requiring substantial changes to the underlying model architecture.
Critical weaknesses
  1. In many places, the writing resembles the style of auto-research agents, with densely packed technical details, insufficient explanations, and imprecise terminology. 1) Technical details are compressed without establishing their meaning. "The deterioration, respiratory, and circulatory source tasks use 484,667, 551,232, and 578,274 windows." This statement leaves the reader to infer what constitutes a window, why different source tasks have different counts, and what these counts imply about independent training examples. 2) Technical terms are introduced without sufficient operational detail. For example Eq.6: "A linear scoring map with coefficients θ produces ... " But why the scoring function is defined in this way is not explained. 3) Some wording slips resemble errors commonly seen in LLM-generated text, for example "MIMIC-III supplies the source waveform checkpoints" however, MIMIC-III only supplies the data, and the authors trained checkpoints from these data.
  2. The improvements in Table 2 are suspiciously large and insufficiently explained. AutoAdapt improves the mean AUROC from 0.585 (the strongest baseline) to 0.877, despite using only 1–6 positive patients for adaptation. The improvements are particularly striking for AKI (0.508 → 0.904) and delirium (0.548 → 0.842), which are challenging to predict from physiological waveforms alone. Given that AutoAdapt primarily selects and combines existing adaptation strategies rather than introducing a fundamentally new representation-learning method, such dramatic gains are difficult to explain. Furthermore, the evaluation cohorts are extremely small, with only 6–7 positive patients for mortality, and no confidence intervals are reported. These results raise substantial concerns about statistical reliability, baseline fairness, and potential overfitting to the development benchmarks. In particular, comparing table 1 and 2, we see that the reported improvement becomes dramatically larger when the number of positive adaptation patients becomes much smaller.
  3. The candidate recipe library is manually specified, which creates a maintenance burden. Moreover, as the design space grows, manually constructing and maintaining the recipe library becomes increasingly difficult and does not scale well. This limits the framework's extensibility, despite extensibility being presented as one of its main contributions.
Questions for authors
  1. Eq.6 is written as at(r)=θ⊤[ϕt(r);ϕt(r)⊗ht].a_t(r)=\theta^\top[\phi_t(r);\phi_t(r)\otimes h_t]. which is not intuitive. For better clarity, it could be rewritten as at(r)=w⊤ϕt(r)+ϕt(r)⊤Whta_t(r)=w^\top\phi_t(r)+\phi_t(r)^\top W h_t.
2 comments
AuthorsOpenReview ↗

Response to Reviewer tcUG (1/2)

Thank you for your careful reading and for recognizing the practical importance of few-shot adaptation across hospitals and the modular design of AutoAdapt. We address your concerns and questions below.

W1/Q1: Clarity, compressed technical details, and Eq. 6

We have revised the Method and Experimental Setup sections so that each technical detail is introduced together with its meaning and purpose, and we have rewritten Eq. 6 in the clearer form you suggest. We address your three examples below.

Source windows. We have rewritten this passage as follows: "Each source checkpoint is trained on labeled five-minute waveform windows from MIMIC-III. The deterioration, respiratory, and circulatory source tasks contain 484,667, 551,232, and 578,274 windows from 5,693, 4,421, and 4,955 patients, respectively. These counts differ because each task has its own label definition and eligibility criteria. Windows from the same patient are correlated, so patient counts, rather than window counts, indicate the amount of independent training evidence."

Scoring function in Eq. 6. The original form at(r)=θ⊤[ϕt(r); ϕt(r)⊗ht]a_t(r)=\theta^\top[\phi_t(r);\,\phi_t(r)\otimes h_t] is equivalent to

at(r)=βt⊤ϕt(r),βt=θ0+Θ ht,a_t(r)=\beta_t^\top\phi_t(r),\qquad \beta_t=\theta_0+\Theta\,h_t ,

where θ=[θ0;vec⁡(Θ)]\theta=[\theta_0;\operatorname{vec}(\Theta)]. Each recipe is scored from its features ϕt(r)\phi_t(r): its adaptation evidence, task similarity, distribution-shift descriptors, prediction geometry, stability, and update complexity. The feature weights βt\beta_t depend on the evidence available for the task, hth_t, which contains the numbers of positive and negative adaptation patients and the adaptation prevalence. The score is then mapped to a task-specific combination weight, which allows AutoAdapt to combine complementary recipes instead of selecting only one winner. We have adopted this form in the revision and explain each feature in Section 4.2.

Terminology. We agree that "MIMIC-III supplies the source waveform checkpoints" is imprecise. MIMIC-III supplies the source data, and we trained the four source checkpoints on these data (Appendix Table 5). We have corrected this sentence and revised similar wording throughout the manuscript.

AuthorsOpenReview ↗

Response to Reviewer tcUG (2/2)

W2: Size of the MIMIC-IV improvements

The MIMIC-IV gains come from components that the baselines do not have, and Figure 3C shows where each part arises: on the same patients and seeds, mean AUROC rises from .567 for the pretrained encoder with a target head to .626 for the strongest fixed recipe, .704 with reliability routing, .846 with the weighted recipe combination, and .877 with candidate-level prediction geometry. No single component accounts for the gain; the largest step comes from combining recipes instead of committing to one.

Why the baselines are low. The MIMIC-IV adaptation cohorts contain only 1–6 positive patients per task, so methods that fit one target predictor have very little to learn from. On the same six tasks, the fixed baselines reach .567 with a frozen encoder and target head, .560 with full fine-tuning, and .572 with LoRA. The existing-work baselines reach .585 for PRAM, .580 for PRAM-MI, .519 for OTTEHR, .561 for LoraHub, .553 for AdaMerging, and .564 for HOSO, against .877 for AutoAdapt (Table 2). AutoAdapt does not need to learn the target relation from these few patients alone: it combines recipes built on source checkpoints specialized for related clinical outcomes, and weights them using patient-held-out evidence, the task prior, and the reliability rule (Section 4.2).

Comparison with a selector that uses evaluation labels. In Figure 3B, the frozen-grid oracle chooses, for each task, the best of the 16 frozen checkpoint–head recipes using evaluation labels. AutoAdapt exceeds this oracle on every task without using evaluation labels. The gain therefore does not come from finding a better single frozen recipe, but from combining complementary recipes.

Why the gain is larger with fewer positive patients. Fewer positive patients mainly hurt the baselines. On ALOTT, with 32 positive adaptation patients, full fine-tuning and LoRA reach .655 and .651 mean AUROC, close to the strongest baseline (.656). On MIMIC-IV, with 1–6 positive patients, the same methods fall to .560 and .572. Methods that depend on fitting the target task degrade as positive patients become scarce, which widens the gap to AutoAdapt. The two benchmarks contain different tasks, so absolute AUROC values are not directly comparable between Tables 1 and 2.

No evaluation labels in fitting. Every method uses the same adaptation and evaluation patients in each run. Candidate predictors and numerical combination weights are fitted only on adaptation patients, and the weights, fitted recipes, and aggregation rules are locked before evaluation patients are scored (Section 4.2).

Patient-level support. Mortality is the most extreme case, with only 1–2 positive adaptation patients and 6–7 positive evaluation patients. We therefore report patient-level support explicitly and interpret mortality as a particularly high-variance few-shot setting.

W3: Manually specified recipe library

The recipe library is generated from short lists of choices rather than written recipe by recipe. The current lists cover four source checkpoints, several update rules, data policies, and input routes. Valid recipes are generated from their Cartesian product and filtered by modality and data-availability constraints (Eq. 2). Adding a new checkpoint or update rule therefore adds one entry to an existing choice list without changing the Automator interface.

Unhelpful entries do not need to be removed by hand. In the checkpoint-pool analysis (Section 6.8), the respiratory specialist is never selected and adds no gain, while the deterioration and circulatory specialists raise mean routing AUROC from .562 to .641 and .646. The Automator assigns no weight to entries that the adaptation evidence does not support.

Scaling. Because recipes are enumerated, the cost of fitting and validating them grows with the size of the library. Very large design spaces would require search rather than enumeration, which we now note as a limitation in Section 7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.