- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses domain adaptation for multivariate time-series models under a locked-predictor setting, where a deployed model's weights are completely frozen due to regulatory certification or closed inference APIs, but the deploying hospital or site has its own target labels. Instead of updating the predictor's parameters, the authors propose PILOT (Per-timestep Input-space adapter for LOcked predicTors), an upstream adapter that transforms the target input data before feeding it into the strictly frozen predictor. The PILOT architecture uses an encoder-decoder structure built with axial attention blocks (across time and features) and FiLM conditioning for static covariates. It operates in two regimes: a source-assisted setting that uses a k-NN memory bank with cross-attention to retrieve local physiological states from source training data, and a source-free setting for strict privacy constraints where no source data is used. Training follows a two-phase protocol: Phase 1 pretrains the adapter using unmasked identity reconstruction, and Phase 2 optimizes the adapter using target labels by back-propagating the task loss directly through the frozen predictor, combined with auxiliary losses for clinical signal preservation and marginal alignment. The authors evaluate PILOT on five clinical YAIB tasks (three classification: mortality, AKI, and sepsis; two regression: length-of-stay and kidney function), adapting a frozen MIMIC-IV LSTM predictor to eICU and HiRID, as well as five non-clinical AdaTime sensor benchmarks using a frozen 1D-CNN. Across both source-assisted and source-free regimes, PILOT outperforms end-to-end domain adaptation and test-time adaptation baselines, matches natively trained target models, and can transfer zero-shot to other frozen architectures like GRU and TCN.
Critical strengths
I like the core idea of focusing on getting the input space (x) right instead of always trying to modify the model weights (f). This provides a nice perspective that solely chasing model-level generalizability or robustness might not be the only way forward for clinical deployment. Furthermore, the paper does not only test on ICU datasets (including both classification and regression tasks), but also evaluates on the AdaTime sensor benchmarks, which nicely shows that this input-adaptation concept can generalize across different time-series domains.
Critical weaknesses
-
Complex training pipeline and need for clearer component ablations: Learning the input adapter (g_theta) requires a fairly heavy pipeline, including an axial-attention encoder-decoder, FiLM conditioning, an optional k-NN memory bank, a two-phase training protocol, and a combination of six different loss terms. Because there are so many moving parts in both the architecture design and the combined loss function, there should be more thorough ablation experiments in the main text isolating each architectural part and loss term to clearly show which specific component contributes the most to the adaptation performance. Furthermore, even though the downstream predictor is frozen, training the PILOT module itself still requires back-propagating through the frozen model and tuning task-specific hyperparameters. The paper lacks a clear analysis of the trade-off between the compute and effort needed to train g_theta versus the actual performance gain.
-
Incomplete experimental directions and simple frozen backbones: For the ICU experiments in the main text, the authors only show adaptation in one direction (using MIMIC-IV as the source and adapting to eICU or HiRID) rather than testing all source-target pairs. In addition, the frozen predictors used in the experiments are relatively basic architectures like standard LSTMs and 1D-CNNs. Given recent progress in time-series foundation models and state-of-the-art clinical forecasters (such as MIRA from Microsoft), the paper would be much stronger if it showed that PILOT can also adapt modern, large pretrained time-series backbones.
Questions for authors
-
Given that g_theta combines multiple architectural modules and six loss terms, could the authors provide a clearer, step-by-step ablation in the main text showing which specific architecture components and loss terms contribute the most to the adaptation gains?
-
Could the authors provide additional experiments using more modern, state-of-the-art frozen time-series predictors or foundation models (e.g., MIRA or Transformer-based backbones) beyond simple LSTMs and 1D-CNNs? Testing other source-target directions in the ICU benchmarks would also make the evaluation more complete.
-
Could the authors provide a clear trade-off analysis comparing the compute cost and parameter count of training PILOT (g_theta) against the performance gains achieved, as well as against the compute cost of training the frozen predictor from scratch?
-
Figure 2 is currently a bit too abstract and not illustrative enough, especially regarding how the memory bank and k-NN cross-attention retrieval work at each timestep. Could the authors redraw or expand Figure 2 to show this mechanism more clearly?
-
This might be a stretch, but have the authors considered using pretrained LLM-based or foundation embedding models to replace the learned g_theta module, potentially turning this into a completely training-free input adaptation method?
-
Minor wording question: In the abstract and main text, the authors use the term "AUCPR". Do you mean the more standard term "AUPRC" (Area Under the Precision-Recall Curve)?