- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors train a family of hybrid Gated DeltaNet/attention foundation models on MIMIC-III bedside waveforms with masked reconstruction. They fit a parametric law relating linear-probe ranking error (1 − AUROC) on eight ICU prediction tasks to model size, pretraining hours and labeled patients. From this law they derive compute-optimal allocations and estimates of label savings.
Critical strengths
This is one of the first attempts to model downstream clinical performance, rather than pretraining loss, as a joint function of model capacity, pretraining exposure and labeled-patient budget. The authors also test their predictions on held-out resource levels and a larger model, which is more than most scaling studies in this domain do.
Critical weaknesses
- The headline "pretraining and supervision reinforce each other" finding follows from the functional form rather than from the data. Equation 9 multiplies the capacity and pretraining terms by P^(-g). Any positive fit under this form will show that more labels increase the return to pretraining and that more pretraining reduces labeling needs. The law also has no irreducible error term, so predicted error tends to zero as resources grow. Given label noise in EHR-derived outcomes and the limited information in a one-hour waveform window, this is implausible, and it likely inflates the extrapolated savings.
- The exponent b = 0.050 is very small, and the allocation exponents depend on the ratio a/(a+b). Modest uncertainty in b would therefore change the recommended split substantially. No confidence intervals are reported for the fitted coefficients, the allocation exponents or the derived savings, even though a patient bootstrap is described. All model sizes also share one learning rate, batch size and very short warmup and decay phases.
- The 61% label saving is computed for the 720M model at 36.3M hours, but longer pretraining was only run for the 20M, 60M and 180M models. Labeled patients span four levels within one decade, which makes the claim that supervision is "the leading downstream scaling factor" difficult to separate from the chosen ranges. The held-out MAE of 0.5 to 1.0 percentage points is reported without the bootstrap uncertainty of the observed errors. Because the whole response range is only about 0.30 to 0.42, readers cannot judge whether the law predicts better than sampling noise. The nearest-configuration baseline is also weak.
- Most baselines are univariate forecasting models (TimesFM, Chronos-Bolt) or wearable-signal models. None were pretrained on 62-channel ICU bedside data at 125 Hz. The paper does not explain how multichannel hour-long inputs were adapted to models with short or single-channel contexts. Waveform foundation models closer to this setting, several of which are cited in the related work (ECG-FM, PaPaGei, BIOT, QualityFM), are not evaluated.
Questions for authors
See weaknesses above.
4 comments
Response 2/3
[Table R7] Excess ranking error of each model size over the optimum of every bootstrap refit at the same compute.
| Model size used at compute | FLOPs | FLOPs |
|---|---|---|
| Our (37M / 46M) | 0.07 [0.67] | 0.06 [0.64] |
| 720M | 0.99 [2.00] | 0.87 [1.82] |
| 2.1B | 1.58 [2.85] | 1.41 [2.62] |
<Hyperparameter settings.> Thanks for the comments. We have tried the learning-rate transfer methods commonly used for LLMs, i.e. μP (Yang et al., 2021) and depth-μP (Yang et al., 2024), and found that they do not improve the downstream performance at the same compute budget, so we keep one shared setting across model sizes. On the other hand, as the reviewer kindly noted, our work is “one of the first attempts to model downstream clinical performance”, and our established scaling law formula also provides a principled way to investigate the best hyperparameters in the future: specifically, by training a scaling ladder under each hyperparameter adaptation method, fitting our scaling law to each ladder, and comparing their predictions at the target scale.
[W3. Label-Saving Range, Held-Out Uncertainty and Nearest-Configuration Baseline]
Thanks for the detailed and constructive comments. We address the three points as follows.
<The 720M label saving.> The 61% saving for 720M at 36.3M hours is an extrapolation along , which we will mark as such together with its bootstrap CI ([27%, 82%]) in our revised paper. It is supported by the long runs at 20M, 60M and 180M (up to 36.3M hours), which the law predicts with an MAE of 1.03%, and the saving stays essentially unchanged under every alternative law form (60.9% to 65.7%, Table R5). Moreover, measured directly from the probes without any law, the label saving within the observed range is at least as large as the law predicts. As shown in Table R8, to match the error that the same FM reaches after 119K hours with all 4,071 patients, 720M after 4.78M hours needs 54.9% fewer patients (law: 45.7%), and 20M after 36.3M hours needs 71.8% fewer (law: 57.8%), so our law does not inflate the savings.
[Table R8] Labeled patients saved after long pretraining to match the error the same FM reaches after 119K hours with all 4,071 patients, measured vs. predicted by Eq. 2.
| Model | Pretraining hours | Measured saving | Law (Eq. 2) |
|---|---|---|---|
| 20M | 36.3M | 71.8% | 57.8% |
| 60M | 17.0M | 69.3% | 54.4% |
| 180M | 5.88M | 55.4% | 47.0% |
| 720M | 4.78M | 54.9% | 45.7% |
| 2.1B | 1.59M | 64.5% | 35.1% |
<Wider label range.> We further extend the label budgets to 1%, 3% and 5% of patients (41, 123 and 204 per task), so that labeled patients span two decades, and refit the probes for all 35 checkpoints. As shown in Table R9, labeled patients remain the steepest axis over the full range, and the law extrapolates to these unseen budgets with a bias of at most 0.011 in ranking error.
[Table R9] Eight-task mean AUROC over the 35 checkpoints by label budget, and the law's bias (predicted − observed ranking error).
| Label budget | Patients per task | AUROC | Law bias |
|---|---|---|---|
| 1% | 41 | 0.568 | +0.005 |
| 3% | 123 | 0.594 | +0.011 |
| 5% | 204 | 0.601 | +0.007 |
| 10% | 408 | 0.609 | −0.002 |
| 30% | 1,222 | 0.642 | +0.001 |
| 60% | 2,443 | 0.665 | +0.002 |
| 100% | 4,071 | 0.677 | −0.003 |
<Held-out MAE versus sampling noise.> To quantify the MAE caused by sampling noise alone, we simulate a perfect law: we take the fitted law's predictions as the ground truth, add the patient-bootstrap deviations of the measured ranking errors (1,999 draws), and rerun the held-out protocol of our paper on each simulated dataset. Even this perfect law attains an MAE of 0.64% [0.46%, 1.04%] at held-out resource levels. As shown in Table R10, our law's MAE (0.48%) lies at the low end of this range, and its MAE on the long runs (1.03%) also lies within the corresponding range [0.33%, 1.17%], i.e., the law predicts at the sampling-noise level.
<Stronger baselines.> We further compare against per-axis power laws, an additive Chinchilla-style law (Hoffmann et al., 2022) and log-linear interpolation. As shown in Table R10, our law matches the best interpolators at held-out levels (0.48% vs. 0.46% to 0.47%) and is the most accurate on the long runs (1.03% vs. 1.13% to 1.26%), while also providing the compute allocation and label savings.
[Table R10] Held-out and extrapolation MAE against the noise floor and stronger baselines.
| Method | Held-out levels | Long runs |
|---|---|---|
| Sampling-noise floor | 0.64% [0.46%, 1.04%] | 0.62% [0.33%, 1.17%] |
| Our law (Eq. 2) | 0.48% | 1.03% |
| Nearest configuration | 0.98% | 1.26% |
| Per-axis power laws | 0.47% | 1.21% |
| Additive Chinchilla-style | 0.64% | 1.15% |
| Log-linear interpolation | 0.46% | 1.13% |
Response 3/3
[W4. Baseline Adaptation and Closer Waveform FMs]
Thanks for the great suggestion. We address the two points as follows.
<Waveform FMs closer to this setting.> We further compare Aether with ECG-FM, AnyPPG, PaPaGei and BIOT, as well as the concatenation of ECG-FM and PaPaGei, the strongest ECG+PPG combination, under all bedside channels and matched input channels, using the same evaluation setup as in our paper. As shown in Table R11, with all bedside channels, Aether-720M (0.706) outperforms every ECG/PPG FM. Within each matched-channel group, Aether-720M also outperforms the specialized FMs. Moreover, adding the other bedside channels lifts Aether-720M from 0.666 to 0.706, which further confirms its capability of jointly modeling multiple channels. QualityFM has released neither code nor pretrained weights, but we will add it once its weights or code are released.
[Table R11] Eight-task mean AUROC at 100% labeled patients.
| Model | Input channels | AUROC |
|---|---|---|
| Aether-720M (ours) | All bedside channels | 0.706 |
| Aether-720M (ours) | ECG + PPG | 0.666 |
| ECG-FM + PaPaGei | ECG + PPG | 0.656 |
| BIOT | ECG + PPG | 0.574 |
| Aether-720M (ours) | ECG | 0.649 |
| ECG-FM | ECG | 0.639 |
| Aether-720M (ours) | PPG | 0.606 |
| AnyPPG | PPG | 0.599 |
| PaPaGei | PPG | 0.591 |
<Adaptation of multichannel one-hour inputs.> All baselines are frozen and produce one feature vector per one-hour window for the same linear probe. ECG-FM, AnyPPG and PaPaGei receive their trained channels, encode every non-overlapping segment of their pretraining length (5 s or 10 s), and average the segment embeddings. For the general-purpose and wearable FMs, whose context lengths are far shorter than a one-hour channel (450K samples at 125 Hz), each valid channel is resampled over the full hour to their maximal context length. The univariate models (TimesFM 2/2.5, Chronos-Bolt and MOMENT) encode each channel separately, while TimesFM 3, SOTER and NormWear 1/2 encode all channels jointly. We have added these details for every baseline in an appendix of our revised paper.
<General-purpose baselines.> To our knowledge, no public FM is pretrained on multichannel ICU bedside waveforms at this scale, which is part of the motivation for Aether. The original comparison therefore covers the public FMs that accept arbitrary channels and long inputs.
We have included all the revisions and additional experiments into our revised paper draft. We sincerely thank the reviewer again for the expert and constructive feedback, and we would be happy to address any further questions.
References
Hoffmann, J. et al. (2022) 'An empirical analysis of compute-optimal large language model training', Advances in Neural Information Processing Systems, 35, pp. 30016–30030.
Yang, G., Hu, E.J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W. and Gao, J. (2021) 'Tuning large neural networks via zero-shot hyperparameter transfer', Advances in Neural Information Processing Systems, 34, pp. 17084–17097.
Yang, G., Yu, D., Zhu, C. and Hayou, S. (2024) 'Tensor Programs VI: feature learning in infinite-depth neural networks', in International Conference on Learning Representations.
Thanks to the authors for their reply. Most of my questions have been addressed. I'll keep my score as it is already positive.
Response 1/3
We sincerely thank reviewer xzau for the expert and detailed review, and deeply appreciate the recognition of our work as “one of the first attempts to model downstream clinical performance”, and that testing predictions on held-out resource levels is “more than most scaling studies in this domain do”. The insightful comments helped us further strengthen the scaling law, and we provide pointwise responses below.
[W1. Interaction and Error Floor in the Functional Form]
Thanks for this insightful comment. We would like to clarify that the interaction in Eq. 2 is supported by the data rather than imposed by the functional form, and that an error floor does not change our conclusions.
<The interaction is supported by the data.> We further fit three alternative law forms on the same configurations. Writing Eq. 2 as R=1/(2+Pg/h) with h=AN−a+BD−b, the three forms are (i) separable additive: R=E+h+CP−c, the additive form of Hoffmann et al. (2022) extended with a label term, with no coupling between P and h; (ii) free interaction strength: R/(1−2R)=h1+δln(P/P0)P−g, where δ=0 recovers Eq. 2 and P0 is the geometric mean of the four label levels; and (iii) additive + interaction: R=E+h+u+ρhu with u=CP−c, where ρ=0 recovers the separable form. We compare the forms by the held-out MAE under the protocol of Figure 4d in our paper, which holds out all configurations at one value of N, D or P, refits on the rest and predicts them; we report it for held-out P and averaged over the three axes. As shown in Table R5, removing the coupling between P and the capacity and pretraining terms (separable additive form) nearly doubles the error at held-out P (1.01% vs. 0.56%), while adding an interaction term back to the separable form significantly improves the fit.
[Table R5] Alternative law forms fitted on the same 140 configurations.
<Law-free evidence.> Without any law, the raw measurements also show that the return to model size grows with labeled patients: scaling from 20M to 2.1B lowers the ranking error by 0.012 with 4,071 labeled patients, while it slightly raises the error with 408 patients (by 0.008), i.e., a 0.020 larger gain with more labels. For pretraining hours, extending pretraining from 119K to 4.78M hours lowers the ranking error by a similar relative amount at every P (4.2% to 7.7%), which is what Eq. 2 predicts, so the same target error is reached with fewer labeled patients.
<Error floor.> Unlike the pretraining loss, whose data entropy sets a floor, our fitting target is the downstream ranking error (1 − AUROC), which can possibly get very close to zero, i.e., AUROC close to 100%, given ideally unlimited waveform data, compute and labeled patients. We further add a floor term to Eq. 2, R=E0+(1−2E0)/(2+Pg/h), and the results show very close conclusions: the fitted floor E0 is not significantly different from zero (95% CI [0, 0.31]); the held-out MAE stays at 0.48%, and the 61% label saving becomes 62.0%. We agree that the floor term is a principled extension of the law, and since it leads to very similar conclusions as our paper, we will include it in our future work.
[W2. Coefficient Uncertainty and Hyperparameter Settings]
Thanks for the suggestion and the careful reading of our scaling law.
<Confidence intervals.> We further refit the law on 1,999 patient-bootstrap resamples and report CIs for the key quantities in Table R6. The label exponent is tightly estimated (g=0.296 [0.217, 0.382]), the 61% label saving stays substantial ([27%, 82%]), and the local elasticities of the error with respect to model size and pretraining hours are both positive. Although a and b individually trade off along a flat ridge of the fit, this flatness makes the recommended split robust: following our N⋆(C) costs only 0.06 to 0.07 points over each refit's own optimum, versus 0.9 to 1.6 points for 720M or 2.1B (as shown in Table R7). In future work, we will extend the scaling ladder to larger model scales to further sharpen a and b.
[Table R6] Patient-bootstrap CIs (1,999 refits) for the key quantities.