acceptodds
Under review as a conference paper at ICLR 2027

Training and Scaling Compute-Optimal Physiological Waveform Foundation Models

17 views

Abstract

We investigate the scaling laws and compute-optimal training of physiological waveform foundation models (FMs). We train Aether, a family of over one hundred FMs ranging from 20M to 2.1B parameters, on up to 36.3M hours of physiological waveforms. We construct eight clinical prediction tasks from MIMIC-III and evaluate the FMs through linear probing. The 720M FM outperforms all existing baseline FMs across all eight tasks. A scaling law of model size, pretraining hours, and labeled patients predicts downstream ranking error, *i.e.*, , effectively with prediction MAE at held-out resource scales and MAE when extrapolating to 2.1B parameters. We present three findings: (1) Compute-optimal training scales both FM size and pretraining hours. Under the fitted law, a increase in compute FLOPs scales model size by and pretraining hours by . (2) Larger FMs use waveform data more efficiently, and greater pretraining exposure increases the benefit of model scaling. Starting from 25M parameters and 4.8M pretraining hours, doubling FM size reduces the predicted hours needed for the same performance by . (3) Pretraining and clinical supervision reinforce each other: more labeled patients increase the return to pretraining, while larger FMs and longer pretraining reduce labeling requirements. For the example of the 720M FM, extending pretraining from 120K to 36.3M hours reduces the predicted patient requirement by at a target ranking error. These findings provide a quantitative training recipe and a promising and durable scaling path for physiological modeling and clinical prediction.

Primary area:
foundation or frontier models, including LLMs
Keywords:
scaling law, foundation models, pretraining, AI for healthcare

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_xzauOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors train a family of hybrid Gated DeltaNet/attention foundation models on MIMIC-III bedside waveforms with masked reconstruction. They fit a parametric law relating linear-probe ranking error (1 − AUROC) on eight ICU prediction tasks to model size, pretraining hours and labeled patients. From this law they derive compute-optimal allocations and estimates of label savings.

Critical strengths

This is one of the first attempts to model downstream clinical performance, rather than pretraining loss, as a joint function of model capacity, pretraining exposure and labeled-patient budget. The authors also test their predictions on held-out resource levels and a larger model, which is more than most scaling studies in this domain do.

Critical weaknesses
  1. The headline "pretraining and supervision reinforce each other" finding follows from the functional form rather than from the data. Equation 9 multiplies the capacity and pretraining terms by P^(-g). Any positive fit under this form will show that more labels increase the return to pretraining and that more pretraining reduces labeling needs. The law also has no irreducible error term, so predicted error tends to zero as resources grow. Given label noise in EHR-derived outcomes and the limited information in a one-hour waveform window, this is implausible, and it likely inflates the extrapolated savings.
  2. The exponent b = 0.050 is very small, and the allocation exponents depend on the ratio a/(a+b). Modest uncertainty in b would therefore change the recommended split substantially. No confidence intervals are reported for the fitted coefficients, the allocation exponents or the derived savings, even though a patient bootstrap is described. All model sizes also share one learning rate, batch size and very short warmup and decay phases.
  3. The 61% label saving is computed for the 720M model at 36.3M hours, but longer pretraining was only run for the 20M, 60M and 180M models. Labeled patients span four levels within one decade, which makes the claim that supervision is "the leading downstream scaling factor" difficult to separate from the chosen ranges. The held-out MAE of 0.5 to 1.0 percentage points is reported without the bootstrap uncertainty of the observed errors. Because the whole response range is only about 0.30 to 0.42, readers cannot judge whether the law predicts better than sampling noise. The nearest-configuration baseline is also weak.
  4. Most baselines are univariate forecasting models (TimesFM, Chronos-Bolt) or wearable-signal models. None were pretrained on 62-channel ICU bedside data at 125 Hz. The paper does not explain how multichannel hour-long inputs were adapted to models with short or single-channel contexts. Waveform foundation models closer to this setting, several of which are cited in the related work (ECG-FM, PaPaGei, BIOT, QualityFM), are not evaluated.
Questions for authors

See weaknesses above.

4 comments
AuthorsOpenReview ↗

Response 1/3

We sincerely thank reviewer xzau for the expert and detailed review, and deeply appreciate the recognition of our work as “one of the first attempts to model downstream clinical performance”, and that testing predictions on held-out resource levels is “more than most scaling studies in this domain do”. The insightful comments helped us further strengthen the scaling law, and we provide pointwise responses below.

[W1. Interaction and Error Floor in the Functional Form]

Thanks for this insightful comment. We would like to clarify that the interaction in Eq. 2 is supported by the data rather than imposed by the functional form, and that an error floor does not change our conclusions.

<The interaction is supported by the data.> We further fit three alternative law forms on the same configurations. Writing Eq. 2 as R=1/(2+Pg/h)R = 1/(2 + P^{g}/h) with h=AN−a+BD−bh = A N^{-a} + B D^{-b}, the three forms are (i) separable additive: R=E+h+CP−cR = E + h + C P^{-c}, the additive form of Hoffmann et al. (2022) extended with a label term, with no coupling between PP and hh; (ii) free interaction strength: R/(1−2R)=h1+δln⁡(P/P0)P−gR/(1 - 2R) = h^{1 + \delta \ln(P/P_0)} P^{-g}, where δ=0\delta = 0 recovers Eq. 2 and P0P_0 is the geometric mean of the four label levels; and (iii) additive + interaction: R=E+h+u+ρhuR = E + h + u + \rho h u with u=CP−cu = C P^{-c}, where ρ=0\rho = 0 recovers the separable form. We compare the forms by the held-out MAE under the protocol of Figure 4d in our paper, which holds out all configurations at one value of NN, DD or PP, refits on the rest and predicts them; we report it for held-out PP and averaged over the three axes. As shown in Table R5, removing the coupling between PP and the capacity and pretraining terms (separable additive form) nearly doubles the error at held-out PP (1.01% vs. 0.56%), while adding an interaction term back to the separable form significantly improves the fit.

[Table R5] Alternative law forms fitted on the same 140 configurations.

FormParamsHeld-out MAE (unseen PP)Held-out MAE (mean of NN, DD, PP)Patients saved, 720M10× compute: size / hours
Eq. 250.56%0.48%60.9%1.24× / 8.09×
Separable additive71.01%0.64%61.7%1.21× / 8.24×
Free interaction strength60.62%0.50%61.8%1.24× / 8.07×
Additive + interaction81.02%0.64%65.7%1.40× / 7.15×

<Law-free evidence.> Without any law, the raw measurements also show that the return to model size grows with labeled patients: scaling from 20M to 2.1B lowers the ranking error by 0.012 with 4,071 labeled patients, while it slightly raises the error with 408 patients (by 0.008), i.e., a 0.020 larger gain with more labels. For pretraining hours, extending pretraining from 119K to 4.78M hours lowers the ranking error by a similar relative amount at every PP (4.2% to 7.7%), which is what Eq. 2 predicts, so the same target error is reached with fewer labeled patients.

<Error floor.> Unlike the pretraining loss, whose data entropy sets a floor, our fitting target is the downstream ranking error (1 − AUROC), which can possibly get very close to zero, i.e., AUROC close to 100%, given ideally unlimited waveform data, compute and labeled patients. We further add a floor term to Eq. 2, R=E0+(1−2E0)/(2+Pg/h)R = E_0 + (1 - 2E_0)/(2 + P^{g}/h), and the results show very close conclusions: the fitted floor E0E_0 is not significantly different from zero (95% CI [0, 0.31]); the held-out MAE stays at 0.48%, and the 61% label saving becomes 62.0%. We agree that the floor term is a principled extension of the law, and since it leads to very similar conclusions as our paper, we will include it in our future work.

[W2. Coefficient Uncertainty and Hyperparameter Settings]

Thanks for the suggestion and the careful reading of our scaling law.

<Confidence intervals.> We further refit the law on 1,999 patient-bootstrap resamples and report CIs for the key quantities in Table R6. The label exponent is tightly estimated (g=0.296g = 0.296 [0.217, 0.382]), the 61% label saving stays substantial ([27%, 82%]), and the local elasticities of the error with respect to model size and pretraining hours are both positive. Although aa and bb individually trade off along a flat ridge of the fit, this flatness makes the recommended split robust: following our N⋆(C)N^{\star}(C) costs only 0.06 to 0.07 points over each refit's own optimum, versus 0.9 to 1.6 points for 720M or 2.1B (as shown in Table R7). In future work, we will extend the scaling ladder to larger model scales to further sharpen aa and bb.

[Table R6] Patient-bootstrap CIs (1,999 refits) for the key quantities.

QuantityPaperRefit95% CI
gg0.2960.296[0.217, 0.382]
Local elasticity in NN0.02550.0255[0.0006, 0.0507]
Local elasticity in DD0.04750.0472[0.0151, 0.0785]
Patients saved, 720M61%60.9%[27%, 82%]
AuthorsOpenReview ↗

Response 2/3

[Table R7] Excess ranking error of each model size over the optimum of every bootstrap refit at the same compute.

Model size used at compute CC101910^{19} FLOPs102010^{20} FLOPs
Our N⋆(C)N^{\star}(C) (37M / 46M)0.07 [0.67]0.06 [0.64]
720M0.99 [2.00]0.87 [1.82]
2.1B1.58 [2.85]1.41 [2.62]

<Hyperparameter settings.> Thanks for the comments. We have tried the learning-rate transfer methods commonly used for LLMs, i.e. μP (Yang et al., 2021) and depth-μP (Yang et al., 2024), and found that they do not improve the downstream performance at the same compute budget, so we keep one shared setting across model sizes. On the other hand, as the reviewer kindly noted, our work is “one of the first attempts to model downstream clinical performance”, and our established scaling law formula also provides a principled way to investigate the best hyperparameters in the future: specifically, by training a scaling ladder under each hyperparameter adaptation method, fitting our scaling law to each ladder, and comparing their predictions at the target scale.

[W3. Label-Saving Range, Held-Out Uncertainty and Nearest-Configuration Baseline]

Thanks for the detailed and constructive comments. We address the three points as follows.

<The 720M label saving.> The 61% saving for 720M at 36.3M hours is an extrapolation along DD, which we will mark as such together with its bootstrap CI ([27%, 82%]) in our revised paper. It is supported by the long runs at 20M, 60M and 180M (up to 36.3M hours), which the law predicts with an MAE of 1.03%, and the saving stays essentially unchanged under every alternative law form (60.9% to 65.7%, Table R5). Moreover, measured directly from the probes without any law, the label saving within the observed range is at least as large as the law predicts. As shown in Table R8, to match the error that the same FM reaches after 119K hours with all 4,071 patients, 720M after 4.78M hours needs 54.9% fewer patients (law: 45.7%), and 20M after 36.3M hours needs 71.8% fewer (law: 57.8%), so our law does not inflate the savings.

[Table R8] Labeled patients saved after long pretraining to match the error the same FM reaches after 119K hours with all 4,071 patients, measured vs. predicted by Eq. 2.

ModelPretraining hoursMeasured savingLaw (Eq. 2)
20M36.3M71.8%57.8%
60M17.0M69.3%54.4%
180M5.88M55.4%47.0%
720M4.78M54.9%45.7%
2.1B1.59M64.5%35.1%

<Wider label range.> We further extend the label budgets to 1%, 3% and 5% of patients (41, 123 and 204 per task), so that labeled patients span two decades, and refit the probes for all 35 checkpoints. As shown in Table R9, labeled patients remain the steepest axis over the full range, and the law extrapolates to these unseen budgets with a bias of at most 0.011 in ranking error.

[Table R9] Eight-task mean AUROC over the 35 checkpoints by label budget, and the law's bias (predicted − observed ranking error).

Label budgetPatients per taskAUROCLaw bias
1%410.568+0.005
3%1230.594+0.011
5%2040.601+0.007
10%4080.609−0.002
30%1,2220.642+0.001
60%2,4430.665+0.002
100%4,0710.677−0.003

<Held-out MAE versus sampling noise.> To quantify the MAE caused by sampling noise alone, we simulate a perfect law: we take the fitted law's predictions as the ground truth, add the patient-bootstrap deviations of the measured ranking errors (1,999 draws), and rerun the held-out protocol of our paper on each simulated dataset. Even this perfect law attains an MAE of 0.64% [0.46%, 1.04%] at held-out resource levels. As shown in Table R10, our law's MAE (0.48%) lies at the low end of this range, and its MAE on the long runs (1.03%) also lies within the corresponding range [0.33%, 1.17%], i.e., the law predicts at the sampling-noise level.

<Stronger baselines.> We further compare against per-axis power laws, an additive Chinchilla-style law (Hoffmann et al., 2022) and log-linear interpolation. As shown in Table R10, our law matches the best interpolators at held-out levels (0.48% vs. 0.46% to 0.47%) and is the most accurate on the long runs (1.03% vs. 1.13% to 1.26%), while also providing the compute allocation and label savings.

[Table R10] Held-out and extrapolation MAE against the noise floor and stronger baselines.

MethodHeld-out levelsLong runs
Sampling-noise floor0.64% [0.46%, 1.04%]0.62% [0.33%, 1.17%]
Our law (Eq. 2)0.48%1.03%
Nearest configuration0.98%1.26%
Per-axis power laws0.47%1.21%
Additive Chinchilla-style0.64%1.15%
Log-linear interpolation0.46%1.13%
AuthorsOpenReview ↗

Response 3/3

[W4. Baseline Adaptation and Closer Waveform FMs]

Thanks for the great suggestion. We address the two points as follows.

<Waveform FMs closer to this setting.> We further compare Aether with ECG-FM, AnyPPG, PaPaGei and BIOT, as well as the concatenation of ECG-FM and PaPaGei, the strongest ECG+PPG combination, under all bedside channels and matched input channels, using the same evaluation setup as in our paper. As shown in Table R11, with all bedside channels, Aether-720M (0.706) outperforms every ECG/PPG FM. Within each matched-channel group, Aether-720M also outperforms the specialized FMs. Moreover, adding the other bedside channels lifts Aether-720M from 0.666 to 0.706, which further confirms its capability of jointly modeling multiple channels. QualityFM has released neither code nor pretrained weights, but we will add it once its weights or code are released.

[Table R11] Eight-task mean AUROC at 100% labeled patients.

ModelInput channelsAUROC
Aether-720M (ours)All bedside channels0.706
Aether-720M (ours)ECG + PPG0.666
ECG-FM + PaPaGeiECG + PPG0.656
BIOTECG + PPG0.574
Aether-720M (ours)ECG0.649
ECG-FMECG0.639
Aether-720M (ours)PPG0.606
AnyPPGPPG0.599
PaPaGeiPPG0.591

<Adaptation of multichannel one-hour inputs.> All baselines are frozen and produce one feature vector per one-hour window for the same linear probe. ECG-FM, AnyPPG and PaPaGei receive their trained channels, encode every non-overlapping segment of their pretraining length (5 s or 10 s), and average the segment embeddings. For the general-purpose and wearable FMs, whose context lengths are far shorter than a one-hour channel (450K samples at 125 Hz), each valid channel is resampled over the full hour to their maximal context length. The univariate models (TimesFM 2/2.5, Chronos-Bolt and MOMENT) encode each channel separately, while TimesFM 3, SOTER and NormWear 1/2 encode all channels jointly. We have added these details for every baseline in an appendix of our revised paper.

<General-purpose baselines.> To our knowledge, no public FM is pretrained on multichannel ICU bedside waveforms at this scale, which is part of the motivation for Aether. The original comparison therefore covers the public FMs that accept arbitrary channels and long inputs.

We have included all the revisions and additional experiments into our revised paper draft. We sincerely thank the reviewer again for the expert and constructive feedback, and we would be happy to address any further questions.

References

Hoffmann, J. et al. (2022) 'An empirical analysis of compute-optimal large language model training', Advances in Neural Information Processing Systems, 35, pp. 30016–30030.

Yang, G., Hu, E.J., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W. and Gao, J. (2021) 'Tuning large neural networks via zero-shot hyperparameter transfer', Advances in Neural Information Processing Systems, 34, pp. 17084–17097.

Yang, G., Yu, D., Zhu, C. and Hayou, S. (2024) 'Tensor Programs VI: feature learning in infinite-depth neural networks', in International Conference on Learning Representations.

Thanks to the authors for their reply. Most of my questions have been addressed. I'll keep my score as it is already positive.

Reviewer_5v5mOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates scaling laws for physiological waveform foundation models. The authors train a family of Aether models ranging from 20M to 2.1B parameters and evaluate them on eight clinical prediction tasks. They propose a scaling law relating model size, pretraining exposure, and labeled patient count, and derive compute-optimal training allocations.

Critical strengths
  1. Extensive empirical study. The paper presents large-scale experiments across model sizes, pretraining durations, and labeled data budgets, providing useful insights into physiological foundation model scaling.

  2. Practical significance. The proposed scaling law offers useful guidance for allocating pretraining compute and potentially reducing downstream labeling requirements.

Critical weaknesses
  1. Data quality is insufficiently investigated. The paper primarily focuses on data quantity without systematically examining the influence of waveform quality, noise, and missing channels. It remains unclear whether the reported scaling trends hold across different data quality levels.

  2. Limited analysis of data diversity. All experiments rely on MIMIC-III waveforms. The paper does not investigate how differences in patient populations, recording devices, or data distributions affect the observed scaling behavior.

  3. Repeated data exposure versus unique data. The pretraining corpus contains approximately 398K hours of recordings, while larger training budgets involve repeated exposure to the same data. It is unclear whether increasing unique waveform data would produce similar scaling trends.

Questions for authors

Q1. Data quality: How is the quality of the pretraining waveform data assessed and controlled? Were noisy or corrupted recordings filtered out? Does the scaling behavior change with waveform quality?

Q2. Data diversity: Would the proposed scaling trends hold for data collected from different hospitals, patient populations, or recording devices? Have the authors examined the impact of data distribution shifts?

Q3. Repeated exposure versus unique data: Since larger pretraining budgets rely on repeated waveform exposure, how would the scaling behavior differ if additional unique patient recordings were available?

7 comments
AuthorsOpenReview ↗

Response 1/2

We sincerely thank reviewer 5v5m for the careful and encouraging review, and deeply appreciate the recognition of our “extensive empirical study” with “practical significance”, and that the reviewer has “no substantive concerns about the paper’s soundness”. We provide pointwise responses below.

[W1 & Q1. Waveform Data Quality]

Thanks for this question. We would like to first clarify the scope of our scaling law and how waveform quality is handled in our pipeline, and then provide additional experiments on how data quality affects scaling.

<Our scope.> We would like to clarify that our main contribution is to establish the functional form of the downstream scaling law for physiological waveform FMs, i.e., how ranking error jointly depends on model size, pretraining hours and labeled patients (Eq. 2 in our paper), instead of a universal set of coefficients. Similar to scaling laws for language modeling, where the fitted coefficients and the compute-optimal allocation shift with model architecture (Tay et al., 2023) and with the quality and distribution of the pretraining data (Sorscher et al., 2022; Bi et al., 2024), we hypothesize that the coefficients of our law also vary with the model architecture and the data distribution (quality, diversity, etc.), while the functional form remains.

<Quality control in our paper.> We primarily focus on excluding abnormal data while preserving the natural signals as much as possible, since, unlike language data (e.g., curated v.s. web text), physiological waveforms have not much clear notion of high- versus low-quality content, and noise and artifacts are inherent to real bedside monitoring. Specifically, our reconstruction loss averages only over observed values (Eq. 1 in our paper), so missing channels and gaps will not be included in loss calculation. During pretraining, a channel is excluded when it is observed for less than 10% of the hour, contains non-finite values, or is an abnormal flat line. We apply no further artifact filtering, so that the corpus keeps the noise found at the bedside.

<Additional experiments of pretraining on higher- v.s. lower-quality data.> We further split the pretraining corpus into a higher-quality and a lower-quality half of equal size (about 199K hours each) with pre-defined quality metrics, and pretrain Aether models on each half, compared with a random half of the same size. We define the quality of each one-hour window by a composite index of four properties: the number of observed channels, completeness, and the ECG and PPG signal quality indices (SQIs), each measured as the mean correlation of individual beats with the average beat template over twelve 5-minute segments. Compared with the lower-quality half, the higher-quality half has more observed channels (3.9 vs. 1.5 on average), higher completeness (0.92 vs. 0.47) and many more 5-minute segments with a valid ECG (10.9 vs. 0.5 out of 12). As shown in Table R4, the higher-quality half matches the random half, while pretraining only on the lower-quality half reduces AUROC by up to 0.018 (20M at 398K hours), which seems to indicate that quality matters mainly through the low-quality tail. We will keep in-depth investigation of waveform data quality as future research questions.

[Table R4] Pretraining on quality halves.

ModelD (hours)High − lowLow − randomHigh − random
20M398K+0.026−0.018+0.007
60M398K+0.013−0.006+0.007
180M398K+0.006−0.008−0.002
20M1.59M+0.002+0.003+0.005
60M1.59M+0.013−0.009+0.004

Thanks for raising this question. More in-depth study of how the coefficients and the compute-optimal allocation shift with data quality, along with how to define the data quality of waveform, is an interesting research direction, which we will keep as future work. We have also added this analysis and discussion to our revised paper.

[W2 & Q2. Data Diversity beyond MIMIC-III]

Thanks for raising this insightful point. We would like to clarify that our study measures scaling within one source by design, which isolates model size, pretraining hours and labeled patients from distribution shift. MIMIC-III is the largest public bedside waveform resource with linked clinical outcomes, which the eight downstream tasks require.

AuthorsOpenReview ↗

Response 2/2

[W3 & Q3. Unique Data versus Repeated Exposure]

Thanks for the question. To establish the value of unique data, we further run experiments by varying corpus size and repetition independently at fixed exposure.

Specifically, we draw nested subsets of 50K, 100K and 199K unique hours from our corpus, and pretrain Aether models on each subset with the paper's recipe at the same exposure as the full-corpus runs (D = 398K and 1.59M hours, each with its own decay). This varies the repeat ratio from 2× to 32× at fixed D, at 20M, 60M and 180M parameters (18 new pretraining runs), all probed with the identical protocol.

As shown by the results in Table R3, at every model size and exposure, the model trained on fewer unique hours reaches the same downstream AUROC as the full-corpus model at the same exposure. All differences stay within ±0.01 AUROC, even at 32× repetition. Within the studied range, a repeated hour is worth as much as a new hour for downstream ranking error, which supports exposure D as the resource variable of the law. Our hypothesis is that physiological waveforms, which are highly redundant and quasi-periodic, require many more epochs to saturate than text, for which repeated data loses its value after about 4 epochs (Muennighoff et al., 2023). This is consistent with masked autoencoding on redundant image and video data, which keeps benefiting from hundreds to thousands of pretraining epochs (He et al., 2022; Tong et al., 2022). Therefore, if additional unique recordings were available, our results indicate that the scaling in exposure D would follow the same law, while reaching larger D at lower repetition.

[Table R3] Unique v.s. repeated data at fixed exposure D: AUROC difference to the full-corpus model of the same size and exposure D, mean over the four downstream label budgets.

ModelD (hours)Unique hoursRepeats EΔAUROC vs. full corpus (Aether)
20M398K50K8+0.005
20M398K100K4+0.000
20M398K199K2+0.005
20M1.59M50K32+0.002
20M1.59M100K16+0.004
20M1.59M199K8+0.001
60M398K50K8−0.008
60M398K100K4+0.001
60M398K199K2−0.002
60M1.59M50K32−0.004
60M1.59M100K16+0.004
60M1.59M199K8−0.002
180M398K50K8+0.002
180M398K100K4+0.005
180M398K199K2+0.001
180M1.59M50K32+0.001
180M1.59M100K16+0.008
180M1.59M199K8+0.004

We have included all the revisions and additional experiments into our revised paper draft. We sincerely thank the reviewer again for the encouraging feedback, and we would be happy to address any further questions.

References

Bi, X. et al. (2024) 'DeepSeek LLM: scaling open-source language models with longtermism', arXiv preprint arXiv:2401.02954.

He, K., Chen, X., Xie, S., Li, Y., Dollár, P. and Girshick, R. (2022) 'Masked autoencoders are scalable vision learners', in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16000–16009.

Muennighoff, N., Rush, A.M., Barak, B., Le Scao, T., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T. and Raffel, C. (2023) 'Scaling data-constrained language models', Advances in Neural Information Processing Systems, 36, pp. 50358–50376.

Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S. and Morcos, A.S. (2022) 'Beyond neural scaling laws: beating power law scaling via data pruning', Advances in Neural Information Processing Systems, 35, pp. 19523–19536.

Tay, Y., Dehghani, M., Abnar, S., Chung, H.W., Fedus, W., Rao, J., Narang, S., Tran, V.Q., Yogatama, D. and Metzler, D. (2023) 'Scaling laws vs model architectures: how does inductive bias influence scaling?', in Findings of the Association for Computational Linguistics: EMNLP 2023. Singapore: Association for Computational Linguistics, pp. 12342–12364.

Tong, Z., Song, Y., Wang, J. and Wang, L. (2022) 'VideoMAE: masked autoencoders are data-efficient learners for self-supervised video pre-training', Advances in Neural Information Processing Systems, 35, pp. 10078–10093.

AuthorsOpenReview ↗

We are keen to further discuss with you

Dear Reviewer 5v5m,

We thank reviewer 5v5m for the time of reviewing and the constructive comments again. We really hope to discuss further with you to see if our response solves your concerns.

As AC kindly reminded, we genuinely hope reviewer 5v5m could kindly check our response. Thank you so much!

Warmest regards,

Authors

Reviewer_cfR1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Aether, a family of physiological waveform foundation models ranging from 20M to 2.1B parameters. Using eight clinical tasks, it fits downstream ranking error as a function of model size, pretraining exposure, and labeled-patient count, then derives compute allocations and predicted data and label savings.

Critical strengths
  1. The study systematically varies three important resource dimensions and evaluates predictions on held-out model sizes and longer training runs. Lots of experiments provide meaningful evidence beyond interpolation within the fitting grid.

  2. The downstream protocol includes patient-disjoint pretraining and evaluation, validation-selected linear probes, patient-balanced AUROC, and patient-level bootstrapping. These controls strengthen the reported clinical comparisons.

Critical weaknesses
  1. These are no ECG/PPG foundation model comparisons. Relevant models such as ECG-FM and AnyPPG are omitted without clear justification. Comparisons with matched input modalities would clarify whether Aether’s gains hold against these specialized models.

  2. These are also no conventional supervised baselines. The paper lacks comparisons using pyPPG-derived PPG features or NeuroKit2-derived ECG/HRV features with classifiers such as logistic regression or XGBoost. Their performance under the same patient splits and label budgets would clarify the gains over simpler approaches.

  3. The resource variable combines unique data with repeated training. The 398,204-hour corpus is reused up to approximately 91 times. Without independently varying corpus size and repetition, the law characterizes training exposure but does not establish the value of acquiring additional waveform data.

  4. The fitting grid requires clarification. Four model sizes, six exposure levels, and four patient levels imply 96 configurations, whereas Appendix B reports 140; Figure 6 reports 57 model–exposure configurations per patient level. A complete run manifest is necessary to audit the fitting and held-out evaluations.

  5. Figure 5(d) contains a discrepancy between its displayed points and reported MAE, alongside metadata describing a horizontal shift of 0.007. The original measurements and plotting code are needed to establish which values support the extrapolation results.

Questions for authors
  1. Does the predicted compute-optimal allocation remain stable under measured training FLOPs, alternative scaling-law forms, and uncertainty in the fitted coefficients?
  2. Can the authors provide the complete run manifest and numerical plotting data, reconciling the configuration counts and Figure 5(d) discrepancy?
8 comments
AuthorsOpenReview ↗

Response 1/2

Thanks to reviewer cfR1 for the questions and for recognizing that our study “systematically varies three important resource dimensions” with controls that “strengthen the reported clinical comparisons”. To address reviewer cfR1's concerns, we provide pointwise responses below.

[W1. ECG/PPG Foundation Model Baselines]

Thanks for the suggestion. We would like to make the following clarifications and provide additional comparisons with ECG/PPG FMs:

  • <Multichannel pretraining.> We would like to clarify that one of the key strengths of our pretraining is to leverage multiple bedside channels so that each channel improves the understanding of the others. As described in Section 3.1 and Figure 2(b) of our paper, every Aether token jointly projects a 200 ms patch of all 62 channels, so the FM learns the dynamics of all channels together rather than of each modality in isolation. Modality-specific FMs such as ECG-FM, AnyPPG and PaPaGei are designed for one modality and cannot use the other bedside channels.
  • <The core contribution.> One of the core contributions of this work is to demonstrate the scaling law of downstream clinical performance as a joint function of FM size, pretraining hours and labeled patients, together with the compute-optimal allocations and label savings derived from it (contributions (1) to (3) in Section 1 of our paper). As recognized by reviewer xzau, our work is “one of the first attempts to model downstream clinical performance, rather than pretraining loss, as a joint function of model capacity, pretraining exposure and labeled-patient budget”. The FM comparison in Section 4.1 confirms that the models whose scaling we study are strong waveform FMs.
  • <All-channel and matched-channel comparisons.> We further conduct experiments under all channels and matched channels to compare Aether with ECG-FM, AnyPPG, PaPaGei and BIOT, as well as the concatenation of ECG-FM and PaPaGei, the strongest ECG+PPG combination. We use the same evaluation setup as in our paper for fair comparison. Results are shown in Table R1. With all bedside channels, Aether-720M (0.706) outperforms every ECG/PPG FM. Within each matched-channel group, Aether-720M also outperforms the specialized FMs: 0.666 vs. 0.656 (ECG-FM + PaPaGei) and 0.574 (BIOT) on ECG+PPG, 0.649 vs. 0.639 (ECG-FM) on ECG, and 0.606 vs. 0.599 (AnyPPG) and 0.591 (PaPaGei) on PPG. Moreover, adding the other bedside channels lifts Aether-720M from 0.666 to 0.706, which further confirms its capability of jointly modeling multiple channels.

[Table R1] Eight-task mean AUROC at 100% labeled patients.

ModelInput channelsAUROC
Aether-720M (ours)All bedside channels0.706
Aether-720M (ours)ECG + PPG0.666
ECG-FM + PaPaGeiECG + PPG0.656
BIOTECG + PPG0.574
Aether-720M (ours)ECG0.649
ECG-FMECG0.639
Aether-720M (ours)PPG0.606
AnyPPGPPG0.599
PaPaGeiPPG0.591

[W2. Conventional Supervised Feature Baselines]

Thanks for the suggestion, we additionally compare against conventional supervised pipelines on hand-engineered physiological features. For every one-hour window, we extract NeuroKit2 ECG and HRV features and pyPPG PPG features, and train L2-regularized logistic regression and XGBoost on the same patient splits at all four label budgets (10/30/60/100%). Other settings are kept consistent with the paper for fair comparison. As shown in Table R2, Aether-720M outperforms the strongest feature pipeline (ECG+PPG features with XGBoost) at every label budget.

[Table R2] Comparison with baselines with eight-task mean AUROC by label budget.

Model10%30%60%100%
Aether-720M0.6250.6670.6820.706
ECG+PPG XGBoost0.5930.6550.6620.673
ECG+PPG LR0.5800.6340.6470.657
ECG LR0.5920.6190.6320.636
ECG XGBoost0.5870.6310.6510.642
PPG XGBoost0.5670.5910.6030.621
PPG LR0.5570.5980.5930.604
AuthorsOpenReview ↗

Response 2/2

[W3. Unique Data versus Repeated Exposure]

Following your suggestions, to establish the value of unique data, we further run experiments by varying corpus size and repetition independently at fixed exposure.

Specifically, we draw nested subsets of 50K, 100K and 199K unique hours from our corpus, and pretrain Aether models on each subset with the paper's recipe at the same exposure as the full-corpus runs (D = 398K and 1.59M hours, each with its own decay). This varies the repeat ratio from 2× to 32× at fixed D, at 20M, 60M and 180M parameters (18 new pretraining runs), all probed with the identical protocol.

As shown by the results in Table R3, at every model size and exposure, the model trained on fewer unique hours reaches the same downstream AUROC as the full-corpus model at the same exposure. All differences stay within ±0.01 AUROC, even at 32× repetition. Within the studied range, a repeated hour is worth as much as a new hour for downstream ranking error, which supports exposure D as the resource variable of the law. Our hypothesis is that physiological waveforms, which are highly redundant and quasi-periodic, require many more epochs to saturate than text, for which repeated data loses its value after about 4 epochs (Muennighoff et al., 2023). This is consistent with masked autoencoding on redundant image and video data, which keeps benefiting from hundreds to thousands of pretraining epochs (He et al., 2022; Tong et al., 2022).

[Table R3] Unique v.s. repeated data at fixed exposure D.

ModelD (hours)Unique hoursRepeats EΔAUROC vs. full corpus (Aether)
20M398K50K8+0.005
20M398K100K4+0.000
20M398K199K2+0.005
20M1.59M50K32+0.002
20M1.59M100K16+0.004
20M1.59M199K8+0.001
60M398K50K8−0.008
60M398K100K4+0.001
60M398K199K2−0.002
60M1.59M50K32−0.004
60M1.59M100K16+0.004
60M1.59M199K8−0.002
180M398K50K8+0.002
180M398K100K4+0.005
180M398K199K2+0.001
180M1.59M50K32+0.001
180M1.59M100K16+0.008
180M1.59M199K8+0.004

[W4. Fitting Grid and Run Manifest]

Thanks for the question. The three numbers count runs with different purposes: (1) The law is fitted on K = 140 configurations, i.e., 35 (N, D) pairs × 4 patient levels, which consist of all combinations of 4 model sizes and 6 exposure levels (24 pairs, i.e., the 96 configurations) plus 11 additional pairs at intermediate exposure levels that sample the pretraining-hours axis more densely. (2) The 5 checkpoints at 2.1B and the long runs at 20M, 60M and 180M are never used for fitting and serve as the held-out extrapolations in Section 4.3, and the 14 iso-FLOPs runs are trained only for the iso-FLOPs analysis. (3) Figure 6 compares the compute-optimal frontier with the trained models, so besides the 35 fitted pairs it also shows the 2.1B checkpoints, the iso-FLOPs runs, 2 of the long runs and the 720M checkpoint at 1.99M hours, giving 57 markers per panel.

We have added this clarification to Appendix B of our revised paper.

[W5. Figure 5(d) Displayed Points]

Thanks for the reminder. The displayed points in Figure 5(d) are typos that come from a preview version of the figure that was included by mistake when compiling the paper. We have corrected Figure 5(d) in our revised paper, and provide the per-configuration observed and predicted values together with the plotting code at https://anonymous.4open.science/r/iclr2027-28750-rebuttal-data/

[Q1. Stability of the Compute-Optimal Allocation]

Yes, they remain stable.

<Measured training FLOPs.> We further refit the scaling law by replacing the approximation C = 6qND with measured training FLOPs, and the results show a very close size exponent of 0.093, compared with 0.088 under C = 6qND (Section 5 in our paper).

<Alternative law forms.> We further fit four alternative law forms on the same configurations (adding an error floor, a separable additive form and two forms with a free interaction), and the size exponents remain close, ranging from 0.084 to 0.146.

<Coefficient uncertainty.> We refit the law on patient-bootstrap resamples, and the resulting compute-optimal allocation closely matches our paper. At 1e19, 1e20 and 1e21 FLOPs, the bootstrap median of the optimal FM size is 38M, 45M and 55M (vs. 37M, 46M and 56M in our paper), and that of the optimal pretraining hours is 2.4M, 20.5M and 169M (vs. 2.5M, 20.3M and 165M). The median size exponent is 0.097, very close to 0.088 in our paper.

[Q2. Run Manifest and Plotting Data]

Please see our responses to W4 and W5.

We have included all the revisions and additional experiments into our revised paper draft.

AuthorsOpenReview ↗

We are keen to further discuss with you

Dear Reviewer cfR1,

We genuinely thank you again for your time and efforts and your constructive comments.

In our response, we have (1) conducted additional comparisons with ECG/PPG FMs under all and matched input channels; (2) conducted additional comparisons with hand-engineered feature baselines; (3) conducted additional experiments on unique vs. repeated data at fixed exposure; (4) verified the stability of the compute-optimal allocation under measured FLOPs, alternative law forms and bootstrap refits; (5) clarified the fitting grid, run manifest and Figure 5(d).

As the AC kindly reminded, that the discussion period is approaching its end, we would really appreciate it if you could kindly let us know whether there are any further questions. We will be more than happy to address them.

Best wishes,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.