acceptodds
Under review as a conference paper at ICLR 2027

Pretraining Strategies and Scaling for ECG Foundation Models: A Systematic Study

Abstract

Specialized foundation models are beginning to emerge in various medical subdomains, but pretraining methodologies and parametric scaling with the size of the pretraining dataset are rarely assessed systematically and in a like-for-like manner. This work focuses on foundation models for electrocardiography (ECG) data, one of the most widely captured physiological time series world-wide. We present a comprehensive assessment of pretraining methodologies, covering five different contrastive and non-contrastive self-supervised learning objectives for ECG foundation models, and investigate their scaling behavior with pretraining dataset sizes up to 11M input samples, exclusively from publicly available sources. Pretraining strategy has a meaningful and consistent impact on downstream performance, with contrastive predictive coding (slightly ahead of JEPA) yielding the strongest overall downstream performance across diverse clinical tasks. Scaling pretraining data continues to yield meaningful improvements up to 11M samples for most objectives and translates into improved downstream performance. We also compare model architectures across all pretraining methodologies and find evidence for a clear superiority of structured state space sequence (S4) models compared to transformers and CNN models. We hypothesize that the strong inductive biases of S4 models, rather than pretraining scale alone, are the primary driver of effective ECG representation learning, with important implications for future foundation model development in this and potentially other physiological signal domains.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
ECG, foundation model, scaling analysis, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9HqEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper systematically studies the effects of pretraining objectives, model architectures, and data scaling on ECG foundation models. The authors compare five self-supervised objectives across multiple architectures and investigate scaling with up to 11M samples. Experiments show that S4 and CPC generally achieve the strongest downstream performance, while CPC and JEPA exhibit the clearest pretraining loss scaling.

Critical strengths

The paper provides a systematic comparison of pretraining objectives and model architectures under controlled settings, addressing an important gap in existing ECG foundation model research. It provides useful empirical insights for ECG foundation model development.

Critical weaknesses

The abstract suggests that scaling data up to 11M samples leads to improved downstream performance, while Sec. 4 reports consistent improvements mainly on PTB-XL and mixed results on other datasets. The additional gains from scaling from 753K to 11M appear limited or inconsistent across tasks, making the benefits of further scaling less clear.

Questions for authors

Could the authors further clarify the downstream benefits of scaling from 753K to 11M samples, given the mixed improvements across tasks?

3 comments

We thank the reviewer for this helpful comments and for the positive assessment of our work. We address W1 and Q1 jointly, as both concern the downstream benefit of scaling from 753K to 11M samples.

1. W1 + Q1 (Downstream benefit of scaling from 753K to 11M)

To make this explicit, we report finetuning performance (macro-AUROC) for all five objectives at 753K and 11M samples, together with the gain, using identical training configurations (4-layer S4, 10 epochs).

CPC

Dataset753K11MGain
Ningbo0.9740.9740.000
CPSC20180.9720.967-0.005
CPSC-Extra0.9060.897-0.009
Georgia0.9120.920+0.008
Chapman0.9590.963+0.004
SPH0.9820.9820.000
PTB-XL (all)0.9430.945+0.002
PTB-XL (sub)0.9370.941+0.004
PTB-XL (super)0.9320.934+0.002
ZZU pECG0.8990.896-0.003
EchoNext0.8320.8320.000

JEPA

Dataset753K11MGain
Ningbo0.9710.970-0.001
CPSC20180.9700.971+0.001
CPSC-Extra0.8880.8880.000
Georgia0.9130.921+0.008
Chapman0.9570.964+0.007
SPH0.9760.974-0.002
PTB-XL (all)0.9430.938-0.005
PTB-XL (sub)0.9240.939+0.015
PTB-XL (super)0.9320.934+0.002
ZZU pECG0.8910.895+0.004
EchoNext0.8260.830+0.004

DinoSR

Dataset753K11MGain
Ningbo0.9640.968+0.004
CPSC20180.9680.962-0.006
CPSC-Extra0.8830.884+0.001
Georgia0.9060.922+0.016
Chapman0.9600.959-0.001
SPH0.9750.981+0.006
PTB-XL (all)0.9350.943+0.008
PTB-XL (sub)0.9380.948+0.010
PTB-XL (super)0.9320.937+0.005
ZZU pECG0.8880.891+0.003
EchoNext0.8290.834+0.005

HuBERT++

Dataset753K11MGain
Ningbo0.9700.969-0.001
CPSC20180.9700.962-0.008
CPSC-Extra0.8910.892+0.001
Georgia0.9150.911-0.004
Chapman0.9580.962+0.004
SPH0.9820.984+0.002
PTB-XL (all)0.9390.9390.000
PTB-XL (sub)0.9400.942+0.002
PTB-XL (super)0.9300.936+0.006
ZZU pECG0.8960.892-0.004
EchoNext0.8300.831+0.001

data2vec

Dataset753K11MGain
Ningbo0.9650.9650.000
CPSC20180.9600.953-0.007
CPSC-Extra0.8390.873+0.034
Georgia0.8940.8940.000
Chapman0.9460.945-0.001
SPH0.9760.974-0.002
PTB-XL (all)0.9240.929+0.005
PTB-XL (sub)0.9340.936+0.002
PTB-XL (super)0.9260.929+0.003
ZZU pECG0.8680.875+0.007
EchoNext0.8160.824+0.008

We agree that the gains from 753K to 11M under full finetuning are modest. We would, however, like to offer a complementary perspective. First, this is the behavior expected from a power law with a residual error floor. In this regime, each additional order of magnitude of data yields smaller absolute improvements, and the purpose of a scaling analysis is precisely to quantify these diminishing returns rather than to assume linear gains. Second, the clearest evidence for scaling lies in the pretraining loss, which decreases consistently with dataset size for CPC and JEPA, and in its strong rank correlation with downstream error for CPC, JEPA and DinoSR on all three investigated datasets. Lower pretraining loss therefore translates into better downstream performance, even though the absolute gains under full finetuning become small at large scale.

We will revise the abstract and Section 4 to state more precisely that scaling yields consistent improvements in pretraining loss and its correlation with downstream error, with diminishing downstream gains under full finetuning between 753K and 11M samples. The tables above will be added to the appendix.

Thank you for the detailed response. The proposed revisions would address my main concern by more clearly distinguishing pre-training loss scaling from downstream performance improvements. I will therefore maintain my current score.

We thank the reviewer for the positive response. We have uploaded the revised manuscript, in which all changes are highlighted in blue.

  • Abstract. We revised the scaling statement to make explicit that downstream gains under full finetuning diminish at large scale.

  • Section 4 ("Scaling with dataset size"). We added that downstream gains from 753K to 11M samples are modest and task-dependent, as expected for a power law with a residual error floor, where each additional order of magnitude of data yields smaller absolute improvements.

  • Appendix. The 753K vs. 11M comparison tables for all five objectives are included in the Tables 40-44.

Reviewer_ayyWOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper runs systematic experiments around five pretraining objectives. The main comparisons use a CNN stem plus a four-layer S4 backbone and pretrain on about 11 million 12-lead ECG records. Downstream evaluation covers 10 public datasets, 26 tasks, and 7 clinical task categories, using three strategies: full fine-tuning, freezing the encoder with query attention pooling, and freezing the encoder with linear probing. The authors also design experiments on backbone comparison, scaling up pretraining data, cross-dataset transfer, and downstream adaptation strategies to further analyze what factors affect model performance.

Critical strengths

The paper systematically compares five SSL pretraining objectives across multiple downstream evaluation strategies and seven clinical task categories, and further examines different backbones, pretraining data scales, cross-dataset transfer capability, and downstream adaptation strategies. This kind of multi-dimensional, thorough experimental design helps provide a more comprehensive understanding of how different factors affect the performance of ECG foundation models.

Critical weaknesses

1)This paper offers no new methodological ideas. The authors' justification for the rationality of each experimental design mainly focuses on "why this comparison is fair" rather than "why this choice was made." In other words, the experimental design lacks sufficient rationale. 2) The paper states that "All models share a common encoder." What is the rationale for the design of this encoder? Would these conclusions still hold if a different encoder were used? 3)The scope of some experimental conclusions is still limited. For example, the cross-dataset transfer experiment fine-tunes on PTB-XL and then directly tests on external datasets, while the downstream adaptation strategy experiment mainly compares CPC and HuBERT++. Therefore, although these results provide valuable empirical findings, they are not yet sufficient to show that the corresponding conclusions hold for all pretraining objectives, data distributions, and downstream tasks. 4) There is a lack of in-depth analysis of why different pretraining objectives perform differently. The paper compares five self-supervised pretraining objectives and finds that CPC is quite competitive on multiple downstream tasks, but it mainly focuses on reporting experimental results, and the explanation of why different objectives perform differently under different evaluation strategies is still insufficient. For example, CPC performs well under frozen-encoder evaluation but shows a clear performance drop under linear evaluation. The authors link this phenomenon to the possibility that CPC's local prediction objective may not sufficiently optimize global representations, but there is still a lack of targeted experimental validation of this explanation, and there is also insufficient analysis of the specific differences in the representations learned by different pretraining objectives in terms of semantic information, temporal dependencies, and global separability.

Questions for authors
  1. Would the paper conclusions still hold if a different shared encoder were used? Would different experimental designs lead to conclusions that differ from those in this paper?

  2. CPC uses a causal backbone, whereas the other pretraining objectives use a non-causal backbone. How do the authors rule out the influence of the backbone's information modeling approach on the comparison results across different pretraining objectives?

  3. The paper evaluates cross-dataset transfer capability using clinically equivalent label mappings and mentions that generative AI was used to assist in drafting the mappings. Did the authors clinically validate these mappings?

3 comments

We thank the reviewer for the thoughtful feedback. New results will be added to the revised manuscript.

1. W1 (Rationale for design choices)

Our contribution is empirical by design. We thank the reviewer for the valuable suggestion to stress the rationale behind the design choices taken. We will add a paragraph to the main text explaining why each choice was made. The five objectives cover the main SSL families (contrastive, latent prediction, masked regression, masked clustering). S4, Transformer and Net1D each represent an architecture class used for ECG, with S4 motivated by [1]. The CNN stem plus sequence backbone follows speech SSL (wav2vec, HuBERT), from which most objectives originate. 240 Hz is the lowest observed sampling frequency observed on HEEDB and therefore the largest possible sampling frequency. 2.5 s windows follows prior work [2] and is also supported by our input-length ablation. The scales are log-spaced to fit scaling curves, up to the largest public ECG data. The three evaluation modes reflect how FMs are used in practice.

2. W2 + Q1 (Shared encoder)

Several objectives require a local encoder, e.g. CPC and masking operate on local tokens. A CNN stem is the natural, standard choice in audio SSL (wav2vec, HuBERT). To assess robustness, we compared a narrow stem (2 layers, kernel sizes [3,1], strides [2,1]) with the wide stem used in the paper (4 layers, kernel sizes [3,1,1,1], strides [2,1,1,1]) (macro-AUROC; bold = not significantly worse, bold italic = best).

DatasetJEPA NarrowJEPA WideCPC NarrowCPC Wide
Ningbo0.9650.9610.9600.955
CPSC20180.9690.9660.9640.964
CPSC-Extra0.8780.8680.8870.884
Georgia0.9020.9080.9020.900
Chapman0.9500.9480.9500.944
SPH0.9700.9680.9670.972
PTB-XL (all)0.9310.9330.9210.921
PTB-XL (sub)0.9270.9270.9280.925
PTB-XL (super)0.9290.9240.9270.924
ZZU pECG0.8820.8830.8810.886
EchoNext0.8260.8230.8250.826

The differences are mostly not significant, with a slight preference for the narrow stem, indicating robustness to the stem.

3. W3 (Continual Pretraining)

We will extend the continual pretraining comparison to data2vec, DinoSR and JEPA and state the scope of each conclusion more explicitly.

4. W4 (Why objectives differ)

We will add (i) linear probing on the last token of CPC, which summarizes the full causal context, to directly test whether its linear-evaluation drop stems from mean pooling of local representations (global separability), and (ii) layer-wise linear probing for S4 and Transformer across all objectives on PTB-XL (all), reported separately for form, rhythm and diagnostic statements (semantic information), where the rhythm statements specifically reflect temporal dependencies. A full mechanistic analysis is beyond scope and will be made explicit in Limitations.

5. Q2 (Causal vs. non-causal backbone)

A causal backbone has a restricted model capacity compared to a non-causal one and therefore has a disadvantage. In supervised training and for JEPA, the causal S4 is never significantly better than the non-causal one and significantly worse on several tasks.

DatasetSup. CausalSup. Non-causalJEPA CausalJEPA Non-causal
Ningbo0.9700.9720.9500.961
CPSC20180.9570.9620.9640.966
CPSC-Extra0.8500.8520.8680.868
Georgia0.9050.9030.8940.908
Chapman0.9570.9630.9440.948
SPH0.9770.9810.9620.968
PTB-XL (all)0.9370.9410.9250.933
PTB-XL (sub)0.9340.9380.9300.927
PTB-XL (super)0.9310.9320.9230.924
ZZU pECG0.8850.8970.8780.883
EchoNext0.8230.8190.8250.823

Restricting other objectives to a causal backbone would only weaken them; CPC performs best despite this handicap.

6. Q3 (Label mappings)

Most mappings are direct matches of the same diagnosis. All mappings were verified manually against the clinical label descriptions and the SNOMED-CT mappings provided with the PhysioNet/CinC Challenge 2021 [3]. We will add PTB-XL label descriptions to the mapping table and explicitly highlight conditions where the mapped conditions are not fully equivalent.

References

[1] Al-Masud et al., ICLR 2026.

[2] Mehari & Strodthoff, IEEE JBHI 2023.

[3] https://physionet.org/content/challenge-2021/1.0.3/

We thank the reviewer again for the thoughtful feedback. We have uploaded a revised manuscript in which all changes are highlighted in blue. Below we summarize how each point has been addressed.

1. W1 (Rationale for design choices)

We added a "Design rationale" paragraph at the beginning of Section 3. It explains the choice of the five objectives (covering contrastive, latent prediction, masked regression and masked clustering families), the CNN stem plus sequence backbone design following speech SSL (wav2vec, HuBERT), the three backbone classes, the 240 Hz sampling rate, the 2.5 s input windows (supported by our input-length ablation), the log-spaced pretraining scales and the three evaluation modes.

2. W2 + Q1 (Shared encoder)

The comparison between the narrow (2-layer) and wide (4-layer) CNN stem for JEPA and CPC is now included in the appendix (Tables 88 and 89) and referenced in Section 3 ("Backbone architectures"). The differences are mostly not statistically significant, indicating robustness to the stem design.

3. W3 (Continual pretraining)

The finetuning strategy comparison (Appendix A.8.4) now covers all five pretraining objectives, and we revised the corresponding paragraph in Section 4 to state the scope of the conclusion explicitly. Continual pretraining followed by a linear head falls within the top statistical group in 54 of 55 objective-task combinations, while a non-linear query-attention head never significantly outperforms the linear head. The size of the benefit depends on the objective. Continual pretraining yields the best result on 9 of 11 tasks for both CPC and HuBERT++, whereas for data2vec, DinoSR and JEPA it performs on par with standard finetuning. We have also toned down the corresponding recommendation in the Conclusion.

4. W4 (Why objectives differ)

  • Last-token readout for CPC. The appendix section A.4.4 shows that using the last token, which summarizes the full causal context, lowers linear evaluation performance on all tasks compared to mean pooling.

  • Layer-wise linear probing. The new appendix section A.6.4 reports linear probes on the CNN stem output and on each backbone layer for S4 and Transformer across all five objectives, evaluated on the diagnostic, form and rhythm statements of PTB-XL (all). The results confirm the advantage of CPC intermediate representations for both backbones and show that linear performance often peaks at an intermediate layer, which we discuss in Section 4 ("Representation analysis").

  • Limitations. We now state that the representational analyses provide supporting rather than causal evidence and that a mechanistic understanding is left for future work.

5. Q2 (Causal vs. non-causal backbone)

The comparisons of causal and non-causal S4 backbones in supervised training and JEPA pretraining are now included in the appendix (Tables 72 and 90) and referenced in Section 3.

6. Q3 (Label mappings)

All mapping tables (Tables 61-66) now include the PTB-XL target descriptions, and footnotes highlight the cases where source and target classes are not fully equivalent. The text now cites the SNOMED-CT mappings provided with the PhysioNet/CinC Challenge 2021 dataset, against which all mappings were verified.

The authors’ response partially resolves my concerns. I maintain my initial score.

Reviewer_CUVnOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This work presents a systematic empirical study of ECG foundation models, comparing five self-supervised pre-training objectives, multiple backbone architectures, and data scales up to 11 million public samples within a unified framework. Results show that CPC achieves the best overall performance, followed by JEPA; S4 generally outperforms Transformers and CNNs across the tested settings. Scaling up pre-training data is generally beneficial, but improvements in pre-training loss do not consistently translate to better downstream performance. Through transfer learning, label efficiency, and representation analysis, the paper further demonstrates that the choice of architecture and pre-training objective deserves as much attention as data scale.

Critical strengths
  1. Controlled comparisons offer practical value. Comparing five pre-training objectives, multiple backbone families, and various data scales within a unified framework makes it easier to isolate the sources of performance differences than evaluating existing models in isolation. The overall superiority of CPC and S4 provides concrete guidance for the design of ECG foundation models.

  2. Broad evaluation coverage; findings extend beyond "more data is better." The evaluation encompasses 26 downstream tasks, three adaptation methods, as well as transfer learning, label efficiency, and representation analysis. In particular, the observation that "improvements in pre-training loss do not necessarily translate to better downstream performance" offers valuable insights for model selection and pre-training evaluation. Test-set bootstrapping also provides some statistical support.

Critical weaknesses
  1. Claims of architectural advantage outpace the evidence. The experiments support that "the tested S4 configuration performs relatively well," but this is insufficient to establish its inductive bias as the primary driver of performance or to generalize the conclusion to the broader SSM family. While inter-layer CKA differences offer a correlative clue, they do not constitute proof of the performance advantage's origin. The distinct results for S4, Mamba-2, and Transformer need to be carefully disentangled.

  2. Scale and data source changes are confounded. The small-scale setting is drawn from a HEEDB subset, whereas the largest scale incorporates HEEDB-Emory and CODE-15%. As a result, the observed improvements may arise from both increased sample size and shifts in data distribution/diversity, meaning they cannot be attributed purely to quantitative scaling. Furthermore, test-set bootstrapping cannot replace an uncertainty analysis accounting for training randomness or subset sampling variation.

Questions for authors
  1. Fairness of the architecture comparison and scope of the conclusions. The paper finds that S4 generally outperforms the evaluated Transformer and CNN backbones and hypothesizes that its inductive biases are a primary driver. Could the authors clarify how parameter counts, computational costs, and hyperparameter search budgets were matched across architectures? Considering the Mamba-2 results, should this advantage be interpreted as specific to the evaluated S4 configuration or as a broader advantage of the SSM family?

  2. Effects of dataset size versus source diversity. Smaller-scale pretraining uses HEEDB subsets, whereas the 11M-sample setting additionally incorporates HEEDB-Emory and CODE-15% (§3; Figure 4). Can the authors disentangle the contributions of increased sample size and changes in data sources? Could they provide a scaling comparison with consistent source proportions, or explicitly discuss how this confounding factor limits the scaling conclusions?

  3. Robustness to training and subsampling randomness. The paper assesses statistical significance through test-set bootstrapping, which does not capture variability from pretraining randomness or subset selection. Have the main method rankings and scaling trends been validated across different random seeds or repeated subsampling? If not, could the authors provide repeated runs for key settings or discuss how this limitation affects confidence in the conclusions?

2 comments

We thank the reviewer for the careful and constructive feedback. Below we address each point, including new results that will be incorporated into the revised manuscript.

1. W1 (Scope of the architectural claim)

We appreciate the opportunity to state the scope of this empirical finding more precisely. Our results indicate that the advantage is specific to S4 rather than shared across the SSM family, as Mamba-1 and Mamba-2 underperform S4 on all 11 tasks in the supervised comparison. We will consistently refer to S4 rather than SSMs, and present the role of S4's inductive biases as a hypothesis motivated by the results. We see the CKA analysis as supporting evidence consistent with the performance results, not as proof of their origin. To complement it, we will add layer-wise linear probing experiments for S4 and Transformer across all five pretraining objectives on PTB-XL (all), reported separately for form, rhythm and diagnostic statements. This characterizes how semantic information, temporal dependencies (rhythm) and global separability develop across layers. To strenghten the architectural comparison within the SSM family, we will further report finetuning results for all pretraining objectives with a Mamba-2 backbone on the 753K HEEDB subset, alongside S4 and Transformer. Finally, we will expand the Limitations section to state that our architectural conclusions are empirical and that the representational analyses provide supporting rather than causal evidence.

2. W2 + Q2 (Dataset size vs. data source)

The smaller scales (18K to 753K) are drawn exclusively from HEEDB, while the 11M setting additionally includes HEEDB-Emory and CODE-15%, a minor fraction of the data. To disentangle size from source, we will pretrain CPC and JEPA (S4) on HEEDB only, excluding HEEDB-Emory and CODE-15%, and report the HEEDB-only scaling analysis for validation loss and downstream error. This complements our existing matched-scale comparison of HEEDB and MIMIC-IV-ECG (753K vs. 759K), which showed largely similar downstream performance. We will also discuss the remaining influence of dataset composition explicitly in the Limitations.

3. W2 + Q3 (Training randomness)

We agree that test-set bootstrapping captures only statistical uncertainty but not variability across training runs. We therefore pretrained CPC and JEPA (S4) twice more on the 753K HEEDB subset and finetuned each run (macro-AUROC; "Rep." is the run reported in the manuscript; bold = not significantly worse than the best run per objective, bold italic = best).

DatasetJEPA Rep.Run 1Run 2CPC Rep.Run 1Run 2
Ningbo0.9710.9720.9690.9740.9710.972
CPSC20180.9700.9710.9730.9720.9720.971
CPSC-Extra0.8880.8950.8930.9060.9090.909
Georgia0.9130.9110.9120.9120.9170.916
Chapman0.9570.9580.9640.9590.9610.962
SPH0.9760.9740.9790.9820.9820.984
PTB-XL (all)0.9430.9410.9420.9430.9410.944
PTB-XL (sub)0.9240.9420.9380.9370.9400.938
PTB-XL (super)0.9320.9320.9310.9320.9300.930
ZZU pECG0.8910.8870.8900.8990.9000.899
EchoNext0.8260.8280.8280.8320.8310.831

The differences between runs are not statistically significant, with the exception of Chapman for JEPA and Ningbo for both JEPA and CPC. We take this as sign that fluctuations due the randomness of the pretraining process do not undermine the robustness of our results. Due to computational constraints to investigate training fluctuations across all investigates backbones and pretraining objectives.

4. Q1 (Fairness of the architecture comparison)

We did not match parameter counts or compute explicitly. Instead, each architecture family received the same supervised hyperparameter search (model size, learning rate, input length), and the best configuration per family was selected. As noted in our Limitations, this selection relies on supervised performance, which need not fully align with the best downstream performance after pretraining performance. Importantly, this procedure did not favour S4 in terms of capacity, as the selected S4 is the smallest and cheapest backbone (3M parameters vs. 19.2M for the Transformer and 10M for Net1D, with about 16 times fewer forward FLOPs than the Transformer). We will add the computational cost of Mamba-2 and the Transformer with sinusoidal PE to the complexity table.

We thank the reviewer again for the constructive feedback. We have uploaded a revised manuscript in which all changes are highlighted in blue. Below we summarize how each point has been addressed.

1. W1 (Scope of the architectural claim)

  • Mamba-2 comparison across all objectives. The appendix section A.6.2 now reports finetuning results for all five pretraining objectives with S4 and Mamba-2 backbones on the 753K HEEDB subset. S4 falls within the top statistical group in 51 of 55 objective-task combinations, compared to 28 for Mamba-2. The gap is largest for CPC and HuBERT++, whereas both backbones perform on par for JEPA. Together with the supervised comparison, where S4 outperforms Mamba-1 and Mamba-2 on all 11 tasks, this indicates that the advantage is largely specific to S4 rather than shared across the SSM family. We state this explicitly in Section 4 ("Backbone analysis: performance").

  • Layer-wise linear probing. The appendix section A.6.4 reports linear probes on the CNN stem output and on each backbone layer for S4 and Transformer across all five objectives, evaluated on the diagnostic, form and rhythm statements of PTB-XL. The results confirm the advantage of CPC intermediate representations for both backbones and show that linear performance often peaks at an intermediate layer, which we discuss in Section 4 ("Representation analysis").

  • Limitations. We now state that our architectural conclusions are empirical and that the representational analyses provide supporting rather than causal evidence. We fully agree that a mechanistic understanding of the observed finding would a worthwhile direction for future research.

2. W2 + Q2 (Dataset size vs. data source)

The new appendix section A.7.2 repeats the scaling analysis for CPC and JEPA with HEEDB alone (10M samples) as the largest scale. Both the loss scaling (CPC α=0.14\alpha = 0.14, R2=0.926R^2 = 0.926; JEPA α=0.0346\alpha = 0.0346, R2=0.8904R^2 = 0.8904) and the strong rank correlation between validation loss and downstream error (Spearman ρ≥0.9\rho \geq 0.9 on all three datasets) are preserved. We reference this in Section 4 ("Scaling with dataset size") and discuss the remaining influence of dataset composition in the Limitations.

3. W2 + Q3 (Training randomness)

The repeated pretraining runs for CPC and JEPA are now included in the appendix (Tables 45 and 46). Section 3 ("Evaluation") states that bootstrapping does not capture this source of variability and refers to these results. With very few exceptions the results of different pretraining runs do not show any statistically significant differences.

4. Q1 (Fairness of the architecture comparison)

The backbone complexity table (Table 23) now includes Mamba-2 (7.7M parameters) and the Transformer with sinusoidal positional encodings (19.7M parameters). It confirms that the selected S4 backbone (3M parameters) is the smallest and cheapest of all evaluated backbones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.