acceptodds
Under review as a conference paper at ICLR 2027

PhysioTRACE: Provenance-Aware Stress Tests for Physiological Foundation Models

Abstract

Physiological foundation models encode how a signal was recorded alongside the physiology it reflects. When recording conditions are associated with diagnosis, this acquisition provenance can become a shortcut, yet the two measurements usually offered as evidence, shifted transfer and provenance decodability, do not show whether a predictor uses it. We introduce PhysioTRACE, a four-axis behavioral audit for frozen encoders that separates what a probe can decode from what a fixed task head relies on. Recover scores how decodable provenance is; Stress reverses only the provenance-target association on the same held-out records; Intervene removes a train-localized provenance component; and Verify certifies that removal only if it beats matched random projections within a declared utility margin. Each audit thus ends in one of three verdicts: no reliance, or reliance with the remedy certified or refused. Across electroencephalography (EEG) and electrocardiography (ECG), five training objectives, and five frozen foundation models, the relation between Recover's calibrated score (Linear Provenance Information) and out-of-distribution utility changes sign between datasets, so neither predicts the other and neither can stand in for a reliance test. On paired EEG views where the shortcut is known by construction, the audit detects it (the head fitted under the association loses about 0.2 AUROC when the association is reversed, while a control head fitted without it is unaffected), localizes a rank-two component whose removal returns that head to control-level behavior without measurable loss of ordinary utility, and certifies the remedy for both encoder objectives tested. On real ECG device metadata it returns all three verdicts: it certifies a remedy that removes 91% of one model's excess vulnerability, finds no reliance where device and diagnosis are barely associated, and refuses the remedy for a second model whose localized direction also carries task signal. A claim that a physiological model is robust to how its inputs were recorded therefore needs a behavioral test, and PhysioTRACE turns it into a test that can pass, fail, or refuse a remedy. We release the protocol, splits, probes, and machine-readable results.

Primary area:
datasets and benchmarks
Keywords:
physiological foundation models, acquisition provenance, shortcut learning, robustness evaluation, distribution shift

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MuszOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Physiological foundation models may retrain information of signal provenance which means how signals were records or from which devices or sites. These provenance information can be associated with diagnosis and provide a shortcut for downstream prediction. This paper then propose PhysioTRACE, an evaluation protocol that distinguish whether recording information can be decoded from embedding. The protocol changes the recording-condition–target association through reweighting, removes a selected representation component, and checks the resulting vulnerability and predictive performance. Experiments include controlled EEG reference changes and observational ECG device comparisons.

Critical strengths

The paper makes a useful distinction between recording information that can be decoded from an embedding and information that a task classifier actually relies on. The controlled EEG experiments, matched random-removal baselines, and ECG evaluations provide evidence for the proposed audit.

Critical weaknesses
  1. The paper writing and presentation especially the abstract and introduction section makes the research problem and contribution difficult to follow. The intro combines and mix terms and provide indirect claims, make the readers difficult to find is it necessary to keep the provenance. It should state clearly that retaining recording information is not necessarily harmful. Also, a simple example in physiological domain would help clarify rather than use image domain hues as example of shortcut. Moreover, these limitation do not really link to specific ml/dl problem, does the paper target the general out-of-distribution problem but want to keep the new tasks recording distribution to improve the adaptation.
  2. The 'shortcut' is also ill-defined and hard to follow, how it benefit the downstream task should be clearly state.
  3. The experiment setups and results do not clearly separate or clarify the claim. For out-of-distribution dataset, only use one dataset for each modality is not sufficiently to support the claim. For EEG, there are a lot of sleep data and ECG there are a lot of single-lead or 12-lead datasets. The evidence for the complete protocol is narrower than the broad model comparisons initially suggest.
Questions for authors

See the weakness section.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Musz

Thank you for pointing out where the problem and contribution were difficult to follow. The abstract is rewritten and shortened from 333 to 264 words. The revised introduction gives a physiological example, introduces each dataset and model where it is first used, and states explicitly that retaining recording information is not harmful in itself (changes are marked in blue).

W1–W2. The problem and how a shortcut helps prediction. Consider the controlled EEG experiment with pretrained CBraMod. When the EEG reference co-occurs with diagnosis, the exposed head reaches 0.833 AUROC, against 0.759 for a control trained without that association. When the association reverses, the exposed head falls to 0.576, while the control stays at 0.759. The first comparison is how the shortcut helps and the second is how it fails, on the same held-out records. The exposed head's loss beyond the control's, 0.257 here, is what we call excess vulnerability, GG.

PhysioTRACE tests that sensitivity and evaluates a candidate remedy. It does not adapt the model to a new domain or solve general OOD generalization.

Removing recording information can also discard information the task needs. Marginal LEACE, which targets linearly decodable device information, brings GG near zero in our ECG models but costs 0.068–0.107 AUROC. Verify therefore accepts a removal only if it reduces GG more than matched random controls and preserves task performance within a declared margin (§3.4; App. K.5).

W3. Evidence for the complete protocol across datasets. Full audits now cover two EEG datasets (NMT and HMC) and two ECG datasets (PTB-XL and Chapman–Shaoxing–Ningbo). These are behavioral audits through Verify, in addition to the broader accessibility and transfer screens listed separately in App. K.1.

On paired NMT views, all three pretrained EEG models (CBraMod, BIOT and LaBraM) show reliance and certified remedies within the declared utility margin (Table 1; App. K.3). The second EEG dataset is sleep data, as you suggested: on HMC, reliance is detected in all three models, with two remedies certified and LaBraM's refused because its utility interval crosses the allowed margin (App. K.9).

For ECG, paired acquisition views of each PTB-XL recording yield certified remedies for ECGFounder and MERL, reducing GG from 0.071 and 0.041 to below 0.001 (App. K.10). We then repeated this design on a second, independent 12-lead database, Chapman–Shaoxing–Ningbo (two hospitals in China; 20,000 patients sampled, 1,999 of them for testing), with three acquisition views of each recording (as distributed, band-limited, and resampled) and sinus rhythm as the target.

EncoderGG [95% CI]Residual GG [95% CI]Audit outcome
ECGFounder0.0031 [0.0015, 0.0051]not applicableNo reliance (G<ε=0.01G<\varepsilon=0.01)
MERL0.032 [0.027, 0.037]0.0004 [0.0002, 0.0007]Reliance; remedy certified

MERL's removal takes out 99% of GG, beats all 100 matched controls and stays within the utility margin (change −0.0012 [−0.0019, −0.0004]). The recording view can be read out of both models about equally well (87.5% and 84.6% probe accuracy), yet only MERL shows reliance above the declared threshold. High decodability alone does not tell us how strongly a predictor relies on recording information (§4.3; App. K.10).

The contribution is to test a particular predictor's sensitivity to an association shift, then determine whether a proposed remedy reduces that sensitivity without losing too much task information. If this resolves your concerns, we ask you to raise your score. If the problem statement is still unclear at any point, please tell us where.

Reviewer_Fvh1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PhysioTRACE, a four-step audit of whether a fixed linear task head on a frozen encoder relies on acquisition provenance (reference, device, site):

  • Recover measures provenance decodability with a calibrated linear V-information score (LPI).
  • Stress reweights held-out data to reverse the provenance–target association at fixed marginals. It compares an "exposed" head, fitted under the association, with a "control" head fitted with independence weights. The difference is the excess vulnerability G.
  • Intervene removes a train-localized low-rank provenance subspace.
  • Verify certifies that removal only if it beats rank- and energy-matched random subspaces and keeps utility within a declared margin δ.

The authors present three results. First, LPI and OOD transfer are not consistently related across ERP/P300 and PTB-XL. Second, on paired NMT reference views with an induced shortcut, the audit detects and removes it. Third, on PTB-XL device metadata, the audit returns "certified" (ECGFounder), "no reliance" (a weakly associated device pair), and "not certified" (CLEF-Small).

Critical strengths
  • The core distinction between accessibility and reliance is correct and well motivated, and the control-head comparator is a sensible way to separate association-specific sensitivity from ordinary shift.
  • The statistics are careful. Identity-aware bootstrap intervals (subjects/patients, crossed with encoder seeds), rank- and energy-matched random controls, and locked selection before test scoring are all better practice than is typical in this literature. The ERP analysis showing that naive trial-level resampling narrows intervals 14–31× is a useful cautionary result.
  • The audit can return a negative answer. Reporting a refused remedy (CLEF-Small) and a null case alongside a certified one is commendable.
  • The paper is transparent about scope, confounding, and limitations, and code and machine-readable results are released.
Critical weaknesses
  1. The methodological novelty is limited. Each component is established:

    • V-information probing (Xu et al.)
    • association-shift and nuisance-randomized stress tests (Veitch et al.; Puli et al.; Makar et al.)
    • linear erasure evaluated against random-direction controls (Elazar et al.; Ravfogel et al.; Belrose et al.)

    Table 2 acknowledges that the projection operator and random controls are reused. The contribution is mainly the assembly into a decision procedure with a control head and a utility margin. That is useful, but it is an evaluation protocol rather than a new method or a new empirical finding about PFMs.

  2. The known-ground-truth validation (Q2) is close to circular, and it does not involve foundation models.

    • The shortcut is induced by training the exposed head where reference co-occurs with diagnosis, on embeddings where reference is 85–94% decodable. That a linear head exploits it, and that removing the paired-view contrast subspace undoes it, is expected by construction.
    • The audited encoders are EEGNetSmall (d=128) models trained by the authors, not PFMs. The frozen EEG foundation models (LaBraM, BIOT, CBraMod) appear only in the transfer and LPI screens and never go through Stress, Intervene, or Verify. The title and abstract therefore overstate what was audited.
    • Paired localization requires exact multi-view recordings of the same data. These rarely exist outside re-referencing, which limits its practical relevance.
  3. In observational data, G does not isolate provenance reliance. Reversing the y–u association by reweighting penalizes any head that uses features correlated with u, including legitimate physiology that differs between the populations recorded on each device. The paper's own sensitivity analyses suggest the observational effect is largely patient mix and workflow rather than acquisition:

    • A head using only age, sex, and recording year reaches G = 0.061. This exceeds ECGFounder (0.023) and matches CLEF-Small, whose excess over the metadata head is −0.002 [−0.010, 0.007].
    • On human-validated ECGs, ECGFounder's G falls to 0.0022, below the paper's own ε = 0.01.
    • On the calendar-matched cohort, ECGFounder's excess over the metadata comparator is only 0.004.

    Under these conditions, "reliance on provenance" and "certified remedy" are hard to interpret. The certified ECGFounder remedy reduces an effect of about 0.02 AUROC that may itself be mostly confounding.

  4. The three verdicts are less informative than presented.

    • The "no reliance" case uses a device pair with a natural odds ratio of 1.04. With essentially no association in training, the exposed and control heads nearly coincide, so G ≈ 0 by construction. This is a sanity check, not a discovery.
    • The CLEF-Small refusal concerns a specific rank-1 linear remedy. It shows that this localization captures task signal, not that the reliance is irremediable.
    • Verdicts depend on ε, δ (0.02 for NMT, 0.01 for ECG; the NMT δ was prespecified only for the reconstruction encoder), and the rank p, and no sensitivity analysis over these choices is reported.
  5. The Q1 evidence is weak. The claim that the LPI–transfer relation "changes sign between datasets" rests on Spearman ρ = −0.29 (n = 8) and +0.07 (n = 7). Neither is distinguishable from zero, so a sign change between them carries little information. The point that probe accessibility does not imply use is also already established in the probing literature (Hewitt & Liang; Elazar et al.). In addition, the frozen FMs perform poorly on ERP (e.g., CBraMod OOD AUROC 0.547, LaBraM 0.567) relative to small task-trained ResNets (about 0.82). This suggests input adaptation issues (e.g., resampling ERP epochs to 200 samples, single-channel adapters on HMC), which weakens model-level comparisons.

  6. The scope is narrow relative to how PFMs are used.

    • The audit covers only linear heads on frozen embeddings and binary or multiclass endpoints. In practice PFMs are usually fine-tuned, and the protocol does not address reliance that emerges during fine-tuning or in nonlinear heads.
    • It requires provenance labels with sufficient joint support in every cell, which many clinical datasets lack.
  7. Presentation. The main text is dense and heavy with numbers. Key setup details are deferred to the appendix, notably that the NMT audit uses author-trained EEGNetSmall rather than PFMs, and the metadata comparator results that substantially qualify Q3. The abstract's mention of "five frozen foundation models" invites a misreading of what was audited.

Questions for authors
  1. Can the full four-step audit be run on frozen LaBraM, BIOT, and CBraMod embeddings on NMT, and on at least one fine-tuned PFM?
  2. How should G be interpreted when u indexes a population difference? Would a head built only on known physiological features (e.g., ECG intervals, EEG band power) also show G > 0 under the same stress? A positive-control head that uses legitimate but u-correlated features would help calibrate G.
  3. How sensitive are the verdicts to ε, δ, the rank p, and the number of random controls?
  4. How does the remedy compare with LEACE or with last-layer retraining under independence weights (Kirichenko et al.)?
  5. For the ECG audits, what happens when stratifying by recording era and validation status jointly, rather than in separate sensitivity analyses?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Fvh1 (1/2): new audits and comparisons (W1, W2, W6; Q1, Q4)

Thank you for the concrete tests you proposed. We added analyses for all five questions: full audits of the three frozen EEG models you named and a secondary fine-tuning study with CBraMod (Q1), a physiological positive control (Q2), sensitivity analyses (Q3), comparisons with LEACE and last-layer retraining (Q4), and the joint stratification by era and validation status (Q5). Changes in the revised PDF are marked in blue.

W1 / Q4. Contribution and comparison with existing remedies. PhysioTRACE contributes a decision procedure for testing whether a fitted head is vulnerable to a provenance–target association shift and whether a proposed remedy reduces that vulnerability specifically while preserving utility. Its probing and erasure components are established and credited in §2. The procedure is released as a standalone Python library that runs on embeddings from any frozen encoder (link in the general response).

In the LEACE comparison you asked for, class-conditional LEACE reduces CLEF-Small's GG from 0.060 to 0.001 and beats 100/100 matched controls, yet loses 0.0175 AUROC (95% CI for the loss: 0.0137–0.0213), which exceeds δ=0.01\delta=0.01. For ECGFounder it reduces GG from 0.023 to 0.001 with a utility change of −0.0049 [−0.0075, −0.0023], which passes. Marginal LEACE brings GG near zero on both models but costs 0.068 and 0.107 AUROC. An eraser can thus leave almost no residual vulnerability, beat every matched control, and still fail as a remedy (App. K.5).

On paired CBraMod, empirical LEACE matches our projection within 2.2×10−122.2\times10^{-12} in the common numerical support. Independence-weighted last-layer retraining, which replaces the head instead of changing the representation, reproduces the control head and improves mean within-reference AUROC by 0.022 [0.008, 0.036]. Its GG is zero by construction, so it is not a certification of a fixed head.

W2 / Q1. Pretrained models and controlled validation. The complete audit now covers frozen CBraMod, BIOT and LaBraM. Utility is the mean same-reference ID AUROC of the fixed source-reference heads.

EncoderGG before → afterΔ ID utility [95% CI]
CBraMod0.257 → −0.010+0.0048 [−0.0055, 0.0155]
BIOT + adapter0.135 → 0.004−0.0010 [−0.0073, 0.0051]
LaBraM0.202 → 0.003−0.0029 [−0.0097, 0.0036]

All three pass the declared utility margin and beat 100/100 matched controls (Table 1; App. K.3). The controlled design tests detection, label-free localization on training views and held-out certification separately. Certification requires the localized removal to outperform matched controls while preserving utility; the induced association alone does not establish either requirement. Each encoder has one checkpoint.

The same three models were also audited on a second paired dataset, HMC sleep staging. Reliance is detected in all three (GG 0.013–0.035). The remedies for CBraMod and BIOT are certified, and LaBraM's is refused because its utility lower bound (−0.028) is below −δ-\delta (App. K.9).

Paired views are not limited to re-referencing: filtering and resampling a stored recording also produce them. We presented each PTB-XL ECG as distributed, band-limited to 0.5–40 Hz, and at 100 Hz sampling, and audited ECGFounder and MERL in this paired mode (App. K.10; results under W3 in part 2).

We repeated this design on a second database, Chapman–Shaoxing–Ningbo (1,999 test patients; target: sinus rhythm). There the outcome is not fixed by construction. Under the same induced association (p=0.60p=0.60), the view is decodable at 87.5% from ECGFounder, inside the 85–94% range you cite for NMT, and at 84.6% from MERL, but only MERL shows reliance above the declared threshold (G=0.032G=0.032 [0.027, 0.037]; remedy certified, residual 0.0004). ECGFounder's G=0.0031G=0.0031 [0.0015, 0.0051] lies entirely below ε=0.01\varepsilon=0.01, so the audit returns no reliance (§4.3; App. K.10).

W6 / Q1. Nonlinear heads and a fine-tuned PFM. Both are now examined as secondary analyses. With an MLP head in place of the linear head, all three EEG models remain certified: GG falls from 0.230 to 0.012 (CBraMod), from 0.046 to 0.011 (BIOT) and from 0.185 to 0.005 (LaBraM), so the linear removal takes out 76–97% of an MLP head's GG (App. K.8).

We also fine-tuned CBraMod end to end under the association, for six seeds, and audited each network with its final layer as the head. Fine-tuning raises GG from 0.259 at initialization to 0.292–0.307 while LPI stays at 1.24–1.29 bits, and the same rank-two removal is certified on all six seeds, with residual GG between −0.013 and −0.010. The fine-tuned networks stay close to the pretrained encoder, so we report this as a secondary test of the audit after fine-tuning (App. K.11).

AuthorsOpenReview ↗

Response to Reviewer Fvh1 (2/2): interpretation, sensitivity, presentation (W3-7; Q2-3, Q5)

W3 / Q2 / Q5. Population differences and joint support. We ran your physiological positive control on the same ECGs, patient folds and stress regimes, with training-fold imputation:

HeadGG [95% CI]
12SL physiological intervals/axes0.0051 [0.0032, 0.0069]
Physiology + age/sex0.0187 [0.0155, 0.0219]
ECGFounder0.0233 [0.0209, 0.0261]

Physiology alone gives a positive GG below ε\varepsilon, and adding demographics takes it above ε\varepsilon. This is why observational GG has to be read as sensitivity to a device-associated collection and population signal. These comparators now accompany the ECG results (§4.3; App. K.6). Their differences are not a causal decomposition. The measured benefit of the remedy remains 0.023→0.0020.023\to0.002 under the specified stress.

A paired design removes the population pathway altogether. With the same ECG of each of 1,907 test patients presented under the three acquisition views listed under W2, GG is 0.071 [0.064, 0.079] for ECGFounder and 0.041 [0.037, 0.046] for MERL, and a certified rank-two removal leaves 0.0004 and 0.0006 with utility changes within 0.001. These views are produced in software, so the result shows that reliance on an acquisition setting can be detected and removed with the population held fixed; it does not apportion the device effect in the observational cohort (App. K.10).

The joint era × human-validation cohort fails the support requirements: the smallest held-out cell is 6 against 75 required, and the record ESS is 181 against 500. Outside human validation one device has no observations. A jointly adjusted verdict is therefore not evaluable on this pair, and we do not treat insufficient overlap as no reliance.

W4 / Q3. Verdicts and sensitivity. For CBraMod, BIOT and LaBraM, all ten task-head CC values (10−610^{-6}–10310^3) remain certified, and removal beats 1,000/1,000 new controls. All three pass for any δ≥0.0097\delta\ge0.0097. With the evaluation stress held fixed, GG increases monotonically with the training association (App. K.4).

For ECG the verdict depends on rank: ECGFounder passes at the declared rank 1 and with conditional LEACE, while ranks 2, 3 and 5 lose too much utility. CLEF-Small fails at every tested rank and under both LEACE variants. As you say, this shows that these linear remedies remove task signal; other remedies remain possible.

The near-independent device pair is a negative control. The Chapman audit (W2, part 1) adds a no-reliance verdict under the induced association (p=0.60p=0.60): the exposed head is trained with the view tied to the label and still loses only 0.003 AUROC when the association is reversed. We read this verdict as specific to that head and task. §3.4 now states that "no reliance" means reliance was not detected under the declared threshold and uncertainty criteria.

W5 / W7. Evidence and presentation. Gap correlations replace the sign-change claim (ERP ρ=0.762\rho=0.762; PTB-XL 0.464, inconclusive). The ERP result supports a positive association between LPI and the shift gap. Our evidence that accessibility does not determine a head's reliance is behavioral: under fixed evaluation stress, unchanged CBraMod embeddings support heads with GG from 0.017 to 0.299 as training changes (Fig. 9). We do not rank architectures from ERP scores that are affected by adaptation. §1 now credits the probing literature (Hewitt & Liang; Elazar et al.) for the point that accessibility does not imply use. The abstract is rewritten and shortened from 333 to 264 words, and it now names only models that were fully audited. Table 1 and App. K.1 separate screens from full audits. §4.2 now says in the main text that the five-seed NMT audit uses EEGNetSmall encoders, apart from the three pretrained models, and §4.3 reports the metadata and physiological comparators there as well. §1 gives a physiological example.

W6. Provenance labels and support. Paired views support controlled validation. The observational mode still requires provenance labels and adequate overlap.

All five of your questions now have an analysis in the revised PDF, and the audit both accepts and refuses existing remedies on pretrained models. If this resolves your concerns, we ask you to raise your score. If a concern remains, please tell us which result or interpretation still prevents you from supporting the paper.

Reviewer_mZYMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper studies whether PFMs rely on information about how data were collected, such as the device or recording setup, when making predictions. It proposes a four-step audit to measure provenance decodability, test whether predictions depend on it, remove related features, and check whether task performance is preserved. The authors evaluate the framework using paired EEG recordings with a known, artificially introduced shortcut and observational ECG data with real device metadata. The results show that provenance decodability does not necessarily reflect the degree of reliance, and the audit produces three possible outcomes: no detected reliance, reliance with a certified remedy, and reliance without a certified remedy.

Critical strengths

S1. The paper addresses an important problem in evaluating physiological foundation models by distinguishing provenance decodability from actual reliance in downstream predictions. The paired EEG experiment provides convincing evidence that the proposed audit can detect a known shortcut and reduce its influence without noticeably affecting task performance. I also appreciate that the framework considers both shortcut reduction and utility preservation, allowing it to reject interventions that may remove useful task information, as demonstrated by the CLEF-Small result.

Critical weaknesses

W1. The paper argues that the relationship between provenance accessibility and transfer differs across datasets, but this result depends on using absolute OOD AUROC, which also reflects a model's in-domain performance. When using the paper's own ID–OOD performance gap (Eq. 5), the correlations have the same direction on both datasets, although the PTB-XL result is not significant (see Q1). This makes the claimed difference between datasets less convincing. The full audit is also tested on only two pretrained foundation models, both ECG. The EEG experiment uses an encoder trained by the authors, leaving the framework's applicability to other foundation models largely untested.

W2. In the observational ECG experiment, device information is closely tied to patient characteristics and recording workflow. The authors acknowledge this, but the abstract's claim of a certified 91% reduction in excess vulnerability may suggest a more specific result than the evidence supports. The proposed projection is also compared only with random subspaces, rather than existing concept-erasure methods. It is therefore difficult to tell whether the method offers an advantage over existing alternatives, or whether CLEF-Small's refused verdict is specific to the chosen intervention.

Questions for authors

Q1. The paper acknowledges that reliance is a property of the fitted task head rather than the foundation model itself. Since the measured reliance depends on the training association and potentially on head regularization, how stable are the audit verdicts across these settings for the same frozen encoder? What conclusions can be drawn about the foundation model itself, rather than the particular encoder–head configuration being evaluated?

Q2. Figure 3 compares LPI with absolute OOD AUROC, which reflects overall predictive performance rather than just the loss caused by domain shift. The reported correlations are weak (ρ=−0.29\rho = -0.29 on ERP; ρ=+0.07\rho = +0.07 on PTB-XL). Using the ID–OOD gap (Eq. 5) and Tables 5, 9, and 10, I obtain ρ≈0.78\rho \approx 0.78 (p≈0.02p \approx 0.02) on ERP and ρ≈0.45\rho \approx 0.45 (p≈0.31p \approx 0.31) on PTB-XL. For the worst-OOD gap, the correlations are ρ≈0.74\rho \approx 0.74 (p≈0.04p \approx 0.04) and ρ≈0.02\rho \approx 0.02 (p≈0.97p \approx 0.97), respectively. The low OOD performance of masked reconstruction on PTB-XL also appears partly due to its low in-domain AUROC (0.720). How do these results affect the conclusions in Q1 and Contribution 2?

Q3. On PTB-XL, a head trained only on age, sex, and recording year has a larger GG than ECGFounder (0.061 vs. 0.023). On the calendar-matched cohort, ECGFounder's excess over the metadata comparator is only 0.004 (Table 18), although this subset has limited cell support. Would a metadata-adjusted analysis with sufficient support change the reliance or certification verdict? If the result mainly reflects device-associated case mix and workflow, should the abstract qualify its claim about the certified remedy?

Q4. Could the authors run the full audit on at least one pretrained EEG foundation model using the paired NMT views already evaluated in Table 8? This would provide more direct evidence that the results extend beyond the author-trained EEGNetSmall.

2 comments
AuthorsOpenReview ↗

Response to Reviewer mZYM (1/2), (W1, Q1, Q2, Q4)

Thank you for the review, and for noting that the audit weighs shortcut removal against utility and can refuse a remedy, as it does for CLEF-Small. Each of your four questions, and the comparison with existing erasers raised in W2, led to a new analysis in the revised PDF, where changes are marked in blue.

Q4 / W1. Full audits of pretrained EEG models. All four steps now run on frozen CBraMod, BIOT and LaBraM, using paired NMT views, train-only rank-two localization, 100 matched controls, 2,000 record-bootstrap draws and the declared utility margin δ=0.02\delta=0.02. Utility is the mean same-reference ID AUROC of the fixed source-reference heads.

EncoderGG before → afterΔ ID utility [95% CI]
CBraMod0.257 → −0.010+0.0048 [−0.0055, 0.0155]
BIOT + adapter0.135 → 0.004−0.0010 [−0.0073, 0.0051]
LaBraM0.202 → 0.003−0.0029 [−0.0097, 0.0036]

All three beat 100/100 controls and stay within the utility margin (Table 1; App. K.3). Each evaluation uses one checkpoint and record-disjoint splits.

To check that this result is not specific to NMT, we repeated the audit on a second paired dataset, HMC sleep staging (sleep against wake, three reference derivations, 22 test recordings). It detects reliance in all three models (GG of 0.035, 0.013 and 0.015), certifies the remedy for CBraMod and BIOT, and refuses it for LaBraM: the lower bound of its utility change, −0.018 [−0.028, −0.010], is below the allowed −δ=−0.02-\delta=-0.02 (App. K.9).

Q1. Dependence on the head and training association. For CBraMod, BIOT and LaBraM, certification holds at all ten tested CC values (10−610^{-6}–10310^3), and removal beats 1,000/1,000 new controls. With the evaluation stress held fixed at p=0.60p=0.60, CBraMod's GG rises from 0.017 to 0.299 as the training association increases from 0.35 to 0.65, while its embedding and LPI remain unchanged (App. K.4, Fig. 9). The encoder exposes a removable reference-associated component, and whether a fitted head exploits it depends on training. We therefore report a verdict for each configuration of encoder, head and training association. The verdicts also hold for a different head family: with an MLP head, all three models remain certified, and the same linear removal takes out 95%, 76% and 97% of GG (App. K.8).

Q2 / W1. Accessibility and transfer. We agree that the ID−OOD gap is the relevant comparison for performance lost under shift. Your approximate correlations reproduce from the rounded tables. In full precision, which the revision now reports (also in Fig. 3), the ID−OOD gap correlations are +0.762 on ERP (p=0.028p=0.028, n=8n=8) and +0.464 on PTB-XL (p=0.294p=0.294, n=7n=7); the worst-gap correlations are +0.667 and +0.071. The ERP result supports a positive association between LPI and the shift gap, while the PTB-XL estimate is uncertain. These tests are exploratory and their p-values nominal. We removed the sign-change claim, §1 and §4.1 now describe the correlations this way, and the masked-reconstruction example distinguishes low ID performance from a large shift gap (§4.1; App. K.2). Contribution 2, which you ask about, no longer rests on these correlations: it now cites the fixed-embedding experiment above, in which the same LPI accompanies heads with very different GG. A correlation across models does not establish whether an individual head relies on provenance.

AuthorsOpenReview ↗

Response to Reviewer mZYM (2/2), (W2, Q3)

Q3 / W2. Observational ECG interpretation. The abstract and main text no longer quote the 91% reduction. The abstract, §1 and §4.3 state the absolute effect, G=0.023→0.002G=0.023\to0.002, as a reduction in sensitivity to a device-associated collection and population signal. A head on physiological ECG features plus age and sex reaches G=0.0187G=0.0187 [0.0155, 0.0219]. Neither this comparison nor the remedy identifies a hardware-specific cause.

Restricting jointly to 1990–1999 and to human-validated records leaves a smallest held-out cell of 6, against 75 required, and a record ESS of 181, against 500 required. Outside human validation, one device is absent entirely. A jointly adjusted verdict with sufficient support is therefore unavailable for this pair, and we report that limit explicitly (App. K.6).

A paired design removes the population pathway by construction. We presented each ECG of 1,907 test patients under three acquisition views (as distributed at 500 Hz, band-limited to 0.5–40 Hz, and at PTB-XL's 100 Hz sampling), so patients are identical across views. Heads on ECGFounder and MERL still rely on the view once it is associated with diagnosis (GG of 0.071 and 0.041), and a certified rank-two removal leaves 0.0004 and 0.0006. The views are produced in software, so this isolates an acquisition pathway and does not apportion the observational device effect.

Repeating this design on a second database (Chapman–Shaoxing–Ningbo, 1,999 test patients, target: sinus rhythm) separates the two models: the view is decodable from both (probe accuracy 87.5% for ECGFounder and 84.6% for MERL), but only MERL shows reliance above the declared threshold (G=0.032G=0.032, remedy certified), while ECGFounder's GG is 0.003 [0.0015, 0.0051], below ε=0.01\varepsilon=0.01 (§4.3; App. K.10).

W2. Is CLEF-Small's refusal specific to our remedy? No. It persists at all tested ranks (1, 2, 3, 5) and under both class-conditional and marginal LEACE. Conditional LEACE reduces GG to 0.001 but costs 0.0175 AUROC (95% CI for the loss: 0.0137–0.0213), which exceeds δ=0.01\delta=0.01. For ECGFounder, conditional LEACE passes, while marginal LEACE loses 0.068 AUROC and is refused. On CBraMod, empirical LEACE matches our projection numerically. Other remedies may pass, and the audit would certify them by the same rule (App. K.5–K.6).

If these results and the clarified claims resolve your concerns, we ask you to raise your score. If something still keeps you from supporting the paper, please tell us which concern remains.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision summary

We thank the reviewers. All three find the distinction between decodable provenance and a predictor's reliance on it useful, and mZYM and Fvh1 note that the audit can refuse a remedy. The revision adds full audits of pretrained EEG models and tests whether the verdicts hold across datasets, heads, and existing erasers. It also clarifies what the observational ECG result establishes and shortens the abstract. Changes in the revised PDF are marked in blue.

Full pretrained EEG audits (mZYM Q4; Fvh1 Q1). All four steps now run on CBraMod, BIOT and LaBraM, the three models named by Fvh1. Each remedy is specific and stays within the declared utility margin (Table 1; App. K.3).

Existing remedies and the value of Verify (mZYM W2; Fvh1 Q4). Class-conditional LEACE reduces CLEF-Small's excess vulnerability GG from 0.060 to 0.001 and beats all 100 random controls, but its utility loss exceeds the 0.01 margin. The same eraser passes for ECGFounder. Both erasers bring GG near zero; the utility margin identifies which one also preserves task performance (App. K.5).

Interpretation of the observational ECG result (mZYM Q3; Fvh1 Q2, Q5). The physiological comparator and the joint-support analysis that Fvh1 requested set the scope of the observational result: it measures sensitivity to a device-associated collection and population signal and does not identify a hardware-specific cause. Gap correlations, which mZYM asked about (Q2), replace the sign-change claim. The distinction between accessibility and reliance is supported directly by controlled head-training experiments.

Coverage across datasets and heads (Musz W3; Fvh1 W2, W6). Complete audits now span two EEG datasets (NMT and HMC) and two ECG datasets (PTB-XL and Chapman–Shaoxing–Ningbo). On HMC, the audit detects reliance in all three pretrained models, certifies two remedies and refuses the third on utility. With an MLP head, all three NMT verdicts are unchanged. Paired acquisition views of the same ECG (filtering and resampling) yield certified remedies for ECGFounder and MERL on PTB-XL. On Chapman–Shaoxing–Ningbo the view is about equally decodable from both models (view-probe accuracy 87.5% and 84.6%), yet only MERL shows reliance above the declared threshold: its GG falls from 0.032 to 0.0004 and the remedy is certified, while ECGFounder's GG of 0.003 is below ε=0.01\varepsilon=0.01 and the audit returns no reliance (§4.3; App. K.8–K.10).

PhysioTRACE contributes a decision procedure that tests a fitted predictor's sensitivity to a provenance–target association shift and determines whether a proposed remedy reduces it beyond matched controls while preserving utility within a declared margin. The audit is also released as a standalone, pip-installable Python library that runs on embeddings from any frozen encoder (anonymized source, tutorial and tests: https://anonymous.4open.science/r/physiotrace-731B). The paired audits added in this revision (App. K.3, K.4, K.8–K.11) were computed with this library's estimators and decision rule. The revised PDF has a shorter abstract (264 words, down from 333), distinguishes screening from full audits, defines the certification criteria, and reports sensitivity to the head, the thresholds and the rank. We ask the reviewers to reconsider their scores in light of these results and the clarified scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.