- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Physiological foundation models may retrain information of signal provenance which means how signals were records or from which devices or sites. These provenance information can be associated with diagnosis and provide a shortcut for downstream prediction. This paper then propose PhysioTRACE, an evaluation protocol that distinguish whether recording information can be decoded from embedding. The protocol changes the recording-condition–target association through reweighting, removes a selected representation component, and checks the resulting vulnerability and predictive performance. Experiments include controlled EEG reference changes and observational ECG device comparisons.
Critical strengths
The paper makes a useful distinction between recording information that can be decoded from an embedding and information that a task classifier actually relies on. The controlled EEG experiments, matched random-removal baselines, and ECG evaluations provide evidence for the proposed audit.
Critical weaknesses
- The paper writing and presentation especially the abstract and introduction section makes the research problem and contribution difficult to follow. The intro combines and mix terms and provide indirect claims, make the readers difficult to find is it necessary to keep the provenance. It should state clearly that retaining recording information is not necessarily harmful. Also, a simple example in physiological domain would help clarify rather than use image domain hues as example of shortcut. Moreover, these limitation do not really link to specific ml/dl problem, does the paper target the general out-of-distribution problem but want to keep the new tasks recording distribution to improve the adaptation.
- The 'shortcut' is also ill-defined and hard to follow, how it benefit the downstream task should be clearly state.
- The experiment setups and results do not clearly separate or clarify the claim. For out-of-distribution dataset, only use one dataset for each modality is not sufficiently to support the claim. For EEG, there are a lot of sleep data and ECG there are a lot of single-lead or 12-lead datasets. The evidence for the complete protocol is narrower than the broad model comparisons initially suggest.
Questions for authors
See the weakness section.
Response to Reviewer Musz
Thank you for pointing out where the problem and contribution were difficult to follow. The abstract is rewritten and shortened from 333 to 264 words. The revised introduction gives a physiological example, introduces each dataset and model where it is first used, and states explicitly that retaining recording information is not harmful in itself (changes are marked in blue).
W1–W2. The problem and how a shortcut helps prediction. Consider the controlled EEG experiment with pretrained CBraMod. When the EEG reference co-occurs with diagnosis, the exposed head reaches 0.833 AUROC, against 0.759 for a control trained without that association. When the association reverses, the exposed head falls to 0.576, while the control stays at 0.759. The first comparison is how the shortcut helps and the second is how it fails, on the same held-out records. The exposed head's loss beyond the control's, 0.257 here, is what we call excess vulnerability, G.
PhysioTRACE tests that sensitivity and evaluates a candidate remedy. It does not adapt the model to a new domain or solve general OOD generalization.
Removing recording information can also discard information the task needs. Marginal LEACE, which targets linearly decodable device information, brings G near zero in our ECG models but costs 0.068–0.107 AUROC. Verify therefore accepts a removal only if it reduces G more than matched random controls and preserves task performance within a declared margin (§3.4; App. K.5).
W3. Evidence for the complete protocol across datasets. Full audits now cover two EEG datasets (NMT and HMC) and two ECG datasets (PTB-XL and Chapman–Shaoxing–Ningbo). These are behavioral audits through Verify, in addition to the broader accessibility and transfer screens listed separately in App. K.1.
On paired NMT views, all three pretrained EEG models (CBraMod, BIOT and LaBraM) show reliance and certified remedies within the declared utility margin (Table 1; App. K.3). The second EEG dataset is sleep data, as you suggested: on HMC, reliance is detected in all three models, with two remedies certified and LaBraM's refused because its utility interval crosses the allowed margin (App. K.9).
For ECG, paired acquisition views of each PTB-XL recording yield certified remedies for ECGFounder and MERL, reducing G from 0.071 and 0.041 to below 0.001 (App. K.10). We then repeated this design on a second, independent 12-lead database, Chapman–Shaoxing–Ningbo (two hospitals in China; 20,000 patients sampled, 1,999 of them for testing), with three acquisition views of each recording (as distributed, band-limited, and resampled) and sinus rhythm as the target.
MERL's removal takes out 99% of G, beats all 100 matched controls and stays within the utility margin (change −0.0012 [−0.0019, −0.0004]). The recording view can be read out of both models about equally well (87.5% and 84.6% probe accuracy), yet only MERL shows reliance above the declared threshold. High decodability alone does not tell us how strongly a predictor relies on recording information (§4.3; App. K.10).
The contribution is to test a particular predictor's sensitivity to an association shift, then determine whether a proposed remedy reduces that sensitivity without losing too much task information. If this resolves your concerns, we ask you to raise your score. If the problem statement is still unclear at any point, please tell us where.