- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a benchmark on Speech clinical AI, on 12 public datasets with 27 clinical speech tasks, grouped by the stage of speech production each condition affects. It evaluates 12 frozen audio encoders with linear probes, plus cross-dataset transfer. The finding is that no encoder works well across all conditions, large general speech models do best overall.
Critical strengths
- the benchmark provides a useful resource for the community, as a common protocol and released code makes future comparisons much easier. The task and models covered are quite board.
Critical weaknesses
- No naive baseline is provided as reference. Nor does the paper provide controls such as non-speech-only baseline, simple acoustic-feature baseline metadata-only baseline. Some results are in fact worse than a constant prediction.
- The evaluation confirms that existing models fall short on these tasks, but it offers little insight into why certain models do better on certain tasks, or how to build a more general and robust clinical speech model. Some conclusions are also not supported. The paper uses OPERA's weak results on the respiratory tasks as evidence that "narrow domain alignment" is not enough. However, these tasks use speech recordings, while OPERA was pretrained on breathing and cough sounds. Its weak results reflect a mismatch in input type, not a failure of domain alignment.
Questions for authors
- Please add a constant or majority-class baseline for every task, plus simple controls.
- The transfer figure reports the best of 12 models per pair, chosen on the target test set. Can you report a model chosen on source validation data, or the average over models?
- Many tasks have very few speakers. For example, TORGO has 15 speakers (only 3 test speakers per fold). How does this affect how much we can trust the results?
- The probe uses only the final layer, and wav2vec 2.0 and HuBERT are speech-recognition fine-tuned checkpoints, which tend to lose voice information in their last layers. Would using a weighted sum of layers, or the checkpoints without speech-recognition fine-tuning, change the ranking?
- What would be the insight or suggestion to the community moving towards more generalizable or practically useful clinical speech models?
6 comments
Transfer results for acoustic baselines (W1/ Q1). We additionally evaluate all three acoustic-feature models (LDA, linear probe, GBT) on every cross-category transfer task. The comparison includes the strongest encoder obtained retrospectively for each target, and the encoder selected using source validation alone (see Table R7). Each acoustic configuration is reported separately, without selecting its classifier based on target performance.
As expected, the pretrained encoders generally achieve stronger cross-category transfer than conventional acoustic features. Even when encoders are selected using source validation alone, their mean AUC is .602, compared with .526–.544 across the three eGeMAPS configurations. This pattern is consistent with a central motivation for large-scale pretraining: exposure to varied audio allows models to learn representations that can be reused across tasks and recording conditions. Our results demonstrate the usefulness of these representations when transferring a source-trained predictor to a different target collection without target-specific adaptation.
This advantage nevertheless varies across representations and transfer directions. For example, on P → F, eGeMAPS–LDA achieves .626, exceeding source-selected WavJEPA at .567, although another pretrained encoder achieves .876. Thus, broad pretraining provides an overall advantage in these experiments, while the choice of representation remains consequential. Acoustic baselines remain useful references for identifying where a particular pretrained representation fails to deliver that advantage. We will incorporate these findings into Section 5.2, provide full per-transfer results and uncertainty in Appendix F, and discuss the limits of attributing the observed advantage to pretraining scale or diversity in Section 6.
Table R7. All 12 cross-category transfer tasks. Each baseline cell reports pooled target AUC (ΔT, ΔS), where ΔT = baseline − target-selected encoder AUC and ΔS = baseline − source-selected encoder AUC. Differences are descriptive point estimates, not paired significance tests. C/F/N/P denote Conceptualization/Formulation/Neuromuscular/Phonatory–Respiratory. Source choices are fixed across targets: emotion2vec+/CLAP/Whisper/WavJEPA, respectively.
| Transfer | Target-selected T | Source-selected S | Linear AUC (ΔT, ΔS) | LDA AUC (ΔT, ΔS) | GBT AUC (ΔT, ΔS) |
|---|---|---|---|---|---|
| C → F | 0.6696 | 0.6107 | 0.5046 (-0.1650, -0.1061) | 0.5187 (-0.1509, -0.0920) | 0.4592 (-0.2104, -0.1515) |
| C → N | 0.6409 | 0.6186 | 0.6326 (-0.0083, +0.0140) | 0.6169 (-0.0240, -0.0017) | 0.5020 (-0.1389, -0.1166) |
| C → P | 0.6048 | 0.6048 | 0.4683 (-0.1365, -0.1365) | 0.4660 (-0.1388, -0.1388) | 0.4901 (-0.1147, -0.1147) |
| F → C | 0.7361 | 0.5007 | 0.4729 (-0.2632, -0.0278) | 0.5014 (-0.2347, +0.0007) | 0.5325 (-0.2036, +0.0318) |
| F → N | 0.8660 | 0.8660 | 0.5032 (-0.3628, -0.3628) | 0.5766 (-0.2894, -0.2894) | 0.5513 (-0.3147, -0.3147) |
| F → P | 0.5807 | 0.5425 | 0.4708 (-0.1099, -0.0717) | 0.5127 (-0.0680, -0.0298) | 0.5213 (-0.0594, -0.0212) |
| N → C | 0.5705 | 0.5574 | 0.5680 (-0.0025, +0.0106) | 0.5556 (-0.0149, -0.0018) | 0.5572 (-0.0133, -0.0002) |
| N → F | 0.8731 | 0.7772 | 0.5927 (-0.2804, -0.1845) | 0.6235 (-0.2496, -0.1537) | 0.4701 (-0.4030, -0.3071) |
| N → P | 0.5834 | 0.5407 | 0.4925 (-0.0909, -0.0482) | 0.4933 (-0.0901, -0.0474) | 0.4957 (-0.0877, -0.0450) |
| P → C | 0.8325 | 0.4503 | 0.4751 (-0.3574, +0.0248) | 0.5008 (-0.3317, +0.0505) | 0.5517 (-0.2808, +0.1014) |
| P → F | 0.8761 | 0.5672 | 0.6011 (-0.2750, +0.0339) | 0.6262 (-0.2499, +0.0590) | 0.6234 (-0.2527, +0.0562) |
| P → N | 0.7536 | 0.5928 | 0.5249 (-0.2287, -0.0679) | 0.5353 (-0.2183, -0.0575) | 0.5232 (-0.2304, -0.0696) |
W2. Interpreting model differences and OPERA’s pretraining alignment
SpeechDx evaluates the predictive performance of existing speech representations developed for different objectives and input distributions. As a datasets and benchmarks contribution, it provides a common framework for identifying their strengths, limitations, and sensitivity to evaluation choices. We included OPERA because its respiratory-acoustic pretraining makes it relevant to the question of whether representations learned from non-speech respiratory sounds transfer to conditions expressed in speech. OPERA’s original benchmark also evaluates modalities absent from pretraining, including sustained vowels for lung-function estimation. However, we agree that clinical-domain alignment and acoustic-input alignment are distinct. OPERA’s pretraining uses breathing, cough, and lung sounds, whereas SpeechDx evaluates speech. Features useful for non-speech respiratory events may differ from those required to discriminate clinical conditions in speech.
Consequently, input mismatch is a plausible contributor to OPERA’s results. Our comparison does not isolate its contribution from differences in training data, objectives, architecture, or representation extraction. We will replace the broad statement about “narrow domain alignment” with the more precise finding that OPERA-GT did not show a consistent advantage over other encoders on the evaluated phonatory/respiratory speech tasks under our linear probing protocol. These results characterize the transferability of this checkpoint in this setting; they do not establish that respiratory-domain pretraining is ineffective.
Insights beyond the leaderboard. Our additional analyses identify concrete factors that affect clinical-speech evaluation. First, performance depends on how pretrained representations are accessed, alongside which encoder is selected. For instance, Appendix G reports that replacing mean pooling with attentive statistical pooling (ASP) changes the best encoder for 19 of 27 tasks, despite keeping the encoder weights frozen. This motivates evaluating temporal aggregation and adaptation explicitly before attributing weak performance solely to an encoder’s pretraining objective. ASP adds trainable capacity and changes optimization; the comparison does not establish which physiological cues it captures.
Second, within-dataset performance and transferability are distinct properties. Across the 12 cross-category transfers, source-selected pretrained encoders achieve a mean AUC of 0.602, compared with 0.526, 0.544, and 0.523 for eGeMAPS with linear probing, LDA, and gradient boosting, respectively. This average advantage is consistent with the intended generalization benefits of pretrained representations, although our comparison does not isolate the contribution of pretraining scale or diversity. The advantage is not universal: each acoustic configuration outperforms the source-selected encoder on some transfer tasks, as reported in Table R7. These results motivate the development of representations whose strengths persist across cohorts and prediction targets, and of model-selection procedures that identify those strengths without target labels. Complete encoder results are in Table 6, Appendix F; Table R7 adds the acoustic and source-selected comparison, discussed further under Q2.
Third, improving encoder rankings does not necessarily mean improving clinically relevant prediction. The added constant baselines (Table R6) show that none of the 12 encoders improves on the training-median predictor on T8, T14, or T15. Meanwhile, competitive demographic baselines (Table R5) expose substantial label–population associations in some cohorts. These findings identify settings where a stronger encoder score alone provides insufficient evidence of useful progress. They motivate demonstrating gains over simple references and testing whether discrimination persists within demographic groups and across datasets. Complete baseline results will appear in Appendix I.
Together, these analyses identify concrete development priorities: improve access to clinically relevant information within representations, establish transfer beyond the training cohort, and demonstrate predictive value beyond simple acoustic and population-based references. Physiologically informed training is one promising hypothesis to evaluate against these requirements; our present results do not establish it as a solution. SpeechDx provides the standardized tasks, controls, and transfer evaluations needed to test whether future approaches deliver these improvements.
We will revise the OPERA interpretation and development implications in Section 5.1, and clarify their scope in Section 6.
W1/Q1. Reference baselines and controls
We thank the reviewer for suggesting these experiments. They substantially strengthen SpeechDx by making cohort-level associations more visible. We have added three baselines: a constant predictor, which uses no individual information; demographic predictors, which use age and/or sex without audio; and acoustic-feature models using openSMILE eGeMAPS with linear probing, LDA for classification or ridge regression for regression, and gradient-boosted trees. These comparisons establish reference points for interpreting predictive performance and identifying the advantages and limitations of the evaluated approaches. We will describe their protocols in Sections 4.2–4.3, incorporate the findings into Section 5.1, and provide full task-level results, uncertainty, and metadata coverage in Appendix I.
Table R5. Baseline coverage and classification summaries. Clinical means exclude the four auxiliary emotion tasks (T3–T6). The common 11-task summary aligns tasks with age/sex results; metadata exclusions may still change participant coverage.
Table R6. MAE for all six regression tasks (lower is better). The encoder row gives the lowest point estimate among all 12 encoders separately for each task. † denotes CV fold means under the current protocol; the probe estimates retain the validation-selection limitation explained under Q3.
I. Constant predictor. Classification scores use training-set class proportions, with the training majority class used for accuracy/F1; regression predicts the training-set median. Each cross-validation fold fits its constant using only its training partition. All 21 classification baselines have AUC .500, and 7 of 204 clinical encoder–task AUC point estimates fall below this reference.
For regression, Table R6 shows that no encoder improves on the constant MAE point estimate on T8, T14, or T15. On T8, the constant predictor achieves an MAE of 4.130, compared with 6.098 for the lowest-error encoder. Conversely, all encoders improve on the constant predictor on T2 and T11. These results identify tasks where the evaluated representations and probes reduce prediction error, alongside tasks where they have yet to demonstrate an improvement over assigning everyone the same value.
II. Demographic predictors. We additionally evaluate age-only, sex-only, and combined-demographic predictors wherever metadata are available. On T1, sex alone achieves AUC 0.671, compared with 0.656 for the highest-scoring encoder. On T7, combined demographics and Whisper both achieve 0.747, with age alone at 0.746. The task-level results underlying Table R5 therefore reveal cohorts in which substantial label predictability is present without using the recordings.
These findings identify where population composition warrants closer attention in dataset construction and evaluation. Performance in cohorts with different demographic distributions or label–demographic associations remains an empirical question, motivating matched, stratified, and external-cohort assessments. These associations may reflect disease-related population differences, recruitment practices, or both. Establishing additional value requires evaluating demographics with and without audio on the same participants, accounting for missing metadata.
III. Acoustic-feature models. We examined what conventional acoustic features achieve using the speech recordings. Table R5 reports mean AUCs of 0.660, 0.685, and 0.687 across the 17 clinical classification tasks for openSMILE eGeMAPS with linear probing, LDA, and gradient boosting, respectively. Task-level comparisons reveal substantial variation in both classification and regression results. On T17, WavLM achieves 0.856, compared with 0.726 for the highest-scoring acoustic feature model; on T8, eGeMAPS–GBT achieves lower MAE than every encoder (4.932 versus 6.098), while remaining worse than the constant predictor (4.130). Thus, established acoustic features can provide competitive predictions on some tasks, while pretrained representations achieve higher performance on others. Their relative strengths depend on the task and evaluated configuration.