acceptodds
Under review as a conference paper at ICLR 2027

SpeechDx: A Multi-Task Benchmark for Clinical Speech AI

Abstract

Speech offers a uniquely informative window into health by simultaneously engaging neurological, motor, respiratory, and vocal systems. Yet, current clinical speech AI methods have largely progressed through isolated condition-specific studies, making results difficult to compare and generalization difficult to assess. We introduce SpeechDx, a large-scale benchmark for clinical speech AI spanning 12 datasets and 27 tasks across diverse health conditions. To enable evaluation across shared clinical mechanisms, SpeechDx structures tasks by the stage of speech production they disrupt, namely conceptualization, formulation, and articulation. The benchmark tests generalization by including tasks with limited labeled data and evaluating the same health condition across multiple datasets, distinguishing clinically meaningful patterns from dataset artefacts. We systematically evaluate 12 state-of-the-art speech and audio encoders across all tasks and under zero-shot cross-condition transfer. Results show that large-scale speech models represent the strongest overall baselines, domain-specific models improve performance only on closely matched tasks, and no current representation generalizes reliably across the clinical speech landscape. SpeechDx establishes a shared evaluation framework for tracking progress toward general-purpose clinical speech representations.

Primary area:
datasets and benchmarks
Keywords:
clinical speech AI, speech biomarkers, audio foundation models, benchmark, linear probing, speech production taxonomy, generalization, zero-shot transfer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_nWKDOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a benchmark on Speech clinical AI, on 12 public datasets with 27 clinical speech tasks, grouped by the stage of speech production each condition affects. It evaluates 12 frozen audio encoders with linear probes, plus cross-dataset transfer. The finding is that no encoder works well across all conditions, large general speech models do best overall.

Critical strengths
  1. the benchmark provides a useful resource for the community, as a common protocol and released code makes future comparisons much easier. The task and models covered are quite board.
Critical weaknesses
  1. No naive baseline is provided as reference. Nor does the paper provide controls such as non-speech-only baseline, simple acoustic-feature baseline metadata-only baseline. Some results are in fact worse than a constant prediction.
  2. The evaluation confirms that existing models fall short on these tasks, but it offers little insight into why certain models do better on certain tasks, or how to build a more general and robust clinical speech model. Some conclusions are also not supported. The paper uses OPERA's weak results on the respiratory tasks as evidence that "narrow domain alignment" is not enough. However, these tasks use speech recordings, while OPERA was pretrained on breathing and cough sounds. Its weak results reflect a mismatch in input type, not a failure of domain alignment.
Questions for authors
  1. Please add a constant or majority-class baseline for every task, plus simple controls.
  2. The transfer figure reports the best of 12 models per pair, chosen on the target test set. Can you report a model chosen on source validation data, or the average over models?
  3. Many tasks have very few speakers. For example, TORGO has 15 speakers (only 3 test speakers per fold). How does this affect how much we can trust the results?
  4. The probe uses only the final layer, and wav2vec 2.0 and HuBERT are speech-recognition fine-tuned checkpoints, which tend to lose voice information in their last layers. Would using a weighted sum of layers, or the checkpoints without speech-recognition fine-tuning, change the ranking?
  5. What would be the insight or suggestion to the community moving towards more generalizable or practically useful clinical speech models?
6 comments

W1/Q1. Reference baselines and controls

We thank the reviewer for suggesting these experiments. They substantially strengthen SpeechDx by making cohort-level associations more visible. We have added three baselines: a constant predictor, which uses no individual information; demographic predictors, which use age and/or sex without audio; and acoustic-feature models using openSMILE eGeMAPS with linear probing, LDA for classification or ridge regression for regression, and gradient-boosted trees. These comparisons establish reference points for interpreting predictive performance and identifying the advantages and limitations of the evaluated approaches. We will describe their protocols in Sections 4.2–4.3, incorporate the findings into Section 5.1, and provide full task-level results, uncertainty, and metadata coverage in Appendix I.

Table R5. Baseline coverage and classification summaries. Clinical means exclude the four auxiliary emotion tasks (T3–T6). The common 11-task summary aligns tasks with age/sex results; metadata exclusions may still change participant coverage.

SummaryeGeMAPS lineareGeMAPS LDA / RidgeeGeMAPS GBTAge onlySex onlyCombined metadataConstant baseline
Number of classification tasks21212111202021
Number of regression tasks6661336
Clinical mean AUC: all 17 tasks0.6600.6850.687———0.5
Mean AUC: common 11 metadata-supported tasks0.6270.6540.6410.5870.5330.6010.5

Table R6. MAE for all six regression tasks (lower is better). The encoder row gives the lowest point estimate among all 12 encoders separately for each task. † denotes CV fold means under the current protocol; the probe estimates retain the validation-selection limitation explained under Q3.

MethodT2T8T11†T14†T15†T16†
Training-median constant5.6254.1301.4980.3410.3951.085
Lowest encoder MAE among all 125.2706.0980.4310.4060.4100.840
eGeMAPS + GBT5.1654.9320.4860.4220.4390.763

I. Constant predictor. Classification scores use training-set class proportions, with the training majority class used for accuracy/F1; regression predicts the training-set median. Each cross-validation fold fits its constant using only its training partition. All 21 classification baselines have AUC .500, and 7 of 204 clinical encoder–task AUC point estimates fall below this reference.

For regression, Table R6 shows that no encoder improves on the constant MAE point estimate on T8, T14, or T15. On T8, the constant predictor achieves an MAE of 4.130, compared with 6.098 for the lowest-error encoder. Conversely, all encoders improve on the constant predictor on T2 and T11. These results identify tasks where the evaluated representations and probes reduce prediction error, alongside tasks where they have yet to demonstrate an improvement over assigning everyone the same value.

II. Demographic predictors. We additionally evaluate age-only, sex-only, and combined-demographic predictors wherever metadata are available. On T1, sex alone achieves AUC 0.671, compared with 0.656 for the highest-scoring encoder. On T7, combined demographics and Whisper both achieve 0.747, with age alone at 0.746. The task-level results underlying Table R5 therefore reveal cohorts in which substantial label predictability is present without using the recordings.

These findings identify where population composition warrants closer attention in dataset construction and evaluation. Performance in cohorts with different demographic distributions or label–demographic associations remains an empirical question, motivating matched, stratified, and external-cohort assessments. These associations may reflect disease-related population differences, recruitment practices, or both. Establishing additional value requires evaluating demographics with and without audio on the same participants, accounting for missing metadata.

III. Acoustic-feature models. We examined what conventional acoustic features achieve using the speech recordings. Table R5 reports mean AUCs of 0.660, 0.685, and 0.687 across the 17 clinical classification tasks for openSMILE eGeMAPS with linear probing, LDA, and gradient boosting, respectively. Task-level comparisons reveal substantial variation in both classification and regression results. On T17, WavLM achieves 0.856, compared with 0.726 for the highest-scoring acoustic feature model; on T8, eGeMAPS–GBT achieves lower MAE than every encoder (4.932 versus 6.098), while remaining worse than the constant predictor (4.130). Thus, established acoustic features can provide competitive predictions on some tasks, while pretrained representations achieve higher performance on others. Their relative strengths depend on the task and evaluated configuration.

Transfer results for acoustic baselines (W1/ Q1). We additionally evaluate all three acoustic-feature models (LDA, linear probe, GBT) on every cross-category transfer task. The comparison includes the strongest encoder obtained retrospectively for each target, and the encoder selected using source validation alone (see Table R7). Each acoustic configuration is reported separately, without selecting its classifier based on target performance.

As expected, the pretrained encoders generally achieve stronger cross-category transfer than conventional acoustic features. Even when encoders are selected using source validation alone, their mean AUC is .602, compared with .526–.544 across the three eGeMAPS configurations. This pattern is consistent with a central motivation for large-scale pretraining: exposure to varied audio allows models to learn representations that can be reused across tasks and recording conditions. Our results demonstrate the usefulness of these representations when transferring a source-trained predictor to a different target collection without target-specific adaptation.

This advantage nevertheless varies across representations and transfer directions. For example, on P → F, eGeMAPS–LDA achieves .626, exceeding source-selected WavJEPA at .567, although another pretrained encoder achieves .876. Thus, broad pretraining provides an overall advantage in these experiments, while the choice of representation remains consequential. Acoustic baselines remain useful references for identifying where a particular pretrained representation fails to deliver that advantage. We will incorporate these findings into Section 5.2, provide full per-transfer results and uncertainty in Appendix F, and discuss the limits of attributing the observed advantage to pretraining scale or diversity in Section 6.

Table R7. All 12 cross-category transfer tasks. Each baseline cell reports pooled target AUC (ΔT, ΔS), where ΔT = baseline − target-selected encoder AUC and ΔS = baseline − source-selected encoder AUC. Differences are descriptive point estimates, not paired significance tests. C/F/N/P denote Conceptualization/Formulation/Neuromuscular/Phonatory–Respiratory. Source choices are fixed across targets: emotion2vec+/CLAP/Whisper/WavJEPA, respectively.

TransferTarget-selected TSource-selected SLinear AUC (ΔT, ΔS)LDA AUC (ΔT, ΔS)GBT AUC (ΔT, ΔS)
C → F0.66960.61070.5046 (-0.1650, -0.1061)0.5187 (-0.1509, -0.0920)0.4592 (-0.2104, -0.1515)
C → N0.64090.61860.6326 (-0.0083, +0.0140)0.6169 (-0.0240, -0.0017)0.5020 (-0.1389, -0.1166)
C → P0.60480.60480.4683 (-0.1365, -0.1365)0.4660 (-0.1388, -0.1388)0.4901 (-0.1147, -0.1147)
F → C0.73610.50070.4729 (-0.2632, -0.0278)0.5014 (-0.2347, +0.0007)0.5325 (-0.2036, +0.0318)
F → N0.86600.86600.5032 (-0.3628, -0.3628)0.5766 (-0.2894, -0.2894)0.5513 (-0.3147, -0.3147)
F → P0.58070.54250.4708 (-0.1099, -0.0717)0.5127 (-0.0680, -0.0298)0.5213 (-0.0594, -0.0212)
N → C0.57050.55740.5680 (-0.0025, +0.0106)0.5556 (-0.0149, -0.0018)0.5572 (-0.0133, -0.0002)
N → F0.87310.77720.5927 (-0.2804, -0.1845)0.6235 (-0.2496, -0.1537)0.4701 (-0.4030, -0.3071)
N → P0.58340.54070.4925 (-0.0909, -0.0482)0.4933 (-0.0901, -0.0474)0.4957 (-0.0877, -0.0450)
P → C0.83250.45030.4751 (-0.3574, +0.0248)0.5008 (-0.3317, +0.0505)0.5517 (-0.2808, +0.1014)
P → F0.87610.56720.6011 (-0.2750, +0.0339)0.6262 (-0.2499, +0.0590)0.6234 (-0.2527, +0.0562)
P → N0.75360.59280.5249 (-0.2287, -0.0679)0.5353 (-0.2183, -0.0575)0.5232 (-0.2304, -0.0696)

W2. Interpreting model differences and OPERA’s pretraining alignment

SpeechDx evaluates the predictive performance of existing speech representations developed for different objectives and input distributions. As a datasets and benchmarks contribution, it provides a common framework for identifying their strengths, limitations, and sensitivity to evaluation choices. We included OPERA because its respiratory-acoustic pretraining makes it relevant to the question of whether representations learned from non-speech respiratory sounds transfer to conditions expressed in speech. OPERA’s original benchmark also evaluates modalities absent from pretraining, including sustained vowels for lung-function estimation. However, we agree that clinical-domain alignment and acoustic-input alignment are distinct. OPERA’s pretraining uses breathing, cough, and lung sounds, whereas SpeechDx evaluates speech. Features useful for non-speech respiratory events may differ from those required to discriminate clinical conditions in speech.

Consequently, input mismatch is a plausible contributor to OPERA’s results. Our comparison does not isolate its contribution from differences in training data, objectives, architecture, or representation extraction. We will replace the broad statement about “narrow domain alignment” with the more precise finding that OPERA-GT did not show a consistent advantage over other encoders on the evaluated phonatory/respiratory speech tasks under our linear probing protocol. These results characterize the transferability of this checkpoint in this setting; they do not establish that respiratory-domain pretraining is ineffective.

Insights beyond the leaderboard. Our additional analyses identify concrete factors that affect clinical-speech evaluation. First, performance depends on how pretrained representations are accessed, alongside which encoder is selected. For instance, Appendix G reports that replacing mean pooling with attentive statistical pooling (ASP) changes the best encoder for 19 of 27 tasks, despite keeping the encoder weights frozen. This motivates evaluating temporal aggregation and adaptation explicitly before attributing weak performance solely to an encoder’s pretraining objective. ASP adds trainable capacity and changes optimization; the comparison does not establish which physiological cues it captures.

Second, within-dataset performance and transferability are distinct properties. Across the 12 cross-category transfers, source-selected pretrained encoders achieve a mean AUC of 0.602, compared with 0.526, 0.544, and 0.523 for eGeMAPS with linear probing, LDA, and gradient boosting, respectively. This average advantage is consistent with the intended generalization benefits of pretrained representations, although our comparison does not isolate the contribution of pretraining scale or diversity. The advantage is not universal: each acoustic configuration outperforms the source-selected encoder on some transfer tasks, as reported in Table R7. These results motivate the development of representations whose strengths persist across cohorts and prediction targets, and of model-selection procedures that identify those strengths without target labels. Complete encoder results are in Table 6, Appendix F; Table R7 adds the acoustic and source-selected comparison, discussed further under Q2.

Third, improving encoder rankings does not necessarily mean improving clinically relevant prediction. The added constant baselines (Table R6) show that none of the 12 encoders improves on the training-median predictor on T8, T14, or T15. Meanwhile, competitive demographic baselines (Table R5) expose substantial label–population associations in some cohorts. These findings identify settings where a stronger encoder score alone provides insufficient evidence of useful progress. They motivate demonstrating gains over simple references and testing whether discrimination persists within demographic groups and across datasets. Complete baseline results will appear in Appendix I.

Together, these analyses identify concrete development priorities: improve access to clinically relevant information within representations, establish transfer beyond the training cohort, and demonstrate predictive value beyond simple acoustic and population-based references. Physiologically informed training is one promising hypothesis to evaluate against these requirements; our present results do not establish it as a solution. SpeechDx provides the standardized tasks, controls, and transfer evaluations needed to test whether future approaches deliver these improvements.

We will revise the OPERA interpretation and development implications in Section 5.1, and clarify their scope in Section 6.

Reviewer_8os6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SpeechDx brings together 12 existing speech datasets covering 23 clinical tasks and four auxiliary emotion tasks. The authors group the tasks by stages of speech production and compare 12 pretrained audio encoders, mainly using frozen representations and linear probes. They also examine cross-dataset and cross-condition transfer, training-set size, attentive statistical pooling, and LoRA. The main contribution is a common evaluation setup for a fragmented field. Large general-purpose speech models perform well overall, but the best model varies considerably across tasks and evaluation settings.

Critical strengths
  1. Bringing these datasets under one evaluation protocol is useful. Comparisons across clinical speech studies are often difficult because the splits and training setups differ. The use of speaker-disjoint splits, including the correction to COVID-19 Sounds, is a particular strength.
  2. The additional experiments help explain where the baseline results do and do not hold. The ASP experiment is especially useful: changing the pooling method changes the best encoder on 19 of 27 tasks. This gives readers a reason to look beyond the main leaderboard.
Critical weaknesses
  1. I am not yet convinced by the interpretation of the cross-condition results. The common affected/control label covers diseases, symptoms, negative emotions, and disfluent utterances. These are quite different prediction targets. Section 5.2 pools datasets within each stage, but it is unclear how much of the reported AUC comes from within-dataset discrimination rather than differences in dataset identity, language, demographics, or recording conditions. Figure 2 also reports the best encoder for each pair based on target-set performance. This is useful as an exploratory result, but does not show what would happen if the encoder had to be chosen using source data alone. Appendix F's acoustic-hierarchy and screening interpretations therefore seem premature.
  2. Several evaluation details need to be resolved before I would trust the rankings. Some cohorts are very small, only one global seed is used, and the validation procedure within cross-validation folds is not clear. Transfer is tested on the complete target dataset, while the in-domain results use held-out splits or cross-validation; claims that one exceeds the other need a comparison on the same subjects. There are also reporting inconsistencies: Table 5 calls its intervals bootstrap 95% CIs but gives (0.44, 1.06) for T10/Qwen3 at 50% data, and Figure 4's regression signs do not agree with its caption and the improvement counts in Appendix G.
Questions for authors
  1. Is cross-stage AUC computed on pooled target recordings or averaged across datasets? Please report per-target results and check whether they remain strong after controlling for dataset identity and available demographic or recording variables.
  2. What happens when the encoder is selected using only source validation data, and transfer and in-domain models are evaluated on the same target test subjects?
  3. How are validation, hyperparameter selection, and early stopping handled within each outer fold? Are confidence intervals resampled by recording or by subject? Please also clarify the Table 5 intervals and Figure 4 sign convention.
  4. Are the main rankings stable across seeds, with clinical tasks only, after excluding known IEMOCAP pretraining overlap, and when using dataset-balanced scores or ASP?
8 comments
AuthorsOpenReview ↗

Response to Reviewer 8os6

We thank the reviewer for recognizing the value of a common evaluation protocol and the analyses beyond the main leaderboard. We also appreciate the questions identifying important distinctions between exploratory transfer, source-only encoder selection, dataset composition, and evaluation uncertainty. We have added source-validation encoder selection, per-target transfer results, evaluation on the target test participants used by the in-domain benchmark, six-seed reruns, and an age/sex-restricted transfer analysis. We address these concerns in the responses below.

W1, Q1–Q2 (1/2). Task definitions, aggregation, and selection rationale

Task definitions and aggregation (W1/Q1). The experiment evaluates whether a probe trained on one source-stage collection retains discrimination on different target tasks without target-label training. Mapping diseases, symptoms, emotions, and disfluencies to a common binary orientation permits this comparison while retaining their distinct task-specific meanings; it does not establish clinical equivalence. The production taxonomy organizes benchmark construction and analysis, but the transfer matrix does not independently establish a physiological hierarchy. We use the following notation throughout the text: C = Conceptualization, F = Formulation, N = Articulation–Neuromuscular, and P = Articulation–Phonatory/Respiratory.

The original cross-stage AUC is computed on pooled target recordings, rather than averaged across datasets, and therefore includes affected–control comparisons across datasets. We now report predictions separately for every target task and provide an equal-dataset mean AUC alongside the pooled AUC. Per-target evaluation excludes cross-dataset comparisons from the AUC calculation, while equal-dataset averaging prevents larger datasets from dominating the aggregate. Table R1 in the next response part reports all 12 cross-stage transfer experiments, and Table R2 reports the 24 comparable held-out evaluations; complete per-target results, including CV targets, will be included in Appendix F.

Source-only selection and within-dataset discrimination (W1/Q1/Q2). Figure 2 in the paper presents the highest target AUC among the 12 encoders to characterize the best transfer performance observed across the evaluated representations; results for all encoders are already provided in Appendix F. This descriptive comparison identifies transfer directions supported by at least one representation, but does not estimate performance when encoder selection must proceed without target labels. We therefore supplement it with source-only selection: for each source stage, we select the encoder with the highest source-validation AUC and evaluate that fixed encoder on all three target stages. Its per-target AUCs and equal-dataset averages address whether discrimination persists within individual target datasets.

Cross-stage discrimination remains evident under source-validation selection. For F → N, source-selected CLAP yields a pooled AUC of 0.866 (95% CI 0.796–0.921) and an equal-dataset mean AUC of 0.762 (0.698–0.823), with per-target AUCs ranging from 0.671 to 0.897. For N → F, source-selected Whisper yields a pooled AUC of 0.777 (0.743–0.811) and an equal-dataset mean AUC of 0.756 (0.720–0.793). These results demonstrate discrimination within individual target datasets in these directions, beyond comparisons across dataset identities.

The effect of encoder selection varies across directions. For P → F, the retrospectively target-selected encoder achieves a pooled AUC of 0.876, compared with 0.567 under source-validation selection; for P → C, the corresponding values are 0.832 and 0.450. Source-validation performance therefore does not consistently identify the representation with the strongest target discrimination. Table R1 distinguishes retrospectively observed transfer performance from performance obtained using source-only selection. C-source runs retain the auxiliary emotion tasks and the known emotion2vec+ pretraining exposure to IEMOCAP; the IEMOCAP-excluded ranking analysis below does not remove that exposure from these transfer runs.

Table R1. All 12 cross-stage transfer tasks across different encoder selection criteria, AUC aggregation techniques, and target populations. The target-selected column is the retrospective maximum pooled AUC on the full target collection. All other scores use the encoder selected by source-validation AUC, fixed across that source stage’s three target stages. Parentheses are participant-bootstrap 95% CIs. “Target-test” restricts tasks with held-out splits to their test participants; CV tasks retain all participants.

DirectionTarget-selected encoder / full pooled AUCSource-selected encoderFull pooled AUCFull equal-dataset AUCTarget-test pooled AUCTarget-test equal-dataset AUC
C → FAST / 0.670emotion2vec+0.611 (0.567, 0.654)0.566 (0.522, 0.611)0.611 (0.513, 0.706)0.610 (0.503, 0.710)
C → NAudioMAE / 0.641emotion2vec+0.619 (0.530, 0.708)0.636 (0.577, 0.693)0.619 (0.530, 0.708)0.636 (0.577, 0.693)
C → Pemotion2vec+ / 0.605emotion2vec+0.605 (0.594, 0.615)0.616 (0.603, 0.628)0.610 (0.588, 0.633)0.620 (0.592, 0.643)
F → Cemotion2vec+ / 0.736CLAP0.501 (0.477, 0.528)0.522 (0.490, 0.553)0.494 (0.471, 0.521)0.497 (0.439, 0.557)
F → NCLAP / 0.866CLAP0.866 (0.796, 0.921)0.762 (0.698, 0.823)0.866 (0.796, 0.921)0.762 (0.698, 0.823)
F → Pemotion2vec+ / 0.581CLAP0.542 (0.532, 0.553)0.539 (0.527, 0.551)0.550 (0.531, 0.569)0.543 (0.519, 0.565)
N → CCLAP / 0.570Whisper0.557 (0.539, 0.593)0.549 (0.521, 0.580)0.546 (0.529, 0.575)0.560 (0.498, 0.624)
N → FHuBERT / 0.873Whisper0.777 (0.743, 0.811)0.756 (0.720, 0.793)0.718 (0.610, 0.811)0.683 (0.589, 0.779)
N → Pemotion2vec+ / 0.583Whisper0.541 (0.529, 0.552)0.597 (0.583, 0.610)0.528 (0.504, 0.552)0.590 (0.564, 0.619)
P → Cemotion2vec+ / 0.832WavJEPA0.450 (0.424, 0.480)0.504 (0.470, 0.536)0.452 (0.426, 0.482)0.460 (0.407, 0.515)
P → FHuBERT / 0.876WavJEPA0.567 (0.509, 0.629)0.565 (0.516, 0.617)0.505 (0.360, 0.648)0.555 (0.423, 0.678)
P → NWhisper / 0.754WavJEPA0.593 (0.457, 0.715)0.599 (0.538, 0.662)0.593 (0.457, 0.715)0.599 (0.538, 0.662)

C = Conceptualization; F = Formulation; N = Articulation–Neuromuscular; P = Articulation–Phonatory/Respiratory. The preceding part explains the equal-dataset weighting conventions.

Age/sex-restricted evaluation (Q1). To test whether the transfer scores remain informative among demographically comparable speakers, we added a participant-level analysis for seven targets with usable age and sex metadata. We evaluated the source-validation-selected encoder for each source stage and included the second- and third-ranked encoders as supplementary sensitivity analyses; inclusion depends exclusively on source-validation performance.

We kept source-trained probes and predictions fixed and averaged recording-level probabilities per participant. Ordinary AUC and restricted concordance use the same metadata-eligible participants and each task’s own labels. Restricted concordance includes only affected–control pairs with the same recorded sex and an age difference ≤5 years, a prespecified threshold. COVID-19 Sounds instead requires the same recorded 10-year age bin, while AphasiaBank uses participants' mean age across visits. Participants with missing age/sex or recorded sex values “Other,” “pnts,” or “unknown” were excluded, as were Coswara participants with conflicting task labels. Eligible pairs receive equal weight, with ties contributing a score of 0.5. 95% CIs are estimated using 1,000 paired participant-bootstrap resamples per task, excluding draws with one class or no eligible pairs.

Table R3 reports all 63 comparisons. We observe that a probe trained on Whisper representations from neuromuscular articulation datasets achieves an ordinary AUC of 0.921 and restricted concordance of 0.918 (95% CI 0.855–0.962) on AphasiaBank. On AVFAD, the corresponding values are 0.685 and 0.705 (0.663–0.749). Greater sensitivity is observed in Coswara’s COVID-19 detection task (T25), with values of 0.630 and 0.576 (0.542–0.614). Across the 63 source-rank/target comparisons, the median absolute change is 0.010, with 50/63 changes ≤0.020.

These results show that transfer discrimination persists in age- and sex-restricted comparisons in specific settings, whereas demographic composition affects performance estimates in others. The analysis addresses measured age and sex within the included populations; it does not establish independence from recording conditions, language, or unmeasured characteristics. Because this analysis uses speaker-level scores and metadata-eligible cohorts, its values should not be directly equated with the recording-level or held-out-test results in Tables R1–R2.

Reviewer_6hoROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SpeechDx, a clinical speech benchmark comprising 27 tasks from 12 publicly available datasets. The authors organize these tasks according to different stages of speech production and evaluate 12 pretrained audio encoders using a standardized linear probing protocol, alongside cross-dataset and cross-condition transfer experiments and additional analyses.

The results suggest that large-scale general-purpose speech models achieve the strongest overall performance, while domain-specific models show advantages on selected tasks.

Critical strengths

The paper addresses the fragmentation of clinical speech evaluation by providing a unified evaluation pipeline, speaker-disjoint data splits, and reproducible experimental protocols. In particular, the authors' effort to correct speaker leakage in the original COVID-19 Sounds data splits is appreciated.

Organizing tasks according to stages of speech production provides a potentially useful framework for evaluating diverse clinical speech tasks. Furthermore, the inclusion of cross-dataset and cross-condition transfer experiments extends the analysis beyond conventional within-dataset performance comparisons.

Critical weaknesses

Although the speech-production taxonomy provides a theoretically motivated organization of clinical tasks, its additional explanatory value beyond dataset grouping remains unclear.

The task composition differs substantially across categories. For example, the phonatory/respiratory category is dominated by COVID-19 and symptom-related tasks, making it difficult to distinguish production-stage effects from dataset-specific characteristics such as recording conditions and label noise. The authors also acknowledge this potential confounding in Section 5.1.

While the paper reports category-level model rankings and cross-condition transfer results, these analyses do not clearly establish whether the proposed taxonomy explains differences in model performance or transferability. Since the taxonomy is presented as a central contribution, stronger evidence of its explanatory value would help establish the broader significance of the benchmark beyond standardizing existing datasets.

Questions for authors
  1. Could the authors further clarify the rationale behind the current task selection and composition, particularly the predominance of COVID-19 and symptom-related tasks in the phonatory/respiratory category? How does this choice affect the interpretation of category-level conclusions?

  2. The paper acknowledges pretraining overlap between emotion2vec+ and IEMOCAP. OPERA-GT also uses COVID-19 Sounds as a pretraining data source, although it uses cough and breathing recordings, whereas SpeechDx uses speech recordings from the same dataset. Such source-level overlap does not necessarily imply that the same recordings were included in pretraining, but it highlights the importance of systematically examining potential overlap. Could the authors clarify whether such overlaps were systematically examined across the evaluated models and datasets? For future models pretrained on some of the benchmark datasets, how should researchers exclude potentially overlapping tasks while maintaining fair and comparable evaluations across models?

Minor Issues

T1 (Table 3): The reported class distribution (24.0% healthy / 76.0% depressed) appears inconsistent with the mean PHQ-8 score of 6.94 and the depression threshold of 10. The authors are encouraged to verify these statistics.

T3 (Appendix B.1): The task description states that there are seven emotion categories, but eight are listed.

6 comments

We thank the reviewer for the constructive assessment and for recognizing the value of the standardized evaluation pipeline, speaker-disjoint splits, and transfer analyses. We address each concern below and describe additional sensitivity analyses conducted in response to the reviews.

W1/W3. Value of the speech-production framework

Our choice to frame SpeechDx around Berisha and Liss’ taxonomy follows established recommendations for constructing benchmarks. BetterBench [1] emphasizes defining the concepts being evaluated, explaining their translation into benchmark tasks, and grounding those decisions in domain literature. Gevers et al. [2] similarly recommend defining tasks through an explicit taxonomy of intended capabilities and assumptions that extends beyond surface-level task types. For SpeechDx specifically, the taxonomy offers two key benefits:

1. Stratified analysis and clinically motivated comparisons. The framework provides a consistent structure for investigating whether changes in representations, training objectives, and clinical coverage produce reproducible stage-level strengths and transfer relationships. Across six seeds, for example, CLAP achieves an overall MRR of 0.275 but a neuromuscular MRR of 0.463, while Qwen3 achieves its highest category-level MRR on phonatory/respiratory tasks (0.547). These profiles reveal variation obscured by a leaderboard combining all tasks; complete seed/category summaries will accompany Appendix D. They do not, however, establish production stage as an independent explanation of performance or transferability, given its association with dataset composition, recording conditions, and label characteristics.

2. Scaffolding for benchmark expansion. New datasets can be selected based on the processes and clinical conditions they add, as well as their population and acquisition characteristics. This helps prioritize additions that broaden within-stage diversity and enable better-matched comparisons across stages. As suitable datasets become available, SpeechDx can extend its clinical coverage while progressively separating production-stage associations from dataset-specific factors.

SpeechDx’s core contributions are dataset integration, clinically grounded task design, and standardized evaluation; encoder comparisons provide reference experiments. We will strengthen this motivation in the Introduction and Section 6, and revise Section 5.1 to distinguish observed category profiles from claims that production stage independently explains task difficulty or transferability.

References

  1. Reuel, Anka, et al. "Betterbench: Assessing AI benchmarks, uncovering issues, and establishing best practices." Advances in Neural Information Processing Systems 37 (2024): 21763-21813.

  2. Gevers, Ine, et al. "In Benchmarks We Trust... Or Not?." Proceedings of the 2025 conference on empirical methods in natural language processing. 2025.

W2/Q1. Task composition across categories

We selected datasets and tasks based on what is readily available and prevalent for clinical speech AI, using the speech-production framework to guide coverage across clinical processes. Our philosophy during this process was to mirror and organize the research landscape rather than manipulate it. The overrepresentation of COVID-19 tasks is a direct consequence of public research interest and the corresponding surge in the number of openly available telehealth datasets during the pandemic. Instead of discarding datasets to achieve artificial category balance, we retained them to enable comparisons not only across distinct diseases, but also across different population compositions, acquisition protocols, and recording environments.

Within the current benchmark, the speech-production framework organizes clinically motivated comparisons. Grouping conditions by their primary disruption of speech production supports interpretation of category-level performance patterns. We agree, however, that these patterns reflect both clinical processes and the characteristics of the available datasets. We will refine Section 5.1 to preserve this interpretation while distinguishing observed associations from production stage effects.

Sensitivity to dataset weighting. To assess whether datasets contributing multiple tasks disproportionately influence the overall comparison, we compute dataset-balanced MRR: task reciprocal ranks are averaged within each dataset, then across all 12 datasets with equal weight (see Table R8). This reduces the combined contribution of COVID-19 Sounds and Coswara from 29.6% (8/27 tasks) to 16.7% (2/12 datasets).

Table R8. Sensitivity of encoder rankings to dataset weighting. Lower ranks indicate better performance.

EncoderTask-weighted rankDataset-balanced rank
Whisper12
Qwen321
WavLM33
emotion2vec+55
AST44
CLAP67
WavJEPA88
MMS76
AudioMAE99
HuBERT1010
OPERA-GT1111
wav2vec 2.01212

The same three encoders occupy the leading positions, although Whisper and Qwen3 exchange places. This supports stability of the leading set under the specified change in dataset weighting. Intermediate ranking changes motivate reporting both aggregation schemes. We will include this analysis in Section 5.1.

Q2 (1/3). Pretraining overlap

The submission reports training sources for all 12 encoders and acknowledges the emotion2vec+–IEMOCAP overlap. Table R9 below consolidates a documentation-based audit, including relevant fine-tuning and parent/teacher lineage. Here, “upstream” covers training before the SpeechDx probe. This review identifies documented intersections and unresolved membership; it is not a recording- or participant-level verification of disjointness.

Table R9. Documentation-based audit of upstream exposure. “No reported SpeechDx corpus intersection” means no benchmark dataset is explicitly identified in the documented sources; it does not certify absence of duplicate recordings or participants.

EncoderReported training sources or lineageAssessment relative to SpeechDx
wav2vec 2.0Libri-Light; LibriSpeech ASR fine-tuningNo reported SpeechDx corpus intersection.
HuBERTLibri-Light; LibriSpeech ASR fine-tuningNo reported SpeechDx corpus intersection.
WavLMLibri-Light, GigaSpeech, VoxPopuliNo reported SpeechDx corpus intersection
MMSMMS corpora, VoxPopuli, MLS, VoxLingua-107, Common Voice, BABELNo reported SpeechDx corpus intersection
Qwen3Codec training with a WavLM semantic teacherNo SpeechDx source identified; tokenizer-specific membership is insufficiently disclosed to exclude exposure.
WhisperWeakly labeled and pseudo-labeled audio; large-v2 pseudo-labeling lineageTraining-recording membership unavailable; SpeechDx exposure unresolved.
AudioMAEAudioSetNo reported SpeechDx corpus intersection.
WavJEPAAudioSet, simulated acoustic scenes, WHAMR! noiseNo reported SpeechDx corpus intersection.
ASTImageNet initialization; supervised AudioSet trainingNo reported SpeechDx corpus intersection.
CLAPLAION-Audio-630K and AudioSet, including audio-encoder initializationNo reported SpeechDx corpus intersection
emotion2vec+Academic emotion-data lineage followed by pseudo-labeled trainingAcknowledged IEMOCAP overlap: T5–T6.
OPERA-GTCOVID-19 Sounds cough/breath, UK COVID-19, COUGHVID, ICBHI, HF LungShared COVID-19 Sounds source: T19–T23. SpeechDx participant overlap remains unverified across modalities and OPERA’s held-out exclusions.

OPERA-GT illustrates why source-level overlap alone is insufficient to determine the evaluation’s interpretation. Its reported pretraining uses cough/breath recordings, whereas SpeechDx uses speech from COVID-19 Sounds. A different input modality does not establish participant-level disjointness. OPERA reports holding out its own downstream test data before pretraining, but this does not verify alignment with SpeechDx’s evaluation participants. We therefore flag the shared source without asserting that SpeechDx’s speech recordings were used upstream. We will add these exposure qualifications to Section 6 and the model documentation in Appendix C.

Comments on the paper

2 comments

Dear Reviewer,

Thank you very much for your time and thoughtful review. As the discussion deadline approaches, could you please take a look at the authors' rebuttal and share your thoughts in the discussion?

Many thanks for your continued help!

AuthorsOpenReview ↗

General Response to All Reviewers

We thank the reviewers for their constructive feedback and for recognizing the value of SpeechDx’s common evaluation framework, broad task coverage, and analyses beyond the main leaderboard. The questions have helped us strengthen the evaluation and clarify important challenges for clinical-speech research. SpeechDx integrates 12 datasets into 23 clinical tasks and four auxiliary emotion tasks. Its core contributions are dataset integration, clinically grounded task design, and standardized evaluation; the evaluated encoders provide reproducible reference experiments.

In response, we added constant, demographic and acoustic controls; five additional probe-training seeds; source-validation encoder selection; per-target and equal-dataset transfer analyses; comparisons on matched target test participants; and age/sex-restricted transfer evaluation. We also examined ranking sensitivity and audited documented upstream training sources. These analyses establish where observed findings persist and where they depend on model selection, benchmark composition, or evaluation choices. We acknowledge the remaining limitations, including small cohorts and unresolved upstream training exposure, and will make their implications explicit in the manuscript.

The encoder results characterize particular checkpoints and evaluation configurations. Their rankings can change as representations, training data, and methods for accessing encoder information evolve. SpeechDx’s lasting value is the shared resource through which these developments can be assessed: common clinical tasks, participant partitions, evaluation procedures, and reference controls. The speech-production framework supports clinically motivated benchmark construction, stratified analysis, and structured expansion.

The reviews also highlight substantive research opportunities. Sensitivity to pooling and checkpoint choice, competitive simple baselines, and differences between source-selected and retrospectively target-selected transfer identify capabilities that future models and evaluation methods should address. These findings motivate developing models that demonstrate additional value beyond demographic and acoustic references, perform reliably on unseen speakers and independently collected cohorts, and support model selection without final-target labels. Physiological and clinical grounding provides a motivated direction for investigating these capabilities.

We therefore view the additional analyses as strengthening both the benchmark and its research agenda. By making current strengths, dependencies, and unresolved challenges measurable within a common framework, SpeechDx provides a foundation for fairer comparisons and more effective clinical-speech models. We encourage the community to build on these reference experiments, broaden clinical and population coverage, and demonstrate reproducible progress across complementary evaluation settings.

The table below identifies tables across all responses. R9–R10 concern documentation and reporting guidance, rather than new experiment results.

TableExperiment or analysisDiscussed in responses to
R1Source-validation encoder selection and pooled/equal-dataset transfer summaries8os6 W1/Q1–Q2; nWKD Q2
R2Transfer and in-domain comparison on the same target participants: all 24 held-out comparisons8os6 Q2
R3Age/sex-restricted transfer: 63 source-rank/target comparisons on seven tasks8os6 Q1
R4Six-seed reruns and ranking sensitivity to clinical-only tasks, IEMOCAP exclusion, dataset weighting and ASP8os6 Q4; nWKD Q3; 6hoR W1/W3 (seed profiles)
R5Constant, demographic and eGeMAPS controls: coverage and classification summariesnWKD W1/Q1, W2, Q5
R6Regression controls: all six tasks, including training-median constantsnWKD W1/Q1, W2
R7All 12 cross-category acoustic-transfer comparisons against target- and source-selected encodersnWKD W1/Q1, W2, Q5
R8Task-weighted versus dataset-balanced ranking summary6hoR W2/Q1; related analysis in 8os6 Q4
R9Documentation-based audit of training sources and lineage for all 12 encoders6hoR Q2
R10Upstream-exposure scenarios and the evaluation claims they permit6hoR Q2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.