- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies a clinical ASR pipeline where ASR backends independently transcribe speech chunks, and an LLM reconciles their outputs using clinical and document context. The proposed training pipeline uses synthetic clinical scripts, text-to-speech, acoustic perturbations, production ASR hypotheses, and teacher-generated targets. Eval is conducted on 226 synthetic clinical encounters and a separate 61-case behavioral suite.
Critical strengths
-
Medical ASR is a practical problem because general-domain ASR might not work very well in clinical settings where clinical terms might give some more difficulties for the joint Speech-Language models to transcribe well
-
Self-hostable and real-time ASR is a interesting approach because in clinical settings users might not want their datasets uploaded online which violates patients-doctors privacy
Critical weaknesses
I suspect this paper is an AI slop, based on some main points here:
- Related works are not sufficiently included. The authors conduct study on medical ASR but there is no work on medical ASR cited. I asked ChatGPT and it easily gave me many publicly available datasets in just 1 min. For example:
Publicly available medical ASR datasets
-
VietMed: A Dataset and Benchmark for Automatic Speech Recognition of Vietnamese in the Medical Domain: https://arxiv.org/abs/2404.05659
-
MultiMed: Multilingual Medical Speech Recognition via Attention Encoder Decoder: https://arxiv.org/abs/2409.14074
Real-time medical ASR setting
- Real-time Speech Summarization for Medical Conversations: https://arxiv.org/abs/2406.15888
-
The synthetic generation of the dataset is very naive and thus makes almost no real-world relevant. The authors put some dictation script and used synthetic TTS to generate synthetic audio files, and then augmented with noise and pertubations. Such medical ASR dataset is very far from real-world settings where noise, speaker speed and vocab is very diverse and varying (see the real-world datasets by VietMed and MultiMed).
-
The experimental results are not convincing if it is implemented in real-world hospitals. As I can see in main Table 2, WER is below 0.1xxx. Such perfect results are generated simply based on the synthetic dataset which is simply a reverse-engineered of the same frontier models. So I'm not surprised about such low error rates reported in the paper.
Questions for authors
N/A
2 comments
Thank you to the authors for taking the time to prepare a thoughtful rebuttal!
I appreciate the authors' efforts, but I feel that only W3 adequately addresses my concerns.
Regarding W1, I believe that when working on a domain X, it is important to discuss and cite relevant prior work in that domain, so that readers can understand the current state of the art and the research directions being explored by the community. Furthermore, when proposing a new method Y, I would expect evaluations on both the newly introduced benchmark and existing benchmarks in X to demonstrate the generalizability of the proposed method (VietMed and MultiMed in your case).
For these reasons, I will respectfully maintain my original score. That said, I hope the authors will not be discouraged by this assessment. I appreciate the effort that went into this work and encourage the authors to continue developing and improving it. I wish them all the best with their future research!
Question 1: Related work
Thank you for these suggestions. VietMed and MultiMed focus on multilingual medical ASR datasets and recognition models, while the third work addresses clinical speech summarization. Our paper instead studies reconciliation of multiple ASR outputs and the distillation of a teacher model into a student reconciler.
We focus our related work discussion on multi-backend ASR reconciliation and generative ASR error correction, which are most directly relevant to our research question (Section 2.1). Additionally, we do not focus on multilingual ASR in this paper, but agree that it would be an interesting direction for future work, as noted in Section 7 (Discussion).
Question 2: Real-world relevance of synthetic data
We agree that TTS cannot fully reproduce the acoustic diversity of real clinical speech. However, we use synthetic audio for training supervision, not evaluation (Sections 4–5). We also apply acoustic augmentation to introduce more diverse ASR disagreements, including noise, reverberation, and speed perturbations (Section 4; Appendix D), and evaluate its contribution through a controlled ablation (Section 6.4).
Importantly, all models are evaluated on the same 226 held-out real clinical encounters, spanning 15 clinicians and 28 specialties, with no clinician overlap between training and evaluation (Section 5, Datasets; Table 2).
We acknowledge that synthetic speech does not fully capture real-world acoustic variability, but our evaluation on held-out real clinical speech directly measures how well synthetic supervision transfers to the target setting.
Question 3: Real-world validity of the experimental results
Thank you for raising this concern. To clarify, all results in Table 2 are evaluated on real clinical speech. ClinStandard comprises 226 held-out real clinical encounters from 15 clinicians across 28 specialties, with independently annotated references and no clinician overlap with training (Section 5, Datasets; Table 2).
We have clarified this distinction in the manuscript and specified that WER values are proportions, not percentages (e.g., 0.1336 corresponds to 13.36% WER; Section 5, Metrics; Table 2 caption).
While these results demonstrate performance on held-out real clinical speech, we do not claim generalization to all hospitals or acoustic environments (Section 7, Limitations).