- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
"HealthTTS: Symptom-Aware Text-to-Speech with Clinical Acoustic Cues" tackles the problem of generating realistic synthetic speech data for the clinical setting. In particular, they focus on the injection of 7 acoustic elements symptom elements such as coughing and sneezing. The paper shows that their symptom-aware TTS model named HealthTTS outperforms existing TTS models across a variety of dimensions including realism and speech naturalness. Their contributions are threefold: 1) Formulate this TTS problem as a structured generation problem 2) Construct a training an evaluation set named SymCue and SumCueBench 3) Introduce HealthTTS which is constructed in a novel 3-stage process. The paper's model outperforms the other chosen baseline models across the reported metrics and they show the importance of each stage in isolation.
Critical strengths
1: Strong motivation - The paper addresses a real limitation of general purpose TTS. There is a difficulty in reliably generating synthetic speech with real acoustic-aware symptoms while staying true to natural speech. There is a a well-defined distinction between structured cue control and naturally perceived speech. The authors additionally tie back in their motivation with the integration virtualized standard patient applications. They do a good job of explaining why non-verbal cues are important to healthcare interactions.
- Novel 3-stage methodology - The proposed training procedure separates cue-sequence learning, acoustic adaptation, and global preference refinement. The ablation results provide additional evidence that these components are individually helpful in strengthening the overall system. For example, removing LM cue weighting reduces Cue Sequence Accuracy by about 16%, while removing flow weighting primarily affects perceptual quality. These are intuitive findings as well. This is a meaningful result because it supports the proposed decomposition of the training process. I would also consider the construction of SymCue and SymCueBench a third strength, although the dataset’s limitations should be acknowledged.
Critical weaknesses
-
Reliance on Automated Evals - The paper relies extensively on Qwen3-Omni throughout its methodology, including dataset verification, temporal annotation, preference-pair construction, and final evaluation. While the authors report human–model agreement on a subset of generated utterances, the validation is relatively limited, allocating little discussion and analysis. In particular, given the subjective nature of metrics such as Cue Realism and Speech Naturalness this would be extra important. The authors do not sufficiently justify their choice of Qwen3-Omni or investigate whether the automated evaluator exhibits systematic biases across different synthesis models. A more detailed analysis of human–model agreement, including system-level correspondence and potential evaluator biases, would strengthen confidence in the reported improvements.
-
Limited Generalizability and Insufficient Justification of Design Choices - The evaluation is based on a relatively small and highly controlled speech resource. Although the training and evaluation partitions use disjoint texts and recordings, they share the same six speaker identities, leaving generalization to unseen speakers untested. Furthermore, several experimental choices are insufficiently motivated, including the selection of seven cue categories, the use of Qwen3-TTS for dataset construction, and the choice of CosyVoice3 as the backbone. The paper would benefit from a clearer justification of these choices and a more detailed discussion of how the proposed system might generalize to naturally occurring patient speech and more diverse speakers. These two weaknesses are more consequential than presentation issues because they directly concern the reliability and scope of the experimental evidence.
Questions for authors
-
Given that Qwen3-Omni is used for data verification, temporal annotation, preference construction, and final evaluation, can the authors clarify how they ensure that the reported improvements are not influenced by systematic evaluator biases? In particular, do the human validation results demonstrate consistent agreement across the different synthesis models, rather than only aggregate correlation? Finally, why is Qwen3-Omni the right model choice here, especially since Qwen3-TTS is from the same model family?
-
Since SymCue and SymCueBench share the same six speaker identities, can the authors clarify what conclusions can be drawn about generalization to unseen speakers? Is there any evidence from the existing experiments that HealthTTS can reliably generate symptom-related cues for speakers outside the training pool?
-
Can the authors clarify what exactly constitutes the “Cue Insertion” baseline and how it differs from the cue-insertion procedure used to construct the training data? Given its substantially lower Speech Naturalness, how do the authors ensure that it provides a fair comparison to HealthTTS?
4 comments
Q2: Generalization to Unseen Speakers
To examine transferability beyond the original recording pool, we conducted an additional evaluation using four reference speakers (two male and two female) outside the SymCue recording pool and HealthTTS fine-tuning data. The evaluation covered 126 requests across all seven cue types, with three generation seeds, yielding 378 utterances.
| Speaker Pool | Cue Match (%) | Cue Seq. Acc. (%) | WER (%) | Spk. Sim. | Speech Nat. | Cue Realism |
|---|---|---|---|---|---|---|
| Seen speakers (full SymCueBench) | 83.87 | 79.41 | 7.33 | 0.87 | 4.13 | 4.84 |
| Four additional speakers | 82.54 | 77.57 | 7.48 | 0.81 | 3.86 | 4.80 |
| Relative Change (%) | ↓1.59% | ↓2.32% | ↑2.05% | ↓6.90% | ↓6.54% | ↓0.83% |
Relative changes are calculated with respect to the seen-speaker results. An increase in WER indicates degradation.
The results show that HealthTTS maintains relatively consistent performance on the four additional speakers, with relative changes below 2.5% in Cue Match, Cue Sequence Accuracy, WER, and Cue Realism. Although Speaker Similarity and Speech Naturalness show somewhat larger changes, the overall results suggest that HealthTTS generalizes reasonably well to reference voices outside the original recording pool.
Q3: Cue Insertion Baseline and Comparison Fairness
To clarify the baseline construction and comparison fairness, we provide a detailed comparison between the Cue Insertion evaluation procedure and the SymCue training-data construction pipeline.(Appendix D.1).
| Processing | SymCue Construction | Cue Insertion Baseline |
|---|---|---|
| Pause adjustment | Adaptive pauses (0.1–0.8 s) | Up to 80 ms when necessary |
| Cross-fading | 40–65 ms | 40 ms |
| Room tone | Matching and adjustment | Local loudness normalization |
| Quality control | Multi-round verification and reconstruction | Fixed insertion pipeline |
SymCue construction applies more extensive acoustic tuning to produce high-quality paired training data, whereas Cue Insertion represents a practical post-hoc editing baseline. Both use recorded symptom cues, but their base speech and acoustic processing procedures differ.
Cue Insertion achieves 95.71% symptom-cue recognition and 4.62 Cue Realism, but substantially lower Speech Naturalness (1.55 vs. 4.13 for HealthTTS). These results indicate that, under the evaluated settings, HealthTTS achieves more natural integration of symptom cues into surrounding speech. We therefore interpret Cue Insertion as a practical baseline rather than an upper bound on post-hoc editing.
Clarification of Design Choices
We further clarify the rationale behind the three design choices raised by the reviewer.
- Seven cue categories: We select seven representative transient, non-lexical vocalizations relevant to patient simulation, covering respiratory cues, throat/physiological cues, and distress-related vocalizations. These categories capture representative symptom manifestations within the scope of this study.
- Qwen3-TTS for data construction: It provides speaker-conditioned base-speech synthesis for constructing paired cue-free and cue-conditioned utterances. Its zero-shot voice cloning and speech generation capabilities support the construction of consistent base speech [6].
- CosyVoice3 as the backbone: Its reference-conditioned generation and separate autoregressive language-model and Flow Matching modules support our sequential adaptation of cue-sequence generation and acoustic realization [4].
We thank the reviewer again for these valuable suggestions, which have helped us clarify our model selection, experimental design, and the scope of our contributions.
References
[4] Du, Z., et al. CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training. 2025.
[5] Xu, J., et al. Qwen3-Omni Technical Report. arXiv:2509.17765, 2025.
[6] Hu, H., et al. Qwen3-TTS Technical Report. arXiv:2601.15621, 2026.
Thank you for answering my questions! I took a look at the paper to see if the information presented in the answers was included in a newer version of the paper, but it was not (unless I missed something). I think it would be good to include this in the paper (at least as appendices). I will keep my score.
Thank you for pointing this out. We have uploaded a revised version of the manuscript that incorporates the additional details and experimental results discussed in our rebuttal responses, including the unseen-speaker evaluation, annotation agreement analysis, stage-wise comparisons, and additional implementation details. These have been included in the main text and appendices where appropriate.
We appreciate your helpful suggestion, which has helped us improve the completeness and clarity of the manuscript.
Dear Reviewer wwpA,
Thank you for your constructive feedback and positive evaluation. We address your concerns regarding evaluator reliability, speaker generalization, baseline fairness, and design choices below.
Q1: Evaluator Reliability and Model Selection
We address the evaluator-dependence concern through three aspects: model selection, blinded evaluation, and independent human validation.
Model selection. We selected Qwen3-Omni for automatic evaluation because of its strong general audio-understanding capabilities. Its technical report demonstrates competitive performance across 36 audio and audio-visual benchmarks, achieving open-source state-of-the-art results on 32 of them [5]. These capabilities make it suitable for identifying non-verbal symptom cues and assessing perceptual speech quality.
Qwen3-TTS serves a different purpose: generating base speech during dataset construction. Its technical report demonstrates strong zero-shot voice cloning, speaker consistency, and controllable speech generation [6], making it suitable for constructing speaker-conditioned, cue-free and cue-conditioned speech pairs. Although both models belong to the Qwen family, they serve distinct generation and evaluation roles.
Blinded evaluation. In human evaluation, the two medical student annotators are unaware of whether an utterance originates from a human or an AI system and do not know the identity of the TTS model. In automatic evaluation, Qwen3-Omni receives generated audio and fixed evaluation instructions without access to the requested cue configuration or generation model identity. Speaker-reference audio is additionally provided for Speaker Similarity.
Human–Omni agreement. Two medical students independently annotated 140 utterances. We compare human–human agreement with human–Omni correspondence using the same evaluation metrics:
We further examine Human–Omni agreement across TTS systems and cue categories using exact cue-sequence agreement and perceptual-rating MAE (Mean Absolute Error).
By TTS model (N=140):
By cue category (N=20 per category):
Across the audited samples, Qwen3-Omni shows substantial correspondence with human judgments. Exact sequence agreement ranges from 84.0% to 88.6% across TTS models and from 85.0% to 90.0% across cue categories. Perceptual-rating MAEs also remain relatively low across the evaluated subgroups. These results further support the reliability of Qwen3-Omni as an automatic evaluator.