acceptodds
Under review as a conference paper at ICLR 2027

HealthTTS: Symptom-Aware Text-to-Speech with Clinical Acoustic Cues

13 views

Abstract

Speech in healthcare interactions conveys not only verbal content but also symptom-related acoustic cues, such as coughs, throat clearing, and sighs. However, our evaluation of representative text-to-speech (TTS) models shows that existing models often fail to reproduce the requested cue type, count, and order, while even correct cue sequences may not integrate naturally into speech. To address these challenges, we propose **HealthTTS**, a symptom-aware TTS model that directly generates complete utterances from text, a speaker reference, and an optional cue configuration. HealthTTS follows a three-stage training procedure that combines cue-aware language-model adaptation, localized acoustic adaptation, and global preference refinement to improve both cue generation and natural cue–speech integration. We construct SymCue, a paired speech resource for training, and **SymCueBench**, an evaluation benchmark covering seven symptom-related acoustic cue types. Under the shared-cue setting, HealthTTS achieves 87.3% Cue Sequence Accuracy, outperforming Fish Speech S2 Pro by 25.1 percentage points while maintaining high Speech Naturalness and Cue Realism. Beyond benchmark evaluation, a virtual standardized patient study with 30 medical students across 14 scenarios shows that HealthTTS achieves higher Patient Realism and Perceived Usefulness than Cue Insertion, a post-hoc insertion reference. To support future research and the open-source community, we will make our code publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Symptom-aware Text-to-Speech, Controllable Speech Generation, Paralinguistic Speech Synthesis, Preference Optimization, Healthcare Interaction, Virtual Standardized Patients

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wwpAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

"HealthTTS: Symptom-Aware Text-to-Speech with Clinical Acoustic Cues" tackles the problem of generating realistic synthetic speech data for the clinical setting. In particular, they focus on the injection of 7 acoustic elements symptom elements such as coughing and sneezing. The paper shows that their symptom-aware TTS model named HealthTTS outperforms existing TTS models across a variety of dimensions including realism and speech naturalness. Their contributions are threefold: 1) Formulate this TTS problem as a structured generation problem 2) Construct a training an evaluation set named SymCue and SumCueBench 3) Introduce HealthTTS which is constructed in a novel 3-stage process. The paper's model outperforms the other chosen baseline models across the reported metrics and they show the importance of each stage in isolation.

Critical strengths

1: Strong motivation - The paper addresses a real limitation of general purpose TTS. There is a difficulty in reliably generating synthetic speech with real acoustic-aware symptoms while staying true to natural speech. There is a a well-defined distinction between structured cue control and naturally perceived speech. The authors additionally tie back in their motivation with the integration virtualized standard patient applications. They do a good job of explaining why non-verbal cues are important to healthcare interactions.

  1. Novel 3-stage methodology - The proposed training procedure separates cue-sequence learning, acoustic adaptation, and global preference refinement. The ablation results provide additional evidence that these components are individually helpful in strengthening the overall system. For example, removing LM cue weighting reduces Cue Sequence Accuracy by about 16%, while removing flow weighting primarily affects perceptual quality. These are intuitive findings as well. This is a meaningful result because it supports the proposed decomposition of the training process. I would also consider the construction of SymCue and SymCueBench a third strength, although the dataset’s limitations should be acknowledged.
Critical weaknesses
  1. Reliance on Automated Evals - The paper relies extensively on Qwen3-Omni throughout its methodology, including dataset verification, temporal annotation, preference-pair construction, and final evaluation. While the authors report human–model agreement on a subset of generated utterances, the validation is relatively limited, allocating little discussion and analysis. In particular, given the subjective nature of metrics such as Cue Realism and Speech Naturalness this would be extra important. The authors do not sufficiently justify their choice of Qwen3-Omni or investigate whether the automated evaluator exhibits systematic biases across different synthesis models. A more detailed analysis of human–model agreement, including system-level correspondence and potential evaluator biases, would strengthen confidence in the reported improvements.

  2. Limited Generalizability and Insufficient Justification of Design Choices - The evaluation is based on a relatively small and highly controlled speech resource. Although the training and evaluation partitions use disjoint texts and recordings, they share the same six speaker identities, leaving generalization to unseen speakers untested. Furthermore, several experimental choices are insufficiently motivated, including the selection of seven cue categories, the use of Qwen3-TTS for dataset construction, and the choice of CosyVoice3 as the backbone. The paper would benefit from a clearer justification of these choices and a more detailed discussion of how the proposed system might generalize to naturally occurring patient speech and more diverse speakers. These two weaknesses are more consequential than presentation issues because they directly concern the reliability and scope of the experimental evidence.

Questions for authors
  1. Given that Qwen3-Omni is used for data verification, temporal annotation, preference construction, and final evaluation, can the authors clarify how they ensure that the reported improvements are not influenced by systematic evaluator biases? In particular, do the human validation results demonstrate consistent agreement across the different synthesis models, rather than only aggregate correlation? Finally, why is Qwen3-Omni the right model choice here, especially since Qwen3-TTS is from the same model family?

  2. Since SymCue and SymCueBench share the same six speaker identities, can the authors clarify what conclusions can be drawn about generalization to unseen speakers? Is there any evidence from the existing experiments that HealthTTS can reliably generate symptom-related cues for speakers outside the training pool?

  3. Can the authors clarify what exactly constitutes the “Cue Insertion” baseline and how it differs from the cue-insertion procedure used to construct the training data? Given its substantially lower Speech Naturalness, how do the authors ensure that it provides a fair comparison to HealthTTS?

4 comments

Dear Reviewer wwpA,

Thank you for your constructive feedback and positive evaluation. We address your concerns regarding evaluator reliability, speaker generalization, baseline fairness, and design choices below.

Q1: Evaluator Reliability and Model Selection

We address the evaluator-dependence concern through three aspects: model selection, blinded evaluation, and independent human validation.

Model selection. We selected Qwen3-Omni for automatic evaluation because of its strong general audio-understanding capabilities. Its technical report demonstrates competitive performance across 36 audio and audio-visual benchmarks, achieving open-source state-of-the-art results on 32 of them [5]. These capabilities make it suitable for identifying non-verbal symptom cues and assessing perceptual speech quality.

Qwen3-TTS serves a different purpose: generating base speech during dataset construction. Its technical report demonstrates strong zero-shot voice cloning, speaker consistency, and controllable speech generation [6], making it suitable for constructing speaker-conditioned, cue-free and cue-conditioned speech pairs. Although both models belong to the Qwen family, they serve distinct generation and evaluation roles.

Blinded evaluation. In human evaluation, the two medical student annotators are unaware of whether an utterance originates from a human or an AI system and do not know the identity of the TTS model. In automatic evaluation, Qwen3-Omni receives generated audio and fixed evaluation instructions without access to the requested cue configuration or generation model identity. Speaker-reference audio is additionally provided for Speaker Similarity.

Human–Omni agreement. Two medical students independently annotated 140 utterances. We compare human–human agreement with human–Omni correspondence using the same evaluation metrics:

DimensionHuman–Human AgreementHuman–Omni Agreement
Cue Sequence (Exact Match %)89.29%86.79%
Speech Naturalness (Pearson's rr)0.820.77
Cue Realism (Pearson's rr)0.840.81
Speaker Similarity (Pearson's rr)0.830.80

We further examine Human–Omni agreement across TTS systems and cue categories using exact cue-sequence agreement and perceptual-rating MAE (Mean Absolute Error).

By TTS model (N=140N=140):

ModelNSequence Agreement (%)Naturalness MAERealism MAESpeaker Sim. MAE
HealthTTS3588.57%0.380.340.08
Fish Speech S2 Pro3086.67%0.410.380.09
CosyVoice33086.67%0.400.410.08
Chatterbox-Turbo2584.00%0.440.420.10
Dia2085.00%0.450.430.11

By cue category (N=20N=20 per category):

Cue CategoryNSequence Agreement (%)Naturalness MAERealism MAESpeaker Sim. MAE
Cough2090.00%0.370.350.08
Throat clearing2085.00%0.410.380.09
Sigh2085.00%0.390.390.08
Pain-related groan2085.00%0.420.400.09
Sneeze2090.00%0.380.340.08
Retch2085.00%0.440.410.09
Hiccup2085.00%0.400.380.08

Across the audited samples, Qwen3-Omni shows substantial correspondence with human judgments. Exact sequence agreement ranges from 84.0% to 88.6% across TTS models and from 85.0% to 90.0% across cue categories. Perceptual-rating MAEs also remain relatively low across the evaluated subgroups. These results further support the reliability of Qwen3-Omni as an automatic evaluator.

Q2: Generalization to Unseen Speakers

To examine transferability beyond the original recording pool, we conducted an additional evaluation using four reference speakers (two male and two female) outside the SymCue recording pool and HealthTTS fine-tuning data. The evaluation covered 126 requests across all seven cue types, with three generation seeds, yielding 378 utterances.

Speaker PoolCue Match (%)Cue Seq. Acc. (%)WER (%)Spk. Sim.Speech Nat.Cue Realism
Seen speakers (full SymCueBench)83.8779.417.330.874.134.84
Four additional speakers82.5477.577.480.813.864.80
Relative Change (%)↓1.59%↓2.32%↑2.05%↓6.90%↓6.54%↓0.83%

Relative changes are calculated with respect to the seen-speaker results. An increase in WER indicates degradation.

The results show that HealthTTS maintains relatively consistent performance on the four additional speakers, with relative changes below 2.5% in Cue Match, Cue Sequence Accuracy, WER, and Cue Realism. Although Speaker Similarity and Speech Naturalness show somewhat larger changes, the overall results suggest that HealthTTS generalizes reasonably well to reference voices outside the original recording pool.

Q3: Cue Insertion Baseline and Comparison Fairness

To clarify the baseline construction and comparison fairness, we provide a detailed comparison between the Cue Insertion evaluation procedure and the SymCue training-data construction pipeline.(Appendix D.1).

ProcessingSymCue ConstructionCue Insertion Baseline
Pause adjustmentAdaptive pauses (0.1–0.8 s)Up to 80 ms when necessary
Cross-fading40–65 ms40 ms
Room toneMatching and adjustmentLocal loudness normalization
Quality controlMulti-round verification and reconstructionFixed insertion pipeline

SymCue construction applies more extensive acoustic tuning to produce high-quality paired training data, whereas Cue Insertion represents a practical post-hoc editing baseline. Both use recorded symptom cues, but their base speech and acoustic processing procedures differ.

Cue Insertion achieves 95.71% symptom-cue recognition and 4.62 Cue Realism, but substantially lower Speech Naturalness (1.55 vs. 4.13 for HealthTTS). These results indicate that, under the evaluated settings, HealthTTS achieves more natural integration of symptom cues into surrounding speech. We therefore interpret Cue Insertion as a practical baseline rather than an upper bound on post-hoc editing.

Clarification of Design Choices

We further clarify the rationale behind the three design choices raised by the reviewer.

  • Seven cue categories: We select seven representative transient, non-lexical vocalizations relevant to patient simulation, covering respiratory cues, throat/physiological cues, and distress-related vocalizations. These categories capture representative symptom manifestations within the scope of this study.
  • Qwen3-TTS for data construction: It provides speaker-conditioned base-speech synthesis for constructing paired cue-free and cue-conditioned utterances. Its zero-shot voice cloning and speech generation capabilities support the construction of consistent base speech [6].
  • CosyVoice3 as the backbone: Its reference-conditioned generation and separate autoregressive language-model and Flow Matching modules support our sequential adaptation of cue-sequence generation and acoustic realization [4].

We thank the reviewer again for these valuable suggestions, which have helped us clarify our model selection, experimental design, and the scope of our contributions.

References

[4] Du, Z., et al. CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training. 2025.

[5] Xu, J., et al. Qwen3-Omni Technical Report. arXiv:2509.17765, 2025.

[6] Hu, H., et al. Qwen3-TTS Technical Report. arXiv:2601.15621, 2026.

Thank you for answering my questions! I took a look at the paper to see if the information presented in the answers was included in a newer version of the paper, but it was not (unless I missed something). I think it would be good to include this in the paper (at least as appendices). I will keep my score.

Thank you for pointing this out. We have uploaded a revised version of the manuscript that incorporates the additional details and experimental results discussed in our rebuttal responses, including the unseen-speaker evaluation, annotation agreement analysis, stage-wise comparisons, and additional implementation details. These have been included in the main text and appendices where appropriate.

We appreciate your helpful suggestion, which has helped us improve the completeness and clarity of the manuscript.

Reviewer_GsMSOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies speech generation with symptom-related acoustic cues, such as cough, throat clearing. This paper introduces SymCue and SymCueBench for training and evaluting control over cue types, counts and order. Based on the CosyVoice3 model, HealthTTS combines cue-weighted language-model adaption, acoustic adaption and preference optimization. the authors reported improve cue-sequence accuracy across different settings.

Critical strengths

The paper is well-written and easy to follow. The research question is clearly stated. Authors provide paired dataset and provide comprehensive experiments and ablations to support the claim.

Critical weaknesses
  1. The method novelty and contribution remains limited. It reads like the framework mainly combines weighted versions of existing language model and fow-matching losses with standard DPO. Alghough the individual ablation shows their effectiveness, but the intuition and novelty behind the framework is limited and not clearly justified.
  2. The evaluation depends heavily on the same model used to construct the supervision. Generalization is evaluated under a restricted setting. For the ablation study, if just use symptom-only, what's the performance will be?
Questions for authors
  1. How does HealthTTS compare with ordinary CosyVoice3 fine-tuning on the same data and control tokens, with both cue weights set to zero and without DPO?
  2. Why are language-model and acoustic adaptation performed sequentially?
4 comments

Dear Reviewer GsMS,

Thank you for your constructive feedback. We address your questions and concerns below.

Q1: Comparison with Ordinary CosyVoice3 Fine-Tuning

To isolate the contributions of cue weighting and preference refinement, we compare ordinary fine-tuning with successive HealthTTS training stages under the same data and evaluation settings. The vanilla baseline uses α=β=0\alpha=\beta=0 without DPO.

CheckpointLM Cue WeightingFlow Cue WeightingDPOCue Match (%)Cue Seq. Acc. (%)Speech Nat.Cue RealismWER (%)
Stage 0: Base CosyVoice3N/AN/AN/A33.6428.312.95–7.93
Vanilla Fine-tuning✗✗✗71.4558.623.653.927.58
Stage I: LM Adaptation✓✗✗77.2066.853.824.157.51
Stage II: Flow Adaptation✓✓✗79.1173.554.104.557.39
Stage III: Full HealthTTS✓✓✓83.8779.414.134.847.33

The two cue-weighted adaptation stages improve Cue Sequence Accuracy by 14.93 percentage points over vanilla fine-tuning (73.55% vs. 58.62%), while DPO provides a further 5.86-point improvement (79.41% vs. 73.55%). These results demonstrate the complementary contributions of weighted adaptation and preference refinement.

Q2: Rationale for Sequential Training

Following established designs in controllable and emotion-aware TTS systems [1–3], we adopt CosyVoice3's hybrid architecture, which combines an autoregressive language model (LM) with a Flow Matching module [4].

This architecture is particularly suitable for symptom-cue generation, which involves both discrete structural control (cue types, counts, order, and position) and continuous acoustic realization. The LM handles structured cue generation, while the Flow module models localized acoustic characteristics of symptoms (e.g., cough bursts and breathy sighs) and their integration into surrounding speech.

Accordingly, we adopt sequential training: Stage I adapts the LM for structured cue generation, Stage II refines acoustic realization through localized Flow adaptation, and Stage III applies preference optimization to complete utterances for global refinement. This design aligns the training objectives with the complementary roles of the two modules.

References

[1] Wu, H., et al. Laugh Now Cry Later: Controlling Time-Varying Emotional States of Flow-Matching-Based Zero-Shot Text-to-Speech. IEEE Spoken Language Technology Workshop (SLT), 2024.

[2] Jeong, J., et al. TTS-CtrlNet: Time-Varying Emotion Aligned Text-to-Speech Generation with ControlNet. arXiv preprint arXiv:2507.04349, 2025.

[3] Wang, S., et al. CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering. International Conference on Machine Learning (ICML), 2026.

[4] Du, Z., et al. CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training. arXiv preprint arXiv:2505.17589, 2025.

W1: Contribution and Design Rationale

Our work is motivated by the specific requirements of clinical patient simulation. Unlike conventional TTS, which primarily focuses on lexical content and speech naturalness, virtual patients must also produce clinically meaningful non-verbal symptom cues with explicit control over their types, counts, order, and positions.

These requirements introduce two major challenges for existing TTS systems: (1) cue-sequence non-compliance, including cue omission and ordering errors due to the sparse occurrence of non-lexical events; and (2) unnatural acoustic integration, as transient symptom sounds must be seamlessly integrated into surrounding speech without compromising naturalness.

To address these challenges, we develop HealthTTS as a unified task-oriented framework. Specifically, cue-weighted LM adaptation improves structured cue generation, localized Flow weighting enhances acoustic realization within symptom regions, and DPO further refines complete utterances through preference optimization. Together with paired cue-free and cue-conditioned supervision and SymCueBench, these components address both controllability and perceptual quality. Our component ablations further demonstrate their complementary contributions.

W2: Evaluation Dependence, Generalization, and Symptom-Only Training

1. Evaluator dependence.

We acknowledge that Qwen3-Omni is used for data verification, temporal annotation, preference construction, and automatic evaluation. During evaluation, it receives generated audio and fixed instructions without access to the requested cue configuration or generation identity.

To assess evaluator reliability, two medical students independently annotated 140 utterances. We compare human–human agreement with human–Omni correspondence using the same metrics:

DimensionHuman–Human AgreementHuman–Omni Agreement
Cue Sequence (Exact Match %)89.29%86.79%
Speech Naturalness (Pearson's rr)0.820.77
Cue Realism (Pearson's rr)0.840.81
Speaker Similarity (Pearson's rr)0.830.80

Qwen3-Omni achieves correspondence with human judgments close to the agreement observed between human annotators, supporting its use as an automatic evaluator.

2. Speaker generalization.

We evaluated HealthTTS on four additional reference speakers (two male and two female) outside the original recording pool and fine-tuning data. The evaluation covered 126 requests across all seven cue types, with three generation seeds, yielding 378 utterances.

Compared with the full SymCueBench results on seen speakers, the additional-speaker evaluation achieves the following:

MetricSeen SpeakersFour Additional Speakers
Cue Match (%)83.8782.54
Cue Sequence Accuracy (%)79.4177.57
WER (%)7.337.48
Speaker Similarity0.870.81
Speech Naturalness4.133.86
Cue Realism4.844.80

These results show relatively consistent cue controllability and realism on the four additional speakers, suggesting that HealthTTS generalizes reasonably well to reference voices outside the original recording pool.

Thank you again for your valuable suggestions.

Thanks authors for providing extra details and explanations. I will keep my score.

Thank you for your thoughtful evaluation and constructive feedback. We sincerely appreciate the time and effort you invested in reviewing our work. Your comments have been very helpful in improving the quality and clarity of our manuscript, and we will carefully address your suggestions in the revised version.

Reviewer_AtQKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HealthTTS, a symptom-aware text-to-speech framework for generating speech with symptom-related acoustic cues, such as coughing, throat clearing, and sighs. The authors formulate symptom-aware TTS as a structured generation task involving control over cue type, count, order, and position. They construct the SymCue dataset and SymCueBench benchmark and introduce a three-stage training strategy based on CosyVoice3, comprising cue-aware language-model adaptation, localized acoustic adaptation, and preference refinement. Experiments demonstrate improvements in cue controllability and speech naturalness over existing TTS baselines, while a virtual standardized patient study shows higher perceived patient realism and usefulness compared with post-hoc cue insertion.

Critical strengths
  1. The paper formulates symptom-aware TTS as a structured speech generation task, enabling explicit control over symptom-related acoustic cues in terms of type, count, order, and position. This provides a meaningful extension of controllable TTS toward healthcare applications.

  2. The authors construct SymCue and SymCueBench with explicit symptom cue annotations, providing a systematic framework for training and evaluating cue controllability. These resources offer a useful benchmark for future research on symptom-aware speech generation.

Critical weaknesses

Weakness 1: SymCue relies on synthesized speech and recorded symptom cues from only six speakers, leaving the realism of generated patient speech and generalization to unseen speakers insufficiently validated.

Weakness 2: The evaluation relies heavily on Qwen3-Omni, while the limited human validation and lack of inter-annotator agreement statistics leave some uncertainty about evaluation reliability.

Weakness 4: While the component-level ablations are informative, reporting intermediate-stage performance could further clarify the incremental benefits of the three-stage framework.

Weakness 4: Although HealthTTS improves symptom cue recognition and perceived patient realism in VSP scenarios, its impact on actual medical training outcomes and clinical communication remains unverified.

Questions for authors

Question 1: When the reference audio comes from speakers outside the original symptom-cue recording pool, can HealthTTS still generate natural symptom-aware speech while preserving the reference speaker's voice characteristics?

Question 2: Could the authors report inter-annotator agreement and further validate Qwen3-Omni's consistency with human judgments across different cue types and TTS models?

Question 3: Could the authors provide additional stage-wise results to clarify the contribution of each training stage beyond the existing ablation studies?

Question 4: Could the authors provide task-oriented evidence of HealthTTS's practical benefits in medical training or explore its potential for downstream medical audio understanding tasks?

4 comments

Dear Reviewer AtQK,

Thank you for your constructive comments and positive evaluation. We deeply appreciate your insightful questions regarding generalization, evaluator reliability, stage-wise progression, and downstream impact. We address each question directly below.

Q1: Generalization to Speakers Outside the Recording Pool

To examine transferability, we evaluated HealthTTS using four additional reference speakers (two male and two female) outside the original SymCue recording pool and HealthTTS fine-tuning data. The evaluation covered 126 requests across all seven cue types, with three generation seeds, yielding 378 utterances.

Speaker PoolCue Match (%)Cue Seq. Acc. (%)WER (%)Spk. Sim.Speech Nat.Cue Realism
Seen speakers (full SymCueBench)83.8779.417.330.874.134.84
Four additional speakers82.5477.577.480.813.864.80
Relative Change (%)↓1.59%↓2.32%↑2.05%↓6.90%↓6.54%↓0.83%

The results show that HealthTTS maintains relatively consistent performance on the four additional speakers, with relative changes below 2.5% in Cue Match, Cue Sequence Accuracy, WER, and Cue Realism. Although Speaker Similarity and Speech Naturalness show slightly larger changes, the overall results suggest that HealthTTS generalizes reasonably well to reference voices outside the original recording pool.

Q2: Inter-Annotator Agreement and Human–Omni Validation

We extend the human audit of 140 utterances (Appendix D.3) to examine inter-annotator reliability and Human–Omni correspondence under a unified evaluation framework.

Inter-annotator agreement. Two medical students independently evaluated the 140 utterances before consensus resolution.

Human–Omni correspondence. Qwen3-Omni's blinded predictions are compared against consensus human cue sequences and averaged human perceptual ratings.

To enable direct comparison, we evaluate both human–human and human–Omni consistency using the same metrics:

DimensionHuman–Human AgreementHuman–Omni Agreement
Cue Sequence (Exact Match %)89.29%86.79%
Speech Naturalness (Pearson's rr)0.820.77
Cue Realism (Pearson's rr)0.840.81
Speaker Similarity (Pearson's rr)0.830.80

The results show that Qwen3-Omni achieves correspondence with human judgments close to the agreement observed between the two human annotators, supporting its use as an automatic evaluator on the audited samples.

Fine-grained subgroup analysis.

We further examine Human–Omni agreement across TTS systems and cue categories using exact cue-sequence agreement and Mean Absolute Error (MAE). MAE is calculated on the original 1–5 scales for Speech Naturalness and Cue Realism, and the normalized 0–1 scale for Speaker Similarity.

1. Stratification by TTS model (N=140N=140):

ModelNNSequence Agreement (%)Naturalness MAERealism MAESpeaker Sim. MAE
HealthTTS3588.57%0.380.340.08
Fish Speech S2 Pro3086.67%0.410.380.09
CosyVoice33086.67%0.400.410.08
Chatterbox-Turbo2584.00%0.440.420.10
Dia2085.00%0.450.430.11

2. Stratification by cue category (N=20N=20 per category):

Cue CategoryNNSequence Agreement (%)Naturalness MAERealism MAESpeaker Sim. MAE
Cough2090.00%0.370.350.08
Throat clearing2085.00%0.410.380.09
Sigh2085.00%0.390.390.08
Pain-related groan2085.00%0.420.400.09
Sneeze2090.00%0.380.340.08
Retch2085.00%0.440.410.09
Hiccup2085.00%0.400.380.08

Across the audited samples, Qwen3-Omni shows substantial correspondence with human judgments, with overall agreement close to that observed between human annotators. Subgroup analyses further demonstrate consistent performance across TTS models, with exact sequence agreement ranging from 84.0% to 88.6%, and across cue categories, ranging from 85.0% to 90.0%. Perceptual-rating MAEs also remain relatively low across the evaluated subgroups. These results further support the reliability of Qwen3-Omni as an automatic evaluator. We will incorporate these analyses and associated uncertainty intervals into Appendix D.3.

Q3: Cumulative Stage-Wise Results

To complement the component ablations, we report cumulative performance across successive training checkpoints on SymCueBench (full-cue setting, three generation seeds). We additionally include vanilla fine-tuning without cue weighting or DPO to distinguish the contributions of standard adaptation and our proposed training stages.

CheckpointLM Cue WeightingFlow Cue WeightingDPOCue Match (%)Cue Seq. Acc. (%)Speech Nat.Cue RealismWER (%)
Stage 0: Base CosyVoice3N/AN/AN/A33.6428.312.95–7.93
Vanilla Fine-tuning✗✗✗71.4558.623.653.927.58
Stage I: LM Adaptation✓✗✗77.2066.853.824.157.51
Stage II: Flow Adaptation✓✓✗79.1173.554.104.557.39
Stage III: Full HealthTTS✓✓✓83.8779.414.134.847.33

Stage I substantially improves cue-sequence control, while Stage II further enhances acoustic quality. Compared with vanilla fine-tuning, the two cue-weighted adaptation stages improve Cue Sequence Accuracy by 14.93 percentage points (73.55% vs. 58.62%). Stage III provides a further 5.86-point improvement through preference refinement (79.41% vs. 73.55%). These results demonstrate the complementary contributions of cue weighting and preference-based refinement.

Q4: Practical Utility in VSP Simulation

Our existing VSP study involved 30 medical students across 14 scenarios. HealthTTS achieved 93.57% symptom-cue recognition, compared with 15.00% for CosyVoice3 and 95.71% for Cue Insertion. It also achieved higher Patient Realism (4.40 vs. 3.50) and Perceived Usefulness (4.50 vs. 3.35) than Cue Insertion.

These findings support HealthTTS's potential as an audio component for VSP simulation. However, the current study measures recognition and perceived usefulness rather than actual learning outcomes or clinical communication improvements. We will clarify this limitation and identify outcome-based evaluation as future work.

Thank you again for your valuable suggestions.

Thank for the detailed responses. Most of my concerns have been addressed, and I will therefore maintain my positive rating.

Thank you for your positive assessment and for the time you devoted to reviewing our manuscript. Your constructive feedback has been invaluable in helping us further strengthen our work. We will carefully incorporate your suggestions into the revised version.

Comments on the paper

2 comments
Area_Chair_Rv5wOpenReview ↗

Author-Reviewer Discussion

Dear Reviewers:

Thank you for your thoughtful reviews. The authors have responded to your reviews. Please check their response and reach out to them if you have any further questions. Please act as soon as possible, as the discussion period with the authors closes on Nov 18.

Best regards,

  • AC.
AuthorsOpenReview ↗

General Response to the Area Chair and Reviewers

Dear Area Chair and Reviewers,

We sincerely thank the Area Chair and all reviewers for their thoughtful feedback and constructive suggestions. During the rebuttal, we conducted additional experiments and analyses to address the reviewers' concerns:

  • Speaker Generalization: Evaluated four additional speakers outside the original speaker pool, showing consistent performance.
  • Evaluation Reliability: Added human annotations on 140 utterances and reported Human–Human and Human–Omni agreement, together with fine-grained analyses.
  • Method Validation: Added vanilla fine-tuning comparisons and cumulative stage-wise results, clarifying the contributions of each training component.
  • Clarifications: Further clarified baseline construction, model selection, training design, and evaluation fairness.
  • Practical Utility: Expanded the VSP analysis and clarified the scope and limitations of our current findings.

We are grateful for the reviewers' engagement throughout the discussion. Following our responses and additional analyses, all three reviewers maintained their positive ratings (3/3/3, Weak Accept).

We sincerely appreciate their valuable feedback, which has helped us strengthen the experimental validation and clarity of our work. We will incorporate these additions and clarifications into the revised manuscript.

Thank you again for your time and consideration.

Sincerely,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.